From 1779df2fb3c743a93ddf779162e05ed1347c2c63 Mon Sep 17 00:00:00 2001 From: Roshan Ramani Date: Sun, 18 Oct 2026 20:26:50 +0530 Subject: [PATCH 001/518] docs(vpc): generate README parameters from values.yaml The vpc Makefile never passed -r README.md to cozyvalues-gen, unlike every other app, so its Parameters table was never regenerated. It still described subnets as a map after the switch to a list, and the README example, which also feeds the website reference page, fails values.schema.json. Assisted-by: LLM Signed-off-by: Roshan Ramani --- packages/apps/vpc/Makefile | 2 +- packages/apps/vpc/README.md | 22 +++++++++++++++------- 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/packages/apps/vpc/Makefile b/packages/apps/vpc/Makefile index fce015f3e..6cc55dbed 100644 --- a/packages/apps/vpc/Makefile +++ b/packages/apps/vpc/Makefile @@ -1,7 +1,7 @@ include ../../../hack/package.mk generate: - cozyvalues-gen -m 'vpc' -v values.yaml -s values.schema.json -g ../../../api/apps/v1alpha1/vpc/types.go + cozyvalues-gen -m 'vpc' -v values.yaml -s values.schema.json -r README.md -g ../../../api/apps/v1alpha1/vpc/types.go ../../../hack/update-crd.sh update: diff --git a/packages/apps/vpc/README.md b/packages/apps/vpc/README.md index 890f12d89..f2dd2a5f9 100644 --- a/packages/apps/vpc/README.md +++ b/packages/apps/vpc/README.md @@ -27,10 +27,18 @@ A VM or a pod may be connected to multiple secondary Subnets at once. Each secon ### Common parameters -| Name | Description | Type | Value | -| -------------------- | -------------------------------- | ------------------- | ------- | -| `subnets` | Subnets of a VPC | `map[string]object` | `{...}` | -| `subnets[name].cidr` | Subnet CIDR, e.g. 192.168.0.0/24 | `cidr` | `{}` | +| Name | Description | Type | Value | +| -------------------------- | ----------------------------------------------------------------------------------------------------------------- | ---------- | ----- | +| `subnets` | Subnets of a VPC | `[]object` | `[]` | +| `subnets[i].name` | Subnet name | `string` | `""` | +| `subnets[i].cidr` | IP address range | `string` | `""` | +| `subnets[i].allowSubnets` | Additional source CIDRs admitted into this private subnet, such as remote networks routed in through a gateway VM | `[]string` | `[]` | +| `peers` | VPC peering connections (bidirectional declaration required) | `[]object` | `[]` | +| `peers[i].vpcName` | Logical name of the remote VPC (without "virtualprivatecloud-" prefix) | `string` | `""` | +| `peers[i].tenantNamespace` | Namespace of the remote tenant | `string` | `""` | +| `routes` | Static routes for the VPC | `[]object` | `[]` | +| `routes[i].cidr` | Destination CIDR | `string` | `""` | +| `routes[i].nextHopIP` | Next hop IP address | `string` | `""` | ## Examples @@ -41,10 +49,10 @@ metadata: name: vpc00 spec: subnets: - sub00: + - name: sub00 cidr: 172.16.0.0/24 - sub01: + - name: sub01 cidr: 172.16.1.0/24 - sub02: + - name: sub02 cidr: 172.16.2.0/24 ``` From a2c9c07a1629eeb78e7cd454753e0e864cbf8a29 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 18 Oct 2026 22:56:06 +0300 Subject: [PATCH 002/518] fix(apps): keep Job and CronJob names within the Kubernetes limits The apps API admits release names of up to 53 characters, and several charts name a Job -. Kubernetes copies a Job name into the job-name label of its pods, and a label value takes at most 63 characters, so the apiserver refused these Jobs for the longest names. A Kafka whose name has 42 to 44 characters could not be installed, a VMDisk of 37 or more could not grow, a VMInstance of 40 or 41 could not take an instance type, profile or external change, and a Bucket of 44 or more or a Qdrant of 42 or more failed its post-delete hook. Those Job names, and the pod label vm-instance copies from one, are now cut to 63 characters. A release name is never longer than 53, so only the fixed suffix is cut and two releases still get two Jobs. A CronJob name is refused over 52 characters, because the controller names each Job it starts -. With the backup CronJob on, a MariaDB whose name has 38 characters or more and a ClickHouse of 35 or more failed the whole release. A cut there reaches into the release name and could give two releases one CronJob, so a name that does not fit becomes the release name cut to 38 characters, a hash of the full release name and -backup, the way the ClickHouse Keeper name is already shortened. The tenant cleanup Job is named after the tenant namespace, which a nested tenant can fill to 63 characters by itself. The Job runs in cozy-system next to that of every other tenant, where a plain cut could give a parent and its child one name and let the child's before-hook-creation delete the parent's Job while it runs, so it is shortened with a hash of the namespace the same way. Names that fit render exactly as before, and a name that did not fit never produced an object, so nothing is renamed on upgrade. The hook ServiceAccounts, Roles and RoleBindings have no such limit and keep their names. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../bucket/templates/hooks/cleanup-acme.yaml | 2 +- .../bucket/tests/job_name_budget_test.yaml | 80 +++++++++++ .../clickhouse/templates/backup-cronjob.yaml | 10 +- .../tests/cronjob_name_budget_test.yaml | 94 +++++++++++++ .../apps/kafka/templates/migration-hook.yaml | 2 +- .../kafka/tests/job_name_budget_test.yaml | 66 +++++++++ .../mariadb/templates/backup-cronjob.yaml | 10 +- .../tests/cronjob_name_budget_test.yaml | 92 +++++++++++++ .../qdrant/templates/hooks/cleanup-pvc.yaml | 2 +- .../qdrant/tests/job_name_budget_test.yaml | 64 +++++++++ .../apps/tenant/templates/cleanup-job.yaml | 15 ++- .../tenant/tests/job_name_budget_test.yaml | 127 ++++++++++++++++++ .../vm-disk/templates/pvc-resize-hook.yaml | 2 +- .../vm-disk/tests/job_name_budget_test.yaml | 127 ++++++++++++++++++ .../vm-instance/templates/vm-update-hook.yaml | 5 +- .../tests/job_name_budget_test.yaml | 107 +++++++++++++++ 16 files changed, 795 insertions(+), 10 deletions(-) create mode 100644 packages/apps/bucket/tests/job_name_budget_test.yaml create mode 100644 packages/apps/clickhouse/tests/cronjob_name_budget_test.yaml create mode 100644 packages/apps/kafka/tests/job_name_budget_test.yaml create mode 100644 packages/apps/mariadb/tests/cronjob_name_budget_test.yaml create mode 100644 packages/apps/qdrant/tests/job_name_budget_test.yaml create mode 100644 packages/apps/tenant/tests/job_name_budget_test.yaml create mode 100644 packages/apps/vm-disk/tests/job_name_budget_test.yaml create mode 100644 packages/apps/vm-instance/tests/job_name_budget_test.yaml diff --git a/packages/apps/bucket/templates/hooks/cleanup-acme.yaml b/packages/apps/bucket/templates/hooks/cleanup-acme.yaml index 94814a2c1..7680df52e 100644 --- a/packages/apps/bucket/templates/hooks/cleanup-acme.yaml +++ b/packages/apps/bucket/templates/hooks/cleanup-acme.yaml @@ -2,7 +2,7 @@ apiVersion: batch/v1 kind: Job metadata: - name: {{ .Release.Name }}-cleanup-acme + name: {{ printf "%s-cleanup-acme" .Release.Name | trunc 63 | trimSuffix "-" }} labels: app.kubernetes.io/instance: {{ .Release.Name }} annotations: diff --git a/packages/apps/bucket/tests/job_name_budget_test.yaml b/packages/apps/bucket/tests/job_name_budget_test.yaml new file mode 100644 index 000000000..94a295b5a --- /dev/null +++ b/packages/apps/bucket/tests/job_name_budget_test.yaml @@ -0,0 +1,80 @@ +suite: bucket cleanup Job name budget + +# The API server copies a Job's name into the job-name label of its pods, and a +# label value takes 63 characters at most, while a release name may take 53. +# The ServiceAccount, Role and RoleBinding have no such limit and keep the full +# name, so a hook that ran before keeps the same objects. + +templates: + - templates/hooks/cleanup-acme.yaml + +set: + _namespace: + host: example.org + +tests: + - it: keeps -cleanup-acme for a short release name + release: + name: my-bucket + namespace: tenant-root + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: my-bucket-cleanup-acme + + - it: keeps the name when it lands exactly on 63 characters + release: + name: bucket-abcdefghijklmnopqrstuvwxyz0123456789abcdefg + namespace: tenant-root + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: bucket-abcdefghijklmnopqrstuvwxyz0123456789abcdefg-cleanup-acme + + - it: cuts the name to 63 characters for a 53-character release name + release: + name: bucket-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij + namespace: tenant-root + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: bucket-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij-cleanup-a + - matchRegex: + path: metadata.name + pattern: ^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$ + - equal: + path: spec.template.spec.serviceAccountName + value: bucket-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij-cleanup-acme + + - it: leaves the RBAC objects on the full name for a 53-character release name + release: + name: bucket-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij + namespace: tenant-root + asserts: + - equal: + path: metadata.name + value: bucket-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij-cleanup-acme + documentSelector: + path: kind + value: ServiceAccount + - equal: + path: roleRef.name + value: bucket-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij-cleanup-acme + documentSelector: + path: kind + value: RoleBinding + - equal: + path: subjects[0].name + value: bucket-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij-cleanup-acme + documentSelector: + path: kind + value: RoleBinding diff --git a/packages/apps/clickhouse/templates/backup-cronjob.yaml b/packages/apps/clickhouse/templates/backup-cronjob.yaml index f792e8f9d..2a0bbd0b2 100644 --- a/packages/apps/clickhouse/templates/backup-cronjob.yaml +++ b/packages/apps/clickhouse/templates/backup-cronjob.yaml @@ -18,11 +18,19 @@ */}} {{- if and .Values.backup.enabled .Values.backup.schedule (not .Values.backup.useSystemBucket) }} {{ $image := .Files.Get "images/backup.json" | fromJson }} +{{- /* The CronJob controller starts each Job as -, so the API + refuses a CronJob name over 52 characters. Cutting the release name to + fit could give two releases one CronJob, so a name that does not fit + keeps a hash of the full release name instead. */}} +{{- $cronJobName := printf "%s-backup" .Release.Name }} +{{- if gt (len $cronJobName) 52 }} +{{- $cronJobName = printf "%s-%s-backup" (.Release.Name | trunc 38 | trimSuffix "-") (.Release.Name | sha256sum | trunc 6) }} +{{- end }} apiVersion: batch/v1 kind: CronJob metadata: - name: {{ .Release.Name }}-backup + name: {{ $cronJobName }} spec: schedule: "{{ .Values.backup.schedule }}" concurrencyPolicy: Forbid diff --git a/packages/apps/clickhouse/tests/cronjob_name_budget_test.yaml b/packages/apps/clickhouse/tests/cronjob_name_budget_test.yaml new file mode 100644 index 000000000..0ff3e2305 --- /dev/null +++ b/packages/apps/clickhouse/tests/cronjob_name_budget_test.yaml @@ -0,0 +1,94 @@ +suite: clickhouse legacy backup CronJob name budget + +# The API refuses a CronJob name over 52 characters, because the controller +# names each Job it starts -, while a release name may take 53. +# Cutting the release name alone would give two long names one CronJob, so a +# name that does not fit is the release name cut to 38 characters, a hash of +# the full release name and -backup. + +templates: + - templates/backup-cronjob.yaml + - templates/backup-script.yaml + - templates/backup-secret.yaml + +set: + _cluster: {} + backup: + enabled: true + schedule: "0 2 * * *" + +tests: + - it: keeps -backup for a short release name + release: + name: ch-test + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: ch-test-backup + + - it: keeps the name when it lands exactly on 52 characters + release: + name: clickhouse-abcdefghijklmnopqrstuvwxyz01234567 + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: clickhouse-abcdefghijklmnopqrstuvwxyz01234567-backup + + - it: shortens the name one character past the budget + release: + name: clickhouse-abcdefghijklmnopqrstuvwxyz012345678 + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: clickhouse-abcdefghijklmnopqrstuvwxyz0-bec95e-backup + + - it: fits a 53-character release name into 52 characters + release: + name: clickhouse-abcdefghijklmnopqrstuvwxyz0123456789abcdef + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: clickhouse-abcdefghijklmnopqrstuvwxyz0-3e8cf9-backup + - matchRegex: + path: metadata.name + pattern: ^[a-z0-9]([-a-z0-9]{0,50}[a-z0-9])?$ + + - it: gives a release name differing only in its last character another CronJob + release: + name: clickhouse-abcdefghijklmnopqrstuvwxyz0123456789abcdex + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: clickhouse-abcdefghijklmnopqrstuvwxyz0-bd3283-backup + + - it: does not end the cut release name with a dash + release: + name: clickhouse-abcdefghijklmnopqrstuvwxyz-0123456789 + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: clickhouse-abcdefghijklmnopqrstuvwxyz-07ba9e-backup diff --git a/packages/apps/kafka/templates/migration-hook.yaml b/packages/apps/kafka/templates/migration-hook.yaml index 9877f699b..997b28ea1 100644 --- a/packages/apps/kafka/templates/migration-hook.yaml +++ b/packages/apps/kafka/templates/migration-hook.yaml @@ -79,7 +79,7 @@ subjects: apiVersion: batch/v1 kind: Job metadata: - name: {{ .Release.Name }}-kafka-migration + name: {{ printf "%s-kafka-migration" .Release.Name | trunc 63 | trimSuffix "-" }} annotations: helm.sh/hook: pre-install,pre-upgrade helm.sh/hook-weight: "2" diff --git a/packages/apps/kafka/tests/job_name_budget_test.yaml b/packages/apps/kafka/tests/job_name_budget_test.yaml new file mode 100644 index 000000000..908ef75ed --- /dev/null +++ b/packages/apps/kafka/tests/job_name_budget_test.yaml @@ -0,0 +1,66 @@ +suite: kafka migration Job name budget + +# The API server copies a Job's name into the job-name label of its pods, and a +# label value takes 63 characters at most. The Kafka name cap leaves a release +# name of up to 50 characters, and the migration hook runs on install, so an +# overlong Job name would fail every install of the longest names. The +# ServiceAccount, Role and RoleBinding have no such limit and keep the full +# name, so the ones an earlier attempt left behind are replaced, not orphaned. + +templates: + - templates/migration-hook.yaml + +tests: + - it: keeps -kafka-migration for a short release name + release: + name: test-kafka + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: test-kafka-kafka-migration + + - it: keeps the name when it lands exactly on 63 characters + release: + name: kafka-abcdefghijklmnopqrstuvwxyz0123456789abcde + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: kafka-abcdefghijklmnopqrstuvwxyz0123456789abcde-kafka-migration + + - it: cuts the name to 63 characters for the longest release name the cap admits + release: + name: kafka-abcdefghijklmnopqrstuvwxyz0123456789abcdefgh + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: kafka-abcdefghijklmnopqrstuvwxyz0123456789abcdefgh-kafka-migrat + - matchRegex: + path: metadata.name + pattern: ^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$ + - equal: + path: spec.template.spec.serviceAccountName + value: kafka-abcdefghijklmnopqrstuvwxyz0123456789abcdefgh-kafka-migration + + - it: leaves the ServiceAccount on the full name for the longest release name + release: + name: kafka-abcdefghijklmnopqrstuvwxyz0123456789abcdefgh + namespace: tenant-test + documentSelector: + path: kind + value: ServiceAccount + asserts: + - equal: + path: metadata.name + value: kafka-abcdefghijklmnopqrstuvwxyz0123456789abcdefgh-kafka-migration diff --git a/packages/apps/mariadb/templates/backup-cronjob.yaml b/packages/apps/mariadb/templates/backup-cronjob.yaml index d39d5a8dd..1072a7c85 100644 --- a/packages/apps/mariadb/templates/backup-cronjob.yaml +++ b/packages/apps/mariadb/templates/backup-cronjob.yaml @@ -1,10 +1,18 @@ {{- if .Values.backup.enabled }} {{ $image := .Files.Get "images/backup.json" | fromJson }} +{{- /* The CronJob controller starts each Job as -, so the API + refuses a CronJob name over 52 characters. Cutting the release name to + fit could give two releases one CronJob, so a name that does not fit + keeps a hash of the full release name instead. */}} +{{- $cronJobName := printf "%s-backup" .Release.Name }} +{{- if gt (len $cronJobName) 52 }} +{{- $cronJobName = printf "%s-%s-backup" (.Release.Name | trunc 38 | trimSuffix "-") (.Release.Name | sha256sum | trunc 6) }} +{{- end }} apiVersion: batch/v1 kind: CronJob metadata: - name: {{ .Release.Name }}-backup + name: {{ $cronJobName }} spec: schedule: "{{ .Values.backup.schedule }}" concurrencyPolicy: Forbid diff --git a/packages/apps/mariadb/tests/cronjob_name_budget_test.yaml b/packages/apps/mariadb/tests/cronjob_name_budget_test.yaml new file mode 100644 index 000000000..18e5e40cf --- /dev/null +++ b/packages/apps/mariadb/tests/cronjob_name_budget_test.yaml @@ -0,0 +1,92 @@ +suite: mariadb backup CronJob name budget + +# The API refuses a CronJob name over 52 characters, because the controller +# names each Job it starts -, while a release name may take 53. +# Cutting the release name alone would give two long names one CronJob, so a +# name that does not fit is the release name cut to 38 characters, a hash of +# the full release name and -backup. + +templates: + - templates/backup-cronjob.yaml + - templates/backup-script.yaml + - templates/backup-secret.yaml + +set: + backup: + enabled: true + +tests: + - it: keeps -backup for a short release name + release: + name: mariadb-test + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: mariadb-test-backup + + - it: keeps the name when it lands exactly on 52 characters + release: + name: mariadb-abcdefghijklmnopqrstuvwxyz0123456789a + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: mariadb-abcdefghijklmnopqrstuvwxyz0123456789a-backup + + - it: shortens the name one character past the budget + release: + name: mariadb-abcdefghijklmnopqrstuvwxyz0123456789ab + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: mariadb-abcdefghijklmnopqrstuvwxyz0123-57a159-backup + + - it: fits a 53-character release name into 52 characters + release: + name: mariadb-abcdefghijklmnopqrstuvwxyz0123456789abcdefghi + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: mariadb-abcdefghijklmnopqrstuvwxyz0123-b5132c-backup + - matchRegex: + path: metadata.name + pattern: ^[a-z0-9]([-a-z0-9]{0,50}[a-z0-9])?$ + + - it: gives a release name differing only in its last character another CronJob + release: + name: mariadb-abcdefghijklmnopqrstuvwxyz0123456789abcdefghx + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: mariadb-abcdefghijklmnopqrstuvwxyz0123-a49389-backup + + - it: does not end the cut release name with a dash + release: + name: mariadb-abcdefghijklmnopqrstuvwxyz012-3456789abc + namespace: tenant-test + documentSelector: + path: kind + value: CronJob + asserts: + - equal: + path: metadata.name + value: mariadb-abcdefghijklmnopqrstuvwxyz012-2985e6-backup diff --git a/packages/apps/qdrant/templates/hooks/cleanup-pvc.yaml b/packages/apps/qdrant/templates/hooks/cleanup-pvc.yaml index bbc9547f9..3e2038e91 100644 --- a/packages/apps/qdrant/templates/hooks/cleanup-pvc.yaml +++ b/packages/apps/qdrant/templates/hooks/cleanup-pvc.yaml @@ -2,7 +2,7 @@ apiVersion: batch/v1 kind: Job metadata: - name: {{ .Release.Name }}-qdrant-cleanup + name: {{ printf "%s-qdrant-cleanup" .Release.Name | trunc 63 | trimSuffix "-" }} labels: app.kubernetes.io/instance: {{ .Release.Name }} annotations: diff --git a/packages/apps/qdrant/tests/job_name_budget_test.yaml b/packages/apps/qdrant/tests/job_name_budget_test.yaml new file mode 100644 index 000000000..76416b387 --- /dev/null +++ b/packages/apps/qdrant/tests/job_name_budget_test.yaml @@ -0,0 +1,64 @@ +suite: qdrant cleanup Job name budget + +# The API server copies a Job's name into the job-name label of its pods, and a +# label value takes 63 characters at most, while a release name may take 53. +# The ServiceAccount, Role and RoleBinding have no such limit and keep the full +# name. + +templates: + - templates/hooks/cleanup-pvc.yaml + +tests: + - it: keeps -qdrant-cleanup for a short release name + release: + name: qdrant-test + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: qdrant-test-qdrant-cleanup + + - it: keeps the name when it lands exactly on 63 characters + release: + name: qdrant-abcdefghijklmnopqrstuvwxyz0123456789abcde + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: qdrant-abcdefghijklmnopqrstuvwxyz0123456789abcde-qdrant-cleanup + + - it: cuts the name to 63 characters for a 53-character release name + release: + name: qdrant-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: qdrant-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij-qdrant-cl + - matchRegex: + path: metadata.name + pattern: ^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$ + - equal: + path: spec.template.spec.serviceAccountName + value: qdrant-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij-qdrant-cleanup + + - it: leaves the ServiceAccount on the full name for a 53-character release name + release: + name: qdrant-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij + namespace: tenant-test + documentSelector: + path: kind + value: ServiceAccount + asserts: + - equal: + path: metadata.name + value: qdrant-abcdefghijklmnopqrstuvwxyz0123456789abcdefghij-qdrant-cleanup diff --git a/packages/apps/tenant/templates/cleanup-job.yaml b/packages/apps/tenant/templates/cleanup-job.yaml index ffa07d0fe..7285a4d79 100644 --- a/packages/apps/tenant/templates/cleanup-job.yaml +++ b/packages/apps/tenant/templates/cleanup-job.yaml @@ -40,11 +40,22 @@ subjects: - kind: ServiceAccount name: {{ include "tenant.name" . }}-cleanup namespace: cozy-system +{{- /* The Job name becomes the job-name label of its pod, which takes 63 + characters at most, and a nested tenant namespace can use all 63 by + itself. Cutting the name to fit would let a parent and one of its + children land on the same Job in cozy-system, where the child's + before-hook-creation would delete the parent's running cleanup, so a + name that does not fit keeps a hash of the whole namespace instead. */}} +{{- $tenantName := include "tenant.name" . }} +{{- $jobName := printf "%s-cleanup" $tenantName }} +{{- if gt (len $jobName) 63 }} +{{- $jobName = printf "%s-%s-cleanup" ($tenantName | trunc 48 | trimSuffix "-") ($tenantName | sha256sum | trunc 6) }} +{{- end }} --- apiVersion: batch/v1 kind: Job metadata: - name: {{ include "tenant.name" . }}-cleanup + name: {{ $jobName }} namespace: cozy-system annotations: helm.sh/hook: pre-delete @@ -98,7 +109,7 @@ spec: ttlSecondsAfterFinished: 300 template: metadata: - name: {{ include "tenant.name" . }}-cleanup + name: {{ $jobName }} labels: policy.cozystack.io/allow-to-apiserver: "true" spec: diff --git a/packages/apps/tenant/tests/job_name_budget_test.yaml b/packages/apps/tenant/tests/job_name_budget_test.yaml new file mode 100644 index 000000000..5cf4b93f6 --- /dev/null +++ b/packages/apps/tenant/tests/job_name_budget_test.yaml @@ -0,0 +1,127 @@ +suite: tenant cleanup Job name budget + +# The API server copies a Job's name into the job-name label of its pods, and a +# label value takes 63 characters at most, while a nested tenant namespace may +# take all 63 by itself. Every tenant's cleanup Job runs in cozy-system, where +# cutting the namespace alone could give a parent and its child one Job, and the +# child's before-hook-creation would delete the parent's while it runs. So a +# name that does not fit is the namespace cut to 48 characters, a hash of the +# whole namespace and -cleanup. The ServiceAccount, Role and RoleBinding have no +# such limit and keep the full name. + +templates: + - templates/cleanup-job.yaml + +tests: + - it: keeps -cleanup for a tenant under tenant-root + release: + name: tenant-test + namespace: tenant-root + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: tenant-test-cleanup + + - it: keeps -cleanup for a short nested tenant + release: + name: tenant-child + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: tenant-test-child-cleanup + + - it: keeps the name when it lands exactly on 63 characters + release: + name: tenant-0123456789abcdefghijk + namespace: tenant-abcdefghijklmnopqrstuvwxyz + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcdefghijk-cleanup + + - it: shortens the name for a 63-character namespace + release: + name: tenant-0123456789abcdefghijklmnopqrs + namespace: tenant-abcdefghijklmnopqrstuvwxyz + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcd-7b3e3e-cleanup + - matchRegex: + path: metadata.name + pattern: ^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$ + - equal: + path: spec.template.metadata.name + value: tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcd-7b3e3e-cleanup + - equal: + path: spec.template.spec.serviceAccountName + value: tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcdefghijklmnopqrs-cleanup + + # Cut to 63 characters, the parent below and its child named clea would both + # be tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcdefghijklmn-clea. + - it: gives a parent its own Job name + release: + name: tenant-0123456789abcdefghijklmn + namespace: tenant-abcdefghijklmnopqrstuvwxyz + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcd-571020-cleanup + + - it: gives the child that would collide with that parent another Job name + release: + name: tenant-clea + namespace: tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcdefghijklmn + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcd-ec5e9b-cleanup + + - it: does not end the cut namespace with a dash + release: + name: tenant-abcdefghijklmno + namespace: tenant-abcdefghijklmnopqrstuvwxyz0123456789abcd + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: tenant-abcdefghijklmnopqrstuvwxyz0123456789abcd-477f49-cleanup + + - it: leaves the RBAC objects on the full name for a 63-character namespace + release: + name: tenant-0123456789abcdefghijklmnopqrs + namespace: tenant-abcdefghijklmnopqrstuvwxyz + asserts: + - equal: + path: metadata.name + value: tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcdefghijklmnopqrs-cleanup + documentSelector: + path: kind + value: ServiceAccount + - equal: + path: subjects[0].name + value: tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcdefghijklmnopqrs-cleanup + documentSelector: + path: kind + value: RoleBinding diff --git a/packages/apps/vm-disk/templates/pvc-resize-hook.yaml b/packages/apps/vm-disk/templates/pvc-resize-hook.yaml index 5e327c6f5..224da33ba 100644 --- a/packages/apps/vm-disk/templates/pvc-resize-hook.yaml +++ b/packages/apps/vm-disk/templates/pvc-resize-hook.yaml @@ -15,7 +15,7 @@ apiVersion: batch/v1 kind: Job metadata: - name: "{{ .Release.Name }}-volume-resize-hook" + name: {{ printf "%s-volume-resize-hook" .Release.Name | trunc 63 | trimSuffix "-" | quote }} annotations: helm.sh/hook: pre-install,pre-upgrade helm.sh/hook-weight: "0" diff --git a/packages/apps/vm-disk/tests/job_name_budget_test.yaml b/packages/apps/vm-disk/tests/job_name_budget_test.yaml new file mode 100644 index 000000000..5922dcc7b --- /dev/null +++ b/packages/apps/vm-disk/tests/job_name_budget_test.yaml @@ -0,0 +1,127 @@ +suite: vm-disk resize Job name budget + +# The API server copies a Job's name into the job-name label of its pods, and a +# label value takes 63 characters at most, while a release name may take 53. +# The hook renders only when an existing PVC grows, so each release below gets +# a smaller PVC to look up. The ServiceAccount, Role and RoleBinding have no +# such limit and keep the full name, so the ones an earlier resize left behind +# are replaced, not orphaned. + +templates: + - templates/pvc-resize-hook.yaml + +set: + storage: 2Gi + +kubernetesProvider: + scheme: + "v1/PersistentVolumeClaim": + gvr: + group: "" + version: "v1" + resource: "persistentvolumeclaims" + namespaced: true + objects: + - kind: PersistentVolumeClaim + apiVersion: v1 + metadata: + name: test-vm-disk + namespace: tenant-test + spec: + resources: + requests: + storage: 1Gi + - kind: PersistentVolumeClaim + apiVersion: v1 + metadata: + name: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789 + namespace: tenant-test + spec: + resources: + requests: + storage: 1Gi + - kind: PersistentVolumeClaim + apiVersion: v1 + metadata: + name: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789abcd + namespace: tenant-test + spec: + resources: + requests: + storage: 1Gi + - kind: PersistentVolumeClaim + apiVersion: v1 + metadata: + name: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789abcdefghi + namespace: tenant-test + spec: + resources: + requests: + storage: 1Gi + +tests: + - it: keeps -volume-resize-hook for a short release name + release: + name: test-vm-disk + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: test-vm-disk-volume-resize-hook + + - it: keeps the name when it lands exactly on 63 characters + release: + name: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789 + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789-volume-resize-hook + + - it: does not end the cut name with a dash + release: + name: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789abcd + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789abcd-volume-resize + + - it: cuts the name to 63 characters for a 53-character release name + release: + name: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789abcdefghi + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789abcdefghi-volume-re + - matchRegex: + path: metadata.name + pattern: ^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$ + - equal: + path: spec.template.spec.serviceAccountName + value: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789abcdefghi-volume-resize-hook + + - it: leaves the ServiceAccount on the full name for a 53-character release name + release: + name: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789abcdefghi + namespace: tenant-test + documentSelector: + path: kind + value: ServiceAccount + asserts: + - equal: + path: metadata.name + value: vm-disk-abcdefghijklmnopqrstuvwxyz0123456789abcdefghi-volume-resize-hook diff --git a/packages/apps/vm-instance/templates/vm-update-hook.yaml b/packages/apps/vm-instance/templates/vm-update-hook.yaml index 72bc617e2..ab77f21b1 100644 --- a/packages/apps/vm-instance/templates/vm-update-hook.yaml +++ b/packages/apps/vm-instance/templates/vm-update-hook.yaml @@ -1,5 +1,6 @@ {{- $vmName := include "virtual-machine.fullname" . -}} {{- $namespace := .Release.Namespace -}} +{{- $jobName := printf "%s-update-hook" .Release.Name | trunc 63 | trimSuffix "-" -}} {{- $existingVM := lookup "kubevirt.io/v1" "VirtualMachine" $namespace $vmName -}} {{- $existingService := lookup "v1" "Service" $namespace $vmName -}} @@ -48,7 +49,7 @@ apiVersion: batch/v1 kind: Job metadata: - name: "{{ $.Release.Name }}-update-hook" + name: "{{ $jobName }}" annotations: helm.sh/hook: pre-install,pre-upgrade helm.sh/hook-weight: "0" @@ -57,7 +58,7 @@ spec: template: metadata: labels: - app: "{{ $.Release.Name }}-update-hook" + app: "{{ $jobName }}" policy.cozystack.io/allow-to-apiserver: "true" spec: serviceAccountName: {{ $.Release.Name }}-update-hook diff --git a/packages/apps/vm-instance/tests/job_name_budget_test.yaml b/packages/apps/vm-instance/tests/job_name_budget_test.yaml new file mode 100644 index 000000000..fb4190930 --- /dev/null +++ b/packages/apps/vm-instance/tests/job_name_budget_test.yaml @@ -0,0 +1,107 @@ +suite: vm-instance update Job name budget + +# The API server copies a Job's name into the job-name label of its pods, and a +# label value takes 63 characters at most, while a release name may take 53. +# The pod's own app label carries the same name and is held to the same limit. +# The hook renders only when an existing VM or Service needs reconciling, so +# each release below gets a Service of the wrong type to look up. The +# ServiceAccount, Role and RoleBinding have no such limit and keep the full +# name, so the ones an earlier update left behind are replaced, not orphaned. + +templates: + - templates/vm-update-hook.yaml + +set: + external: false + +kubernetesProvider: + scheme: + "v1/Service": + gvr: + group: "" + version: "v1" + resource: "services" + namespaced: true + objects: + - kind: Service + apiVersion: v1 + metadata: + name: vm-instance-test + namespace: tenant-test + spec: + type: LoadBalancer + - kind: Service + apiVersion: v1 + metadata: + name: vm-instance-abcdefghijklmnopqrstuvwxyz0123456789abc + namespace: tenant-test + spec: + type: LoadBalancer + - kind: Service + apiVersion: v1 + metadata: + name: vm-instance-abcdefghijklmnopqrstuvwxyz0123456789abcde + namespace: tenant-test + spec: + type: LoadBalancer + +tests: + - it: keeps -update-hook for a short release name + release: + name: vm-instance-test + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: vm-instance-test-update-hook + - equal: + path: spec.template.metadata.labels.app + value: vm-instance-test-update-hook + + - it: keeps the name when it lands exactly on 63 characters + release: + name: vm-instance-abcdefghijklmnopqrstuvwxyz0123456789abc + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: vm-instance-abcdefghijklmnopqrstuvwxyz0123456789abc-update-hook + + - it: cuts the Job name and the app label to 63 characters for a 53-character release name + release: + name: vm-instance-abcdefghijklmnopqrstuvwxyz0123456789abcde + namespace: tenant-test + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: vm-instance-abcdefghijklmnopqrstuvwxyz0123456789abcde-update-ho + - matchRegex: + path: metadata.name + pattern: ^[a-z0-9]([-a-z0-9]{0,61}[a-z0-9])?$ + - equal: + path: spec.template.metadata.labels.app + value: vm-instance-abcdefghijklmnopqrstuvwxyz0123456789abcde-update-ho + - equal: + path: spec.template.spec.serviceAccountName + value: vm-instance-abcdefghijklmnopqrstuvwxyz0123456789abcde-update-hook + + - it: leaves the ServiceAccount on the full name for a 53-character release name + release: + name: vm-instance-abcdefghijklmnopqrstuvwxyz0123456789abcde + namespace: tenant-test + documentSelector: + path: kind + value: ServiceAccount + asserts: + - equal: + path: metadata.name + value: vm-instance-abcdefghijklmnopqrstuvwxyz0123456789abcde-update-hook From b74029cb390c79c1c7cf3a1a697e0f6b70d353d1 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 18 Oct 2026 23:11:02 +0300 Subject: [PATCH 003/518] fix(keycloak): apply keycloak-operator CRD updates on upgrade keycloak-operator ships its CRDs in the chart's crds/ directory and its PackageSource sets no upgradeCRDs policy, so helm-controller never replaces them. A cluster installed before v1.2.0 therefore runs the v1.32.0 operator against the v1.25.0 CRDs, whose ClusterKeycloakRealm schema has no spec.sessions, spec.login or spec.organizationsEnabled. The apiserver drops those fields from the realm keycloak-configure renders, so the operator never applies the session limits the platform sets. On a v1.6.4 cluster in that state Keycloak also rejected the dashboard client with HTTP 400, and replacing the CRDs from the chart by hand, with an operator restart, cleared it. The v1.32.0 CRDs are not purely additive. They drop the realm, targetRealm and keycloakOwner fields that v1.25.0 marked deprecated and require realmRef and keycloakRef instead. Every Keycloak object the platform renders already uses the refs, and each CRD keeps its single version, so replacing them loses nothing the platform relies on. helm-controller applies the policy only when it upgrades the release, and it runs an upgrade only when the release artifact or its values change, which a PackageSource edit does not do. The package test added here also changes the keycloak-operator artifact, so the replace runs on the upgrade that ships this change even where nothing else in the package changes. The test itself pins the realm fields keycloak-configure sets against the CRD the chart ships. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../platform/sources/keycloak-operator.yaml | 7 ++++++ .../sources_keycloak_operator_crds_test.yaml | 22 +++++++++++++++++++ packages/system/keycloak-operator/Makefile | 3 +++ .../keycloak-operator/tests/crds_test.yaml | 19 ++++++++++++++++ 4 files changed, 51 insertions(+) create mode 100644 packages/core/platform/tests/sources_keycloak_operator_crds_test.yaml create mode 100644 packages/system/keycloak-operator/tests/crds_test.yaml diff --git a/packages/core/platform/sources/keycloak-operator.yaml b/packages/core/platform/sources/keycloak-operator.yaml index ce3a5f7e0..5b116e5e6 100644 --- a/packages/core/platform/sources/keycloak-operator.yaml +++ b/packages/core/platform/sources/keycloak-operator.yaml @@ -20,3 +20,10 @@ spec: install: namespace: cozy-keycloak releaseName: keycloak-operator + # helm-controller leaves existing CRDs alone on upgrade, so a cluster + # installed before v1.2.0 keeps the v1.25.0 schema and the apiserver + # prunes the realm session settings keycloak-configure sets. v1.32.0 + # drops the realm, targetRealm and keycloakOwner fields deprecated in + # favour of realmRef and keycloakRef. The platform renders only the + # refs, so replace is safe. + upgradeCRDs: CreateReplace diff --git a/packages/core/platform/tests/sources_keycloak_operator_crds_test.yaml b/packages/core/platform/tests/sources_keycloak_operator_crds_test.yaml new file mode 100644 index 000000000..5cccfc757 --- /dev/null +++ b/packages/core/platform/tests/sources_keycloak_operator_crds_test.yaml @@ -0,0 +1,22 @@ +suite: keycloak-operator applies CRD updates on upgrade +# keycloak-configure sets spec.sessions and spec.organizationsEnabled, and +# spec.login when configured, on the cozy ClusterKeycloakRealm. The CRDs of +# keycloak-operator v1.25.0 declare none of them. Helm never upgrades CRDs +# shipped in crds/, so without CreateReplace a cluster installed before v1.2.0 +# keeps those CRDs, the apiserver drops the fields without an error, and the +# operator never applies them to the realm. Pin the policy so an upgrade cannot +# silently regress. +templates: + - templates/sources.yaml +release: + name: cozystack + namespace: cozy-system +tests: + - it: keycloak-operator PackageSource sets upgradeCRDs to CreateReplace + documentSelector: + path: metadata.name + value: cozystack.keycloak-operator + asserts: + - equal: + path: spec.variants[0].components[0].install.upgradeCRDs + value: CreateReplace diff --git a/packages/system/keycloak-operator/Makefile b/packages/system/keycloak-operator/Makefile index e2f7f45d5..303f37eeb 100644 --- a/packages/system/keycloak-operator/Makefile +++ b/packages/system/keycloak-operator/Makefile @@ -4,6 +4,9 @@ export NAMESPACE=cozy-keycloak include ../../../hack/common-envs.mk include ../../../hack/package.mk +test: + helm unittest . + image: docker buildx build images/keycloak-operator \ $(call image-tags,keycloak-operator,v$(COZYSTACK_VERSION)) \ diff --git a/packages/system/keycloak-operator/tests/crds_test.yaml b/packages/system/keycloak-operator/tests/crds_test.yaml new file mode 100644 index 000000000..1be3fe611 --- /dev/null +++ b/packages/system/keycloak-operator/tests/crds_test.yaml @@ -0,0 +1,19 @@ +suite: keycloak-operator CRDs declare the realm settings the platform sets +# keycloak-configure renders the cozy ClusterKeycloakRealm with spec.sessions +# and spec.organizationsEnabled, and spec.login when configured. The apiserver +# drops any field the CRD does not declare without an error, so with a chart +# that lacks them the realm never gets those settings. +includeCrds: true +templates: + - charts/keycloak-operator/crds/v1.edp.epam.com_clusterkeycloakrealms.yaml +tests: + - it: declares the session, organization and login settings + asserts: + - exists: + path: spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.sessions.properties.ssoSessionSettings.properties.idleTimeout + - exists: + path: spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.sessions.properties.ssoSessionSettings.properties.maxLifespan + - exists: + path: spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.organizationsEnabled + - exists: + path: spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.login From 8e96a3e9b73b127a116131327b73925e7c2ffa9a Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 18 Oct 2026 22:59:52 +0300 Subject: [PATCH 004/518] fix(foundationdb): apply foundationdb-operator CRD updates on upgrade The foundationdb-operator PackageSource sets no upgradeCRDs policy, so a cluster installed before v1.6.0 runs the v2.30.0 operator on the v2.13.0 CRDs it first installed. The apiserver prunes every field added since, status.backupDetails.restorable on FoundationDBBackup among them, so a backup never shows as restorable however long it has run. The v2.30.0 CRDs only add fields. They keep the single v1beta2 version and drop or retype nothing, and the one new constraint, a pattern on perpetual_storage_wiggle_locality, sits on a field the platform does not set, so CreateReplace is safe. Nothing in the foundationdb-operator package changes here, so a cluster picks the new CRDs up at the next upgrade of that release, which comes with the next change to the package. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../sources/foundationdb-operator.yaml | 6 ++++++ ...urces_foundationdb_operator_crds_test.yaml | 20 +++++++++++++++++++ 2 files changed, 26 insertions(+) create mode 100644 packages/core/platform/tests/sources_foundationdb_operator_crds_test.yaml diff --git a/packages/core/platform/sources/foundationdb-operator.yaml b/packages/core/platform/sources/foundationdb-operator.yaml index 0af881f60..28328c3d1 100644 --- a/packages/core/platform/sources/foundationdb-operator.yaml +++ b/packages/core/platform/sources/foundationdb-operator.yaml @@ -20,3 +20,9 @@ spec: install: namespace: cozy-foundationdb-operator releaseName: foundationdb-operator + # helm-controller leaves existing CRDs alone on upgrade, so a cluster + # installed before v1.6.0 keeps the v2.13.0 schema under the v2.30.0 + # operator, and the apiserver prunes the fields added since, + # status.backupDetails.restorable among them. The schema diff adds + # fields and drops none, so replace is safe. + upgradeCRDs: CreateReplace diff --git a/packages/core/platform/tests/sources_foundationdb_operator_crds_test.yaml b/packages/core/platform/tests/sources_foundationdb_operator_crds_test.yaml new file mode 100644 index 000000000..a0976ae57 --- /dev/null +++ b/packages/core/platform/tests/sources_foundationdb_operator_crds_test.yaml @@ -0,0 +1,20 @@ +suite: foundationdb-operator applies CRD updates on upgrade +# The fdb-operator v2.30.0 CRDs add fields the v2.13.0 ones lack, among them +# status.backupDetails.restorable on FoundationDBBackup. Helm never upgrades +# CRDs shipped in crds/, so without CreateReplace a cluster installed before +# v1.6.0 keeps the v2.13.0 CRDs and the apiserver prunes those fields on every +# write. Pin the policy so an upgrade cannot silently regress. +templates: + - templates/sources.yaml +release: + name: cozystack + namespace: cozy-system +tests: + - it: foundationdb-operator PackageSource sets upgradeCRDs to CreateReplace + documentSelector: + path: metadata.name + value: cozystack.foundationdb-operator + asserts: + - equal: + path: spec.variants[0].components[0].install.upgradeCRDs + value: CreateReplace From 89028b455ef2fbf4772672c2fe6342150efc3200 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 18 Oct 2026 23:16:45 +0300 Subject: [PATCH 005/518] fix(monitoring): route InfoInhibitor to the blackhole receiver InfoInhibitor exists to be the source side of an Alertmanager inhibit rule and, as its own description says, to go to a null receiver. The platform Alertmanager config has no inhibit rule, and the rule has carried severity major since it was imported alongside Alerta, so Alerta shows it one level below critical. Alerta's Telegram and Slack plugins forward it by default, and a notify_email label inherited from the info alert gets it emailed too. It fires whenever a rule labelled severity="info" is active with no warning or critical alert in the same namespace and cluster. The platform's own rules no longer use info, but rules it does not write still do. CDI generates CDIStorageProfilesIncomplete, KubeVirt generates KubeVirtVMGuestMemoryAvailableLow, and operators add their own. Each of them turns into a major InfoInhibitor in Alerta. none is the severity upstream kube-prometheus gives this rule, and the platform config already sends severity="none" to the blackhole ahead of every route that delivers. The severity contract refuses none because such an alert is never delivered, so InfoInhibitor is named as the one alert for which that is the point. Tests hold both ends, so neither the label nor the route order can drift back to delivering it. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- hack/alert-severity-contract.bats | 89 +++++++++++++++++++ .../alerts/general.rules.yaml | 2 +- .../tests/alertmanager_blackhole_test.yaml | 45 ++++++++++ 3 files changed, 135 insertions(+), 1 deletion(-) create mode 100644 packages/system/monitoring/tests/alertmanager_blackhole_test.yaml diff --git a/hack/alert-severity-contract.bats b/hack/alert-severity-contract.bats index e1315eb96..44f197000 100644 --- a/hack/alert-severity-contract.bats +++ b/hack/alert-severity-contract.bats @@ -9,6 +9,10 @@ load test_helper # scope: they are re-vendored by `make update`, so a bad severity there is # fixed upstream or in patches/, not by editing the label. ALERT_SEVERITY_ALLOWED=" security critical major minor warning indeterminate informational normal ok cleared debug trace unknown " +# The alerts that exist only to be dropped, the one place `none` is right. +# InfoInhibitor is the source side of an Alertmanager inhibit rule, which the +# platform config does not have, so delivering it at any severity is noise. +ALERT_SEVERITY_BLACKHOLED=" InfoInhibitor " helm_actions_stripped() { awk ' @@ -104,6 +108,14 @@ alert_severity_contract() { break fi done + if [ "$severity" = "none" ]; then + for blackholed in $ALERT_SEVERITY_BLACKHOLED; do + if [ "$blackholed" = "$alert" ]; then + accepted=1 + break + fi + done + fi if [ "$accepted" -eq 0 ]; then echo "$file: $alert -> '$severity'" >&2 bad=$((bad + 1)) @@ -128,6 +140,83 @@ alert_severity_contract() { alert_severity_contract } +@test "InfoInhibitor is labelled for the blackhole route" { + rows=$(mktemp) || return 1 + if ! alert_severity_rows > "$rows"; then + rm -f "$rows" + return 1 + fi + tab=$(printf '\t') + found=$(grep -c "${tab}InfoInhibitor${tab}" "$rows" || true) + delivered=$(grep "${tab}InfoInhibitor${tab}" "$rows" | grep -vc "${tab}none\$" || true) + rm -f "$rows" + [ "$found" -gt 0 ] + [ "$delivered" -eq 0 ] +} + +@test "severity none passes only on an alert that exists to be dropped" { + fixture=$(mktemp -d) + mkdir -p "$fixture/packages/system/example/alerts" + printf '%s\n' \ + 'apiVersion: operator.victoriametrics.com/v1beta1' \ + 'kind: VMRule' \ + 'spec:' \ + ' groups:' \ + ' - name: general' \ + ' rules:' \ + ' - alert: InfoInhibitor' \ + ' labels:' \ + ' severity: none' \ + > "$fixture/packages/system/example/alerts/00-inhibitor.yaml" + git -C "$fixture" init -q + git -C "$fixture" add packages/system/example/alerts/00-inhibitor.yaml + + inhibitor_succeeded=0 + ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || inhibitor_succeeded=$? + + printf '%s\n' \ + 'apiVersion: operator.victoriametrics.com/v1beta1' \ + 'kind: VMRule' \ + 'spec:' \ + ' groups:' \ + ' - name: silent' \ + ' rules:' \ + ' - alert: SilentAlert' \ + ' labels:' \ + ' severity: none' \ + > "$fixture/packages/system/example/alerts/01-silent.yaml" + git -C "$fixture" add packages/system/example/alerts/01-silent.yaml + + silent_succeeded=0 + ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || silent_succeeded=$? + rm -rf "$fixture" + [ "$inhibitor_succeeded" -eq 0 ] + [ "$silent_succeeded" -ne 0 ] +} + +@test "an alert that exists to be dropped still fails on a refused severity" { + fixture=$(mktemp -d) + mkdir -p "$fixture/packages/system/example/alerts" + printf '%s\n' \ + 'apiVersion: operator.victoriametrics.com/v1beta1' \ + 'kind: VMRule' \ + 'spec:' \ + ' groups:' \ + ' - name: general' \ + ' rules:' \ + ' - alert: InfoInhibitor' \ + ' labels:' \ + ' severity: info' \ + > "$fixture/packages/system/example/alerts/00-inhibitor.yaml" + git -C "$fixture" init -q + git -C "$fixture" add packages/system/example/alerts/00-inhibitor.yaml + + contract_succeeded=0 + ALERT_SEVERITY_ROOT="$fixture" alert_severity_contract >/dev/null 2>&1 || contract_succeeded=$? + rm -rf "$fixture" + [ "$contract_succeeded" -ne 0 ] +} + @test "a parser failure cannot leave the contract green" { fixture=$(mktemp -d) mkdir -p "$fixture/packages/system/example/alerts" diff --git a/packages/system/monitoring-agents/alerts/general.rules.yaml b/packages/system/monitoring-agents/alerts/general.rules.yaml index 924870c40..45dd6100f 100644 --- a/packages/system/monitoring-agents/alerts/general.rules.yaml +++ b/packages/system/monitoring-agents/alerts/general.rules.yaml @@ -52,6 +52,6 @@ spec: != "InfoInhibitor", severity =~ "warning|critical", alertstate="firing"} == 1 labels: - severity: major + severity: none exported_instance: global service: general.rules diff --git a/packages/system/monitoring/tests/alertmanager_blackhole_test.yaml b/packages/system/monitoring/tests/alertmanager_blackhole_test.yaml new file mode 100644 index 000000000..e987bd405 --- /dev/null +++ b/packages/system/monitoring/tests/alertmanager_blackhole_test.yaml @@ -0,0 +1,45 @@ +suite: Alertmanager blackhole route +templates: + - templates/alerta/alerta.yaml + +release: + name: monitoring + namespace: tenant-root + +set: + _cluster: + root-host: example.org + _namespace: + host: tenant-root.example.org + ingress: tenant-root + +# Rules labelled severity="none", InfoInhibitor among them, exist to be +# dropped, and Alerta answers that severity with a 500. Only the heartbeat +# route may come ahead of the blackhole, so no delivering route claims them. +tests: + - it: drops severity none ahead of every delivering route + asserts: + - documentSelector: + path: metadata.labels.app + value: alertmanager + matchRegex: + path: stringData["alertmanager.yaml"] + pattern: '\n routes:\n - receiver: alerta_no_resolve\n repeat_interval: 1m\n matchers:\n - .event="Heartbeat".\n - receiver: blackhole\n matchers:\n - .severity="none".\n' + + - it: keeps severity none out of email even when its alert name is listed + set: + alerta: + alerts: + email: + to: "oncall@example.org" + from: "alerts@example.org" + smarthost: "smtp.example.org:465" + alertnames: + - InfoInhibitor + asserts: + - documentSelector: + path: metadata.labels.app + value: alertmanager + matchRegex: + path: stringData["alertmanager.yaml"] + pattern: '\n routes:\n - receiver: alerta_no_resolve\n repeat_interval: 1m\n matchers:\n - .event="Heartbeat".\n - receiver: blackhole\n matchers:\n - .severity="none".\n' From 8d82348e85fb9588ad213d763ece81f7b88e84a8 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Mon, 19 Oct 2026 16:04:58 +0300 Subject: [PATCH 006/518] fix(vm-disk): refuse a disk smaller than the golden image it clones CDI checks a clone against its source before it creates the claim. When the golden image's PVC is in Block mode and requests more than the disk, it records CloneValidationFailed and leaves the DataVolume pending for good. The DataVolume spec is immutable, so raising storage on the VMDisk afterwards changes nothing, and the resize hook finds no claim to patch. Every image in vm-default-images is 20Gi on replicated, where CDI picks Block mode for the LINSTOR driver, while the chart defaults storage to 5Gi, so a disk created from an image at the default size never comes up. The chart now refuses such a disk at render time and names both sizes, so the user sets storage to the golden's size before any DataVolume exists. It compares what validateClone compares in CDI v1.66.1, the golden PVC's request against the disk's request raised to the target StorageProfile's minimumSupportedPvcSize. A Filesystem golden is not compared, because CDI sizes that clone from the image inside it or with a filesystem overhead, and neither is a golden whose PVC does not exist yet. An existing disk is exempt, as for the other golden checks. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- packages/apps/vm-disk/templates/dv.yaml | 20 ++ .../vm-disk/tests/dv_golden_size_test.yaml | 252 ++++++++++++++++++ 2 files changed, 272 insertions(+) create mode 100644 packages/apps/vm-disk/tests/dv_golden_size_test.yaml diff --git a/packages/apps/vm-disk/templates/dv.yaml b/packages/apps/vm-disk/templates/dv.yaml index 09831fc79..9bec73f5f 100644 --- a/packages/apps/vm-disk/templates/dv.yaml +++ b/packages/apps/vm-disk/templates/dv.yaml @@ -146,6 +146,26 @@ {{- fail (printf "source.image %q: golden image %q lives in StorageClass %q, and a clone into StorageClass %q of the same provisioner gets no host-assisted fallback from CDI. LINSTOR creates the clone next to the source data, and StorageClass %q does not allow remote volume access from every node, so the clone may never bind. Set storageClass to %q, or ask the platform administrator to set cloneStrategyOverride to copy in the kubevirt-cdi package." .Values.source.image.name $desiredSource.pvc.name $goldenSC $targetSC $targetSC $goldenSC) }} {{- end }} {{- end }} + {{- /* CDI v1.66.1 refuses to clone a Block source PVC into a smaller + request before it creates the claim (validateClone in + pkg/controller/datavolume/pvc-clone-controller.go), and the + DataVolume then waits forever. It raises the request to the target + StorageProfile's minimumSupportedPvcSize before comparing. A + Filesystem source is not compared, as CDI sizes that clone from + the image inside it or with a filesystem overhead the chart does + not reproduce. */}} + {{- $goldenSize := dig "spec" "resources" "requests" "storage" "" ($goldenPVC | default dict) | toString }} + {{- if and .Values.storage $goldenSize (eq (dig "spec" "volumeMode" "" ($goldenPVC | default dict) | toString) "Block") }} + {{- $goldenBytes := include "cozy-lib.resources.toFloat" $goldenSize | float64 }} + {{- $diskBytes := include "cozy-lib.resources.toFloat" (.Values.storage | toString) | float64 }} + {{- if and (lt $diskBytes $goldenBytes) $targetSC }} + {{- $minimum := dig "metadata" "annotations" "cdi.kubevirt.io/minimumSupportedPvcSize" "" (lookup "cdi.kubevirt.io/v1beta1" "StorageProfile" "" $targetSC | default dict) | toString }} + {{- $diskBytes = maxf $diskBytes (include "cozy-lib.resources.toFloat" $minimum | float64) }} + {{- end }} + {{- if lt $diskBytes $goldenBytes }} + {{- fail (printf "storage %s is smaller than golden image %q (%s), so CDI refuses to clone it and the disk would stay pending forever. Set storage to %s or more." (.Values.storage | toString) $desiredSource.pvc.name $goldenSize $goldenSize) }} + {{- end }} + {{- end }} {{- end }} {{- else if hasKey .Values.source "http" }} {{- $desiredSource = dict "http" (dict "url" (required "A valid .Values.source.http.url entry required!" .Values.source.http.url)) }} diff --git a/packages/apps/vm-disk/tests/dv_golden_size_test.yaml b/packages/apps/vm-disk/tests/dv_golden_size_test.yaml new file mode 100644 index 000000000..6a68874c7 --- /dev/null +++ b/packages/apps/vm-disk/tests/dv_golden_size_test.yaml @@ -0,0 +1,252 @@ +suite: vm-disk refuses a disk smaller than the golden image it clones +release: + name: test-vm-disk + namespace: tenant-test +templates: + - templates/dv.yaml +tests: + - it: refuses a new disk at the chart default size from a larger Block golden + set: + source: {image: {name: ubuntu}} + kubernetesProvider: + scheme: &scheme + "cdi.kubevirt.io/v1beta1/DataVolume": + gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes} + namespaced: true + "cdi.kubevirt.io/v1beta1/StorageProfile": + gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: storageprofiles} + namespaced: false + "v1/Namespace": + gvr: {group: "", version: v1, resource: namespaces} + namespaced: false + "v1/PersistentVolumeClaim": + gvr: {group: "", version: v1, resource: persistentvolumeclaims} + namespaced: true + "storage.k8s.io/v1/StorageClass": + gvr: {group: storage.k8s.io, version: v1, resource: storageclasses} + namespaced: false + objects: + - &namespace + kind: Namespace + apiVersion: v1 + metadata: {name: tenant-test} + - &goldenPVC + kind: PersistentVolumeClaim + apiVersion: v1 + metadata: {name: vm-default-images-ubuntu, namespace: cozy-public} + spec: + storageClassName: replicated + volumeMode: Block + resources: {requests: {storage: 20Gi}} + asserts: + - failedTemplate: + errorPattern: 'storage 5Gi is smaller than golden image "vm-default-images-ubuntu" \(20Gi\), so CDI refuses to clone it and the disk would stay pending forever\. Set storage to 20Gi or more\.' + + - it: clones into a disk the size of the golden + set: + source: {image: {name: ubuntu}} + storage: 20Gi + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + asserts: + - equal: + path: spec.storage.resources.requests.storage + value: 20Gi + + - it: clones into a disk larger than the golden + set: + source: {image: {name: ubuntu}} + storage: 25Gi + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + asserts: + - equal: + path: spec.storage.resources.requests.storage + value: 25Gi + + - it: compares bytes, so 21G is smaller than a 20Gi golden + set: + source: {image: {name: ubuntu}} + storage: 21G + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + asserts: + - failedTemplate: + errorPattern: 'storage 21G is smaller than golden image "vm-default-images-ubuntu" \(20Gi\)' + + - it: compares bytes, so 20480Mi is the size of a 20Gi golden + set: + source: {image: {name: ubuntu}} + storage: 20480Mi + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + asserts: + - equal: + path: spec.storage.resources.requests.storage + value: 20480Mi + + - it: clones when the target StorageProfile's minimum PVC size covers the golden + set: + source: {image: {name: ubuntu}} + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + - kind: StorageProfile + apiVersion: cdi.kubevirt.io/v1beta1 + metadata: + name: replicated + annotations: {cdi.kubevirt.io/minimumSupportedPvcSize: 20Gi} + asserts: + - notFailedTemplate: {} + + - it: refuses when the target StorageProfile's minimum PVC size is still below the golden + set: + source: {image: {name: ubuntu}} + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + - kind: StorageProfile + apiVersion: cdi.kubevirt.io/v1beta1 + metadata: + name: replicated + annotations: {cdi.kubevirt.io/minimumSupportedPvcSize: 10Gi} + asserts: + - failedTemplate: + errorPattern: 'storage 5Gi is smaller than golden image "vm-default-images-ubuntu" \(20Gi\)' + + - it: reads the minimum PVC size of the default class for an empty storageClass + set: + source: {image: {name: ubuntu}} + storageClass: "" + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + - kind: StorageClass + apiVersion: storage.k8s.io/v1 + metadata: + name: replicated + annotations: {storageclass.kubernetes.io/is-default-class: "true"} + provisioner: linstor.csi.linbit.com + - kind: StorageProfile + apiVersion: cdi.kubevirt.io/v1beta1 + metadata: + name: replicated + annotations: {cdi.kubevirt.io/minimumSupportedPvcSize: 20Gi} + asserts: + - notFailedTemplate: {} + + - it: does not compare sizes for a Filesystem golden + set: + source: {image: {name: ubuntu}} + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - kind: PersistentVolumeClaim + apiVersion: v1 + metadata: {name: vm-default-images-ubuntu, namespace: cozy-public} + spec: + storageClassName: replicated + volumeMode: Filesystem + resources: {requests: {storage: 20Gi}} + asserts: + - notFailedTemplate: {} + + - it: renders without a cluster to look the golden up in + set: + source: {image: {name: ubuntu}} + asserts: + - equal: + path: spec.storage.resources.requests.storage + value: 5Gi + + - it: clones a golden whose PVC does not exist yet + set: + source: {image: {name: ubuntu}} + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - kind: DataVolume + apiVersion: cdi.kubevirt.io/v1beta1 + metadata: {name: vm-default-images-ubuntu, namespace: cozy-public} + status: {phase: ImportScheduled} + asserts: + - notFailedTemplate: {} + + - it: leaves a disk without a size to CDI, which takes the golden's + set: + source: {image: {name: ubuntu}} + storage: null + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + asserts: + - notFailedTemplate: {} + + - it: leaves an existing disk alone + set: + source: {image: {name: ubuntu}} + storage: 25Gi + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + - kind: DataVolume + apiVersion: cdi.kubevirt.io/v1beta1 + metadata: + name: test-vm-disk + namespace: tenant-test + annotations: + vm-disk.cozystack.io/source: '{"image":{"name":"ubuntu"}}' + vm-disk.cozystack.io/storage-class: replicated + spec: + source: {pvc: {name: vm-default-images-ubuntu, namespace: cozy-public}} + storage: {resources: {requests: {storage: 10Gi}}, storageClassName: replicated} + asserts: + - notFailedTemplate: {} + + - it: does not compare sizes for an http source + set: + source: {http: {url: "https://example.com/disk.img"}} + storage: 1Gi + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + asserts: + - notFailedTemplate: {} + + - it: does not compare sizes for an upload source + set: + source: {upload: {}} + storage: 1Gi + kubernetesProvider: + scheme: *scheme + objects: + - *namespace + - *goldenPVC + asserts: + - notFailedTemplate: {} From 85026299958edba4ccdcfaafef5b17fad7f055fc Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Mon, 19 Oct 2026 19:33:25 +0300 Subject: [PATCH 007/518] fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves linstor_error_reports_count comes from the LINSTOR controller, so its series carry the controller pod's instance and controller_node labels. When the pod moves to another node, or an external label of the scrape changes, the existing report counts come back as new series. VictoriaMetrics' increase() counts the first samples of a series from zero when they are small, and report counts are small, so linstorSatelliteErrorRate fires for every node with an old report and keeps firing for 15 minutes. The up guard matches on instance and lets the new series through. The controller also leaves a satellite out of the metric while it is not connected, so a satellite that comes back after more than five minutes has its old count counted again in the same way. Keyed on hostname, a moved controller or a changed label continues the same series, and holding the last value for an hour bridges a disconnect or a slow controller restart. The up guard matches on() because the result no longer has an instance label. The controller rule takes the same form. Its hostname is the pod name and a new pod starts with no reports, but a label change hit it the same way. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- hack/linstor-error-report-alerts.bats | 32 +++ .../linstor-error-report-alerts/unittest.yaml | 198 ++++++++++++++++++ .../alerts/piraeus-datastore.yaml | 6 +- 3 files changed, 233 insertions(+), 3 deletions(-) create mode 100644 hack/linstor-error-report-alerts.bats create mode 100644 hack/testdata/linstor-error-report-alerts/unittest.yaml diff --git a/hack/linstor-error-report-alerts.bats b/hack/linstor-error-report-alerts.bats new file mode 100644 index 000000000..4c6124aad --- /dev/null +++ b/hack/linstor-error-report-alerts.bats @@ -0,0 +1,32 @@ +#!/usr/bin/env bats +# Replays the LINSTOR error-report alerts from +# packages/system/piraeus-operator/alerts/piraeus-datastore.yaml through +# vmalert-tool, pinned to the VictoriaMetrics release that the +# victoria-metrics-operator deploys by default. Its increase() counts the first +# samples of a series as new when they are small, and the report counts are +# small, so the cases pin that a series which starts over after a controller +# move, a label change or a disconnect does not report old error reports again. +# Only pulling the image touches the network. + +load test_helper + +REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)" +VMALERT_TOOL_IMAGE="victoriametrics/vmalert-tool:v1.136.0@sha256:417346d7b6fdb66d8e3984887c2ad114535ec7d983eeef53381b56c776f333f1" + +@test "LINSTOR error-report alerts pass their vmalert-tool cases" { + docker info >/dev/null 2>&1 || { + echo "docker is required to run the cases in $VMALERT_TOOL_IMAGE" >&2 + return 1 + } + work=$(mktemp -d) + yq '.spec' "$REPO_ROOT/packages/system/piraeus-operator/alerts/piraeus-datastore.yaml" > "$work/rules.yaml" + cp "$REPO_ROOT/hack/testdata/linstor-error-report-alerts/unittest.yaml" "$work/unittest.yaml" + rc=0 + docker run --rm --network none -v "$work:/w:ro" "$VMALERT_TOOL_IMAGE" \ + unittest --files /w/unittest.yaml > "$work/out" 2>&1 || rc=$? + cat "$work/out" + passed=$(grep -cx 'SUCCESS' "$work/out" || true) + rm -rf "$work" + [ "$rc" -eq 0 ] + [ "$passed" -eq 1 ] +} diff --git a/hack/testdata/linstor-error-report-alerts/unittest.yaml b/hack/testdata/linstor-error-report-alerts/unittest.yaml new file mode 100644 index 000000000..41e332169 --- /dev/null +++ b/hack/testdata/linstor-error-report-alerts/unittest.yaml @@ -0,0 +1,198 @@ +rule_files: + - rules.yaml + +evaluation_interval: 15s + +tests: + - name: controller pod moves to another node + interval: 30s + input_series: + - series: 'up{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a"}' + values: '1x38 0 stale' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a", hostname="node-a", module="SATELLITE"}' + values: '1x38 stale' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", controller_node="node-a", hostname="node-b", module="SATELLITE"}' + values: '2x38 stale' + - series: 'up{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b"}' + values: '_x41 1x79' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b", hostname="node-a", module="SATELLITE"}' + values: '_x41 1x58 2x20' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.2:3370", controller_node="node-b", hostname="node-b", module="SATELLITE"}' + values: '_x41 2x79' + alert_rule_test: + - eval_time: 21m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + - eval_time: 28m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + - eval_time: 35m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + - eval_time: 45m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + - eval_time: 51m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + exp_alerts: + - exp_labels: + hostname: node-a + severity: warning + exp_annotations: + description: | + LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes. + Use "linstor error-reports list --nodes node-a" to inspect the reports. + + - name: an external label of the scrape changes + interval: 30s + input_series: + - series: 'up{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack"}' + values: '1x39 stale' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack", hostname="node-a", module="SATELLITE"}' + values: '1x39 stale' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cozystack", hostname="linstor-controller-abc", module="CONTROLLER"}' + values: '1x39 stale' + - series: 'up{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a"}' + values: '_x41 1x79' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a", hostname="node-a", module="SATELLITE"}' + values: '_x41 1x79' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", cluster="cluster-a", hostname="linstor-controller-abc", module="CONTROLLER"}' + values: '_x41 1x79' + alert_rule_test: + - eval_time: 21m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + - eval_time: 30m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + - eval_time: 21m + groupname: linstor.rules + alertname: linstorControllerErrorRate + - eval_time: 30m + groupname: linstor.rules + alertname: linstorControllerErrorRate + + - name: a satellite drops out of the metric while it is disconnected + interval: 30s + input_series: + - series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}' + values: '1x220' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}' + values: '1x38 stale _x80 1x100' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-b", module="SATELLITE"}' + values: '2x220' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-c", module="SATELLITE"}' + values: '1x38 stale _x80 2x100' + alert_rule_test: + - eval_time: 61m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + exp_alerts: + - exp_labels: + hostname: node-c + severity: warning + exp_annotations: + description: | + LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes. + Use "linstor error-reports list --nodes node-c" to inspect the reports. + - eval_time: 70m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + exp_alerts: + - exp_labels: + hostname: node-c + severity: warning + exp_annotations: + description: | + LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes. + Use "linstor error-reports list --nodes node-c" to inspect the reports. + - eval_time: 76m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + + - name: a new report fires for the 15 minute window + interval: 30s + input_series: + - series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}' + values: '1x120' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}' + values: '1x39 2x80' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-b", module="SATELLITE"}' + values: '2x120' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-c", module="SATELLITE"}' + values: '0x39 1x80' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="linstor-controller-abc", module="CONTROLLER"}' + values: '0x39 1x80' + alert_rule_test: + - eval_time: 21m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + exp_alerts: + - exp_labels: + hostname: node-a + severity: warning + exp_annotations: + description: | + LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes. + Use "linstor error-reports list --nodes node-a" to inspect the reports. + - exp_labels: + hostname: node-c + severity: warning + exp_annotations: + description: | + LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes. + Use "linstor error-reports list --nodes node-c" to inspect the reports. + - eval_time: 34m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + exp_alerts: + - exp_labels: + hostname: node-a + severity: warning + exp_annotations: + description: | + LINSTOR Satellite "node-a" reports 1 errors in the last 15 minutes. + Use "linstor error-reports list --nodes node-a" to inspect the reports. + - exp_labels: + hostname: node-c + severity: warning + exp_annotations: + description: | + LINSTOR Satellite "node-c" reports 1 errors in the last 15 minutes. + Use "linstor error-reports list --nodes node-c" to inspect the reports. + - eval_time: 36m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + - eval_time: 21m + groupname: linstor.rules + alertname: linstorControllerErrorRate + exp_alerts: + - exp_labels: + hostname: linstor-controller-abc + severity: warning + exp_annotations: + description: | + LINSTOR Controller reports 1 errors in the last 15 minutes. + Use "linstor error-reports list --since 15minutes" to see them. + - eval_time: 36m + groupname: linstor.rules + alertname: linstorControllerErrorRate + + - name: a failed controller scrape still holds the alert back + interval: 30s + input_series: + - series: 'up{job="linstor-controller", instance="10.0.0.1:3370"}' + values: '1x19 0x140 1x60' + - series: 'linstor_error_reports_count{job="linstor-controller", instance="10.0.0.1:3370", hostname="node-a", module="SATELLITE"}' + values: '1x19 stale _x140 1x60' + alert_rule_test: + - eval_time: 85m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + - eval_time: 93m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate + - eval_time: 100m + groupname: linstor.rules + alertname: linstorSatelliteErrorRate diff --git a/packages/system/piraeus-operator/alerts/piraeus-datastore.yaml b/packages/system/piraeus-operator/alerts/piraeus-datastore.yaml index ab679d663..8be364a72 100644 --- a/packages/system/piraeus-operator/alerts/piraeus-datastore.yaml +++ b/packages/system/piraeus-operator/alerts/piraeus-datastore.yaml @@ -29,8 +29,8 @@ spec: LINSTOR Satellite "{{ $labels.hostname }}" reports {{ $value }} errors in the last 15 minutes. Use "linstor error-reports list --nodes {{ $labels.hostname }}" to inspect the reports. expr: | - increase(linstor_error_reports_count{module="SATELLITE"}[15m]) > 0 - and on(instance, job) + increase((max by (hostname) (last_over_time(linstor_error_reports_count{module="SATELLITE"}[1h])))[15m:1m]) > 0 + and on() min_over_time(up{job="linstor-controller"}[15m]) == 1 labels: severity: warning @@ -39,7 +39,7 @@ spec: description: | LINSTOR Controller reports {{ $value }} errors in the last 15 minutes. Use "linstor error-reports list --since 15minutes" to see them. - expr: increase(linstor_error_reports_count{module="CONTROLLER"}[15m]) > 0 + expr: increase((max by (hostname) (last_over_time(linstor_error_reports_count{module="CONTROLLER"}[1h])))[15m:1m]) > 0 labels: severity: warning - alert: linstorSatelliteNotOnline From 78cadf53f9f88a2b5569024031c399a5aba6e075 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Mon, 19 Oct 2026 19:55:08 +0300 Subject: [PATCH 008/518] fix(tcp-balancer): render the kubernetes and talos Service ports only with kubernetesAndTalos The Service rendered ports 6443 and 50000 under httpAndHttps, while the Deployment declares the kubernetes and talos container ports and HAProxy binds those frontends only under kubernetesAndTalos. So with the default values every Service carried two ports whose targetPort names no container declares, with no endpoints behind them, and with external: true the LoadBalancer Service carried them as well. A release with kubernetesAndTalos and without httpAndHttps got no Service port for those frontends at all. kubernetesAndTalos is not documented in values.yaml, but the apps API keeps the top level of spec open and hands it to the chart as values, so the key reaches the chart and the Deployment and the ConfigMap already follow it. Gating the Service on the same key keeps the three templates in step, while dropping the block would leave HAProxy listening on ports that no Service exposes. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../apps/tcp-balancer/templates/service.yaml | 2 +- .../apps/tcp-balancer/tests/service_test.yaml | 97 +++++++++++++++++++ 2 files changed, 98 insertions(+), 1 deletion(-) create mode 100644 packages/apps/tcp-balancer/tests/service_test.yaml diff --git a/packages/apps/tcp-balancer/templates/service.yaml b/packages/apps/tcp-balancer/templates/service.yaml index 8384cc85b..689db7e16 100644 --- a/packages/apps/tcp-balancer/templates/service.yaml +++ b/packages/apps/tcp-balancer/templates/service.yaml @@ -27,7 +27,7 @@ spec: port: 443 targetPort: https {{- end }} - {{- with .Values.httpAndHttps }} + {{- with .Values.kubernetesAndTalos }} - name: kubernetes protocol: TCP port: 6443 diff --git a/packages/apps/tcp-balancer/tests/service_test.yaml b/packages/apps/tcp-balancer/tests/service_test.yaml new file mode 100644 index 000000000..57191115c --- /dev/null +++ b/packages/apps/tcp-balancer/tests/service_test.yaml @@ -0,0 +1,97 @@ +suite: haproxy service + +templates: + - templates/service.yaml + - templates/deployment.yaml + - templates/configmap.yaml + +tests: + - it: publishes only http and https by default, both on container ports the Deployment declares + asserts: + - equal: + path: spec.ports + value: + - name: http + protocol: TCP + port: 80 + targetPort: http + - name: https + protocol: TCP + port: 443 + targetPort: https + template: templates/service.yaml + - equal: + path: spec.template.spec.containers[0].ports + value: + - containerPort: 8080 + name: http + - containerPort: 8443 + name: https + template: templates/deployment.yaml + + - it: publishes kubernetes and talos for kubernetesAndTalos alone, both on container ports the Deployment declares + set: + httpAndHttps: null + kubernetesAndTalos: + endpoints: + - 10.1.1.1 + asserts: + - equal: + path: spec.ports + value: + - name: kubernetes + protocol: TCP + port: 6443 + targetPort: kubernetes + - name: talos + protocol: TCP + port: 50000 + targetPort: talos + template: templates/service.yaml + - equal: + path: spec.template.spec.containers[0].ports + value: + - containerPort: 6443 + name: kubernetes + - containerPort: 50000 + name: talos + template: templates/deployment.yaml + + - it: publishes all four ports when both are set, each on a container port the Deployment declares + set: + kubernetesAndTalos: + endpoints: + - 10.1.1.1 + asserts: + - equal: + path: spec.ports + value: + - name: http + protocol: TCP + port: 80 + targetPort: http + - name: https + protocol: TCP + port: 443 + targetPort: https + - name: kubernetes + protocol: TCP + port: 6443 + targetPort: kubernetes + - name: talos + protocol: TCP + port: 50000 + targetPort: talos + template: templates/service.yaml + - equal: + path: spec.template.spec.containers[0].ports + value: + - containerPort: 8080 + name: http + - containerPort: 8443 + name: https + - containerPort: 6443 + name: kubernetes + - containerPort: 50000 + name: talos + template: templates/deployment.yaml From f1f30b3071254e3640292b4eb483b04f5d40d77a Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Tue, 20 Oct 2026 12:05:51 +0300 Subject: [PATCH 009/518] fix(etcd): require an even spread of members over nodes by default The default spread of the etcd members was ScheduleAnyway, which only adds to the score of a node, and other scores can outweigh it. A new etcd on a three-node cluster got its first two members on one node. On a node-local StorageClass a member cannot leave its node, so losing that node takes two of the three members and the quorum. DoNotSchedule with maxSkew 1 still lets an etcd with more members than nodes start, because the skew is counted between the nodes that exist. One node takes all three members and two nodes take two and one. nodeTaintsPolicy is Ignore, so a cordoned node still counts and a member waits for it instead of landing next to another member. The operator copies the constraints into an EtcdMember when it creates it and builds the Pod from that copy, so members that exist keep the old setting and only members created later get the new one. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../extra/etcd/templates/etcd-cluster.yaml | 3 +- .../etcd/tests/topology-spread_test.yaml | 46 ++++++++++++++----- 2 files changed, 37 insertions(+), 12 deletions(-) diff --git a/packages/extra/etcd/templates/etcd-cluster.yaml b/packages/extra/etcd/templates/etcd-cluster.yaml index 2824ce371..67dd34bd9 100644 --- a/packages/extra/etcd/templates/etcd-cluster.yaml +++ b/packages/extra/etcd/templates/etcd-cluster.yaml @@ -69,7 +69,8 @@ spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: "kubernetes.io/hostname" - whenUnsatisfiable: ScheduleAnyway + whenUnsatisfiable: DoNotSchedule + nodeTaintsPolicy: Ignore labelSelector: matchLabels: app.kubernetes.io/instance: etcd diff --git a/packages/extra/etcd/tests/topology-spread_test.yaml b/packages/extra/etcd/tests/topology-spread_test.yaml index 4b96fc310..11bb92243 100644 --- a/packages/extra/etcd/tests/topology-spread_test.yaml +++ b/packages/extra/etcd/tests/topology-spread_test.yaml @@ -8,7 +8,7 @@ templates: - templates/etcd-cluster.yaml tests: - - it: uses a soft per-host spread by default + - it: uses a strict per-host spread by default set: _cluster: {} documentSelector: @@ -16,17 +16,41 @@ tests: value: EtcdCluster asserts: - equal: - path: spec.topologySpreadConstraints[0].maxSkew - value: 1 + path: spec.topologySpreadConstraints[0] + value: + maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: DoNotSchedule + nodeTaintsPolicy: Ignore + labelSelector: + matchLabels: + app.kubernetes.io/instance: etcd + - lengthEqual: + path: spec.topologySpreadConstraints + count: 1 + + - it: renders a soft cluster-wide spread policy as written + set: + _cluster: + scheduling: + globalAppTopologySpreadConstraints: | + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + documentSelector: + path: kind + value: EtcdCluster + asserts: - equal: - path: spec.topologySpreadConstraints[0].topologyKey - value: kubernetes.io/hostname - - equal: - path: spec.topologySpreadConstraints[0].whenUnsatisfiable - value: ScheduleAnyway - - equal: - path: spec.topologySpreadConstraints[0].labelSelector.matchLabels["app.kubernetes.io/instance"] - value: etcd + path: spec.topologySpreadConstraints[0] + value: + maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/instance: etcd - lengthEqual: path: spec.topologySpreadConstraints count: 1 From 5632596fe84fa358191c62f5f316df62f32c865c Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Tue, 20 Oct 2026 13:38:55 +0300 Subject: [PATCH 010/518] feat(monitoring): alert when pods of one workload share a node Two etcd members, two Redis Sentinels or both replicas of a storage tier can end up on one node through a soft spread rule, a drain, or a placement made before a spread rule existed. Nothing reported it, and the scheduler does not move running pods, so it stays like that until a pod is recreated. One node failure then costs the quorum or every copy at once. etcd-operator owns each member pod by its own EtcdMember, so members are grouped by the EtcdCluster name their names start with. Owner and node are read from kube_pod_info because kube_pod_owner and kube_pod_status_phase get the node of the kube-state-metrics pod as a target label. Nodes with a NoSchedule or NoExecute taint do not count, which covers cordoned, not ready and unreachable nodes. The alert waits an hour, longer than a replacement pod runs next to its terminating predecessor or a node takes to drain and come back. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../workload-placement-alerts/unittest.yaml | 572 ++++++++++++++++++ hack/workload-placement-alerts.bats | 26 + .../alerts/workload-placement.yaml | 81 +++ 3 files changed, 679 insertions(+) create mode 100644 hack/testdata/workload-placement-alerts/unittest.yaml create mode 100644 hack/workload-placement-alerts.bats create mode 100644 packages/system/monitoring-agents/alerts/workload-placement.yaml diff --git a/hack/testdata/workload-placement-alerts/unittest.yaml b/hack/testdata/workload-placement-alerts/unittest.yaml new file mode 100644 index 000000000..2565a8241 --- /dev/null +++ b/hack/testdata/workload-placement-alerts/unittest.yaml @@ -0,0 +1,572 @@ +rule_files: + - rules.yaml + +evaluation_interval: 15s + +tests: + - name: two of three StatefulSet pods on one node + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-0", node="node-a", created_by_kind="StatefulSet", created_by_name="db"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-1", node="node-a", created_by_kind="StatefulSet", created_by_name="db"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-2", node="node-b", created_by_kind="StatefulSet", created_by_name="db"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-0", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-1", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-2", phase="Running", node="node-c"}' + values: '1x150' + alert_rule_test: + - eval_time: 59m + groupname: workload-placement + alertname: PodReplicasColocated + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + exp_alerts: + - exp_labels: + cluster: cozystack + namespace: tenant-a + owner_kind: StatefulSet + owner_name: db + node: node-a + severity: warning + exported_instance: tenant-a/db + service: workload-placement + exp_annotations: + summary: Pods of one workload share a node. + description: >- + 2 running pods of StatefulSet tenant-a/db are on node node-a in cluster cozystack, + although the cluster has a schedulable node for each of its pods. Losing this node + stops these pods together, which can cost a quorum-based workload its quorum. The + scheduler does not move running pods, so they stay together until they are recreated. + + - name: StatefulSet pods spread over three nodes + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-0", node="node-a", created_by_kind="StatefulSet", created_by_name="db"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-1", node="node-b", created_by_kind="StatefulSet", created_by_name="db"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-2", node="node-c", created_by_kind="StatefulSet", created_by_name="db"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-0", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-1", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="db-2", phase="Running", node="node-c"}' + values: '1x150' + alert_rule_test: + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + - eval_time: 120m + groupname: workload-placement + alertname: PodReplicasColocated + + - name: five ReplicaSet pods on three nodes + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-aaaaa", node="node-a", created_by_kind="ReplicaSet", created_by_name="worker-7c8d9"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-bbbbb", node="node-a", created_by_kind="ReplicaSet", created_by_name="worker-7c8d9"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-ccccc", node="node-b", created_by_kind="ReplicaSet", created_by_name="worker-7c8d9"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-ddddd", node="node-b", created_by_kind="ReplicaSet", created_by_name="worker-7c8d9"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-eeeee", node="node-c", created_by_kind="ReplicaSet", created_by_name="worker-7c8d9"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-aaaaa", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-bbbbb", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-ccccc", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-ddddd", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="worker-7c8d9-eeeee", phase="Running", node="node-c"}' + values: '1x150' + alert_rule_test: + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + - eval_time: 120m + groupname: workload-placement + alertname: PodReplicasColocated + + - name: DaemonSet surge pod next to the old one + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-x7k2p", node="node-a", created_by_kind="DaemonSet", created_by_name="node-agent"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-q9z4m", node="node-a", created_by_kind="DaemonSet", created_by_name="node-agent"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-b5n8w", node="node-b", created_by_kind="DaemonSet", created_by_name="node-agent"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-x7k2p", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-q9z4m", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="cozy-system", pod="node-agent-b5n8w", phase="Running", node="node-c"}' + values: '1x150' + alert_rule_test: + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + - eval_time: 120m + groupname: workload-placement + alertname: PodReplicasColocated + + - name: two pods of one Job on one node + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="backup-29345678-h2k4m", node="node-a", created_by_kind="Job", created_by_name="backup-29345678"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="backup-29345678-p8r6t", node="node-a", created_by_kind="Job", created_by_name="backup-29345678"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="backup-29345678-h2k4m", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="backup-29345678-p8r6t", phase="Running", node="node-c"}' + values: '1x150' + alert_rule_test: + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + + - name: etcd members of one EtcdCluster on one node + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-x7k2p", node="node-a", created_by_kind="EtcdMember", created_by_name="etcd-x7k2p"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-q9z4m", node="node-a", created_by_kind="EtcdMember", created_by_name="etcd-q9z4m"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-b5n8w", node="node-b", created_by_kind="EtcdMember", created_by_name="etcd-b5n8w"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-0", node="node-b", created_by_kind="EtcdMember", created_by_name="etcd-0"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-1", node="node-c", created_by_kind="EtcdMember", created_by_name="etcd-1"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-2", node="node-c", created_by_kind="EtcdMember", created_by_name="etcd-2"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-x7k2p", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-q9z4m", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="etcd-b5n8w", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-0", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-1", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-b", pod="etcd-2", phase="Running", node="node-c"}' + values: '1x150' + alert_rule_test: + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + exp_alerts: + - exp_labels: + cluster: cozystack + namespace: tenant-a + owner_kind: EtcdCluster + owner_name: etcd + node: node-a + severity: warning + exported_instance: tenant-a/etcd + service: workload-placement + exp_annotations: + summary: Pods of one workload share a node. + description: >- + 2 running pods of EtcdCluster tenant-a/etcd are on node node-a in cluster cozystack, + although the cluster has a schedulable node for each of its pods. Losing this node + stops these pods together, which can cost a quorum-based workload its quorum. The + scheduler does not move running pods, so they stay together until they are recreated. + - exp_labels: + cluster: cozystack + namespace: tenant-b + owner_kind: EtcdCluster + owner_name: etcd + node: node-c + severity: warning + exported_instance: tenant-b/etcd + service: workload-placement + exp_annotations: + summary: Pods of one workload share a node. + description: >- + 2 running pods of EtcdCluster tenant-b/etcd are on node node-c in cluster cozystack, + although the cluster has a schedulable node for each of its pods. Losing this node + stops these pods together, which can cost a quorum-based workload its quorum. The + scheduler does not move running pods, so they stay together until they are recreated. + + - name: replacement pod next to its terminating predecessor for twenty minutes + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-aaaaa", node="node-a", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-bbbbb", node="node-b", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}' + values: '1x30 stale' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-ccccc", node="node-b", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}' + values: '_x10 1x140' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-aaaaa", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-bbbbb", phase="Running", node="node-c"}' + values: '1x30 stale' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-ccccc", phase="Running", node="node-c"}' + values: '_x10 1x140' + alert_rule_test: + - eval_time: 29m + groupname: workload-placement + alertname: PodReplicasColocated + - eval_time: 75m + groupname: workload-placement + alertname: PodReplicasColocated + - eval_time: 120m + groupname: workload-placement + alertname: PodReplicasColocated + + - name: the same overlap held past the hour + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-aaaaa", node="node-a", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-bbbbb", node="node-b", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}' + values: '1x100 stale' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-ccccc", node="node-b", created_by_kind="ReplicaSet", created_by_name="web-6b8f4"}' + values: '_x10 1x140' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-aaaaa", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-bbbbb", phase="Running", node="node-c"}' + values: '1x100 stale' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="web-6b8f4-ccccc", phase="Running", node="node-c"}' + values: '_x10 1x140' + alert_rule_test: + - eval_time: 69m + groupname: workload-placement + alertname: PodReplicasColocated + - eval_time: 71m + groupname: workload-placement + alertname: PodReplicasColocated + exp_alerts: + - exp_labels: + cluster: cozystack + namespace: tenant-a + owner_kind: ReplicaSet + owner_name: web-6b8f4 + node: node-b + severity: warning + exported_instance: tenant-a/web-6b8f4 + service: workload-placement + exp_annotations: + summary: Pods of one workload share a node. + description: >- + 2 running pods of ReplicaSet tenant-a/web-6b8f4 are on node node-b in cluster cozystack, + although the cluster has a schedulable node for each of its pods. Losing this node + stops these pods together, which can cost a quorum-based workload its quorum. The + scheduler does not move running pods, so they stay together until they are recreated. + - eval_time: 105m + groupname: workload-placement + alertname: PodReplicasColocated + + - name: a cordoned node does not count until it is uncordoned + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '0x150' + - series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '0x150' + - series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x30 0x120' + - series: 'kube_node_spec_taint{job="kube-state-metrics", cluster="cozystack", node="node-c", key="node.kubernetes.io/unschedulable", value="", effect="NoSchedule"}' + values: '1x30 stale' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-0", node="node-a", created_by_kind="StatefulSet", created_by_name="kv"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-1", node="node-a", created_by_kind="StatefulSet", created_by_name="kv"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-2", node="node-b", created_by_kind="StatefulSet", created_by_name="kv"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-0", phase="Running", node="node-a"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-1", phase="Running", node="node-a"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-2", phase="Running", node="node-a"}' + values: '1x150' + alert_rule_test: + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + - eval_time: 90m + groupname: workload-placement + alertname: PodReplicasColocated + - eval_time: 92m + groupname: workload-placement + alertname: PodReplicasColocated + exp_alerts: + - exp_labels: + cluster: cozystack + namespace: tenant-a + owner_kind: StatefulSet + owner_name: kv + node: node-a + severity: warning + exported_instance: tenant-a/kv + service: workload-placement + exp_annotations: + summary: Pods of one workload share a node. + description: >- + 2 running pods of StatefulSet tenant-a/kv are on node node-a in cluster cozystack, + although the cluster has a schedulable node for each of its pods. Losing this node + stops these pods together, which can cost a quorum-based workload its quorum. The + scheduler does not move running pods, so they stay together until they are recreated. + + - name: a node that is down does not count + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '0x150' + - series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '0x150' + - series: 'kube_node_spec_unschedulable{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '0x150' + - series: 'kube_node_spec_taint{job="kube-state-metrics", cluster="cozystack", node="node-c", key="node.kubernetes.io/unreachable", value="", effect="NoSchedule"}' + values: '1x150' + - series: 'kube_node_spec_taint{job="kube-state-metrics", cluster="cozystack", node="node-c", key="node.kubernetes.io/unreachable", value="", effect="NoExecute"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-0", node="node-a", created_by_kind="StatefulSet", created_by_name="kv"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-1", node="node-a", created_by_kind="StatefulSet", created_by_name="kv"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-2", node="node-b", created_by_kind="StatefulSet", created_by_name="kv"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-0", phase="Running", node="node-a"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-1", phase="Running", node="node-a"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kv-2", phase="Running", node="node-a"}' + values: '1x150' + alert_rule_test: + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + - eval_time: 120m + groupname: workload-placement + alertname: PodReplicasColocated + + - name: pods that are not running do not count + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-aaaaa", node="node-a", created_by_kind="ReplicaSet", created_by_name="api-5c6d7"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-bbbbb", node="node-a", created_by_kind="ReplicaSet", created_by_name="api-5c6d7"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-ccccc", node="node-b", created_by_kind="ReplicaSet", created_by_name="api-5c6d7"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-aaaaa", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-bbbbb", phase="Running", node="node-c"}' + values: '0x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-bbbbb", phase="Failed", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="api-5c6d7-ccccc", phase="Running", node="node-c"}' + values: '1x150' + alert_rule_test: + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + + - name: one pod each in two clusters that share workload and node names + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="kubernetes-example", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="kubernetes-example", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="kubernetes-example", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="kube-system", pod="coredns-5d78c9869d-x7k2p", node="node-a", created_by_kind="ReplicaSet", created_by_name="coredns-5d78c9869d"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="kubernetes-example", namespace="kube-system", pod="coredns-5d78c9869d-q9z4m", node="node-a", created_by_kind="ReplicaSet", created_by_name="coredns-5d78c9869d"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="kube-system", pod="coredns-5d78c9869d-x7k2p", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="kubernetes-example", namespace="kube-system", pod="coredns-5d78c9869d-q9z4m", phase="Running", node="node-b"}' + values: '1x150' + alert_rule_test: + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + + - name: ReplicaSet, CNPG Cluster and StrimziPodSet pods on one node + interval: 1m + input_series: + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-a"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-b"}' + values: '1x150' + - series: 'kube_node_info{job="kube-state-metrics", cluster="cozystack", node="node-c"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-h2k4m", node="node-a", created_by_kind="ReplicaSet", created_by_name="rfs-cache-6f7d8b9c5d"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-p8r6t", node="node-a", created_by_kind="ReplicaSet", created_by_name="rfs-cache-6f7d8b9c5d"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-w3x5z", node="node-b", created_by_kind="ReplicaSet", created_by_name="rfs-cache-6f7d8b9c5d"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-1", node="node-a", created_by_kind="Cluster", created_by_name="postgres-db"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-2", node="node-b", created_by_kind="Cluster", created_by_name="postgres-db"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-3", node="node-b", created_by_kind="Cluster", created_by_name="postgres-db"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-0", node="node-c", created_by_kind="StrimziPodSet", created_by_name="kafka-kafka"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-1", node="node-c", created_by_kind="StrimziPodSet", created_by_name="kafka-kafka"}' + values: '1x150' + - series: 'kube_pod_info{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-2", node="node-a", created_by_kind="StrimziPodSet", created_by_name="kafka-kafka"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-h2k4m", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-p8r6t", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="rfs-cache-6f7d8b9c5d-w3x5z", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-1", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-2", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="postgres-db-3", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-0", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-1", phase="Running", node="node-c"}' + values: '1x150' + - series: 'kube_pod_status_phase{job="kube-state-metrics", cluster="cozystack", namespace="tenant-a", pod="kafka-kafka-2", phase="Running", node="node-c"}' + values: '1x150' + alert_rule_test: + - eval_time: 61m + groupname: workload-placement + alertname: PodReplicasColocated + exp_alerts: + - exp_labels: + cluster: cozystack + namespace: tenant-a + owner_kind: ReplicaSet + owner_name: rfs-cache-6f7d8b9c5d + node: node-a + severity: warning + exported_instance: tenant-a/rfs-cache-6f7d8b9c5d + service: workload-placement + exp_annotations: + summary: Pods of one workload share a node. + description: >- + 2 running pods of ReplicaSet tenant-a/rfs-cache-6f7d8b9c5d are on node node-a in cluster + cozystack, although the cluster has a schedulable node for each of its pods. Losing this + node stops these pods together, which can cost a quorum-based workload its quorum. The + scheduler does not move running pods, so they stay together until they are recreated. + - exp_labels: + cluster: cozystack + namespace: tenant-a + owner_kind: Cluster + owner_name: postgres-db + node: node-b + severity: warning + exported_instance: tenant-a/postgres-db + service: workload-placement + exp_annotations: + summary: Pods of one workload share a node. + description: >- + 2 running pods of Cluster tenant-a/postgres-db are on node node-b in cluster cozystack, + although the cluster has a schedulable node for each of its pods. Losing this node + stops these pods together, which can cost a quorum-based workload its quorum. The + scheduler does not move running pods, so they stay together until they are recreated. + - exp_labels: + cluster: cozystack + namespace: tenant-a + owner_kind: StrimziPodSet + owner_name: kafka-kafka + node: node-c + severity: warning + exported_instance: tenant-a/kafka-kafka + service: workload-placement + exp_annotations: + summary: Pods of one workload share a node. + description: >- + 2 running pods of StrimziPodSet tenant-a/kafka-kafka are on node node-c in cluster + cozystack, although the cluster has a schedulable node for each of its pods. Losing + this node stops these pods together, which can cost a quorum-based workload its quorum. + The scheduler does not move running pods, so they stay together until they are + recreated. diff --git a/hack/workload-placement-alerts.bats b/hack/workload-placement-alerts.bats new file mode 100644 index 000000000..3f7b9b961 --- /dev/null +++ b/hack/workload-placement-alerts.bats @@ -0,0 +1,26 @@ +#!/usr/bin/env bats + +load test_helper + +REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)" +VMALERT_TOOL_IMAGE="victoriametrics/vmalert-tool:v1.136.0@sha256:417346d7b6fdb66d8e3984887c2ad114535ec7d983eeef53381b56c776f333f1" + +@test "workload placement alerts pass their vmalert-tool cases" { + docker info >/dev/null 2>&1 || { + echo "docker is required to run the cases in $VMALERT_TOOL_IMAGE" >&2 + return 1 + } + work=$(mktemp -d) + rc=0 + yq '.spec' "$REPO_ROOT/packages/system/monitoring-agents/alerts/workload-placement.yaml" > "$work/rules.yaml" || rc=$? + cp "$REPO_ROOT/hack/testdata/workload-placement-alerts/unittest.yaml" "$work/unittest.yaml" || rc=$? + if [ "$rc" -eq 0 ]; then + docker run --rm --network none -v "$work:/w:ro" "$VMALERT_TOOL_IMAGE" \ + unittest --files /w/unittest.yaml > "$work/out" 2>&1 || rc=$? + cat "$work/out" + fi + passed=$(grep -cx 'SUCCESS' "$work/out" 2>/dev/null || true) + rm -rf "$work" + [ "$rc" -eq 0 ] + [ "$passed" -eq 1 ] +} diff --git a/packages/system/monitoring-agents/alerts/workload-placement.yaml b/packages/system/monitoring-agents/alerts/workload-placement.yaml new file mode 100644 index 000000000..3ae9c8f50 --- /dev/null +++ b/packages/system/monitoring-agents/alerts/workload-placement.yaml @@ -0,0 +1,81 @@ +apiVersion: operator.victoriametrics.com/v1beta1 +kind: VMRule +metadata: + name: alerts-workload-placement +spec: + groups: + - name: workload-placement + params: {} + rules: + - alert: PodReplicasColocated + expr: |- + count by (cluster, namespace, owner_kind, owner_name, node) ( + label_replace( + label_replace( + ( + max by (cluster, namespace, pod, node, created_by_kind, created_by_name) ( + kube_pod_info{job="kube-state-metrics", created_by_kind=~"StatefulSet|ReplicaSet|Cluster|StrimziPodSet"} + ) + or + label_replace( + label_replace( + max by (cluster, namespace, pod, node, created_by_kind, created_by_name) ( + kube_pod_info{job="kube-state-metrics", created_by_kind="EtcdMember"} + ), + "created_by_name", "$1", "created_by_name", "(.+)-[^-]+" + ), + "created_by_kind", "EtcdCluster", "", "" + ) + ) + and on (cluster, namespace, pod) + kube_pod_status_phase{job="kube-state-metrics", phase="Running"} == 1, + "owner_kind", "$1", "created_by_kind", "(.+)" + ), + "owner_name", "$1", "created_by_name", "(.+)" + ) + ) > 1 + and on (cluster, namespace, owner_kind, owner_name) + count by (cluster, namespace, owner_kind, owner_name) ( + label_replace( + label_replace( + ( + max by (cluster, namespace, pod, node, created_by_kind, created_by_name) ( + kube_pod_info{job="kube-state-metrics", created_by_kind=~"StatefulSet|ReplicaSet|Cluster|StrimziPodSet"} + ) + or + label_replace( + label_replace( + max by (cluster, namespace, pod, node, created_by_kind, created_by_name) ( + kube_pod_info{job="kube-state-metrics", created_by_kind="EtcdMember"} + ), + "created_by_name", "$1", "created_by_name", "(.+)-[^-]+" + ), + "created_by_kind", "EtcdCluster", "", "" + ) + ) + and on (cluster, namespace, pod) + kube_pod_status_phase{job="kube-state-metrics", phase="Running"} == 1, + "owner_kind", "$1", "created_by_kind", "(.+)" + ), + "owner_name", "$1", "created_by_name", "(.+)" + ) + ) + <= on (cluster) group_left () + count by (cluster) ( + max by (cluster, node) (kube_node_info{job="kube-state-metrics"}) + unless on (cluster, node) + kube_node_spec_taint{job="kube-state-metrics", effect=~"NoSchedule|NoExecute"} + ) + for: 1h + labels: + severity: warning + exported_instance: '{{ $labels.namespace }}/{{ $labels.owner_name }}' + service: workload-placement + annotations: + summary: Pods of one workload share a node. + description: >- + {{ $value }} running pods of {{ $labels.owner_kind }} {{ $labels.namespace }}/{{ $labels.owner_name }} + are on node {{ $labels.node }} in cluster {{ $labels.cluster }}, although the cluster has a + schedulable node for each of its pods. Losing this node stops these pods together, which can cost + a quorum-based workload its quorum. The scheduler does not move running pods, so they stay + together until they are recreated. From 4a706901b501aa0a5d620de73c352659a9768027 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Wed, 21 Oct 2026 04:14:26 +0300 Subject: [PATCH 011/518] fix(cozy-lib)!: refuse memory below 1Mi instead of rendering a few bytes A quantity without a unit counts bytes. The app schemas take a bare number for memory and the dashboard sends it as typed, so 8 meant as 8Gi rendered a memory limit and request of 8 bytes and the app did not work. Charts size their containers through sanitize, directly or from defaultingSanitize, so the floor goes there and the render stops with a message that names the value and asks for a unit. 1Mi is far below the smallest preset, 128Mi, and above any bare number typed for Mi or Gi. 0 is refused as well, since kubelet sets no cgroup memory limit for a zero limit. flatten renders tenant quotas from the same conversion without the floor, because a quota of 0 memory is a valid way to allow none. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../library/cozy-lib/templates/_resources.tpl | 29 ++- .../templates/tests/resources-defaulting.yaml | 11 + .../tests/resources_memory_floor_test.yaml | 203 ++++++++++++++++++ 3 files changed, 242 insertions(+), 1 deletion(-) create mode 100644 packages/tests/cozy-lib-tests/templates/tests/resources-defaulting.yaml create mode 100644 packages/tests/cozy-lib-tests/tests/resources_memory_floor_test.yaml diff --git a/packages/library/cozy-lib/templates/_resources.tpl b/packages/library/cozy-lib/templates/_resources.tpl index 48b0718fd..6ee63af0e 100644 --- a/packages/library/cozy-lib/templates/_resources.tpl +++ b/packages/library/cozy-lib/templates/_resources.tpl @@ -68,6 +68,8 @@ number of milli-CPUs (a CPU quantity finer than 1m is invalid for the VPA admission webhook and meaningless for the scheduler). + A memory below 1Mi fails the render, see cozy-lib.resources.checkMemory. + The template now expects **one flat map** as input (no nested `requests:` / `limits:` sections). Each value in that map is taken as the *limit* for the corresponding resource. Usage example: @@ -92,6 +94,31 @@ devices.com/nvidia: "1" # = limit */}} {{- define "cozy-lib.resources.sanitize" }} +{{- include "cozy-lib.resources.checkMemory" (index . 0) }} +{{- include "cozy-lib.resources.requestsAndLimits" . }} +{{- end }} + +{{- /* + A quantity without a unit counts bytes, so a memory of `8` typed for 8Gi + sizes the container at 8 bytes and nothing else refuses it. Every preset is + 128Mi or more, so a floor of 1Mi only stops that kind of slip. +*/}} +{{- define "cozy-lib.resources.checkMemory" }} +{{- if hasKey . "memory" }} +{{- $memory := index . "memory" }} +{{- if not (kindIs "invalid" $memory) }} +{{- if lt (include "cozy-lib.resources.toFloat" $memory | float64) (include "cozy-lib.resources.toFloat" "1Mi" | float64) }} +{{- fail (printf "memory (%v) must be at least 1Mi. A value without a unit is read as bytes, so add the unit, such as 512Mi or 8Gi." $memory) }} +{{- end }} +{{- end }} +{{- end }} +{{- end }} + +{{- /* + sanitize without the memory floor. flatten renders a ResourceQuota with it, + where a memory of 0 is a valid quota that allows none. +*/}} +{{- define "cozy-lib.resources.requestsAndLimits" }} {{- $cpuAllocationRatio := include "cozy-lib.resources.cpuAllocationRatio" . | float64 }} {{- $memoryAllocationRatio := include "cozy-lib.resources.memoryAllocationRatio" . | float64 }} {{- $ephemeralStorageAllocationRatio := include "cozy-lib.resources.ephemeralStorageAllocationRatio" . | float64 }} @@ -204,7 +231,7 @@ {{- define "cozy-lib.resources.flatten" -}} {{- $out := dict -}} -{{- $res := include "cozy-lib.resources.sanitize" . | fromYaml -}} +{{- $res := include "cozy-lib.resources.requestsAndLimits" . | fromYaml -}} {{- range $section, $values := $res }} {{- range $k, $v := $values }} {{- with include "cozy-lib.resources.flattenResource" (list $section $k) }} diff --git a/packages/tests/cozy-lib-tests/templates/tests/resources-defaulting.yaml b/packages/tests/cozy-lib-tests/templates/tests/resources-defaulting.yaml new file mode 100644 index 000000000..3e6d9a9eb --- /dev/null +++ b/packages/tests/cozy-lib-tests/templates/tests/resources-defaulting.yaml @@ -0,0 +1,11 @@ +{{- with .Values.defaulting }} +apiVersion: v1 +kind: Pod +metadata: + name: {{ $.Release.Name }}-defaulting +spec: + containers: + - name: main + image: busybox + resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .preset .resources $) | nindent 6 }} +{{- end }} diff --git a/packages/tests/cozy-lib-tests/tests/resources_memory_floor_test.yaml b/packages/tests/cozy-lib-tests/tests/resources_memory_floor_test.yaml new file mode 100644 index 000000000..485829054 --- /dev/null +++ b/packages/tests/cozy-lib-tests/tests/resources_memory_floor_test.yaml @@ -0,0 +1,203 @@ +# ./tests/resources_memory_floor_test.yaml +suite: resources memory floor + +# A quantity without a unit counts bytes, so a memory of 8 typed for 8Gi would +# size the container at 8 bytes. sanitize refuses anything below 1Mi, which +# every preset clears by far. flatten renders tenant quotas without the floor, +# because a quota of 0 memory is a valid way to allow none. + +templates: + - templates/tests/resources.yaml + - templates/tests/resources-defaulting.yaml + - templates/tests/quota.yaml + +release: + name: myrelease + namespace: default + +tests: + - it: refuses a quoted number without a unit + template: templates/tests/resources.yaml + set: + resources: + cpu: "1" + memory: "8" + asserts: + - failedTemplate: + errorMessage: memory (8) must be at least 1Mi. A value without a unit is read as bytes, so add the unit, such as 512Mi or 8Gi. + + - it: refuses an unquoted number without a unit + template: templates/tests/resources.yaml + set: + resources: + cpu: "1" + memory: 8 + asserts: + - failedTemplate: + errorPattern: '^memory \(8\) must be at least 1Mi\.' + + - it: refuses a zero memory + template: templates/tests/resources.yaml + set: + resources: + cpu: "1" + memory: 0 + asserts: + - failedTemplate: + errorPattern: '^memory \(0\) must be at least 1Mi\.' + + - it: refuses a byte count one below 1Mi + template: templates/tests/resources.yaml + set: + resources: + cpu: "1" + memory: "1048575" + asserts: + - failedTemplate: + errorPattern: '^memory \(1048575\) must be at least 1Mi\.' + + - it: refuses a suffixed value below 1Mi + template: templates/tests/resources.yaml + set: + resources: + cpu: "1" + memory: 1023Ki + asserts: + - failedTemplate: + errorPattern: '^memory \(1023Ki\) must be at least 1Mi\.' + + - it: refuses a negative memory + template: templates/tests/resources.yaml + set: + resources: + cpu: "1" + memory: -1Gi + asserts: + - failedTemplate: + errorPattern: '^memory \(-1Gi\) must be at least 1Mi\.' + + - it: renders exactly 1Mi + template: templates/tests/resources.yaml + set: + resources: + cpu: "1" + memory: 1Mi + asserts: + - equal: + path: spec.containers[0].resources.limits.memory + value: 1Mi + - equal: + path: spec.containers[0].resources.requests.memory + value: "1048576" + + - it: renders an unquoted byte count of exactly 1Mi + template: templates/tests/resources.yaml + set: + resources: + cpu: "1" + memory: 1048576 + asserts: + - notFailedTemplate: {} + - equal: + path: spec.containers[0].resources.requests.memory + value: "1048576" + + - it: renders 512Mi + template: templates/tests/resources.yaml + set: + resources: + cpu: "1" + memory: 512Mi + asserts: + - equal: + path: spec.containers[0].resources.limits.memory + value: 512Mi + - equal: + path: spec.containers[0].resources.requests.memory + value: "536870912" + + - it: renders 8Gi + template: templates/tests/resources.yaml + set: + resources: + cpu: "1" + memory: 8Gi + asserts: + - equal: + path: spec.containers[0].resources.limits.memory + value: 8Gi + - equal: + path: spec.containers[0].resources.requests.memory + value: "8589934592" + + - it: leaves a map without memory alone + template: templates/tests/resources.yaml + set: + resources: + ephemeral-storage: 2Gi + asserts: + - notFailedTemplate: {} + - notExists: + path: spec.containers[0].resources.limits.memory + + - it: refuses a number without a unit set over a preset + template: templates/tests/resources-defaulting.yaml + set: + defaulting: + preset: t1.nano + resources: + memory: "8" + asserts: + - failedTemplate: + errorPattern: '^memory \(8\) must be at least 1Mi\.' + + - it: renders the smallest preset unchanged + template: templates/tests/resources-defaulting.yaml + set: + defaulting: + preset: t1.nano + asserts: + - equal: + path: spec.containers[0].resources.limits.memory + value: 128Mi + - equal: + path: spec.containers[0].resources.requests.memory + value: "134217728" + + - it: renders the smallest legacy preset unchanged + template: templates/tests/resources-defaulting.yaml + set: + defaulting: + preset: nano + asserts: + - equal: + path: spec.containers[0].resources.limits.memory + value: 128Mi + + - it: renders an explicit memory with a unit over a preset + template: templates/tests/resources-defaulting.yaml + set: + defaulting: + preset: t1.nano + resources: + memory: 8Gi + asserts: + - equal: + path: spec.containers[0].resources.limits.memory + value: 8Gi + - equal: + path: spec.containers[0].resources.limits.cpu + value: 250m + + - it: still renders a tenant quota of 0 memory + template: templates/tests/quota.yaml + set: + quota: + memory: 0 + asserts: + - equal: + path: spec.hard["limits.memory"] + value: "0" + - equal: + path: spec.hard["requests.memory"] + value: "0" From 6bb0fda706e9ccaec5f433f6cd81eb2cdb7e6ef0 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Wed, 21 Oct 2026 04:30:33 +0300 Subject: [PATCH 012/518] fix(cozystack-basics): refuse a direct delete of a tenant namespace A tenant namespace is meant to go away with its Tenant. The tenant chart's pre-delete hook deletes the applications first and the tenant modules such as etcd after them, and helm-controller deletes the namespace at the end. A direct delete hands the namespace to the namespace controller instead. It deletes the pods first and then everything else together, so the tenant etcd and its etcd-ca-tls Secret can be gone before the TenantControlPlanes that use them. Kamaji then cannot reach the datastore to clean up, the TenantControlPlane finalizer never clears and the namespace stays Terminating. The policy refuses DELETE of a namespace labelled tenant.cozystack.io/tenant-root, which the tenant chart puts on every tenant namespace below tenant-root. tenant-root carries no such label and stays with the no-delete guardrail. Two callers are let through. helm-controller and its shards run as the flux ServiceAccount in cozy-fluxcd and delete the namespace during a Tenant uninstall. The garbage collector runs in kube-system and removes a child namespace through its ownerReference once the parent namespace is gone. The namespace controller only finalizes a namespace and never sends a DELETE for it, so it is not affected. Everyone else is refused, system:masters included, as with the no-delete guardrail. The message names the Tenant to delete and the label to remove when the Tenant is already gone. The policy test helpers take the template path now, so the new test compiles the shipped template with the apiserver CEL compiler like the external TLS policy test does. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- hack/e2e-install-cozystack.bats | 37 ++++ .../tenant-namespace-delete-policy.yaml | 59 +++++++ ...ostname-policies-capability-gate_test.yaml | 4 + .../tenant-namespace-delete-policy_test.yaml | 51 ++++++ .../tests/tenant-root_test.yaml | 6 + .../application/external_tls_policy_test.go | 22 +-- .../tenant_namespace_delete_policy_test.go | 163 ++++++++++++++++++ 7 files changed, 331 insertions(+), 11 deletions(-) create mode 100644 packages/system/cozystack-basics/templates/tenant-namespace-delete-policy.yaml create mode 100644 packages/system/cozystack-basics/tests/tenant-namespace-delete-policy_test.yaml create mode 100644 pkg/registry/apps/application/tenant_namespace_delete_policy_test.go diff --git a/hack/e2e-install-cozystack.bats b/hack/e2e-install-cozystack.bats index 47315ac85..4f0884269 100644 --- a/hack/e2e-install-cozystack.bats +++ b/hack/e2e-install-cozystack.bats @@ -561,3 +561,40 @@ data: version: "${version}" EOF } + +@test "Tenant namespace delete policy refuses a direct delete and keeps the Tenant uninstall" { + if ! kubectl api-resources --api-group=admissionregistration.k8s.io \ + 2>/dev/null | grep -qw validatingadmissionpolicies; then + skip "ValidatingAdmissionPolicy API not available on this cluster" + fi + kubectl get validatingadmissionpolicy cozystack-tenant-namespace-delete-policy + kubectl get validatingadmissionpolicybinding cozystack-tenant-namespace-delete-policy + + kubectl apply -f - </dev/null 2>&1; do sleep 2; done' + kubectl wait hr/tenant-deleteprobe -n tenant-root --timeout=1m --for=condition=ready + + local selected output rc still_there + selected=$(kubectl get namespaces --selector tenant.cozystack.io/tenant-root --field-selector metadata.name=tenant-deleteprobe -o name) + [ "$selected" = "namespace/tenant-deleteprobe" ] + + output=$(kubectl delete namespace tenant-deleteprobe --dry-run=server 2>&1) && rc=0 || rc=$? + echo "kubectl delete exit=$rc, output=$output" + [ "$rc" -ne 0 ] + echo "$output" | grep -qF "ValidatingAdmissionPolicy 'cozystack-tenant-namespace-delete-policy'" + echo "$output" | grep -qF 'kubectl delete tenants.apps.cozystack.io deleteprobe --namespace tenant-root.' + + kubectl delete tenants.apps.cozystack.io deleteprobe -n tenant-root --timeout=5m + kubectl wait namespace/tenant-deleteprobe --for=delete --timeout=5m + if ! still_there=$(kubectl get namespace tenant-deleteprobe --ignore-not-found -o name); then + echo "FAIL: reading the tenant-deleteprobe namespace did not answer"; false + fi + if [ -n "$still_there" ]; then echo "FAIL: deleting the Tenant must remove its namespace"; false; fi +} diff --git a/packages/system/cozystack-basics/templates/tenant-namespace-delete-policy.yaml b/packages/system/cozystack-basics/templates/tenant-namespace-delete-policy.yaml new file mode 100644 index 000000000..cbc928dc4 --- /dev/null +++ b/packages/system/cozystack-basics/templates/tenant-namespace-delete-policy.yaml @@ -0,0 +1,59 @@ +{{- if .Capabilities.APIVersions.Has "admissionregistration.k8s.io/v1/ValidatingAdmissionPolicy" }} +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicy +metadata: + name: cozystack-tenant-namespace-delete-policy + labels: + internal.cozystack.io/managed-by-cozystack: "" +spec: + failurePolicy: Fail + matchConstraints: + resourceRules: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["DELETE"] + resources: ["namespaces"] + variables: + - name: platformCaller + expression: >- + has(request.userInfo.groups) && request.userInfo.groups.exists(g, + g == "system:serviceaccounts:cozy-fluxcd" || + g == "system:serviceaccounts:kube-system") + - name: name + expression: oldObject.metadata.name + - name: cut + expression: variables.name.lastIndexOf("-") + - name: tenant + expression: variables.name.substring(variables.cut + 1) + - name: parentNamespace + expression: >- + variables.cut > 0 && variables.name.substring(0, variables.cut) != "tenant" + ? variables.name.substring(0, variables.cut) + : "tenant-root" + validations: + - expression: variables.platformCaller + messageExpression: >- + "Namespace " + variables.name + " belongs to the Tenant " + variables.tenant + + " in " + variables.parentNamespace + ". Delete the Tenant instead with kubectl delete tenants.apps.cozystack.io " + + variables.tenant + " --namespace " + variables.parentNamespace + + ". Deleting the namespace directly can remove the tenant etcd before the Kubernetes clusters that use it and leave the namespace stuck in Terminating." + + " If the Tenant is already gone, remove the label first with kubectl label namespace " + variables.name + + " tenant.cozystack.io/tenant-root-" + reason: Forbidden +--- +apiVersion: admissionregistration.k8s.io/v1 +kind: ValidatingAdmissionPolicyBinding +metadata: + name: cozystack-tenant-namespace-delete-policy + labels: + internal.cozystack.io/managed-by-cozystack: "" +spec: + policyName: cozystack-tenant-namespace-delete-policy + validationActions: [Deny] + matchResources: + objectSelector: + matchExpressions: + - key: tenant.cozystack.io/tenant-root + operator: Exists +{{- end }} diff --git a/packages/system/cozystack-basics/tests/hostname-policies-capability-gate_test.yaml b/packages/system/cozystack-basics/tests/hostname-policies-capability-gate_test.yaml index 61b8be507..96cfdfaa3 100644 --- a/packages/system/cozystack-basics/tests/hostname-policies-capability-gate_test.yaml +++ b/packages/system/cozystack-basics/tests/hostname-policies-capability-gate_test.yaml @@ -1,5 +1,6 @@ suite: cozystack-basics VAP policies are gated on the ValidatingAdmissionPolicy API templates: + - templates/tenant-namespace-delete-policy.yaml - templates/route-hostname-policy.yaml - templates/gateway-hostname-policy.yaml - templates/ingress-hostname-policy.yaml @@ -20,6 +21,9 @@ capabilities: tests: - it: none of the VAP policies render when the API is unavailable asserts: + - hasDocuments: + count: 0 + template: templates/tenant-namespace-delete-policy.yaml - hasDocuments: count: 0 template: templates/route-hostname-policy.yaml diff --git a/packages/system/cozystack-basics/tests/tenant-namespace-delete-policy_test.yaml b/packages/system/cozystack-basics/tests/tenant-namespace-delete-policy_test.yaml new file mode 100644 index 000000000..a0e53c77f --- /dev/null +++ b/packages/system/cozystack-basics/tests/tenant-namespace-delete-policy_test.yaml @@ -0,0 +1,51 @@ +suite: tenant namespace delete policy renders with its binding +templates: + - templates/tenant-namespace-delete-policy.yaml +release: + name: cozystack-basics + namespace: cozy-system +capabilities: + apiVersions: + - admissionregistration.k8s.io/v1/ValidatingAdmissionPolicy +tests: + - it: renders the policy and a Deny binding to it + asserts: + - hasDocuments: + count: 2 + - documentIndex: 0 + equal: + path: kind + value: ValidatingAdmissionPolicy + - documentIndex: 0 + equal: + path: metadata.name + value: cozystack-tenant-namespace-delete-policy + - documentIndex: 1 + equal: + path: spec.policyName + value: cozystack-tenant-namespace-delete-policy + - documentIndex: 1 + equal: + path: spec.validationActions + value: [Deny] + + - it: matches only DELETE of core namespaces + asserts: + - documentIndex: 0 + equal: + path: spec.matchConstraints.resourceRules + value: + - apiGroups: [""] + apiVersions: ["v1"] + operations: ["DELETE"] + resources: ["namespaces"] + + - it: binds to the namespaces the tenant chart labels as descendants of tenant-root + asserts: + - documentIndex: 1 + equal: + path: spec.matchResources.objectSelector + value: + matchExpressions: + - key: tenant.cozystack.io/tenant-root + operator: Exists diff --git a/packages/system/cozystack-basics/tests/tenant-root_test.yaml b/packages/system/cozystack-basics/tests/tenant-root_test.yaml index e22dbb649..54210174c 100644 --- a/packages/system/cozystack-basics/tests/tenant-root_test.yaml +++ b/packages/system/cozystack-basics/tests/tenant-root_test.yaml @@ -79,3 +79,9 @@ tests: equal: path: spec.chartRef.name value: cozystack-tenant-application-default-tenant + + - it: tenant-root namespace carries no tenant.cozystack.io/tenant-root label, which the tenant namespace delete policy selects on + asserts: + - documentIndex: 0 + notExists: + path: metadata.labels["tenant.cozystack.io/tenant-root"] diff --git a/pkg/registry/apps/application/external_tls_policy_test.go b/pkg/registry/apps/application/external_tls_policy_test.go index 1cbc808fc..ff5704a23 100644 --- a/pkg/registry/apps/application/external_tls_policy_test.go +++ b/pkg/registry/apps/application/external_tls_policy_test.go @@ -41,14 +41,14 @@ import ( // expressions a cluster actually receives rather than a copy of them. const externalTLSPolicyPath = "../../../../packages/system/cozystack-basics/templates/database-external-tls-policy.yaml" -// loadExternalTLSPolicy returns the policy and its binding from the chart -// template. The template's only directives are the capability guard lines -// around the manifests; any other directive would mean the test is no longer -// looking at what renders, so it fails instead of guessing. -func loadExternalTLSPolicy(t *testing.T) (*admissionregistrationv1.ValidatingAdmissionPolicy, *admissionregistrationv1.ValidatingAdmissionPolicyBinding) { +// loadPolicy returns the policy and its binding from a chart template. The +// template's only directives are the capability guard lines around the +// manifests; any other directive would mean the test is no longer looking at +// what renders, so it fails instead of guessing. +func loadPolicy(t *testing.T, path string) (*admissionregistrationv1.ValidatingAdmissionPolicy, *admissionregistrationv1.ValidatingAdmissionPolicyBinding) { t.Helper() - raw, err := os.ReadFile(externalTLSPolicyPath) + raw, err := os.ReadFile(path) if err != nil { t.Fatalf("read policy template: %v", err) } @@ -92,11 +92,11 @@ func loadExternalTLSPolicy(t *testing.T) (*admissionregistrationv1.ValidatingAdm return policy, binding } -// compileExternalTLSPolicy mirrors compilePolicy in +// compilePolicy mirrors compilePolicy in // k8s.io/apiserver/pkg/admission/plugin/policy/validating, which is // unexported: the same composition environment, the same variable, match // condition, validation and message compilers. -func compileExternalTLSPolicy(t *testing.T, policy *admissionregistrationv1.ValidatingAdmissionPolicy) validating.Validator { +func compilePolicy(t *testing.T, policy *admissionregistrationv1.ValidatingAdmissionPolicy) validating.Validator { t.Helper() compiler, err := plugincel.NewCompositedCompiler(environment.MustBaseEnvSet(environment.DefaultCompatibilityVersion())) @@ -139,7 +139,7 @@ func compileExternalTLSPolicy(t *testing.T, policy *admissionregistrationv1.Vali } func TestDatabaseExternalTLSPolicy_Binding(t *testing.T) { - policy, binding := loadExternalTLSPolicy(t) + policy, binding := loadPolicy(t, externalTLSPolicyPath) if binding.Spec.PolicyName != policy.Name { t.Errorf("binding names policy %q, want %q", binding.Spec.PolicyName, policy.Name) @@ -171,8 +171,8 @@ func TestDatabaseExternalTLSPolicy_Binding(t *testing.T) { } func TestDatabaseExternalTLSPolicy_Decisions(t *testing.T) { - policy, _ := loadExternalTLSPolicy(t) - validator := compileExternalTLSPolicy(t, policy) + policy, _ := loadPolicy(t, externalTLSPolicyPath) + validator := compilePolicy(t, policy) tlsOn := map[string]any{"enabled": true} tlsOff := map[string]any{"enabled": false} diff --git a/pkg/registry/apps/application/tenant_namespace_delete_policy_test.go b/pkg/registry/apps/application/tenant_namespace_delete_policy_test.go new file mode 100644 index 000000000..6c8870ea2 --- /dev/null +++ b/pkg/registry/apps/application/tenant_namespace_delete_policy_test.go @@ -0,0 +1,163 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package application + +import ( + "context" + "slices" + "strings" + "testing" + + admissionregistrationv1 "k8s.io/api/admissionregistration/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/labels" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apiserver/pkg/admission" + "k8s.io/apiserver/pkg/admission/plugin/policy/validating" + celconfig "k8s.io/apiserver/pkg/apis/cel" + "k8s.io/apiserver/pkg/authentication/user" +) + +const tenantNamespaceDeletePolicyPath = "../../../../packages/system/cozystack-basics/templates/tenant-namespace-delete-policy.yaml" + +func TestTenantNamespaceDeletePolicy_Binding(t *testing.T) { + policy, binding := loadPolicy(t, tenantNamespaceDeletePolicyPath) + + if binding.Spec.PolicyName != policy.Name { + t.Errorf("binding names policy %q, want %q", binding.Spec.PolicyName, policy.Name) + } + if !slices.Equal(binding.Spec.ValidationActions, []admissionregistrationv1.ValidationAction{admissionregistrationv1.Deny}) { + t.Errorf("binding validationActions = %v, want [Deny]", binding.Spec.ValidationActions) + } + if policy.Spec.FailurePolicy == nil || *policy.Spec.FailurePolicy != admissionregistrationv1.Fail { + t.Errorf("failurePolicy must be Fail") + } + + rules := policy.Spec.MatchConstraints.ResourceRules + if len(rules) != 1 { + t.Fatalf("want one resource rule, got %d", len(rules)) + } + rule := rules[0] + if !slices.Equal(rule.APIGroups, []string{""}) || !slices.Equal(rule.APIVersions, []string{"v1"}) { + t.Errorf("rule must match core v1 only, got apiGroups %v apiVersions %v", rule.APIGroups, rule.APIVersions) + } + if !slices.Equal(rule.Operations, []admissionregistrationv1.OperationType{admissionregistrationv1.Delete}) { + t.Errorf("operations = %v, want [DELETE]", rule.Operations) + } + if !slices.Equal(rule.Resources, []string{"namespaces"}) { + t.Errorf("resources = %v, want [namespaces]", rule.Resources) + } + + if binding.Spec.MatchResources == nil || binding.Spec.MatchResources.ObjectSelector == nil { + t.Fatal("binding must select namespaces by label") + } + selector, err := metav1.LabelSelectorAsSelector(binding.Spec.MatchResources.ObjectSelector) + if err != nil { + t.Fatalf("binding objectSelector: %v", err) + } + selected := []struct { + name string + labels labels.Set + want bool + }{ + {name: "tenant namespace under tenant-root", labels: labels.Set{"tenant.cozystack.io/tenant-root": "", "tenant.cozystack.io/tenant-foo": "", "namespace.cozystack.io/host": "foo.example.org"}, want: true}, + {name: "nested tenant namespace", labels: labels.Set{"tenant.cozystack.io/tenant-root": "", "tenant.cozystack.io/tenant-foo": "", "tenant.cozystack.io/tenant-foo-bar": ""}, want: true}, + {name: "tenant-root namespace", labels: labels.Set{"platform.cozystack.io/no-delete": "true", "namespace.cozystack.io/host": "example.org", "namespace.cozystack.io/gateway": "tenant-root"}}, + {name: "namespace named tenant-* without tenant labels", labels: labels.Set{"namespace.cozystack.io/host": "case-probe.example.org"}}, + {name: "unlabelled namespace", labels: labels.Set{}}, + } + for _, tc := range selected { + if got := selector.Matches(tc.labels); got != tc.want { + t.Errorf("%s: selected = %v, want %v", tc.name, got, tc.want) + } + } +} + +func TestTenantNamespaceDeletePolicy_Decisions(t *testing.T) { + policy, _ := loadPolicy(t, tenantNamespaceDeletePolicyPath) + validator := compilePolicy(t, policy) + + fluxGroups := []string{"system:serviceaccounts", "system:serviceaccounts:cozy-fluxcd", "system:authenticated"} + kubeSystemGroups := []string{"system:serviceaccounts", "system:serviceaccounts:kube-system", "system:authenticated"} + adminGroups := []string{"system:masters", "system:authenticated"} + + cases := []struct { + name string + namespace string + user string + groups []string + denied bool + tenant string + parent string + }{ + {name: "helm-controller uninstalling a Tenant", namespace: "tenant-foo", user: "system:serviceaccount:cozy-fluxcd:flux", groups: fluxGroups}, + {name: "helm-controller uninstalling a nested Tenant", namespace: "tenant-foo-bar", user: "system:serviceaccount:cozy-fluxcd:flux", groups: fluxGroups}, + {name: "garbage collector following a deleted parent namespace", namespace: "tenant-foo-bar", user: "system:serviceaccount:kube-system:generic-garbage-collector", groups: kubeSystemGroups}, + {name: "cluster admin deleting a tenant namespace", namespace: "tenant-foo", user: "admin", groups: adminGroups, denied: true, tenant: "foo", parent: "tenant-root"}, + {name: "cluster admin deleting a nested tenant namespace", namespace: "tenant-foo-bar", user: "admin", groups: adminGroups, denied: true, tenant: "bar", parent: "tenant-foo"}, + {name: "cluster admin deleting a third-level tenant namespace", namespace: "tenant-foo-bar-baz", user: "admin", groups: adminGroups, denied: true, tenant: "baz", parent: "tenant-foo-bar"}, + {name: "cluster admin deleting a labelled namespace without a dash", namespace: "standalone", user: "admin", groups: adminGroups, denied: true, tenant: "standalone", parent: "tenant-root"}, + {name: "OIDC user bound to cluster-admin", namespace: "tenant-foo", user: "cluster-admin-user", groups: []string{"cozystack-cluster-admin", "system:authenticated"}, denied: true, tenant: "foo", parent: "tenant-root"}, + {name: "cozy-system service account", namespace: "tenant-foo", user: "system:serviceaccount:cozy-system:cozystack-api", groups: []string{"system:serviceaccounts", "system:serviceaccounts:cozy-system", "system:authenticated"}, denied: true, tenant: "foo", parent: "tenant-root"}, + {name: "tenant service account", namespace: "tenant-foo-bar", user: "system:serviceaccount:tenant-foo:default", groups: []string{"system:serviceaccounts", "system:serviceaccounts:tenant-foo", "system:authenticated"}, denied: true, tenant: "bar", parent: "tenant-foo"}, + {name: "user without groups", namespace: "tenant-foo", user: "no-groups", denied: true, tenant: "foo", parent: "tenant-root"}, + } + + gvk := schema.GroupVersionKind{Version: "v1", Kind: "Namespace"} + gvr := schema.GroupVersionResource{Version: "v1", Resource: "namespaces"} + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + oldObj := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "v1", + "kind": "Namespace", + "metadata": map[string]any{ + "name": tc.namespace, + "labels": map[string]any{"tenant.cozystack.io/tenant-root": ""}, + }, + }} + var newObj runtime.Object + attrs := admission.NewAttributesRecord(newObj, oldObj, gvk, tc.namespace, tc.namespace, gvr, "", admission.Delete, &metav1.DeleteOptions{}, false, &user.DefaultInfo{Name: tc.user, Groups: tc.groups}) + versioned := &admission.VersionedAttributes{Attributes: attrs, VersionedObject: admission.NewLazyObject(newObj), VersionedOldObject: admission.NewLazyObject(oldObj), VersionedKind: gvk} + + result := validator.Validate(context.Background(), gvr, versioned, nil, nil, celconfig.RuntimeCELCostBudget, nil) + var denials []validating.PolicyDecision + for _, d := range result.Decisions { + if d.Evaluation == validating.EvalError { + t.Fatalf("evaluation error: %s", d.Message) + } + if d.Action == validating.ActionDeny { + denials = append(denials, d) + } + } + if got := len(denials) > 0; got != tc.denied { + t.Fatalf("denied = %v, want %v (decisions %+v)", got, tc.denied, result.Decisions) + } + for _, d := range denials { + for _, want := range []string{ + "kubectl delete tenants.apps.cozystack.io " + tc.tenant + " --namespace " + tc.parent + ".", + "kubectl label namespace " + tc.namespace + " tenant.cozystack.io/tenant-root-", + } { + if !strings.Contains(d.Message, want) { + t.Errorf("denial message must carry %q, got %q", want, d.Message) + } + } + } + }) + } +} From b83cc087cc4828fa8236de8347e22af4639ddf29 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 21 Oct 2026 15:48:13 +0500 Subject: [PATCH 013/518] docs: move the repo-only guides onto the website Seven guides in docs/ were written next to the code and left in the source tree, where no operator looks. The website carries them now, so a second copy here only lets the two drift again: the Backup Classes page had already fallen behind its original. docs/release.md, docs/deprecating-applications.md and the two SeaweedFS runbooks in docs/operations stay. A migration, a bats suite, a workflow or AGENTS.md reads each of them by path, which is the test for keeping a document in this repository. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- api/apps/v1alpha1/mongodb/types.go | 2 +- api/backups/v1alpha1/DESIGN.md | 2 +- cmd/cozystack-operator/main.go | 4 +- docs/gateway-classes.md | 125 ----- docs/observability/distributed-tracing.md | 123 ----- docs/operations/backup-classes.md | 461 ------------------ docs/operations/resource-presets.md | 52 -- .../seaweedfs-capacity-for-backups.md | 37 -- docs/operations/system-memory-limits.md | 179 ------- docs/storage-immutability.md | 45 -- examples/backups/clickhouse/README.md | 2 +- examples/backups/etcd/README.md | 2 +- examples/backups/mariadb/README.md | 2 +- examples/backups/mongodb/05-mongodb-src.yaml | 2 +- examples/backups/mongodb/README.md | 2 +- .../postgres/35-restorejob-in-place.yaml | 2 +- .../backups/postgres/45-restorejob-pitr.yaml | 2 +- hack/check-rd-presets.sh | 2 +- hack/migration-50-etcd-adopt.bats | 4 +- .../backupcontroller/backupjob_controller.go | 4 +- .../backupcontroller/credentials_projector.go | 2 +- .../psmdbstrategy_controller.go | 8 +- packages/apps/clickhouse/README.md | 2 +- packages/apps/foundationdb/README.md | 2 +- packages/apps/harbor/README.md | 2 +- packages/apps/kafka/README.md | 2 +- .../kubernetes-nodes/templates/_helpers.tpl | 2 +- packages/apps/kubernetes/README.md | 2 +- packages/apps/mariadb/README.md | 4 +- packages/apps/mongodb/README.md | 26 +- packages/apps/mongodb/templates/mongodb.yaml | 2 +- packages/apps/mongodb/templates/restore.yaml | 2 +- packages/apps/mongodb/tests/mongodb_test.yaml | 2 +- packages/apps/mongodb/tests/restore_test.yaml | 2 +- packages/apps/mongodb/values.schema.json | 2 +- packages/apps/mongodb/values.yaml | 2 +- packages/apps/nats/README.md | 4 +- packages/apps/openbao/README.md | 2 +- packages/apps/opensearch/README.md | 2 +- packages/apps/postgres/README.md | 4 +- packages/apps/postgres/templates/db.yaml | 4 +- .../postgres/tests/backup_storage_test.yaml | 2 +- packages/apps/qdrant/README.md | 4 +- packages/apps/rabbitmq/README.md | 6 +- packages/apps/redis/README.md | 4 +- packages/apps/tcp-balancer/README.md | 2 +- packages/apps/valkey/README.md | 4 +- packages/apps/vm-disk/README.md | 2 +- packages/apps/vm-disk/templates/dv.yaml | 4 +- .../vm-disk/tests/dv_immutability_test.yaml | 2 +- packages/apps/vpn/README.md | 2 +- packages/core/installer/values.yaml | 2 +- .../core/platform/templates/bundles/iaas.yaml | 2 +- packages/core/platform/values.yaml | 4 +- packages/extra/gateway/README.md | 2 +- .../cozy-lib/templates/_resourcepresets.tpl | 2 +- .../templates/backupclass-default.yaml | 2 +- .../templates/rbac.yaml | 2 +- .../templates/strategy-kafka-default.yaml | 2 +- .../templates/strategy-mongodb-default.yaml | 4 +- .../templates/strategy-redis-default.yaml | 6 +- .../system/mongodb-rd/cozyrds/mongodb.yaml | 2 +- packages/system/monitoring-agents/values.yaml | 2 +- pkg/apis/apps/presets/legacy.go | 2 +- 64 files changed, 89 insertions(+), 1111 deletions(-) delete mode 100644 docs/gateway-classes.md delete mode 100644 docs/observability/distributed-tracing.md delete mode 100644 docs/operations/backup-classes.md delete mode 100644 docs/operations/resource-presets.md delete mode 100644 docs/operations/seaweedfs-capacity-for-backups.md delete mode 100644 docs/operations/system-memory-limits.md delete mode 100644 docs/storage-immutability.md diff --git a/api/apps/v1alpha1/mongodb/types.go b/api/apps/v1alpha1/mongodb/types.go index 44764350c..1ef060b85 100644 --- a/api/apps/v1alpha1/mongodb/types.go +++ b/api/apps/v1alpha1/mongodb/types.go @@ -84,7 +84,7 @@ type Backup struct { // Cron schedule for automated backups. Ignored when useSystemBucket is true (scheduling moves to the BackupClass Plan). // +kubebuilder:default:="0 2 * * *" Schedule string `json:"schedule,omitempty"` - // Opt-in: back up to the platform system bucket without supplying S3 credentials. The cozy-default MongoDB BackupClass injects the storage at BackupJob time; tenants do not set s3AccessKey/s3SecretKey/destinationPath/endpointURL, and backups scope to `/`. Scheduling moves to a BackupClass Plan; `schedule`, `retentionPolicy` and PITR are ignored while this is on, and this flow has no retention yet (see docs/operations/backup-classes.md). Requires `backup.enabled: true` (the pbm agents run on that flag); the chart refuses the combination with `enabled: false` at render time; the schema carries the same rule as a marker for the UI and for the day the apiserver enforces CEL on application schemas (cozystack/cozystack#2657), which it does not yet. A release that has not yet rendered this flag (its cluster still carries the chart's scheduled tasks and PITR) holds a BackupJob until it does, so the tenant's own storage is never injected over. + // Opt-in: back up to the platform system bucket without supplying S3 credentials. The cozy-default MongoDB BackupClass injects the storage at BackupJob time; tenants do not set s3AccessKey/s3SecretKey/destinationPath/endpointURL, and backups scope to `/`. Scheduling moves to a BackupClass Plan; `schedule`, `retentionPolicy` and PITR are ignored while this is on, and this flow has no retention yet (see https://cozystack.io/docs/v1/operations/services/backup-classes/). Requires `backup.enabled: true` (the pbm agents run on that flag); the chart refuses the combination with `enabled: false` at render time; the schema carries the same rule as a marker for the UI and for the day the apiserver enforces CEL on application schemas (cozystack/cozystack#2657), which it does not yet. A release that has not yet rendered this flag (its cluster still carries the chart's scheduled tasks and PITR) holds a BackupJob until it does, so the tenant's own storage is never injected over. // +kubebuilder:default:=false UseSystemBucket bool `json:"useSystemBucket,omitempty"` } diff --git a/api/backups/v1alpha1/DESIGN.md b/api/backups/v1alpha1/DESIGN.md index 628b0d7b4..faaffbcde 100644 --- a/api/backups/v1alpha1/DESIGN.md +++ b/api/backups/v1alpha1/DESIGN.md @@ -31,7 +31,7 @@ Tenants reference `cozy-default` from `BackupJob`/`Plan`/`RestoreJob`; they neve The platform-managed flow is layered *on top of* — not in place of — the core API. Custom `BackupClass`/Strategy CRs targeting other buckets continue to work; tenants picking `cozy-default` simply opt into the platform's pre-wired stack. -**Open gap (FoundationDB):** the `cozy-default-foundationdb` Strategy CR is shipped but **not** bound by `cozy-default`. Restore for FoundationDB runs `fdbrestore` from inside the `cozy-foundationdb-operator` deployment, which does not yet mount `cozy-backups-creds` with the blob credentials file. Admins who want platform-default FDB backups today must either wire the operator deployment manually or use a custom BackupClass — see [Backup Classes](../../../docs/operations/backup-classes.md) for the follow-up plan. +**Open gap (FoundationDB):** the `cozy-default-foundationdb` Strategy CR is shipped but **not** bound by `cozy-default`. Restore for FoundationDB runs `fdbrestore` from inside the `cozy-foundationdb-operator` deployment, which does not yet mount `cozy-backups-creds` with the blob credentials file. Admins who want platform-default FDB backups today must either wire the operator deployment manually or use a custom BackupClass — see [Backup Classes](https://cozystack.io/docs/v1/operations/services/backup-classes/) for the follow-up plan. **Tenant visibility:** projected Secrets carry no `internal.cozystack.io/tenantresource` label, so the lineage-controller webhook does not promote them to a `TenantSecret` view. Combined with the default RBAC (no core/v1.Secret verbs on tenant ServiceAccounts), the projected credentials are unreadable from a tenant kubeconfig even when they sit in the tenant's own namespace; the operator Pods mount the Secret via kubelet, which bypasses tenant RBAC. diff --git a/cmd/cozystack-operator/main.go b/cmd/cozystack-operator/main.go index 8ffa37411..07e3ca292 100644 --- a/cmd/cozystack-operator/main.go +++ b/cmd/cozystack-operator/main.go @@ -152,10 +152,10 @@ func main() { "Default container memory limit in eligible system namespaces, including kube-system. "+ "Lowering or an oversized request creates a durable hold requiring an administrator's "+ "exact-value acknowledgement. Empty or 0 disables and removes managed LimitRanges; "+ - "safety state is retained. See docs/operations/system-memory-limits.md.") + "safety state is retained. See https://cozystack.io/docs/v1/operations/configuration/system-memory-limits/.") flag.StringVar(&systemNamespaceMemoryRequest, "system-namespace-memory-request", DefaultSystemNamespaceMemoryRequest, "Default memory request paired with the system namespace limit. Keep small; 0 is supported. "+ - "An empty installer value selects the operator default. See docs/operations/system-memory-limits.md.") + "An empty installer value selects the operator default. See https://cozystack.io/docs/v1/operations/configuration/system-memory-limits/.") opts := zap.Options{ Development: true, diff --git a/docs/gateway-classes.md b/docs/gateway-classes.md deleted file mode 100644 index 5aafb31b2..000000000 --- a/docs/gateway-classes.md +++ /dev/null @@ -1,125 +0,0 @@ -# GatewayClass selection and TLS termination for tenant Gateways - -A tenant that owns a Gateway (`tenant.spec.gateway: true`) gets one `Gateway` object bound to one `GatewayClass`. Which class that is decides who programs the dataplane and, with it, whether TLS is terminated by the Gateway itself or by something in front of it. This page covers the operator-facing surface: the platform values that bound the choice, the tenant field that makes it, what each resulting certificate mode implies, what a move between classes does to certificates and Secrets, and how the mechanism fails when the configured names and the installed classes disagree. - -The chart-level reference for the per-tenant Gateway — inheritance, listener shapes, the security model, ACME rate limits — is [`packages/extra/gateway/README.md`](../packages/extra/gateway/README.md). - -## Platform values - -Three keys under `gateway` in the platform values bound which class a tenant Gateway can land on: - -```yaml -gateway: - enabled: true - className: cilium - tenantSelectableClasses: [] - edgeTerminatedClasses: [] -``` - -- `className` — the class every tenant Gateway uses unless the tenant names another one. Defaults to the Cilium class the platform ships. -- `tenantSelectableClasses` — the additional class names a tenant may name for itself. Empty (the default) means a tenant cannot name anything but the platform default. -- `edgeTerminatedClasses` — the class names whose provider terminates TLS upstream of the Gateway. Membership in this list is the only thing that puts a tenant into edge mode. - -All three reach the per-tenant gateway chart through the platform values channel, as `_cluster.gateway-class-name`, `_cluster.gateway-tenant-classes` and `_cluster.gateway-edge-classes`. The two lists are joined with commas on the way, so a name containing a comma is rejected at platform render time rather than silently split into two names. - -`gateway.enabled` is a separate switch and not a prerequisite for any of this: it moves the platform's own published services from Ingress onto Gateway API and points the ACME `ClusterIssuer`s' HTTP-01 solver at the publishing tenant's Gateway. A tenant Gateway is created from `tenant.spec.gateway` alone. - -## How a tenant picks a class - -```yaml -apiVersion: apps.cozystack.io/v1alpha1 -kind: Tenant -metadata: - name: acme - namespace: tenant-root -spec: - gateway: true - gatewayClass: edge-tunnel -``` - -`edge-tunnel` throughout this page stands for the name of a GatewayClass whose controller terminates TLS outside the cluster and forwards cleartext to the Gateway. Substitute the name your own edge-terminating controller registers. - -The field is read only for a tenant that owns a Gateway. A tenant that inherits — `gateway` left unset or set to `false` — publishes through the nearest ancestor that owns one and runs on that ancestor's class, so its own `gatewayClass` has no effect. Leaving `gatewayClass` empty selects `gateway.className`. - -The set a tenant may name is `gateway.tenantSelectableClasses` plus the current `gateway.className`, so a tenant may always name the default explicitly. Anything else fails that tenant's own gateway release at render time, with a message naming the class and the allowed set, and reaches no other tenant. The allowlist exists because `Tenant` is a tenant-writable object while a `GatewayClass` is cluster-scoped: without it a tenant could attach its Gateway to any class the cluster registers, including one an admin keeps for internal-only traffic. - -## What the class decides: the certificate mode - -`TenantGateway.spec.certMode` is derived, never written by the tenant. The gateway chart resolves it in this order, first match winning: - -| Condition | `certMode` | What the controller issues | -| --- | --- | --- | -| the resolved class is in `gateway.edgeTerminatedClasses` | `edge` | nothing | -| `publishing.certificates.wildcardSecretName` is set | `existingSecret` | nothing; the operator's Secret is referenced | -| `publishing.certificates.solver: dns01` | `dns01` | one wildcard `Certificate` per tenant | -| otherwise | `http01` | one `Certificate` per published hostname, except the passthrough names below | - -Under `http01` a hostname held by a TLS-passthrough listener is the exception to the per-hostname rule. The hostname earns neither an HTTPS-terminate listener nor a `Certificate`, whether or not a `TLSRoute` is attached to that listener yet, because the backend presents its own certificate and the Gateway never holds a key for the name. The shipped `tlsPassthroughServices` defaults put `api`, `vm-exportproxy` and `cdi-uploadproxy` in that position on the publishing tenant's Gateway, for the names `publishing.exposedServices` lists while `gateway.enabled` is on, and the `Certificate` objects that covered them are collected on upgrade. No other tenant renders those listeners, so a child tenant's `HTTPRoute` on one of those names keeps its listener. `packages/extra/gateway/README.md` carries what that changes and the ways forward. - -Edge therefore wins over an operator-supplied wildcard Secret, which wins over the solver. In edge mode the solver, DNS-01 provider and issuer-name inputs are not read at all, so an unsupported value in any of them does not fail the tenant's release. `existingSecret` skips the same validations; only `http01` and `dns01` reject a bad solver, provider or issuer name. - -## What edge mode changes on the Gateway - -- The Gateway serves port 80 only. Its listeners are `edge` for `*.`, `edge-apex` for ``, and one `edge-child-*` per inheriting child tenant carrying that child's `*.`. There is no port-443 listener, and no hostname-less port-80 listener either. -- No per-tenant ACME `Issuer` and no `Certificate` are created. -- No http-to-https redirect `HTTPRoute` is created. The edge is expected to perform that redirect. -- `tlsPassthroughServices` is not written onto the `TenantGateway` and no passthrough listener is rendered, so the tenant Kubernetes API, VM export and CDI upload endpoints are not published through such a Gateway. The packages that own those endpoints still create their `TLSRoute` objects, which then match no listener on that Gateway. -- Application `HTTPRoute`s attach by hostname exactly as in the other modes — the edge listeners carry the same namespace selector the HTTPS listeners carry elsewhere. - -That last point has a blast radius worth stating on its own, and it applies only where those endpoints are published through the Gateway at all. Each of them renders its `TLSRoute` only when `gateway.enabled` is on and its name is still in `publishing.exposedServices`; at the default `gateway.enabled: false` there is no such route and nothing to lose. Where both hold, setting `gateway.className` to an edge-terminated class puts the publishing tenant on it too, and then the endpoints that go unpublished are the platform's own: the `TLSRoute` for `api.` is pinned to a `tls-api` listener, an edge Gateway has no such listener, so the tenant Kubernetes API stops being served — VM export and CDI upload go the same way. Nothing fails at render time, and the loss surfaces only on the routes themselves, which the controller flips to `Accepted=False` with reason `NoMatchingParent`. Putting only child tenants on the edge class avoids it, because the platform's routes keep pointing at the publishing tenant's Gateway. - -Listing a class in `gateway.edgeTerminatedClasses` is an assertion about that class, and nothing verifies it. Put the bundled `cilium` class in the list and its Gateways serve every application hostname over plain HTTP on a public address, with no redirect and no certificate. Only list a class whose provider really does terminate TLS in front of the Gateway. - -## Moving a tenant between classes - -Onto an edge-terminated class, on the next reconcile: - -| Object | What happens | -| --- | --- | -| the per-tenant ACME `Issuer` | deleted | -| per-listener and wildcard `Certificate` objects | deleted | -| the TLS Secrets those `Certificate`s issued | left in the namespace, unreferenced | -| the http-to-https redirect `HTTPRoute` | deleted | -| the replica of `publishing.certificates.wildcardSecretName` | deleted from that tenant's namespace | -| route status conditions the controller wrote earlier | an attached `TLSRoute` that declares hostnames is flipped to `Accepted=False`, reason `NoMatchingParent`; on a hostname-less `TLSRoute`, and on every attached `HTTPRoute`, the controller's own entry is removed instead | - -Every deletion is guarded by ownership: an object of the same name that the controller did not create is left in place. - -The TLS Secrets survive because cert-manager ships here with `enableCertificateOwnerRef: false`, so a `Certificate` does not own the Secret it issued into. That is how every mode transition behaves and is not something edge introduces. Removing the leftovers is an operator action. - -The wildcard replica is withdrawn because the namespace stops counting as a TLS termination point. Three things keep it anyway: a namespace that also owns an ingress controller still terminates locally; a namespace holding more than one `TenantGateway` keeps the replica while any of them is not on an edge-terminated class; and a namespace whose `TenantGateway` objects cannot be listed, or which carries the Gateway owner label but has no `TenantGateway` yet, keeps it as well, so that a transient read failure never withdraws a key a Gateway may still be serving. - -Moving back off an edge-terminated class restores whatever the new mode calls for: the ACME `Issuer` and its `Certificate`s under `http01` or `dns01`, the redirect route under any non-edge mode, and the wildcard replica whenever `publishing.certificates.wildcardSecretName` is set. The `TLSRoute` conditions go back too, since the passthrough listeners do: `http01` judges those routes again from the hostname claims it collects, while `dns01` and `existingSecret` collect none and so drop the entry rather than leave a withdrawal standing on a listener that is serving. `existingSecret` mints no `Issuer` and no `Certificate` of its own, so a move from `edge` into it brings back the replica and the redirect route and nothing else. - -The two halves of the move are not ordered against each other. The wildcard-secret controller watches the `TenantGateway` and prunes the replica as soon as `certMode` changes, while the `Gateway` itself is re-rendered when the tenant's HelmRelease reconciles, and the class controller reprograms after that. A tenant leaving `existingSecret` can therefore be briefly serving HTTPS listeners whose Secret has already gone. - -## When the configured names and the installed classes disagree - -Nothing in this mechanism checks that a class name corresponds to a `GatewayClass` the cluster actually has. Run `kubectl get gatewayclass` before putting a name into any of the three values. - -- **A name in `className` or `tenantSelectableClasses` that matches no installed class.** The tenant's release renders and the `Gateway` object is created, but no controller claims it, so it gets no address and no listener status. Nothing fails at render time; the symptom is the `TenantGateway` holding `Ready=False` with reason `GatewayNotAccepted`. -- **A misspelled name in `edgeTerminatedClasses`.** It matches no class, so edge mode never engages for it and nothing says so. A tenant whose TLS really is terminated upstream stays in whichever mode the solver and wildcard-secret settings select, and its Gateway keeps terminating TLS itself. -- **A class removed from `tenantSelectableClasses` while a tenant still names it.** That tenant's gateway release starts failing to render. Other tenants are unaffected. -- **`className` changed while a tenant pins the outgoing name.** The allowed set is the *current* default plus `tenantSelectableClasses`, so the pin stops being valid and that tenant's gateway release fails. Add the outgoing class to `tenantSelectableClasses` before changing the default, or update the tenant first. -- **A class whose `status.supportedFeatures` lists features but not `TLSRoute`.** A tenant Gateway that renders TLS-passthrough listeners on it reads `Ready=False` with reason `GatewayClassUnsupported`, naming the class, because nothing would serve those listeners. A class that lists no features at all is not judged, since implementations predating the field leave it empty. -- **The publishing tenant selecting a class that terminates TLS differently from `className`.** Refused at render time, in both directions. The cluster-wide ACME `ClusterIssuer`s point their HTTP-01 solver at the publishing tenant's Gateway, and whether that solver pins `sectionName: http` is decided from `gateway.className` alone — a tenant's own choice does not travel on that channel. A publishing tenant that moved itself onto an edge class would keep a solver pinned to the `http` listener its Gateway no longer renders, and every HTTP-01 certificate in the cluster would stop issuing with nothing in any status naming the cause. Change `gateway.className` instead. The publishing tenant is the namespace named by `publishing.ingressName`, `tenant-root` by default. -- **A comma or a non-string entry in either list.** The platform chart fails to render, naming the entry. - -## Checking what a tenant landed on - -```bash -kubectl get gatewayclass - -kubectl --namespace tenant-acme get tenantgateway cozystack \ - --output jsonpath='{.spec.gatewayClassName}{"\t"}{.spec.certMode}{"\n"}' - -kubectl --namespace tenant-acme get tenantgateway cozystack \ - --output jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{"\t"}{.reason}{"\n"}{end}' - -kubectl --namespace tenant-acme get gateway cozystack \ - --output jsonpath='{range .spec.listeners[*]}{.name}{"\t"}{.port}{"\t"}{.hostname}{"\n"}{end}' - -kubectl --namespace tenant-acme get certificate,issuer -``` - -A tenant on an edge-terminated class reports `certMode: edge`, shows port-80 listeners only, and has no `Certificate` or `Issuer` that this controller created. Leftover TLS Secrets from an earlier mode are not shown by the last command — list Secrets of type `kubernetes.io/tls` in the namespace to find those. diff --git a/docs/observability/distributed-tracing.md b/docs/observability/distributed-tracing.md deleted file mode 100644 index d4e6ad80a..000000000 --- a/docs/observability/distributed-tracing.md +++ /dev/null @@ -1,123 +0,0 @@ -# Distributed tracing - -Distributed tracing is the third observability signal in Cozystack, alongside metrics (VictoriaMetrics) and logs (VictoriaLogs). This guide explains what the platform provides, how to send traces from your workloads, what each managed engine can and cannot emit on its own, and how tenants can share one traces store instead of running their own. The design is [cozystack/community `design-proposals/distributed-tracing`](https://github.com/cozystack/community/tree/main/design-proposals/distributed-tracing). - -## What the platform provides - -A tenant's Monitoring application stands up two pieces once tracing is configured: - -- A **VictoriaTraces backend**, one per `tracingStorages` entry (a `VTCluster` in `cluster` mode, a `VTSingle` in `single` mode), with a Grafana Jaeger datasource named `vtraces-` for viewing its spans. -- A **per-tenant OpenTelemetry Collector**, the app-facing OTLP ingest gateway, reachable in the tenant namespace at: - - **`otel-traces:4317`**: OTLP over gRPC - - **`otel-traces:4318`**: OTLP over HTTP - -The collector applies head sampling (10% by default) and optional attribute redaction, then forwards spans to the backend. It lives in your tenant namespace, so traffic from your workloads to it stays inside the namespace. With the default per-tenant backend the spans stay there too; with [shared-central tracing](#shared-central-tracing) the collector forwards them to a store in tenant-root. - -What the platform does **not** do is emit spans for you: producing spans is the workload's job. The platform delivers the transport, storage and tenancy; the *depth* of what shows up is a property of how each application or engine is instrumented. - -## Sending traces from your application (client-side) - -This is the primary path and it works for any workload with OpenTelemetry instrumentation (an OTel SDK in your code, or an auto-instrumentation agent). You do **not** hard-code the endpoint: point your workload at the in-namespace collector through the standard OpenTelemetry environment variables: - -```yaml -env: - # Send to the per-tenant collector over OTLP/HTTP (use :4317 for gRPC). - - name: OTEL_EXPORTER_OTLP_ENDPOINT - value: "http://otel-traces:4318" - - name: OTEL_EXPORTER_OTLP_PROTOCOL - value: "http/protobuf" - # Name this service and the environment it runs in. - - name: OTEL_SERVICE_NAME - value: "my-app" - - name: OTEL_RESOURCE_ATTRIBUTES - value: "deployment.environment.name=production" -``` - -The variables are defined by the OpenTelemetry specification, and the official SDKs and agents read them, but support for each variable differs per language: check your SDK against the [specification compliance matrix](https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md). For a JVM workload you can add the OpenTelemetry Java agent (`-javaagent:/otel/opentelemetry-javaagent.jar`), and the same variables drive it with no code changes. To use gRPC instead, set the endpoint to `http://otel-traces:4317` **and** `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`; with [shared-central tracing](#shared-central-tracing), stay on HTTP, since spans refused over the tenant's write rate are retried only over HTTP, and keep each export request below 3 MiB. - -You do not need to tag your tenant: the backend a tenant writes to is its own, and in shared-central mode the platform attributes every span to its tenant itself. `service.namespace` is for grouping your own services (for example by team), not for the tenant. - -Once spans arrive, they are visible in Grafana under **Explore**, in the traces datasource. - -> **Correlation is not wired yet.** Cross-signal pivots, span to logs by `trace_id` and span to RED metrics, are part of the design but not configured: the traces datasource carries no links to logs or metrics, and no `spanmetrics` connector produces span metrics. Traces are viewable on their own until that follow-up work lands. - -## What each managed engine can emit - -Tracing depth differs per engine, because a sidecar cannot see *inside* a server process. Most managed servers do **not** emit OTLP spans themselves; their tracing is client-side (instrument the application that talks to them). - -| Engine | Server-side OTLP spans? | How you get traces | -| --- | --- | --- | -| **ClickHouse** | Partial: native span *table*, no push, disabled | The server can record internal query spans into `system.opentelemetry_span_log`, which the Cozystack chart removes (see below). Client-side propagation gives end-to-end traces. | -| **PostgreSQL** | No (extension-gated) | Statement-level spans need the `pg_tracing` extension, which is not in the `sharedPreloadLibraries` allowlist, and `shared_preload_libraries` cannot be set through `parameters`. Use client-side instrumentation. | -| **Kafka** | No (brokers) | Brokers do not emit request spans; instrument producers/consumers (client-side). JVM clients can use the OTel Java agent. | -| **RabbitMQ** | No (non-OTLP) | The `rabbitmq_tracing` plugin emits RabbitMQ-format events, not OTLP. Instrument publishers/consumers (client-side). | -| **NATS** | No | Server message-tracing emits NATS-format events, not OTLP. Instrument clients (context is propagated in message headers). | -| **MariaDB** | No | No server-side OTLP. Instrument the application (client-side). | -| **Redis** | No | No server-side OTLP. Instrument the application (client-side). | - -The takeaway: for every engine here, the way to get a useful request trace is to **instrument the application** that issues the queries or messages and point it at `otel-traces` as shown above. The engine then appears as a span in your application's trace when context is propagated. - -## ClickHouse: internal query spans - -ClickHouse is the one engine that produces spans natively: it can record query-execution spans into the `system.opentelemetry_span_log` system table and take part in a trace whose context a client propagates through the W3C `traceparent` header. The Cozystack ClickHouse chart removes that log in its server configuration, alongside the other system logs it disables, and the ClickHouse application exposes no setting to turn it back on, so these spans are not available today. - -Even with the log enabled, ClickHouse has **no native OTLP push**: getting those spans into a collector, and so into Grafana, needs a shipper that reads `system.opentelemetry_span_log` and forwards OTLP, which neither Vector nor Fluent Bit does. Both are follow-up work; end-to-end request traces come from client-side instrumentation, as for every other engine. - -## Sampling, redaction, and tenancy - -- **Sampling:** the collector head-samples traces at 10% by default (`tracingCollector.samplingPercentage`). The decision is made per trace ID, so a kept trace keeps all of its spans and several collector replicas agree on it. A busy application keeps about one trace in ten, so the backend is not overwhelmed. -- **Redaction (PII):** span attributes can carry sensitive data (SQL text with literals, parameters, connection strings). List such keys in `tracingCollector.redactAttributes` to drop them before export. -- **Tenancy:** with the default per-tenant backend, spans never leave the tenant namespace, and only the tenant's own Grafana reads them. With shared-central tracing they are stored in tenant-root; see below for how tenants are kept apart there. - -## Shared-central tracing - -Instead of a VictoriaTraces backend per tenant, tenants can send their spans to one store in tenant-root. Applications do not change: they keep sending to `otel-traces` in their own namespace, and the collector forwards to the shared store. - -### Hosting the store (tenant-root) - -tenant-root opts in with `tracingCentralHost: true` on its Monitoring application, next to a `tracingStorages` entry named `generic` in `cluster` mode; that store is then shared with every tenant that opts in. A vmauth in tenant-root fronts it, and every central tenant's collector and Grafana go through that vmauth. - -All central tenants share the store's disk and retention with tenant-root, and nothing reserves space per tenant. Every traces store is bounded by disk usage: without `retentionDiskUsageBytes` it drops its oldest day once its volume is 80% full. VictoriaTraces always keeps the last two days, though, so the disk cap alone does not stop a fast writer from filling the volume. - -There is no per-tenant allow-list: with hosting on, every tenant that sets `tracingCentral` is admitted, nested tenants included. A tenant-root that needs to limit who shares its store keeps `tracingCentralHost` off. - -### The per-tenant write rate - -Because the disk cap alone does not bound a fast writer, each central tenant is also held to a write rate, which the platform sets with `monitoring.tracingCentralTenantBytesPerSecond` in the platform package, as a whole number of bytes per second up to 1099511627776 (1 TiB/s); any other value fails the platform render. By default it is about 97 KiB/s, at which one tenant writes at most the disk cap of a default 10Gi store over the two days, so raise it with the store. The rate is counted uncompressed and before the collector samples, so the store receives only the sampled share of it and keeps that compressed. A change reaches every tenant's collector through the `cozystack-values` Secret, and upgrades every release in every tenant once. - -The collector Cozystack ships is built with a rate-limit processor, and each central tenant's collector enforces the rate, split across its replicas: a sender whose connection stays on one replica gets that replica's share. Data above the rate is refused. An OTLP/HTTP exporter gets a 429 and retries; an OTLP/gRPC exporter gets `RESOURCE_EXHAUSTED` without the `RetryInfo` the OTLP specification requires for a retry, so it drops those spans. A request larger than 3 MiB, decompressed, is refused with a 400, which exporters do not retry. Applications of central tenants should therefore export over HTTP, in batches below 3 MiB. - -The limiter keeps its state in each collector pod's memory, and every pod starts with a full allowance, so a pod start lets through above the rate up to 5 MiB, or one second of the pod's share of the rate when that is larger. Shared-central tracing therefore allows at most 10 collector replicas, and the tenant's Monitoring render fails above that. The rate bounds each tenant, not their sum: several tenants writing at their limits together can still exceed the cap, so size the volume for the tenants you admit. - -### Opting a tenant in - -Set `tracingCentral: true` on the Tenant, which needs `monitoring: true` on the same tenant. The tenant chart then renders the egress rules the collector and Grafana need to reach the vmauth, and switches the tenant's Monitoring to the shared store. If tenant-root does not host the store, the tenant's Monitoring render fails with a message naming the prerequisite, rather than dropping spans. - -In the tenant's Grafana the shared store appears as the `traces-central` datasource. The tenant's own local stores, if it lists any, keep their datasources, so traces stored before the switch stay readable until the tenant removes those entries. - -### Isolation and the trust boundary - -The store itself authorizes nothing; it takes the tenant from request headers. Each central tenant gets two VMUsers that pin one account, derived from its namespace name and UID: the collector's, which may only write, and its Grafana's, which may only read, each with its own Secret (`traces-central-credentials` and `traces-central-reader-credentials`). vmauth replaces any account a client sends. Tenant pods can reach only the vmauth proxy port, and only from pods carrying the collector's or Grafana's label, with the tenant's credentials. So tenants sharing the store cannot read or write each other's traces. - -Within its own account, a tenant writes only through its collector, so the write rate holds. Grafana's login cannot write, so a write sent through the Grafana datasource proxy, which vmauth would otherwise route to the store, finds no route. The collector and Grafana images are the platform's and cannot be set from the Monitoring application, and of a Secret named in `oidc.customConfig.secretRef` only its `auth.ini` key is mounted into Grafana, so the collector's password does not reach a pod the tenant can shape. - -tenant-root is inside the store's trust boundary. Its workloads reach the store directly, and one that learns a tenant's account can read and write it. Whoever tenant-root admits to its Grafana can read every account through tenant-root's own datasource for the store, which does not go through vmauth. Who that is follows tenant-root Grafana's OIDC mode: with `System`, tenant-root's own groups, which already hold the same access level in every tenant under it; with `CustomConfig` or `None`, whoever that configuration admits. - -### Lifecycle - -- **The account follows the namespace.** It is derived from the namespace name and UID, so a re-created namespace, a restored one included, starts a new account and does not see the spans stored before. -- **Turning central on** keeps the tenant's local stores and their datasources. Right after it, spans the collector sends before the vmauth has loaded the tenant's login get a 401, which the exporter does not retry, so those spans are dropped. The same happens after the tenant's `traces-central-credentials` Secret is re-created with a new password; after `traces-central-reader-credentials` is, the tenant's Grafana gets a 401 on reads until the vmauth loads the new login. -- **Turning central off** removes the tenant's login to the shared store. The spans it wrote there stay until the store's retention drops them and cannot be read from the tenant while it is off; turning it on again on the same namespace makes them readable again. Spans sent between turning it off and the collector switching back to a local store may be dropped, since the egress rule to the shared store goes first. -- **Deleting a tenant** leaves its spans in the shared store until retention drops them. - -### Stopping hosting - -Turn `tracingCentral` off on every central tenant and let its Monitoring reconcile before tenant-root stops hosting. tenant-root's render refuses to stop hosting while a central tenant's VMUsers remain and lists the tenants still holding them. Removing tenant-root's Monitoring altogether is not a render and is not refused; until the central tenants turn central off, every Monitoring render of theirs fails, their metrics, logs and alerting changes included. - -The two refusals read each other's objects when they render, so a tenant turning central on while tenant-root stops hosting can pass both. That tenant's collector then exports to a vmauth that is gone while every release reports Ready, until its Monitoring renders again. Recover by hosting again; turning the tenant's `tracingCentral` off instead stops the silent drop, and turning it on again is then refused until tenant-root hosts. - -## Prerequisites - -- The tenant's Monitoring application with a `tracingStorages` entry, or shared-central tracing turned on for the tenant, and `tracingCollector.enabled` left on (the default). -- For shared-central tracing: tenant-root hosting the store, as described above, and at most 10 collector replicas (`tracingCollector.replicas`). -- A workload with OpenTelemetry instrumentation (SDK or auto-instrumentation agent). diff --git a/docs/operations/backup-classes.md b/docs/operations/backup-classes.md deleted file mode 100644 index a65e717ba..000000000 --- a/docs/operations/backup-classes.md +++ /dev/null @@ -1,461 +0,0 @@ -# Backup Classes - -Cozystack ships a single platform-managed `BackupClass` named `cozy-default`. It is provisioned automatically when the `backupstrategy-controller` package is installed and references the system-managed bucket provisioned through the `apps.cozystack.io/Bucket` CR `cozy-backups` in the `tenant-root` namespace (the real S3 bucket name is the COSI-assigned one from `BucketClaim.status.bucketName`). - -Tenants reference `cozy-default` from `BackupJob`, `Plan`, and `RestoreJob` resources — they do **not** supply S3 credentials, endpoints, or paths. The platform projects the system-managed credentials Secret into the tenant namespace per BackupJob (or, for long-lived references like Velero's `BackupStorageLocation`, into a fixed list of system namespaces on a periodic tick), and the default strategy templates encode `/` into every S3 path so two tenants with the same application name never collide. - -## Supported applications - -### Bound by `cozy-default` (work out-of-the-box) - -| Application Kind | Driver | Strategy CR | -|----------------------------------|--------------------------------------|----------------------------------------------------------------------------| -| `apps.cozystack.io/Postgres` | CloudNativePG (barman) | `strategy.backups.cozystack.io/CNPG` `cozy-default-cnpg` | -| `apps.cozystack.io/MariaDB` | mariadb-operator dump | `strategy.backups.cozystack.io/MariaDB` `cozy-default-mariadb` | -| `apps.cozystack.io/ClickHouse` | Altinity `clickhouse-backup` sidecar | `strategy.backups.cozystack.io/Altinity` `cozy-default-altinity` | -| `apps.cozystack.io/MongoDB` | Percona psmdb operator (pbm) dump | `strategy.backups.cozystack.io/MongoDB` `cozy-default-mongodb` | -| `apps.cozystack.io/Kafka` | Kafka Admin API (topic metadata) | `strategy.backups.cozystack.io/Kafka` `cozy-default-kafka` | -| `apps.cozystack.io/Etcd` | etcd-operator snapshot | `strategy.backups.cozystack.io/Etcd` `cozy-default-etcd` | -| `apps.cozystack.io/RabbitMQ` | RabbitMQ definitions (management API) | `strategy.backups.cozystack.io/Rabbitmq` `cozy-default-rabbitmq` | -| `apps.cozystack.io/Redis` | RDB dump Job (sentinel-discovered master) | `strategy.backups.cozystack.io/Redis` `cozy-default-redis` | -| `apps.cozystack.io/Valkey` | RDB dump Job (sentinel-discovered master) | `strategy.backups.cozystack.io/Redis` `cozy-default-redis` | -| `apps.cozystack.io/VMInstance` | Velero + kubevirt-velero-plugin | `strategy.backups.cozystack.io/Velero` `cozy-default-velero-vminstance` | -| `apps.cozystack.io/VMDisk` | Velero | `strategy.backups.cozystack.io/Velero` `cozy-default-velero-vmdisk` | - -Valkey runs on the same RedisFailover operator as Redis, so `cozy-default-redis` serves both: the driver hands the template `.Release.Prefix` (`redis-` or `valkey-`, after the app's Kind), from which it builds the Sentinel Service and the auth and CA Secret names. A Valkey backup restores only into a Valkey app, a Redis backup only into a Redis app. The Valkey chart has no TLS option, so a Valkey dump always connects in plaintext. - -### Shipped but NOT bound (admin opt-in required) - -| Application Kind | Driver | Strategy CR | -|----------------------------------|--------------------------------------|----------------------------------------------------------------------------| -| `apps.cozystack.io/FoundationDB` | FoundationDB operator backup_agent | `strategy.backups.cozystack.io/FoundationDB` `cozy-default-foundationdb` | - -The FoundationDB strategy CR is rendered by the chart so admins can reference it from a custom BackupClass once the operator-side plumbing (mounting `cozy-backups-creds` into the `cozy-foundationdb-operator` Deployment) is wired manually. See "FoundationDB caveat" below. - -### Endpoint format per driver - -Different operators expect different endpoint shapes; the strategy templates rendered by `backupstrategy-controller` resolve one S3 endpoint (via the `backupstrategy-controller.endpoint` helper) and adapt it to each consumer's contract. For a provisioned bucket (`provisionBucket: true`, the default) the endpoint is **derived from the COSI bucket's system-credentials Secret** (`backupStorage.systemSecretName`) and forced to `https://` — the external S3 ingress with an ACME cert, the only endpoint the backup operators can verify. SeaweedFS's in-cluster S3 serves TLS on `:8333` behind the self-signed "SeaweedFS CA", and the Etcd Strategy S3 schema has no `caCert` field, so the in-cluster endpoint cannot be targeted directly. The `backupStorage.endpoint` chart value (a full URL like `http://seaweedfs-s3.tenant-root.svc:8333`) is the **fallback**, used for external S3 (`provisionBucket: false`) and for offline `helm template`/pre-reconcile renders where the Secret lookup returns nothing. The resolved endpoint is adapted per consumer: - -| Driver | Strategy template field | Form | -|--------|-------------------------|------| -| CNPG (Postgres) | `barmanObjectStore.endpointURL` | full URL (scheme preserved) | -| Etcd | `destination.s3.endpoint` | full URL (scheme preserved) | -| MariaDB | `storage.s3.endpoint` | bare host:port (scheme stripped); `tls.enabled` derived from the scheme | -| MongoDB | `s3.endpointUrl` | full URL (scheme preserved); on the default flow the app's `backup.endpointURL`, on `useSystemBucket` the strategy coordinate injected by the driver | -| FoundationDB | `blobStoreConfiguration.accountName` + `urlParameters.secure_connection` | bare host:port + derived secure flag | -| Velero | `BackupStorageLocation.spec.config.s3Url` | full URL (scheme preserved) | -| ClickHouse sidecar | `S3_ENDPOINT` env | bare host:port (from projected Secret) | -| Redis (dump Job) | `S3_ENDPOINT` env | bare host:port (from projected Secret); `https://` prepended when unscheme'd | -| Kafka | `S3_ENDPOINT` env on the strategy Job | full URL (scheme preserved); the Job's `curl --aws-sigv4` prepends `https://` if the endpoint carries no scheme, honours `backupStorage.forcePathStyle`, and (for the image's curl 7.76.1) signs a portless URL with `--connect-to` to the real port so a ported endpoint still verifies | - -The projected `cozy-backups-creds.endpoint` key is **stripped of scheme** so chart-emitted sidecars (ClickHouse) consume it directly. Drivers that need the full URL receive the resolved endpoint described above — derived from the COSI system Secret (forced `https://`) for a provisioned bucket, or the `backupStorage.endpoint` fallback for external S3. - -VM-driven (Velero) backups land in the same `cozy-backups` bucket under the `velero/` prefix. A `BackupStorageLocation` named `cozy-default` is shipped by the `backupstrategy-controller` chart (`packages/system/backupstrategy-controller/templates/velero-bsl.yaml`) so endpoint/bucket/region come from the same `backupStorage` values block used by Strategy CRs and the projector. - -### FoundationDB caveat - -The strategy CR `cozy-default-foundationdb` is shipped, but it is **not** bound by `cozy-default` yet. Restore runs `fdbrestore` from inside the `cozy-foundationdb-operator` Deployment, which does not yet mount `cozy-backups-creds`. Until the operator deployment is updated to mount the projected Secret, FDB platform-default restore silently fails — admins who need it today should keep using a per-app `Bucket` plus a custom `BackupClass`, or wire the credentials file into the operator deployment themselves. - -**Cleanup gotcha (zombie backup_agent).** Unlike CNPG/MariaDB/Altinity (one-shot operator-side Backup CRs), the FoundationDB driver creates an `apps.foundationdb.org/FoundationDBBackup` CR that drives a **long-lived** `backup_agent` Deployment streaming continuously to S3. Deleting a Cozystack `Backup` (e.g. via retention sweeping) does NOT stop that Deployment — the agent keeps writing until the next BackupJob's `stopOtherFoundationDBBackups` call swaps it out, until an admin invokes `examples/backups/foundationdb/cleanup.sh`, or until the operator-side CR is deleted by hand. If a tenant deletes their last Cozystack Backup and never submits another BackupJob, the agent pods will continue running indefinitely and accumulate S3 PUTs. This is intentional today (the driver has no RBAC verb to stop the operator-side CR on Cozystack-Backup deletion) but admins should be aware of it. - -## Postgres and MariaDB: a restored copy converges its application passwords shortly after recovery - -Managed Postgres no longer accepts a plaintext `users[].password`; every application-user password is chart-generated into `-credentials` and read only from there. The CNPG backup snapshot therefore carries no passwords — deliberate, since a snapshot that did was the one place a tenant password sat in cleartext. - -**Residue from before the upgrade — this is forward-looking, not retroactive.** The removal scrubs new renders and new snapshots; it does not rewrite copies made earlier, and upgrading preserves the live password (the chart reads the existing `-credentials` through `lookup`), so those older copies still expose the *current* credential. Two sinks retained it: a Postgres `Backup` object created before this release snapshotted `app.spec.users` while that spec still carried a plaintext `password`, so its `status.underlyingResources` held that password in cleartext — a field any tenant with `get backups` in the namespace can read (see the `underlyingResources` caution below); and every pre-upgrade Helm revision of `-init-script` rendered `ALTER ROLE … WITH PASSWORD ''` into its manifest, retained for `MaxHistory` revisions. There is no in-band rotation to invalidate the exposed value (chart-based rotation was intentionally not added — a chart-rendered Secret cannot revoke a leaked credential; rotation is being reworked as a controller, cozystack/community#72). The backup strategy controller removes `users.*.password` from every CNPG `Backup` snapshot each time its leader starts, so the first sink is cleaned in-cluster after the upgrade, including a `Backup` written by a pre-upgrade replica during the rollout; the start-up log line `legacy snapshot password scrub finished` reports how many it changed. That does not reach copies made outside the cluster — apiserver audit logs, etcd backups, or object-level exports of the `Backup` resources — and the old init-script Secret persists in Helm history until those revisions age out of `MaxHistory`. So after upgrading, treat any password that was ever set through `users[].password` as exposed until it is rotated. - -The consequence for a **restore into a copy** (a differently-named target): the recovered database keeps the source roles with their source password hashes, but the target chart generates fresh random passwords into the target's `-credentials`, and the role-reconciling init-job is skipped while `bootstrap.enabled` is set on the restored app. The restore driver clears `bootstrap.enabled` automatically once recovery is healthy, so the next HelmRelease reconcile runs the init-job, which applies `ALTER ROLE … WITH PASSWORD` for every `spec.users` entry and converges the recovered roles onto the target Secret. Convergence is therefore automatic but not instant: between the RestoreJob reaching `Succeeded` and that init-job completing, the passwords advertised in `-credentials` do not yet match the recovered roles, so a client reading that Secret cannot log in as an application user for that brief window. The chart flags the Secret `postgres.cozystack.io/credentials-pending: "true"` and the NOTES print the same caution while bootstrap is still on, so a reader does not treat the advertised password as usable mid-restore; both clear on their own once the driver clears bootstrap. The data is intact and reachable as the CNPG superuser throughout. After a to-copy restore, wait for the application login to succeed rather than assuming the advertised password works the instant the RestoreJob succeeds. The driver clears `bootstrap.enabled` by writing it onto the Postgres app's own spec, so a GitOps source (Argo CD, Flux, Terraform) that pins `bootstrap.enabled: true` for that app will revert the clear on its next apply and the copy's credentials never converge — exclude the restored app from such a source until convergence completes, or clear `bootstrap.enabled` in the source itself. If the driver cannot clear the flag at all (the app was deleted or renamed mid-restore), the RestoreJob fails with reason `BootstrapDisableFailed` once its deadline elapses rather than sitting `Running` forever with no signal. An in-place restore is unaffected: the release keeps its existing `-credentials`, whose passwords already match the roles the recovery brings back. One-time migration: an app restored to a copy under an earlier release — before this auto-clear existed — may still be sitting with `bootstrap.enabled` set and its init-job permanently skipped; the driver clears the flag only for a restore it drives to completion, never retroactively for one that already finished, so such an app keeps advertising unconverged credentials until you clear `bootstrap.enabled` on it once by hand. - -Databases and users after a Postgres restore, in place or into a copy: the driver sets the app's `spec.databases` and `spec.users` to the snapshot taken when the Backup was created, while recovery replays WAL past that point (to the end of the archive when no `recoveryTime` is set), so the recovered cluster can hold databases and users the snapshot does not list. On the first init-job run after the restore, a database the chart manages but the restored spec does not declare is released instead of dropped: its data stays, and it and its `_admin` and `_readonly` roles lose the `managed by helm` comment, so the chart no longer grants on it or drops it. Add it to `databases` to manage it again. Users are not released: a recovered user the restored spec does not declare, including a source user the target of a copy does not declare, is dropped on that run, so a client still logging in as that user loses the login. Every object it owned is reassigned to `cozystack:orphaned`, a role that cannot log in and that the chart grants nothing. Functions it owns are switched to `SECURITY INVOKER`, and on PostgreSQL 15 and later its views to `security_invoker`. A materialized view, a rule, and a view on PostgreSQL 13 or 14 have no such switch and keep reading with the rights of `cozystack:orphaned`. The user's grants and memberships do not carry over, so a view or function of that user can lose access it had through them. Declare the user in `users` to keep it; the chart then generates a new password for it into `-credentials`. - -**Objects reassigned by earlier releases stay as they were.** Before this change the chart reassigned a dropped user's or role's objects to the `postgres` superuser, and upgrading does not move them. After upgrading, run the query below in each database: it lists the relations and routines `postgres` owns outside the system catalog and extensions, leaving out a sequence that belongs to a column, which moves with its table. Of those, the chart itself owns only its `auto_grant_schema_privileges()` event-trigger function, so anything else came from a tenant role or was created by hand. Hand what a tenant role created to `cozystack:orphaned` with the matching `ALTER TABLE`, `ALTER VIEW`, `ALTER ROUTINE` or other `ALTER … OWNER TO "cozystack:orphaned"`; the next init-job run switches its functions and, on PostgreSQL 15 and later, its views as above. Handle first the rows with `security_definer` set, and views, materialized views and tables with rules, since those run with their owner's rights: - -```sql -SELECT c.oid::regclass::text AS object, NULL::boolean AS security_definer FROM pg_class c -WHERE c.relowner = 'postgres'::regrole AND c.relkind NOT IN ('i', 'I', 't') - AND c.relnamespace NOT IN ('pg_catalog'::regnamespace, 'information_schema'::regnamespace) - AND NOT EXISTS (SELECT FROM pg_depend d WHERE d.classid = 'pg_class'::regclass AND d.objid = c.oid - AND (d.deptype = 'e' OR (c.relkind = 'S' AND d.refclassid = 'pg_class'::regclass AND d.deptype IN ('a', 'i')))) -UNION ALL -SELECT p.oid::regprocedure::text, p.prosecdef FROM pg_proc p -WHERE p.proowner = 'postgres'::regrole - AND p.pronamespace NOT IN ('pg_catalog'::regnamespace, 'information_schema'::regnamespace) - AND NOT EXISTS (SELECT FROM pg_depend d WHERE d.classid = 'pg_proc'::regclass AND d.objid = p.oid AND d.deptype = 'e'); -``` - -MariaDB avoids the same problem at backup time: the driver sets `ignoreGlobalPriv` on every operator `Backup` it creates, so `mysql.global_priv` — the grant table holding every user's password hash, root included — is excluded from the logical dump and a restore into a copy never overwrites the target's chart-managed users. The same exclusion is also a limitation, and it applies to every restore, not only a to-copy one: because the grant table is never in the dump, only accounts Cozystack declares through `User` CRs (the chart's `users` map) are recreated on restore — an account created out of band, directly in MySQL rather than through `users`, is in no backup, so a restore brings back its data and its `mysql.db` grants but not the account that owns them, and that login fails until the account is recreated. Cozystack treats the `User` CRs as the source of truth for accounts precisely so the grant table need not be, so keep managed accounts in `users` rather than creating them by hand. That matters most for `root`, which the operator only applies at datadir bootstrap via `rootPasswordSecretKeyRef` and could never repair afterwards; keeping the source's grants out of the dump is what lets the target's advertised `root` password stay correct through a restore. This is a **backup-side** fix, so it only protects backups captured from this release onward. A restore from a backup taken **before** this change still replays the source's `mysql.global_priv` and overwrites the target's application users and `root` with the source's hashes; after restoring such an older backup, treat the target's advertised credentials as unreliable and reset them — application users converge on the next reconcile through their `User` CRs, but `root` must be reset out of band (`ALTER USER 'root' … IDENTIFIED BY …` against the restored server) since the operator applies `rootPasswordSecretKeyRef` only at datadir bootstrap. That once-only write is also why the Secret itself now matters: `root` lives solely in `-credentials` with no plaintext escape hatch left in values, so if that Secret is lost after bootstrap — a tenant deletes it, or a partial Velero restore drops it — the advertised `root` password and the live server's diverge with no in-band way to reconcile them; back the Secret up alongside the data, and reset `root` out of band if it is ever lost. - -## ClickHouse: opt-in to the system bucket - -The `clickhouse-backup` sidecar runs inside the ClickHouse Pod itself, so the Helm chart is what wires its S3 credentials. Existing tenants on the legacy `backup.s3*` values continue to work unchanged. To switch a release onto the platform bucket, set: - -```yaml -backup: - enabled: true - useSystemBucket: true -``` - -When `useSystemBucket: true`: - -- The chart-emitted `-backup-s3` Secret is no longer rendered. -- The sidecar consumes `cozy-backups-creds` (projected by the platform). -- `S3_PATH` is set to `/` so two tenants with the same ClickHouse release name never share a prefix. - -`s3Region`, `s3Bucket`, `endpoint`, `s3AccessKey`, `s3SecretKey`, and `s3CredentialsSecret` are ignored in this mode. - -## MongoDB: backup storage - -MongoDB supports two storage flows. By default the S3 target lives on the app CR and the tenant supplies its own credentials; alternatively the tenant opts into the platform system bucket and the driver injects the storage onto the live cluster at backup time. - -### Default: the application owns the backup storage - -The Percona psmdb operator runs the percona-backup-mongodb (pbm) agents whenever the `PerconaServerMongoDB` cluster has `spec.backup.enabled: true` (a declared storage is not needed for the agents to start), and services a `PerconaServerMongoDBBackup` only once the storage it references by name is declared on the cluster. On the default flow the MongoDB application **opts into backups** and points at the bucket in its own chart values: - -```yaml -apiVersion: apps.cozystack.io/v1alpha1 -kind: MongoDB -metadata: - name: orders-db -spec: - backup: - enabled: true - destinationPath: "s3:///orders-db/" - endpointURL: "https://seaweedfs-s3.tenant-root:8333" - insecureSkipTLSVerify: true # self-signed in-cluster seaweedfs; psmdb s3 storage has no CA-bundle field - s3AccessKey: "" - s3SecretKey: "" -``` - -The chart declares that storage as `s3-storage`, which the `cozy-default-mongodb` strategy names (`storageName: s3-storage`, `type: logical`). The driver then drives on-demand backups and restores through the unified `BackupJob` / `RestoreJob` / `Plan` interface on top of the operator's storage config — adding restore-to-differently-named instances (via `PerconaServerMongoDBRestore.spec.backupSource`) and point-in-time recovery that the raw scheduled-task / `mongodump` paths do not offer. When a target cluster has backups disabled, the driver surfaces a clear `Ready=False` precondition on the BackupJob/RestoreJob rather than hanging until the deadline. - -For a to-copy restore the operator reads the dump from the **source** backup's bucket/endpoint. The driver re-points the restore at the **target** cluster's own S3 credentials only when the target declares a storage on that same bucket (the common case where both share one bucket), since the source release's Secret dies with it in a DR scenario; a system-bucket source keeps its projected `cozy-backups-creds` reference, which outlives the source app. A target whose storages are all on a different bucket gets no credential swap: the restore keeps the source's Secret reference and is held at `Ready=False RestoreCredentialsMissing` naming that Secret until it exists in the restore namespace, so a cross-bucket restore never runs with a credential for the wrong bucket. Re-create the source release's `-s3-creds` Secret by name to let it proceed. - -A full, scripted example (write a marker document, back up, restore to a copy, assert the round-trip while the source stays untouched) is in [`examples/backups/mongodb/`](../../examples/backups/mongodb/) — driven by `run-all.sh`. - -### Opt-in to the system bucket - -Unlike ClickHouse's sidecar, PBM takes the bucket/endpoint/prefix as static fields on the `PerconaServerMongoDB` CR, and the platform bucket name is only known at BackupJob time — so the chart cannot render them. Instead the `cozy-default-mongodb` driver injects the storage onto the live cluster. To back up to the platform bucket without supplying S3 credentials, set: - -```yaml -backup: - enabled: true - useSystemBucket: true -``` - -When `useSystemBucket: true`: - -- The chart-emitted `-s3-creds` Secret is no longer rendered, and the chart leaves `spec.backup.storages`, `spec.backup.tasks` and `spec.backup.pitr` unset on the `PerconaServerMongoDB`. -- On every BackupJob the driver SSA-injects the `s3-storage` entry from the strategy coordinates (bucket/endpoint from the platform system bucket, `credentialsSecret: cozy-backups-creds`, prefix `/`) under its own field manager, so a Flux re-render never reverts it and a later change to the coordinates is picked up on the next backup rather than frozen at the first. The backup itself is not started until the operator reports that it has pushed that `spec.backup` into pbm's own configuration (`status.observedGeneration` caught up with the cluster's generation and the `PBMReady` condition `True`): pbm writes to whatever its config last said, not to the spec, so a backup started before that push would land at the previous coordinates while its `Backup` recorded the new ones. The job waits at `Ready=False PerconaServerMongoDBPBMConfigPending` meanwhile, and fails at the backup deadline if the operator never gets there. -- The injection waits for the release to actually render the flag. The app's `backup.useSystemBucket` is a desired value (it reads `true` as soon as it is written, before helm-controller renders the revision), while the tenant's own storage, scheduled tasks and PITR stream from the previous render stay on the `PerconaServerMongoDB` until it does. As long as the cluster still carries `spec.backup.tasks` or `spec.backup.pitr`, the driver does not inject and holds the BackupJob at `Ready=False` (`PerconaServerMongoDBLegacyRender`), failing it at the backup deadline; a release wedged on an old revision therefore never has its nightly dump and oplog redirected into the shared bucket. `useSystemBucket: true` together with `enabled: false` is refused by the chart at render time; the schema carries the same rule as an `x-kubernetes-validations` marker, which the aggregated apiserver does not evaluate yet ([#2657](https://github.com/cozystack/cozystack/issues/2657)), so until then the render-time refusal is what stops such a release, and a release wedged on it shows up as a failed HelmRelease rather than a redirected backup. - -`destinationPath`, `endpointURL`, `insecureSkipTLSVerify`, `s3AccessKey` and `s3SecretKey` are ignored in this mode. - -Because the chart drops `spec.backup.tasks` and `spec.backup.pitr`, the psmdb operator's own scheduled backups and oplog PITR do **not** run on this flow — `backup.schedule` and `backup.retentionPolicy` stay in the schema for the default flow but have no effect here, and the `recoveryTime` PITR restores described below are not available. Migrate scheduled backups to a `backups.cozystack.io/Plan` against `cozy-default` instead. **Flipping this flag on a running app** removes the storage, tasks and pitr the app had, so a release that was taking scheduled backups + PITR before the flip stops until a `Plan` takes over. - -Dropping `spec.backup.tasks` also drops the psmdb `keep` retention it carried, so on this flow **the driver owns the archive** the way the Rabbitmq/Redis drivers own theirs (an on-demand `PerconaServerMongoDBBackup` is not covered by psmdb task retention). The driver stamps the operator's `percona.com/delete-backup` finalizer onto each backup it mints against a storage that carries the platform coordinates (the strategy's bucket and `credentialsSecret`, as read back from the live cluster after the injection; a strategy without an `s3` block, or a storage pointing anywhere else, holds the job at `Ready=False` instead of minting an unowned dump), and deleting a Cozystack `Backup` (by hand today; by platform retention once it lands) deletes that `PerconaServerMongoDBBackup` CR — whose finalizer removes the pbm object from the shared bucket before the `Backup` is released, so deleting `Backup` objects is what reclaims `cozy-backups`. A legacy backup writes to the tenant's own bucket and keeps the pre-existing hands-off contract (its lifecycle stays the tenant's / operator's; nothing is deleted on `Backup` removal). Note this prunes only when a `Backup` is deleted: unlike CNPG, whose shipped strategy sets a `retentionPolicy` the engine trims on, MongoDB has no engine-side scheduled trim on this flow, so keeping `cozy-backups` bounded depends on a `Plan`'s retention actually deleting old `Backup` objects. **A `Plan` has no retention field yet** (`PlanSpec` carries only `applicationRef`, `backupClassName` and `schedule`), so today a nightly `useSystemBucket` Plan grows the shared bucket unbounded — `schedule` and `retentionPolicy` on the app are inert here (see [Opt-in to the system bucket](#opt-in-to-the-system-bucket)), and platform-level retention is tracked in the [`BackupRetentionPolicy` design (#4192)](https://github.com/cozystack/cozystack/pull/4192). Teardown GC of orphaned `Backup`s / S3 data is [#3966](https://github.com/cozystack/cozystack/issues/3966). Until those land, bound `cozy-backups` with an S3 lifecycle policy on the object store. Two cases release the `Backup` without waiting for the object to be pruned, so a wedged prune never pins a namespace in `Terminating`: a terminating namespace (where the operator's delete-backup step cannot run) and the `backups.cozystack.io/skip-artifact-cleanup: "true"` annotation. In both the driver strips its own finalizer and leaves the object for a bucket lifecycle policy to reclaim. A down operator or unreachable storage in a live namespace is NOT released automatically — the `Backup` stays `Terminating` as a visible signal until the operator recovers or you set that annotation to force the release. - -A dump that has started streaming on this flow is not cancelled on the 30-minute backup deadline (a real dataset routinely outruns it, and deleting the CR mid-stream would prune a partial), but it is not unbounded either: after 24 hours in `running` the `BackupJob` fails with a `BackupAbandoned` warning naming the `PerconaServerMongoDBBackup`, which is left in place. If that dump later completes, its archive is reachable only through that CR; delete the CR to prune it. - -**Opting back out** (`useSystemBucket` true→false) is not automatic in reverse: the driver stops injecting, but the `s3-storage` entry it already applied stays on the live `PerconaServerMongoDB` under the driver's own field manager (`cozystack-psmdb-backup-driver`) — the chart does not own it and so never prunes it. It is inert once `backup.enabled=false`; to fully return a release to the legacy flow, delete that storage entry from the `PerconaServerMongoDB` by hand before re-declaring the tenant's own `backup.*` S3 values. Until it is removed, a new `BackupJob` on that release is refused rather than written into the platform bucket unowned: the driver recognises the injected entry by its `credentialsSecret` and holds the job at `Ready=False` (`PerconaServerMongoDBStorageStale`) naming that manual step, failing it at the backup deadline. A backup already streaming when the flag was flipped is unaffected — its flow is fixed by the finalizer on its own operator CR. - -### Point-in-time recovery (MongoDB) - -psmdb records an oplog stream between logical backups. A MongoDB `RestoreJob` recovers to a timestamp via `spec.options.recoveryTime` — the same option name and RFC3339 format the Postgres/CNPG driver uses, so the two are uniform: - -```yaml -spec: - options: - recoveryTime: "2026-08-05T12:34:56Z" # RFC3339 (UTC), same as the CNPG driver -``` - -The driver converts `recoveryTime` to the psmdb oplog target internally. Unlike CNPG (whose empty `recoveryTime` replays WAL to the latest archived point), an empty `recoveryTime` here restores the backup snapshot as taken — a psmdb logical backup is already a consistent point, so "restore this backup" is the safe default. `spec.options.restoreTimeoutSeconds` caps how long the driver waits for the operator restore before failing (default 30m), matching the CNPG option. Any unrecognised key under `spec.options` (e.g. a `recoverytime` typo) is ignored but surfaced as a `UnknownRestoreOption` Warning event on the RestoreJob rather than silently dropped. - -## Inspecting the defaults - -```bash -kubectl get backupclasses -kubectl get backupclass cozy-default -o yaml -kubectl -n tenant-root get bucket cozy-backups -kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials -kubectl -n cozy-velero get backupstoragelocation cozy-default -``` - -The bucket lives in `tenant-root` and is provisioned through the `apps.cozystack.io/Bucket` CR. The system-managed credentials Secret never leaves that namespace. The backupstrategy-controller projects a copy under the name `cozy-backups-creds` into a tenant namespace right before each BackupJob or RestoreJob runs, and refreshes the same Secret in `cozy-velero` (and any other namespace listed in `backupStorage.systemNamespaces`) on a 1-minute tick. The projected Secret carries multiple key formats so each driver finds what it needs in one place: - -| Key | Consumer | -|-----------------------------------------------|-------------------------------------------| -| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | CNPG, MariaDB, Etcd, RabbitMQ, Redis | -| `accessKey` / `secretKey` (plus `bucketName`, `endpoint`, `region`) | ClickHouse sidecar | -| `cloud` | Velero (AWS credentials file format) | -| `blob_credentials.json` | FoundationDB backup_agent | - -The Redis dump Job additionally reads the `endpoint`, `bucketName`, and `region` keys (alongside the `AWS_*` pair) via `secretKeyRef`, the same set the ClickHouse sidecar consumes. - -The dump Job also speaks to the Redis app's own Sentinel and master, which is a separate TLS axis from the S3 endpoint above. When the `Redis` app has `tls.enabled`, the operator moves both to a TLS-only listener and publishes the CA at `redis-.ca-cert`; the Job mounts that Secret optionally and connects with `--tls --cacert` exactly when its `ca.crt` is present, so a plaintext (non-TLS) app is unaffected. This covers the default `tls.authClients: no`. Mutual TLS (`authClients: yes`) is not supported for backup — it requires a client certificate signed by the release CA, which the platform does not issue. - -### Bootstrap window - -On a fresh-cluster install, the Velero `BackupStorageLocation` `cozy-default` is rendered before the credentials projector has had a chance to copy `cozy-backups-creds` into `cozy-velero`. The BSL reports `Unavailable` until the projector's first synchronous round completes (which runs as soon as the `backupstrategy-controller` acquires leadership — in practice moments after the Pod becomes Ready, typically tens of seconds after `helm install` returns, not minutes). Velero rejects new `Backup` AND `Restore` requests against `storageLocation: cozy-default` during that window. Plan VM backup automation accordingly, or wait for the BSL to become ready before submitting backups: `kubectl -n cozy-velero wait backupstoragelocation cozy-default --for=jsonpath='{.status.phase}'=Available --timeout=5m`. - -**Note on controller restarts.** The BSL flickers `Unavailable` on every `backupstrategy-controller` pod restart while the projector replays its first synchronous round. The window is short (single-digit seconds) but operators who alert on BSL availability should suppress alerts during the controller's `kube_pod_container_status_restarts_total{container=backupstrategy-controller}` events or use a longer evaluation window than the projector tick (60s). - -### Cozy-default Bucket bootstrap - -`cozy-default` ships an `apps.cozystack.io/Bucket cozy-backups` CR in `tenant-root`, which the bucket-application chart turns into a `BucketClaim`; the COSI driver then assigns the real S3 bucket name (`bucket-`, so it cannot be computed in advance) and writes it to the BucketClaim's `.status.bucketName`. The strategy templates and the Velero BSL all read that real bucket name (Helm `lookup` against the BucketClaim). On a fresh install the BucketClaim takes a reconcile cycle to populate its status — until it does, the strategy templates render empty and only the `Bucket` CR + `BackupClass` are present in the cluster. - -**That skip does not repair itself on a reconcile.** helm-controller re-renders a release only when its chart or values change; the interval reconcile is a no-op for a healthy release, and drift detection is off on operator-generated HelmReleases. A cluster that lost the race at install time therefore keeps `BackupClass cozy-default` with **no** `Strategy` CRs and **no** Velero BSL indefinitely — which also fail-closes the pre-adoption snapshot in the v1.6.0 etcd migration. - -The same trap sits one release earlier, on the Secret everything else depends on. `bucket-cozy-backups-system-credentials` is rendered by the `bucket-cozy-backups-system` release behind a `lookup` of the COSI Secret, and is skipped just as permanently when that lookup is empty. Without it the credentials projector has no source at all, so no strategy, no Velero and no v1.6.0 etcd migration can resolve — and the gate cannot even determine the bucket name, which it reads from that Secret. - -Convergence is driven instead by the controller's default-objects gate (`backupStorage.reconcileDefaultObjects`, on by default), which repairs both, in order. While the credentials Secret is absent or carries no bucket name, the gate stamps `reconcile.fluxcd.io/forceAt` + `requestedAt` on the **bucket's** `bucket--system` HelmRelease. Once the bucket name resolves it checks that every object `cozy-default` routes to exists and stamps the same pair on the **`backupstrategy-controller`** HelmRelease, forcing the real Helm upgrade that re-runs the lookups. The two are throttled independently (one force per 5 minutes each), so the second is not delayed by the first. Expect the objects within a minute or two of the bucket becoming ready. - -Each default `Strategy` CR also waits for its own CRD. The chart ships the `strategy.backups.cozystack.io` CRDs as templates in the same release, and Helm resolves every object's kind before it applies any, so a CR rendered next to a CRD the apiserver does not serve yet would fail the whole release, CRDs included. Each strategy template therefore renders only once its CRD is `Established`. That holds the CRs back on the first render of every install, including an external-S3 one where the bucket name is known up front, and on the upgrade that first ships a new strategy kind: that revision creates the CRDs, and the gate's forced upgrade above renders the CRs. - -The gate does not force a suspended HelmRelease — helm-controller ignores both annotations while `spec.suspend` is true, so it logs the skip and leaves the force counter alone. A release left suspended (`cozyhr suspend`) is therefore never repaired until it is resumed. - -Watch it with: - -```bash -kubectl -n cozy-backup-controller logs deploy/backupstrategy-controller | grep default-objects-gate -``` - -#### Manual recovery on an affected cluster - -Only needed on a cluster running a version without the gate (or with `reconcileDefaultObjects: false`). A plain `flux reconcile helmrelease` does **nothing** here — it does not re-render. You need a forced upgrade, and **both** annotations: `forceAt` is what makes helm-controller run a real Helm upgrade, and it is only honoured together with `requestedAt`. - -First confirm the bucket name is actually resolvable — forcing before that just re-runs the same empty lookup: - -```bash -kubectl -n tenant-root get bucketclaim bucket-cozy-backups -o jsonpath='{.status.bucketName}' -``` - -Then force the two releases, **in this order**: - -```bash -# 1. The credentials Secret. It is rendered by the -system release, not by -# bucket-cozy-backups — easy to miss, and the projector (hence every -# strategy and Velero) has no source without it. -ts=$(date -u +%Y-%m-%dT%H:%M:%SZ) -kubectl -n tenant-root annotate helmrelease bucket-cozy-backups-system \ - reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite -kubectl -n tenant-root get secret bucket-cozy-backups-system-credentials - -# 2. The Strategy CRs and the Velero BSL. -ts=$(date -u +%Y-%m-%dT%H:%M:%SZ) -kubectl -n cozy-backup-controller annotate helmrelease backupstrategy-controller \ - reconcile.fluxcd.io/forceAt="$ts" reconcile.fluxcd.io/requestedAt="$ts" --overwrite -kubectl get $(kubectl get crd -o name | grep strategy.backups.cozystack.io) 2>/dev/null -kubectl -n cozy-velero get backupstoragelocation cozy-default -``` - -The race is nested: step 2's `lookup` resolves from the BucketClaim status, but the projector — and the v1.6.0 etcd migration — read the Secret from step 1, so a cluster missing both needs both. - -### Observability - -The credentials projector emits two Prometheus counters labelled by `namespace` (and `reason` for failures): - -- `cozystack_backup_credentials_projection_successes_total` -- `cozystack_backup_credentials_projection_failures_total` - -Alert on `rate(cozystack_backup_credentials_projection_failures_total[5m]) > 0` or `absent_over_time(cozystack_backup_credentials_projection_successes_total[10m])` to catch a stale BSL credential or a malformed source Secret without log scraping. - -The default-objects gate emits three more: - -- `cozystack_backup_default_objects_missing{backupclass="cozy-default"}` — how many of the objects the platform default backups depend on are absent: the credentials Secret, the Strategy CRs `cozy-default` routes to, and the Velero BSL. **This is the alert that would have caught the missing Strategy CRs**: it is non-zero whatever the HelmRelease's `Ready` condition says. Alert on `min_over_time(cozystack_backup_default_objects_missing[15m]) > 0`. -- `cozystack_backup_default_objects_check_errors_total{backupclass="cozy-default"}` — checks that could not reach a conclusion (an API error reading the source Secret, the BackupClass, or one of the routed objects). The gauge above is deliberately **not** written on those ticks, so that it does not flap on a transient API error — which means that while this counter climbs, the gauge is stale and a `0` on it proves nothing. Pair the two: `min_over_time(cozystack_backup_default_objects_missing[15m]) > 0 or rate(cozystack_backup_default_objects_check_errors_total[15m]) > 0`. -- `cozystack_backup_default_objects_force_reconciles_total{namespace,name}` — forced Helm upgrades issued, labelled by the release forced (this chart's own, or the bucket's `-system` release). A counter that keeps climbing means the forced render is not producing the objects (a missing CRD, for instance), which is a different problem from the install-time race. A suspended release is skipped before the patch and is **not** counted here, so a paused release cannot masquerade as a render that keeps failing — look for the `skipped forcing a suspended HelmRelease` log line instead. - -## Admin overrides for `cozy-default` - -`cozy-default` is rendered by the `backupstrategy-controller` chart and owned by Flux's helm-controller. **Direct `kubectl edit backupclass cozy-default` is overwritten on the next helm reconcile** — the same applies to its companion `strategy.backups.cozystack.io/*` CRs (`cozy-default-cnpg`, `cozy-default-etcd`, `cozy-default-mariadb`, `cozy-default-altinity`, `cozy-default-mongodb`, `cozy-default-foundationdb`, `cozy-default-rabbitmq`, `cozy-default-redis`, the two `cozy-default-velero-*`). The supported override path is the `backupStorage` block on the **`platform` component** of the `cozystack.cozystack-platform` Package CR: - -```yaml -apiVersion: cozystack.io/v1alpha1 -kind: Package -metadata: - name: cozystack.cozystack-platform -spec: - components: - platform: - values: - backupStorage: - provisionBucket: true # default; set false for external S3 - bucketName: cozy-backups # apps.cozystack.io/Bucket release name - endpoint: http://seaweedfs-s3.tenant-root.svc.cozy.local:8333 - region: us-east-1 - forcePathStyle: true - systemSecretName: bucket-cozy-backups-system-credentials - systemNamespaces: - - cozy-velero -``` - -The platform chart forwards this block into the child `Package cozystack.backupstrategy-controller` as `components.backupstrategy-controller.values.backupStorage` (`packages/core/platform/templates/bundles/system.yaml`), from where the cozystack operator merges it into the `backupstrategy-controller` HelmRelease over the chart defaults. Two paths that look plausible do **not** work: `spec.components.backupstrategy-controller` on the `cozystack.cozystack-platform` Package is silently ignored (the only component under that PackageSource is `platform`), and patching the child `Package cozystack.backupstrategy-controller` directly is reverted whenever the platform helm-reconcile re-renders it. - -A sibling `backupStrategyController` block on the same `platform` component is forwarded the same way, for the controller's own knobs rather than the bucket. The one an operator reaches for is `redisBackupResources`, which sizes the Redis strategy Pod — restore loads the whole dataset into a throwaway loader, so a large Redis needs more memory / ephemeral-storage than the defaults (the loader-not-ready log line points here): - -```yaml -spec: - components: - platform: - values: - backupStrategyController: - redisBackupResources: - limits: - memory: 8Gi - ephemeral-storage: 32Gi -``` - -| Knob | Effect | -|---|---| -| `provisionBucket` | Toggle creation of the in-cluster `apps.cozystack.io/Bucket` CR. Set `false` for external S3 (see [Disabling the platform-managed bucket](#disabling-the-platform-managed-bucket)). | -| `bucketName` | Two modes. With `provisionBucket: true` (default): K8s name of the Bucket CR + lookup key for the COSI BucketClaim — the actual S3 bucket name is the COSI-assigned UUID, surfaced through `BucketClaim.status.bucketName`. With `provisionBucket: false`: taken **verbatim as the real S3 bucket name** and baked into every strategy CR + the Velero BSL. | -| `namespace` | Namespace the Bucket CR (and its system-credentials Secret) lives in — `tenant-root` by default. Must be a tenant namespace (`tenant-*`): the Bucket chart's RBAC helper fails the Helm render for any other prefix. | -| `bucketNameOverride` | Escape hatch for offline `helm template` renders — bypasses the live-cluster BucketClaim lookup. Leave empty in production. | -| `endpoint` | **Fallback** S3 endpoint. For a provisioned bucket the strategy CRs + Velero BSL derive the endpoint from the COSI system Secret (external ACME ingress, forced `https://`) instead; this value is used only for external S3 (`provisionBucket: false`) and offline renders. For external S3, switching it to `https://` enables TLS in the MariaDB/FoundationDB strategies, which derive TLS from the scheme. The Redis dump Job is not scheme-driven: the projector delivers a bare host and the script prepends `https://` unconditionally, so the Job always connects over TLS — point `endpointCASecretName` (below) at the private CA rather than relying on the scheme, and ensure that CA is reachable to the relevant Pods. | -| `endpointCASecretName` | Optional Secret (key `ca.crt`) in the app namespace the Redis Job trusts for a self-signed S3 endpoint. Empty by default: the projected endpoint is always `https://` and the platform bucket's ACME cert verifies against the image's system CA store unaided. Set it only for a private CA. The Job mounts it optionally (nothing projects this Secret automatically), so a name typo does not wedge the Pod on `FailedMount`: a missing `ca.crt` falls through to the system CA store and the Job fails fast at the TLS handshake with a legible error — it does not skip verification, which still needs the explicit `insecureSkipTLSVerify` opt-in. | -| `insecureSkipTLSVerify` | Disables S3 certificate verification for the Redis Job (`curl -k`). `false` by default and an explicit opt-in, never a fallback — an untrusted-cert endpoint fails closed unless this is set. Prefer `endpointCASecretName`. | -| `region` | Re-projected into `cozy-backups-creds` on the next reconcile. Pod-restart required for chart-emitted clients consuming the region via env (ClickHouse sidecar today). | -| `forcePathStyle` | Path-style addressing; SeaweedFS S3 requires it, AWS S3 typically doesn't. | -| `systemSecretName` | Name of the human-friendly Secret produced by the Bucket app (or pre-created manually for external S3). The projector also accepts the raw COSI Secret format. | -| `systemNamespaces` | Namespaces where the controller eagerly projects `cozy-backups-creds` (Velero BSL, FDB operator). Tenants are projected lazily during BackupJob reconcile. | - -When the override needs to go beyond storage coordinates — different retention, different driver→Kind binding, multi-region split — create a **sibling BackupClass** with a unique name (anything but `cozy-default`). Sibling BackupClasses live outside the chart, are admin-owned, and Flux will not touch them. Tenants opt in by setting `backupClassName: ` on their `BackupJob`s. - -## Tuning via a custom BackupClass - -The defaults aim at a reasonable middle (gzip compression where applicable; engine-side retention only where the strategy carries it, which today is the CNPG `retentionPolicy: "30d"` and the two Velero `ttl: 720h`; every other shipped strategy, MongoDB included, sets none, so its archives on the shared bucket are bounded only by an S3 lifecycle policy or a manual `Backup` delete: a `Plan` only creates BackupJobs and never deletes a `Backup`). To override for a specific tenant or workload, create your own `BackupClass` pointing at the same strategy CRs but with tweaked `parameters`, or a fresh strategy CR. Common knobs: - -- **CNPG strategy**: `barmanObjectStore.retentionPolicy`, `data.compression`, `wal.compression`. -- **MariaDB strategy**: `compression`, `maxRetention`, `databases[]`. -- **MongoDB strategy**: `storageName` (which `spec.backup.storages` entry on the psmdb cluster to use), `type` (`logical`), `compressionType` / `compressionLevel`, and — for the `useSystemBucket` flow — the `s3` coordinates the driver injects. On the default flow the S3 target is tuned via `backup.*` values on the MongoDB release instead (see [MongoDB: backup storage](#mongodb-backup-storage)). -- **Altinity strategy**: tune the `clickhouse-backup` sidecar via `backup.*` values on the ClickHouse release; the strategy Pod is a thin HTTP client. When the S3 endpoint's certificate is signed by a private CA rather than a publicly-trusted one — SeaweedFS's in-cluster `:8333` being the case in point — point `backup.endpointCA` at a Secret holding that CA bundle; the chart mounts it into the sidecar and adds it to the trust store via `SSL_CERT_DIR`, which supplements the system CA set rather than replacing it. -- **FoundationDB strategy**: `snapshotPeriodSeconds`, `agentCount`, `urlParameters[]`. -- **Rabbitmq strategy**: `artifactURITemplate` (the `///definitions.json` object-key layout) and the pod `template` (image, resources). The backup is a definitions export over the management API — vhosts, users, permissions, queues, exchanges, bindings, policies, parameters — so message payloads are out of scope and there is no point-in-time recovery; a full message-data backup is a Velero volume snapshot instead. **Restore is a merge, not a reset**: importing definitions (`POST /api/definitions`) creates or updates what the export contains and never deletes, so anything created after the backup survives the restore and the broker is not returned to its exact backup-time state. A to-copy restore imports the source's users (with password hashes) into the target, so those source credentials become valid logins there. Unlike the operator-backed strategies (whose engine owns archive retention), the Rabbitmq driver owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not normally accumulate objects. The delete is **best-effort**: when it cannot run from here the Backup is released without deleting the object (with a `Warning` Event naming what was left behind), so the Backup — and any namespace being torn down — never wedges. The named give-up conditions are: the namespace is terminating (`CREATE` is forbidden there), the strategy CR or the projected credentials are unavailable, the rendered Job name is invalid, or the skip annotation below is set. Otherwise a genuinely failing delete (object storage unreachable) keeps the `Backup` in `Terminating` (a visible signal to act on) and retries. To release such a stuck `Backup`, annotate it `backups.cozystack.io/skip-artifact-cleanup: "true"` — cleanup then skips the delete and lets the `Backup` go, leaving the object in the bucket to be reclaimed manually or by a bucket lifecycle policy (`kubectl annotate backup -n backups.cozystack.io/skip-artifact-cleanup=true`). -- **Velero strategy (VMInstance / VMDisk)**: `ttl`, `includedResources[]`, `excludedResources[]`. -- **Etcd strategy**: today the strategy is path-only, and a `Plan` carries no retention field yet (see [#4192](https://github.com/cozystack/cozystack/pull/4192)), so nothing trims etcd snapshots on the shared bucket automatically either. - -The system-managed credentials Secret is the **only** way for in-cluster strategies to reach `cozy-backups`. Do not embed access keys in `BackupClass.parameters` — the security model relies on Secret references, and `parameters` end up in `Backup.status.underlyingResources`, which tenants can read. - -## Disabling the platform-managed bucket - -If a deployment runs against an external S3 (no SeaweedFS), set `backupStorage.provisionBucket: false` via the `platform` component override described above and create the source credentials Secret in `tenant-root` manually (flat-key format: `accessKey` / `secretKey` / `endpoint` / `bucketName`; or the raw COSI `BucketInfo` JSON). In the same `backupStorage` block, update `endpoint`, `region`, **and `bucketName`**: with `provisionBucket: false` the strategies and the Velero BSL take `bucketName` verbatim as the real S3 bucket name (no COSI lookup), so it must name the actual bucket on the external S3 — the `bucketName` key inside the Secret alone is not enough. The Velero `BackupStorageLocation` picks the same values up automatically (the chart renders it from the same `backupStorage` block), so no separate BSL configuration is needed. Note that disabling the cluster-default BSL itself (the chart's `velero.bslEnabled` value) is **not** carried by the `backupStorage` override path — the platform Package forwards only the `backupStorage` block. - -## Upgrade notes from chart-managed backups - -> **Postgres backups now use the CloudNativePG Barman Cloud plugin, not native `spec.backup.barmanObjectStore`.** -> -> Postgres backups were migrated off the deprecated native `barmanObjectStore` (removed in CNPG 1.29) onto the [Barman Cloud plugin](https://github.com/cloudnative-pg/plugin-barman-cloud). The `barman-cloud-*` binaries the native path needs are absent from the `standard` image variant `keycloak-db` pins; `apps/postgres` pins bare (system-flavor) tags that still ship them — its breakage was the operator/CRD skew below, not missing binaries — but the system flavor is deprecated upstream, so both move to the plugin. The chart now renders `spec.plugins` on the `cnpg.io/Cluster` plus a `barmancloud.cnpg.io/ObjectStore` CR that carries the S3 configuration, and the CNPG backup driver SSA-applies an `ObjectStore` and patches `spec.plugins` (instead of `spec.backup.barmanObjectStore`) for the platform flow; `Backup`/`ScheduledBackup` use `method: plugin`. On `helm upgrade` a legacy chart-managed Cluster's `spec.backup.barmanObjectStore` is replaced by `spec.plugins` + an `ObjectStore`. This only works once the `plugin-barman-cloud` operator is installed and co-located with the CNPG operator in `cozy-postgres-operator` (shipped by the `postgres-operator` chart) and its cert-manager dependency is present — otherwise the Cluster references a plugin that is not registered and backups / WAL archiving stall. -> -> **Mixed-version window: BackupJobs fail until the app re-renders with the new chart.** A `BackupJob` that fires while a `Cluster` still carries the pre-upgrade chart's native `spec.backup.barmanObjectStore` is rejected by the CNPG admission webhook (`Cannot enable a WAL archiver plugin when barmanObjectStore is configured`) — the driver cannot attach `spec.plugins` alongside the native config. This resolves itself once Flux re-renders the Postgres app with the new chart (which replaces `barmanObjectStore` with `spec.plugins`); re-run the BackupJob afterward. Upgrading only the operators/driver without the app charts leaves every `backup.enabled: true` Postgres in this state permanently. -> -> **Postgres `backup.enabled: true` with placeholder credentials no longer wires backups on upgrade.** -> -> The pre-v1.5 defaults for `backup.s3AccessKey` / `backup.s3SecretKey` in `packages/apps/postgres/values.yaml` were the literal `""` / `""` placeholders, so the Postgres chart still rendered a backup block on the `cnpg.io/Cluster` (with junk credentials, WAL archiving failing at runtime). Starting with v1.5 those defaults are empty strings and the chart NO LONGER renders any backup wiring when the placeholders are unmodified. Tenants on the legacy chart-managed flow who relied on those placeholders see `spec.plugins` and the `ObjectStore` disappear from the live `Cluster` on `helm upgrade`. Action — pick one: -> -> - **Move to the platform flow (recommended).** Set `backup.useSystemBucket: true`; the chart leaves `spec.plugins` unset and the CNPG backup driver SSA-applies the `ObjectStore` and patches `spec.plugins` onto the live `Cluster` at first BackupJob time. No tenant-side keys required. -> - **Stay on the legacy chart-managed flow.** Supply real `backup.s3AccessKey` / `backup.s3SecretKey` (or a pre-existing `backup.s3CredentialsSecret.name`); the chart renders `spec.plugins` + the `ObjectStore` from those coordinates. -> -> **Two platform-level changes ride along with this migration (they are required for the plugin to run):** -> -> - **CloudNativePG operator bumped to 1.28.1** (`postgres-operator` chart). A minor operator upgrade for the whole fleet; the CNPG CRDs move with it. This also fixes a latent bug where the operator image was newer than its bundled CRDs (`status.instanceID.sessionID` pruned), which made the operator believe the instance manager restarted and fail **every** backup cluster-wide with `instance manager was restarted during backup`. The vendored chart ships the CNPG CRDs as ordinary release manifests (gated by `crds.create`), so a normal `helm upgrade` updates them together with the operator — the `helm.sh/resource-policy: keep` annotation only prevents their deletion, not updates. The historical skew came from the `image.tag` pin outrunning the vendored chart's CRDs; that pin has since been removed, so no out-of-band CRD apply is needed unless the HelmRelease itself is held back. -> - **LINSTOR scheduler admission webhook must not strip native-sidecar fields.** Older `linstor-scheduler` extender builds (≤ v0.3.5's predecessors) decoded every pod through a stale vendored Kubernetes API and re-encoded it, stripping `initContainers[].restartPolicy` cluster-wide — which breaks the plugin's restartable barman sidecar ("startupProbe forbidden without restartPolicy=Always"). The platform already ships the fixed webhook (`linstor-scheduler` re-vendored to upstream chart 0.3.1 / extender v0.3.6, which stopped stripping unknown pod fields), so no action is needed on an up-to-date platform; just do not hold the `linstor-scheduler` package back on an older version when rolling out the plugin. -> -> The same `useSystemBucket` opt-in applies to ClickHouse — see [ClickHouse: opt-in to the system bucket](#clickhouse-opt-in-to-the-system-bucket). When `useSystemBucket: true` is set on ClickHouse, the legacy `-backup` CronJob, credential Secret, and backup script are no longer rendered (they are mutually exclusive with the platform flow); migrate scheduled backups to a `backups.cozystack.io/Plan` against `cozy-default`. - -## Tenant workflow - -Tenants only ever see the BackupClass name. Typical apply: - -```yaml -apiVersion: backups.cozystack.io/v1alpha1 -kind: BackupJob -metadata: - name: ad-hoc - namespace: tenant-acme -spec: - backupClassName: cozy-default - applicationRef: - apiGroup: apps.cozystack.io - kind: Postgres - name: orders-db -``` - -An application takes one restore at a time. A RestoreJob that targets an application another unfinished RestoreJob is already restoring fails with reason `ConcurrentRestore` before it changes anything, and its message names the job that goes first. It is not queued: once that job has finished, create a new RestoreJob if the restore is still wanted. A restore that has started always goes first. Between two that have not, the older one runs; creation times have one-second precision, so of two created in the same second, the one whose name sorts first runs. - -## Point-in-time recovery (PostgreSQL) - -A `RestoreJob` restores a `Postgres` application from a `Backup`. Omit `spec.options.recoveryTime` to recover to the latest point in the WAL archive; set it (RFC3339) to recover the database to an exact instant — a point-in-time recovery (PITR). The CNPG barman-cloud plugin restores a base backup and replays archived WAL from it: up to `recoveryTime` when one is set, so the restored cluster reflects the database as of that instant and later writes are absent, and to the end of the archive otherwise. - -The base backup a restore starts from is the one the `Backup` refers to: the driver passes its barman backup ID to CNPG as `bootstrap.recovery.recoveryTarget.backupID`. There are two exceptions, and for both the choice is left to the plugin. The first is a `recoveryTime` earlier than one second after that backup ended, which the backup cannot serve (recovery would have to stop before the backup is consistent); the plugin then takes the newest base backup that ended at or before `recoveryTime`. The second is a `Backup` whose `cnpg.io/Backup` no longer exists: retention (`barmanObjectStore.retentionPolicy`) deletes it together with the base backup it describes, so the archive can no longer start from that backup, and the `RestoreJob` records a `BaseBackupGone` warning event. A base backup removed from the bucket by anything else — a lifecycle rule, a manual delete — is still pinned for as long as its `cnpg.io/Backup` stays. The plugin deletes a `cnpg.io/Backup` whose base backup has left the catalog on its next maintenance pass (every `instanceSidecarConfiguration.retentionPolicyIntervalSeconds` of the `ObjectStore`, 30 minutes by default, retention policy or not), but only from the primary of the cluster that took it, running under the same UID. Until then a restore from it fails after the target has been purged, with `no backup found with ID` in the recovery log, and a retry fails the same way; deleting that `cnpg.io/Backup` by hand, which leaves the object store untouched, lets the next `RestoreJob` go through the plugin's choice. Left to itself for every restore, the plugin would start from the newest base backup in the archive whenever `recoveryTime` is empty, and from the newest one ending by `recoveryTime` otherwise, whichever `Backup` the `RestoreJob` names and on any timeline — see [Archives with several timelines](#archives-with-several-timelines) for why that matters. - -```yaml -apiVersion: backups.cozystack.io/v1alpha1 -kind: RestoreJob -metadata: - name: orders-db-pitr - namespace: tenant-acme -spec: - backupRef: - name: orders-db-adhoc # a completed Backup of the source - targetApplicationRef: # omit for a destructive in-place restore - apiGroup: apps.cozystack.io - kind: Postgres - name: orders-db-copy # a pre-deployed, empty Postgres app - options: - recoveryTime: "2026-07-21T09:30:00Z" -``` - -A full, scripted example (write a marker, capture the timestamp, restore to it, assert what survived) is in [`examples/backups/postgres/`](../../examples/backups/postgres/) — `45-restorejob-pitr.yaml`, driven by `run-all.sh`. - -### The recoverable window - -`recoveryTime` must fall inside the window the archive can reconstruct: - -- **Earliest** — the completion of the oldest base backup still in the archive. You cannot recover to an instant before the first base backup: WAL replay always starts from a base backup, and retention (`barmanObjectStore.retentionPolicy`) eventually trims the oldest ones together with the WAL that predates them. Aim at least one second after a backup's `status.stoppedAt`: `stoppedAt` is truncated to the second, while the plugin compares `recoveryTime` with the end time barman records in `backup.info`, to the microsecond, and keeps only a backup that ended at or before it. A `recoveryTime` equal to `stoppedAt` therefore falls before that backup's actual end: an older backup is used instead, and when there is none the restore fails with `no target backup found`. -- **Latest** — the last transaction, commit or abort, contained in the WAL shipped to object storage. It is not the newest WAL segment: a time-targeted recovery ends just before the first transaction recorded after `recoveryTime`, and a segment switch, a checkpoint or the end of a base backup is not one. A `recoveryTime` at or after the last archived transaction leaves PostgreSQL nothing to stop on. On a busy database the last transaction trails "now" by the archive lag; on a database at rest it is its last write, however recent the WAL archive and however recent the latest base backup. Once the source is deleted, the window is frozen at the last transaction that made it to S3 before it went away. - -A `recoveryTime` **at or after the last archived transaction cannot converge**: PostgreSQL replays every available WAL, never finds a transaction after the target, exits with `FATAL: recovery ended before configured recovery target was reached`, and CNPG re-creates the recovery instance in a loop. Such a wedged restore is failed by the restore deadline (`spec.options.restoreTimeoutSeconds`, default 30m) — and the driver reads the recovery pod's log at that point, so instead of a generic timeout the RestoreJob ends `status.phase: Failed` with conditions `RecoveryConverged=False` and `Ready=False`, both reason `RecoveryTargetUnreachable`, and a message naming the target. The driver deliberately does **not** fail earlier off that FATAL: a *reachable* near-now target hits the identical FATAL transiently — often repeatedly, on a slow archiver — before it converges, so an earlier trip would reject a recoverable restore. To reject an unreachable target quickly, set a short `restoreTimeoutSeconds`; otherwise pick a time before the last archived transaction. The state as of the last write needs no `recoveryTime` at all: omitting it replays the whole archive, and that is the state it recovers. - -Restoring to a **very recent** instant is safe as long as the database keeps writing and WAL archiving is current: the segment covering the target, or the transaction after it, may not be in object storage yet and the same FATAL fires transiently, but because the driver only fails at the deadline (not on that FATAL), the recovery has the whole window to catch up — the next attempt promotes once the WAL ships and the cluster goes healthy, which completes the restore. Only a target that never becomes reachable within the deadline is failed: on a database that has stopped writing, that is every target after its last transaction. If archiving is badly behind (a stalled `archive_command`, an overloaded cluster) a near-now target can exceed even the default 30m window; restore to a point you can confirm is followed by an archived transaction (see the discovery section below). - -A `recoveryTime` **before the earliest base backup** fails differently: PostgreSQL cannot begin replay before the base backup it restored, so recovery never reaches a consistent state and never emits the "recovery ended before …" FATAL the driver classifies on. That case also surfaces at the deadline, but with the generic reason `RestoreFailed` rather than `RecoveryTargetUnreachable`. Choosing a `recoveryTime` at least one second after the oldest base backup's `stoppedAt` (below) avoids it. - -### Discovering the earliest / latest restorable time - -CNPG's `Cluster.status.firstRecoverabilityPoint` exists in the status schema but is unreliable under the barman-cloud plugin — it was observed empty on every plugin-backed cluster in a test environment (CNPG 1.28.1) — so read the window from the backup catalog instead. Each completed base backup records its WAL range and timestamps on the underlying `cnpg.io/Backup`: - -```bash -# Completed base backups, oldest first: STOP plus one second is the earliest -# instant that backup alone can restore to; the oldest one bounds the window. -kubectl -n get backups.postgresql.cnpg.io \ - --sort-by=.status.stoppedAt \ - -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,ID:.status.backupId,START:.status.startedAt,STOP:.status.stoppedAt,BEGINWAL:.status.beginWal,ENDWAL:.status.endWal - -# The cozystack Backup points at its underlying cnpg.io/Backup and S3 prefix. -kubectl -n get backup.backups.cozystack.io -o jsonpath='{.spec.driverMetadata}' -``` - -The upper bound is the timestamp of the last `COMMIT` or `ABORT` record in the archived WAL, which nothing in the cluster status reports. `pg_waldump` prints it from the newest segments under `//wals/`, working back from the newest segment until one holds such a record; on a database at rest that can be several segments back, because every segment switch and every base backup adds one with no transaction in it. Any `recoveryTime` before that timestamp is reachable; for the state as of that last transaction, omit `recoveryTime`. - -### Archives with several timelines - -A WAL archive can hold several PostgreSQL timelines under one `serverName`. Every failover starts one, and so did every in-place restore made before restored clusters archived to a fresh prefix (`bootstrap.newServerName`): those wrote the new timeline back into the prefix they had recovered from, often without its `.history` file, so the archive ends up with branches that are siblings rather than a single line of descent. - -Recovery follows `recovery_target_timeline: latest`, which CNPG leaves as PostgreSQL's default: starting from the base backup's timeline, PostgreSQL looks for the history file of the next timeline number, then the next, stops at the first one missing, and targets the last one it found, replaying only the segments on that line of descent. Without any history file that is the base backup's own timeline alone. When the timeline it targets does not descend from the base backup — a sibling branch whose history file happens to carry the next number — recovery does not fall back to another timeline: it stops with `FATAL: requested timeline N is not a child of this server's history` (or `... is not in this server's history`). Two consequences: - -- the last archived transaction that bounds a restore is the last one on the timeline that restore follows, not the newest one in the archive; -- a restore must start from a base backup on the branch it means to bring back. The plugin's own choice ignores timelines — CNPG passes no `targetTLI` — so the newest backup, or the newest one ending by `recoveryTime`, can lie on another branch and restore that branch's data instead. The driver pins the `Backup`'s own base backup with `recoveryTarget.backupID` for that reason; a `recoveryTime` earlier than that backup's end still goes through the plugin's time-based choice, and on such an archive can land on another branch. - -### Idempotency under GitOps - -An in-progress restore is safe to reconcile. The driver purges the target `Cluster` + PVCs exactly once per RestoreJob (guarded by the `TargetPurged` condition and by the `postgres.cozystack.io/restored-server-name` annotation, which identifies the RestoreJob whose purge produced the live recovery `Cluster`), suspends the target's HelmRelease across the purge so Flux cannot race the bootstrap swap, and resumes it once the recovery cluster is rendered. A Flux reconcile (or a controller restart) mid-restore therefore re-attaches to the recovering cluster rather than deleting it and starting over. - -## Kafka: topic metadata only - -The `cozy-default-kafka` strategy backs up **topic metadata**, not message data. Its Job talks to the Kafka Admin API (`kafka-topics --describe` on backup, `kafka-topics --create` + `kafka-configs --alter` on restore) and stores one small object per run at `s3://////kafka-metadata.txt`. It captures every non-internal topic's partition count, replication factor and non-default configs; message payloads, consumer-group offsets, ACLs, quotas and `KafkaUser`s are out of scope, and there is no point-in-time recovery. The driver gates each run on the Strimzi `Kafka` cluster reporting `Ready` and never mutates it. - -This is the "reconstruct a cluster's topic topology" flow: an in-place restore recreates dropped topics into the source, and a to-copy restore applies the source's topics onto a freshly-bootstrapped empty `Kafka` (partitions and configs preserved). Restore is additive at the topic-set level — it never deletes topics that exist live but not in the backup — but it does not silently accept a divergent existing topic: it grows the partition count when the backup asks for more, and fails loudly (naming the topic and both values) when the backup asks for fewer partitions or a different replication factor, since Kafka cannot shrink partitions or change RF in place and a restore that cannot reach the recorded state must not report `Succeeded`. Restored topics are created directly through the Admin API, so on the target they are **unmanaged** (no `KafkaTopic` CR); that is the accepted tradeoff of an Admin-API-only driver. Restoring a topic whose replication factor exceeds the target cluster's broker count fails at `--create`. Topic configs are restored **additively** (`kafka-configs --add-config`): a config the backup recorded is re-applied, but a config added to a live topic after the backup is not removed, so an in-place restore does not return a topic's config set to its exact backup-time state (partition count and replication factor are the only shape the restore reconciles or fails loudly on). - -One consequence of the Admin-API approach: if a topic is declared in the Kafka app's `spec.topics`, the Strimzi Topic Operator owns it and reconciles its config from the `KafkaTopic` CR, reverting any dynamic config an in-place restore sets that the CR does not list. So this strategy is durable for **out-of-band** topics (created directly, no CR); for CR-declared topics the `KafkaTopic` CR / GitOps is the source of truth. A to-copy restore onto a fresh empty cluster is unaffected. For message-data durability, use a volume-snapshot strategy instead. See `examples/backups/kafka-metadata/` for the end-to-end flow. - -Restore is not transactional: it applies topics one at a time as it reads the object, so a failure part-way through (an RF mismatch, an unreachable broker) leaves the topics already created behind — re-running the restore is safe (existing topics are reconciled, not recreated). Like the Rabbitmq driver, this one owns its object outright and deletes it from the bucket when its `Backup` is deleted — via a one-shot Job the Backup's removal waits on — so a retention-pruned `Plan` does not accumulate objects; the delete is **best-effort** with the same give-up conditions and `backups.cozystack.io/skip-artifact-cleanup` escape hatch documented for Rabbitmq above. One caveat specific to this driver: its image ships an older `curl` (7.76.1) whose `--aws-sigv4` omits a non-default port from the SigV4 canonical host, which every S3 backend then rejects with `403 SignatureDoesNotMatch` on a ported endpoint (fixed in curl 7.86.0). The strategy script works around it with `--connect-to` — it signs and sends a portless URL and redirects the connection to the real port — so a ported endpoint (the `backupStorage.endpoint` fallback `…:8333`, `provisionBucket: false`, or SeaweedFS with `s3.ingress.enabled: false`) works too; the sibling drivers run curl 8.x and need no such workaround. Export time scales with topic count — the backup spends two Kafka CLI invocations (each a fresh JVM, ~1.5–3s) per topic — so a cluster with many hundreds of topics takes tens of minutes; the run is bounded at the Job layer by `activeDeadlineSeconds` (default 30 minutes, from the same wait the readiness precondition uses), so a wedged or unschedulable run is failed by Kubernetes as `DeadlineExceeded` — the pod is killed before the script's final upload, so no object is left orphaned — rather than requeuing forever. A cluster whose export legitimately runs longer raises the bound with the `backupTimeout` BackupClass parameter (a Go duration, e.g. `2h`), which round-trips through the `Backup` so a restore is bounded by the same value. diff --git a/docs/operations/resource-presets.md b/docs/operations/resource-presets.md deleted file mode 100644 index 95a8fb59a..000000000 --- a/docs/operations/resource-presets.md +++ /dev/null @@ -1,52 +0,0 @@ -# Resource presets - -Every cozystack app exposes a `resourcesPreset` field whose value selects a CPU and memory reservation. Presets follow a cloud-style `.` naming convention. Setting `resources:` explicitly overrides the preset. - -## Series - -| Series | CPU:Memory ratio | When to use | -| ------ | ---------------- | ------------------------------------------------- | -| `t1` | 1:0.5 | Tiny / burstable, low memory | -| `c1` | 1:1 | Compute-balanced, CPU-bound workloads | -| `s1` | 1:2 | Standard — proxies, caches, lightweight services | -| `u1` | 1:4 | Universal — databases and messaging | -| `m1` | 1:8 | Memory-heavy — search, analytics, large caches | - -## Watch out: legacy and instance-type `medium` differ - -Legacy `medium` had **1 CPU / 1Gi**. The new `*.medium` sizes have **2 CPU** (in any series). The names overlap but the resources do not. The migration table below stays correct — `medium → c1.small (1 CPU / 1Gi)` — but if you read the instance-type sizing matrix first and pick `c1.medium` "to keep things the same", you will double your CPU. When in doubt, consult the legacy-to-instance-type mapping at the bottom of this page. - -## Sizes - -| Size | CPU | t1 mem | c1 mem | s1 mem | u1 mem | m1 mem | -| ---------- | ---- | ------ | ------ | ------ | ------ | ------ | -| `nano` | 250m | 128Mi | 256Mi | 512Mi | 1Gi | 2Gi | -| `micro` | 500m | 256Mi | 512Mi | 1Gi | 2Gi | 4Gi | -| `small` | 1 | 512Mi | 1Gi | 2Gi | 4Gi | 8Gi | -| `medium` | 2 | 1Gi | 2Gi | 4Gi | 8Gi | 16Gi | -| `large` | 4 | 2Gi | 4Gi | 8Gi | 16Gi | 32Gi | -| `xlarge` | 8 | 4Gi | 8Gi | 16Gi | 32Gi | 64Gi | -| `2xlarge` | 16 | 8Gi | 16Gi | 32Gi | 64Gi | 128Gi | -| `4xlarge` | 32 | 16Gi | 32Gi | 64Gi | 128Gi | 256Gi | - -Ephemeral storage is 2Gi for every preset. - -## Legacy flat names (deprecated) - -The following short names existed before the instance-type rename and remain accepted for backward compatibility. They render exactly the CPU and memory they did before — the legacy block in `cozy-lib` returns the original values verbatim. The 1:1 mapping is: - -| Legacy | CPU | Memory | Instance-type equivalent | -| -------- | ---- | ------ | ------------------------ | -| `nano` | 250m | 128Mi | `t1.nano` | -| `micro` | 500m | 256Mi | `t1.micro` | -| `small` | 1 | 512Mi | `t1.small` | -| `medium` | 1 | 1Gi | `c1.small` | -| `large` | 2 | 2Gi | `c1.medium` | -| `xlarge` | 4 | 4Gi | `c1.large` | -| `2xlarge`| 8 | 8Gi | `c1.xlarge` | - -Legacy names are scheduled for removal in a future cozystack release. New manifests should use the instance-type form. `cozystack-api` emits a `klog` warning whenever an app CR carries a legacy value, naming the suggested replacement. - -## Automatic migration - -Platform upgrades run `migration 39` as a pre-upgrade hook. It walks every `HelmRelease` `spec.values` and every app CR under `apps.cozystack.io/v1alpha1`, rewriting legacy `resourcesPreset` values in place using the table above. The conversion is idempotent and best-effort: a failure on one object logs a warning and leaves the rest of the migration to complete. Set `MIGRATION_DRY_RUN=1` to preview the patches without applying them. diff --git a/docs/operations/seaweedfs-capacity-for-backups.md b/docs/operations/seaweedfs-capacity-for-backups.md deleted file mode 100644 index eb6e5c189..000000000 --- a/docs/operations/seaweedfs-capacity-for-backups.md +++ /dev/null @@ -1,37 +0,0 @@ -# SeaweedFS capacity for backups - -A SeaweedFS instance starts with two volume servers of `volume.size: 10Gi` each and `replicationFactor: 2`, so every byte it stores lands on both servers and the instance holds a little under 10 GiB of objects. That is enough for occasional backups and restores. It is not enough for continuous WAL archiving from a Postgres cluster that writes steadily: WAL is archived around the clock and kept for the whole retention window, so the bucket grows until the window is full, and an instance sized by default runs out long before then. - -A volume server stops taking writes in one of two ways. Once its free disk falls below `minFreeSpacePercent` (5%), which it checks every minute, every volume on it turns read-only and it takes no new volume, so every bucket with a copy there stops at once. Before that, it can run out of slots: the volumes that already exist keep taking writes until each reaches `master.volumeSizeLimitMB`, a bucket whose volumes are all full needs a new one, and a server with no room left for another volume of the size limit cannot host it. When no server can, the master logs `Not enough data nodes found` and every upload into that bucket fails. The slots can run out while the disk is still half empty: a volume server counts every volume it holds at the full size limit, so the nearly empty volumes of other buckets keep one bucket from growing. `barman-cloud-wal-archive` then fails on every segment, WAL segments stay on the Postgres volume, and the database goes down when that volume is full. - -## Estimating what a bucket needs - -For each Postgres cluster that archives into the instance, the bucket holds the WAL of the whole retention window plus every base backup inside it: - -```text -bucket ≈ WAL per day × retention days + database size × (base backups in the window + 1) -``` - -The WAL rate of a running cluster, in bytes per day before compression, is roughly the number of segments it archived over a day times the 16 MiB segment size. The count also includes the small history and backup-label files, and the sum covers every pod that was primary during the day: - -```promql -sum by (namespace, job) (increase(cnpg_pg_stat_archiver_archived_count{namespace=""}[1d])) * 16 * 1024 * 1024 -``` - -The platform's default backup strategy compresses both WAL and base backups with gzip, so the stored size is lower, by an amount that depends on the data; a cluster backed up to an object store of its own choosing stores them uncompressed unless its ObjectStore says otherwise. The default `retentionPolicy` is `30d`, so a cluster that writes 4 GB of WAL a day needs about 120 GB of WAL alone before its base backups are counted. - -Each volume server must hold the sum of those buckets, since at `replicationFactor: 2` with two servers each server carries a full copy. Size `volume.size` so that this sum stays below 95% of it, and leave room on top for every bucket that is still growing: at `replicationFactor: 2` a grow claims three volumes of `master.volumeSizeLimitMB` on each server. - -## Alerts that watch the instance - -The generic `KubePersistentVolumeFillingUp` alert does not cover this. It pages below 3% free, while a volume server stops at 5% or, out of slots, with its disk half empty, and its early warning needs the disk to keep filling, which it stops doing once writes fail. Each instance carries its own rules instead: - -- `SeaweedFSVolumeServerDiskLow` (warning): a volume server has less than 10% of its disk left before the `minFreeSpacePercent` reserve. -- `SeaweedFSVolumeServerFull` (warning): a volume server cannot host another volume, because it is at the reserve, where all its volumes are read-only, or has no slot left, where buckets whose volumes on it are full stop taking writes. -- `SeaweedFSWritesFailing` (critical): the master has failed to place a write in every 5-minute window for 15 minutes. Other writes may still succeed in between. The usual cause is a bucket with no writable volume left that the master cannot grow, whose uploads all fail; a steady stream of newly created buckets, each failing a few picks while its first volumes grow, looks the same. - -On the Postgres side, `LastFailedArchiveTime` fires while WAL archiving keeps failing, and `KubePersistentVolumeFillingUp` fires as the WAL backlog fills the Postgres volume. - -## Growing an instance - -Raise `volume.size` on the SeaweedFS application. The upgrade expands each volume server's PVC in place, which needs a StorageClass that allows volume expansion. Postgres keeps retrying the segments that waited on its volume, so archiving catches up once the master can grow the bucket again. diff --git a/docs/operations/system-memory-limits.md b/docs/operations/system-memory-limits.md deleted file mode 100644 index af3076f0e..000000000 --- a/docs/operations/system-memory-limits.md +++ /dev/null @@ -1,179 +0,0 @@ -# System Component Memory Limits - -Cozystack defaults container memory in eligible system namespaces and sets explicit requests and limits on the node DaemonSets. This page explains why that is not merely resource hygiene, how to tune it, and what it deliberately does not cover. - -## Why a memory limit, and not a PriorityClass - -Talos Linux v1.12 introduced a userspace OOM handler, enabled by default, that reacts to memory pressure before the kernel OOM killer does. Its only input from the machine is its own config document; everything else it reads straight out of cgroupfs, and it holds no Kubernetes client at all. `PriorityClass` — `system-node-critical` included — therefore has no bearing on which cgroup it picks. - -Setting a priority on a system component to stop these kills looks like a fix and changes nothing about them. Take that claim narrowly, because priority is far from inert elsewhere: it still drives scheduler preemption, kubelet eviction ordering, and the `oom_score_adj` kubelet hands the kernel OOM killer. None of those three is what kills these pods. The one ordering input this handler has is the Kubernetes **QoS class**, inferred from the cgroup path rather than from any pod object — a different axis from priority, so a `system-node-critical` pod with no memory limit is an ordinary candidate like any other. - -The handler scores each pod cgroup with a CEL expression. The default in Talos v1.14.2, the version Cozystack currently ships for host nodes, is: - -``` -memory_max.hasValue() ? 0.0 : - {Besteffort: 1.0, Burstable: 0.5, Guaranteed: 0.0, Podruntime: 0.0, System: 0.0}[class] * - double(memory_current.orValue(0u)) -``` - -Any cgroup scoring zero is dropped from the candidate set entirely. A pod whose containers all carry a memory limit has `memory.max` set on its pod cgroup, scores `0.0`, and can never be selected. A pod without one stays a candidate no matter how little memory it is using. - -Three consequences follow, and all three are easy to get wrong: - -- Only a **limit** grants immunity. A memory **request** merely moves the pod from BestEffort to Burstable, which under the default `strictCgroupClassOrdering: true` means "killed second" rather than "not killed" — Burstable cgroups are considered only once no BestEffort one is eligible, and the score then breaks ties within the class. That setting arrived in v1.13.4; on v1.13.0 through v1.13.3 the score alone decided, so a large Burstable pod could outrank a small BestEffort one. -- **Every** container in the pod needs a limit, init containers included. Kubelet only sets pod-level `memory.max` when all of them have one, so a single limit-free sidecar puts the whole pod back in the candidate set. -- CPU limits are irrelevant here. The ranking expression is given the cgroup's path, its QoS class, and `memory.max`, `memory.current` and `memory.peak` — no CPU information of any kind. - -Victim selection is also decoupled from the trigger. The cgroup that caused the pressure and the cgroup that gets `SIGKILL`ed are unrelated by design. In practice the pods carrying limits are overwhelmingly tenant workloads — managed applications are sized through resource presets, and a tenant namespace with `resourceQuotas` configured gets a default limit of its own — while system components carried none until the change this page describes. That inverts the intended order: whatever fires the trigger, `metallb-speaker` or `linstor-satellite`, using a few dozen megabytes and entirely uninvolved, is the one killed for it, repeatedly, until the pressure subsides. On Talos v1.13.x a tenant workload thrashing against its own multi-gigabyte limit is enough to fire it, through the node-wide PSI clause described [below](#when-the-trigger-itself-is-the-problem). - -## What Cozystack does - -**A default LimitRange in eligible system namespaces.** The `cozystack-operator` maintains a `LimitRange` named `cozystack-system-defaults` in each namespace it reconciles whose name does not begin with `tenant-`, defaulting container memory for anything that declares none. It covers containers whose upstream chart exposes no `resources` knob. Namespaces with a foreign memory policy or an unresolved safety hold are excluded. - -**Explicit requests and limits on node DaemonSets.** Charts additionally set real values on the DaemonSets that run on every node — the cilium agent and its init containers, metallb speaker, the frr-k8s controller and its sidecars, the linstor satellite along with plunger and drbd-logger, virt-handler, fluent-bit, node-exporter, multus and its init container, velero node-agent, and all three kubevirt-csi-node containers. Requests there are fitted to observed usage, which is a real signal for the scheduler; the LimitRange cannot supply that, because it applies one number to every container in the namespace and so has to keep its default request deliberately tiny. kube-ovn is absent from that list because its vendored chart already sets both. - -The operator's LimitRange stops at the `tenant-` prefix, so tenant namespaces never receive it. A tenant namespace gets a default of its own only from the tenant chart's `tenant-range-limits`, which is rendered only when `resourceQuotas` is configured on the tenant and is empty by default. A tenant workload left without a memory limit therefore stays an eviction candidate, which is the upstream design working as intended and is what restores the ordering described above. - -## Tuning - -Two installer values, both empty by default so the operator's own defaults apply: - -| Value | Operator flag | Default | -|---|---|---| -| `cozystackOperator.systemNamespaceMemoryLimit` | `--system-namespace-memory-limit` | `32Gi` | -| `cozystackOperator.systemNamespaceMemoryRequest` | `--system-namespace-memory-request` | `32Mi` | - -The limit is a ceiling, not a reservation, and the default is deliberately far above any real working set. The handler discards a cgroup for carrying a `memory.max` at all, whatever its value, so what the default buys is immunity rather than a fitted ceiling. That is also why it is set so high: the scan described below sees *declared requests*, never a container's actual resident set, so a ceiling low enough to bind is a ceiling that can convert a rare pressure-driven kill into a deterministic one for any component whose real usage nobody measured. Raising it is close to free; lowering it is where the risk lives, and it is worth doing per component in that component's own chart before doing it fleet-wide here. - -**The limit must stay above the largest memory request in any system namespace.** A defaulted limit below a container's own request is rejected at admission, and the pod simply will not start. List the requests before lowering it — the operator marks exactly the namespaces it treats as system with `cozystack.io/system=true`, the same condition under which it creates the LimitRange: - -```bash -for ns in $(kubectl get ns -l cozystack.io/system=true -o jsonpath='{.items[*].metadata.name}'); do - kubectl get pods -n "$ns" -o json | jq -r --arg ns "$ns" ' - .items[] - | [.spec.containers[], (.spec.initContainers // [])[]][] - | select(.resources.requests.memory != null) - | "\($ns)\t\(.name)\t\(.resources.requests.memory)"' -done | sort -u -``` - -**Keep the request small.** The operator always pairs the default limit with a default request, because a `LimitRange` that sets `default` without `defaultRequest` makes each container's request equal its limit and reserves the whole ceiling at schedule time. Note that leaving `systemNamespaceMemoryRequest` empty does not produce that state: an empty installer value omits the flag, which selects the operator's own `32Mi`. The knob is there to be raised or lowered, and clearing it is not a way to switch the request off. - -The operator also refuses to start when the request exceeds the limit. A `LimitRange` whose `defaultRequest` is above its `default` is rejected by the API server, which would wedge namespace reconciliation for every system package, so the check happens at startup rather than at apply time. - -Setting the limit to `0` disables the feature, so the knob is reversible. Leaving it empty is not the same thing: that selects the operator's `32Gi` default. - -On the next Package reconcile after the feature is disabled, the operator lists LimitRanges cluster-wide by `app.kubernetes.io/managed-by: cozystack-package-controller` and removes every object it owns, including leftovers in namespaces no active Package targets. A same-named LimitRange without that ownership label is left alone. The same ownership boundary applies while the feature is enabled: a foreign object already holding the `cozystack-system-defaults` name is neither overwritten nor adopted, even if it contains only CPU or storage policy, so that namespace receives no Cozystack memory default until the name is free. - -## Withholding and acknowledging a default - -**An unsafe default stays withheld until a cluster administrator acknowledges it.** Before applying, the operator scans Deployments, StatefulSets, DaemonSets, CronJobs, Jobs and Pods, including init containers, for requests above the configured limit without an explicit limit. A visible blocker withdraws the managed LimitRange and records a hold on the Namespace. Removing a Pod or template does not release the hold: the same empty snapshot can occur between deletion and recreation by another controller. - -Lowering a previously applied default also creates a hold, even if no blocker is visible. LimitRanger persists its default on admitted Pods, so a stored limit cannot prove that the workload's source declares one. The operator remembers the previous limit independently of the LimitRange and does not infer permission to tighten from a Pod's `limits` field or `kubernetes.io/limit-ranger` annotation. - -The cluster-scoped Namespace carries `operator.cozystack.io/system-memory-state`, a JSON object with `version`, `lastLimit`, and, when held, `hold`, `target` and `reason`. This is operator-managed safety state. It survives controller restart, removal of the LimitRange, removal of the Package target, and disabling the feature. Do not remove or edit it to clear a hold: doing so discards the evidence protecting future Pod admission. Invalid state blocks new defaults until a cluster administrator repairs it. - -Inspect the hold and the required acknowledgement: - -```bash -kubectl get namespace -o json | jq -r ' - .metadata.annotations["operator.cozystack.io/system-memory-state"] - | fromjson | {reason, target, acknowledgement: (.hold + ":" + .target)}' -``` - -Before acknowledging, inspect the source of every affected workload, including custom resources and static-pod manifests that this scanner cannot read. Give the affected container an explicit limit at least as large as its request, lower its request only when appropriate, or raise the configured default. A currently running Pod's defaulted limit is not that source. Also check actual memory usage: admission safety does not make a small runtime ceiling safe. - -Copy the exact `acknowledgement` value reported for the current target into a separate annotation: - -```bash -kubectl annotate namespace \ - operator.cozystack.io/system-memory-ack=':' --overwrite -``` - -This requires permission to patch the cluster-scoped Namespace; a tenant's namespaced RoleBinding does not grant it. The operator grants no additional permission for acknowledgement. A token authorizes only its hold generation and canonical target quantity. Changing the configured target rotates the hold, even when returning to an earlier value. The operator consumes a matching token once; if a concrete request-above-default blocker still exists, the token is rejected and removed, and the hold stays. Acknowledgement never overrides a foreign LimitRange. - -After successful acknowledgement and a clean scan, the configured default is applied. Existing Pods keep their resources; the new default reaches later admissions. Package reconciles run after ordinary events and are additionally scheduled every five minutes, so acknowledgement and workload changes also progress on a quiet cluster, subject to controller queue and API availability. The reason and required token are visible on the Namespace and in operator logs: - -```bash -kubectl -n cozy-system logs deploy/cozystack-operator | grep "withholding the default container memory limit" -``` - -A hold remains after disable/re-enable, and changing the configured value while held requires acknowledging the new token. The operator also removes its LimitRange when no active Package targets the namespace, considering all Packages sharing it; the Namespace safety state remains for a later reinstall. Setting the limit to zero removes managed defaults even if their safety state is invalid. - -**A namespace another LimitRange already governs is left alone.** If a system namespace — `kube-system` included — already carries a LimitRange that says anything about memory, the operator writes nothing there and withdraws anything it wrote earlier. Any memory-bearing field counts, at either the `Container` or the `Pod` scope: a second `default` leaves the effective ceiling to the order `LimitRanger` happens to iterate them, a `max` below this default or a `min` above the default request rejects the pod outright, and a `maxLimitRequestRatio` rejects it too, since a 32Gi default against a 32Mi default request is a ratio of 1024:1. The `Pod` scope is included because the collision there is not decidable in advance at all — whether a per-container default breaches a pod-wide bound depends on how many containers a not-yet-created pod will have. LimitRanges that bound only cpu or storage say nothing about memory and coexist normally. Note that withdrawing hands the namespace to the administrator's policy in full: if that LimitRange defaults container memory below some container's own request, the resulting admission rejection is theirs to resolve, and it is one that would have occurred without Cozystack in the picture at all. - -A LimitRange only mutates at admission. Existing pods keep running without limits until they restart, so the protection lands progressively as workloads roll rather than the moment the setting is applied. - -## Verifying - -Confirm a pod cgroup actually carries `memory.max` — this is the property that matters, not the QoS class. Talos runs kubelet with the `cgroupfs` driver on a unified cgroup v2 hierarchy, so a pod cgroup is a directory named for the pod UID, dashes and all: - -```bash -POD_UID=$(kubectl get pod -n -o jsonpath='{.metadata.uid}') -NODE=$(kubectl get pod -n -o jsonpath='{.spec.nodeName}') -NODE_IP=$(kubectl get node "$NODE" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}') -talosctl -n "$NODE_IP" read "/sys/fs/cgroup/kubepods/burstable/pod$POD_UID/memory.max" -``` - -A byte count rather than the literal `max` means the pod is out of the victim set. Note that `talosctl -n` takes a machine address rather than a Kubernetes node name, which is why the address is looked up instead of being reused from `nodeName`. - -Mind the QoS segment of that path, and do not extrapolate it. BestEffort and Burstable pods live under `kubepods/besteffort/` and `kubepods/burstable/`, but there is no `guaranteed` directory: kubelet creates QoS-level cgroups for those two classes only, so a Guaranteed pod sits directly at `kubepods/pod`. To avoid guessing, list the pod directories two levels down instead: - -```bash -talosctl -n "$NODE_IP" ls -d 2 -t d /sys/fs/cgroup/kubepods -``` - -`talosctl read` wants an `os:admin` talosconfig and a single node per call. Reading the same file through `kubectl debug node/$NODE --profile=sysadmin` under `/host/sys/fs/cgroup` also works, but it depends on the kubelet, on scheduling, and on a namespace that admits a privileged pod — all shakier than the Talos API in precisely the situation that makes you run this check. - -Inspect what the handler has actually killed: - -```bash -talosctl -n get oomactions -o yaml -``` - -Each entry carries the score the victim was selected on, the command lines of the processes in it, and a dump of the trigger context it fired under. Ask for `-o yaml`: the default table shows only a score, and its `Time` column reads a field the spec never sets, so it renders empty. This is a ring buffer of the last 50 actions held in memory, so it does not survive a `machined` restart and is no substitute for an alert — but it is still the better of the two signals here, being structured and complete where the logs are neither. - -Once every eligible pod in a namespace carries a limit, the healthy signature is the controller still triggering under pressure and finding nothing it is allowed to kill: - -```bash -talosctl -n logs controller-runtime | grep OOMController -``` - -Both halves of that signature — `OOM controller triggered`, then `no eligible cgroup to kill` — are emitted at the default log level, so neither needs a debug flag. Note that the structured fields are encoded as JSON rather than logfmt, so the second line reads `no eligible cgroup to kill {"component": "controller-runtime", "controller": "runtime.OOMController", "ranked": 0}`. Grep for `"ranked"` or for the message text; `ranked=` matches nothing and reads as though the handler were not running. The trigger firing is not itself a fault. - -`talosctl dmesg` carries the same lines and is the more common reflex, but it is the weaker source: that path strips log levels and timestamps, truncates at 976 bytes, and suppresses the first four occurrences of each ranking error — precisely the lines worth having if scoring ever misbehaves. - -## What this does not cover - -**Kubernetes static pods.** `kube-apiserver`, `kube-controller-manager` and `kube-scheduler` are managed by Talos rather than by any chart. A `cozystack-system-defaults` LimitRange does reach `kube-system`, since `cozystack-scheduler` installs there, but it cannot touch them: a LimitRange defaults resources at API-server admission, and kubelet builds static pods straight from files on disk without ever passing through it. Sizing those is a Talos machine-config matter. Their mirror pods pass admission. An already persisted mirror Pod with an oversized request and no limit creates a hold, and lowering the previous default requires acknowledgement. A mirror Pod rejected before persistence provides no scan evidence; its absence does not mean the static manifest is safe. - -**Future Pods outside the scanned templates.** A custom controller can change its desired request without producing a Deployment, StatefulSet, DaemonSet, CronJob or Job. If its first replacement Pod is rejected at admission, there is no new Pod for this scanner to inspect. An ownerless ReplicaSet at zero replicas has the same gap. A `RequestsOnly` VPA can likewise introduce an unseen request after defaulting. The hold prevents automatic lowering and automatic recovery after an observed blocker, but does not validate arbitrary future custom-resource changes. Configure explicit resources for those workloads and inspect their admission errors. - -**A handful of vendored containers with no upstream `resources` knob** — the four frr-k8s `cp-*` init containers, the kube-ovn `hostpath-init` and `install-cni` init containers, and `cozy-proxy`, whose chart exposes no resources value at all. These are covered by the namespace LimitRange and nothing else. That is enough for OOM immunity, since the defaulted limit is what puts `memory.max` on the cgroup, but their request is then the generic default rather than a figure fitted to measured usage, so the scheduler gets a weaker signal for them than for the DaemonSets above. Lifting that needs changes upstream. - -**Optional components** `hami` and `kilo` carry no chart-level values, on the grounds that inventing numbers for components with no usage measurements behind them is worse than the blanket default. - -## When the trigger itself is the problem - -Giving system components limits stops them being *victims*. It does not stop the handler *triggering*, and a node under genuine sustained pressure will keep firing. The v1.14.2 default trigger is: - -``` -(multiply_qos_vectors(d_qos_memory_full_total, {System: 8.0, Podruntime: 4.0}) > 3000.0 && - multiply_qos_vectors(qos_memory_full_avg10, {System: 1.0, Podruntime: 1.0}) > 5.0 && - time_since_trigger > duration("5s")) -``` - -It fires only on memory stalls in the `System` and `Podruntime` cgroups, so pressure confined to pods does not trip it. It reached that shape over four changes: [siderolabs/talos#12602](https://github.com/siderolabs/talos/pull/12602) replaced the original global-PSI trigger with the per-QoS form in v1.13.0, [#12632](https://github.com/siderolabs/talos/pull/12632) added the `qos_memory_full_avg10 > 5.0` conjunct to make it less sensitive, [#13725](https://github.com/siderolabs/talos/pull/13725) added the `5s` cooldown in v1.13.6, and [#13895](https://github.com/siderolabs/talos/pull/13895) dropped a second clause, `memory_full_avg10 > 75.0 && time_since_trigger > duration("10s")`, in v1.14.0. That clause was a cause-blind global-PSI backstop: a single workload thrashing inside its own cgroup limit could drive root `memory_full` above 75 while the node had free RAM. Any node still running a v1.13 release carries it, tenant Kubernetes workers included. A tenant worker's machine config is rendered by the `kubernetes-nodes` chart, which accepts no extra documents, so the workaround below is for host nodes only. - -On a v1.13 host node the backstop can be removed without upgrading, through an `OOMConfig` machine-config document that sets the v1.14 trigger — at the cost of the last-resort guard before the kernel OOM killer takes over. The document also accepts `cgroupRankingExpression`, `strictCgroupClassOrdering` and `sampleInterval`; anything left out keeps its default, so overriding the trigger alone is enough here: - -```yaml -apiVersion: v1alpha1 -kind: OOMConfig -triggerExpression: |- - (multiply_qos_vectors(d_qos_memory_full_total, {System: 8.0, Podruntime: 4.0}) > 3000.0 && - multiply_qos_vectors(qos_memory_full_avg10, {System: 1.0, Podruntime: 1.0}) > 5.0 && - time_since_trigger > duration("5s")) -``` - -Prefer fixing the workload that is generating the pressure. Persistent triggering on v1.14 means the system and runtime cgroups keep stalling for memory, and capping the pod that is growing into the node's memory addresses the cause rather than the symptom. On v1.13 it can also mean a pod sitting at its own memory ceiling and reclaiming constantly, which raising that pod's limit addresses. diff --git a/docs/storage-immutability.md b/docs/storage-immutability.md deleted file mode 100644 index 35c03a515..000000000 --- a/docs/storage-immutability.md +++ /dev/null @@ -1,45 +0,0 @@ -# `storageClass` is immutable after creation - -Every stateful Cozystack application that exposes a `storageClass` parameter declares it immutable in the chart schema (`x-kubernetes-validations: [{rule: "self == oldSelf"}]`). This document explains the contract and where it is enforced. - -## Why - -`storageClass` binds the StatefulSet's PVC template. Kubernetes fixes `PersistentVolumeClaim.spec.storageClassName` at PVC creation time — editing `volumeClaimTemplates[].spec.storageClassName` on an existing StatefulSet does not retroactively migrate any data. A user who edits `storageClass` on an existing resource may believe their data is being moved to the new class; it is not. Locking the field at the schema layer makes that contract explicit. - -## What enforces it - -| Consumer | Behavior | -| --- | --- | -| Cozystack UI (`packages/system/dashboard/images/console`) | Reads the CEL rule from the chart's `openAPISchema` and renders `storageClass` as a disabled, helper-text-annotated field on edit forms. Save-time overlay reinstates the original value before PUT. | -| `kubectl edit` / `kubectl patch` against the cozystack aggregated apiserver | **Currently accepted** — the apiserver does not yet evaluate CEL rules embedded in `ApplicationDefinition.openAPISchema`. The change passes through but does not propagate to existing PVCs (see "Why" above), so no data corruption is possible. Apiserver enforcement is tracked in cozystack/cozystack#2657. | -| `kubectl edit` / `kubectl patch` against native CRDs | Enforced today by the standard apiextensions apiserver. | - -## Apps covered - -The following charts annotate at least one `storageClass` field as immutable: - -- `clickhouse`, `foundationdb`, `harbor`, `kafka` (Kafka broker + KRaft controller), `kubernetes`, `kubernetes-nodes`, `mariadb`, `mongodb`, `nats`, `openbao`, `opensearch`, `postgres`, `qdrant`, `rabbitmq`, `redis`, `valkey`, `vm-disk`. - -The `kubernetes-nodes` chart annotates both its per-pool `storageClass` (the worker node system-disk class, defaulted `replicated`) and its `cluster` field immutable. `storageClass` binds the same PVC-template contract as every other chart in the table, so the plain `self == oldSelf` rule applies for the reason in "Why" above. `cluster` is immutable for a different reason: it wires the pool's CAPI objects to one parent cluster `kubernetes-`, and repointing a live pool at another cluster would orphan its running worker VMs rather than migrate them. - -## `vm-disk` also freezes `source` - -The `vm-disk` chart annotates `source` immutable in addition to `storageClass`, for a mechanism distinct from the PVC-template reasoning above. A `vm-disk` renders a single `DataVolume`, and `DataVolume.spec` is immutable in CDI: once the DataVolume exists, editing `spec.source` (or `spec.storage.storageClassName`) has no effect. The chart therefore looks the existing DataVolume up and reuses its stored `spec`, so an edit to `.Values.source` would otherwise render as a silent no-op while the HelmRelease still reported success — the CR and the cluster diverge with no diagnostic (cozystack/cozystack#2985). - -Because the aggregated apiserver does not yet evaluate the CEL rule (see the table above, cozystack/cozystack#2657), `vm-disk` adds a second, chart-level guard that is the substantive enforcement today: `templates/dv.yaml` fails the render when the requested `storageClass` or `source` differs from what the disk was created with. Both checks compare the operator's input recorded in an annotation at creation — `vm-disk.cozystack.io/storage-class` and `vm-disk.cozystack.io/source` — written only at creation and preserved verbatim afterward, never overwritten with an unapplied edit. The storage-class annotation is written even for the cluster default (`""`), and its key presence — not its value — marks a recorded class, so a disk created with `""` is still caught if later changed to a concrete class. Recording the input rather than reading the live object matters on both axes: the image-clone PVC name carries a chart-version prefix (`vm-image-` was renamed to `vm-default-images-`) that drifts across upgrades with no operator edit, and the live PVC's `storageClassName` on an adopted or foreign DataVolume may legitimately differ from the CR's default without being a change. - -Disks created before these annotations existed (or out-of-band) are handled conservatively so a working release never deadlocks. For `source`, such a disk is compared against its stored `spec.source` directly, but only when that stored source is a shape the chart itself produces (`blank`, `http` with just a `url`, `upload`, or a `pvc` with just `name`/`namespace`). An adopted DataVolume using `spec.sourceRef`, or a source type the chart cannot render (`registry`, `snapshot`, `imageio`, `http` with a `secretRef`, a foreign-namespace `pvc`), has no comparable provenance and is left alone rather than failed on a mismatch. The image-clone exemption fires only when the stored source is a `cozy-public` `pvc` **and** the request is itself an image — the one case where the `vm-image-` → `vm-default-images-` prefix drift is indistinguishable from a real image-name change; a change away from an image source to another type is still caught. For `storageClass` there is no safe live fallback (the CR default need not reflect an adopted disk's real class), so an un-annotated disk is simply not guarded here; apiserver CEL (cozystack/cozystack#2657) is the eventual guard for those. To actually change `storageClass` or `source`, delete and recreate the disk; size growth is the one mutable dimension and is handled separately by `templates/pvc-resize-hook.yaml`. - -The `storageClass` annotation records the effective value applied at creation, including when it came from the chart's `values.yaml` default (`replicated`) rather than an explicit operator value. Applications created through the aggregated apiserver have the default stamped into the CR, so it stays stable; but a direct HelmRelease consumer that omits `storageClass` records the chart default of the day, so changing that default in a future chart version would make every such untouched release start failing the render until its CR pins the old value or the disk is recreated. Treat the `storageClass` default as a frozen contract for the same reason the `source` serialization above is. - -The `source` annotation is a byte-for-byte `toJson` of `.Values.source`, so it is coupled to the exact serialization of the source schema. Adding a subfield with a schema default (which the apiserver would stamp onto every CR) would make the recorded and requested JSON diverge on an untouched disk; such a change needs an annotation-migration story rather than a bare schema bump. - -If you add a new stateful chart that exposes `storageClass`, annotate it the same way: - -```yaml -## @param {string} storageClass - StorageClass used to store the data. -## @immutable -storageClass: "" -``` - -For fields declared inside a `@typedef`, place `## @immutable` on the line below the `## @field` it should attach to. diff --git a/examples/backups/clickhouse/README.md b/examples/backups/clickhouse/README.md index 3b9fc0d3e..1d392df5d 100644 --- a/examples/backups/clickhouse/README.md +++ b/examples/backups/clickhouse/README.md @@ -6,7 +6,7 @@ > `backup.enabled: true` and `backup.useSystemBucket: true` on the > ClickHouse release; tenants do not provision a Bucket or supply S3 > credentials. See -> [Backup Classes](../../../docs/operations/backup-classes.md) +> [Backup Classes](https://cozystack.io/docs/v1/operations/services/backup-classes/) > for the supported BackupJob / Plan flow against `cozy-default`. The > walk-through below covers the **legacy** path that wires a per-app > Bucket, custom strategy CR, and bespoke BackupClass — useful when an diff --git a/examples/backups/etcd/README.md b/examples/backups/etcd/README.md index c0775cb0a..d2b7a7858 100644 --- a/examples/backups/etcd/README.md +++ b/examples/backups/etcd/README.md @@ -5,7 +5,7 @@ > the system bucket `cozy-backups`. Tenants reference `cozy-default` > directly from BackupJob / Plan / RestoreJob without provisioning a > Bucket or supplying S3 credentials. See -> [Backup Classes](../../../docs/operations/backup-classes.md). +> [Backup Classes](https://cozystack.io/docs/v1/operations/services/backup-classes/). > The walk-through below covers the **legacy** path that wires a > per-app Bucket and bespoke BackupClass — useful for tuned non-default > policies. diff --git a/examples/backups/mariadb/README.md b/examples/backups/mariadb/README.md index 97b2aa17e..fce3155a9 100644 --- a/examples/backups/mariadb/README.md +++ b/examples/backups/mariadb/README.md @@ -5,7 +5,7 @@ > the system bucket `cozy-backups`. Tenants reference `cozy-default` > directly from BackupJob / Plan / RestoreJob without provisioning a > Bucket or supplying S3 credentials. See -> [Backup Classes](../../../docs/operations/backup-classes.md). +> [Backup Classes](https://cozystack.io/docs/v1/operations/services/backup-classes/). > The walk-through below covers the **legacy** path that wires a > per-app Bucket and bespoke BackupClass — useful for tuned non-default > policies (custom retention, encryption, secondary buckets). diff --git a/examples/backups/mongodb/05-mongodb-src.yaml b/examples/backups/mongodb/05-mongodb-src.yaml index d828d44bc..3b8b1f3bf 100644 --- a/examples/backups/mongodb/05-mongodb-src.yaml +++ b/examples/backups/mongodb/05-mongodb-src.yaml @@ -8,7 +8,7 @@ # flow, where the application owns the storage and the driver does not inject it # — so, unlike MariaDB, the application must opt into backups. (Set # backup.useSystemBucket: true instead to have the driver inject the platform -# system-bucket storage; see docs/operations/backup-classes.md.) +# system-bucket storage; see https://cozystack.io/docs/v1/operations/services/backup-classes/.) # # The bucket coordinates (destinationPath / endpointURL / s3AccessKey / # s3SecretKey) point the storage at the Bucket provisioned in 00-bucket.yaml. diff --git a/examples/backups/mongodb/README.md b/examples/backups/mongodb/README.md index 42d44737c..f1fbf6118 100644 --- a/examples/backups/mongodb/README.md +++ b/examples/backups/mongodb/README.md @@ -23,7 +23,7 @@ Files are numbered so that `kubectl apply -f` order matches the dependency graph ## How MongoDB backups differ from the other drivers -The psmdb operator runs the pbm agents when the cluster has `backup.enabled: true`, and services an on-demand `PerconaServerMongoDBBackup` only once the storage the CR names is declared on the cluster. Where that storage comes from depends on the flow the application picked. On the default flow (this example) the chart declares it from the application's own `backup.*` values, and the `cozy-default-mongodb` Strategy only names it (`storageName: s3-storage`); the driver injects nothing. With `backup.useSystemBucket: true` the chart leaves the storage out and the driver injects the platform bucket's coordinates onto the cluster before each backup. Either way the driver surfaces a clear `Ready=False` precondition (rather than a silent hang) when the target cluster has backups disabled; see `docs/operations/backup-classes.md` for both flows. +The psmdb operator runs the pbm agents when the cluster has `backup.enabled: true`, and services an on-demand `PerconaServerMongoDBBackup` only once the storage the CR names is declared on the cluster. Where that storage comes from depends on the flow the application picked. On the default flow (this example) the chart declares it from the application's own `backup.*` values, and the `cozy-default-mongodb` Strategy only names it (`storageName: s3-storage`); the driver injects nothing. With `backup.useSystemBucket: true` the chart leaves the storage out and the driver injects the platform bucket's coordinates onto the cluster before each backup. Either way the driver surfaces a clear `Ready=False` precondition (rather than a silent hang) when the target cluster has backups disabled; see `https://cozystack.io/docs/v1/operations/services/backup-classes/` for both flows. Restores use `PerconaServerMongoDBRestore.spec.backupSource`, carrying the S3 destination read back from the source backup, so the same artifact restores diff --git a/examples/backups/postgres/35-restorejob-in-place.yaml b/examples/backups/postgres/35-restorejob-in-place.yaml index 16d7253e7..933d9625d 100644 --- a/examples/backups/postgres/35-restorejob-in-place.yaml +++ b/examples/backups/postgres/35-restorejob-in-place.yaml @@ -6,7 +6,7 @@ # recovery, same as the to-copy variant in 45-restorejob-pitr.yaml (which the # e2e drives with a real, in-window timestamp). Uncomment and set a timestamp # inside your recoverable window - see 45-restorejob-pitr.yaml and -# docs/operations/backup-classes.md ("Point-in-time recovery") for how to find +# https://cozystack.io/docs/v1/operations/services/backup-classes/ ("Point-in-time recovery") for how to find # the earliest/latest restorable time. --- apiVersion: backups.cozystack.io/v1alpha1 diff --git a/examples/backups/postgres/45-restorejob-pitr.yaml b/examples/backups/postgres/45-restorejob-pitr.yaml index 7c9434f0c..b8e1e883d 100644 --- a/examples/backups/postgres/45-restorejob-pitr.yaml +++ b/examples/backups/postgres/45-restorejob-pitr.yaml @@ -11,7 +11,7 @@ # backup's completion and at/before the latest WAL shipped to object storage. # A target past the latest archived WAL cannot converge - the RestoreJob fails # with condition Ready=False/RecoveryTargetUnreachable instead of hanging. See -# docs/operations/backup-classes.md ("Point-in-time recovery") for how to +# https://cozystack.io/docs/v1/operations/services/backup-classes/ ("Point-in-time recovery") for how to # discover the earliest/latest restorable time. # # run-all.sh substitutes REPLACE_WITH_RECOVERY_TIME with a timestamp it captures diff --git a/hack/check-rd-presets.sh b/hack/check-rd-presets.sh index f91ef893a..c0fae2884 100755 --- a/hack/check-rd-presets.sh +++ b/hack/check-rd-presets.sh @@ -73,7 +73,7 @@ legacy_hits=$(grep -rEn 'defaultingSanitize.*\(list "(nano|micro|small|medium|la if [ -n "$legacy_hits" ]; then echo "FAIL: hardcoded legacy preset literals in chart templates:" >&2 echo "$legacy_hits" >&2 - echo "Replace with the instance-type form from docs/operations/resource-presets.md." >&2 + echo "Replace with the instance-type form from https://cozystack.io/docs/v1/guides/resource-management/." >&2 exit 1 fi echo "No hardcoded legacy preset literals in chart templates." diff --git a/hack/migration-50-etcd-adopt.bats b/hack/migration-50-etcd-adopt.bats index 22422ea77..2d3f7336b 100644 --- a/hack/migration-50-etcd-adopt.bats +++ b/hack/migration-50-etcd-adopt.bats @@ -264,7 +264,7 @@ lineno() { @test "external S3 (no COSI at all) keeps the Strategy CR endpoint scheme verbatim" { prep # provisionBucket: false on a cluster with no SeaweedFS/COSI at all - # (docs/operations/backup-classes.md describes exactly this flip), so the + # (https://cozystack.io/docs/v1/operations/services/backup-classes/ describes exactly this flip), so the # BucketClaim CRD itself is absent and every claim query fails the way real # kubectl fails on an unknown resource type. That must classify as external # and must NOT abort the script under `set -euo pipefail`. @@ -382,7 +382,7 @@ lineno() { @test "COSI with overridden bucket coordinates is still detected (no hardcoded names)" { prep # backupStorage.namespace / .bucketName are supported Package-CR overrides - # (docs/operations/backup-classes.md), and this hook cannot see chart values — + # (https://cozystack.io/docs/v1/operations/services/backup-classes/), and this hook cannot see chart values — # nothing plumbs them into the migration Job's env. So the claim may sit in any # namespace under any name. Keying on the COSI-assigned .status.bucketName, # which is what the projector republishes, makes the probe independent of both. diff --git a/internal/backupcontroller/backupjob_controller.go b/internal/backupcontroller/backupjob_controller.go index 791524240..c6b743356 100644 --- a/internal/backupcontroller/backupjob_controller.go +++ b/internal/backupcontroller/backupjob_controller.go @@ -101,7 +101,7 @@ func (r *BackupJobReconciler) Reconcile(ctx context.Context, req ctrl.Request) ( // so the tenant sees the misbinding instead of an infinite // requeue loop with no status signal. if errors.Is(err, ErrNoMatchingStrategy) { - return r.markBackupJobFailed(ctx, j, fmt.Sprintf("BackupClass %q does not bind a strategy for Kind %q; see docs/operations/backup-classes.md", j.Spec.BackupClassName, normalizedAppRef.Kind)) + return r.markBackupJobFailed(ctx, j, fmt.Sprintf("BackupClass %q does not bind a strategy for Kind %q; see https://cozystack.io/docs/v1/operations/services/backup-classes/", j.Spec.BackupClassName, normalizedAppRef.Kind)) } // BackupClass not found yet is transient on a fresh install: // during the bootstrap window the cozy-default BackupClass is @@ -290,7 +290,7 @@ func (r *BackupJobReconciler) handleProjectionError(ctx context.Context, j *back // Tenant-side projection failures must be visible in the same // Prometheus counter as system-side failures so the // "rate(failures_total) > 0 or absent_over_time(successes_total[10m])" - // alert in docs/operations/backup-classes.md catches them. The system + // alert in https://cozystack.io/docs/v1/operations/services/backup-classes/ catches them. The system // projector reports against system namespaces (cozy-velero etc.); // here we attribute against the BackupJob's tenant namespace. credentialsProjectionFailures.WithLabelValues(j.Namespace, classifyReason(err)).Inc() diff --git a/internal/backupcontroller/credentials_projector.go b/internal/backupcontroller/credentials_projector.go index a60666c38..4e0c3b75a 100644 --- a/internal/backupcontroller/credentials_projector.go +++ b/internal/backupcontroller/credentials_projector.go @@ -157,7 +157,7 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr // Scheme-strip the env-var fallback to match the source-Secret path // (parseSourceSecret strips there). The projected `endpoint` key is // consumed verbatim as S3_ENDPOINT by the ClickHouse sidecar, which - // expects a bare host:port — see docs/operations/backup-classes.md. + // expects a bare host:port — see https://cozystack.io/docs/v1/operations/services/backup-classes/. creds.endpoint = stripScheme(cfg.Endpoint) } if creds.endpoint == "" { diff --git a/internal/backupcontroller/psmdbstrategy_controller.go b/internal/backupcontroller/psmdbstrategy_controller.go index 702a3a1a0..0ead1bc58 100644 --- a/internal/backupcontroller/psmdbstrategy_controller.go +++ b/internal/backupcontroller/psmdbstrategy_controller.go @@ -482,14 +482,14 @@ func (r *BackupJobReconciler) reconcileMongoDB(ctx context.Context, j *backupsv1 // Opting out (useSystemBucket true→false) stops the injection but // leaves the driver's storage entry on the live cluster: the chart never // owned it and does not prune it, so it stays until removed by hand - // (backup-classes.md). A CR minted against it would carry no finalizer + // (https://cozystack.io/docs/v1/operations/services/backup-classes/). A CR minted against it would carry no finalizer // and land the dump in the platform bucket as an object nothing owns or // prunes. The injected entry is identified by its credentialsSecret, the // one field on it the driver chose: bucket names collide with an admin's // external bucket and drift when the platform bucket is re-provisioned. if psmdbBackupDeadlineExceeded(j.Status.StartedAt) { return r.markBackupJobFailed(ctx, j, fmt.Sprintf( - "psmdb.percona.com/PerconaServerMongoDB %s/%s storage %q is still the entry injected for the system bucket (credentialsSecret %q) while backup.useSystemBucket=false (%s after the job started); delete that storage entry from the PerconaServerMongoDB by hand or set backup.useSystemBucket=true again (see docs/operations/backup-classes.md)", + "psmdb.percona.com/PerconaServerMongoDB %s/%s storage %q is still the entry injected for the system bucket (credentialsSecret %q) while backup.useSystemBucket=false (%s after the job started); delete that storage entry from the PerconaServerMongoDB by hand or set backup.useSystemBucket=true again (see https://cozystack.io/docs/v1/operations/services/backup-classes/)", j.Namespace, psmdbName, storageName, cred, psmdbDefaultBackupDeadline)) } return r.requeueMongoDBBackupWaiting(ctx, j, "PerconaServerMongoDBStorageStale", @@ -1634,7 +1634,7 @@ func (r *RestoreJobReconciler) reconcileMongoDBRestore(ctx context.Context, rest // Point-in-time recovery needs an oplog stream, which the useSystemBucket flow // never captures — the chart leaves spec.backup.pitr unset there (see - // backup-classes.md). Refuse a recoveryTime restore of a system-bucket backup + // https://cozystack.io/docs/v1/operations/services/backup-classes/). Refuse a recoveryTime restore of a system-bucket backup // with a named error rather than hand the operator a PITR target it cannot // serve. Identify the flow from the flag recorded on the snapshot at backup // time, not from the resolved credentialsSecret: a BackupClass may name any @@ -1652,7 +1652,7 @@ func (r *RestoreJobReconciler) reconcileMongoDBRestore(ctx context.Context, rest } if snap != nil && snap.UseSystemBucket { return r.markRestoreJobFailed(ctx, restoreJob, - "restoreJob.spec.options.recoveryTime is set, but point-in-time recovery is not available for a backup taken on the useSystemBucket flow (no oplog is captured there); omit recoveryTime to restore the backup as taken. See docs/operations/backup-classes.md.") + "restoreJob.spec.options.recoveryTime is set, but point-in-time recovery is not available for a backup taken on the useSystemBucket flow (no oplog is captured there); omit recoveryTime to restore the backup as taken. See https://cozystack.io/docs/v1/operations/services/backup-classes/.") } } diff --git a/packages/apps/clickhouse/README.md b/packages/apps/clickhouse/README.md index 02d2dbb9e..93f5594c3 100644 --- a/packages/apps/clickhouse/README.md +++ b/packages/apps/clickhouse/README.md @@ -102,7 +102,7 @@ chart-emitted CronJob renders only when `backup.schedule` is non-empty. [altinity-clickhouse-backup]: https://github.com/Altinity/clickhouse-backup -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## Parameters diff --git a/packages/apps/foundationdb/README.md b/packages/apps/foundationdb/README.md index c253bc67e..85480003e 100644 --- a/packages/apps/foundationdb/README.md +++ b/packages/apps/foundationdb/README.md @@ -149,7 +149,7 @@ For issues related to FoundationDB itself, refer to the [FoundationDB documentat For Cozystack-specific issues, consult the Cozystack documentation or support channels. -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## Parameters diff --git a/packages/apps/harbor/README.md b/packages/apps/harbor/README.md index 7a036d7a0..10d6e5a04 100644 --- a/packages/apps/harbor/README.md +++ b/packages/apps/harbor/README.md @@ -14,7 +14,7 @@ seaweedfs: true Without object storage in the tenant chain, Harbor cannot provision its registry bucket: the `-registry` `BucketClaim`/`BucketAccess` never produces the `-registry-bucket` credentials secret, so the Harbor `HelmRelease` stays unreconciled, waiting on `BucketInfo`. -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## Parameters diff --git a/packages/apps/kafka/README.md b/packages/apps/kafka/README.md index 3691eabca..651e6e8d2 100644 --- a/packages/apps/kafka/README.md +++ b/packages/apps/kafka/README.md @@ -56,7 +56,7 @@ This setting is ignored if the corresponding `resources` value is set. Presets follow a cloud-style `.` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases and keep their original sizes, which do not follow one series: `nano` through `small` equal the `t1` sizes of the same name, while `medium` equals `c1.small` rather than `c1.medium`. -See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping. +See [Resource management](https://cozystack.io/docs/v1/guides/resource-management/) for the full size matrix and the legacy-to-instance-type mapping. ### topics diff --git a/packages/apps/kubernetes-nodes/templates/_helpers.tpl b/packages/apps/kubernetes-nodes/templates/_helpers.tpl index 56286b278..08a74b7ce 100644 --- a/packages/apps/kubernetes-nodes/templates/_helpers.tpl +++ b/packages/apps/kubernetes-nodes/templates/_helpers.tpl @@ -231,7 +231,7 @@ it reconciles. name several (.Values.cluster, pool) splits reconstruct the SAME object name kubernetes--, so an operator who edits spec.cluster to another such value (its immutability is enforced only by the dashboard, - not the aggregated apiserver, see docs/storage-immutability.md) does not + not the aggregated apiserver, see https://cozystack.io/docs/v1/operations/storage-class-immutability/) does not prune or delete a single worker VM, but silently flips spec.clusterName and the pool WorkloadMonitor selector. CAPI rejects the immutable MachineDeployment.spec.clusterName Update loudly; the WorkloadMonitor diff --git a/packages/apps/kubernetes/README.md b/packages/apps/kubernetes/README.md index d2534a8fa..9187ded09 100644 --- a/packages/apps/kubernetes/README.md +++ b/packages/apps/kubernetes/README.md @@ -99,7 +99,7 @@ See the reference for components utilized in this service: - **`controlPlane.apiServer.extraArgs` refuses flags the tenant control plane owns** (Kamaji 26.9.4-edge): the tenant control plane now sets a list of kube-apiserver flags itself and drops an `extraArgs` entry naming one, where the previous build let the entry win. The entry stops taking effect once the new build reconciles the control plane, whatever the chart renders. Two of those flags have a field the control plane renders them from, and the chart moves an entry written as `--flag=value` into it while the field is empty: `--enable-admission-plugins` into `controlPlane.apiServer.admissionControllers`, `--kubelet-preferred-address-types` into `controlPlane.kubelet.preferredAddressTypes`. The KamajiControlPlane CRD accepts only the plugin names it lists, so a plugin outside that list that the apiserver leaves off by default can no longer be enabled, and a moved name outside it fails when the KamajiControlPlane is applied, with an error that names it, rather than at render. The chart refuses to render any other entry naming one of the flags, so a tenant carrying one stops reconciling until the entry is removed; the description of `controlPlane.apiServer.extraArgs` lists the flags. The same Kamaji update passes every `extraArgs` entry to the apiserver, so repeats of a flag no longer collapse to the last one: naming the kube feature gates both bare and `kube:`-prefixed across entries, or giving two `--emulated-version` entries that name kube, now stops the apiserver from starting. The chart refuses neither, in any `spec.oidc.mode`: both reach the apiserver as written. Keep one spelling for kube gate terms and one kube emulated version. -> The top-level `storageClass` field is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. The per-pool worker `storageClass` now lives on the `KubernetesNodes` resource (see the `kubernetes-nodes` chart). +> The top-level `storageClass` field is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. The per-pool worker `storageClass` now lives on the `KubernetesNodes` resource (see the `kubernetes-nodes` chart). ## Proxmox substrate: credentials, privileges and tenant scope diff --git a/packages/apps/mariadb/README.md b/packages/apps/mariadb/README.md index 65eaf7a56..9f07540c0 100644 --- a/packages/apps/mariadb/README.md +++ b/packages/apps/mariadb/README.md @@ -10,7 +10,7 @@ This managed service is controlled by mariadb-operator, ensuring efficient manag - Docs: https://mariadb.com/kb/en/documentation/ - GitHub: https://github.com/mariadb-operator/mariadb-operator -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## HowTos @@ -246,7 +246,7 @@ This setting is ignored if the corresponding `resources` value is set. Presets follow a cloud-style `.` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases and keep their original sizes, which do not follow one series: `nano` through `small` equal the `t1` sizes of the same name, while `medium` equals `c1.small` rather than `c1.medium`. -See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping. +See [Resource management](https://cozystack.io/docs/v1/guides/resource-management/) for the full size matrix and the legacy-to-instance-type mapping. ### users diff --git a/packages/apps/mongodb/README.md b/packages/apps/mongodb/README.md index a6f236315..39ff631ae 100644 --- a/packages/apps/mongodb/README.md +++ b/packages/apps/mongodb/README.md @@ -193,7 +193,7 @@ Scaling an external replica set down can also produce transient reconfiguration Releases with `external: false`, and sharded releases in any configuration, are unaffected: neither renders `splitHorizons`, so no SAN changes and nothing restarts. -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## Parameters @@ -247,18 +247,18 @@ Releases with `external: false`, and sharded releases in any configuration, are ### Backup parameters -| Name | Description | Type | Value | -| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------------- | -| `backup` | Backup configuration. | `object` | `{}` | -| `backup.enabled` | Enable regular backups. | `bool` | `false` | -| `backup.useSystemBucket` | Opt-in: back up to the platform system bucket without supplying S3 credentials. The cozy-default MongoDB BackupClass injects the storage at BackupJob time; tenants do not set s3AccessKey/s3SecretKey/destinationPath/endpointURL, and backups scope to `/`. Scheduling moves to a BackupClass Plan; `schedule`, `retentionPolicy` and PITR are ignored while this is on, and this flow has no retention yet (see docs/operations/backup-classes.md). Requires `backup.enabled: true` (the pbm agents run on that flag); the chart refuses the combination with `enabled: false` at render time; the schema carries the same rule as a marker for the UI and for the day the apiserver enforces CEL on application schemas (cozystack/cozystack#2657), which it does not yet. A release that has not yet rendered this flag (its cluster still carries the chart's scheduled tasks and PITR) holds a BackupJob until it does, so the tenant's own storage is never injected over. | `bool` | `false` | -| `backup.schedule` | Cron schedule for automated backups. Ignored when useSystemBucket is true (scheduling moves to the BackupClass Plan). | `string` | `0 2 * * *` | -| `backup.retentionPolicy` | Retention policy (e.g. "30d"). Ignored when useSystemBucket is true (no engine-side retention on that flow). | `string` | `30d` | -| `backup.destinationPath` | Destination path for backups (e.g. s3://bucket/path/). Ignored when useSystemBucket is true (the driver injects the platform bucket). | `string` | `s3://bucket/path/to/folder/` | -| `backup.endpointURL` | S3 endpoint URL for uploads. Ignored when useSystemBucket is true (the strategy supplies the platform endpoint). | `string` | `http://minio-gateway-service:9000` | -| `backup.insecureSkipTLSVerify` | Skip TLS verification for the S3 endpoint (self-signed, e.g. in-cluster seaweedfs). Ignored when useSystemBucket is true (the platform backupStorage knob applies instead). | `bool` | `false` | -| `backup.s3AccessKey` | Access key for S3 authentication. Ignored when useSystemBucket is true (credentials come from the projected cozy-backups-creds). | `string` | `""` | -| `backup.s3SecretKey` | Secret key for S3 authentication. Ignored when useSystemBucket is true (credentials come from the projected cozy-backups-creds). | `string` | `""` | +| Name | Description | Type | Value | +| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----------------------------------- | +| `backup` | Backup configuration. | `object` | `{}` | +| `backup.enabled` | Enable regular backups. | `bool` | `false` | +| `backup.useSystemBucket` | Opt-in: back up to the platform system bucket without supplying S3 credentials. The cozy-default MongoDB BackupClass injects the storage at BackupJob time; tenants do not set s3AccessKey/s3SecretKey/destinationPath/endpointURL, and backups scope to `/`. Scheduling moves to a BackupClass Plan; `schedule`, `retentionPolicy` and PITR are ignored while this is on, and this flow has no retention yet (see https://cozystack.io/docs/v1/operations/services/backup-classes/). Requires `backup.enabled: true` (the pbm agents run on that flag); the chart refuses the combination with `enabled: false` at render time; the schema carries the same rule as a marker for the UI and for the day the apiserver enforces CEL on application schemas (cozystack/cozystack#2657), which it does not yet. A release that has not yet rendered this flag (its cluster still carries the chart's scheduled tasks and PITR) holds a BackupJob until it does, so the tenant's own storage is never injected over. | `bool` | `false` | +| `backup.schedule` | Cron schedule for automated backups. Ignored when useSystemBucket is true (scheduling moves to the BackupClass Plan). | `string` | `0 2 * * *` | +| `backup.retentionPolicy` | Retention policy (e.g. "30d"). Ignored when useSystemBucket is true (no engine-side retention on that flow). | `string` | `30d` | +| `backup.destinationPath` | Destination path for backups (e.g. s3://bucket/path/). Ignored when useSystemBucket is true (the driver injects the platform bucket). | `string` | `s3://bucket/path/to/folder/` | +| `backup.endpointURL` | S3 endpoint URL for uploads. Ignored when useSystemBucket is true (the strategy supplies the platform endpoint). | `string` | `http://minio-gateway-service:9000` | +| `backup.insecureSkipTLSVerify` | Skip TLS verification for the S3 endpoint (self-signed, e.g. in-cluster seaweedfs). Ignored when useSystemBucket is true (the platform backupStorage knob applies instead). | `bool` | `false` | +| `backup.s3AccessKey` | Access key for S3 authentication. Ignored when useSystemBucket is true (credentials come from the projected cozy-backups-creds). | `string` | `""` | +| `backup.s3SecretKey` | Secret key for S3 authentication. Ignored when useSystemBucket is true (credentials come from the projected cozy-backups-creds). | `string` | `""` | ### Bootstrap (recovery) parameters diff --git a/packages/apps/mongodb/templates/mongodb.yaml b/packages/apps/mongodb/templates/mongodb.yaml index c12dbca07..26e23054e 100644 --- a/packages/apps/mongodb/templates/mongodb.yaml +++ b/packages/apps/mongodb/templates/mongodb.yaml @@ -1,6 +1,6 @@ {{- $clusterDomain := (index .Values._cluster "cluster-domain") | default "cozy.local" }} {{- if and .Values.backup.useSystemBucket (not .Values.backup.enabled) }} -{{- fail "mongodb: backup.useSystemBucket=true requires backup.enabled=true: the percona-backup-mongodb agents only run when backups are enabled, so this combination renders a cluster that can never service a BackupJob. Set backup.enabled=true (or drop useSystemBucket). See docs/operations/backup-classes.md." }} +{{- fail "mongodb: backup.useSystemBucket=true requires backup.enabled=true: the percona-backup-mongodb agents only run when backups are enabled, so this combination renders a cluster that can never service a BackupJob. Set backup.enabled=true (or drop useSystemBucket). See https://cozystack.io/docs/v1/operations/services/backup-classes/." }} {{- end }} {{- /* TLS IMPLEMENTATION NOTE — operator-managed cert-manager chain diff --git a/packages/apps/mongodb/templates/restore.yaml b/packages/apps/mongodb/templates/restore.yaml index 2742cc81f..067739e9c 100644 --- a/packages/apps/mongodb/templates/restore.yaml +++ b/packages/apps/mongodb/templates/restore.yaml @@ -1,6 +1,6 @@ {{- if .Values.bootstrap.enabled }} {{- if .Values.backup.useSystemBucket }} -{{- fail "mongodb: bootstrap.enabled=true is incompatible with backup.useSystemBucket=true. bootstrap.enabled is not reset after a bootstrap restore completes, so set bootstrap.enabled=false before opting in; to restore from the system bucket use a backups.cozystack.io/RestoreJob against the cozy-default BackupClass instead. See docs/operations/backup-classes.md." }} +{{- fail "mongodb: bootstrap.enabled=true is incompatible with backup.useSystemBucket=true. bootstrap.enabled is not reset after a bootstrap restore completes, so set bootstrap.enabled=false before opting in; to restore from the system bucket use a backups.cozystack.io/RestoreJob against the cozy-default BackupClass instead. See https://cozystack.io/docs/v1/operations/services/backup-classes/." }} {{- end }} {{- if not .Values.bootstrap.backupName }} {{- fail "bootstrap.backupName is required when bootstrap.enabled is true" }} diff --git a/packages/apps/mongodb/tests/mongodb_test.yaml b/packages/apps/mongodb/tests/mongodb_test.yaml index d6f4f039d..6b48a0a5c 100644 --- a/packages/apps/mongodb/tests/mongodb_test.yaml +++ b/packages/apps/mongodb/tests/mongodb_test.yaml @@ -941,4 +941,4 @@ tests: useSystemBucket: true asserts: - failedTemplate: - errorMessage: "mongodb: backup.useSystemBucket=true requires backup.enabled=true: the percona-backup-mongodb agents only run when backups are enabled, so this combination renders a cluster that can never service a BackupJob. Set backup.enabled=true (or drop useSystemBucket). See docs/operations/backup-classes.md." + errorMessage: "mongodb: backup.useSystemBucket=true requires backup.enabled=true: the percona-backup-mongodb agents only run when backups are enabled, so this combination renders a cluster that can never service a BackupJob. Set backup.enabled=true (or drop useSystemBucket). See https://cozystack.io/docs/v1/operations/services/backup-classes/." diff --git a/packages/apps/mongodb/tests/restore_test.yaml b/packages/apps/mongodb/tests/restore_test.yaml index 8337873b5..91ccc53d4 100644 --- a/packages/apps/mongodb/tests/restore_test.yaml +++ b/packages/apps/mongodb/tests/restore_test.yaml @@ -71,7 +71,7 @@ tests: useSystemBucket: true asserts: - failedTemplate: - errorMessage: "mongodb: bootstrap.enabled=true is incompatible with backup.useSystemBucket=true. bootstrap.enabled is not reset after a bootstrap restore completes, so set bootstrap.enabled=false before opting in; to restore from the system bucket use a backups.cozystack.io/RestoreJob against the cozy-default BackupClass instead. See docs/operations/backup-classes.md." + errorMessage: "mongodb: bootstrap.enabled=true is incompatible with backup.useSystemBucket=true. bootstrap.enabled is not reset after a bootstrap restore completes, so set bootstrap.enabled=false before opting in; to restore from the system bucket use a backups.cozystack.io/RestoreJob against the cozy-default BackupClass instead. See https://cozystack.io/docs/v1/operations/services/backup-classes/." - it: fails when destinationPath is missing release: diff --git a/packages/apps/mongodb/values.schema.json b/packages/apps/mongodb/values.schema.json index 880ffb619..e0416e97b 100644 --- a/packages/apps/mongodb/values.schema.json +++ b/packages/apps/mongodb/values.schema.json @@ -313,7 +313,7 @@ "default": "0 2 * * *" }, "useSystemBucket": { - "description": "Opt-in: back up to the platform system bucket without supplying S3 credentials. The cozy-default MongoDB BackupClass injects the storage at BackupJob time; tenants do not set s3AccessKey/s3SecretKey/destinationPath/endpointURL, and backups scope to `\u003cnamespace\u003e/\u003capplication\u003e`. Scheduling moves to a BackupClass Plan; `schedule`, `retentionPolicy` and PITR are ignored while this is on, and this flow has no retention yet (see docs/operations/backup-classes.md). Requires `backup.enabled: true` (the pbm agents run on that flag); the chart refuses the combination with `enabled: false` at render time; the schema carries the same rule as a marker for the UI and for the day the apiserver enforces CEL on application schemas (cozystack/cozystack#2657), which it does not yet. A release that has not yet rendered this flag (its cluster still carries the chart's scheduled tasks and PITR) holds a BackupJob until it does, so the tenant's own storage is never injected over.", + "description": "Opt-in: back up to the platform system bucket without supplying S3 credentials. The cozy-default MongoDB BackupClass injects the storage at BackupJob time; tenants do not set s3AccessKey/s3SecretKey/destinationPath/endpointURL, and backups scope to `\u003cnamespace\u003e/\u003capplication\u003e`. Scheduling moves to a BackupClass Plan; `schedule`, `retentionPolicy` and PITR are ignored while this is on, and this flow has no retention yet (see https://cozystack.io/docs/v1/operations/services/backup-classes/). Requires `backup.enabled: true` (the pbm agents run on that flag); the chart refuses the combination with `enabled: false` at render time; the schema carries the same rule as a marker for the UI and for the day the apiserver enforces CEL on application schemas (cozystack/cozystack#2657), which it does not yet. A release that has not yet rendered this flag (its cluster still carries the chart's scheduled tasks and PITR) holds a BackupJob until it does, so the tenant's own storage is never injected over.", "type": "boolean", "default": false } diff --git a/packages/apps/mongodb/values.yaml b/packages/apps/mongodb/values.yaml index 3ad91f2b4..74a6f9e9b 100644 --- a/packages/apps/mongodb/values.yaml +++ b/packages/apps/mongodb/values.yaml @@ -155,7 +155,7 @@ databases: {} ## @typedef {struct} Backup - Backup configuration. ## @field {bool} enabled - Enable regular backups. -## @field {bool} [useSystemBucket] - Opt-in: back up to the platform system bucket without supplying S3 credentials. The cozy-default MongoDB BackupClass injects the storage at BackupJob time; tenants do not set s3AccessKey/s3SecretKey/destinationPath/endpointURL, and backups scope to `/`. Scheduling moves to a BackupClass Plan; `schedule`, `retentionPolicy` and PITR are ignored while this is on, and this flow has no retention yet (see docs/operations/backup-classes.md). Requires `backup.enabled: true` (the pbm agents run on that flag); the chart refuses the combination with `enabled: false` at render time; the schema carries the same rule as a marker for the UI and for the day the apiserver enforces CEL on application schemas (cozystack/cozystack#2657), which it does not yet. A release that has not yet rendered this flag (its cluster still carries the chart's scheduled tasks and PITR) holds a BackupJob until it does, so the tenant's own storage is never injected over. +## @field {bool} [useSystemBucket] - Opt-in: back up to the platform system bucket without supplying S3 credentials. The cozy-default MongoDB BackupClass injects the storage at BackupJob time; tenants do not set s3AccessKey/s3SecretKey/destinationPath/endpointURL, and backups scope to `/`. Scheduling moves to a BackupClass Plan; `schedule`, `retentionPolicy` and PITR are ignored while this is on, and this flow has no retention yet (see https://cozystack.io/docs/v1/operations/services/backup-classes/). Requires `backup.enabled: true` (the pbm agents run on that flag); the chart refuses the combination with `enabled: false` at render time; the schema carries the same rule as a marker for the UI and for the day the apiserver enforces CEL on application schemas (cozystack/cozystack#2657), which it does not yet. A release that has not yet rendered this flag (its cluster still carries the chart's scheduled tasks and PITR) holds a BackupJob until it does, so the tenant's own storage is never injected over. ## @field {string} [schedule] - Cron schedule for automated backups. Ignored when useSystemBucket is true (scheduling moves to the BackupClass Plan). ## @field {string} [retentionPolicy] - Retention policy (e.g. "30d"). Ignored when useSystemBucket is true (no engine-side retention on that flow). ## @field {string} [destinationPath] - Destination path for backups (e.g. s3://bucket/path/). Ignored when useSystemBucket is true (the driver injects the platform bucket). diff --git a/packages/apps/nats/README.md b/packages/apps/nats/README.md index f79a9b0fb..73cf0aacb 100644 --- a/packages/apps/nats/README.md +++ b/packages/apps/nats/README.md @@ -3,7 +3,7 @@ NATS is an open-source, simple, secure, and high performance messaging system. It provides a data layer for cloud native applications, IoT messaging, and microservices architectures. -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## Parameters @@ -54,7 +54,7 @@ This setting is ignored if the corresponding `resources` value is set. Presets follow a cloud-style `.` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases and keep their original sizes, which do not follow one series: `nano` through `small` equal the `t1` sizes of the same name, while `medium` equals `c1.small` rather than `c1.medium`. -See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping. +See [Resource management](https://cozystack.io/docs/v1/guides/resource-management/) for the full size matrix and the legacy-to-instance-type mapping. ### tls diff --git a/packages/apps/openbao/README.md b/packages/apps/openbao/README.md index 2ba51bddf..ace9d6ebe 100644 --- a/packages/apps/openbao/README.md +++ b/packages/apps/openbao/README.md @@ -3,7 +3,7 @@ OpenBAO is an open-source secrets management solution forked from HashiCorp Vault. It provides identity-based secrets and encryption management for cloud infrastructure. -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## Auto-unseal diff --git a/packages/apps/opensearch/README.md b/packages/apps/opensearch/README.md index 74fe3caa6..94830d569 100644 --- a/packages/apps/opensearch/README.md +++ b/packages/apps/opensearch/README.md @@ -1,6 +1,6 @@ # Managed OpenSearch Service -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## TLS and client verification diff --git a/packages/apps/postgres/README.md b/packages/apps/postgres/README.md index bdc13ea9c..50c300aff 100644 --- a/packages/apps/postgres/README.md +++ b/packages/apps/postgres/README.md @@ -160,7 +160,7 @@ See: - -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ### TLS for server connections @@ -341,7 +341,7 @@ This setting is ignored if the corresponding `resources` value is set. Presets follow a cloud-style `.` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases and keep their original sizes, which do not follow one series: `nano` through `small` equal the `t1` sizes of the same name, while `medium` equals `c1.small` rather than `c1.medium`. -See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping. +See [Resource management](https://cozystack.io/docs/v1/guides/resource-management/) for the full size matrix and the legacy-to-instance-type mapping. ### users diff --git a/packages/apps/postgres/templates/db.yaml b/packages/apps/postgres/templates/db.yaml index c087037b9..8538b326b 100644 --- a/packages/apps/postgres/templates/db.yaml +++ b/packages/apps/postgres/templates/db.yaml @@ -211,13 +211,13 @@ spec: backups.cozystack.io/RestoreJob targeting a fresh Postgres app — the backup-controller's CNPG driver SSA-patches the Cluster's spec.bootstrap before CNPG sees it. See the BackupClass and Restore - flow in docs/operations/backup-classes.md. + flow in https://cozystack.io/docs/v1/operations/services/backup-classes/. Fail fast at chart-render time so tenants see the misconfiguration immediately rather than during a Postgres bootstrap pod crash loop. */}} {{- if and .Values.bootstrap.enabled .Values.backup.useSystemBucket }} - {{- fail "postgres: bootstrap.enabled=true is incompatible with backup.useSystemBucket=true. Use a backups.cozystack.io/RestoreJob against the cozy-default BackupClass to restore from the system bucket; see docs/operations/backup-classes.md." }} + {{- fail "postgres: bootstrap.enabled=true is incompatible with backup.useSystemBucket=true. Use a backups.cozystack.io/RestoreJob against the cozy-default BackupClass to restore from the system bucket; see https://cozystack.io/docs/v1/operations/services/backup-classes/." }} {{- end }} {{/* The externalClusters bootstrap block below references $credSecretName diff --git a/packages/apps/postgres/tests/backup_storage_test.yaml b/packages/apps/postgres/tests/backup_storage_test.yaml index 4aba2222b..1a4312133 100644 --- a/packages/apps/postgres/tests/backup_storage_test.yaml +++ b/packages/apps/postgres/tests/backup_storage_test.yaml @@ -308,7 +308,7 @@ tests: oldName: old asserts: - failedTemplate: - errorMessage: "postgres: bootstrap.enabled=true is incompatible with backup.useSystemBucket=true. Use a backups.cozystack.io/RestoreJob against the cozy-default BackupClass to restore from the system bucket; see docs/operations/backup-classes.md." + errorMessage: "postgres: bootstrap.enabled=true is incompatible with backup.useSystemBucket=true. Use a backups.cozystack.io/RestoreJob against the cozy-default BackupClass to restore from the system bucket; see https://cozystack.io/docs/v1/operations/services/backup-classes/." - it: "bootstrap without credentials: render fails fast" template: templates/db.yaml diff --git a/packages/apps/qdrant/README.md b/packages/apps/qdrant/README.md index 84fd84a4a..51b01bcf8 100644 --- a/packages/apps/qdrant/README.md +++ b/packages/apps/qdrant/README.md @@ -9,7 +9,7 @@ Service deploys Qdrant as a StatefulSet with automatic cluster mode when multipl - Docs: https://qdrant.tech/documentation/ - GitHub: https://github.com/qdrant/qdrant -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## Parameters @@ -53,7 +53,7 @@ This setting is ignored if the corresponding `resources` value is set. Presets follow a cloud-style `.` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases and keep their original sizes, which do not follow one series: `nano` through `small` equal the `t1` sizes of the same name, while `medium` equals `c1.small` rather than `c1.medium`. -See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping. +See [Resource management](https://cozystack.io/docs/v1/guides/resource-management/) for the full size matrix and the legacy-to-instance-type mapping. ### tls diff --git a/packages/apps/rabbitmq/README.md b/packages/apps/rabbitmq/README.md index 9d0cbb0ad..130e9af0a 100644 --- a/packages/apps/rabbitmq/README.md +++ b/packages/apps/rabbitmq/README.md @@ -9,11 +9,11 @@ The service utilizes official RabbitMQ operator. This ensures the reliability an - Github: https://github.com/rabbitmq/cluster-operator/ - Docs: https://www.rabbitmq.com/kubernetes/operator/operator-overview.html -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## Backups -RabbitMQ is bound out-of-the-box by the platform `cozy-default` BackupClass, so a `Plan` or an ad-hoc `BackupJob` referencing `cozy-default` backs the instance up with no per-release wiring — see [Backup Classes](../../../docs/operations/backup-classes.md). +RabbitMQ is bound out-of-the-box by the platform `cozy-default` BackupClass, so a `Plan` or an ad-hoc `BackupJob` referencing `cozy-default` backs the instance up with no per-release wiring — see [Backup Classes](https://cozystack.io/docs/v1/operations/services/backup-classes/). The scope is **definitions only** — vhosts, users, permissions, queues, exchanges, bindings, policies and parameters, exported through the management API. **Message payloads are not included**, and there is no point-in-time recovery; back up the messages on disk with a Velero volume snapshot instead if you need them. Restore is a **merge**, not a reset: importing definitions creates or updates what the backup contains and never deletes, so anything created after the backup survives a restore, and a to-copy restore into a separate instance imports the source's users (with their password hashes) into the target. @@ -141,5 +141,5 @@ This setting is ignored if the corresponding `resources` value is set. Presets follow a cloud-style `.` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases and keep their original sizes, which do not follow one series: `nano` through `small` equal the `t1` sizes of the same name, while `medium` equals `c1.small` rather than `c1.medium`. -See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping. +See [Resource management](https://cozystack.io/docs/v1/guides/resource-management/) for the full size matrix and the legacy-to-instance-type mapping. diff --git a/packages/apps/redis/README.md b/packages/apps/redis/README.md index 0922bf821..b1ba98d7c 100644 --- a/packages/apps/redis/README.md +++ b/packages/apps/redis/README.md @@ -9,7 +9,7 @@ Service utilizes the freshworks-oss Redis Operator (a maintained fork of the arc - Docs: https://redis.io/docs/ - GitHub: https://github.com/freshworks-oss/redis-operator -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## Parameters @@ -75,7 +75,7 @@ This setting is ignored if the corresponding `resources` value is set. Presets follow a cloud-style `.` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases and keep their original sizes, which do not follow one series: `nano` through `small` equal the `t1` sizes of the same name, while `medium` equals `c1.small` rather than `c1.medium`. -See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping. +See [Resource management](https://cozystack.io/docs/v1/guides/resource-management/) for the full size matrix and the legacy-to-instance-type mapping. ### sentinel diff --git a/packages/apps/tcp-balancer/README.md b/packages/apps/tcp-balancer/README.md index f7b03483d..c8b240c13 100644 --- a/packages/apps/tcp-balancer/README.md +++ b/packages/apps/tcp-balancer/README.md @@ -54,4 +54,4 @@ This setting is ignored if the corresponding `resources` value is set. Presets follow a cloud-style `.` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases and keep their original sizes, which do not follow one series: `nano` through `small` equal the `t1` sizes of the same name, while `medium` equals `c1.small` rather than `c1.medium`. -See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping. +See [Resource management](https://cozystack.io/docs/v1/guides/resource-management/) for the full size matrix and the legacy-to-instance-type mapping. diff --git a/packages/apps/valkey/README.md b/packages/apps/valkey/README.md index 56a55354f..ac9b4b101 100644 --- a/packages/apps/valkey/README.md +++ b/packages/apps/valkey/README.md @@ -10,7 +10,7 @@ Service runs Valkey on the freshworks-oss redis-operator, which manages Sentinel - Valkey: https://github.com/valkey-io/valkey - Operator: https://github.com/freshworks-oss/redis-operator -> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. +> `storageClass` is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. ## Parameters @@ -54,4 +54,4 @@ This setting is ignored if the corresponding `resources` value is set. Presets follow a cloud-style `.` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases and keep their original sizes, which do not follow one series: `nano` through `small` equal the `t1` sizes of the same name, while `medium` equals `c1.small` rather than `c1.medium`. -See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping. +See [Resource management](https://cozystack.io/docs/v1/guides/resource-management/) for the full size matrix and the legacy-to-instance-type mapping. diff --git a/packages/apps/vm-disk/README.md b/packages/apps/vm-disk/README.md index 4700a8ee9..a736be0db 100644 --- a/packages/apps/vm-disk/README.md +++ b/packages/apps/vm-disk/README.md @@ -2,7 +2,7 @@ A Virtual Machine Disk -> `storageClass` and `source` are annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. `DataVolume.spec` is immutable in CDI, so the chart also fails the release on an attempted `storageClass`/`source` edit of an existing disk rather than silently reusing the old spec; delete and recreate the disk to change either field. +> `storageClass` and `source` are annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. `DataVolume.spec` is immutable in CDI, so the chart also fails the release on an attempted `storageClass`/`source` edit of an existing disk rather than silently reusing the old spec; delete and recreate the disk to change either field. > Whether CDI has finished populating the disk shows up in the `WorkloadsReady` condition, not in `Ready`, which follows the Helm release and does not wait on the DataVolume: `WorkloadsReady` stays `False` while an import, clone or upload is in flight and when the DataVolume fails. Its message names the DataVolume and the phase it is in, followed by the error CDI reports for the pod populating the disk when there is one, such as a 404 or an unresolvable host on an HTTP source: CDI keeps retrying such an import, so its phase stays `ImportInProgress`, and the error is shown between retries rather than while a retry is running. `kubectl get vmdisks` shows the condition in the `WORKLOADS` column, and the console badge reads `DataVolumeNotReady` until it is `True`. A disk created before this behaviour was introduced keeps its old release settings, under which `Ready` may wait on the DataVolume, until its application is next updated. A `source.upload` disk reads `False` from the moment it is created until `virtctl image-upload` completes, because CDI reports waiting for the upload and receiving it as one phase. A disk counts as ready while it waits for its consumer with `bindImmediately` off on a `WaitForFirstConsumer` class, because that is a state such a disk is supposed to rest in. diff --git a/packages/apps/vm-disk/templates/dv.yaml b/packages/apps/vm-disk/templates/dv.yaml index 09831fc79..f61045327 100644 --- a/packages/apps/vm-disk/templates/dv.yaml +++ b/packages/apps/vm-disk/templates/dv.yaml @@ -164,7 +164,7 @@ existing disk is a silent no-op that diverges the CR from the cluster (#2985); fail instead. Both are compared against the operator input recorded in an annotation at creation (legacy disks fall back to stored spec.source). Full - contract + caveats: docs/storage-immutability.md. Size growth: pvc-resize-hook. + contract + caveats: https://cozystack.io/docs/v1/operations/storage-class-immutability/. Size growth: pvc-resize-hook. */}} {{- if $existingDV }} {{- $existingAnnotations := $existingDV.metadata.annotations | default dict }} @@ -184,7 +184,7 @@ {{- /* Legacy disk (no annotation): compare only chart-producible stored shapes; anything else (sourceRef, non-map, foreign source) is left alone, never crashed or deadlocked. Vars are declared before the map - guards so they stay in scope below. See docs/storage-immutability.md. */}} + guards so they stay in scope below. See https://cozystack.io/docs/v1/operations/storage-class-immutability/. */}} {{- $storedType := "" }} {{- $chartShaped := false }} {{- $storedIsImageClone := false }} diff --git a/packages/apps/vm-disk/tests/dv_immutability_test.yaml b/packages/apps/vm-disk/tests/dv_immutability_test.yaml index 13808ae99..1085360d1 100644 --- a/packages/apps/vm-disk/tests/dv_immutability_test.yaml +++ b/packages/apps/vm-disk/tests/dv_immutability_test.yaml @@ -358,7 +358,7 @@ tests: # rendered spec, so image is exempt from the annotation-less fallback. The # edit is silently reused rather than failed. Not a regression (all edits # were previously silent); all disks created by this chart version onward - # carry the annotation and are fully guarded. See docs/storage-immutability.md. + # carry the annotation and are fully guarded. See https://cozystack.io/docs/v1/operations/storage-class-immutability/. set: storageClass: replicated source: {image: {name: debian}} diff --git a/packages/apps/vpn/README.md b/packages/apps/vpn/README.md index f8032f621..53d48a774 100644 --- a/packages/apps/vpn/README.md +++ b/packages/apps/vpn/README.md @@ -57,7 +57,7 @@ This setting is ignored if the corresponding `resources` value is set. Presets follow a cloud-style `.` naming convention. Five series cover the full CPU-to-memory ratio range (`t1` 1:0.5, `c1` 1:1, `s1` 1:2, `u1` 1:4, `m1` 1:8) and each series ships eight sizes (`nano` through `4xlarge`). The legacy flat names (`nano`, `micro`, `small`, `medium`, `large`, `xlarge`, `2xlarge`) remain accepted as deprecated aliases and keep their original sizes, which do not follow one series: `nano` through `small` equal the `t1` sizes of the same name, while `medium` equals `c1.small` rather than `c1.medium`. -See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-presets.md) for the full size matrix and the legacy-to-instance-type mapping. +See [Resource management](https://cozystack.io/docs/v1/guides/resource-management/) for the full size matrix and the legacy-to-instance-type mapping. ### users diff --git a/packages/core/installer/values.yaml b/packages/core/installer/values.yaml index b080fd433..43639ab5c 100644 --- a/packages/core/installer/values.yaml +++ b/packages/core/installer/values.yaml @@ -48,7 +48,7 @@ cozystackOperator: # Lowering or an oversized request creates a durable hold. A cluster administrator # must acknowledge its exact target before defaulting resumes; disabling retains holds. # Explicit container limits and foreign namespace memory policies are preserved. - # See docs/operations/system-memory-limits.md for sizing, exclusions and acknowledgement. + # See https://cozystack.io/docs/v1/operations/configuration/system-memory-limits/ for sizing, exclusions and acknowledgement. systemNamespaceMemoryLimit: "" # Default memory request paired with the limit. Empty selects 32Mi; 0 is supported. # Keep small to avoid reserving the full limit for every previously limitless container. diff --git a/packages/core/platform/templates/bundles/iaas.yaml b/packages/core/platform/templates/bundles/iaas.yaml index 9dca40f0a..9042d20e9 100644 --- a/packages/core/platform/templates/bundles/iaas.yaml +++ b/packages/core/platform/templates/bundles/iaas.yaml @@ -18,7 +18,7 @@ Feature gates the cluster declines, forwarded verbatim. This include is the only writer of the cozystack.kubevirt Package, and patching that Package by hand is reverted the next time the platform re-renders it (see - docs/operations/backup-classes.md for the same trap on backupStorage), so + https://cozystack.io/docs/v1/operations/services/backup-classes/ for the same trap on backupStorage), so without this hop the chart value has no supported setter. The parenthesized walk is nil-safe for the reason the gpu block below spells out. */ -}} diff --git a/packages/core/platform/values.yaml b/packages/core/platform/values.yaml index 122dac747..c4f8337be 100644 --- a/packages/core/platform/values.yaml +++ b/packages/core/platform/values.yaml @@ -173,7 +173,7 @@ networking: # # On a running cluster this is set on the `platform` component of the # `cozystack.cozystack-platform` Package CR, the same path as backupStorage - # (see docs/operations/backup-classes.md). Editing the multus HelmRelease or + # (see https://cozystack.io/docs/v1/operations/services/backup-classes/). Editing the multus HelmRelease or # the child Package cozystack.multus directly is reverted by the platform # reconcile. # stageCniPlugins: true @@ -712,7 +712,7 @@ monitoring: # documents every knob: provisionBucket, bucketName, endpoint, region, # forcePathStyle, systemSecretName, systemNamespaces, …). Admins set this via # spec.components.platform.values.backupStorage on the -# cozystack.cozystack-platform Package — see docs/operations/backup-classes.md. +# cozystack.cozystack-platform Package — see https://cozystack.io/docs/v1/operations/services/backup-classes/. # Empty (the default) forwards nothing and keeps the child chart defaults. backupStorage: {} diff --git a/packages/extra/gateway/README.md b/packages/extra/gateway/README.md index c47ebfb14..b72dd223b 100644 --- a/packages/extra/gateway/README.md +++ b/packages/extra/gateway/README.md @@ -16,7 +16,7 @@ Owning a separate Gateway makes sense for: a tenant that needs its own LB IP (DN The platform-wide `publishing.certificates.solver` value selects how the controller sources TLS certificates for the tenant Gateway. Setting `publishing.certificates.wildcardSecretName` switches to a third mode (existing Secret) that overrides the solver entirely, and putting the tenant's GatewayClass in `gateway.edgeTerminatedClasses` switches to a fourth that issues nothing at all — see below. -The sections below are the chart's own reference. For the operator's view of the same surface — the three platform values that bound which class a tenant lands on, what a tenant writes to pick one, what moving between classes does to certificates and to the wildcard replica, and how the mechanism fails when the configured names and the installed classes disagree — see [`docs/gateway-classes.md`](../../../docs/gateway-classes.md). +The sections below are the chart's own reference. For the operator's view of the same surface — the three platform values that bound which class a tenant lands on, what a tenant writes to pick one, what moving between classes does to certificates and to the wildcard replica, and how the mechanism fails when the configured names and the installed classes disagree — see [GatewayClass selection](https://cozystack.io/docs/v1/networking/gateway-classes/). ### Default — HTTP-01 diff --git a/packages/library/cozy-lib/templates/_resourcepresets.tpl b/packages/library/cozy-lib/templates/_resourcepresets.tpl index f26c6f9c2..c08a89a5e 100644 --- a/packages/library/cozy-lib/templates/_resourcepresets.tpl +++ b/packages/library/cozy-lib/templates/_resourcepresets.tpl @@ -19,7 +19,7 @@ are retained as backward-compatibility aliases with their original resource values. They are deprecated and will be removed in a future release; migrate to instance-type names. See migration table in - docs/operations/resource-presets.md. + https://cozystack.io/docs/v1/guides/resource-management/. */}} {{- define "cozy-lib.resources.unsanitizedPreset" }} diff --git a/packages/system/backupstrategy-controller/templates/backupclass-default.yaml b/packages/system/backupstrategy-controller/templates/backupclass-default.yaml index 750d564ab..3f3b271e3 100644 --- a/packages/system/backupstrategy-controller/templates/backupclass-default.yaml +++ b/packages/system/backupstrategy-controller/templates/backupclass-default.yaml @@ -91,7 +91,7 @@ spec: # foundationdb operator pod itself (cozy-foundationdb-operator). That # operator deployment does not yet mount cozy-backups-creds, so a # platform-default FDB restore would silently fail. See DESIGN.md / - # backup-classes.md for the open gap and follow-up work. + # https://cozystack.io/docs/v1/operations/services/backup-classes/ for the open gap and follow-up work. - application: apiGroup: apps.cozystack.io kind: VMInstance diff --git a/packages/system/backupstrategy-controller/templates/rbac.yaml b/packages/system/backupstrategy-controller/templates/rbac.yaml index ac9124392..8d90980a8 100644 --- a/packages/system/backupstrategy-controller/templates/rbac.yaml +++ b/packages/system/backupstrategy-controller/templates/rbac.yaml @@ -172,7 +172,7 @@ rules: # finalizer frees the object), and cancel a not-yet-started system-bucket CR at # the BackupJob timeout (a legacy CR is left in place). patch: strip the driver's own delete-backup finalizer # to release a wedged cleanup. See reconcileMongoDB / cleanupMongoDBBackup and - # docs/operations/backup-classes.md. + # https://cozystack.io/docs/v1/operations/services/backup-classes/. verbs: ["get", "list", "watch", "create", "delete", "patch"] - apiGroups: ["psmdb.percona.com"] resources: ["perconaservermongodbrestores"] diff --git a/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml index d5c88bdf5..c65462295 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml @@ -14,7 +14,7 @@ on the target they are unmanaged (no KafkaTopic CR) — and for a topic that IS owned by a KafkaTopic CR, the Strimzi Topic Operator reconciles its configs from the CR, so an in-place config restore of such a topic is transient. See - docs/operations/backup-classes.md and the example README. + https://cozystack.io/docs/v1/operations/services/backup-classes/ and the example README. */}} {{- $bucketName := include "backupstrategy-controller.bucketName" . -}} {{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "kafkas.strategy.backups.cozystack.io") -}} diff --git a/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml index 3bf8ff407..7509f4bf7 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml @@ -45,7 +45,7 @@ hang): the MongoDB application must have backup.enabled=true so the percona-backup-mongodb agents run. On the legacy flow the "s3-storage" entry must also already exist on the app; on the useSystemBucket flow the driver - injects it at BackupJob time. See docs/operations/backup-classes.md. + injects it at BackupJob time. See https://cozystack.io/docs/v1/operations/services/backup-classes/. */}} {{- $bucketName := include "backupstrategy-controller.bucketName" . -}} {{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "mongodbs.strategy.backups.cozystack.io") -}} @@ -66,7 +66,7 @@ spec: A legacy app leaves the flag false and is never injected, so it keeps its own static s3-storage. endpointUrl carries the scheme from the endpoint helper so PBM's aws-sdk client gets a valid URL; credentials come from the - projected cozy-backups-creds. See docs/operations/backup-classes.md. + projected cozy-backups-creds. See https://cozystack.io/docs/v1/operations/services/backup-classes/. */}} s3: bucket: {{ $bucketName | quote }} diff --git a/packages/system/backupstrategy-controller/templates/strategy-redis-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-redis-default.yaml index f2f9a54b0..3db27f7e2 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-redis-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-redis-default.yaml @@ -31,7 +31,7 @@ spec: keeps stacking behind; the job-in-progress path has no other deadline. int so a large values-file override (parsed as float64) renders as an integer, not exponent form (1e+06), which would fail schema validation. Full - rationale: docs/operations/backup-classes.md. + rationale: https://cozystack.io/docs/v1/operations/services/backup-classes/. */}} activeDeadlineSeconds: {{ int .Values.backupStrategyController.redisBackupActiveDeadlineSeconds }} volumes: @@ -58,7 +58,7 @@ spec: activeDeadlineSeconds (never a Failed Job) when the admin has not created it. A missing ca.crt does NOT skip verification: the script falls through to the image's system CA store and curl fails the handshake fast; - skip-verify still needs its explicit opt-in. See backup-classes.md. + skip-verify still needs its explicit opt-in. See https://cozystack.io/docs/v1/operations/services/backup-classes/. */}} - name: s3-ca secret: @@ -346,7 +346,7 @@ spec: cluster-scoped and Helm-managed, so an admin raises the ceiling once via backupStrategyController.redisBackupResources in chart values (an in-place edit of the CR reverts on upgrade). Defaults suit - small-to-medium instances. See backup-classes.md. + small-to-medium instances. See https://cozystack.io/docs/v1/operations/services/backup-classes/. */}} resources: {{- toYaml .Values.backupStrategyController.redisBackupResources | nindent 12 }} diff --git a/packages/system/mongodb-rd/cozyrds/mongodb.yaml b/packages/system/mongodb-rd/cozyrds/mongodb.yaml index 9d42f5e20..af8aa5389 100644 --- a/packages/system/mongodb-rd/cozyrds/mongodb.yaml +++ b/packages/system/mongodb-rd/cozyrds/mongodb.yaml @@ -8,7 +8,7 @@ spec: singular: mongodb plural: mongodbs openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of MongoDB replicas in replica set.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration for each MongoDB replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster.","type":"boolean","default":false},"version":{"description":"MongoDB major version to deploy.","type":"string","default":"v8","enum":["v8","v7","v6"]},"sharding":{"description":"Enable sharded cluster mode. When disabled, deploys a replica set.","type":"boolean","default":false},"shardingConfig":{"description":"Configuration for sharded cluster mode.","type":"object","default":{},"required":["configServerSize","configServers","mongos"],"properties":{"configServerSize":{"description":"PVC size for config servers.","default":"3Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"configServers":{"description":"Number of config server replicas.","type":"integer","default":3},"mongos":{"description":"Number of mongos router replicas.","type":"integer","default":2},"shards":{"description":"List of shard configurations.","type":"array","default":[{"name":"rs0","replicas":3,"size":"10Gi"}],"items":{"type":"object","required":["name","replicas","size"],"properties":{"name":{"description":"Shard name.","type":"string"},"replicas":{"description":"Number of replicas in this shard.","type":"integer"},"size":{"description":"PVC size for this shard.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}}}}},"users":{"description":"Users configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"password":{"description":"Password for the user (auto-generated if omitted).","type":"string"}}}},"databases":{"description":"Databases configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"roles":{"description":"Roles assigned to users.","type":"object","properties":{"admin":{"description":"List of users with admin privileges (readWrite + dbAdmin).","type":"array","items":{"type":"string"}},"readonly":{"description":"List of users with read-only privileges.","type":"array","items":{"type":"string"}}}}}}},"backup":{"description":"Backup configuration.","type":"object","default":{},"required":["enabled"],"properties":{"destinationPath":{"description":"Destination path for backups (e.g. s3://bucket/path/). Ignored when useSystemBucket is true (the driver injects the platform bucket).","type":"string","default":"s3://bucket/path/to/folder/"},"enabled":{"description":"Enable regular backups.","type":"boolean","default":false},"endpointURL":{"description":"S3 endpoint URL for uploads. Ignored when useSystemBucket is true (the strategy supplies the platform endpoint).","type":"string","default":"http://minio-gateway-service:9000"},"insecureSkipTLSVerify":{"description":"Skip TLS verification for the S3 endpoint (self-signed, e.g. in-cluster seaweedfs). Ignored when useSystemBucket is true (the platform backupStorage knob applies instead).","type":"boolean","default":false},"retentionPolicy":{"description":"Retention policy (e.g. \"30d\"). Ignored when useSystemBucket is true (no engine-side retention on that flow).","type":"string","default":"30d"},"s3AccessKey":{"description":"Access key for S3 authentication. Ignored when useSystemBucket is true (credentials come from the projected cozy-backups-creds).","type":"string","default":""},"s3SecretKey":{"description":"Secret key for S3 authentication. Ignored when useSystemBucket is true (credentials come from the projected cozy-backups-creds).","type":"string","default":""},"schedule":{"description":"Cron schedule for automated backups. Ignored when useSystemBucket is true (scheduling moves to the BackupClass Plan).","type":"string","default":"0 2 * * *"},"useSystemBucket":{"description":"Opt-in: back up to the platform system bucket without supplying S3 credentials. The cozy-default MongoDB BackupClass injects the storage at BackupJob time; tenants do not set s3AccessKey/s3SecretKey/destinationPath/endpointURL, and backups scope to `/`. Scheduling moves to a BackupClass Plan; `schedule`, `retentionPolicy` and PITR are ignored while this is on, and this flow has no retention yet (see docs/operations/backup-classes.md). Requires `backup.enabled: true` (the pbm agents run on that flag); the chart refuses the combination with `enabled: false` at render time; the schema carries the same rule as a marker for the UI and for the day the apiserver enforces CEL on application schemas (cozystack/cozystack#2657), which it does not yet. A release that has not yet rendered this flag (its cluster still carries the chart's scheduled tasks and PITR) holds a BackupJob until it does, so the tenant's own storage is never injected over.","type":"boolean","default":false}},"x-kubernetes-validations":[{"message":"backup.useSystemBucket requires backup.enabled: the percona-backup-mongodb agents only run when backups are enabled","rule":"!(has(self.useSystemBucket) && self.useSystemBucket) || (has(self.enabled) && self.enabled)"}]},"bootstrap":{"description":"Bootstrap configuration.","type":"object","default":{},"required":["backupName","enabled"],"properties":{"backupName":{"description":"Name of backup to restore from.","type":"string","default":""},"enabled":{"description":"Whether to restore from a backup.","type":"boolean","default":false},"recoveryTime":{"description":"Timestamp for point-in-time recovery; empty means latest.","type":"string","default":""}}}}} + {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of MongoDB replicas in replica set.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration for each MongoDB replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster.","type":"boolean","default":false},"version":{"description":"MongoDB major version to deploy.","type":"string","default":"v8","enum":["v8","v7","v6"]},"sharding":{"description":"Enable sharded cluster mode. When disabled, deploys a replica set.","type":"boolean","default":false},"shardingConfig":{"description":"Configuration for sharded cluster mode.","type":"object","default":{},"required":["configServerSize","configServers","mongos"],"properties":{"configServerSize":{"description":"PVC size for config servers.","default":"3Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"configServers":{"description":"Number of config server replicas.","type":"integer","default":3},"mongos":{"description":"Number of mongos router replicas.","type":"integer","default":2},"shards":{"description":"List of shard configurations.","type":"array","default":[{"name":"rs0","replicas":3,"size":"10Gi"}],"items":{"type":"object","required":["name","replicas","size"],"properties":{"name":{"description":"Shard name.","type":"string"},"replicas":{"description":"Number of replicas in this shard.","type":"integer"},"size":{"description":"PVC size for this shard.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}}}}},"users":{"description":"Users configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"password":{"description":"Password for the user (auto-generated if omitted).","type":"string"}}}},"databases":{"description":"Databases configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"roles":{"description":"Roles assigned to users.","type":"object","properties":{"admin":{"description":"List of users with admin privileges (readWrite + dbAdmin).","type":"array","items":{"type":"string"}},"readonly":{"description":"List of users with read-only privileges.","type":"array","items":{"type":"string"}}}}}}},"backup":{"description":"Backup configuration.","type":"object","default":{},"required":["enabled"],"properties":{"destinationPath":{"description":"Destination path for backups (e.g. s3://bucket/path/). Ignored when useSystemBucket is true (the driver injects the platform bucket).","type":"string","default":"s3://bucket/path/to/folder/"},"enabled":{"description":"Enable regular backups.","type":"boolean","default":false},"endpointURL":{"description":"S3 endpoint URL for uploads. Ignored when useSystemBucket is true (the strategy supplies the platform endpoint).","type":"string","default":"http://minio-gateway-service:9000"},"insecureSkipTLSVerify":{"description":"Skip TLS verification for the S3 endpoint (self-signed, e.g. in-cluster seaweedfs). Ignored when useSystemBucket is true (the platform backupStorage knob applies instead).","type":"boolean","default":false},"retentionPolicy":{"description":"Retention policy (e.g. \"30d\"). Ignored when useSystemBucket is true (no engine-side retention on that flow).","type":"string","default":"30d"},"s3AccessKey":{"description":"Access key for S3 authentication. Ignored when useSystemBucket is true (credentials come from the projected cozy-backups-creds).","type":"string","default":""},"s3SecretKey":{"description":"Secret key for S3 authentication. Ignored when useSystemBucket is true (credentials come from the projected cozy-backups-creds).","type":"string","default":""},"schedule":{"description":"Cron schedule for automated backups. Ignored when useSystemBucket is true (scheduling moves to the BackupClass Plan).","type":"string","default":"0 2 * * *"},"useSystemBucket":{"description":"Opt-in: back up to the platform system bucket without supplying S3 credentials. The cozy-default MongoDB BackupClass injects the storage at BackupJob time; tenants do not set s3AccessKey/s3SecretKey/destinationPath/endpointURL, and backups scope to `/`. Scheduling moves to a BackupClass Plan; `schedule`, `retentionPolicy` and PITR are ignored while this is on, and this flow has no retention yet (see https://cozystack.io/docs/v1/operations/services/backup-classes/). Requires `backup.enabled: true` (the pbm agents run on that flag); the chart refuses the combination with `enabled: false` at render time; the schema carries the same rule as a marker for the UI and for the day the apiserver enforces CEL on application schemas (cozystack/cozystack#2657), which it does not yet. A release that has not yet rendered this flag (its cluster still carries the chart's scheduled tasks and PITR) holds a BackupJob until it does, so the tenant's own storage is never injected over.","type":"boolean","default":false}},"x-kubernetes-validations":[{"message":"backup.useSystemBucket requires backup.enabled: the percona-backup-mongodb agents only run when backups are enabled","rule":"!(has(self.useSystemBucket) && self.useSystemBucket) || (has(self.enabled) && self.enabled)"}]},"bootstrap":{"description":"Bootstrap configuration.","type":"object","default":{},"required":["backupName","enabled"],"properties":{"backupName":{"description":"Name of backup to restore from.","type":"string","default":""},"enabled":{"description":"Whether to restore from a backup.","type":"boolean","default":false},"recoveryTime":{"description":"Timestamp for point-in-time recovery; empty means latest.","type":"string","default":""}}}}} release: prefix: mongodb- labels: diff --git a/packages/system/monitoring-agents/values.yaml b/packages/system/monitoring-agents/values.yaml index 78a6ec9fd..5fce4df58 100644 --- a/packages/system/monitoring-agents/values.yaml +++ b/packages/system/monitoring-agents/values.yaml @@ -15,7 +15,7 @@ kube-state-metrics: # A memory limit is what sets memory.max on the pod cgroup, and the Talos # userspace OOM handler only ever selects cgroups that lack one, so running # this without a limit leaves it an eviction candidate on every node. See - # docs/operations/system-memory-limits.md. + # https://cozystack.io/docs/v1/operations/configuration/system-memory-limits/. # # No CPU limit: it is irrelevant to that OOM ranking, and a ceiling that binds # only buys CFS throttling, which for a metrics scraper turns into scrape diff --git a/pkg/apis/apps/presets/legacy.go b/pkg/apis/apps/presets/legacy.go index 41ff0124c..bbf183840 100644 --- a/pkg/apis/apps/presets/legacy.go +++ b/pkg/apis/apps/presets/legacy.go @@ -22,7 +22,7 @@ limitations under the License. // The same table is mirrored in three other places — keep them in sync: // - packages/library/cozy-lib/templates/_resourcepresets.tpl (legacy block) // - packages/core/platform/images/migrations/migrations/39 -// - docs/operations/resource-presets.md +// - https://cozystack.io/docs/v1/guides/resource-management/ package presets import "fmt" From 9f6fdd77c6ca964e0adcc387e50993487f2e594e Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sat, 24 Oct 2026 15:52:37 +0300 Subject: [PATCH 014/518] docs(etcd): update the affinity notes for the hard member spread The affinity description and the comment above the affinity block explained the soft default by a member that would stay Pending on its node-local volume. Now that the spread is DoNotSchedule by default, the spread keeps such a member Pending as well, once its node holds another member and some other node holds none. Both texts now explain the soft default by what is left, that a required rule forbids two members on one node even when every node has one, and state the node-local case for both rules. The description also says that the operator copies affinity into each EtcdMember when it creates it, so a recreated Pod keeps the rule its member was created with. README, values.schema.json and the ApplicationDefinition are regenerated from it. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- packages/extra/etcd/README.md | 20 +++++++++---------- .../extra/etcd/templates/etcd-cluster.yaml | 14 ++++++++----- packages/extra/etcd/values.schema.json | 2 +- packages/extra/etcd/values.yaml | 2 +- packages/system/etcd-rd/cozyrds/etcd.yaml | 2 +- 5 files changed, 22 insertions(+), 18 deletions(-) diff --git a/packages/extra/etcd/README.md b/packages/extra/etcd/README.md index e96c63832..91341ff32 100644 --- a/packages/extra/etcd/README.md +++ b/packages/extra/etcd/README.md @@ -34,14 +34,14 @@ coordinates). ### Common parameters -| Name | Description | Type | Value | -| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | -------- | -| `version` | etcd version (X.Y.Z) to run, set on the EtcdCluster's spec.version; the operator pulls etcd from its own configured image. | `string` | `3.6.11` | -| `size` | Persistent Volume size. | `quantity` | `4Gi` | -| `storageClass` | StorageClass used to store the data. | `string` | `""` | -| `replicas` | Number of etcd replicas. | `int` | `3` | -| `resources` | Resource configuration for etcd. | `object` | `{}` | -| `resources.cpu` | Number of CPU cores allocated. | `quantity` | `1000m` | -| `resources.memory` | Amount of memory allocated. | `quantity` | `512Mi` | -| `affinity` | Affinity rules set verbatim on the EtcdCluster's spec.affinity, replacing the chart default. When left empty the chart applies a soft (preferred) podAntiAffinity that spreads members across nodes by kubernetes.io/hostname on a best-effort basis, so a single node failure is less likely to take out more than one member. The default is intentionally soft: a hard rule would leave a member Pending when there are fewer schedulable nodes than replicas, or when node-local storage pins a member to a node. Set a hard requiredDuringSchedulingIgnoredDuringExecution rule here to forbid co-location outright (only on clusters with enough nodes and non-node-pinned storage). Note that the operator applies affinity only when it creates or replaces a member Pod and does not reschedule running members, so on a cluster that already has co-located members the rule takes effect only as each member is replaced, not retroactively. | `object` | `{}` | +| Name | Description | Type | Value | +| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | -------- | +| `version` | etcd version (X.Y.Z) to run, set on the EtcdCluster's spec.version; the operator pulls etcd from its own configured image. | `string` | `3.6.11` | +| `size` | Persistent Volume size. | `quantity` | `4Gi` | +| `storageClass` | StorageClass used to store the data. | `string` | `""` | +| `replicas` | Number of etcd replicas. | `int` | `3` | +| `resources` | Resource configuration for etcd. | `object` | `{}` | +| `resources.cpu` | Number of CPU cores allocated. | `quantity` | `1000m` | +| `resources.memory` | Amount of memory allocated. | `quantity` | `512Mi` | +| `affinity` | Affinity rules set verbatim on the EtcdCluster's spec.affinity, replacing the chart default. When left empty the chart applies a soft (preferred) podAntiAffinity by kubernetes.io/hostname on top of the topologySpreadConstraints, which by default require an even spread of members over nodes (whenUnsatisfiable DoNotSchedule, maxSkew 1) unless the platform sets scheduling.globalAppTopologySpreadConstraints. The anti-affinity stays soft because a required rule forbids two members on one node even when every node already has one, so an etcd with more members than schedulable nodes could not start all of them. Set a requiredDuringSchedulingIgnoredDuringExecution rule here only on clusters with at least as many schedulable nodes as members. With node-local storage a recreated member Pod can only return to its own node, so a required rule keeps it Pending while that node holds another member, and the default spread does the same once some other node holds none. The operator copies affinity into each EtcdMember when it creates the member and builds every Pod of that member from that copy, so a change does not reschedule running members and takes effect only as members are replaced by new ones. | `object` | `{}` | diff --git a/packages/extra/etcd/templates/etcd-cluster.yaml b/packages/extra/etcd/templates/etcd-cluster.yaml index 67dd34bd9..3676f53cb 100644 --- a/packages/extra/etcd/templates/etcd-cluster.yaml +++ b/packages/extra/etcd/templates/etcd-cluster.yaml @@ -79,11 +79,15 @@ spec: affinity: {{- toYaml . | nindent 4 }} {{- else }} - # Soft anti-affinity: spread etcd members across nodes on a best-effort - # basis so a single node failure is unlikely to take out more than one - # member. Preferred (not required) because a hard rule would leave a member - # Pending when replicas exceed schedulable nodes, or when node-local storage - # pins a member to a node. Override via .Values.affinity for a hard rule. + # Soft anti-affinity, a preference on top of the spread constraint above, + # which by default is the hard rule that keeps two members off one node + # while another node has none. Preferred, not required, because a required + # rule also forbids two members on one node when every node already has + # one, so an etcd with more members than schedulable nodes could not start + # all of them. With node-local storage a recreated member Pod can only + # return to its own node, and the default spread keeps it Pending there + # while that node holds another member and some other node holds none. + # Override via .Values.affinity for a hard rule. affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: diff --git a/packages/extra/etcd/values.schema.json b/packages/extra/etcd/values.schema.json index b438ca9d7..c33219f4f 100644 --- a/packages/extra/etcd/values.schema.json +++ b/packages/extra/etcd/values.schema.json @@ -67,7 +67,7 @@ } }, "affinity": { - "description": "Affinity rules set verbatim on the EtcdCluster's spec.affinity, replacing the chart default. When left empty the chart applies a soft (preferred) podAntiAffinity that spreads members across nodes by kubernetes.io/hostname on a best-effort basis, so a single node failure is less likely to take out more than one member. The default is intentionally soft: a hard rule would leave a member Pending when there are fewer schedulable nodes than replicas, or when node-local storage pins a member to a node. Set a hard requiredDuringSchedulingIgnoredDuringExecution rule here to forbid co-location outright (only on clusters with enough nodes and non-node-pinned storage). Note that the operator applies affinity only when it creates or replaces a member Pod and does not reschedule running members, so on a cluster that already has co-located members the rule takes effect only as each member is replaced, not retroactively.", + "description": "Affinity rules set verbatim on the EtcdCluster's spec.affinity, replacing the chart default. When left empty the chart applies a soft (preferred) podAntiAffinity by kubernetes.io/hostname on top of the topologySpreadConstraints, which by default require an even spread of members over nodes (whenUnsatisfiable DoNotSchedule, maxSkew 1) unless the platform sets scheduling.globalAppTopologySpreadConstraints. The anti-affinity stays soft because a required rule forbids two members on one node even when every node already has one, so an etcd with more members than schedulable nodes could not start all of them. Set a requiredDuringSchedulingIgnoredDuringExecution rule here only on clusters with at least as many schedulable nodes as members. With node-local storage a recreated member Pod can only return to its own node, so a required rule keeps it Pending while that node holds another member, and the default spread does the same once some other node holds none. The operator copies affinity into each EtcdMember when it creates the member and builds every Pod of that member from that copy, so a change does not reschedule running members and takes effect only as members are replaced by new ones.", "type": "object", "default": {}, "x-kubernetes-preserve-unknown-fields": true diff --git a/packages/extra/etcd/values.yaml b/packages/extra/etcd/values.yaml index 61b994d7c..c5be94909 100644 --- a/packages/extra/etcd/values.yaml +++ b/packages/extra/etcd/values.yaml @@ -23,5 +23,5 @@ resources: cpu: 1000m memory: 512Mi -## @param {object} [affinity] - Affinity rules set verbatim on the EtcdCluster's spec.affinity, replacing the chart default. When left empty the chart applies a soft (preferred) podAntiAffinity that spreads members across nodes by kubernetes.io/hostname on a best-effort basis, so a single node failure is less likely to take out more than one member. The default is intentionally soft: a hard rule would leave a member Pending when there are fewer schedulable nodes than replicas, or when node-local storage pins a member to a node. Set a hard requiredDuringSchedulingIgnoredDuringExecution rule here to forbid co-location outright (only on clusters with enough nodes and non-node-pinned storage). Note that the operator applies affinity only when it creates or replaces a member Pod and does not reschedule running members, so on a cluster that already has co-located members the rule takes effect only as each member is replaced, not retroactively. +## @param {object} [affinity] - Affinity rules set verbatim on the EtcdCluster's spec.affinity, replacing the chart default. When left empty the chart applies a soft (preferred) podAntiAffinity by kubernetes.io/hostname on top of the topologySpreadConstraints, which by default require an even spread of members over nodes (whenUnsatisfiable DoNotSchedule, maxSkew 1) unless the platform sets scheduling.globalAppTopologySpreadConstraints. The anti-affinity stays soft because a required rule forbids two members on one node even when every node already has one, so an etcd with more members than schedulable nodes could not start all of them. Set a requiredDuringSchedulingIgnoredDuringExecution rule here only on clusters with at least as many schedulable nodes as members. With node-local storage a recreated member Pod can only return to its own node, so a required rule keeps it Pending while that node holds another member, and the default spread does the same once some other node holds none. The operator copies affinity into each EtcdMember when it creates the member and builds every Pod of that member from that copy, so a change does not reschedule running members and takes effect only as members are replaced by new ones. affinity: {} diff --git a/packages/system/etcd-rd/cozyrds/etcd.yaml b/packages/system/etcd-rd/cozyrds/etcd.yaml index 90ed8a80a..1ce88e8ae 100644 --- a/packages/system/etcd-rd/cozyrds/etcd.yaml +++ b/packages/system/etcd-rd/cozyrds/etcd.yaml @@ -8,7 +8,7 @@ spec: plural: etcds singular: etcd openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"version":{"description":"etcd version (X.Y.Z) to run, set on the EtcdCluster's spec.version; the operator pulls etcd from its own configured image.","type":"string","default":"3.6.11"},"size":{"description":"Persistent Volume size.","default":"4Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""},"replicas":{"description":"Number of etcd replicas.","type":"integer","default":3},"resources":{"description":"Resource configuration for etcd.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","default":"1000m","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","default":"512Mi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"affinity":{"description":"Affinity rules set verbatim on the EtcdCluster's spec.affinity, replacing the chart default. When left empty the chart applies a soft (preferred) podAntiAffinity that spreads members across nodes by kubernetes.io/hostname on a best-effort basis, so a single node failure is less likely to take out more than one member. The default is intentionally soft: a hard rule would leave a member Pending when there are fewer schedulable nodes than replicas, or when node-local storage pins a member to a node. Set a hard requiredDuringSchedulingIgnoredDuringExecution rule here to forbid co-location outright (only on clusters with enough nodes and non-node-pinned storage). Note that the operator applies affinity only when it creates or replaces a member Pod and does not reschedule running members, so on a cluster that already has co-located members the rule takes effect only as each member is replaced, not retroactively.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}} + {"title":"Chart Values","type":"object","properties":{"version":{"description":"etcd version (X.Y.Z) to run, set on the EtcdCluster's spec.version; the operator pulls etcd from its own configured image.","type":"string","default":"3.6.11"},"size":{"description":"Persistent Volume size.","default":"4Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""},"replicas":{"description":"Number of etcd replicas.","type":"integer","default":3},"resources":{"description":"Resource configuration for etcd.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","default":"1000m","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","default":"512Mi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"affinity":{"description":"Affinity rules set verbatim on the EtcdCluster's spec.affinity, replacing the chart default. When left empty the chart applies a soft (preferred) podAntiAffinity by kubernetes.io/hostname on top of the topologySpreadConstraints, which by default require an even spread of members over nodes (whenUnsatisfiable DoNotSchedule, maxSkew 1) unless the platform sets scheduling.globalAppTopologySpreadConstraints. The anti-affinity stays soft because a required rule forbids two members on one node even when every node already has one, so an etcd with more members than schedulable nodes could not start all of them. Set a requiredDuringSchedulingIgnoredDuringExecution rule here only on clusters with at least as many schedulable nodes as members. With node-local storage a recreated member Pod can only return to its own node, so a required rule keeps it Pending while that node holds another member, and the default spread does the same once some other node holds none. The operator copies affinity into each EtcdMember when it creates the member and builds every Pod of that member from that copy, so a change does not reschedule running members and takes effect only as members are replaced by new ones.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}} release: prefix: "" labels: From dc94660a44d2fa09165223723f5f9a5bd794f896 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Sat, 24 Oct 2026 17:05:20 +0200 Subject: [PATCH 015/518] fix(postgres): render the declared databases in a stable order keys returns a map's keys in Go's randomized iteration order, so a release with several databases rendered DEFINED_DBS in a different order on each upgrade. The init-script Secret, and the checksum the init Job carries, then changed on upgrades that left the databases untouched, which makes a no-op upgrade look like a real change in the release history. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../apps/postgres/templates/init-script.yaml | 2 +- .../tests/init_script_deterministic_test.yaml | 37 +++++++++++++++++++ 2 files changed, 38 insertions(+), 1 deletion(-) create mode 100644 packages/apps/postgres/tests/init_script_deterministic_test.yaml diff --git a/packages/apps/postgres/templates/init-script.yaml b/packages/apps/postgres/templates/init-script.yaml index e78279a2d..395267ff0 100644 --- a/packages/apps/postgres/templates/init-script.yaml +++ b/packages/apps/postgres/templates/init-script.yaml @@ -337,7 +337,7 @@ stringData: RESTORE_MARK={{ with .Values.bootstrap.newServerName }}$(echo {{ b64enc . }} | base64 -d){{ end }} ADOPTED_MARK=$(psql -v ON_ERROR_STOP=1 -t -A --dbname postgres -c "SELECT current_setting('cozystack.adopted_restore', true)") MANAGED_DBS=$(psql -v ON_ERROR_STOP=1 -t -A -c "SELECT datname FROM pg_database d JOIN pg_shdescription s ON d.oid = s.objoid WHERE s.description = 'database managed by helm'") - DEFINED_DBS="{{ join " " (keys .Values.databases) }}" + DEFINED_DBS="{{ join " " (keys .Values.databases | sortAlpha) }}" DELETE_DBS=$(for db in $MANAGED_DBS; do case " $DEFINED_DBS " in *" $db "*) :;; *) echo $db;; esac; done) if [ -n "$RESTORE_MARK" ] && [ "$ADOPTED_MARK" != "$RESTORE_MARK" ]; then diff --git a/packages/apps/postgres/tests/init_script_deterministic_test.yaml b/packages/apps/postgres/tests/init_script_deterministic_test.yaml new file mode 100644 index 000000000..25d0facdd --- /dev/null +++ b/packages/apps/postgres/tests/init_script_deterministic_test.yaml @@ -0,0 +1,37 @@ +suite: init-script renders the same for the same values + +# keys returns a map's keys in Go's randomized iteration order, so a release with +# several databases rendered DEFINED_DBS in a different order on each upgrade. The +# init-script Secret then changed on upgrades that changed nothing in the +# databases, and so did the checksum the init Job carries. + +templates: + - templates/init-script.yaml + +release: + name: pg-test + namespace: tenant-test + +set: + _cluster: + scheduling: {} + +tests: + - it: lists the declared databases in a stable order + set: + databases: + hotel: {} + alpha: {} + golf: {} + charlie: {} + foxtrot: {} + bravo: {} + echo: {} + delta: {} + documentSelector: + path: metadata.name + value: pg-test-init-script + asserts: + - matchRegex: + path: stringData["init.sh"] + pattern: 'DEFINED_DBS="alpha bravo charlie delta echo foxtrot golf hotel"' From 00c11c7a745db9b57d419631c3c88b7ac8c274cb Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 25 Oct 2026 16:29:50 +0300 Subject: [PATCH 016/518] test(monitoring): check the listed alert still reaches email after the blackhole The case lists InfoInhibitor in email.alertnames but only asserted the route order up to the blackhole, so it would still pass if the email route for a listed name stopped being rendered. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../monitoring/tests/alertmanager_blackhole_test.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/packages/system/monitoring/tests/alertmanager_blackhole_test.yaml b/packages/system/monitoring/tests/alertmanager_blackhole_test.yaml index e987bd405..d4e366180 100644 --- a/packages/system/monitoring/tests/alertmanager_blackhole_test.yaml +++ b/packages/system/monitoring/tests/alertmanager_blackhole_test.yaml @@ -43,3 +43,9 @@ tests: matchRegex: path: stringData["alertmanager.yaml"] pattern: '\n routes:\n - receiver: alerta_no_resolve\n repeat_interval: 1m\n matchers:\n - .event="Heartbeat".\n - receiver: blackhole\n matchers:\n - .severity="none".\n' + - documentSelector: + path: metadata.labels.app + value: alertmanager + matchRegex: + path: stringData["alertmanager.yaml"] + pattern: '(?s)\n - receiver: blackhole\n matchers:\n - .severity="none".\n.*\n - receiver: email\n group_wait: 1m\n group_interval: 5m\n repeat_interval: .12h.\n matchers:\n - .alertname=~"\^\(InfoInhibitor\)\$".\n' From 7bda2bdbc07dd941724e5ea12e51e8c808313b1b Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 25 Oct 2026 16:30:14 +0300 Subject: [PATCH 017/518] fix(monitoring): select the vmalert Service the operator creates The VMServiceScrape selected app.kubernetes.io/instance=vmalert, while the chart names the VMAlert vmalert- and the operator puts that name into the instance label of its Service. The scrape matched nothing, so the metrics of vmalert itself were never collected. The instance stays in the selector so the scrape keeps pointing at the one VMAlert this release renders. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../templates/vm/vmalert-scrape.yaml | 4 +- .../monitoring/tests/vmalert_scrape_test.yaml | 38 +++++++++++++++++++ 2 files changed, 41 insertions(+), 1 deletion(-) create mode 100644 packages/system/monitoring/tests/vmalert_scrape_test.yaml diff --git a/packages/system/monitoring/templates/vm/vmalert-scrape.yaml b/packages/system/monitoring/templates/vm/vmalert-scrape.yaml index ee406a275..b467d5564 100644 --- a/packages/system/monitoring/templates/vm/vmalert-scrape.yaml +++ b/packages/system/monitoring/templates/vm/vmalert-scrape.yaml @@ -1,3 +1,4 @@ +{{- with first .Values.metricsStorages }} --- apiVersion: operator.victoriametrics.com/v1beta1 kind: VMServiceScrape @@ -13,5 +14,6 @@ spec: selector: matchLabels: app.kubernetes.io/component: monitoring - app.kubernetes.io/instance: vmalert + app.kubernetes.io/instance: vmalert-{{ .name }} app.kubernetes.io/name: vmalert +{{- end }} diff --git a/packages/system/monitoring/tests/vmalert_scrape_test.yaml b/packages/system/monitoring/tests/vmalert_scrape_test.yaml new file mode 100644 index 000000000..1dfb9ec3e --- /dev/null +++ b/packages/system/monitoring/tests/vmalert_scrape_test.yaml @@ -0,0 +1,38 @@ +suite: vmalert metrics are scraped +templates: + - templates/vm/vmalert-scrape.yaml + +release: + name: monitoring + namespace: tenant-root + +set: + _cluster: + root-host: example.org + _namespace: + host: tenant-root.example.org + ingress: tenant-root + metricsStorages: + - name: shortterm + retentionPeriod: "3d" + deduplicationInterval: "15s" + - name: longterm + retentionPeriod: "14d" + deduplicationInterval: "5m" + +tests: + - it: selects the Service of the VMAlert the chart renders + asserts: + - equal: + path: spec.selector.matchLabels + value: + app.kubernetes.io/component: monitoring + app.kubernetes.io/instance: vmalert-shortterm + app.kubernetes.io/name: vmalert + + - it: renders nothing without metrics storages, like the VMAlert itself + set: + metricsStorages: [] + asserts: + - hasDocuments: + count: 0 From 34766ff0b50af0801f2c94c1435dd744211bbc23 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 25 Oct 2026 17:06:52 +0300 Subject: [PATCH 018/518] fix(vertical-pod-autoscaler): run two admission controllers spread over nodes With one admission controller and failurePolicy Ignore, a pod created while that controller is being rescheduled is admitted without its VPA recommendation, and with updateMode Initial it keeps the defaults for the rest of its life. On a v1.6.5 cluster a vmstorage pod created during a node drain got the operator defaults, a 1500Mi limit against a 3.4Gi recommendation, and was OOMKilled weeks later. A second replica on another node and a PodDisruptionBudget keep one controller serving through a drain. The controller still deletes and recreates its webhook configuration on start, so a restart leaves only that short gap instead of the whole reschedule. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../tests/vpa_test.yaml | 29 +++++++++++++++++++ .../vertical-pod-autoscaler/values.yaml | 12 ++++++++ 2 files changed, 41 insertions(+) diff --git a/packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml b/packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml index 6b8c82675..1d2011330 100644 --- a/packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml +++ b/packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml @@ -242,3 +242,32 @@ tests: count: 0 + + - it: runs two admission controllers spread over nodes + template: charts/vertical-pod-autoscaler/templates/admission-controller/deployment.yaml + asserts: + - equal: + path: spec.replicas + value: 2 + - equal: + path: spec.template.spec.topologySpreadConstraints[0].topologyKey + value: kubernetes.io/hostname + - equal: + path: spec.template.spec.topologySpreadConstraints[0].labelSelector.matchLabels + value: + app.kubernetes.io/name: vertical-pod-autoscaler + app.kubernetes.io/instance: vpa + app.kubernetes.io/component: admission-controller + + - it: keeps one admission controller up through a voluntary disruption + template: charts/vertical-pod-autoscaler/templates/admission-controller/pdb.yaml + asserts: + - equal: + path: spec.minAvailable + value: 1 + - equal: + path: spec.selector.matchLabels + value: + app.kubernetes.io/name: vertical-pod-autoscaler + app.kubernetes.io/instance: vpa + app.kubernetes.io/component: admission-controller diff --git a/packages/system/vertical-pod-autoscaler/values.yaml b/packages/system/vertical-pod-autoscaler/values.yaml index 46678903e..456b99912 100644 --- a/packages/system/vertical-pod-autoscaler/values.yaml +++ b/packages/system/vertical-pod-autoscaler/values.yaml @@ -37,6 +37,18 @@ vertical-pod-autoscaler: prometheus-cadvisor-job-name: cadvisor admissionController: + replicaCount: 2 + pdb: + create: true + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: ScheduleAnyway + labelSelector: + matchLabels: + app.kubernetes.io/name: vertical-pod-autoscaler + app.kubernetes.io/instance: "{{ .Release.Name }}" + app.kubernetes.io/component: admission-controller resources: limits: memory: 256Mi From 61289c721011af0ca4a61eb278ae2664ae7cd8e7 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 25 Oct 2026 17:09:16 +0300 Subject: [PATCH 019/518] fix(etcd): size the defrag job for the client it runs The hourly defrag pod runs the etcd-defrag client, which only sends defragment calls to the members, yet it requested 200m CPU and 256Mi from the tenant quota. On a sandbox tenant whose quota was mostly taken by its own etcd and SeaweedFS, the job could not create a pod and the defrag never ran. The client peaked at 5.1 MiB working set over a day on a v1.6.5 cluster. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- packages/extra/etcd/templates/etcd-defrag.yaml | 8 ++++---- packages/extra/etcd/tests/defrag_test.yaml | 13 +++++++++++++ 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/packages/extra/etcd/templates/etcd-defrag.yaml b/packages/extra/etcd/templates/etcd-defrag.yaml index 52cc63518..c1eefa0db 100644 --- a/packages/extra/etcd/templates/etcd-defrag.yaml +++ b/packages/extra/etcd/templates/etcd-defrag.yaml @@ -29,11 +29,11 @@ spec: drop: ["ALL"] resources: requests: - cpu: 200m - memory: 256Mi + cpu: 10m + memory: 32Mi limits: - cpu: 500m - memory: 512Mi + cpu: 100m + memory: 64Mi args: # The v1alpha2 operator names the headless Service after the # cluster ({{ $.Release.Name }}) and does not use StatefulSet diff --git a/packages/extra/etcd/tests/defrag_test.yaml b/packages/extra/etcd/tests/defrag_test.yaml index 8d74e1a6b..b3acc3401 100644 --- a/packages/extra/etcd/tests/defrag_test.yaml +++ b/packages/extra/etcd/tests/defrag_test.yaml @@ -41,3 +41,16 @@ tests: documentSelector: path: kind value: EtcdCluster + + - it: asks the tenant quota only for what the defrag client uses + template: templates/etcd-defrag.yaml + asserts: + - equal: + path: spec.jobTemplate.spec.template.spec.containers[0].resources + value: + requests: + cpu: 10m + memory: 32Mi + limits: + cpu: 100m + memory: 64Mi From a7ec05f9eb759e0c1025cf4bf95631df338a85c0 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 25 Oct 2026 17:15:04 +0300 Subject: [PATCH 020/518] fix(monitoring-agents): make a silent Watchdog an alarm in Alerta Alerta turns the Watchdog into a heartbeat and stores its severity in the heartbeat attributes. The alerta heartbeats --alert job of the alerta-web image takes the severity of a stale heartbeat from those attributes ahead of its HEARTBEAT_SEVERITY default, so with severity ok the HeartbeatFail it raises is a normal alert and nothing says the alerting pipeline stopped. Without a timeout label the heartbeat also took a day to go stale, while Alertmanager refreshes it every minute. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- .../alerts/general.rules.yaml | 3 +- .../tests/watchdog_heartbeat_test.yaml | 29 +++++++++++++++++++ 2 files changed, 31 insertions(+), 1 deletion(-) create mode 100644 packages/system/monitoring-agents/tests/watchdog_heartbeat_test.yaml diff --git a/packages/system/monitoring-agents/alerts/general.rules.yaml b/packages/system/monitoring-agents/alerts/general.rules.yaml index 924870c40..0ac83fe72 100644 --- a/packages/system/monitoring-agents/alerts/general.rules.yaml +++ b/packages/system/monitoring-agents/alerts/general.rules.yaml @@ -33,7 +33,8 @@ spec: is working properly. expr: vector(1) labels: - severity: ok + severity: major + timeout: "300" exported_instance: global service: general.rules event: Heartbeat diff --git a/packages/system/monitoring-agents/tests/watchdog_heartbeat_test.yaml b/packages/system/monitoring-agents/tests/watchdog_heartbeat_test.yaml new file mode 100644 index 000000000..fe630a80c --- /dev/null +++ b/packages/system/monitoring-agents/tests/watchdog_heartbeat_test.yaml @@ -0,0 +1,29 @@ +suite: Watchdog heartbeat raises an alarm in Alerta when it stops +templates: + - templates/alerts.yaml + +release: + name: monitoring-agents + namespace: cozy-monitoring + +tests: + - it: gives the heartbeat a severity Alerta reports as an alarm + documentSelector: + path: metadata.name + value: alerts-general.rules + asserts: + - equal: + path: spec.groups[0].rules[?(@.alert == "Watchdog")].labels.severity + value: major + + - it: lets the heartbeat go stale within minutes, not a day + documentSelector: + path: metadata.name + value: alerts-general.rules + asserts: + - equal: + path: spec.groups[0].rules[?(@.alert == "Watchdog")].labels.timeout + value: "300" + - equal: + path: spec.groups[0].rules[?(@.alert == "Watchdog")].labels.event + value: Heartbeat From 70e1e8b1454db65222a0b3c0952650fa69239fdc Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 25 Oct 2026 20:33:46 +0300 Subject: [PATCH 021/518] test(vertical-pod-autoscaler): check maxSkew and whenUnsatisfiable of the spread The test only looked at the topology key and the label selector, so a change to maxSkew or to whenUnsatisfiable went unnoticed. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml b/packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml index 1d2011330..0ff338b48 100644 --- a/packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml +++ b/packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml @@ -252,6 +252,12 @@ tests: - equal: path: spec.template.spec.topologySpreadConstraints[0].topologyKey value: kubernetes.io/hostname + - equal: + path: spec.template.spec.topologySpreadConstraints[0].maxSkew + value: 1 + - equal: + path: spec.template.spec.topologySpreadConstraints[0].whenUnsatisfiable + value: ScheduleAnyway - equal: path: spec.template.spec.topologySpreadConstraints[0].labelSelector.matchLabels value: From bcc5f3216a7ea4ec93bf2597ad4821ff97e855ac Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Mon, 26 Oct 2026 18:21:32 +0300 Subject: [PATCH 022/518] fix(kubevirt): let tenant admins upload a VMDisk image with virtctl A VMDisk created with source upload waits in UploadReady for an image, but no tenant role can fill it. virtctl image-upload reads the disk PVC and CDI's PVC Prime and requests an upload token, and the tenant roles grant neither get on persistentvolumeclaims nor create on uploadtokenrequests, so the command stops on a forbidden PVC read. cozy:tenant:admin:base now grants both, so tenant admins and super-admins get them in their own namespaces through the existing RoleBindings, while the use and view tiers do not. Reading the DataVolume is already granted for the disk's own name by the vm-disk chart's dashboard role. The vm-disk notes print the command with --no-create, because the chart has already created the DataVolume, and the security self-assessment and threat model list the new grants. The rules, their tests, the notes change and the docs lines come from #3847, which also carries a console upload panel and an upload proxy certificate that are still in review there. Fixes #3759 Co-authored-by: Myasnikov Daniil Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- docs/security/self-assessment.md | 2 +- docs/security/threat-model.md | 4 +- packages/apps/vm-disk/templates/NOTES.txt | 2 +- .../apps/vm-disk/tests/upload_notes_test.yaml | 18 ++++ .../templates/clusterroles.yaml | 11 +++ .../tests/cdi_upload_rbac_test.yaml | 97 +++++++++++++++++++ 6 files changed, 130 insertions(+), 4 deletions(-) create mode 100644 packages/apps/vm-disk/tests/upload_notes_test.yaml create mode 100644 packages/system/cozystack-basics/tests/cdi_upload_rbac_test.yaml diff --git a/docs/security/self-assessment.md b/docs/security/self-assessment.md index cb8360904..b59cbda4a 100644 --- a/docs/security/self-assessment.md +++ b/docs/security/self-assessment.md @@ -55,7 +55,7 @@ Actors are decomposed by **trust boundary**: two components are distinct actors | GitOps / Flux delivery layer | T1 — platform control plane | Reconciles HelmReleases/Kustomizations into arbitrary privileged manifests; effectively cluster-admin. | Tenants cannot create HelmReleases (RBAC); not tenant-network-reachable. | | `cozystack-api` aggregated API server | T1 — platform control plane | Converts tenant CRs into HelmReleases; can CRUD HelmReleases, Secrets, and Cilium policies cluster-wide. Acts as the policy-enforcing gateway. | Constrains tenants via fixed chart reference, name/quota/host validation, and admission. | | System operators / controllers | T1 — platform control plane | Domain-scoped cluster privileges (provision control planes, mint certs/secrets, program CNI). | Distinct ServiceAccounts; isolated from tenants by RBAC and network policy. | -| Tenant administrator (OIDC `-admin` / `-super-admin`) | T2 — tenant boundary | CRUD on `apps.cozystack.io` application kinds and `sdn.cozystack.io/securitygroups` in its namespace subtree; read on pods/services/ingress plus `delete` on a fixed set (pods/services/endpoints/events); no create/update of arbitrary core resources. (The ClusterRole also lists `persistentvolumes`, but that grant is inert under the namespace-scoped RoleBinding, since PVs are cluster-scoped.) `super-admin` holds the `apps.cozystack.io/*` wildcard (and can create child `Tenant`s); `admin` has an enumerated allowlist excluding some kinds (e.g. `tenants`, `monitoring`, `etcd`, `ingress`). | Namespace-scoped RBAC + admission policies + quota; no HelmRelease or cross-tenant access. | +| Tenant administrator (OIDC `-admin` / `-super-admin`) | T2 — tenant boundary | CRUD on `apps.cozystack.io` application kinds and `sdn.cozystack.io/securitygroups` in its namespace subtree; read on pods/services/ingress plus `delete` on a fixed set (pods/services/endpoints/events); `get` on persistentvolumeclaims and `create` on `upload.cdi.kubevirt.io/uploadtokenrequests`, the pair `virtctl image-upload` needs to upload into an existing VMDisk; no create/update of arbitrary core resources. (The ClusterRole also lists `persistentvolumes`, but that grant is inert under the namespace-scoped RoleBinding, since PVs are cluster-scoped.) `super-admin` holds the `apps.cozystack.io/*` wildcard (and can create child `Tenant`s); `admin` has an enumerated allowlist excluding some kinds (e.g. `tenants`, `monitoring`, `etcd`, `ingress`). | Namespace-scoped RBAC + admission policies + quota; no HelmRelease or cross-tenant access. | | Tenant user (OIDC `-view` / `-use`) | T2 — tenant boundary | Read-only view plus operational verbs (VM console/vnc/portforward, start/stop/restart). Cannot create Application CRs. | Strictly dominated by tenant-admin; cannot escalate. | | Parent vs child (nested) tenant | T2 — tenant boundary | Parent ServiceAccounts bind into descendant namespaces; children do not bind upward. Quota hierarchically capped. | Ancestor→descendant trust is one-way. | | Tenant workloads / managed services | T3 — workload boundary | Run tenant-controlled code; no cluster-API rights beyond their ServiceAccount. | Namespace-scoped RBAC/identity isolate API objects; on the network, Cilium egress confines each tenant pod to its own subtree + platform system services + `world`, so unrelated tenants cannot reach each other over the pod network (enforced on egress; ingress is open to `cluster` — see Non-goals). | diff --git a/docs/security/threat-model.md b/docs/security/threat-model.md index 9790d9591..0a4ea0bb3 100644 --- a/docs/security/threat-model.md +++ b/docs/security/threat-model.md @@ -39,7 +39,7 @@ The actors below are decomposed **by trust boundary**, not by component: two com | GitOps / Flux delivery layer | T1 | Reconciles HelmReleases/Kustomizations into arbitrary privileged manifests; effectively cluster-admin. | Tenants cannot create HelmReleases/Kustomizations (RBAC); not tenant-network-reachable. | | `cozystack-api` aggregated API server (ServiceAccount) | T1 | CRUD HelmReleases, Secrets, and CiliumNetworkPolicies cluster-wide; reads Namespaces/RBAC/quotas/nodes. Acts as the policy-enforcing gateway that converts tenant CRs to HelmReleases. | A distinct, network-exposed attack surface from Flux; constrains tenants via fixed chart ref + name/quota/host validation + admission. | | System operators / controllers | T1 | Domain-scoped cluster privileges (provision control planes, mint certs/secrets, program CNI). | Distinct ServiceAccounts, but all cluster-admin-adjacent; isolated from tenants only by RBAC and network policy. | -| Tenant administrator (OIDC groups `-admin`, `-super-admin`) | T2 | CRUD on `apps.cozystack.io` application kinds and `sdn.cozystack.io/securitygroups` within its namespace subtree; read on pods/services/ingress plus `delete` on a fixed set (pods/services/endpoints/events); no create/update of arbitrary core resources. (The ClusterRole also lists `persistentvolumes`, but that grant is inert under the namespace-scoped RoleBinding, since PVs are cluster-scoped.) The `-super-admin` group holds the `apps.cozystack.io/*` wildcard (and so can create child `Tenant`s); the `-admin` group has an enumerated allowlist that excludes some kinds (e.g. `tenants`, `monitoring`, `etcd`, `ingress`, `external-dns`, `bootbox`). | Namespace-scoped RoleBindings + admission policies + quota; no access to HelmReleases, arbitrary core writes, or other tenants. | +| Tenant administrator (OIDC groups `-admin`, `-super-admin`) | T2 | CRUD on `apps.cozystack.io` application kinds and `sdn.cozystack.io/securitygroups` within its namespace subtree; read on pods/services/ingress plus `delete` on a fixed set (pods/services/endpoints/events); `get` on persistentvolumeclaims and `create` on `upload.cdi.kubevirt.io/uploadtokenrequests`, the pair `virtctl image-upload` needs to upload into an existing VMDisk; no create/update of arbitrary core resources. (The ClusterRole also lists `persistentvolumes`, but that grant is inert under the namespace-scoped RoleBinding, since PVs are cluster-scoped.) The `-super-admin` group holds the `apps.cozystack.io/*` wildcard (and so can create child `Tenant`s); the `-admin` group has an enumerated allowlist that excludes some kinds (e.g. `tenants`, `monitoring`, `etcd`, `ingress`, `external-dns`, `bootbox`). | Namespace-scoped RoleBindings + admission policies + quota; no access to HelmReleases, arbitrary core writes, or other tenants. | | Tenant user (OIDC groups `-view`, `-use`) | T2 | Read-only view of Application CRs, pods, and services, plus operational verbs only (VM console/vnc/portforward, start/stop/restart). Cannot create Application CRs. | Strictly dominated by tenant-admin; cannot escalate to create/delete, so it is a distinct actor. | | Parent tenant vs child (nested) tenant | T2 | Parent ServiceAccounts are bound into descendant namespaces; child ServiceAccounts are not bound upward. Quota is hierarchically capped. | Ancestor→descendant only; compromising a parent compromises its children, not the reverse. | | Tenant workloads / managed services | T3 | Run tenant-controlled code; no cluster-API rights beyond their ServiceAccount. | Namespace-scoped RBAC/identity isolate their API objects. On the network, Cilium egress confines each tenant pod to its own tenant subtree, platform system services, and `world` (out-of-cluster), so unrelated tenants cannot reach each other over the pod network — enforced on the egress side (tenant-pod ingress is open to `cluster`; see Non-goals for residual caveats). | @@ -92,7 +92,7 @@ The following are explicitly out of scope for the platform's security guarantees The single most load-bearing isolation control is the combination of two facts: -1. The tenant ClusterRoles (`packages/system/cozystack-basics/templates/clusterroles.yaml`) grant **no** `helm.toolkit.fluxcd.io` access and **no** create/update of arbitrary core resources — only CRUD on `apps.cozystack.io` application kinds (super-admin holds the `*` wildcard; admin an enumerated allowlist) and `sdn.cozystack.io/securitygroups`, read access to pods/services/ingresses, and (for tenant-admins) `delete` on a fixed set of core objects (pods/services/endpoints/events; the `persistentvolumes` entry in the ClusterRole is inert under the namespace-scoped RoleBinding). Tenant isolation is enforced by the *absence* of privileged grants in RBAC, not by an admission policy that whitelists `apps.cozystack.io/*` writes. +1. The tenant ClusterRoles (`packages/system/cozystack-basics/templates/clusterroles.yaml`) grant **no** `helm.toolkit.fluxcd.io` access and **no** create/update of arbitrary core resources — only CRUD on `apps.cozystack.io` application kinds (super-admin holds the `*` wildcard; admin an enumerated allowlist) and `sdn.cozystack.io/securitygroups`, read access to pods/services/ingresses, and (for tenant-admins) `delete` on a fixed set of core objects (pods/services/endpoints/events; the `persistentvolumes` entry in the ClusterRole is inert under the namespace-scoped RoleBinding), `get` on persistentvolumeclaims and `create` on `upload.cdi.kubevirt.io/uploadtokenrequests` (for uploading an image into an existing VMDisk with `virtctl image-upload`). Tenant isolation is enforced by the *absence* of privileged grants in RBAC, not by an admission policy that whitelists `apps.cozystack.io/*` writes. 2. The Application-to-HelmRelease conversion (`pkg/registry/apps/application/rest.go`) takes the chart reference, release-name prefix, and `valuesFrom` from server-side configuration; the tenant controls only the Helm values (`app.Spec`). If a future `ApplicationDefinition`, RBAC change, or API change ever let a tenant influence the chart reference, `valuesFrom`, or write HelmReleases directly, the T2→T1 isolation would collapse. Changes touching tenant RBAC or the Application REST layer should be reviewed against this invariant. diff --git a/packages/apps/vm-disk/templates/NOTES.txt b/packages/apps/vm-disk/templates/NOTES.txt index 73d07cc59..0940e75ba 100644 --- a/packages/apps/vm-disk/templates/NOTES.txt +++ b/packages/apps/vm-disk/templates/NOTES.txt @@ -1,5 +1,5 @@ {{- if and .Values.source (hasKey .Values.source "upload") }} To upload your disk run: -virtctl image-upload dv -n {{ .Release.Namespace }} {{ .Release.Name }} --image-path +virtctl image-upload dv {{ .Release.Name }} --no-create --namespace {{ .Release.Namespace }} --image-path {{- end }} diff --git a/packages/apps/vm-disk/tests/upload_notes_test.yaml b/packages/apps/vm-disk/tests/upload_notes_test.yaml new file mode 100644 index 000000000..16a6432a3 --- /dev/null +++ b/packages/apps/vm-disk/tests/upload_notes_test.yaml @@ -0,0 +1,18 @@ +suite: VM disk upload notes +templates: + - templates/NOTES.txt + +release: + name: vm-disk-demo + namespace: tenant-root + +tests: + - it: reuses the chart-created DataVolume for local image upload + set: + source: + upload: {} + asserts: + - matchRegexRaw: + pattern: 'virtctl image-upload dv vm-disk-demo --no-create --namespace tenant-root --image-path ' + - notMatchRegexRaw: + pattern: 'virtctl image-upload dv -n' diff --git a/packages/system/cozystack-basics/templates/clusterroles.yaml b/packages/system/cozystack-basics/templates/clusterroles.yaml index 9807d32f1..5df762886 100644 --- a/packages/system/cozystack-basics/templates/clusterroles.yaml +++ b/packages/system/cozystack-basics/templates/clusterroles.yaml @@ -235,6 +235,17 @@ metadata: labels: rbac.cozystack.io/aggregate-to-tenant-admin: "true" rules: +# virtctl reads both the disk PVC and CDI's dynamically named PVC Prime. +- apiGroups: [""] + resources: + - persistentvolumeclaims + verbs: + - get +- apiGroups: ["upload.cdi.kubevirt.io"] + resources: + - uploadtokenrequests + verbs: + - create - apiGroups: [""] resources: - pods diff --git a/packages/system/cozystack-basics/tests/cdi_upload_rbac_test.yaml b/packages/system/cozystack-basics/tests/cdi_upload_rbac_test.yaml new file mode 100644 index 000000000..b99df44bf --- /dev/null +++ b/packages/system/cozystack-basics/tests/cdi_upload_rbac_test.yaml @@ -0,0 +1,97 @@ +suite: tenant admin CDI upload RBAC + +templates: + - templates/clusterroles.yaml + +release: + name: cozystack-basics + namespace: cozy-system + +tests: + - it: lets tenant admins inspect upload PVCs and create CDI upload tokens + documentSelector: + path: metadata.name + value: cozy:tenant:admin:base + asserts: + - equal: + path: metadata.labels["rbac.cozystack.io/aggregate-to-tenant-admin"] + value: "true" + - contains: + path: rules + content: + apiGroups: [""] + resources: ["persistentvolumeclaims"] + verbs: ["get"] + - contains: + path: rules + content: + apiGroups: ["upload.cdi.kubevirt.io"] + resources: ["uploadtokenrequests"] + verbs: ["create"] + + - it: keeps the admin grant out of the use, view and service-account aggregation + documentSelector: + path: metadata.name + value: cozy:tenant:admin:base + asserts: + - notExists: + path: metadata.labels["rbac.cozystack.io/aggregate-to-tenant-use"] + - notExists: + path: metadata.labels["rbac.cozystack.io/aggregate-to-tenant-view"] + - notExists: + path: metadata.labels["rbac.cozystack.io/aggregate-to-tenant"] + + - it: does not grant CDI upload permissions to the use tier + documentSelector: + path: metadata.name + value: cozy:tenant:use:base + asserts: + - notContains: + path: rules + content: + apiGroups: [""] + resources: ["persistentvolumeclaims"] + verbs: ["get"] + - notContains: + path: rules + content: + apiGroups: ["upload.cdi.kubevirt.io"] + resources: ["uploadtokenrequests"] + verbs: ["create"] + + - it: aggregates upload permissions through admin into super-admin + documentSelector: + path: metadata.name + value: cozy:tenant:admin + asserts: + - equal: + path: metadata.labels["rbac.cozystack.io/aggregate-to-tenant-super-admin"] + value: "true" + - contains: + path: aggregationRule.clusterRoleSelectors + content: + matchLabels: + rbac.cozystack.io/aggregate-to-tenant-admin: "true" + + - it: selects admin permissions for super-admin + documentSelector: + path: metadata.name + value: cozy:tenant:super-admin + asserts: + - contains: + path: aggregationRule.clusterRoleSelectors + content: + matchLabels: + rbac.cozystack.io/aggregate-to-tenant-super-admin: "true" + + - it: does not grant CDI upload token creation to the view tier + documentSelector: + path: metadata.name + value: cozy:tenant:view:base + asserts: + - notContains: + path: rules + content: + apiGroups: ["upload.cdi.kubevirt.io"] + resources: ["uploadtokenrequests"] + verbs: ["create"] From 982db83dd22d6a0257beda788cf75839c738ab42 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Mon, 26 Oct 2026 20:10:59 +0300 Subject: [PATCH 023/518] fix(postgres): accept a standard five-field backup schedule The mongodb app and backup Plans take a standard five-field cron, but the postgres chart hands backup.schedule to a CloudNativePG ScheduledBackup, which parses it with robfig/cron v1 Parse. That parser reads the first field as seconds and makes day-of-week optional, so a five-field value passes admission with only a warning and runs one field off. 0 2 * * * runs hourly and 5 * * * * every minute. A five-field schedule now gets a leading 0 seconds field. Six fields and descriptors pass through, and any other field count fails the render. A release that already sets five fields gets the standard meaning on upgrade, so 0 2 * * * goes from hourly to daily. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- api/apps/v1alpha1/postgresql/types.go | 2 +- packages/apps/postgres/README.md | 15 ++-- packages/apps/postgres/templates/backup.yaml | 19 ++++- .../postgres/tests/backup_schedule_test.yaml | 80 +++++++++++++++++++ packages/apps/postgres/values.schema.json | 2 +- packages/apps/postgres/values.yaml | 2 +- .../system/postgres-rd/cozyrds/postgres.yaml | 2 +- 7 files changed, 107 insertions(+), 15 deletions(-) create mode 100644 packages/apps/postgres/tests/backup_schedule_test.yaml diff --git a/api/apps/v1alpha1/postgresql/types.go b/api/apps/v1alpha1/postgresql/types.go index f0415f398..a1e32ed26 100644 --- a/api/apps/v1alpha1/postgresql/types.go +++ b/api/apps/v1alpha1/postgresql/types.go @@ -124,7 +124,7 @@ type Backup struct { // DEPRECATED. See `s3AccessKey`. // +kubebuilder:default:="" S3SecretKey string `json:"s3SecretKey,omitempty"` - // Legacy. Cron schedule (CNPG 6-field format) for the chart-emitted ScheduledBackup. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time. + // Legacy. Cron schedule for the chart-emitted ScheduledBackup, either a standard five-field cron such as `0 2 * * *` for every day at 02:00, or six fields with leading seconds such as `0 0 2 * * *` for the same time. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time. // +kubebuilder:default:="" Schedule string `json:"schedule,omitempty"` // Opt-in: when true, the chart-emitted `-s3-creds` Secret is skipped AND `spec.plugins` (plus the barman-cloud ObjectStore) is left UNSET in the chart-rendered Cluster — the cozy-default BackupClass driver SSA-applies an ObjectStore (carrying destinationPath/endpointURL/credentials) and patches `spec.plugins` on the live Cluster when the first BackupJob runs. Consequence: plugin WAL archiving is NOT active until that first BackupJob fires; WAL accumulates on the PVC in the meantime, so fire an ad-hoc BackupJob immediately after enabling the flag on existing releases. Use together with the platform `cozy-default` BackupClass — tenants do not need to fill `s3AccessKey`/`s3SecretKey` or `destinationPath`/`endpointURL`. The destination path automatically scopes to `s3://cozy-backups///`. diff --git a/packages/apps/postgres/README.md b/packages/apps/postgres/README.md index bdc13ea9c..49f08d441 100644 --- a/packages/apps/postgres/README.md +++ b/packages/apps/postgres/README.md @@ -20,7 +20,7 @@ together rather than picking one: | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | **Archive plumbing** (chart, legacy) | Renders `spec.plugins` (referencing a barman-cloud `ObjectStore`) on the cnpg.io Cluster from helm-install onwards. CNPG runs the barman-cloud plugin as a WAL-archiver sidecar, every WAL switch ships to object storage, and any backup driver gets a complete WAL chain to start replay from. Renders only when `backup.enabled=true`, `useSystemBucket=false`, `destinationPath` is non-empty, AND inline-or-external creds are supplied. **Skipped in the platform `useSystemBucket=true` mode** — the CNPG backup driver SSA-applies the `ObjectStore` and patches `spec.plugins` onto the live Cluster at first BackupJob time; until that first BackupJob fires, plugin WAL archiving is not active and WAL accumulates on the PVC. Fire an ad-hoc BackupJob immediately after enabling the flag on existing releases. | Legacy: `backup.enabled=true` plus `backup.destinationPath`, `backup.endpointURL`, and either `backup.s3AccessKey`+`backup.s3SecretKey` or `backup.s3CredentialsSecret`. Platform: `backup.enabled=true` plus `backup.useSystemBucket=true` — no S3 fields. | | **Backup orchestration** (recommended) | Drives ad-hoc and scheduled backups, retention, and restores from `backups.cozystack.io` resources that can span multiple Postgres apps in a tenant. The driver SSA-applies its own barman-cloud `ObjectStore` and patches `spec.plugins` with `ForceOwnership` (so compression / target settings flow from the strategy template, while the live Cluster's `serverName` — its WAL-archive S3 prefix — is always preserved); chart-managed values cover the same destination so there is no tug-of-war on the live Cluster. | `strategy.backups.cozystack.io/CNPG` + `BackupClass` + `Plan` (recurring) or `BackupJob` (ad-hoc), restores via `RestoreJob` | -| Legacy chart-emitted scheduled backup | The chart can also emit a `cnpg.io/ScheduledBackup` directly. Superseded by the BackupClass + Plan path, kept around for clusters that did not migrate. **Off by default** - rendered only when both `backup.enabled` and `backup.schedule` are non-empty. With `backup.schedule=""` plugin WAL archiving still runs; only the chart-emitted scheduled backup is silent. | `backup.enabled=true` plus `backup.schedule` (CNPG 6-field cron, e.g. `"0 2 * * * *"`) | +| Legacy chart-emitted scheduled backup | The chart can also emit a `cnpg.io/ScheduledBackup` directly. Superseded by the BackupClass + Plan path, kept around for clusters that did not migrate. **Off by default** - rendered only when both `backup.enabled` and `backup.schedule` are non-empty. With `backup.schedule=""` plugin WAL archiving still runs; only the chart-emitted scheduled backup is silent. | `backup.enabled=true` plus `backup.schedule` (five-field cron such as `"0 2 * * *"`, or six fields with leading seconds) | The **canonical setup** depends on whether the cluster ships the platform `cozy-default` BackupClass. @@ -87,16 +87,11 @@ field. ### How to enable chart-managed scheduled backups (legacy) -The chart can also emit a `cnpg.io/ScheduledBackup` directly, without a -BackupClass. Superseded by the BackupClass + Plan path above and kept -around for clusters that did not migrate. It does not run by default - -`backup.schedule` defaults to an empty string, which gates the chart's -ScheduledBackup template off. To turn it on, fill in a CNPG 6-field cron -expression: +The chart can also emit a `cnpg.io/ScheduledBackup` directly, without a BackupClass. Superseded by the BackupClass + Plan path above and kept around for clusters that did not migrate. It does not run by default - `backup.schedule` defaults to an empty string, which gates the chart's ScheduledBackup template off. To turn it on, set a standard five-field cron such as `0 2 * * *` for every day at 02:00, or the CloudNativePG six-field form with a leading seconds field such as `0 0 2 * * *` for the same time. The chart prepends a `0` seconds field to a five-field value, because CloudNativePG reads the first field as seconds and would otherwise run `0 2 * * *` every hour. ```yaml ## @param backup.enabled Enable plugin WAL archiving + render the chart-managed ScheduledBackup -## @param backup.schedule Cron schedule (CNPG 6-field). Empty means no chart-managed ScheduledBackup +## @param backup.schedule Cron schedule, five fields or six with leading seconds. Empty means no chart-managed ScheduledBackup ## @param backup.retentionPolicy Retention policy ## @param backup.destinationPath Path to store the backup (i.e. s3://bucket/path/to/folder) ## @param backup.endpointURL S3 Endpoint used to upload data to the cloud @@ -107,7 +102,7 @@ backup: retentionPolicy: 30d destinationPath: s3://bucket/path/to/folder/ endpointURL: http://minio-gateway-service:9000 - schedule: "0 2 * * * *" # opt in - empty (the default) means no chart-managed schedule + schedule: "0 2 * * *" # opt in - empty (the default) means no chart-managed schedule s3AccessKey: oobaiRus9pah8PhohL1ThaeTa4UVa7gu s3SecretKey: ju3eum4dekeich9ahM1te8waeGai0oog ``` @@ -295,7 +290,7 @@ For `sslmode=verify-full` to work, the CA bundle retrieved above must be saved t | `backup` | Backup configuration. | `object` | `{}` | | `backup.enabled` | Enable regular backups. | `bool` | `false` | | `backup.useSystemBucket` | Opt-in: when true, the chart-emitted `-s3-creds` Secret is skipped AND `spec.plugins` (plus the barman-cloud ObjectStore) is left UNSET in the chart-rendered Cluster — the cozy-default BackupClass driver SSA-applies an ObjectStore (carrying destinationPath/endpointURL/credentials) and patches `spec.plugins` on the live Cluster when the first BackupJob runs. Consequence: plugin WAL archiving is NOT active until that first BackupJob fires; WAL accumulates on the PVC in the meantime, so fire an ad-hoc BackupJob immediately after enabling the flag on existing releases. Use together with the platform `cozy-default` BackupClass — tenants do not need to fill `s3AccessKey`/`s3SecretKey` or `destinationPath`/`endpointURL`. The destination path automatically scopes to `s3://cozy-backups///`. | `bool` | `false` | -| `backup.schedule` | Legacy. Cron schedule (CNPG 6-field format) for the chart-emitted ScheduledBackup. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time. | `string` | `""` | +| `backup.schedule` | Legacy. Cron schedule for the chart-emitted ScheduledBackup, either a standard five-field cron such as `0 2 * * *` for every day at 02:00, or six fields with leading seconds such as `0 0 2 * * *` for the same time. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time. | `string` | `""` | | `backup.retentionPolicy` | Retention policy (e.g. "30d"). | `string` | `30d` | | `backup.destinationPath` | DEPRECATED. Per-tenant S3 configuration is superseded by the platform-managed `cozy-default` BackupClass and the `cozy-backups` system bucket. Leave empty for new installations; the BackupClass driver picks up the system-managed coordinates. Kept for in-place upgrade compatibility. | `string` | `s3://bucket/path/to/folder/` | | `backup.endpointURL` | DEPRECATED. See `destinationPath`. | `string` | `http://minio-gateway-service:9000` | diff --git a/packages/apps/postgres/templates/backup.yaml b/packages/apps/postgres/templates/backup.yaml index aa1d888e3..4025ca571 100644 --- a/packages/apps/postgres/templates/backup.yaml +++ b/packages/apps/postgres/templates/backup.yaml @@ -23,15 +23,32 @@ against a Cluster with no backup configuration and fail. This mirrors the ClickHouse `useSystemBucket` gates on the chart-managed backup resources. + + CNPG parses `schedule` with robfig/cron v1 `Parse`, which takes the first + field as seconds and makes day-of-week optional. A standard five-field + cron passes its webhook with only a warning and then runs one field off, + so `5 * * * *` fires every minute instead of hourly. A five-field value + therefore gets a leading `0` seconds field. Six fields and descriptors + such as `@daily` pass through as they are, and any other field count + fails the render, which CNPG would reject at admission anyway. */}} {{- if and .Values.backup.enabled .Values.backup.schedule (not .Values.backup.useSystemBucket) }} +{{- $schedule := .Values.backup.schedule }} +{{- $fields := regexFindAll `\S+` $schedule -1 }} +{{- if not (hasPrefix "@" (trim $schedule)) }} +{{- if eq (len $fields) 5 }} +{{- $schedule = printf "0 %s" (join " " $fields) }} +{{- else if ne (len $fields) 6 }} +{{- fail (printf "postgres: backup.schedule %q is not a cron schedule with 5 or 6 fields (it has %d). Use a standard five-field cron such as \"0 2 * * *\", a six-field cron with leading seconds such as \"0 0 2 * * *\", or a descriptor such as \"@daily\"." $schedule (len $fields)) }} +{{- end }} +{{- end }} --- apiVersion: postgresql.cnpg.io/v1 kind: ScheduledBackup metadata: name: {{ .Release.Name }} spec: - schedule: {{ .Values.backup.schedule | quote }} + schedule: {{ $schedule | quote }} backupOwnerReference: self method: plugin pluginConfiguration: diff --git a/packages/apps/postgres/tests/backup_schedule_test.yaml b/packages/apps/postgres/tests/backup_schedule_test.yaml new file mode 100644 index 000000000..e78526848 --- /dev/null +++ b/packages/apps/postgres/tests/backup_schedule_test.yaml @@ -0,0 +1,80 @@ +suite: chart-managed scheduled backup takes five-field and six-field cron + +# CNPG parses the ScheduledBackup schedule with robfig/cron v1 Parse, which +# takes the first field as seconds and accepts five fields with only a +# warning, so an unconverted "5 * * * *" would run every minute. + +templates: + - templates/backup.yaml + +release: + name: pg-test + namespace: tenant-test + +tests: + - it: prepends a zero seconds field to a standard five-field schedule + set: + backup: + enabled: true + schedule: "5 * * * *" + asserts: + - equal: + path: spec.schedule + value: "0 5 * * * *" + + - it: joins the fields of a five-field schedule with single spaces + set: + backup: + enabled: true + schedule: " 0 2 * *\t* " + asserts: + - equal: + path: spec.schedule + value: "0 0 2 * * *" + + - it: passes a six-field schedule through unchanged + set: + backup: + enabled: true + schedule: "0 5 * * * *" + asserts: + - equal: + path: spec.schedule + value: "0 5 * * * *" + + - it: passes a descriptor through unchanged + set: + backup: + enabled: true + schedule: "@every 6h" + asserts: + - equal: + path: spec.schedule + value: "@every 6h" + + - it: refuses a schedule with four fields + set: + backup: + enabled: true + schedule: "5 * * *" + asserts: + - failedTemplate: + errorMessage: 'postgres: backup.schedule "5 * * *" is not a cron schedule with 5 or 6 fields (it has 4). Use a standard five-field cron such as "0 2 * * *", a six-field cron with leading seconds such as "0 0 2 * * *", or a descriptor such as "@daily".' + + - it: refuses a schedule with seven fields + set: + backup: + enabled: true + schedule: "0 0 2 * * * 2026" + asserts: + - failedTemplate: + errorMessage: 'postgres: backup.schedule "0 0 2 * * * 2026" is not a cron schedule with 5 or 6 fields (it has 7). Use a standard five-field cron such as "0 2 * * *", a six-field cron with leading seconds such as "0 0 2 * * *", or a descriptor such as "@daily".' + + - it: refuses a schedule of only spaces + set: + backup: + enabled: true + schedule: " " + asserts: + - failedTemplate: + errorMessage: 'postgres: backup.schedule " " is not a cron schedule with 5 or 6 fields (it has 0). Use a standard five-field cron such as "0 2 * * *", a six-field cron with leading seconds such as "0 0 2 * * *", or a descriptor such as "@daily".' diff --git a/packages/apps/postgres/values.schema.json b/packages/apps/postgres/values.schema.json index 91747ba24..799b05776 100644 --- a/packages/apps/postgres/values.schema.json +++ b/packages/apps/postgres/values.schema.json @@ -427,7 +427,7 @@ "default": "" }, "schedule": { - "description": "Legacy. Cron schedule (CNPG 6-field format) for the chart-emitted ScheduledBackup. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time.", + "description": "Legacy. Cron schedule for the chart-emitted ScheduledBackup, either a standard five-field cron such as `0 2 * * *` for every day at 02:00, or six fields with leading seconds such as `0 0 2 * * *` for the same time. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time.", "type": "string", "default": "" }, diff --git a/packages/apps/postgres/values.yaml b/packages/apps/postgres/values.yaml index 83b658a64..a938497d2 100644 --- a/packages/apps/postgres/values.yaml +++ b/packages/apps/postgres/values.yaml @@ -321,7 +321,7 @@ databases: {} ## @typedef {struct} Backup - Backup configuration. ## @field {bool} enabled - Enable regular backups. ## @field {bool} [useSystemBucket] - Opt-in: when true, the chart-emitted `-s3-creds` Secret is skipped AND `spec.plugins` (plus the barman-cloud ObjectStore) is left UNSET in the chart-rendered Cluster — the cozy-default BackupClass driver SSA-applies an ObjectStore (carrying destinationPath/endpointURL/credentials) and patches `spec.plugins` on the live Cluster when the first BackupJob runs. Consequence: plugin WAL archiving is NOT active until that first BackupJob fires; WAL accumulates on the PVC in the meantime, so fire an ad-hoc BackupJob immediately after enabling the flag on existing releases. Use together with the platform `cozy-default` BackupClass — tenants do not need to fill `s3AccessKey`/`s3SecretKey` or `destinationPath`/`endpointURL`. The destination path automatically scopes to `s3://cozy-backups///`. -## @field {string} [schedule] - Legacy. Cron schedule (CNPG 6-field format) for the chart-emitted ScheduledBackup. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time. +## @field {string} [schedule] - Legacy. Cron schedule for the chart-emitted ScheduledBackup, either a standard five-field cron such as `0 2 * * *` for every day at 02:00, or six fields with leading seconds such as `0 0 2 * * *` for the same time. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time. ## @field {string} [retentionPolicy] - Retention policy (e.g. "30d"). ## @field {string} [destinationPath] - DEPRECATED. Per-tenant S3 configuration is superseded by the platform-managed `cozy-default` BackupClass and the `cozy-backups` system bucket. Leave empty for new installations; the BackupClass driver picks up the system-managed coordinates. Kept for in-place upgrade compatibility. ## @field {string} [endpointURL] - DEPRECATED. See `destinationPath`. diff --git a/packages/system/postgres-rd/cozyrds/postgres.yaml b/packages/system/postgres-rd/cozyrds/postgres.yaml index 39adc0f58..aeea68e12 100644 --- a/packages/system/postgres-rd/cozyrds/postgres.yaml +++ b/packages/system/postgres-rd/cozyrds/postgres.yaml @@ -30,7 +30,7 @@ spec: singular: postgres plural: postgreses openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Postgres replicas. Under active autoscaling this is the resting/seed count, not the live count — KEDA drives the live count via `/scale`. Disabling autoscaling does NOT automatically bring the live count back to `replicas`: the client-side merge writes `instances` only when the rendered value changes, so a cluster KEDA grew past `replicas` stays there with no autoscaler left to reduce it — lower `replicas` to the count you want and it rebases down (shedding the surplus replicas + PVCs). Lowering it below the live count, or disabling/dry-running while it sits below the live count, sheds in one step past the scaleDown pacing; stage it to the live count first (suspend the HelmRelease and `kubectl scale` if needed) to avoid a transient. Must stay greater than `quorum.maxSyncReplicas`.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each PostgreSQL replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster.","type":"boolean","default":false},"version":{"description":"PostgreSQL major version to deploy","type":"string","default":"v18","enum":["v18","v17","v16","v15","v14","v13"]},"flavor":{"description":"PostgreSQL image flavor, fixed when the cluster is created. `postgresql` uses the CloudNativePG PostgreSQL image. `postgis` uses the CloudNativePG PostGIS image, which adds postgis, postgis_raster, postgis_topology, postgis_sfcgal, postgis_tiger_geocoder and address_standardizer; enable them per database in `databases..extensions`. Changing the flavor of an existing cluster is refused, and a backup restores only into a cluster of the flavor it was taken from: to change flavor, create a new cluster and move the data with a logical dump (pg_dump / pg_restore).","type":"string","default":"postgresql","enum":["postgresql","postgis"]},"tls":{"description":"TLS configuration for server connections.","type":"object","default":{},"properties":{"enabled":{"description":"Tri-state switch controlling whether the chart injects the external hostnames into the operator-managed CNPG cert via spec.certificates.serverAltDNSNames. When omitted, the chart injects them if `external: true` and skips them otherwise. Set explicitly to `true` to inject regardless of `external` (no-op when `external: false` since there is no external hostname to add). Set to `false` to skip injection. Note that CNPG keeps its built-in TLS on the wire regardless of this flag — this toggle only controls the chart-side SAN injection; to disable PostgreSQL TLS entirely set `postgresql.parameters.ssl = \"off\"` at the CNPG layer.","type":"boolean"},"required":{"description":"Tri-state switch controlling whether the server refuses sessions that are not encrypted. When omitted, it is on if `external: true` and off otherwise. When on, the chart renders `spec.postgresql.pg_hba` with `hostssl all all all scram-sha-256` (`md5` on PostgreSQL 13, matching CloudNativePG's default rule) followed by `host all all all reject`, so a client that connects with `sslmode=disable` is refused instead of getting a plaintext session; in-cluster clients are held to the same rule, since the pod network cannot be told apart from an external client. Set to `false` to keep accepting plaintext sessions on an external instance, or to `true` to require TLS on an in-cluster one. Rendering fails if `postgresql.parameters.ssl` is off while this is in effect.","type":"boolean"}}},"postgresql":{"description":"PostgreSQL server configuration.","type":"object","default":{},"properties":{"parameters":{"description":"PostgreSQL server parameters. Values may be strings or integers; integers are coerced to strings by the template (e.g. both `max_connections: 100` and `max_connections: \"100\"` are accepted). BLOCKED (enable arbitrary code execution): archive_command, restore_command, ssl_passphrase_command, archive_cleanup_command, recovery_end_command, dynamic_library_path, local_preload_libraries, session_preload_libraries, shared_preload_libraries. Do NOT override CloudNativePG-managed parameters: archive_mode, primary_conninfo, wal_level, max_replication_slots.","type":"object","default":{"max_connections":"100"},"additionalProperties":{"anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}},"sharedPreloadLibraries":{"description":"Libraries loaded at server start, restricted to an allowlist of modules already shipped in the PostgreSQL image. CloudNativePG also preloads `pg_stat_statements`, `auto_explain` and `pgaudit` by itself whenever `parameters` sets a key under their prefix (e.g. `pg_stat_statements.track`), so removing one of them from this list does not unload it while such a key is set; `pg_prewarm` is preloaded only through this list. Changing the list restarts PostgreSQL on every instance, the primary included, so writes pause briefly.","type":"array","items":{"type":"string","enum":["pg_stat_statements","auto_explain","pgaudit","pg_prewarm"]}}}},"quorum":{"description":"Quorum configuration for synchronous replication.","type":"object","default":{},"required":["maxSyncReplicas","minSyncReplicas"],"properties":{"maxSyncReplicas":{"description":"Maximum number of synchronous replicas allowed (must be less than total replicas). Under autoscaling it also sets the scale-down floor (the effective minimum is `max(minReplicas, maxSyncReplicas+1, 2)`); the chart fails to render if that floor exceeds `autoscaling.maxReplicas`.","type":"integer","default":0},"minSyncReplicas":{"description":"Minimum number of synchronous replicas required for commit.","type":"integer","default":0}}},"autoscaling":{"description":"Read-replica autoscaling configuration.","type":"object","default":{},"required":["dryRun","enabled","maxReplicas","maxReplicationLagSeconds","metric","minReplicas","serverAddress","target","transition"],"properties":{"dryRun":{"description":"Recommendation mode: keep the static count and render the ScaledObject with both scaling directions paused, so KEDA keeps the HPA and keeps serving its trigger metric but actuates neither way. Note both scaling policies are disabled, so the HPA's desiredReplicas is frozen to the current count - read the recommendation from the matching read-load dashboard panel for your metric (connections or CPU): desired = 1 + ceil(read load / target), clamped to [effectiveMin..maxReplicas] where effectiveMin = max(minReplicas, maxSyncReplicas+1, 2), not from the desired-replicas line. Like disabling, flipping a LIVE autoscaled cluster into dryRun re-renders `instances: replicas` (the raw static value, which can be below the autoscaling floor); if `replicas` is below the live count, stage it to the live count first (see the enablement note) or the cluster sheds down to `replicas`. No shed occurs when the rendered value is unchanged (replicas already equals the active seed) — client-side merge is then a no-op and the live count is preserved. Permanent, unlike `transition`.","type":"boolean","default":false},"enabled":{"description":"Enable horizontal autoscaling of read replicas. Requires the platform `keda` package to be enabled by an administrator first; the chart fails to render (rather than break the release) if KEDA is not installed. Prerequisite: the queried vmselect must hold BOTH the CNPG metrics (cnpg_backends_total / container CPU) AND kube_pod_labels (kube-state-metrics). This holds for a tenant on the shared root monitoring stack (`monitoring: false`, both land in tenant-root). A tenant running its OWN Monitoring app (`monitoring: true`) has the CNPG metrics but not kube_pod_labels (KSM is root-only), so it must set `serverAddress` to a vmselect carrying both series or autoscaling never actuates.","type":"boolean","default":false},"maxReplicas":{"description":"Maximum total instances. Must be >= minReplicas and >= the synchronous-quorum floor (maxSyncReplicas+1); the chart fails to render (rather than silently run an unbounded instance count) if the effective floor exceeds it - raise maxReplicas. HAZARD: lowering maxReplicas below the live count makes the HPA cut straight to the new maximum in one step (the current>max branch bypasses the scale-down pacing), shedding the surplus replicas + PVCs at once; to lower it deliberately, suspend the HelmRelease and pin the live count with `kubectl scale` first.","type":"integer","default":6},"maxReplicationLagSeconds":{"description":"Freeze scaling while replication lag exceeds this (seconds) and the primary is writing. 0 disables the brake. Default 0: the freeze branch has only been validated on a live cluster in its non-braking (pass-through) form, so the lag brake is opt-in until the braking path is exercised under load; set a positive value (e.g. 30) to enable it.","type":"integer","default":0},"metric":{"description":"Read-load signal to scale on.","type":"string","default":"ReadConnections","enum":["ReadConnections","ReadCPUUtilization"]},"minReplicas":{"description":"Minimum total instances; raised to the synchronous-quorum floor (`maxSyncReplicas + 1`) and to 2 when either is higher.","type":"integer","default":2},"serverAddress":{"description":"Override the Prometheus/vmselect URL the ScaledObject queries. Empty (default) derives `http://vmselect-shortterm..svc:8481/select/0/prometheus`. Set it when the derived vmselect does not hold both the CNPG metrics and kube_pod_labels (e.g. `monitoring: true` tenants, or a non-default `metricsStorages` name). SSRF constraint: the host must be an in-cluster Service DNS name (`..svc[.]`) whose namespace is within the tenant's own subtree (its namespace, its parents, or `tenant-root`) — the cluster-privileged KEDA operator fetches this URL every poll, so it is refused a metadata endpoint, an external host, or a sibling/system namespace. Residual surface (accepted trade-off): `tenant-root` is always allowed because the derived default points there, so a tenant can still aim the operator at any Service in `tenant-root` on any port/path (a blind in-cluster reachability probe — the operator reads no response body back to the tenant); the egress NetworkPolicy does not narrow that.","type":"string","default":""},"target":{"description":"Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold.","type":"integer","default":150},"transition":{"description":"Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject in both directions, so KEDA stands up with its HPA present but not actuating. Flip to false (phase 2) to hand the runtime count to KEDA. Because the both-direction pause keeps the HPA (not the HPA-deleting full pause), KEDA is genuinely warm at phase 2 — no HPA recreate — and because the active seed is max(replicas, effectiveMin), phase 2 re-renders the same instance count phase 1 did (when `replicas` is staged at/above the floor), so the flip is a merge no-op with no dip. The HPA's scaleUp stabilization window still smooths the first actuation.","type":"boolean","default":false}}},"users":{"description":"Users configuration map. Passwords are always auto-generated and stored in the `-credentials` Secret; they cannot be set from values — read a user's password from that Secret. A `password` left over in values from before the field was removed is ignored by the render and draws an admission warning, but it is not inert on an upgraded release: the chart preserves whatever password is already in the Secret, which on the first upgrade is the value that was set before removal, so that value stays the live credential until it is rotated (dedicated rotation is tracked in cozystack/community#72). The live password lives only in the Secret.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"bypassRls":{"description":"Whether the user can bypass row-level security policies (BYPASSRLS). A role can only create other roles carrying BYPASSRLS if it holds the attribute itself, so this is required alongside `createRole` to provision a privileged service role.","type":"boolean"},"createRole":{"description":"Whether the user can create and manage roles (CREATEROLE). Requires PostgreSQL 16 or newer. Setting it back to `false` withdraws the attribute only: the roles the user created, and its ADMIN and SET memberships in them, stay. Once any user has held it, the chart no longer adopts an existing role it did not create, so a user or database declared later under the name of such a role fails the init job until a platform administrator drops that role.","type":"boolean"},"replication":{"description":"Whether the user has replication privileges.","type":"boolean"}}}},"databases":{"description":"Databases configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"extensions":{"description":"List of enabled PostgreSQL extensions.","type":"array","items":{"type":"string"}},"roles":{"description":"Roles assigned to users.","type":"object","properties":{"admin":{"description":"List of users with admin privileges.","type":"array","items":{"type":"string"}},"readonly":{"description":"List of users with read-only privileges.","type":"array","items":{"type":"string"}}}}}}},"backup":{"description":"Backup configuration.","type":"object","default":{},"required":["enabled"],"properties":{"destinationPath":{"description":"DEPRECATED. Per-tenant S3 configuration is superseded by the platform-managed `cozy-default` BackupClass and the `cozy-backups` system bucket. Leave empty for new installations; the BackupClass driver picks up the system-managed coordinates. Kept for in-place upgrade compatibility.","type":"string","default":"s3://bucket/path/to/folder/"},"enabled":{"description":"Enable regular backups.","type":"boolean","default":false},"endpointCA":{"description":"DEPRECATED. Pre-existing Secret with the CA bundle the barman-cloud plugin should trust when reaching a self-signed S3 endpoint. Used for both backup and bootstrap recovery in the legacy chart-managed flow.","type":"object","default":{},"properties":{"key":{"description":"Key within the Secret containing the CA bundle. Defaults to `ca.crt`.","type":"string","default":""},"name":{"description":"Name of the Secret in the application namespace. Empty means no endpointCA is emitted (the plugin uses the system trust store).","type":"string","default":""}}},"endpointURL":{"description":"DEPRECATED. See `destinationPath`.","type":"string","default":"http://minio-gateway-service:9000"},"retentionPolicy":{"description":"Retention policy (e.g. \"30d\").","type":"string","default":"30d"},"s3AccessKey":{"description":"DEPRECATED. Tenants no longer supply S3 keys; the system Bucket Secret is projected into the tenant namespace by the backup controller. Ignored when `s3CredentialsSecret.name` is set or `useSystemBucket` is true. The chart skips materialising `-s3-creds` whenever this field is empty so a default install does not leak placeholder credentials into the tenant namespace.","type":"string","default":""},"s3CredentialsSecret":{"description":"DEPRECATED. Pre-existing Secret with S3 credentials. Use the platform-managed `cozy-default` BackupClass instead. When set, the chart references this Secret directly (legacy chart-managed flow). The CNPG backup driver writes this field on restore so credentials never land in the CR `.spec`.","type":"object","default":{},"properties":{"accessKeyIDKey":{"description":"Key in the Secret holding the access key ID. Defaults to `AWS_ACCESS_KEY_ID`.","type":"string","default":""},"name":{"description":"Name of the Secret in the application namespace. Empty means the chart materialises `-s3-creds` from `s3AccessKey`/`s3SecretKey`.","type":"string","default":""},"secretAccessKeyKey":{"description":"Key in the Secret holding the secret access key. Defaults to `AWS_SECRET_ACCESS_KEY`.","type":"string","default":""}}},"s3SecretKey":{"description":"DEPRECATED. See `s3AccessKey`.","type":"string","default":""},"schedule":{"description":"Legacy. Cron schedule (CNPG 6-field format) for the chart-emitted ScheduledBackup. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time.","type":"string","default":""},"useSystemBucket":{"description":"Opt-in: when true, the chart-emitted `-s3-creds` Secret is skipped AND `spec.plugins` (plus the barman-cloud ObjectStore) is left UNSET in the chart-rendered Cluster — the cozy-default BackupClass driver SSA-applies an ObjectStore (carrying destinationPath/endpointURL/credentials) and patches `spec.plugins` on the live Cluster when the first BackupJob runs. Consequence: plugin WAL archiving is NOT active until that first BackupJob fires; WAL accumulates on the PVC in the meantime, so fire an ad-hoc BackupJob immediately after enabling the flag on existing releases. Use together with the platform `cozy-default` BackupClass — tenants do not need to fill `s3AccessKey`/`s3SecretKey` or `destinationPath`/`endpointURL`. The destination path automatically scopes to `s3://cozy-backups///`.","type":"boolean","default":false}}},"bootstrap":{"description":"Bootstrap configuration.","type":"object","default":{},"required":["enabled","oldName"],"properties":{"backupID":{"description":"Barman ID of the base backup to recover from, passed as `recoveryTarget.backupID`. Empty lets the barman-cloud plugin pick one: the newest backup when `recoveryTime` is empty, otherwise the newest one ending at or before it, on any timeline. The CNPG backup driver sets this on restore.","type":"string","default":""},"enabled":{"description":"Whether to restore from a backup.","type":"boolean","default":false},"newServerName":{"description":"WAL-archive server name (S3 path prefix) the RESTORED cluster writes to. Must differ from `serverName` (the recovery source) so a restored cluster archives to a fresh, empty prefix and barman-cloud-check-wal-archive passes while it recovers from the source's prefix. The CNPG backup driver sets this on restore; empty means the cluster archives under its own name.","type":"string","default":""},"oldName":{"description":"Previous cluster name before deletion.","type":"string","default":""},"recoveryTime":{"description":"Timestamp (RFC3339) for point-in-time recovery; empty means latest.","type":"string","default":""},"serverName":{"description":"Server name (S3 path prefix) used by the original cluster when writing backups; passed to the barman-cloud plugin via `externalClusters[].plugin.parameters.serverName`. Defaults to `bootstrap.oldName`. Set this only when the original cluster wrote backups under an explicit server name that differed from its Kubernetes resource name.","type":"string","default":""}}}}} + {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Postgres replicas. Under active autoscaling this is the resting/seed count, not the live count — KEDA drives the live count via `/scale`. Disabling autoscaling does NOT automatically bring the live count back to `replicas`: the client-side merge writes `instances` only when the rendered value changes, so a cluster KEDA grew past `replicas` stays there with no autoscaler left to reduce it — lower `replicas` to the count you want and it rebases down (shedding the surplus replicas + PVCs). Lowering it below the live count, or disabling/dry-running while it sits below the live count, sheds in one step past the scaleDown pacing; stage it to the live count first (suspend the HelmRelease and `kubectl scale` if needed) to avoid a transient. Must stay greater than `quorum.maxSyncReplicas`.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each PostgreSQL replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster.","type":"boolean","default":false},"version":{"description":"PostgreSQL major version to deploy","type":"string","default":"v18","enum":["v18","v17","v16","v15","v14","v13"]},"flavor":{"description":"PostgreSQL image flavor, fixed when the cluster is created. `postgresql` uses the CloudNativePG PostgreSQL image. `postgis` uses the CloudNativePG PostGIS image, which adds postgis, postgis_raster, postgis_topology, postgis_sfcgal, postgis_tiger_geocoder and address_standardizer; enable them per database in `databases..extensions`. Changing the flavor of an existing cluster is refused, and a backup restores only into a cluster of the flavor it was taken from: to change flavor, create a new cluster and move the data with a logical dump (pg_dump / pg_restore).","type":"string","default":"postgresql","enum":["postgresql","postgis"]},"tls":{"description":"TLS configuration for server connections.","type":"object","default":{},"properties":{"enabled":{"description":"Tri-state switch controlling whether the chart injects the external hostnames into the operator-managed CNPG cert via spec.certificates.serverAltDNSNames. When omitted, the chart injects them if `external: true` and skips them otherwise. Set explicitly to `true` to inject regardless of `external` (no-op when `external: false` since there is no external hostname to add). Set to `false` to skip injection. Note that CNPG keeps its built-in TLS on the wire regardless of this flag — this toggle only controls the chart-side SAN injection; to disable PostgreSQL TLS entirely set `postgresql.parameters.ssl = \"off\"` at the CNPG layer.","type":"boolean"},"required":{"description":"Tri-state switch controlling whether the server refuses sessions that are not encrypted. When omitted, it is on if `external: true` and off otherwise. When on, the chart renders `spec.postgresql.pg_hba` with `hostssl all all all scram-sha-256` (`md5` on PostgreSQL 13, matching CloudNativePG's default rule) followed by `host all all all reject`, so a client that connects with `sslmode=disable` is refused instead of getting a plaintext session; in-cluster clients are held to the same rule, since the pod network cannot be told apart from an external client. Set to `false` to keep accepting plaintext sessions on an external instance, or to `true` to require TLS on an in-cluster one. Rendering fails if `postgresql.parameters.ssl` is off while this is in effect.","type":"boolean"}}},"postgresql":{"description":"PostgreSQL server configuration.","type":"object","default":{},"properties":{"parameters":{"description":"PostgreSQL server parameters. Values may be strings or integers; integers are coerced to strings by the template (e.g. both `max_connections: 100` and `max_connections: \"100\"` are accepted). BLOCKED (enable arbitrary code execution): archive_command, restore_command, ssl_passphrase_command, archive_cleanup_command, recovery_end_command, dynamic_library_path, local_preload_libraries, session_preload_libraries, shared_preload_libraries. Do NOT override CloudNativePG-managed parameters: archive_mode, primary_conninfo, wal_level, max_replication_slots.","type":"object","default":{"max_connections":"100"},"additionalProperties":{"anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}},"sharedPreloadLibraries":{"description":"Libraries loaded at server start, restricted to an allowlist of modules already shipped in the PostgreSQL image. CloudNativePG also preloads `pg_stat_statements`, `auto_explain` and `pgaudit` by itself whenever `parameters` sets a key under their prefix (e.g. `pg_stat_statements.track`), so removing one of them from this list does not unload it while such a key is set; `pg_prewarm` is preloaded only through this list. Changing the list restarts PostgreSQL on every instance, the primary included, so writes pause briefly.","type":"array","items":{"type":"string","enum":["pg_stat_statements","auto_explain","pgaudit","pg_prewarm"]}}}},"quorum":{"description":"Quorum configuration for synchronous replication.","type":"object","default":{},"required":["maxSyncReplicas","minSyncReplicas"],"properties":{"maxSyncReplicas":{"description":"Maximum number of synchronous replicas allowed (must be less than total replicas). Under autoscaling it also sets the scale-down floor (the effective minimum is `max(minReplicas, maxSyncReplicas+1, 2)`); the chart fails to render if that floor exceeds `autoscaling.maxReplicas`.","type":"integer","default":0},"minSyncReplicas":{"description":"Minimum number of synchronous replicas required for commit.","type":"integer","default":0}}},"autoscaling":{"description":"Read-replica autoscaling configuration.","type":"object","default":{},"required":["dryRun","enabled","maxReplicas","maxReplicationLagSeconds","metric","minReplicas","serverAddress","target","transition"],"properties":{"dryRun":{"description":"Recommendation mode: keep the static count and render the ScaledObject with both scaling directions paused, so KEDA keeps the HPA and keeps serving its trigger metric but actuates neither way. Note both scaling policies are disabled, so the HPA's desiredReplicas is frozen to the current count - read the recommendation from the matching read-load dashboard panel for your metric (connections or CPU): desired = 1 + ceil(read load / target), clamped to [effectiveMin..maxReplicas] where effectiveMin = max(minReplicas, maxSyncReplicas+1, 2), not from the desired-replicas line. Like disabling, flipping a LIVE autoscaled cluster into dryRun re-renders `instances: replicas` (the raw static value, which can be below the autoscaling floor); if `replicas` is below the live count, stage it to the live count first (see the enablement note) or the cluster sheds down to `replicas`. No shed occurs when the rendered value is unchanged (replicas already equals the active seed) — client-side merge is then a no-op and the live count is preserved. Permanent, unlike `transition`.","type":"boolean","default":false},"enabled":{"description":"Enable horizontal autoscaling of read replicas. Requires the platform `keda` package to be enabled by an administrator first; the chart fails to render (rather than break the release) if KEDA is not installed. Prerequisite: the queried vmselect must hold BOTH the CNPG metrics (cnpg_backends_total / container CPU) AND kube_pod_labels (kube-state-metrics). This holds for a tenant on the shared root monitoring stack (`monitoring: false`, both land in tenant-root). A tenant running its OWN Monitoring app (`monitoring: true`) has the CNPG metrics but not kube_pod_labels (KSM is root-only), so it must set `serverAddress` to a vmselect carrying both series or autoscaling never actuates.","type":"boolean","default":false},"maxReplicas":{"description":"Maximum total instances. Must be >= minReplicas and >= the synchronous-quorum floor (maxSyncReplicas+1); the chart fails to render (rather than silently run an unbounded instance count) if the effective floor exceeds it - raise maxReplicas. HAZARD: lowering maxReplicas below the live count makes the HPA cut straight to the new maximum in one step (the current>max branch bypasses the scale-down pacing), shedding the surplus replicas + PVCs at once; to lower it deliberately, suspend the HelmRelease and pin the live count with `kubectl scale` first.","type":"integer","default":6},"maxReplicationLagSeconds":{"description":"Freeze scaling while replication lag exceeds this (seconds) and the primary is writing. 0 disables the brake. Default 0: the freeze branch has only been validated on a live cluster in its non-braking (pass-through) form, so the lag brake is opt-in until the braking path is exercised under load; set a positive value (e.g. 30) to enable it.","type":"integer","default":0},"metric":{"description":"Read-load signal to scale on.","type":"string","default":"ReadConnections","enum":["ReadConnections","ReadCPUUtilization"]},"minReplicas":{"description":"Minimum total instances; raised to the synchronous-quorum floor (`maxSyncReplicas + 1`) and to 2 when either is higher.","type":"integer","default":2},"serverAddress":{"description":"Override the Prometheus/vmselect URL the ScaledObject queries. Empty (default) derives `http://vmselect-shortterm..svc:8481/select/0/prometheus`. Set it when the derived vmselect does not hold both the CNPG metrics and kube_pod_labels (e.g. `monitoring: true` tenants, or a non-default `metricsStorages` name). SSRF constraint: the host must be an in-cluster Service DNS name (`..svc[.]`) whose namespace is within the tenant's own subtree (its namespace, its parents, or `tenant-root`) — the cluster-privileged KEDA operator fetches this URL every poll, so it is refused a metadata endpoint, an external host, or a sibling/system namespace. Residual surface (accepted trade-off): `tenant-root` is always allowed because the derived default points there, so a tenant can still aim the operator at any Service in `tenant-root` on any port/path (a blind in-cluster reachability probe — the operator reads no response body back to the tenant); the egress NetworkPolicy does not narrow that.","type":"string","default":""},"target":{"description":"Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold.","type":"integer","default":150},"transition":{"description":"Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject in both directions, so KEDA stands up with its HPA present but not actuating. Flip to false (phase 2) to hand the runtime count to KEDA. Because the both-direction pause keeps the HPA (not the HPA-deleting full pause), KEDA is genuinely warm at phase 2 — no HPA recreate — and because the active seed is max(replicas, effectiveMin), phase 2 re-renders the same instance count phase 1 did (when `replicas` is staged at/above the floor), so the flip is a merge no-op with no dip. The HPA's scaleUp stabilization window still smooths the first actuation.","type":"boolean","default":false}}},"users":{"description":"Users configuration map. Passwords are always auto-generated and stored in the `-credentials` Secret; they cannot be set from values — read a user's password from that Secret. A `password` left over in values from before the field was removed is ignored by the render and draws an admission warning, but it is not inert on an upgraded release: the chart preserves whatever password is already in the Secret, which on the first upgrade is the value that was set before removal, so that value stays the live credential until it is rotated (dedicated rotation is tracked in cozystack/community#72). The live password lives only in the Secret.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"bypassRls":{"description":"Whether the user can bypass row-level security policies (BYPASSRLS). A role can only create other roles carrying BYPASSRLS if it holds the attribute itself, so this is required alongside `createRole` to provision a privileged service role.","type":"boolean"},"createRole":{"description":"Whether the user can create and manage roles (CREATEROLE). Requires PostgreSQL 16 or newer. Setting it back to `false` withdraws the attribute only: the roles the user created, and its ADMIN and SET memberships in them, stay. Once any user has held it, the chart no longer adopts an existing role it did not create, so a user or database declared later under the name of such a role fails the init job until a platform administrator drops that role.","type":"boolean"},"replication":{"description":"Whether the user has replication privileges.","type":"boolean"}}}},"databases":{"description":"Databases configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"extensions":{"description":"List of enabled PostgreSQL extensions.","type":"array","items":{"type":"string"}},"roles":{"description":"Roles assigned to users.","type":"object","properties":{"admin":{"description":"List of users with admin privileges.","type":"array","items":{"type":"string"}},"readonly":{"description":"List of users with read-only privileges.","type":"array","items":{"type":"string"}}}}}}},"backup":{"description":"Backup configuration.","type":"object","default":{},"required":["enabled"],"properties":{"destinationPath":{"description":"DEPRECATED. Per-tenant S3 configuration is superseded by the platform-managed `cozy-default` BackupClass and the `cozy-backups` system bucket. Leave empty for new installations; the BackupClass driver picks up the system-managed coordinates. Kept for in-place upgrade compatibility.","type":"string","default":"s3://bucket/path/to/folder/"},"enabled":{"description":"Enable regular backups.","type":"boolean","default":false},"endpointCA":{"description":"DEPRECATED. Pre-existing Secret with the CA bundle the barman-cloud plugin should trust when reaching a self-signed S3 endpoint. Used for both backup and bootstrap recovery in the legacy chart-managed flow.","type":"object","default":{},"properties":{"key":{"description":"Key within the Secret containing the CA bundle. Defaults to `ca.crt`.","type":"string","default":""},"name":{"description":"Name of the Secret in the application namespace. Empty means no endpointCA is emitted (the plugin uses the system trust store).","type":"string","default":""}}},"endpointURL":{"description":"DEPRECATED. See `destinationPath`.","type":"string","default":"http://minio-gateway-service:9000"},"retentionPolicy":{"description":"Retention policy (e.g. \"30d\").","type":"string","default":"30d"},"s3AccessKey":{"description":"DEPRECATED. Tenants no longer supply S3 keys; the system Bucket Secret is projected into the tenant namespace by the backup controller. Ignored when `s3CredentialsSecret.name` is set or `useSystemBucket` is true. The chart skips materialising `-s3-creds` whenever this field is empty so a default install does not leak placeholder credentials into the tenant namespace.","type":"string","default":""},"s3CredentialsSecret":{"description":"DEPRECATED. Pre-existing Secret with S3 credentials. Use the platform-managed `cozy-default` BackupClass instead. When set, the chart references this Secret directly (legacy chart-managed flow). The CNPG backup driver writes this field on restore so credentials never land in the CR `.spec`.","type":"object","default":{},"properties":{"accessKeyIDKey":{"description":"Key in the Secret holding the access key ID. Defaults to `AWS_ACCESS_KEY_ID`.","type":"string","default":""},"name":{"description":"Name of the Secret in the application namespace. Empty means the chart materialises `-s3-creds` from `s3AccessKey`/`s3SecretKey`.","type":"string","default":""},"secretAccessKeyKey":{"description":"Key in the Secret holding the secret access key. Defaults to `AWS_SECRET_ACCESS_KEY`.","type":"string","default":""}}},"s3SecretKey":{"description":"DEPRECATED. See `s3AccessKey`.","type":"string","default":""},"schedule":{"description":"Legacy. Cron schedule for the chart-emitted ScheduledBackup, either a standard five-field cron such as `0 2 * * *` for every day at 02:00, or six fields with leading seconds such as `0 0 2 * * *` for the same time. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time.","type":"string","default":""},"useSystemBucket":{"description":"Opt-in: when true, the chart-emitted `-s3-creds` Secret is skipped AND `spec.plugins` (plus the barman-cloud ObjectStore) is left UNSET in the chart-rendered Cluster — the cozy-default BackupClass driver SSA-applies an ObjectStore (carrying destinationPath/endpointURL/credentials) and patches `spec.plugins` on the live Cluster when the first BackupJob runs. Consequence: plugin WAL archiving is NOT active until that first BackupJob fires; WAL accumulates on the PVC in the meantime, so fire an ad-hoc BackupJob immediately after enabling the flag on existing releases. Use together with the platform `cozy-default` BackupClass — tenants do not need to fill `s3AccessKey`/`s3SecretKey` or `destinationPath`/`endpointURL`. The destination path automatically scopes to `s3://cozy-backups///`.","type":"boolean","default":false}}},"bootstrap":{"description":"Bootstrap configuration.","type":"object","default":{},"required":["enabled","oldName"],"properties":{"backupID":{"description":"Barman ID of the base backup to recover from, passed as `recoveryTarget.backupID`. Empty lets the barman-cloud plugin pick one: the newest backup when `recoveryTime` is empty, otherwise the newest one ending at or before it, on any timeline. The CNPG backup driver sets this on restore.","type":"string","default":""},"enabled":{"description":"Whether to restore from a backup.","type":"boolean","default":false},"newServerName":{"description":"WAL-archive server name (S3 path prefix) the RESTORED cluster writes to. Must differ from `serverName` (the recovery source) so a restored cluster archives to a fresh, empty prefix and barman-cloud-check-wal-archive passes while it recovers from the source's prefix. The CNPG backup driver sets this on restore; empty means the cluster archives under its own name.","type":"string","default":""},"oldName":{"description":"Previous cluster name before deletion.","type":"string","default":""},"recoveryTime":{"description":"Timestamp (RFC3339) for point-in-time recovery; empty means latest.","type":"string","default":""},"serverName":{"description":"Server name (S3 path prefix) used by the original cluster when writing backups; passed to the barman-cloud plugin via `externalClusters[].plugin.parameters.serverName`. Defaults to `bootstrap.oldName`. Set this only when the original cluster wrote backups under an explicit server name that differed from its Kubernetes resource name.","type":"string","default":""}}}}} release: prefix: postgres- labels: From c70f83b375caeaf609884420ecc0ca769184bd74 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 26 Oct 2026 22:11:09 +0100 Subject: [PATCH 024/518] fix(mariadb-operator): derive the cluster name from the platform domain The package pinned clusterName to cozy.local, so on a cluster installed with any other networking.clusterDomain the operator resolved its own MariaDB pods as ..svc.cozy.local and got NXDOMAIN. Replication between replicas was never configured, and User, Database and Grant resources failed to reconcile, with the operator only retrying at a verbose log level. clusterName is a vendored subchart value, out of reach of the _cluster lookup that package templates use, so the platform now carries it under the subchart key in the shared values Secret, the same route KEDA and the OpenSearch operator already take. The literal in the package stays as the default for a render without that Secret. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- packages/core/platform/templates/apps.yaml | 8 ++++ ...ps_mariadb_operator_cluster_name_test.yaml | 33 ++++++++++++++ .../tests/cluster_name_test.yaml | 43 +++++++++++++++++++ packages/system/mariadb-operator/values.yaml | 2 + 4 files changed, 86 insertions(+) create mode 100644 packages/core/platform/tests/apps_mariadb_operator_cluster_name_test.yaml create mode 100644 packages/system/mariadb-operator/tests/cluster_name_test.yaml diff --git a/packages/core/platform/templates/apps.yaml b/packages/core/platform/templates/apps.yaml index 030cfe8b2..4a0f0ae89 100644 --- a/packages/core/platform/templates/apps.yaml +++ b/packages/core/platform/templates/apps.yaml @@ -33,6 +33,14 @@ stringData: opensearch-operator: manager: dnsBase: {{ .Values.networking.clusterDomain | quote }} + {{- /* + The operator resolves its own MariaDB pods, and the webhook certificate + gets its SAN, from CLUSTER_NAME, which the vendored subchart reads from + clusterName. Same subchart-key route as above; a package rendered + without this Secret falls back to the literal in its values.yaml. + */}} + mariadb-operator: + clusterName: {{ .Values.networking.clusterDomain | quote }} _cluster: root-host: {{ $rootHost | quote }} bundle-name: {{ .Values.bundles.system.variant | quote }} diff --git a/packages/core/platform/tests/apps_mariadb_operator_cluster_name_test.yaml b/packages/core/platform/tests/apps_mariadb_operator_cluster_name_test.yaml new file mode 100644 index 000000000..6cd9426ba --- /dev/null +++ b/packages/core/platform/tests/apps_mariadb_operator_cluster_name_test.yaml @@ -0,0 +1,33 @@ +suite: apps.yaml networking.clusterDomain reaches the mariadb-operator cluster name + +templates: + - templates/apps.yaml + +release: + name: cozy-platform + namespace: cozy-system + +# The operator dials ..svc. to set up replication and +# to reconcile User, Database and Grant, and the vendored subchart fills +# CLUSTER_NAME from clusterName. A cluster whose domain is not cozy.local got +# NXDOMAIN on every such lookup while the literal was pinned in the package. +# Every pattern anchors clusterName to its mariadb-operator parent, so a sibling +# block carrying the same domain cannot satisfy the assertion on its own. + +tests: + - it: carries the cozy.local default under the subchart key + asserts: + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^mariadb-operator:\n clusterName: "cozy\.local"$' + + - it: tracks a non-default networking.clusterDomain + set: + networking.clusterDomain: example.test + asserts: + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^mariadb-operator:\n clusterName: "example\.test"$' + - notMatchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^mariadb-operator:\n clusterName: "cozy\.local"$' diff --git a/packages/system/mariadb-operator/tests/cluster_name_test.yaml b/packages/system/mariadb-operator/tests/cluster_name_test.yaml new file mode 100644 index 000000000..45d762727 --- /dev/null +++ b/packages/system/mariadb-operator/tests/cluster_name_test.yaml @@ -0,0 +1,43 @@ +suite: mariadb-operator package resolves its pods in the platform cluster domain +release: + name: mariadb-operator + namespace: cozy-mariadb-operator +# core/platform hands the cluster domain to this package as +# mariadb-operator.clusterName in the shared values Secret. Guard that the +# vendored chart still threads that key into what the operator and the webhook +# certificate use, so a re-vendor that renames it goes red here. +tests: + - it: sets the operator CLUSTER_NAME from the injected domain + templates: + - charts/mariadb-operator/templates/operator/deployment.yaml + set: + mariadb-operator.clusterName: example.test + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: CLUSTER_NAME + value: example.test + + - it: issues the webhook certificate for the injected domain + templates: + - charts/mariadb-operator/templates/webhook/certificate.yaml + documentSelector: + path: kind + value: Certificate + set: + mariadb-operator.clusterName: example.test + asserts: + - contains: + path: spec.dnsNames + content: mariadb-operator-webhook.cozy-mariadb-operator.svc.example.test + + - it: keeps cozy.local as the standalone default + templates: + - charts/mariadb-operator/templates/operator/deployment.yaml + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: CLUSTER_NAME + value: cozy.local diff --git a/packages/system/mariadb-operator/values.yaml b/packages/system/mariadb-operator/values.yaml index 2728c084e..285975440 100644 --- a/packages/system/mariadb-operator/values.yaml +++ b/packages/system/mariadb-operator/values.yaml @@ -1,4 +1,6 @@ mariadb-operator: + # Fallback only: core/platform injects networking.clusterDomain into this key + # through the shared cozystack-values Secret (see its apps.yaml). clusterName: cozy.local metrics: enabled: true From 93faa7a8620da8d00a783f40ceb9090ad1e5c265 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 26 Oct 2026 22:18:50 +0100 Subject: [PATCH 025/518] fix(lineage, kubernetes): resolve CAPK node VMs to their cluster The Cluster API KubeVirt provider creates a node's VirtualMachine without an ownerReference and links it to its KubevirtMachine through the capk.cluster.x-k8s.io/kubevirt-machine-{name,namespace} labels only. The ownership walk therefore stopped at the VM, found no ancestor, and the lineage webhook left every object below it unmanaged: the virt-launcher pods of worker nodes never got internal.cozystack.io/managed-by-cozystack, while still carrying the application lineage labels copied from the machine template. Anything that relies on the webhook having resolved those pods misses them. The securitygroup-controller caches only managed pods, so a SecurityGroup attached to a Kubernetes application labelled the control-plane pods and never the nodes. Follow the CAPK labels from the VM to its KubevirtMachine, whose ownerReferences reach the CAPI Cluster and from there the cluster's HelmRelease. Only a KubevirtMachine in the VM's own namespace is followed: CAPK can target an external infrastructure cluster, and the walk must never attribute an object across namespaces. Signed-off-by: Mattia Eleuteri Assisted-by: LLM --- pkg/lineage/capk_test.go | 101 +++++++++++++++++++++++++++++++++++++++ pkg/lineage/lineage.go | 38 +++++++++++++++ 2 files changed, 139 insertions(+) create mode 100644 pkg/lineage/capk_test.go diff --git a/pkg/lineage/capk_test.go b/pkg/lineage/capk_test.go new file mode 100644 index 000000000..606647eed --- /dev/null +++ b/pkg/lineage/capk_test.go @@ -0,0 +1,101 @@ +package lineage + +import ( + "context" + "testing" + + helmv2 "github.com/fluxcd/helm-controller/api/v2" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + dynamicfake "k8s.io/client-go/dynamic/fake" +) + +const capkNamespace = "tenant-demo" + +// kubernetesAppMapper maps the cluster's HelmRelease to its Kubernetes +// application, as the webhook's ApplicationDefinition mapper does. +type kubernetesAppMapper struct{} + +func (kubernetesAppMapper) Map(*helmv2.HelmRelease) (string, string, string, error) { + return "apps.cozystack.io/v1alpha1", "Kubernetes", "kubernetes-", nil +} + +func capkObject(apiVersion, kind, name string, labels map[string]string, owner *unstructured.Unstructured) *unstructured.Unstructured { + o := &unstructured.Unstructured{} + o.SetAPIVersion(apiVersion) + o.SetKind(kind) + o.SetNamespace(capkNamespace) + o.SetName(name) + o.SetLabels(labels) + if owner != nil { + o.SetOwnerReferences([]metav1.OwnerReference{{APIVersion: owner.GetAPIVersion(), Kind: owner.GetKind(), Name: owner.GetName()}}) + } + return o +} + +// capkNodeChain builds the objects behind the virt-launcher pod of a +// Kubernetes worker node, as observed on a live cluster: the pod is owned by +// its VMI and the VMI by its VM, but CAPK leaves the VM without an +// ownerReference and points at its KubevirtMachine through labels only. From +// the KubevirtMachine up, ownerReferences run to the CAPI Cluster, which the +// cluster's HelmRelease labels. +func capkNodeChain(vmLabels map[string]string) (*unstructured.Unstructured, []runtime.Object, meta.RESTMapper) { + app := capkObject("apps.cozystack.io/v1alpha1", "Kubernetes", "cluster-a", nil, nil) + hr := capkObject(HRAPIVersion, HRKind, "kubernetes-cluster-a", nil, nil) + cluster := capkObject("cluster.x-k8s.io/v1beta1", "Cluster", "kubernetes-cluster-a", map[string]string{HRLabel: "kubernetes-cluster-a"}, nil) + md := capkObject("cluster.x-k8s.io/v1beta1", "MachineDeployment", "kubernetes-cluster-a-md0", nil, cluster) + ms := capkObject("cluster.x-k8s.io/v1beta1", "MachineSet", "kubernetes-cluster-a-md0-x", nil, md) + machine := capkObject("cluster.x-k8s.io/v1beta1", "Machine", "kubernetes-cluster-a-md0-x-y", nil, ms) + kvm := capkObject(KubevirtMachineAPIVersion, KubevirtMachineKind, "kubernetes-cluster-a-md0-x-y", nil, machine) + vm := capkObject(VMAPIVersion, VMKind, "kubernetes-cluster-a-md0-x-y", vmLabels, nil) + vmi := capkObject("kubevirt.io/v1", "VirtualMachineInstance", "kubernetes-cluster-a-md0-x-y", nil, vm) + pod := capkObject("v1", "Pod", "virt-launcher-kubernetes-cluster-a-md0-x-y-abcde", nil, vmi) + + objs := []runtime.Object{app, hr, cluster, md, ms, machine, kvm, vm, vmi} + mapper := meta.NewDefaultRESTMapper(nil) + for _, o := range objs { + mapper.Add(o.GetObjectKind().GroupVersionKind(), meta.RESTScopeNamespace) + } + mapper.Add(schema.GroupVersionKind{Version: "v1", Kind: "Pod"}, meta.RESTScopeNamespace) + return pod, objs, mapper +} + +func walkNodePod(t *testing.T, vmLabels map[string]string) []ObjectID { + t.Helper() + pod, objs, mapper := capkNodeChain(vmLabels) + dyn := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme(), objs...) + return WalkOwnershipGraph(context.Background(), dyn, mapper, kubernetesAppMapper{}, pod) +} + +func TestWalkResolvesCAPKNodeVMToKubernetesApp(t *testing.T) { + got := walkNodePod(t, map[string]string{ + KubevirtMachineNameLabel: "kubernetes-cluster-a-md0-x-y", + KubevirtMachineNamespaceLabel: capkNamespace, + }) + want := ObjectID{APIVersion: "apps.cozystack.io/v1alpha1", Kind: "Kubernetes", Namespace: capkNamespace, Name: "cluster-a"} + if len(got) != 1 || got[0] != want { + t.Fatalf("WalkOwnershipGraph(node pod) = %+v, want [%+v]", got, want) + } +} + +func TestWalkIgnoresKubevirtMachineOfAnotherNamespace(t *testing.T) { + // A VM pointing at a KubevirtMachine outside its own namespace (an external + // infrastructure cluster, or a forged label) must not be resolved through it. + got := walkNodePod(t, map[string]string{ + KubevirtMachineNameLabel: "kubernetes-cluster-a-md0-x-y", + KubevirtMachineNamespaceLabel: "tenant-other", + }) + if len(got) != 0 { + t.Fatalf("WalkOwnershipGraph(node pod) = %+v, want no ancestors", got) + } +} + +func TestWalkStopsAtUnlabelledVM(t *testing.T) { + // A standalone VM with no CAPK labels and no ownerReference has no ancestors. + if got := walkNodePod(t, nil); len(got) != 0 { + t.Fatalf("WalkOwnershipGraph(pod of standalone VM) = %+v, want no ancestors", got) + } +} diff --git a/pkg/lineage/lineage.go b/pkg/lineage/lineage.go index e1f050052..1546df647 100644 --- a/pkg/lineage/lineage.go +++ b/pkg/lineage/lineage.go @@ -20,6 +20,16 @@ const ( HRAPIVersion = "helm.toolkit.fluxcd.io/v2" HRKind = "HelmRelease" HRLabel = "helm.toolkit.fluxcd.io/name" + + // KubeVirt VirtualMachines created by the Cluster API KubeVirt provider + // (CAPK) for cluster nodes carry no ownerReference: CAPK links them to + // their KubevirtMachine through these labels only. + VMAPIVersion = "kubevirt.io/v1" + VMKind = "VirtualMachine" + KubevirtMachineAPIVersion = "infrastructure.cluster.x-k8s.io/v1alpha1" + KubevirtMachineKind = "KubevirtMachine" + KubevirtMachineNameLabel = "capk.cluster.x-k8s.io/kubevirt-machine-name" + KubevirtMachineNamespaceLabel = "capk.cluster.x-k8s.io/kubevirt-machine-namespace" ) // AppMapper maps HelmRelease to application metadata. @@ -91,6 +101,18 @@ func WalkOwnershipGraph( return } + // A node VM of a Kubernetes cluster has no ownerReference, so without this + // step the walk would stop at it and never reach the cluster's application. + if name, ok := kubevirtMachineName(obj); ok { + ownerObj, err := getUnstructuredObject(ctx, client, mapper, state.cache, KubevirtMachineAPIVersion, KubevirtMachineKind, obj.GetNamespace(), name) + if err != nil { + l.Error(err, "couldn't get KubevirtMachine of VirtualMachine", "namespace", obj.GetNamespace(), "virtualMachine", obj.GetName(), "kubevirtMachine", name) + return + } + out = append(out, WalkOwnershipGraph(ctx, client, mapper, appMapper, ownerObj, state)...) + return + } + // I want "if err1 != nil go to next block, if err2 != nil, go to next block, etc semantics", // like an early return from a function, but if all checks succeed, I don't want to do the rest // of the function, so it's a `for { if err { break } if othererr { break } if allgood { return } @@ -138,6 +160,22 @@ func WalkOwnershipGraph( return } +// kubevirtMachineName returns the KubevirtMachine a CAPK-created +// VirtualMachine belongs to. It only follows a KubevirtMachine in the VM's own +// namespace: CAPK can also run against an external infrastructure cluster, +// where the KubevirtMachine lives elsewhere and is not ours to resolve. +func kubevirtMachineName(obj *unstructured.Unstructured) (string, bool) { + if obj.GetAPIVersion() != VMAPIVersion || obj.GetKind() != VMKind { + return "", false + } + labels := obj.GetLabels() + name := labels[KubevirtMachineNameLabel] + if name == "" || labels[KubevirtMachineNamespaceLabel] != obj.GetNamespace() { + return "", false + } + return name, true +} + func parseWalkMemory(memory []interface{}) (*walkState, error) { switch len(memory) { case 0: From dde255b7af102f32ea67cfcf0dcd67b15855d8cd Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Tue, 27 Oct 2026 08:44:43 +0100 Subject: [PATCH 026/518] fix(securitygroups): project a membership-only group to a policy Cilium accepts A SecurityGroup with attachments and no rules still needs its backing CiliumNetworkPolicy, which carries the attachments the controller reconciles. Its spec held only an empty ingress list: the CRD anyOf accepts that, but the agent's api.Rule.Sanitize counts entries, so Cilium 1.19 marks the policy Valid=False. Write one empty ingress rule, which allows nothing, and disable default deny in both directions so the rule does not isolate the member pods. The policy is then valid and still changes nothing in the datapath. The reverse projection recognises that exact spec, enableDefaultDeny included, so the group still reads back with no rules while a tenant-authored empty ingress rule keeps its default deny and stays visible. Fixes #5129 Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- pkg/apis/sdn/DESIGN.md | 16 ++- pkg/registry/sdn/securitygroup/cilium.go | 53 ++++--- pkg/registry/sdn/securitygroup/rest.go | 61 ++++++-- pkg/registry/sdn/securitygroup/rest_test.go | 145 +++++++++++++------- 4 files changed, 197 insertions(+), 78 deletions(-) diff --git a/pkg/apis/sdn/DESIGN.md b/pkg/apis/sdn/DESIGN.md index e7fa62f95..017da9d9a 100644 --- a/pkg/apis/sdn/DESIGN.md +++ b/pkg/apis/sdn/DESIGN.md @@ -130,7 +130,21 @@ spec: The securitygroup-controller stamps `securitygroup.sdn.cozystack.io/db: ""` onto the pods of the attached `Postgres/db` application, which is what makes the policy's `endpointSelector` select them. -A **membership-only SecurityGroup** — one with `attachments` but no `ingress`/`egress` rules — still projects to a backing CiliumNetworkPolicy, because the controller reconciles the policy (not the SecurityGroup) and the attachments live on it. The CiliumNetworkPolicy CRD requires at least one rule section to be present (its `spec` `anyOf` is `ingress | ingressDeny | egress | egressDeny`), so the projection always emits the `ingress` section, an empty list `ingress: []` when the group has no ingress rules. An empty list is the CRD's documented no-op ("if omitted or empty, this rule does not apply at ingress") and `enableDefaultDeny` defaults to `false` for a direction with no rules, so the policy is schema-valid yet datapath-inert: it neither allows nor denies anything, consistent with §7. The empty section is hidden from the SecurityGroup view on read, so a membership-only group round-trips with no `ingress`/`egress`. +A **membership-only SecurityGroup** — one with `attachments` but no `ingress`/`egress` rules — still projects to a backing CiliumNetworkPolicy, because the controller reconciles the policy (not the SecurityGroup) and the attachments live on it. Cilium rejects a policy none of whose rule sections has an entry: the CRD `anyOf` (`ingress | ingressDeny | egress | egressDeny`) is satisfied by an empty list, but the agent's `api.Rule.Sanitize` counts entries and marks such a policy `Valid=False`. The projection therefore writes one empty ingress rule, which allows nothing, together with `enableDefaultDeny: {ingress: false, egress: false}`, so that the rule does not put the member pods into default-deny either: + +```yaml +spec: + endpointSelector: + matchLabels: + securitygroup.sdn.cozystack.io/db: "" + ingress: + - {} + enableDefaultDeny: + ingress: false + egress: false +``` + +The policy is valid yet datapath-inert: it neither allows nor denies anything, consistent with §7. The override depends on the Cilium agent's `enable-non-default-deny-policies` option, which defaults to `true`; with it off, Cilium forces default deny on every policy. The placeholder is hidden from the SecurityGroup view on read, so a membership-only group round-trips with no `ingress`/`egress`. `enableDefaultDeny` is what tells it apart from a group whose single ingress rule is genuinely empty, which projects without the override and so keeps Cilium's ingress default deny. ## 6. RBAC diff --git a/pkg/registry/sdn/securitygroup/cilium.go b/pkg/registry/sdn/securitygroup/cilium.go index fa0ddb65a..906afcdcd 100644 --- a/pkg/registry/sdn/securitygroup/cilium.go +++ b/pkg/registry/sdn/securitygroup/cilium.go @@ -44,25 +44,24 @@ type CiliumNetworkPolicySpec struct { // it to the SecurityGroup's own membership label. EndpointSelector metav1.LabelSelector `json:"endpointSelector,omitempty"` - // Ingress is the list of allowed inbound traffic rules. It is always - // serialized (an empty list, not omitted) and the projection guarantees it is - // non-nil. The backing CiliumNetworkPolicy CRD requires at least one rule - // section to be present (spec anyOf: ingress|ingressDeny|egress|egressDeny), - // so a rules-less (membership-only) SecurityGroup would otherwise project to a - // selector-only spec the cilium.io/v2 API rejects synchronously. An empty - // ingress list is the CRD's documented no-op ("if omitted or empty, this rule - // does not apply at ingress"), and enableDefaultDeny defaults to false for a - // direction with no rules, so the always-present list keeps the policy - // schema-valid while leaving the member pods' connectivity untouched. The - // non-nil guarantee matters because the CRD has no nullable fields: a nil - // slice without omitempty would marshal to null and be rejected. - Ingress []CiliumIngressRule `json:"ingress"` + // Ingress is the list of allowed inbound traffic rules. + Ingress []CiliumIngressRule `json:"ingress,omitempty"` - // Egress is the list of allowed outbound traffic rules. Unlike Ingress it - // keeps omitempty: ingress is the guaranteed-present carrier that satisfies the - // CRD anyOf in every case, so egress is emitted only when there are egress - // rules. + // Egress is the list of allowed outbound traffic rules. Egress []CiliumEgressRule `json:"egress,omitempty"` + + // EnableDefaultDeny overrides Cilium's per-direction default-deny posture. + // The projection sets it only for a membership-only SecurityGroup; see + // membershipOnlySpec. + EnableDefaultDeny *CiliumDefaultDeny `json:"enableDefaultDeny,omitempty"` +} + +// CiliumDefaultDeny mirrors the cilium.io/v2 enableDefaultDeny block. A nil +// direction leaves Cilium's default in place: deny for a direction that has +// rules, allow otherwise. +type CiliumDefaultDeny struct { + Ingress *bool `json:"ingress,omitempty"` + Egress *bool `json:"egress,omitempty"` } // CiliumIngressRule mirrors a single cilium.io/v2 ingress rule. fromApp/fromSG @@ -170,6 +169,26 @@ func (in *CiliumNetworkPolicySpec) DeepCopyInto(out *CiliumNetworkPolicySpec) { in.Egress[i].DeepCopyInto(&out.Egress[i]) } } + if in.EnableDefaultDeny != nil { + out.EnableDefaultDeny = in.EnableDefaultDeny.DeepCopy() + } +} + +// DeepCopy returns a deep copy of the receiver. +func (in *CiliumDefaultDeny) DeepCopy() *CiliumDefaultDeny { + if in == nil { + return nil + } + out := new(CiliumDefaultDeny) + if in.Ingress != nil { + v := *in.Ingress + out.Ingress = &v + } + if in.Egress != nil { + v := *in.Egress + out.Egress = &v + } + return out } // DeepCopy returns a deep copy of the receiver. diff --git a/pkg/registry/sdn/securitygroup/rest.go b/pkg/registry/sdn/securitygroup/rest.go index a06439a75..d4a8d223d 100644 --- a/pkg/registry/sdn/securitygroup/rest.go +++ b/pkg/registry/sdn/securitygroup/rest.go @@ -143,11 +143,10 @@ func sgFromSelector(sel metav1.LabelSelector) (string, bool) { // projectIngress turns the SecurityGroup ingress rules into Cilium ingress // rules: fromApp peers become lineage-label endpointSelectors, fromSG peers // become membership-label endpointSelectors, and fromCIDR/toPorts carry over. -// It always returns a non-nil slice (empty when there are no rules): the backing -// CiliumNetworkPolicy's ingress section must be present on the wire to satisfy -// the CRD anyOf, and a non-nil empty slice serializes to an empty list rather -// than null (which the CRD, having no nullable fields, would reject). func projectIngress(rules []sdnv1alpha1.IngressRule) []CiliumIngressRule { + if rules == nil { + return nil + } out := make([]CiliumIngressRule, len(rules)) for i := range rules { var eps []metav1.LabelSelector @@ -195,9 +194,8 @@ func projectEgress(rules []sdnv1alpha1.EgressRule) []CiliumEgressRule { // the lineage-label nor the membership-label shape are ignored, so a // hand-edited backing policy degrades gracefully rather than erroring. func reconstructIngress(rules []CiliumIngressRule) []sdnv1alpha1.IngressRule { - // Collapse an empty list to nil so the always-present empty ingress section the - // projection emits for a rules-less group (to satisfy the CRD anyOf) is - // invisible in the SecurityGroup view, keeping the rules-less round-trip exact. + // Collapse an empty list to nil: policies written before membershipOnlySpec + // carried an always-present empty ingress section. if len(rules) == 0 { return nil } @@ -224,8 +222,8 @@ func reconstructIngress(rules []CiliumIngressRule) []sdnv1alpha1.IngressRule { // reconstructEgress is the egress counterpart of reconstructIngress. func reconstructEgress(rules []CiliumEgressRule) []sdnv1alpha1.EgressRule { - // Collapse an empty list to nil for the same reason as reconstructIngress, so a - // SecurityGroup that carries no egress rules round-trips with egress unset. + // Collapse an empty list to nil, so a SecurityGroup that carries no egress + // rules round-trips with egress unset. if len(rules) == 0 { return nil } @@ -409,8 +407,10 @@ func policyToSecurityGroup(np *CiliumNetworkPolicy) *sdnv1alpha1.SecurityGroup { // Attachments live in a storage-owned annotation, not the // endpointSelector (which is the SecurityGroup's own membership label). Attachments: decodeAttachments(np.Annotations[attachmentsAnnotation]), - Ingress: reconstructIngress(spec.Ingress), - Egress: reconstructEgress(spec.Egress), + } + if !isMembershipOnlySpec(spec) { + sg.Spec.Ingress = reconstructIngress(spec.Ingress) + sg.Spec.Egress = reconstructEgress(spec.Egress) } } return sg @@ -485,6 +485,10 @@ func securityGroupToPolicy(sg *sdnv1alpha1.SecurityGroup, cur *CiliumNetworkPoli // label onto the attached applications' pods. Rules project app/SG peers into // endpoint selectors. spec := sg.Spec.DeepCopy() + if len(spec.Ingress) == 0 && len(spec.Egress) == 0 { + out.Spec = membershipOnlySpec(sg.Name) + return &out + } out.Spec = &CiliumNetworkPolicySpec{ EndpointSelector: buildEndpointSelector(sg.Name), Ingress: projectIngress(spec.Ingress), @@ -497,6 +501,41 @@ func securityGroupToPolicy(sg *sdnv1alpha1.SecurityGroup, cur *CiliumNetworkPoli return &out } +// membershipOnlySpec is the backing spec of a SecurityGroup with no rules. The +// policy must still exist, since it carries the attachments the +// securitygroup-controller reconciles, but Cilium rejects a rule with no +// ingress, ingressDeny, egress or egressDeny entry (api.Rule.Sanitize counts +// entries, so an empty list is as good as absent) and marks the policy +// Valid=False. One empty ingress rule satisfies that check and allows nothing, +// and disabling default deny in both directions keeps that rule from putting +// the member pods into default-deny, so the policy changes nothing in the +// datapath. That relies on enable-non-default-deny-policies, which defaults to +// true; with it off Cilium forces default deny on every policy. +func membershipOnlySpec(name string) *CiliumNetworkPolicySpec { + off := false + return &CiliumNetworkPolicySpec{ + EndpointSelector: buildEndpointSelector(name), + Ingress: []CiliumIngressRule{{}}, + EnableDefaultDeny: &CiliumDefaultDeny{Ingress: &off, Egress: &off}, + } +} + +// isMembershipOnlySpec reports whether spec is what membershipOnlySpec writes, +// so the reverse projection shows no rules for it. enableDefaultDeny tells it +// apart from a SecurityGroup whose single ingress rule is genuinely empty, +// which the projection writes without it, keeping ingress default deny. +func isMembershipOnlySpec(spec *CiliumNetworkPolicySpec) bool { + d := spec.EnableDefaultDeny + if d == nil || d.Ingress == nil || *d.Ingress || d.Egress == nil || *d.Egress { + return false + } + if len(spec.Egress) != 0 || len(spec.Ingress) != 1 { + return false + } + in := spec.Ingress[0] + return len(in.FromEndpoints) == 0 && len(in.FromCIDR) == 0 && len(in.ToPorts) == 0 +} + // normalizePortProtocols upper-cases every port protocol in place so the value // written to the backing CiliumNetworkPolicy matches its case-sensitive enum. func normalizePortProtocols(spec *CiliumNetworkPolicySpec) { diff --git a/pkg/registry/sdn/securitygroup/rest_test.go b/pkg/registry/sdn/securitygroup/rest_test.go index 5473dfd56..f3ad68fc3 100644 --- a/pkg/registry/sdn/securitygroup/rest_test.go +++ b/pkg/registry/sdn/securitygroup/rest_test.go @@ -7,6 +7,7 @@ import ( "context" "encoding/json" "errors" + "fmt" "reflect" "sort" "strings" @@ -1195,16 +1196,53 @@ func TestCreateAcceptsEmptyAttachments(t *testing.T) { } } +// specWire returns the backing policy's spec as Cilium decodes it: the CRD and +// the agent read the wire form, not the Go struct. +func specWire(t *testing.T, np *CiliumNetworkPolicy) map[string]any { + t.Helper() + raw, err := json.Marshal(np) + if err != nil { + t.Fatalf("marshal backing policy: %v", err) + } + var decoded struct { + Spec map[string]any `json:"spec"` + } + if err := json.Unmarshal(raw, &decoded); err != nil { + t.Fatalf("unmarshal backing policy: %v", err) + } + return decoded.Spec +} + +// ciliumAcceptsRuleSections mirrors the first check of Cilium's +// api.Rule.Sanitize, which marks a policy Valid=False when none of its rule +// sections has an entry. The CRD anyOf only needs one of the keys to be +// present, so it admits an empty list that the agent then rejects. +func ciliumAcceptsRuleSections(spec map[string]any) bool { + for _, k := range []string{"ingress", "ingressDeny", "egress", "egressDeny"} { + if l, _ := spec[k].([]any); len(l) > 0 { + return true + } + } + return false +} + +func getBackingPolicy(t *testing.T, r *REST, name string) *CiliumNetworkPolicy { + t.Helper() + np := &CiliumNetworkPolicy{} + if err := r.c.Get(context.Background(), types.NamespacedName{Namespace: testNamespace, Name: name}, np); err != nil { + t.Fatalf("backing policy not found: %v", err) + } + return np +} + func TestRulesLessSecurityGroupProjectsValidPolicy(t *testing.T) { r := newTestREST(t) // A membership-only SecurityGroup attaches to an application to stamp the - // membership label but carries no ingress or egress rules. The backing - // CiliumNetworkPolicy CRD still requires at least one rule section to be - // present (spec anyOf: ingress|ingressDeny|egress|egressDeny), so a - // selector-only spec is rejected synchronously by the cilium.io/v2 API. The - // projection must emit an explicit empty ingress list — the CRD's documented - // no-op ("if omitted or empty, this rule does not apply at ingress") — so the - // policy is schema-valid yet leaves the member pods' connectivity untouched. + // membership label but carries no ingress or egress rules. Its backing + // policy must be one Cilium accepts and that changes nothing in the + // datapath: a rule section with an entry that allows nothing, and default + // deny disabled in both directions so that entry does not isolate the + // member pods. createSG(t, r, &sdnv1alpha1.SecurityGroup{ ObjectMeta: metav1.ObjectMeta{Name: "sg-membership", Namespace: testNamespace}, Spec: sdnv1alpha1.SecurityGroupSpec{ @@ -1212,50 +1250,22 @@ func TestRulesLessSecurityGroupProjectsValidPolicy(t *testing.T) { }, }) - np := &CiliumNetworkPolicy{} - if err := r.c.Get(context.Background(), types.NamespacedName{Namespace: testNamespace, Name: "sg-membership"}, np); err != nil { - t.Fatalf("backing policy not found: %v", err) + spec := specWire(t, getBackingPolicy(t, r, "sg-membership")) + if !ciliumAcceptsRuleSections(spec) { + t.Fatalf("rules-less policy has no rule entry, Cilium marks it Valid=False: %v", spec) + } + if got, want := fmt.Sprint(spec["ingress"]), "[map[]]"; got != want { + t.Fatalf("rules-less policy ingress = %s, want %s (one rule that allows nothing)", got, want) + } + if _, ok := spec["egress"]; ok { + t.Fatalf("rules-less policy must not carry an egress key: %v", spec) + } + if got, want := fmt.Sprint(spec["enableDefaultDeny"]), "map[egress:false ingress:false]"; got != want { + t.Fatalf("rules-less policy enableDefaultDeny = %s, want %s", got, want) } - // The CRD validates the wire form, not the Go struct, so assert on the - // marshaled bytes: spec.ingress must be present (an empty list satisfies the - // anyOf), and spec.egress must be absent (ingress is the guaranteed-present - // carrier; egress is emitted only when there are egress rules). - raw, err := json.Marshal(np) - if err != nil { - t.Fatalf("marshal backing policy: %v", err) - } - var decoded struct { - Spec map[string]json.RawMessage `json:"spec"` - } - if err := json.Unmarshal(raw, &decoded); err != nil { - t.Fatalf("unmarshal backing policy: %v", err) - } - ing, ok := decoded.Spec["ingress"] - if !ok { - t.Fatalf("backing policy spec has no ingress key; the CRD anyOf would reject it: %s", raw) - } - if string(ing) != "[]" { - t.Fatalf("rules-less policy ingress = %s, want []", ing) - } - if _, ok := decoded.Spec["egress"]; ok { - t.Fatalf("rules-less policy must not carry an egress key: %s", raw) - } - - // The same key must survive the unstructured conversion path (which honors - // omitempty independently of json.Marshal), so the empty list cannot be - // silently dropped there either. - u, err := runtime.DefaultUnstructuredConverter.ToUnstructured(np) - if err != nil { - t.Fatalf("to unstructured: %v", err) - } - spec, _ := u["spec"].(map[string]any) - if _, ok := spec["ingress"]; !ok { - t.Fatalf("unstructured backing policy spec has no ingress key: %v", spec) - } - - // On read, the synthetic empty ingress list collapses back to no rules, so a - // rules-less SecurityGroup round-trips with neither ingress nor egress. + // On read, the placeholder rule is hidden, so a rules-less SecurityGroup + // round-trips with neither ingress nor egress. out, err := r.Get(ctxNS(), "sg-membership", &metav1.GetOptions{}) if err != nil { t.Fatalf("Get returned error: %v", err) @@ -1264,6 +1274,43 @@ func TestRulesLessSecurityGroupProjectsValidPolicy(t *testing.T) { if got.Spec.Ingress != nil || got.Spec.Egress != nil { t.Fatalf("rules-less round-trip should carry no ingress/egress, got: %+v", got.Spec) } + + // Adding a rule must drop the default-deny override, or the group's + // rules would allow traffic without isolating the member pods. + got.Spec.Egress = []sdnv1alpha1.EgressRule{{ToCIDR: []string{"10.0.0.0/8"}}} + if _, _, err := r.Update(ctxNS(), "sg-membership", rest.DefaultUpdatedObjectInfo(got), nil, nil, false, &metav1.UpdateOptions{}); err != nil { + t.Fatalf("Update returned error: %v", err) + } + spec = specWire(t, getBackingPolicy(t, r, "sg-membership")) + if _, ok := spec["enableDefaultDeny"]; ok { + t.Fatalf("policy with rules must not override default deny: %v", spec) + } + if _, ok := spec["ingress"]; ok { + t.Fatalf("egress-only policy must not carry an ingress key: %v", spec) + } +} + +func TestEmptyIngressRuleIsNotMembershipOnly(t *testing.T) { + r := newTestREST(t) + // An explicit empty ingress rule projects to the same rule as the + // membership-only placeholder, but it is tenant-authored: it keeps Cilium's + // ingress default deny and must stay visible on read. + createSG(t, r, &sdnv1alpha1.SecurityGroup{ + ObjectMeta: metav1.ObjectMeta{Name: "sg-deny-in", Namespace: testNamespace}, + Spec: sdnv1alpha1.SecurityGroupSpec{Ingress: []sdnv1alpha1.IngressRule{{}}}, + }) + + spec := specWire(t, getBackingPolicy(t, r, "sg-deny-in")) + if _, ok := spec["enableDefaultDeny"]; ok { + t.Fatalf("an empty ingress rule must keep default deny: %v", spec) + } + out, err := r.Get(ctxNS(), "sg-deny-in", &metav1.GetOptions{}) + if err != nil { + t.Fatalf("Get returned error: %v", err) + } + if got := out.(*sdnv1alpha1.SecurityGroup).Spec.Ingress; len(got) != 1 { + t.Fatalf("empty ingress rule should round-trip as one rule, got: %+v", got) + } } func TestCreateRejectsInvalidAttachmentLabelValue(t *testing.T) { From f21abe0d35ab533aceebbea95a6b76a88b47cefb Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Tue, 27 Oct 2026 09:12:25 +0100 Subject: [PATCH 027/518] test(mariadb-operator): run the package suite from make test The helm-unittest sweep reaches a package only through its make test target, so the suite added with the cluster-name fix would otherwise never run in CI. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- packages/system/mariadb-operator/Makefile | 3 +++ 1 file changed, 3 insertions(+) diff --git a/packages/system/mariadb-operator/Makefile b/packages/system/mariadb-operator/Makefile index 905653cac..e80038778 100644 --- a/packages/system/mariadb-operator/Makefile +++ b/packages/system/mariadb-operator/Makefile @@ -3,6 +3,9 @@ export NAMESPACE=cozy-$(NAME) include ../../../hack/package.mk +test: + helm unittest . + update: rm -rf charts helm repo add mariadb-operator https://mariadb-operator.github.io/mariadb-operator From c0c5f2580fb3d878efd34b91c9e088c9aef4db8d Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Tue, 27 Oct 2026 12:20:45 +0300 Subject: [PATCH 028/518] docs(postgres): note how an existing backup schedule changes on upgrade A release that already sets backup.schedule to "0 2 * * *" runs hourly at minute 2 today and daily at 02:00 after this change. The README now says so and gives "0 2 * * * *" for keeping the hourly run. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- packages/apps/postgres/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/packages/apps/postgres/README.md b/packages/apps/postgres/README.md index 49f08d441..9185b35d5 100644 --- a/packages/apps/postgres/README.md +++ b/packages/apps/postgres/README.md @@ -89,6 +89,8 @@ field. The chart can also emit a `cnpg.io/ScheduledBackup` directly, without a BackupClass. Superseded by the BackupClass + Plan path above and kept around for clusters that did not migrate. It does not run by default - `backup.schedule` defaults to an empty string, which gates the chart's ScheduledBackup template off. To turn it on, set a standard five-field cron such as `0 2 * * *` for every day at 02:00, or the CloudNativePG six-field form with a leading seconds field such as `0 0 2 * * *` for the same time. The chart prepends a `0` seconds field to a five-field value, because CloudNativePG reads the first field as seconds and would otherwise run `0 2 * * *` every hour. +On upgrade, an existing `backup.schedule: "0 2 * * *"` changes from hourly at minute 2 to daily at 02:00. To keep the previous hourly schedule, set `backup.schedule: "0 2 * * * *"` before upgrading. + ```yaml ## @param backup.enabled Enable plugin WAL archiving + render the chart-managed ScheduledBackup ## @param backup.schedule Cron schedule, five fields or six with leading seconds. Empty means no chart-managed ScheduledBackup From 3990268195c1b0133d365bb1c40d8c777cd7e1a1 Mon Sep 17 00:00:00 2001 From: Fedor Batonogov Date: Wed, 28 Oct 2026 11:42:37 +0300 Subject: [PATCH 029/518] feat(platform): expose Keycloak truststores through platform values Assisted-by: LLM Signed-off-by: Fedor Batonogov --- .../platform/templates/bundles/system.yaml | 10 + .../bundles_keycloak_truststores_test.yaml | 203 ++++++++++++++++++ packages/core/platform/values.yaml | 13 ++ 3 files changed, 226 insertions(+) create mode 100644 packages/core/platform/tests/bundles_keycloak_truststores_test.yaml diff --git a/packages/core/platform/templates/bundles/system.yaml b/packages/core/platform/templates/bundles/system.yaml index b175714a2..2d0fa3506 100644 --- a/packages/core/platform/templates/bundles/system.yaml +++ b/packages/core/platform/templates/bundles/system.yaml @@ -178,9 +178,19 @@ {{- $controllerValues := dict "cozystackController" (dict "disableTelemetry" true) -}} {{- $_ := set $cozystackEngineComponents "cozystack-controller" (dict "values" $controllerValues) -}} {{- end -}} +{{- $keycloakTruststores := .Values.authentication.oidc.keycloakTruststores -}} +{{- $hasKeycloakTruststores := and (not (kindIs "invalid" $keycloakTruststores)) (or (not (kindIs "slice" $keycloakTruststores)) (not (empty $keycloakTruststores))) -}} +{{- $keycloakComponents := dict -}} +{{- if $hasKeycloakTruststores }} +{{- $_ := set $keycloakComponents "keycloak" (dict "values" (dict "truststores" $keycloakTruststores)) -}} +{{- end }} {{- if .Values.authentication.oidc.enabled }} {{include "cozystack.platform.package" (list "cozystack.cozystack-engine" "oidc" $ $cozystackEngineComponents) }} +{{- if $hasKeycloakTruststores }} +{{include "cozystack.platform.package" (list "cozystack.keycloak" "default" $ $keycloakComponents) }} +{{- else }} {{include "cozystack.platform.package.default" (list "cozystack.keycloak" $) }} +{{- end }} {{include "cozystack.platform.package.default" (list "cozystack.keycloak-operator" $) }} {{- else }} {{include "cozystack.platform.package" (list "cozystack.cozystack-engine" "default" $ $cozystackEngineComponents) }} diff --git a/packages/core/platform/tests/bundles_keycloak_truststores_test.yaml b/packages/core/platform/tests/bundles_keycloak_truststores_test.yaml new file mode 100644 index 000000000..6319c6dd9 --- /dev/null +++ b/packages/core/platform/tests/bundles_keycloak_truststores_test.yaml @@ -0,0 +1,203 @@ +suite: bundles.system Keycloak truststores wiring +templates: + - templates/bundles/system.yaml +release: + name: cozystack + namespace: cozy-system +tests: + - it: emits cozystack.keycloak without a components block when keycloakTruststores is unset + set: + bundles: + system: + enabled: true + variant: isp-full + authentication: + oidc: + enabled: true + documentSelector: + path: metadata.name + value: cozystack.keycloak + asserts: + - equal: + path: kind + value: Package + - equal: + path: spec.variant + value: default + - notExists: + path: spec.components + + - it: omits the components block when keycloakTruststores is explicitly empty + set: + bundles: + system: + enabled: true + variant: isp-full + authentication: + oidc: + enabled: true + keycloakTruststores: [] + documentSelector: + path: metadata.name + value: cozystack.keycloak + asserts: + - notExists: + path: spec.components + + - it: omits the components block when keycloakTruststores is explicitly null + set: + bundles: + system: + enabled: true + variant: isp-full + authentication: + oidc: + enabled: true + keycloakTruststores: null + documentSelector: + path: metadata.name + value: cozystack.keycloak + asserts: + - notExists: + path: spec.components + + - it: forwards false so the Keycloak chart can reject it instead of silently removing trust + set: + bundles: + system: + enabled: true + variant: isp-full + authentication: + oidc: + enabled: true + keycloakTruststores: false + documentSelector: + path: metadata.name + value: cozystack.keycloak + asserts: + - equal: + path: spec.components.keycloak.values.truststores + value: false + + - it: forwards zero so the Keycloak chart can reject it instead of silently removing trust + set: + bundles: + system: + enabled: true + variant: isp-full + authentication: + oidc: + enabled: true + keycloakTruststores: 0 + documentSelector: + path: metadata.name + value: cozystack.keycloak + asserts: + - equal: + path: spec.components.keycloak.values.truststores + value: 0 + + - it: forwards an empty string so the Keycloak chart can reject it instead of silently removing trust + set: + bundles: + system: + enabled: true + variant: isp-full + authentication: + oidc: + enabled: true + keycloakTruststores: "" + documentSelector: + path: metadata.name + value: cozystack.keycloak + asserts: + - equal: + path: spec.components.keycloak.values.truststores + value: "" + + - it: forwards an empty map so the Keycloak chart can reject it instead of silently removing trust + set: + bundles: + system: + enabled: true + variant: isp-full + authentication: + oidc: + enabled: true + keycloakTruststores: {} + documentSelector: + path: metadata.name + value: cozystack.keycloak + asserts: + - equal: + path: spec.components.keycloak.values.truststores + value: {} + + - it: forwards the complete truststores list without taking over chart defaults + set: + bundles: + system: + enabled: true + variant: isp-full + authentication: + oidc: + enabled: true + keycloakTruststores: + - configMapName: organization-ca + - secretName: partner-ca + key: certificates.pem + - configMapName: secondary-ca + key: chain.pem + documentSelector: + path: metadata.name + value: cozystack.keycloak + asserts: + - equal: + path: spec.components.keycloak.values + value: + truststores: + - configMapName: organization-ca + - secretName: partner-ca + key: certificates.pem + - configMapName: secondary-ca + key: chain.pem + + - it: forwards keycloakTruststores on the isp-slim variant too + set: + bundles: + system: + enabled: true + variant: isp-slim + authentication: + oidc: + enabled: true + keycloakTruststores: + - configMapName: organization-ca + key: ca.crt + documentSelector: + path: metadata.name + value: cozystack.keycloak + asserts: + - equal: + path: spec.components.keycloak.values.truststores + value: + - configMapName: organization-ca + key: ca.crt + + - it: does not emit cozystack.keycloak for a malformed false-y value when OIDC is disabled + set: + bundles: + system: + enabled: true + variant: isp-full + authentication: + oidc: + enabled: false + keycloakTruststores: false + asserts: + - containsDocument: + apiVersion: cozystack.io/v1alpha1 + kind: Package + name: cozystack.keycloak + any: true + not: true diff --git a/packages/core/platform/values.yaml b/packages/core/platform/values.yaml index d60effbc8..c02ad7b23 100644 --- a/packages/core/platform/values.yaml +++ b/packages/core/platform/values.yaml @@ -549,6 +549,19 @@ authentication: # OIDC discovery; any other value -> used verbatim. An explicit value is not rewritten, so a # custom realm paired with an explicit URL must name the same realm. keycloakInternalUrl: null + # Additional CA trust anchors for the bundled Keycloak server (LDAPS, + # external IdPs, SMTP). Each entry selects one key from an existing Secret + # or ConfigMap in cozy-keycloak; key defaults to ca.crt. Trust applies to + # the entire server across all realms. See packages/system/keycloak/values.yaml + # for supported formats, incoming-client-mTLS scope and rotation behavior. + # Ignored when oidc.enabled is false: no Keycloak Package is emitted. An + # existing Package and its rendered truststores remain on the cluster + # (helm.sh/resource-policy: keep). + keycloakTruststores: [] + # - configMapName: organization-ca + # key: ca.crt + # - secretName: partner-ca + # key: certificates.pem # Gateway API configuration gateway: # Enable Gateway API support across the platform. When true: From 48779cc03b23b7debb6dc6e4f39958fe9c165e0a Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Thu, 29 Oct 2026 10:00:55 +0100 Subject: [PATCH 030/518] fix(redis): run one Sentinel when Redis runs a single pod The chart pinned three Sentinels whatever the replica count, so a single-pod instance paid for three pods that have nothing to fail over to. One Sentinel is enough for the operator to locate the master: its quorum is replicas/2+1, which is 1. The operator gives the Sentinel Deployment a PodDisruptionBudget with minAvailable 1 whenever Redis has two replicas or fewer. On a single Sentinel that budget refuses every eviction and would block node drains, so it is disabled in that case. From two Redis pods up nothing changes: three Sentinels, budget kept. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- api/apps/v1alpha1/redis/types.go | 4 +- packages/apps/redis/README.md | 16 +-- .../apps/redis/templates/redisfailover.yaml | 17 ++- .../redis/tests/sentinel_replicas_test.yaml | 103 ++++++++++++++++++ packages/apps/redis/values.schema.json | 4 +- packages/apps/redis/values.yaml | 4 +- packages/system/redis-rd/cozyrds/redis.yaml | 2 +- 7 files changed, 132 insertions(+), 18 deletions(-) create mode 100644 packages/apps/redis/tests/sentinel_replicas_test.yaml diff --git a/api/apps/v1alpha1/redis/types.go b/api/apps/v1alpha1/redis/types.go index 509c1e6ce..c2363749c 100644 --- a/api/apps/v1alpha1/redis/types.go +++ b/api/apps/v1alpha1/redis/types.go @@ -17,7 +17,7 @@ type Config struct { } type ConfigSpec struct { - // Number of Redis replicas. + // Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to. // +kubebuilder:default:=2 Replicas int `json:"replicas"` // Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied. @@ -51,7 +51,7 @@ type ConfigSpec struct { // Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it. // +kubebuilder:validation:Minimum=0 MinReplicasMaxLag *int `json:"minReplicasMaxLag,omitempty"` - // Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the three Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. + // Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. // +kubebuilder:default:={} Sentinel Sentinel `json:"sentinel,omitempty"` } diff --git a/packages/apps/redis/README.md b/packages/apps/redis/README.md index 0922bf821..28152d9f9 100644 --- a/packages/apps/redis/README.md +++ b/packages/apps/redis/README.md @@ -17,7 +17,7 @@ Service utilizes the freshworks-oss Redis Operator (a maintained fork of the arc | Name | Description | Type | Value | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | --------- | -| `replicas` | Number of Redis replicas. | `int` | `2` | +| `replicas` | Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to. | `int` | `2` | | `resources` | Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied. | `object` | `{}` | | `resources.cpu` | CPU available to each replica. | `quantity` | `""` | | `resources.memory` | Memory (RAM) available to each replica. | `quantity` | `""` | @@ -48,13 +48,13 @@ Service utilizes the freshworks-oss Redis Operator (a maintained fork of the arc ### Sentinel parameters -| Name | Description | Type | Value | -| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----- | -| `sentinel` | Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the three Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. | `object` | `{}` | -| `sentinel.resources` | Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`. | `object` | `{}` | -| `sentinel.resources.cpu` | CPU available to each replica. | `quantity` | `""` | -| `sentinel.resources.memory` | Memory (RAM) available to each replica. | `quantity` | `""` | -| `sentinel.resourcesPreset` | Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`. | `string` | `{}` | +| Name | Description | Type | Value | +| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----- | +| `sentinel` | Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. | `object` | `{}` | +| `sentinel.resources` | Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`. | `object` | `{}` | +| `sentinel.resources.cpu` | CPU available to each replica. | `quantity` | `""` | +| `sentinel.resources.memory` | Memory (RAM) available to each replica. | `quantity` | `""` | +| `sentinel.resourcesPreset` | Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`. | `string` | `{}` | ## Parameter examples and reference diff --git a/packages/apps/redis/templates/redisfailover.yaml b/packages/apps/redis/templates/redisfailover.yaml index 27c7e9047..2dd1fecaf 100644 --- a/packages/apps/redis/templates/redisfailover.yaml +++ b/packages/apps/redis/templates/redisfailover.yaml @@ -9,6 +9,14 @@ top-level resources falls back to resourcesPreset. */}} {{- $sentinel := .Values.sentinel | default dict -}} +{{- /* + A lone Redis pod has no replica to promote, so three Sentinels would only + watch it. One Sentinel still lets the operator find the master (its quorum is + replicas/2+1, so 1). The operator gives the Sentinel Deployment a PDB with + minAvailable 1 whenever Redis has two replicas or fewer, which on a single + Sentinel would block every node drain, so that PDB is dropped with it. +*/}} +{{- $sentinelReplicas := ternary 1 3 (le (int .Values.replicas) 1) -}} {{- $sentinelPreset := $sentinel.resourcesPreset | default .Values.resourcesPreset -}} {{- $sentinelResources := .Values.resources -}} {{- if or $sentinel.resources $sentinel.resourcesPreset -}} @@ -65,7 +73,10 @@ spec: # pods they keep no RDB, only a sentinel.conf the init container rewrites # on every start, so a chart rollback to an older engine is safe here. image: {{ $redisImage | quote }} - replicas: 3 + replicas: {{ $sentinelReplicas }} + {{- if eq $sentinelReplicas 1 }} + disablePodDisruptionBudget: true + {{- end }} resources: {{- include "cozy-lib.resources.defaultingSanitize" (list $sentinelPreset $sentinelResources $) | nindent 6 }} redis: image: {{ $redisImage | quote }} @@ -152,8 +163,8 @@ metadata: labels: workloads.cozystack.io/resource-preset: {{ $sentinelPreset | quote }} spec: - minReplicas: 2 - replicas: 3 + minReplicas: {{ ternary 1 2 (eq $sentinelReplicas 1) }} + replicas: {{ $sentinelReplicas }} kind: redis type: sentinel selector: diff --git a/packages/apps/redis/tests/sentinel_replicas_test.yaml b/packages/apps/redis/tests/sentinel_replicas_test.yaml new file mode 100644 index 000000000..012f58247 --- /dev/null +++ b/packages/apps/redis/tests/sentinel_replicas_test.yaml @@ -0,0 +1,103 @@ +suite: Sentinel count follows the Redis replicas + +release: + name: my-redis + namespace: tenant-test + +set: + _cluster: + cluster-domain: cluster.local + _namespace: + host: example.com + +templates: + - templates/redisfailover.yaml + +tests: + - it: "runs a single Sentinel for a single Redis pod" + set: + replicas: 1 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 1 + - equal: + path: spec.redis.replicas + value: 1 + + # The operator's Sentinel PDB is minAvailable 1 at two Redis replicas or + # fewer; on one Sentinel it would refuse every eviction and block node drains. + - it: "drops the Sentinel PDB when only one Sentinel runs" + set: + replicas: 1 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.disablePodDisruptionBudget + value: true + + - it: "keeps three Sentinels and their PDB from two Redis pods up" + set: + replicas: 2 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 3 + - notExists: + path: spec.sentinel.disablePodDisruptionBudget + + - it: "keeps three Sentinels for three Redis pods" + set: + replicas: 3 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 3 + + - it: "keeps three Sentinels at the chart default" + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 3 + + - it: "reports one expected Sentinel pod to the workload monitor" + set: + replicas: 1 + documentSelector: + path: metadata.name + value: my-redis-sentinel + asserts: + - equal: + path: spec.replicas + value: 1 + - equal: + path: spec.minReplicas + value: 1 + + - it: "reports three expected Sentinel pods to the workload monitor" + set: + replicas: 2 + documentSelector: + path: metadata.name + value: my-redis-sentinel + asserts: + - equal: + path: spec.replicas + value: 3 + - equal: + path: spec.minReplicas + value: 2 diff --git a/packages/apps/redis/values.schema.json b/packages/apps/redis/values.schema.json index 4bf3e0964..1a301615f 100644 --- a/packages/apps/redis/values.schema.json +++ b/packages/apps/redis/values.schema.json @@ -3,7 +3,7 @@ "type": "object", "properties": { "replicas": { - "description": "Number of Redis replicas.", + "description": "Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to.", "type": "integer", "default": 2 }, @@ -172,7 +172,7 @@ "minimum": 0 }, "sentinel": { - "description": "Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the three Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.", + "description": "Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.", "type": "object", "default": {}, "properties": { diff --git a/packages/apps/redis/values.yaml b/packages/apps/redis/values.yaml index 2a580c8e6..7917ca34d 100644 --- a/packages/apps/redis/values.yaml +++ b/packages/apps/redis/values.yaml @@ -55,7 +55,7 @@ ## @value xlarge ## @value 2xlarge -## @param {int} replicas - Number of Redis replicas. +## @param {int} replicas - Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to. replicas: 2 ## @param {Resources} [resources] - Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied. @@ -127,5 +127,5 @@ authEnabled: true ## @field {Resources} [resources] - Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`. ## @field {ResourcesPreset} [resourcesPreset] - Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`. -## @param {Sentinel} [sentinel] - Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the three Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. +## @param {Sentinel} [sentinel] - Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. sentinel: {} diff --git a/packages/system/redis-rd/cozyrds/redis.yaml b/packages/system/redis-rd/cozyrds/redis.yaml index a1d619c34..cfcaeb01d 100644 --- a/packages/system/redis-rd/cozyrds/redis.yaml +++ b/packages/system/redis-rd/cozyrds/redis.yaml @@ -8,7 +8,7 @@ spec: plural: redises singular: redis openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Redis replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.nano","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster. Can be enabled only when the instance is created, together with `tls.enabled: true`; turning it on for an existing instance, or changing `tls.enabled` while it is on, is refused at admission.","type":"boolean","default":false},"version":{"description":"Redis major version to deploy. The Redis and Sentinel pods run the same image.","type":"string","default":"v8","enum":["v8","v7"]},"tls":{"description":"TLS configuration. TLS is opt-in and is not inferred from `external`.","type":"object","default":{},"properties":{"authClients":{"description":"Maps to the Redis `tls-auth-clients` directive. Defaults to `no` — the server certificate is presented but client certificates are not validated. `yes` requires a client certificate signed by this release's CA, which the platform does not issue. Changing it restarts the Redis and Sentinel pods, because the directive is read from the config file once at startup.","type":"string","enum":["no","optional","yes"]},"enabled":{"description":"Enable TLS for Redis and Sentinel connections. Disabled unless set. Enabling it moves Redis and Sentinel to a TLS-only listener, so existing plaintext clients must be migrated at the same time. Encryption is provided by the redis-operator fork that mounts the certificate Secret into both Redis and Sentinel pods at `/tls`.","type":"boolean"}}},"authEnabled":{"description":"Enable password generation.","type":"boolean","default":true},"minReplicasToWrite":{"description":"Redis `min-replicas-to-write`: the master refuses writes unless at least this many replicas are connected and within `minReplicasMaxLag`. Unset by default, which leaves the Redis default (disabled). With `replicas: 2` and a value of 1 the master rejects writes while its only replica is down, trading availability for a bound on the writes lost if an isolated master is failed over. Replication stays asynchronous, so this bounds the loss, it does not remove it. Must be 0 or more and below `replicas` (a master with `replicas` pods has `replicas - 1` replicas), otherwise every write is refused; the chart refuses such a value.","type":"integer","minimum":0},"minReplicasMaxLag":{"description":"Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it.","type":"integer","minimum":0},"sentinel":{"description":"Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the three Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.","type":"object","default":{},"properties":{"resources":{"description":"Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`.","type":"object","properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}} + {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.nano","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster. Can be enabled only when the instance is created, together with `tls.enabled: true`; turning it on for an existing instance, or changing `tls.enabled` while it is on, is refused at admission.","type":"boolean","default":false},"version":{"description":"Redis major version to deploy. The Redis and Sentinel pods run the same image.","type":"string","default":"v8","enum":["v8","v7"]},"tls":{"description":"TLS configuration. TLS is opt-in and is not inferred from `external`.","type":"object","default":{},"properties":{"authClients":{"description":"Maps to the Redis `tls-auth-clients` directive. Defaults to `no` — the server certificate is presented but client certificates are not validated. `yes` requires a client certificate signed by this release's CA, which the platform does not issue. Changing it restarts the Redis and Sentinel pods, because the directive is read from the config file once at startup.","type":"string","enum":["no","optional","yes"]},"enabled":{"description":"Enable TLS for Redis and Sentinel connections. Disabled unless set. Enabling it moves Redis and Sentinel to a TLS-only listener, so existing plaintext clients must be migrated at the same time. Encryption is provided by the redis-operator fork that mounts the certificate Secret into both Redis and Sentinel pods at `/tls`.","type":"boolean"}}},"authEnabled":{"description":"Enable password generation.","type":"boolean","default":true},"minReplicasToWrite":{"description":"Redis `min-replicas-to-write`: the master refuses writes unless at least this many replicas are connected and within `minReplicasMaxLag`. Unset by default, which leaves the Redis default (disabled). With `replicas: 2` and a value of 1 the master rejects writes while its only replica is down, trading availability for a bound on the writes lost if an isolated master is failed over. Replication stays asynchronous, so this bounds the loss, it does not remove it. Must be 0 or more and below `replicas` (a master with `replicas` pods has `replicas - 1` replicas), otherwise every write is refused; the chart refuses such a value.","type":"integer","minimum":0},"minReplicasMaxLag":{"description":"Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it.","type":"integer","minimum":0},"sentinel":{"description":"Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.","type":"object","default":{},"properties":{"resources":{"description":"Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`.","type":"object","properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}} release: prefix: redis- labels: From 55066b981d3a93aeb395a529506d765aae6278da Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Thu, 29 Oct 2026 10:06:40 +0100 Subject: [PATCH 031/518] fix(kubernetes): bring up a replacement worker before draining the old one A worker pool rollout (instance type, image or version change) ran with maxUnavailable 1, so CAPI drained and deleted an old Machine as soon as its replacement had been created rather than once it had joined. A one-node pool was left with no node for the few minutes the new VM took to boot and register, and every pool lost one node of capacity for each step of the rollout. With maxUnavailable 0 a node is only drained once its replacement is up. The surge stays at the pool ceiling, with a floor of 1 because CAPI refuses a rolling update where both values are 0 and maxReplicas may be 0. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../kubernetes-nodes/templates/nodegroup.yaml | 9 ++- .../render_snapshot_test.yaml.snap | 6 +- .../tests/rollout_strategy_test.yaml | 67 +++++++++++++++++++ 3 files changed, 77 insertions(+), 5 deletions(-) create mode 100644 packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml diff --git a/packages/apps/kubernetes-nodes/templates/nodegroup.yaml b/packages/apps/kubernetes-nodes/templates/nodegroup.yaml index 18ec0e16b..534b93170 100644 --- a/packages/apps/kubernetes-nodes/templates/nodegroup.yaml +++ b/packages/apps/kubernetes-nodes/templates/nodegroup.yaml @@ -1024,10 +1024,15 @@ spec: # action until the autoscaler notices unschedulable Pods and # scales back up — a real disruption window on every routine # platform bump. + {{- /* maxUnavailable 0 makes a rollout (instance type, image, version) + wait for each replacement Machine to be up before draining an old + one; with 1, a one-node pool lost its only node until the new one + joined. CAPI refuses maxSurge and maxUnavailable both at 0, and + maxReplicas may be 0, hence the floor of 1. */}} strategy: rollingUpdate: - maxSurge: {{ $group.maxReplicas }} - maxUnavailable: 1 + maxSurge: {{ max 1 ($group.maxReplicas | int) }} + maxUnavailable: 0 type: RollingUpdate selector: matchLabels: diff --git a/packages/apps/kubernetes-nodes/tests/__snapshot__/render_snapshot_test.yaml.snap b/packages/apps/kubernetes-nodes/tests/__snapshot__/render_snapshot_test.yaml.snap index 281fc1238..8c7183b22 100644 --- a/packages/apps/kubernetes-nodes/tests/__snapshot__/render_snapshot_test.yaml.snap +++ b/packages/apps/kubernetes-nodes/tests/__snapshot__/render_snapshot_test.yaml.snap @@ -97,7 +97,7 @@ matches the golden render of a GPU pool: strategy: rollingUpdate: maxSurge: 3 - maxUnavailable: 1 + maxUnavailable: 0 type: RollingUpdate template: metadata: @@ -250,7 +250,7 @@ matches the golden render of a kubelet-override pool: strategy: rollingUpdate: maxSurge: 3 - maxUnavailable: 1 + maxUnavailable: 0 type: RollingUpdate template: metadata: @@ -405,7 +405,7 @@ matches the golden render of the pool objects: strategy: rollingUpdate: maxSurge: 3 - maxUnavailable: 1 + maxUnavailable: 0 type: RollingUpdate template: metadata: diff --git a/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml b/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml new file mode 100644 index 000000000..1741d77e1 --- /dev/null +++ b/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml @@ -0,0 +1,67 @@ +suite: worker pool rollout keeps capacity +templates: + - templates/nodegroup.yaml +release: + name: kubernetes-nodes-myk8s-md0 + namespace: tenant-test +set: + cluster: myk8s + _cluster: + cluster-domain: cozy.local + version: "v1.35" + minReplicas: 1 + maxReplicas: 1 + instanceType: "" + diskSize: 20Gi + storageClass: replicated + resources: + cpu: "2" + memory: 4Gi +tests: + # A rollout replaces every Machine. With maxUnavailable 1 a one-node pool + # drained and deleted its only node as soon as the replacement Machine was + # created, leaving the pool empty until the new node joined. + - it: never drains a node before its replacement is ready + documentSelector: + path: kind + value: MachineDeployment + asserts: + - equal: + path: spec.strategy.type + value: RollingUpdate + - equal: + path: spec.strategy.rollingUpdate.maxUnavailable + value: 0 + - equal: + path: spec.strategy.rollingUpdate.maxSurge + value: 1 + + - it: surges up to the pool ceiling + set: + maxReplicas: 5 + documentSelector: + path: kind + value: MachineDeployment + asserts: + - equal: + path: spec.strategy.rollingUpdate.maxSurge + value: 5 + - equal: + path: spec.strategy.rollingUpdate.maxUnavailable + value: 0 + + # CAPI rejects a RollingUpdate with maxSurge and maxUnavailable both 0. + - it: keeps a surge of one on a pool capped at zero + set: + minReplicas: 0 + maxReplicas: 0 + documentSelector: + path: kind + value: MachineDeployment + asserts: + - equal: + path: spec.strategy.rollingUpdate.maxSurge + value: 1 + - equal: + path: spec.strategy.rollingUpdate.maxUnavailable + value: 0 From 392bbb3c96e278c3056ab98f5887d74771173374 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Thu, 29 Oct 2026 10:45:21 +0100 Subject: [PATCH 032/518] fix(kubernetes): keep maxUnavailable 1 on GPU worker pools A replacement VM in a pool with passed-through GPUs may need the very GPU the old node holds. With maxUnavailable 0 it would stay Pending and the rollout would never finish, so those pools keep removing a node first. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../kubernetes-nodes/templates/nodegroup.yaml | 6 ++++-- .../__snapshot__/render_snapshot_test.yaml.snap | 2 +- .../tests/rollout_strategy_test.yaml | 17 +++++++++++++++++ 3 files changed, 22 insertions(+), 3 deletions(-) diff --git a/packages/apps/kubernetes-nodes/templates/nodegroup.yaml b/packages/apps/kubernetes-nodes/templates/nodegroup.yaml index 534b93170..45ef30cc6 100644 --- a/packages/apps/kubernetes-nodes/templates/nodegroup.yaml +++ b/packages/apps/kubernetes-nodes/templates/nodegroup.yaml @@ -1027,12 +1027,14 @@ spec: {{- /* maxUnavailable 0 makes a rollout (instance type, image, version) wait for each replacement Machine to be up before draining an old one; with 1, a one-node pool lost its only node until the new one - joined. CAPI refuses maxSurge and maxUnavailable both at 0, and + joined. A pool with passed-through GPUs keeps 1: its replacement VM + may need the very GPU the old node holds, and would wait for it + forever. CAPI refuses maxSurge and maxUnavailable both at 0, and maxReplicas may be 0, hence the floor of 1. */}} strategy: rollingUpdate: maxSurge: {{ max 1 ($group.maxReplicas | int) }} - maxUnavailable: 0 + maxUnavailable: {{ ternary 1 0 (not (empty $group.gpus)) }} type: RollingUpdate selector: matchLabels: diff --git a/packages/apps/kubernetes-nodes/tests/__snapshot__/render_snapshot_test.yaml.snap b/packages/apps/kubernetes-nodes/tests/__snapshot__/render_snapshot_test.yaml.snap index 8c7183b22..c8e990a9c 100644 --- a/packages/apps/kubernetes-nodes/tests/__snapshot__/render_snapshot_test.yaml.snap +++ b/packages/apps/kubernetes-nodes/tests/__snapshot__/render_snapshot_test.yaml.snap @@ -97,7 +97,7 @@ matches the golden render of a GPU pool: strategy: rollingUpdate: maxSurge: 3 - maxUnavailable: 0 + maxUnavailable: 1 type: RollingUpdate template: metadata: diff --git a/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml b/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml index 1741d77e1..ae44bb03d 100644 --- a/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml @@ -65,3 +65,20 @@ tests: - equal: path: spec.strategy.rollingUpdate.maxUnavailable value: 0 + + # A replacement VM in a pool with passed-through GPUs may need the very GPU + # the old node holds; with maxUnavailable 0 it would wait for it forever. + - it: lets a GPU pool remove a node before its replacement is up + set: + gpus: + - name: nvidia.com/AD102GL_L40S + documentSelector: + path: kind + value: MachineDeployment + asserts: + - equal: + path: spec.strategy.rollingUpdate.maxUnavailable + value: 1 + - equal: + path: spec.strategy.rollingUpdate.maxSurge + value: 1 From a8c4640bea007c19d012bbc332b3803051f31a2a Mon Sep 17 00:00:00 2001 From: Ahmed Kamal Date: Sat, 31 Oct 2026 23:07:29 +0200 Subject: [PATCH 033/518] test(tests): select resources in absence-only Helm checks Signed-off-by: Ahmed Kamal Assisted-by: LLM --- .github/workflows/pull-requests.yaml | 4 + hack/check-helm-document-indices.bats | 141 ++++++++++++++++++ hack/check-helm-document-indices.py | 79 ++++++++++ .../tests/dashboard-resourcemap_test.yaml | 4 +- .../tests/kamaji_managed_args_test.yaml | 8 +- packages/apps/mariadb/tests/tls_test.yaml | 27 +++- packages/apps/mongodb/tests/tls_test.yaml | 32 +++- packages/apps/redis/tests/tls_test.yaml | 8 +- .../tests/solver_test.yaml | 8 +- .../tests/volume_resize_hook_test.yaml | 6 +- 10 files changed, 294 insertions(+), 23 deletions(-) create mode 100644 hack/check-helm-document-indices.bats create mode 100644 hack/check-helm-document-indices.py diff --git a/.github/workflows/pull-requests.yaml b/.github/workflows/pull-requests.yaml index b8b06fedf..96dce2056 100644 --- a/.github/workflows/pull-requests.yaml +++ b/.github/workflows/pull-requests.yaml @@ -364,6 +364,10 @@ jobs: with: path: _out/envtest key: envtest-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('Makefile') }} + - name: Check new Helm document indices + env: + BASE_REF: ${{ github.event.pull_request.base.ref }} + run: python3 hack/check-helm-document-indices.py "$(git merge-base HEAD "origin/$BASE_REF")" # -k is required, not cosmetic: unit-tests and test-controllers are two # independent goals on the same command line, and without -k make stops # handing out new work the moment any unit-tests prerequisite fails — diff --git a/hack/check-helm-document-indices.bats b/hack/check-helm-document-indices.bats new file mode 100644 index 000000000..f797d0b7f --- /dev/null +++ b/hack/check-helm-document-indices.bats @@ -0,0 +1,141 @@ +#!/usr/bin/env bats + +load test_helper + +REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)" + +setup() { + strict_setup + tmp=$(mktemp -d) + cd "$tmp" + git init -q + mkdir -p packages/apps/example/templates packages/apps/example/tests + cat > packages/apps/example/templates/multi.yaml <<'EOF' +apiVersion: v1 +kind: ConfigMap +metadata: + name: first +--- +apiVersion: v1 +kind: Secret +metadata: + name: second +EOF + head -n 4 packages/apps/example/templates/multi.yaml > packages/apps/example/templates/single.yaml + git add . + git -c user.name=Test -c user.email=test@example.org commit -qm fixture + base=$(git rev-parse HEAD) +} + +teardown() { + cd "$REPO_ROOT" + rm -rf "$tmp" +} + +write_suite() { + cat > packages/apps/example/tests/resource_test.yaml < output 2>&1; then + echo "accepted a positional selector on a multi-document template" >&2 + false + fi + grep -q 'resource_test.yaml:6: use documentSelector' output +} + +@test "allows an index on a single-document template and ignores comments" { + write_suite single + printf '\n# documentIndex: 1\n' >> packages/apps/example/tests/resource_test.yaml + python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" +} + +@test "leaves existing indices alone when another assertion changes" { + write_suite multi + git -c user.name=Test -c user.email=test@example.org commit -qm existing + base=$(git rev-parse HEAD) + sed -i 's/spec.missing/spec.other/' packages/apps/example/tests/resource_test.yaml + python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" +} + +@test "follows assertion template overrides in later YAML suites" { + write_suite single + cat >> packages/apps/example/tests/resource_test.yaml <<'EOF' +--- +suite: overridden template +templates: + - templates/single.yaml +tests: + - it: checks another resource + asserts: + - notExists: + path: spec.missing + template: templates/multi.yaml + documentIndex: 1 +EOF + if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then + echo "accepted an assertion-level positional selector" >&2 + false + fi + grep -q 'use documentSelector.*templates/multi.yaml' output +} + +@test "rejects an invalid base rather than passing an empty diff" { + if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" missing-base > output 2>&1; then + echo "accepted an invalid base" >&2 + false + fi +} + +@test "test-level indices follow each assertion's template override" { + write_suite multi + cat > packages/apps/example/tests/resource_test.yaml <<'EOF' +suite: assertion override +templates: [templates/single.yaml, templates/multi.yaml] +tests: + - it: checks only the single-document template + documentIndex: 0 + asserts: + - template: templates/single.yaml + notExists: + path: spec.missing +EOF + python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" +} + +@test "conditional kinds inside one document are not multiple documents" { + write_suite single + cat > packages/apps/example/templates/single.yaml <<'EOF' +apiVersion: v1 +{{ if .Values.secret }} +kind: Secret +{{ else }} +kind: ConfigMap +{{ end }} +metadata: + name: single +EOF + python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" +} + +@test "rejects a positional selector whose value is on a separate line" { + write_suite multi + sed -i 's/documentIndex: 0/documentIndex:\n 0/' packages/apps/example/tests/resource_test.yaml + if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then + echo "accepted a multiline positional selector" >&2 + false + fi + grep -q 'resource_test.yaml:6: use documentSelector' output +} diff --git a/hack/check-helm-document-indices.py b/hack/check-helm-document-indices.py new file mode 100644 index 000000000..bf9c56f47 --- /dev/null +++ b/hack/check-helm-document-indices.py @@ -0,0 +1,79 @@ +#!/usr/bin/env python3 +"""Reject added positional selectors on multi-resource Helm templates. + +Usage: hack/check-helm-document-indices.py +Requires mikefarah/yq, like the chart test tooling. Existing indices are left +alone. This static check counts --- separated blocks with a top-level kind; +it does not evaluate conditional documents or resources emitted by helpers/ranges. +""" + +import json +from pathlib import Path +import re +import subprocess +import sys + + +def git(*args): + return subprocess.check_output(["git", *args], text=True) + + +def check(base): + root = Path(git("rev-parse", "--show-toplevel").strip()) + paths = git("diff", "--name-only", "--diff-filter=ACMR", base, "--", + "packages/*/*/tests/*_test.yaml").splitlines() + errors = [] + for path in paths: + added = set() + for line in git("diff", "--no-ext-diff", "--unified=0", base, "--", path).splitlines(): + if line.startswith("@@"): + number = int(re.search(r"\+(\d+)", line).group(1)) + elif line.startswith(("+++", "---", "diff ", "index ", "\\")): + continue + elif line.startswith("+"): + if "documentIndex" in line: + added.add(number) + number += 1 + elif line.startswith(" "): + number += 1 + if not added: + continue + + # Preserve the key's source line while yq resolves multi-document YAML, + # flow mappings and anchors; a text-only suite parser misses those. + expression = ('[. | (.. | select(tag == "!!map" and has("documentIndex")) ' + '| .documentIndex) |= {"line": key | line}]') + suites = json.loads(subprocess.check_output( + ["yq", "eval-all", "-o=json", expression, str(root / path)], text=True)) + chart = (root / path).parent.parent + for suite in suites: + if not suite: + continue + for test in suite.get("tests", []): + templates = test.get("templates", suite.get("templates", ["templates/*"])) + for assertion in test.get("asserts", []): + index = assertion.get("documentIndex", test.get("documentIndex", {})) + number = index.get("line") + if number not in added: + continue + selected = [assertion["template"]] if "template" in assertion else templates + for pattern in selected: + matches = list(chart.glob(pattern)) + if not matches: + matches = list((chart / "templates").glob(pattern)) + for template in matches: + if not template.is_file(): + continue + documents = re.split(r"(?m)^---[ \t]*(?:#.*)?$", template.read_text()) + if sum(bool(re.search(r"(?m)^kind\s*:", doc)) for doc in documents) > 1: + errors.append(f"{path}:{number}: use documentSelector, not " + f"documentIndex, for {template.relative_to(root)}") + for error in sorted(set(errors)): + print(error, file=sys.stderr) + return bool(errors) + + +if __name__ == "__main__": + if len(sys.argv) != 2: + sys.exit("usage: check-helm-document-indices.py ") + sys.exit(check(sys.argv[1])) diff --git a/packages/apps/kafka/tests/dashboard-resourcemap_test.yaml b/packages/apps/kafka/tests/dashboard-resourcemap_test.yaml index ace053c58..88da0d69a 100644 --- a/packages/apps/kafka/tests/dashboard-resourcemap_test.yaml +++ b/packages/apps/kafka/tests/dashboard-resourcemap_test.yaml @@ -137,11 +137,13 @@ tests: release: name: test-kafka namespace: tenant-test + documentSelector: + path: kind + value: Role asserts: - notContains: path: rules[2].resourceNames content: test-kafka-zookeeper - documentIndex: 0 # External bootstrap service absent when external=false, regardless of tls.enabled - it: does not grant access to external bootstrap service when tls.enabled=true and external=false diff --git a/packages/apps/kubernetes/tests/kamaji_managed_args_test.yaml b/packages/apps/kubernetes/tests/kamaji_managed_args_test.yaml index b09343cf5..85045c9a1 100644 --- a/packages/apps/kubernetes/tests/kamaji_managed_args_test.yaml +++ b/packages/apps/kubernetes/tests/kamaji_managed_args_test.yaml @@ -241,18 +241,22 @@ tests: errorMessage: 'controlPlane.apiServer.extraArgs contains "--secure-port=6444" — the tenant control plane sets --secure-port itself and drops every entry naming it before the kube-apiserver sees the list, so this entry has no effect, and written without =value it leaves its value behind as an argument the apiserver refuses. Remove the entry; the description of controlPlane.apiServer.extraArgs lists every flag the control plane owns.' - it: leaves spec.admissionControllers out by default, so the control plane keeps its own default set + documentSelector: + path: kind + value: KamajiControlPlane asserts: - notExists: path: spec.admissionControllers - documentIndex: 1 - it: leaves spec.admissionControllers out for an explicitly empty list set: controlPlane.apiServer.admissionControllers: [] + documentSelector: + path: kind + value: KamajiControlPlane asserts: - notExists: path: spec.admissionControllers - documentIndex: 1 - it: renders controlPlane.apiServer.admissionControllers into spec.admissionControllers verbatim set: diff --git a/packages/apps/mariadb/tests/tls_test.yaml b/packages/apps/mariadb/tests/tls_test.yaml index 2f5f4548a..ac7b8ae79 100644 --- a/packages/apps/mariadb/tests/tls_test.yaml +++ b/packages/apps/mariadb/tests/tls_test.yaml @@ -530,7 +530,9 @@ tests: - templates/dashboard-resourcemap.yaml set: external: false - documentIndex: 0 + documentSelector: + path: kind + value: Role asserts: - notContains: path: rules[1].resourceNames @@ -544,7 +546,9 @@ tests: - templates/dashboard-resourcemap.yaml set: external: true - documentIndex: 0 + documentSelector: + path: kind + value: Role asserts: - notContains: path: rules[1].resourceNames @@ -560,7 +564,9 @@ tests: external: true tls: issuer: operator - documentIndex: 0 + documentSelector: + path: kind + value: Role asserts: - notContains: path: rules[1].resourceNames @@ -576,7 +582,9 @@ tests: external: false tls: issuer: cert-manager - documentIndex: 0 + documentSelector: + path: kind + value: Role asserts: - notContains: path: rules[1].resourceNames @@ -695,7 +703,9 @@ tests: set: tls: issuer: cert-manager - documentIndex: 3 + documentSelector: + path: metadata.name + value: my-mariadb-tls asserts: - notContains: path: spec.dnsNames @@ -1187,7 +1197,9 @@ tests: - it: "dashboard Role never grants the CA bundle even when TLS is unmanaged" templates: - templates/dashboard-resourcemap.yaml - documentIndex: 0 + documentSelector: + path: kind + value: Role set: tls: issuer: operator @@ -1436,6 +1448,9 @@ tests: enabled: true s3Bucket: s3.example.org/test cleanupStrategy: "--keep-last=3" + documentSelector: + path: kind + value: Secret asserts: - notMatchRegex: path: stringData["backup.sh"] diff --git a/packages/apps/mongodb/tests/tls_test.yaml b/packages/apps/mongodb/tests/tls_test.yaml index 0ddb38e46..8cf64b8cc 100644 --- a/packages/apps/mongodb/tests/tls_test.yaml +++ b/packages/apps/mongodb/tests/tls_test.yaml @@ -53,7 +53,9 @@ tests: value: mg-test-ssl - it: spec.tls has no issuerConf (the operator-managed secret is passed directly) - documentIndex: 0 + documentSelector: + path: kind + value: PerconaServerMongoDB asserts: - notExists: path: spec.tls.issuerConf @@ -127,13 +129,17 @@ tests: # test would pass even with the external check removed. _namespace: host: example.com - documentIndex: 0 + documentSelector: + path: kind + value: PerconaServerMongoDB asserts: - notExists: path: spec.replsets[0].splitHorizons - it: chart does not set allowInvalidCertificates, leaving the operator default - documentIndex: 0 + documentSelector: + path: kind + value: PerconaServerMongoDB asserts: # The README states the permissive default is inherited rather than # chosen here. Pin that, so the claim stays falsifiable. @@ -141,7 +147,9 @@ tests: path: spec.tls.allowInvalidCertificates - it: chart does not set certValidityDuration, leaving the operator default - documentIndex: 0 + documentSelector: + path: kind + value: PerconaServerMongoDB asserts: # Same class as allowInvalidCertificates above: the leaf lifetime is the # operator's to choose, not this chart's. Deliberately left unpinned. @@ -177,7 +185,9 @@ tests: # the CRD and notExists on it can never fail. - it: no TLS-disabling escape hatches with a default replica count - documentIndex: 0 + documentSelector: + path: kind + value: PerconaServerMongoDB asserts: - notExists: path: spec.allowUnsafeConfigurations @@ -235,7 +245,9 @@ tests: - name: rs0 replicas: 3 size: 10Gi - documentIndex: 0 + documentSelector: + path: kind + value: PerconaServerMongoDB asserts: # splitHorizons is replica-set scoped and rendered only for the # non-sharded topology. In sharded mode external access goes through the @@ -255,7 +267,9 @@ tests: replicas: 0 _namespace: host: example.com - documentIndex: 0 + documentSelector: + path: kind + value: PerconaServerMongoDB asserts: # With replicas: 0 the range body renders nothing. Emitting the key with # a null value would be rejected by the API server: the CRD types @@ -269,7 +283,9 @@ tests: external: true _namespace: host: "" - documentIndex: 0 + documentSelector: + path: kind + value: PerconaServerMongoDB asserts: # The controller may create _namespace before populating host, so the # bootstrap window presents host as an empty string rather than as an diff --git a/packages/apps/redis/tests/tls_test.yaml b/packages/apps/redis/tests/tls_test.yaml index 9b1b3ac0b..c8f55ee91 100644 --- a/packages/apps/redis/tests/tls_test.yaml +++ b/packages/apps/redis/tests/tls_test.yaml @@ -557,7 +557,9 @@ tests: external: false tls: enabled: true - documentIndex: 3 + documentSelector: + path: metadata.name + value: my-redis-tls asserts: - notContains: path: spec.dnsNames @@ -840,7 +842,9 @@ tests: external: false tls: enabled: true - documentIndex: 3 + documentSelector: + path: metadata.name + value: my-redis-tls asserts: - notContains: path: spec.dnsNames diff --git a/packages/system/cert-manager-issuers/tests/solver_test.yaml b/packages/system/cert-manager-issuers/tests/solver_test.yaml index 953d978bc..2a9faebbf 100644 --- a/packages/system/cert-manager-issuers/tests/solver_test.yaml +++ b/packages/system/cert-manager-issuers/tests/solver_test.yaml @@ -228,10 +228,14 @@ tests: gateway-class-name: " cloudflare-tunnel " gateway-edge-classes: cloudflare-tunnel asserts: - - documentIndex: 0 + - documentSelector: + path: metadata.name + value: letsencrypt-prod notExists: path: spec.acme.solvers[0].http01.gatewayHTTPRoute.parentRefs[0].sectionName - - documentIndex: 1 + - documentSelector: + path: metadata.name + value: letsencrypt-stage notExists: path: spec.acme.solvers[0].http01.gatewayHTTPRoute.parentRefs[0].sectionName diff --git a/packages/system/seaweedfs/tests/volume_resize_hook_test.yaml b/packages/system/seaweedfs/tests/volume_resize_hook_test.yaml index 402748312..011a3de94 100644 --- a/packages/system/seaweedfs/tests/volume_resize_hook_test.yaml +++ b/packages/system/seaweedfs/tests/volume_resize_hook_test.yaml @@ -156,9 +156,11 @@ tests: resources: requests: storage: 10Gi + documentSelector: + path: kind + value: Job asserts: - - documentIndex: 0 - notMatchRegex: + - notMatchRegex: path: spec.template.spec.containers[0].args[0] pattern: kubectl patch pvc From f12887636c430b0f7ab0b871a968f94205c3aa66 Mon Sep 17 00:00:00 2001 From: Ahmed Kamal Date: Sat, 31 Oct 2026 23:14:29 +0200 Subject: [PATCH 034/518] fix(tests): reject matcher failures in negative file assertions Signed-off-by: Ahmed Kamal Assisted-by: LLM --- hack/run-kubernetes-block-io_test.bats | 30 ------- hack/run-kubernetes-cpu-throttle_test.bats | 39 --------- hack/run-kubernetes-kvm-exits_test.bats | 30 ------- .../run-kubernetes-network-counters_test.bats | 30 ------- hack/run-kubernetes-node-join_test.bats | 30 ------- hack/run-kubernetes-runner-canary_test.bats | 31 ------- hack/run-kubernetes-runner-cpu_test.bats | 31 ------- .../run-kubernetes-sandbox-cpu-time_test.bats | 30 ------- hack/run-kubernetes-serial-console_test.bats | 17 ---- ...run-kubernetes-talos-diagnostics_test.bats | 10 --- hack/run-kubernetes-thread-cpu_test.bats | 31 ------- hack/runner-identity.bats | 31 ------- hack/test-helper_test.bats | 85 +++++++++++++++++++ hack/test_helper.bash | 29 +++++++ 14 files changed, 114 insertions(+), 340 deletions(-) create mode 100644 hack/test-helper_test.bats diff --git a/hack/run-kubernetes-block-io_test.bats b/hack/run-kubernetes-block-io_test.bats index b083f9e4f..18b12d253 100644 --- a/hack/run-kubernetes-block-io_test.bats +++ b/hack/run-kubernetes-block-io_test.bats @@ -151,36 +151,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - # A missing file must fail rather than vacuously pass: a bare `! grep -q` - # succeeds on an unreadable path, which is indistinguishable from "the file - # exists and does not carry the label". - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - # Branched on awk's exact status. awk exits 1 for "no line matched" and 2 for - # "I could not evaluate this"; folded into an `if`, both land in the passing - # branch and the assertion "this is absent" is satisfied by the matcher giving - # up. Every negative claim in this file rests on the three-way split. - local _rc=0 - awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac -} - # The function under test redirects a command's stderr into a report file and # decides which artifact to write from it. cozytest.sh runs under `set -x`, so # the tracer's own line for that command lands on the very stderr being diff --git a/hack/run-kubernetes-cpu-throttle_test.bats b/hack/run-kubernetes-cpu-throttle_test.bats index d254c7d49..c4ed0dfa4 100644 --- a/hack/run-kubernetes-cpu-throttle_test.bats +++ b/hack/run-kubernetes-cpu-throttle_test.bats @@ -94,45 +94,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - # A missing file must fail rather than vacuously pass: a bare `! grep -q` - # succeeds on an unreadable path, which is indistinguishable from "the file - # exists and does not carry the label". awk exits 2 on an unreadable path, - # which folded into an if reads the same way, so the file is checked first. - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - # Branched on awk's exact status, not on truthiness. awk exits 1 for "no line - # matched" and 2 for "I could not evaluate this" -- an unparseable regex, most - # of all. Folded into `if ... then fail; fi`, both land in the passing branch, - # so the assertion "this pattern is absent" is satisfied by the matcher giving - # up. That is the wrong direction for a negative assertion: a positive one - # fails loudly when its matcher breaks and gets fixed, this one goes green and - # stays green. Every claim in this file about something NOT appearing rests on - # the three-way split below. - # `|| _rc=$?` rather than a bare call: both runners set -e, and awk exiting 1 - # for "no match" -- the ordinary success case here -- would otherwise abort the - # test before the branch below ever runs. The `if` form this replaces was - # exempt only because conditions are exempt. - local _rc=0 - awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac -} - # COZY_REPORT_DIR is set inline in each test below rather than through a helper, # which is the opposite of what the node-join suite does. The difference is # deliberate and shrinks to one line: there the assignment carries a ten-line diff --git a/hack/run-kubernetes-kvm-exits_test.bats b/hack/run-kubernetes-kvm-exits_test.bats index 0473e642f..47ff07d9e 100644 --- a/hack/run-kubernetes-kvm-exits_test.bats +++ b/hack/run-kubernetes-kvm-exits_test.bats @@ -81,36 +81,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - # A missing file must fail rather than vacuously pass: a bare `! grep -q` - # succeeds on an unreadable path, which is indistinguishable from "the file - # exists and does not carry the label". - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - # Branched on awk's exact status. awk exits 1 for "no line matched" and 2 for - # "I could not evaluate this"; folded together, a negative assertion is - # satisfied by its own matcher giving up, which is the direction that goes - # green and stays green. - local _rc=0 - awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac -} - # The function under test redirects a command's stderr into a report file and # decides which artifact to write from it. cozytest.sh runs under `set -x`, so # the tracer's own line for that command lands on the very stderr being diff --git a/hack/run-kubernetes-network-counters_test.bats b/hack/run-kubernetes-network-counters_test.bats index 73da4c2b2..42f986821 100644 --- a/hack/run-kubernetes-network-counters_test.bats +++ b/hack/run-kubernetes-network-counters_test.bats @@ -148,36 +148,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - # A missing file must fail rather than vacuously pass: a bare `! grep -q` - # succeeds on an unreadable path, which is indistinguishable from "the file - # exists and does not carry the label". - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - # Branched on awk's exact status. awk exits 1 for "no line matched" and 2 for - # "I could not evaluate this"; folded into an `if`, both land in the passing - # branch and the assertion "this is absent" is satisfied by the matcher giving - # up. Every negative claim in this file rests on the three-way split. - local _rc=0 - awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac -} - # The function under test redirects a command's stderr into a report file and # decides which artifact to write from it. cozytest.sh runs under `set -x`, so # the tracer's own line for that command lands on the very stderr being diff --git a/hack/run-kubernetes-node-join_test.bats b/hack/run-kubernetes-node-join_test.bats index 42dbe2f11..27bf9bdb9 100644 --- a/hack/run-kubernetes-node-join_test.bats +++ b/hack/run-kubernetes-node-join_test.bats @@ -300,36 +300,6 @@ assert_file_contains() { return 1 } -# The sibling suites' three-way split, and here for the same reason. A bare -# `! grep -q` succeeds on a file that does not exist, which is -# indistinguishable from the file existing without the pattern; and awk exits 1 -# for "no line matched" and 2 for "I could not evaluate this", so folding those -# together lets a broken matcher satisfy a negative assertion. A positive -# assertion fails loudly when its matcher breaks and gets fixed; this one would -# go green and stay green. -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - local _rc=0 - - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac -} - # The ceilings both budget guards below price their sums from, read out of the # helper rather than restated in each. They are the same numbers held against # two different bounds -- the phase budget from below, the operation window from diff --git a/hack/run-kubernetes-runner-canary_test.bats b/hack/run-kubernetes-runner-canary_test.bats index 7e2f85f90..65c0626f8 100644 --- a/hack/run-kubernetes-runner-canary_test.bats +++ b/hack/run-kubernetes-runner-canary_test.bats @@ -67,37 +67,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - # A missing file must fail rather than vacuously pass: a bare `! grep -q` - # succeeds on an unreadable path, which is indistinguishable from "the file - # exists and does not carry the label". - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - # Branched on awk's exact status. awk exits 1 for "no line matched" and 2 for - # "I could not evaluate this"; folded together, a negative assertion is - # satisfied by its own matcher giving up, which is the direction that goes - # green and stays green. - local _rc=0 - awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - cat "${file}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac -} - # The clock the arms are timed by, replaced by a scripted sequence. # # Real durations are the one thing this suite must not depend on: they are set diff --git a/hack/run-kubernetes-runner-cpu_test.bats b/hack/run-kubernetes-runner-cpu_test.bats index 4eca0a3fc..eee4da69a 100644 --- a/hack/run-kubernetes-runner-cpu_test.bats +++ b/hack/run-kubernetes-runner-cpu_test.bats @@ -58,37 +58,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - # A missing file must fail rather than vacuously pass: a bare `! grep -q` - # succeeds on an unreadable path, which is indistinguishable from "the file - # exists and does not carry the label". - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - # Branched on awk's exact status. awk exits 1 for "no line matched" and 2 for - # "I could not evaluate this"; folded together, a negative assertion is - # satisfied by its own matcher giving up, which is the direction that goes - # green and stays green. - local _rc=0 - awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - cat "${file}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac -} - # The functions under test redirect a command's stderr into a report file and # decide which artifact to write from it. cozytest.sh runs under `set -x`, so the # tracer's own line for that command lands on the very stderr being redirected, diff --git a/hack/run-kubernetes-sandbox-cpu-time_test.bats b/hack/run-kubernetes-sandbox-cpu-time_test.bats index 0cd1bace4..c6f3028ca 100644 --- a/hack/run-kubernetes-sandbox-cpu-time_test.bats +++ b/hack/run-kubernetes-sandbox-cpu-time_test.bats @@ -83,36 +83,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - # A missing file must fail rather than vacuously pass: a bare `! grep -q` - # succeeds on an unreadable path, which is indistinguishable from "the file - # exists and does not carry the label". - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - # Branched on awk's exact status. awk exits 1 for "no line matched" and 2 for - # "I could not evaluate this"; folded together, a negative assertion is - # satisfied by its own matcher giving up, which is the direction that goes - # green and stays green. - local _rc=0 - awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac -} - # The function under test redirects a command's stderr into a report file and # decides which artifact to write from it. cozytest.sh runs under `set -x`, so # the tracer's own line for that command lands on the very stderr being diff --git a/hack/run-kubernetes-serial-console_test.bats b/hack/run-kubernetes-serial-console_test.bats index 3dd91773d..4c77bd269 100644 --- a/hack/run-kubernetes-serial-console_test.bats +++ b/hack/run-kubernetes-serial-console_test.bats @@ -129,23 +129,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - # A missing file must fail rather than vacuously pass: awk exits 2 on an - # unreadable path, which is indistinguishable from "no line matched" once the - # status is folded into an if. - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - if awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}"; then - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi -} - # The function under test redirects a command's stderr into a report file and # decides which artifact to write from it. cozytest.sh runs under `set -x`, so # the tracer's own line for that command is emitted on the very stderr being diff --git a/hack/run-kubernetes-talos-diagnostics_test.bats b/hack/run-kubernetes-talos-diagnostics_test.bats index c5411837f..5ce8eef66 100644 --- a/hack/run-kubernetes-talos-diagnostics_test.bats +++ b/hack/run-kubernetes-talos-diagnostics_test.bats @@ -114,16 +114,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - if awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}"; then - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi -} - @test "worker VMI rows retain names whose default interface has no IP" { . hack/e2e-chainsaw/_lib/run-kubernetes.sh rows=$(printf '%s\n' '{"items":[{"metadata":{"name":"worker-a"},"status":{"interfaces":[{"name":"default","ipAddress":"10.244.1.92"}]}},{"metadata":{"name":"worker-b"},"status":{"interfaces":[{"name":"other","ipAddress":"192.0.2.1"}]}}]}' | cozy_tenant_worker_vmi_rows) diff --git a/hack/run-kubernetes-thread-cpu_test.bats b/hack/run-kubernetes-thread-cpu_test.bats index 4d9f7c90b..b37c749e5 100644 --- a/hack/run-kubernetes-thread-cpu_test.bats +++ b/hack/run-kubernetes-thread-cpu_test.bats @@ -85,37 +85,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - # A missing file must fail rather than vacuously pass: a bare `! grep -q` - # succeeds on an unreadable path, which is indistinguishable from "the file - # exists and does not carry the label". - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - # Branched on awk's exact status. awk exits 1 for "no line matched" and 2 for - # "I could not evaluate this"; folded into an `if`, both land in the passing - # branch and the assertion "this pattern is absent" is satisfied by the - # matcher giving up. Every claim in this file about something NOT appearing - # rests on the three-way split below. - local _rc=0 - awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac -} - # The function under test redirects a command's stderr into a report file and # decides which artifact to write from it. cozytest.sh runs under `set -x`, so # the tracer's own line for that command lands on the very stderr being diff --git a/hack/runner-identity.bats b/hack/runner-identity.bats index acfa36e32..ea42444f5 100644 --- a/hack/runner-identity.bats +++ b/hack/runner-identity.bats @@ -83,37 +83,6 @@ assert_file_contains() { return 1 } -assert_file_lacks_pattern() { - local pattern="$1" - local file="$2" - - # A missing file must fail rather than vacuously pass: a bare `! grep -q` - # succeeds on an unreadable path, which is indistinguishable from "the file - # exists and does not carry the pattern". - if [ ! -f "${file}" ]; then - printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 - return 1 - fi - # Branched on awk's exact status. awk exits 1 for "no line matched" and 2 for - # "I could not evaluate this"; folded together, a negative assertion is - # satisfied by its own matcher giving up, which is the direction that goes - # green and stays green. - local _rc=0 - awk -v pattern="${pattern}" '$0 ~ pattern { found = 1 } END { exit found ? 0 : 1 }' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - cat "${file}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac -} - # The shape sizing object, as the `shapeConfig` subpath serves it: a flat object of # sizing numbers with one string in it, pretty-printed because the service is free # to answer either way and the reader buffers its whole input for that reason. diff --git a/hack/test-helper_test.bats b/hack/test-helper_test.bats new file mode 100644 index 000000000..d02580099 --- /dev/null +++ b/hack/test-helper_test.bats @@ -0,0 +1,85 @@ +#!/usr/bin/env bats + +load test_helper + +# A separate shell keeps errexit active inside the assertion, unlike calling +# the function directly through Bats' run helper. +run_lacks_pattern() { + run bash -eu -c ' + . hack/test_helper.bash + assert_file_lacks_pattern "$1" "$2" + printf "checked\n" + ' _ "$@" +} + +@test "a valid non-matching pattern passes under errexit" { + printf 'safe content\n' >"$BATS_TEST_TMPDIR/artifact" + + run_lacks_pattern 'secret:|secretName:' "$BATS_TEST_TMPDIR/artifact" + + [ "$status" -eq 0 ] + [ "$output" = checked ] +} + +@test "an empty readable file passes for a valid pattern" { + : >"$BATS_TEST_TMPDIR/artifact" + + run_lacks_pattern 'secret:' "$BATS_TEST_TMPDIR/artifact" + + [ "$status" -eq 0 ] + [ "$output" = checked ] +} + +@test "a matching pattern fails the assertion" { + printf 'safe content\nsecretName: credential\n' >"$BATS_TEST_TMPDIR/artifact" + + run_lacks_pattern 'secret:|secretName:' "$BATS_TEST_TMPDIR/artifact" + + [ "$status" -eq 1 ] + [[ "$output" == *'not to match: secret:|secretName:'* ]] +} + +@test "an invalid pattern fails on a populated file" { + printf 'safe content\n' >"$BATS_TEST_TMPDIR/artifact" + + run_lacks_pattern '[' "$BATS_TEST_TMPDIR/artifact" + + [ "$status" -eq 1 ] + [[ "$output" == *'could not evaluate pattern ['* ]] +} + +@test "an invalid pattern fails even on an empty file" { + : >"$BATS_TEST_TMPDIR/artifact" + + run_lacks_pattern '[' "$BATS_TEST_TMPDIR/artifact" + + [ "$status" -eq 1 ] + [[ "$output" == *'could not evaluate pattern ['* ]] +} + +@test "a missing artifact fails rather than passing as no match" { + run_lacks_pattern 'secret:' "$BATS_TEST_TMPDIR/missing" + + [ "$status" -eq 1 ] + [[ "$output" == *'to exist so it could be checked'* ]] +} + +@test "a directory is not an artifact file" { + run_lacks_pattern 'secret:' "$BATS_TEST_TMPDIR" + + [ "$status" -eq 1 ] + [[ "$output" == *'to exist so it could be checked'* ]] +} + +@test "an unreadable artifact fails rather than passing as no match" { + printf 'safe content\n' >"$BATS_TEST_TMPDIR/artifact" + chmod 000 "$BATS_TEST_TMPDIR/artifact" + if [ -r "$BATS_TEST_TMPDIR/artifact" ]; then + skip 'this user can read files without read permission' + fi + + run_lacks_pattern 'secret:' "$BATS_TEST_TMPDIR/artifact" + + [ "$status" -eq 1 ] + [[ "$output" == *'could not evaluate pattern secret:'* ]] +} diff --git a/hack/test_helper.bash b/hack/test_helper.bash index f3944c9c0..f5f74a8b1 100644 --- a/hack/test_helper.bash +++ b/hack/test_helper.bash @@ -9,3 +9,32 @@ strict_setup() { setup() { strict_setup } + +assert_file_lacks_pattern() { + local pattern="$1" + local file="$2" + local _rc=0 + + if [ ! -f "${file}" ]; then + printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 + return 1 + fi + # Validate the regex even for an empty file. Only status 1 means no match; + # keeping awk in an OR-list lets that normal result survive errexit. + awk -v pattern="${pattern}" ' + BEGIN { match("", pattern) } + $0 ~ pattern { found = 1 } + END { exit found ? 0 : 1 } + ' "${file}" || _rc=$? + case "${_rc}" in + 0) + printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 + return 1 + ;; + 1) return 0 ;; + *) + printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 + return 1 + ;; + esac +} From 8d40fc8e00ef099fd827d643d2485cfa7d316b18 Mon Sep 17 00:00:00 2001 From: Ahmed Kamal Date: Sat, 31 Oct 2026 23:36:26 +0200 Subject: [PATCH 035/518] fix(tests): honor singular Helm test template overrides Signed-off-by: Ahmed Kamal Assisted-by: LLM --- hack/check-helm-document-indices.bats | 33 +++++++++++++++++++++++++++ hack/check-helm-document-indices.py | 4 +++- 2 files changed, 36 insertions(+), 1 deletion(-) diff --git a/hack/check-helm-document-indices.bats b/hack/check-helm-document-indices.bats index f797d0b7f..ca84af677 100644 --- a/hack/check-helm-document-indices.bats +++ b/hack/check-helm-document-indices.bats @@ -106,6 +106,7 @@ suite: assertion override templates: [templates/single.yaml, templates/multi.yaml] tests: - it: checks only the single-document template + template: templates/multi.yaml documentIndex: 0 asserts: - template: templates/single.yaml @@ -139,3 +140,35 @@ EOF fi grep -q 'resource_test.yaml:6: use documentSelector' output } + +@test "rejects a singular test template overriding a single-document suite" { + write_suite single + sed -i '/ - it:/a\ template: templates/multi.yaml' packages/apps/example/tests/resource_test.yaml + if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then + echo "accepted a positional selector on the test template override" >&2 + false + fi + grep -q 'use documentSelector.*templates/multi.yaml' output +} + +@test "allows a singular test template selecting one document" { + write_suite multi + sed -i '/ - it:/a\ template: templates/single.yaml' packages/apps/example/tests/resource_test.yaml + python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" +} + +@test "plural test templates take precedence over the singular test template" { + write_suite multi + sed -i '/ - it:/a\ templates: [templates/single.yaml]\n template: templates/multi.yaml' packages/apps/example/tests/resource_test.yaml + python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" +} + +@test "an empty plural test override falls back to the singular template" { + write_suite single + sed -i '/ - it:/a\ templates: []\n template: templates/multi.yaml' packages/apps/example/tests/resource_test.yaml + if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then + echo "accepted a positional selector after an empty plural override" >&2 + false + fi + grep -q 'use documentSelector.*templates/multi.yaml' output +} diff --git a/hack/check-helm-document-indices.py b/hack/check-helm-document-indices.py index bf9c56f47..212a50964 100644 --- a/hack/check-helm-document-indices.py +++ b/hack/check-helm-document-indices.py @@ -50,7 +50,9 @@ def check(base): if not suite: continue for test in suite.get("tests", []): - templates = test.get("templates", suite.get("templates", ["templates/*"])) + templates = test.get("templates") or ( + [test["template"]] if test.get("template") + else suite.get("templates", ["templates/*"])) for assertion in test.get("asserts", []): index = assertion.get("documentIndex", test.get("documentIndex", {})) number = index.get("line") From e6651ce222c43b84749998257ff3ab7cf148b2ea Mon Sep 17 00:00:00 2001 From: Ahmed Kamal Date: Sat, 31 Oct 2026 23:40:51 +0200 Subject: [PATCH 036/518] fix(tests): separate awk errors from match results Signed-off-by: Ahmed Kamal Assisted-by: LLM --- hack/test-helper_test.bats | 21 +++++++++++++++++++-- hack/test_helper.bash | 31 ++++++++++++++----------------- 2 files changed, 33 insertions(+), 19 deletions(-) diff --git a/hack/test-helper_test.bats b/hack/test-helper_test.bats index d02580099..902ed8af9 100644 --- a/hack/test-helper_test.bats +++ b/hack/test-helper_test.bats @@ -6,10 +6,10 @@ load test_helper # the function directly through Bats' run helper. run_lacks_pattern() { run bash -eu -c ' - . hack/test_helper.bash + . "$3" assert_file_lacks_pattern "$1" "$2" printf "checked\n" - ' _ "$@" + ' _ "$@" "$BATS_TEST_DIRNAME/test_helper.bash" } @test "a valid non-matching pattern passes under errexit" { @@ -83,3 +83,20 @@ run_lacks_pattern() { [ "$status" -eq 1 ] [[ "$output" == *'could not evaluate pattern secret:'* ]] } + +@test "a matcher error with status one cannot pass as no match" { + printf 'safe content\n' >"$BATS_TEST_TMPDIR/artifact" + mkdir "$BATS_TEST_TMPDIR/bin" + cat >"$BATS_TEST_TMPDIR/bin/awk" <<'SH' +#!/bin/sh +printf '0\n' +printf 'awk: evaluation failed\n' >&2 +exit 1 +SH + chmod +x "$BATS_TEST_TMPDIR/bin/awk" + + PATH="$BATS_TEST_TMPDIR/bin:$PATH" run_lacks_pattern 'secret:' "$BATS_TEST_TMPDIR/artifact" + + [ "$status" -eq 1 ] + [[ "$output" == *'could not evaluate pattern secret:'* ]] +} diff --git a/hack/test_helper.bash b/hack/test_helper.bash index f5f74a8b1..592676b8c 100644 --- a/hack/test_helper.bash +++ b/hack/test_helper.bash @@ -13,28 +13,25 @@ setup() { assert_file_lacks_pattern() { local pattern="$1" local file="$2" - local _rc=0 + local found if [ ! -f "${file}" ]; then printf 'expected %s to exist so it could be checked for: %s\n' "${file}" "${pattern}" >&2 return 1 fi - # Validate the regex even for an empty file. Only status 1 means no match; - # keeping awk in an OR-list lets that normal result survive errexit. - awk -v pattern="${pattern}" ' + # Validate even an empty file, and keep the match result separate from awk's + # status: some implementations also use status 1 for evaluation errors. + if ! found=$(awk -v pattern="${pattern}" ' BEGIN { match("", pattern) } $0 ~ pattern { found = 1 } - END { exit found ? 0 : 1 } - ' "${file}" || _rc=$? - case "${_rc}" in - 0) - printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 - return 1 - ;; - 1) return 0 ;; - *) - printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 - return 1 - ;; - esac + END { print found ? 1 : 0 } + ' "${file}"); then + printf 'awk could not evaluate pattern %s against %s\n' "${pattern}" "${file}" >&2 + return 1 + fi + if [ "${found}" != 0 ]; then + printf 'expected %s not to match: %s\n' "${file}" "${pattern}" >&2 + return 1 + fi + return 0 } From 2757e2c60ad35e40d6e378a6049b8a20f6242ea5 Mon Sep 17 00:00:00 2001 From: Ahmed Kamal Date: Sun, 1 Nov 2026 01:23:55 +0200 Subject: [PATCH 037/518] test(tests): tighten Helm selector guard scope and coverage Signed-off-by: Ahmed Kamal Assisted-by: LLM --- .github/workflows/pull-requests.yaml | 9 +- hack/check-helm-document-indices.bats | 174 +++++++++++++++++- hack/check-helm-document-indices.py | 41 +++-- .../apps/kafka/tests/jvm_sizing_test.yaml | 4 +- 4 files changed, 197 insertions(+), 31 deletions(-) diff --git a/.github/workflows/pull-requests.yaml b/.github/workflows/pull-requests.yaml index 96dce2056..1f44fc287 100644 --- a/.github/workflows/pull-requests.yaml +++ b/.github/workflows/pull-requests.yaml @@ -364,10 +364,6 @@ jobs: with: path: _out/envtest key: envtest-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('Makefile') }} - - name: Check new Helm document indices - env: - BASE_REF: ${{ github.event.pull_request.base.ref }} - run: python3 hack/check-helm-document-indices.py "$(git merge-base HEAD "origin/$BASE_REF")" # -k is required, not cosmetic: unit-tests and test-controllers are two # independent goals on the same command line, and without -k make stops # handing out new work the moment any unit-tests prerequisite fails — @@ -380,6 +376,11 @@ jobs: - name: Run Bats unit tests for docs-only changes if: needs.plan.outputs.code == 'false' run: make bats-unit-tests + - name: Check new Helm document indices + if: ${{ !cancelled() }} + env: + BASE_REF: ${{ github.event.pull_request.base.ref }} + run: python3 hack/check-helm-document-indices.py "$(git merge-base HEAD "origin/$BASE_REF")" - name: Upload unit test report if: always() uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 diff --git a/hack/check-helm-document-indices.bats b/hack/check-helm-document-indices.bats index ca84af677..5188f5b26 100644 --- a/hack/check-helm-document-indices.bats +++ b/hack/check-helm-document-indices.bats @@ -66,7 +66,8 @@ EOF write_suite multi git -c user.name=Test -c user.email=test@example.org commit -qm existing base=$(git rev-parse HEAD) - sed -i 's/spec.missing/spec.other/' packages/apps/example/tests/resource_test.yaml + sed 's/spec.missing/spec.other/' packages/apps/example/tests/resource_test.yaml > changed.yaml + mv changed.yaml packages/apps/example/tests/resource_test.yaml python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" } @@ -133,17 +134,37 @@ EOF @test "rejects a positional selector whose value is on a separate line" { write_suite multi - sed -i 's/documentIndex: 0/documentIndex:\n 0/' packages/apps/example/tests/resource_test.yaml + cat > packages/apps/example/tests/resource_test.yaml <<'EOF' +suite: multiline index +templates: [templates/multi.yaml] +tests: + - it: checks a resource + documentIndex: + 0 + asserts: + - notExists: + path: spec.missing +EOF if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then echo "accepted a multiline positional selector" >&2 false fi - grep -q 'resource_test.yaml:6: use documentSelector' output + grep -q 'resource_test.yaml:5: use documentSelector' output } @test "rejects a singular test template overriding a single-document suite" { write_suite single - sed -i '/ - it:/a\ template: templates/multi.yaml' packages/apps/example/tests/resource_test.yaml + cat > packages/apps/example/tests/resource_test.yaml <<'EOF' +suite: test template override +templates: [templates/single.yaml] +tests: + - it: checks a resource + template: templates/multi.yaml + documentIndex: 0 + asserts: + - notExists: + path: spec.missing +EOF if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then echo "accepted a positional selector on the test template override" >&2 false @@ -153,22 +174,161 @@ EOF @test "allows a singular test template selecting one document" { write_suite multi - sed -i '/ - it:/a\ template: templates/single.yaml' packages/apps/example/tests/resource_test.yaml + cat > packages/apps/example/tests/resource_test.yaml <<'EOF' +suite: test template override +templates: [templates/multi.yaml] +tests: + - it: checks a resource + template: templates/single.yaml + documentIndex: 0 + asserts: + - notExists: + path: spec.missing +EOF python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" } @test "plural test templates take precedence over the singular test template" { write_suite multi - sed -i '/ - it:/a\ templates: [templates/single.yaml]\n template: templates/multi.yaml' packages/apps/example/tests/resource_test.yaml + cat > packages/apps/example/tests/resource_test.yaml <<'EOF' +suite: test template override +templates: [templates/multi.yaml] +tests: + - it: checks a resource + templates: [templates/single.yaml] + template: templates/multi.yaml + documentIndex: 0 + asserts: + - notExists: + path: spec.missing +EOF python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" } @test "an empty plural test override falls back to the singular template" { write_suite single - sed -i '/ - it:/a\ templates: []\n template: templates/multi.yaml' packages/apps/example/tests/resource_test.yaml + cat > packages/apps/example/tests/resource_test.yaml <<'EOF' +suite: test template override +templates: [templates/single.yaml] +tests: + - it: checks a resource + templates: [] + template: templates/multi.yaml + documentIndex: 0 + asserts: + - notExists: + path: spec.missing +EOF if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then echo "accepted a positional selector after an empty plural override" >&2 false fi grep -q 'use documentSelector.*templates/multi.yaml' output } + +@test "does not check vendored chart suites" { + mkdir -p packages/apps/example/charts/sub/templates packages/apps/example/charts/sub/tests + cp packages/apps/example/templates/multi.yaml packages/apps/example/charts/sub/templates/ + cat > packages/apps/example/charts/sub/tests/resource_test.yaml <<'EOF' +suite: upstream resource selection +templates: [templates/multi.yaml] +tests: + - it: checks an upstream resource + documentIndex: 0 + asserts: + - notExists: + path: spec.missing +EOF + git add . + python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" +} + +@test "a new test-level index overrides existing assertion indices" { + cat > packages/apps/example/tests/resource_test.yaml <<'EOF' +suite: index inheritance +templates: [templates/multi.yaml] +tests: + - it: checks a resource + asserts: + - notExists: + path: spec.missing + documentIndex: 0 + - notExists: + path: spec.other + documentIndex: 0 +EOF + git add . + git -c user.name=Test -c user.email=test@example.org commit -qm existing + base=$(git rev-parse HEAD) + cat > packages/apps/example/tests/resource_test.yaml <<'EOF' +suite: index inheritance +templates: [templates/multi.yaml] +tests: + - it: checks a resource + documentIndex: 1 + asserts: + - notExists: + path: spec.missing + documentIndex: 0 + - notExists: + path: spec.other + documentIndex: 0 +EOF + if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then + echo "accepted a new inherited positional selector" >&2 + false + fi + grep -q 'resource_test.yaml:5: use documentSelector' output +} + +@test "allows a pure suite rename with existing indices" { + write_suite multi + git -c user.name=Test -c user.email=test@example.org commit -qm existing + base=$(git rev-parse HEAD) + git mv packages/apps/example/tests/resource_test.yaml packages/apps/example/tests/renamed_test.yaml + python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" +} + +@test "checks added index lines in a renamed suite" { + write_suite multi + git -c user.name=Test -c user.email=test@example.org commit -qm existing + base=$(git rev-parse HEAD) + git mv packages/apps/example/tests/resource_test.yaml packages/apps/example/tests/renamed_test.yaml + cat > packages/apps/example/tests/renamed_test.yaml <<'EOF' +suite: resource selection +templates: + - templates/multi.yaml +tests: + - it: checks a resource + documentIndex: 1 + asserts: + - notExists: + path: spec.missing +EOF + if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then + echo "accepted a new positional selector in a renamed suite" >&2 + false + fi + grep -q 'renamed_test.yaml:6: use documentSelector' output +} + +@test "an empty suite template list includes nested chart templates" { + mkdir packages/apps/example/templates/nested + mv packages/apps/example/templates/multi.yaml packages/apps/example/templates/nested/ + cat > packages/apps/example/tests/resource_test.yaml <<'EOF' +suite: default templates +templates: [] +tests: + - it: checks a resource + documentIndex: 0 + asserts: + - notExists: + path: spec.missing +EOF + git add . + if python3 "$REPO_ROOT/hack/check-helm-document-indices.py" "$base" > output 2>&1; then + echo "accepted a positional selector with default templates" >&2 + false + fi + grep -q 'use documentSelector.*templates/nested/multi.yaml' output +} diff --git a/hack/check-helm-document-indices.py b/hack/check-helm-document-indices.py index 212a50964..4fd36b06f 100644 --- a/hack/check-helm-document-indices.py +++ b/hack/check-helm-document-indices.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 """Reject added positional selectors on multi-resource Helm templates. -Usage: hack/check-helm-document-indices.py +Usage: python3 hack/check-helm-document-indices.py Requires mikefarah/yq, like the chart test tooling. Existing indices are left alone. This static check counts --- separated blocks with a top-level kind; it does not evaluate conditional documents or resources emitted by helpers/ranges. @@ -20,22 +20,25 @@ def git(*args): def check(base): root = Path(git("rev-parse", "--show-toplevel").strip()) - paths = git("diff", "--name-only", "--diff-filter=ACMR", base, "--", - "packages/*/*/tests/*_test.yaml").splitlines() + additions = {} + path = None + for line in git("diff", "--no-ext-diff", "-M", "--unified=0", base, "--", + ":(glob)packages/*/*/tests/*_test.yaml").splitlines(): + if line.startswith("diff --git "): + path = None + elif line.startswith("+++ b/"): + path = line[6:] + additions[path] = set() + elif line.startswith("@@"): + number = int(re.search(r"\+(\d+)", line).group(1)) + elif path and line.startswith("+"): + if "documentIndex" in line: + additions[path].add(number) + number += 1 + elif line.startswith(" "): + number += 1 errors = [] - for path in paths: - added = set() - for line in git("diff", "--no-ext-diff", "--unified=0", base, "--", path).splitlines(): - if line.startswith("@@"): - number = int(re.search(r"\+(\d+)", line).group(1)) - elif line.startswith(("+++", "---", "diff ", "index ", "\\")): - continue - elif line.startswith("+"): - if "documentIndex" in line: - added.add(number) - number += 1 - elif line.startswith(" "): - number += 1 + for path, added in additions.items(): if not added: continue @@ -52,9 +55,9 @@ def check(base): for test in suite.get("tests", []): templates = test.get("templates") or ( [test["template"]] if test.get("template") - else suite.get("templates", ["templates/*"])) + else suite.get("templates") or ["templates/**/*"]) for assertion in test.get("asserts", []): - index = assertion.get("documentIndex", test.get("documentIndex", {})) + index = test.get("documentIndex") or assertion.get("documentIndex", {}) number = index.get("line") if number not in added: continue @@ -77,5 +80,5 @@ def check(base): if __name__ == "__main__": if len(sys.argv) != 2: - sys.exit("usage: check-helm-document-indices.py ") + sys.exit("usage: python3 hack/check-helm-document-indices.py ") sys.exit(check(sys.argv[1])) diff --git a/packages/apps/kafka/tests/jvm_sizing_test.yaml b/packages/apps/kafka/tests/jvm_sizing_test.yaml index 26a065d17..f4569a1e2 100644 --- a/packages/apps/kafka/tests/jvm_sizing_test.yaml +++ b/packages/apps/kafka/tests/jvm_sizing_test.yaml @@ -27,7 +27,9 @@ tests: # kafka-run-class.sh then drops its G1 settings. - it: leaves the node pool JVM options to Strimzi and Kafka template: templates/migration-hook.yaml - documentIndex: 3 + documentSelector: + path: kind + value: Job asserts: - notMatchRegex: path: spec.template.spec.containers[0].args[0] From 342283cfc781e343760148fb85775f914d114bcc Mon Sep 17 00:00:00 2001 From: Ahmed Kamal Date: Sun, 1 Nov 2026 02:17:57 +0200 Subject: [PATCH 038/518] docs(tests): document Helm selector guard limits Signed-off-by: Ahmed Kamal Assisted-by: LLM --- hack/check-helm-document-indices.py | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/hack/check-helm-document-indices.py b/hack/check-helm-document-indices.py index 4fd36b06f..dd75e799a 100644 --- a/hack/check-helm-document-indices.py +++ b/hack/check-helm-document-indices.py @@ -2,9 +2,7 @@ """Reject added positional selectors on multi-resource Helm templates. Usage: python3 hack/check-helm-document-indices.py -Requires mikefarah/yq, like the chart test tooling. Existing indices are left -alone. This static check counts --- separated blocks with a top-level kind; -it does not evaluate conditional documents or resources emitted by helpers/ranges. +Requires mikefarah/yq, like the chart test tooling. """ import json @@ -19,6 +17,11 @@ def git(*args): def check(base): + """Check added indices, leaving existing ones alone. + + This static check counts --- separated blocks with a top-level kind; it does + not evaluate conditional documents or resources emitted by helpers/ranges. + """ root = Path(git("rev-parse", "--show-toplevel").strip()) additions = {} path = None From 21fe9d4195848d992fdcb1d67f03a263d1fc716b Mon Sep 17 00:00:00 2001 From: Ahmed Kamal Date: Sun, 1 Nov 2026 02:18:13 +0200 Subject: [PATCH 039/518] docs(tests): document negative file assertion contract Signed-off-by: Ahmed Kamal Assisted-by: LLM --- hack/test_helper.bash | 2 ++ 1 file changed, 2 insertions(+) diff --git a/hack/test_helper.bash b/hack/test_helper.bash index 592676b8c..bbbef8de8 100644 --- a/hack/test_helper.bash +++ b/hack/test_helper.bash @@ -10,6 +10,8 @@ setup() { strict_setup } +# Only a successful matcher explicitly reporting no match passes; missing, +# non-regular or unreadable files and matcher errors fail closed. assert_file_lacks_pattern() { local pattern="$1" local file="$2" From 9d33037566a35979ecdb95ae256f6f209afb078d Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Sun, 25 Oct 2026 16:40:54 +0300 Subject: [PATCH 040/518] fix(workloadmonitor): keep a bucket not ready until its claim is provisioned A bucket WorkloadMonitor has no pods and minReplicas 0, so it stayed operational, and the Bucket app stayed Ready, while its BucketClaim was never provisioned. A tenant whose nearest S3 ancestor runs no SeaweedFS gets a claim for a bucket class that does not exist, and the dashboard showed a ready bucket that was never created. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> --- api/v1alpha1/workloadmonitor_types.go | 4 + .../controller/workloadmonitor_controller.go | 35 ++++- .../workloadmonitor_controller_test.go | 132 ++++++++++++++++++ 3 files changed, 168 insertions(+), 3 deletions(-) diff --git a/api/v1alpha1/workloadmonitor_types.go b/api/v1alpha1/workloadmonitor_types.go index 0a4aa04cc..b5089652a 100644 --- a/api/v1alpha1/workloadmonitor_types.go +++ b/api/v1alpha1/workloadmonitor_types.go @@ -39,6 +39,10 @@ type WorkloadMonitorSpec struct { // that is not operational because a DataVolume it selects is not ready. const WorkloadMonitorReasonDataVolumeNotReady = "DataVolumeNotReady" +// WorkloadMonitorReasonBucketClaimNotReady is the status Reason of a monitor +// that is not operational because a BucketClaim it selects is not provisioned. +const WorkloadMonitorReasonBucketClaimNotReady = "BucketClaimNotReady" + // WorkloadMonitorStatus defines the observed state of WorkloadMonitor type WorkloadMonitorStatus struct { // Operational indicates if the workload meets all operational requirements diff --git a/internal/controller/workloadmonitor_controller.go b/internal/controller/workloadmonitor_controller.go index 984ab7b21..d5d30e4e4 100644 --- a/internal/controller/workloadmonitor_controller.go +++ b/internal/controller/workloadmonitor_controller.go @@ -374,6 +374,25 @@ func (r *WorkloadMonitorReconciler) isBucketClaimReady(bc *cosiv1alpha1.BucketCl return bc.Status.BucketReady } +// bucketClaimsMessage names every BucketClaim that has not been provisioned, +// with its bucket class, and is empty when there is none. +func (r *WorkloadMonitorReconciler) bucketClaimsMessage(claims []cosiv1alpha1.BucketClaim) string { + var pending []string + for i := range claims { + bc := &claims[i] + if r.isBucketClaimReady(bc) { + continue + } + entry := fmt.Sprintf("BucketClaim %s is not provisioned", bc.Name) + if bc.Spec.BucketClassName != "" { + entry = fmt.Sprintf("BucketClaim %s of bucket class %s is not provisioned", bc.Name, bc.Spec.BucketClassName) + } + pending = append(pending, entry) + } + sort.Strings(pending) + return strings.Join(pending, "; ") +} + // isServiceReady checks if the service has an external IP bound func (r *WorkloadMonitorReconciler) isServiceReady(svc *corev1.Service) bool { return len(svc.Status.LoadBalancer.Ingress) > 0 @@ -1157,6 +1176,8 @@ func (r *WorkloadMonitorReconciler) Reconcile(ctx context.Context, req ctrl.Requ } } + bucketsMessage := r.bucketClaimsMessage(bucketClaimList.Items) + // Update WorkloadMonitor status based on observed pods monitor.Status.ObservedReplicas = observedReplicas monitor.Status.AvailableReplicas = availableReplicas @@ -1178,15 +1199,23 @@ func (r *WorkloadMonitorReconciler) Reconcile(ctx context.Context, req ctrl.Requ if fresh.Spec.MinReplicas != nil && availableReplicas < *fresh.Spec.MinReplicas { fresh.Status.Operational = pointer.Bool(false) } - // Only the DataVolume check writes Message and Reason, so the stored ones - // are the last DataVolume verdict: kept while no reader is installed yet, so a - // controller restart does not report a stuck disk as ready. + // A stored DataVolume verdict is the last one: kept while no reader is + // installed yet, so a controller restart does not report a stuck disk as + // ready. BucketClaims are listed on every pass, so their verdict is + // recomputed and only fills fields no DataVolume verdict holds. if dataVolumesRead { fresh.Status.Message = dataVolumesMessage fresh.Status.Reason = "" if dataVolumesMessage != "" { fresh.Status.Reason = cozyv1alpha1.WorkloadMonitorReasonDataVolumeNotReady } + } else if fresh.Status.Reason == cozyv1alpha1.WorkloadMonitorReasonBucketClaimNotReady { + fresh.Status.Message = "" + fresh.Status.Reason = "" + } + if fresh.Status.Message == "" && bucketsMessage != "" { + fresh.Status.Message = bucketsMessage + fresh.Status.Reason = cozyv1alpha1.WorkloadMonitorReasonBucketClaimNotReady } if fresh.Status.Message != "" { fresh.Status.Operational = pointer.Bool(false) diff --git a/internal/controller/workloadmonitor_controller_test.go b/internal/controller/workloadmonitor_controller_test.go index 353066c4b..f8c761da5 100644 --- a/internal/controller/workloadmonitor_controller_test.go +++ b/internal/controller/workloadmonitor_controller_test.go @@ -2975,3 +2975,135 @@ func TestReconcileBucketApp_LegacySizeWinsOverOwnerlessNewWorkload(t *testing.T) t.Errorf("size = %s, want the legacy 689Gi over the stale 10Gi", got.String()) } } + +func newBucketClaim(name string, labels map[string]string, class string, ready bool) *cosiv1alpha1.BucketClaim { + return &cosiv1alpha1.BucketClaim{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "default", Labels: labels}, + Spec: cosiv1alpha1.BucketClaimSpec{ + BucketClassName: class, + Protocols: []cosiv1alpha1.Protocol{cosiv1alpha1.ProtocolS3}, + }, + Status: cosiv1alpha1.BucketClaimStatus{BucketReady: ready}, + } +} + +func reconcileBucketMonitor(t *testing.T, c client.Client, s *runtime.Scheme, withReader bool) *cozyv1alpha1.WorkloadMonitor { + t.Helper() + reconciler := &WorkloadMonitorReconciler{Client: c, Scheme: s} + if withReader { + reconciler.DataVolumeReader = c + } + req := reconcile.Request{NamespacedName: types.NamespacedName{Name: "my-bucket", Namespace: "default"}} + if _, err := reconciler.Reconcile(context.TODO(), req); err != nil { + t.Fatalf("Reconcile returned error: %v", err) + } + updated := &cozyv1alpha1.WorkloadMonitor{} + if err := c.Get(context.TODO(), req.NamespacedName, updated); err != nil { + t.Fatalf("Failed to get updated WorkloadMonitor: %v", err) + } + if updated.Status.Operational == nil { + t.Fatal("Expected Operational to be set, got nil") + } + return updated +} + +func newBucketMonitorClient(s *runtime.Scheme, objs ...client.Object) client.Client { + monitor := &cozyv1alpha1.WorkloadMonitor{ + ObjectMeta: metav1.ObjectMeta{Name: "my-bucket", Namespace: "default"}, + Spec: cozyv1alpha1.WorkloadMonitorSpec{ + Kind: "bucket", + Type: "s3", + Selector: map[string]string{"app.kubernetes.io/instance": "my-bucket"}, + Replicas: ptr.To[int32](0), + MinReplicas: ptr.To[int32](0), + }, + } + return fake.NewClientBuilder(). + WithScheme(s). + WithObjects(append([]client.Object{monitor}, objs...)...). + WithStatusSubresource(monitor). + Build() +} + +func TestReconcile_BucketClaimDecidesOperational(t *testing.T) { + selected := map[string]string{"app.kubernetes.io/instance": "my-bucket"} + cases := []struct { + name string + claim *cosiv1alpha1.BucketClaim + operational bool + message string + }{ + {"provisioned", newBucketClaim("my-bucket", selected, "tenant-root", true), true, ""}, + {"not provisioned", newBucketClaim("my-bucket", selected, "tenant-root", false), false, "BucketClaim my-bucket of bucket class tenant-root is not provisioned"}, + {"not provisioned without a class", newBucketClaim("my-bucket", selected, "", false), false, "BucketClaim my-bucket is not provisioned"}, + {"outside the selector", newBucketClaim("other", map[string]string{"app.kubernetes.io/instance": "other"}, "tenant-root", false), true, ""}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + s := newTestScheme() + got := reconcileBucketMonitor(t, newBucketMonitorClient(s, tc.claim), s, true) + if *got.Status.Operational != tc.operational { + t.Errorf("Operational = %v, want %v", *got.Status.Operational, tc.operational) + } + if got.Status.Message != tc.message { + t.Errorf("Message = %q, want %q", got.Status.Message, tc.message) + } + wantReason := "" + if tc.message != "" { + wantReason = cozyv1alpha1.WorkloadMonitorReasonBucketClaimNotReady + } + if got.Status.Reason != wantReason { + t.Errorf("Reason = %q, want %q", got.Status.Reason, wantReason) + } + }) + } +} + +func TestReconcile_ProvisionedBucketClaimClearsTheVerdict(t *testing.T) { + for _, withReader := range []bool{true, false} { + t.Run(fmt.Sprintf("dataVolumeReader=%v", withReader), func(t *testing.T) { + s := newTestScheme() + claim := newBucketClaim("my-bucket", map[string]string{"app.kubernetes.io/instance": "my-bucket"}, "tenant-root", false) + c := newBucketMonitorClient(s, claim) + + got := reconcileBucketMonitor(t, c, s, withReader) + if *got.Status.Operational || got.Status.Reason != cozyv1alpha1.WorkloadMonitorReasonBucketClaimNotReady { + t.Fatalf("before provisioning: Operational = %v, Reason = %q", *got.Status.Operational, got.Status.Reason) + } + + stored := &cosiv1alpha1.BucketClaim{} + if err := c.Get(context.TODO(), types.NamespacedName{Name: "my-bucket", Namespace: "default"}, stored); err != nil { + t.Fatalf("Failed to get BucketClaim: %v", err) + } + stored.Status.BucketReady = true + if err := c.Update(context.TODO(), stored); err != nil { + t.Fatalf("Failed to update BucketClaim: %v", err) + } + + got = reconcileBucketMonitor(t, c, s, withReader) + if !*got.Status.Operational || got.Status.Message != "" || got.Status.Reason != "" { + t.Errorf("after provisioning: Operational = %v, Message = %q, Reason = %q", *got.Status.Operational, got.Status.Message, got.Status.Reason) + } + }) + } +} + +func TestReconcile_DataVolumeVerdictOutranksBucketClaim(t *testing.T) { + s := newTestScheme() + selected := map[string]string{"app.kubernetes.io/instance": "my-bucket"} + phase := "ImportInProgress" + c := newBucketMonitorClient(s, + newBucketClaim("my-bucket", selected, "tenant-root", false), + newDataVolume("my-bucket", selected, &phase), + ) + got := reconcileBucketMonitor(t, c, s, true) + if *got.Status.Operational { + t.Error("Operational = true, want false") + } + if got.Status.Reason != cozyv1alpha1.WorkloadMonitorReasonDataVolumeNotReady { + t.Errorf("Reason = %q, want %q", got.Status.Reason, cozyv1alpha1.WorkloadMonitorReasonDataVolumeNotReady) + } + if got.Status.Message != "DataVolume my-bucket is ImportInProgress" { + t.Errorf("Message = %q", got.Status.Message) + } +} From 7c1bd33b08a6cb690cf574de7fe392f15cae771d Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 2 Nov 2026 11:23:46 +0100 Subject: [PATCH 041/518] fix(redis): delete the Sentinel PDB left behind on scale-down Setting sentinel.disablePodDisruptionBudget only stops the operator from ensuring the PDB; it never deletes the one it created while three Sentinels ran. An existing instance scaled to one Redis pod therefore kept minAvailable 1 on its single Sentinel, which refuses every eviction. A post-upgrade hook deletes rfs- in that case. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- packages/apps/redis/README.md | 2 +- packages/apps/redis/templates/_resources.tpl | 4 + .../redis/templates/hooks/sentinel-pdb.yaml | 119 ++++++++++++++++++ .../apps/redis/templates/redisfailover.yaml | 6 +- .../redis/tests/sentinel_pdb_hook_test.yaml | 78 ++++++++++++ 5 files changed, 206 insertions(+), 3 deletions(-) create mode 100644 packages/apps/redis/templates/hooks/sentinel-pdb.yaml create mode 100644 packages/apps/redis/tests/sentinel_pdb_hook_test.yaml diff --git a/packages/apps/redis/README.md b/packages/apps/redis/README.md index 28152d9f9..478f75986 100644 --- a/packages/apps/redis/README.md +++ b/packages/apps/redis/README.md @@ -79,7 +79,7 @@ See [`docs/operations/resource-presets.md`](../../../docs/operations/resource-pr ### sentinel -`sentinel.resources` and `sentinel.resourcesPreset` size the three Sentinel pods separately from the Redis pods. A Sentinel holds only its configuration and the failover state, so it needs a small fraction of what a Redis replica holding the dataset does. When the `sentinel` block sets neither, the Sentinel pods are sized like the Redis pods, which is how every instance was sized before these parameters existed. +`sentinel.resources` and `sentinel.resourcesPreset` size the Sentinel pods separately from the Redis pods. A Sentinel holds only its configuration and the failover state, so it needs a small fraction of what a Redis replica holding the dataset does. When the `sentinel` block sets neither, the Sentinel pods are sized like the Redis pods, which is how every instance was sized before these parameters existed. ```yaml resourcesPreset: u1.small diff --git a/packages/apps/redis/templates/_resources.tpl b/packages/apps/redis/templates/_resources.tpl index 6539c99a1..98119dbb2 100644 --- a/packages/apps/redis/templates/_resources.tpl +++ b/packages/apps/redis/templates/_resources.tpl @@ -47,3 +47,7 @@ These presets are for basic testing and not meant to be used in production {{- printf "ERROR: Preset key '%s' invalid. Allowed values are %s" .type (join "," (keys $presets)) | fail -}} {{- end -}} {{- end -}} + +{{- define "redis.sentinelReplicas" -}} +{{- ternary 1 3 (le (int .Values.replicas) 1) -}} +{{- end -}} diff --git a/packages/apps/redis/templates/hooks/sentinel-pdb.yaml b/packages/apps/redis/templates/hooks/sentinel-pdb.yaml new file mode 100644 index 000000000..e29613c8a --- /dev/null +++ b/packages/apps/redis/templates/hooks/sentinel-pdb.yaml @@ -0,0 +1,119 @@ +{{- /* + An instance scaled down to one Redis pod keeps the Sentinel PDB the operator + created while it ran three Sentinels: setting disablePodDisruptionBudget only + stops the operator from ensuring it, never deletes it, and minAvailable 1 on + a single Sentinel refuses every eviction. A fresh install with one replica + never gets that PDB, so this runs on upgrade only. +*/}} +{{- if and .Release.IsUpgrade (eq (include "redis.sentinelReplicas" . | int) 1) }} +{{- $name := printf "%s-sentinel-pdb" .Release.Name }} +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ $name }} + labels: + app.kubernetes.io/instance: {{ .Release.Name }} + annotations: + helm.sh/hook: post-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ $name }} + labels: + app.kubernetes.io/instance: {{ .Release.Name }} + annotations: + helm.sh/hook: post-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +rules: + - apiGroups: ["policy"] + resources: ["poddisruptionbudgets"] + resourceNames: ["rfs-{{ .Release.Name }}"] + verbs: ["get", "delete"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ $name }} + labels: + app.kubernetes.io/instance: {{ .Release.Name }} + annotations: + helm.sh/hook: post-upgrade + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ $name }} +subjects: + - kind: ServiceAccount + name: {{ $name }} + namespace: {{ .Release.Namespace }} +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ $name }} + labels: + app.kubernetes.io/instance: {{ .Release.Name }} + annotations: + helm.sh/hook: post-upgrade + helm.sh/hook-weight: "5" + helm.sh/hook-delete-policy: before-hook-creation,hook-succeeded +spec: + ttlSecondsAfterFinished: 3600 + # Bounds the kubectl call too: a --request-timeout would make kubectl skip + # its in-cluster credentials (hack/check-hook-request-timeout.bats). + activeDeadlineSeconds: 120 + backoffLimit: 2 + template: + metadata: + labels: + app.kubernetes.io/instance: {{ .Release.Name }} + policy.cozystack.io/allow-to-apiserver: "true" + spec: + serviceAccountName: {{ $name }} + restartPolicy: Never + terminationGracePeriodSeconds: 10 + securityContext: + runAsNonRoot: true + runAsUser: 65534 + runAsGroup: 65534 + seccompProfile: + type: RuntimeDefault + containers: + - name: delete-pdb + image: '{{ include "cozy-lib.image" (list "clastix/kubectl:v1.32@sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c" $) }}' + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + cpu: 200m + memory: 128Mi + env: + - name: HOME + value: /tmp + command: + - kubectl + - --namespace={{ .Release.Namespace }} + - delete + - poddisruptionbudget + - rfs-{{ .Release.Name }} + - --ignore-not-found=true + volumeMounts: + - name: tmp + mountPath: /tmp + volumes: + - name: tmp + emptyDir: {} +{{- end }} diff --git a/packages/apps/redis/templates/redisfailover.yaml b/packages/apps/redis/templates/redisfailover.yaml index 2dd1fecaf..05a96e3cc 100644 --- a/packages/apps/redis/templates/redisfailover.yaml +++ b/packages/apps/redis/templates/redisfailover.yaml @@ -14,9 +14,11 @@ watch it. One Sentinel still lets the operator find the master (its quorum is replicas/2+1, so 1). The operator gives the Sentinel Deployment a PDB with minAvailable 1 whenever Redis has two replicas or fewer, which on a single - Sentinel would block every node drain, so that PDB is dropped with it. + Sentinel would block every node drain, so that PDB is dropped with it; the + operator does not delete a PDB it stops managing, hence + hooks/sentinel-pdb.yaml. */}} -{{- $sentinelReplicas := ternary 1 3 (le (int .Values.replicas) 1) -}} +{{- $sentinelReplicas := include "redis.sentinelReplicas" . | int -}} {{- $sentinelPreset := $sentinel.resourcesPreset | default .Values.resourcesPreset -}} {{- $sentinelResources := .Values.resources -}} {{- if or $sentinel.resources $sentinel.resourcesPreset -}} diff --git a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml new file mode 100644 index 000000000..71ef30e23 --- /dev/null +++ b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml @@ -0,0 +1,78 @@ +suite: stale Sentinel PDB removed on scale-down to one Redis pod + +# The operator only stops ensuring the Sentinel PDB once +# disablePodDisruptionBudget is set; it never deletes the one it created while +# three Sentinels ran. Left in place, its minAvailable 1 refuses every eviction +# of the single Sentinel and blocks node drains. + +templates: + - templates/hooks/sentinel-pdb.yaml + +release: + name: my-redis + namespace: tenant-test + +set: + _cluster: + cluster-domain: cluster.local + _namespace: + host: example.com + +tests: + - it: deletes the operator's Sentinel PDB on an upgrade to one replica + release: + upgrade: true + set: + replicas: 1 + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.annotations["helm.sh/hook"] + value: post-upgrade + - equal: + path: spec.template.spec.containers[0].command + value: + - kubectl + - --namespace=tenant-test + - delete + - poddisruptionbudget + - rfs-my-redis + - --ignore-not-found=true + - notMatchRegex: + path: spec.template.spec.containers[0].command[4] + pattern: request-timeout + + - it: lets the hook delete only that PDB + release: + upgrade: true + set: + replicas: 1 + documentSelector: + path: kind + value: Role + asserts: + - equal: + path: rules + value: + - apiGroups: ["policy"] + resources: ["poddisruptionbudgets"] + resourceNames: ["rfs-my-redis"] + verbs: ["get", "delete"] + + - it: renders nothing on install, where no such PDB exists + set: + replicas: 1 + asserts: + - hasDocuments: + count: 0 + + - it: renders nothing while three Sentinels run + release: + upgrade: true + set: + replicas: 2 + asserts: + - hasDocuments: + count: 0 From 670f33d68cac9a3a731bb3baaea89c48f3d3911e Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 2 Nov 2026 11:25:12 +0100 Subject: [PATCH 042/518] feat(kubernetes): let a worker pool set its rollout maxUnavailable A Proxmox template that passes a GPU through leaves gpus empty, so the chart renders maxUnavailable 0 for that pool and its replacement VM waits for the GPU the old worker still holds. The pool can now say so itself with maxUnavailable, which overrides the 0, or 1 with gpus, that the chart derives otherwise. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- api/apps/v1alpha1/kubernetesnodes/types.go | 3 ++ .../kubernetesnodes/zz_generated.deepcopy.go | 5 +++ packages/apps/kubernetes-nodes/README.md | 1 + .../kubernetes-nodes/templates/nodegroup.yaml | 11 +++++-- .../tests/rollout_strategy_test.yaml | 32 +++++++++++++++++++ .../apps/kubernetes-nodes/values.schema.json | 5 +++ packages/apps/kubernetes-nodes/values.yaml | 3 ++ .../cozyrds/kubernetes-nodes.yaml | 2 +- 8 files changed, 58 insertions(+), 4 deletions(-) diff --git a/api/apps/v1alpha1/kubernetesnodes/types.go b/api/apps/v1alpha1/kubernetesnodes/types.go index 59bbe59c6..c5c6dc15b 100644 --- a/api/apps/v1alpha1/kubernetesnodes/types.go +++ b/api/apps/v1alpha1/kubernetesnodes/types.go @@ -51,6 +51,9 @@ type ConfigSpec struct { // List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). // +kubebuilder:default:={} Gpus []GPU `json:"gpus,omitempty"` + // Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. + // +kubebuilder:validation:Minimum=0 + MaxUnavailable *int `json:"maxUnavailable,omitempty"` // Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. KernelModules []KernelModule `json:"kernelModules,omitempty"` // Kubelet resource reservations for this pool. diff --git a/api/apps/v1alpha1/kubernetesnodes/zz_generated.deepcopy.go b/api/apps/v1alpha1/kubernetesnodes/zz_generated.deepcopy.go index 185d2fc22..15f0a8b6a 100644 --- a/api/apps/v1alpha1/kubernetesnodes/zz_generated.deepcopy.go +++ b/api/apps/v1alpha1/kubernetesnodes/zz_generated.deepcopy.go @@ -67,6 +67,11 @@ func (in *ConfigSpec) DeepCopyInto(out *ConfigSpec) { *out = make([]GPU, len(*in)) copy(*out, *in) } + if in.MaxUnavailable != nil { + in, out := &in.MaxUnavailable, &out.MaxUnavailable + *out = new(int) + **out = **in + } if in.KernelModules != nil { in, out := &in.KernelModules, &out.KernelModules *out = make([]KernelModule, len(*in)) diff --git a/packages/apps/kubernetes-nodes/README.md b/packages/apps/kubernetes-nodes/README.md index 900a19a7a..dd2872b93 100644 --- a/packages/apps/kubernetes-nodes/README.md +++ b/packages/apps/kubernetes-nodes/README.md @@ -43,6 +43,7 @@ the parent cluster `kubernetes-` in the same namespace, plus a | `podCpuRequest` | CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. | `quantity` | `""` | | `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` | | `gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` | +| `maxUnavailable` | Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. | `*int` | `null` | | `kernelModules` | Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. | `[]object` | `[]` | | `kernelModules[i].name` | Module name as `modprobe` takes it, e.g. `nvidia_uvm`. | `string` | `""` | | `kernelModules[i].parameters` | Module parameters, each as a bare `key=value` string. | `[]string` | `[]` | diff --git a/packages/apps/kubernetes-nodes/templates/nodegroup.yaml b/packages/apps/kubernetes-nodes/templates/nodegroup.yaml index 45ef30cc6..0581f02f9 100644 --- a/packages/apps/kubernetes-nodes/templates/nodegroup.yaml +++ b/packages/apps/kubernetes-nodes/templates/nodegroup.yaml @@ -1029,12 +1029,17 @@ spec: one; with 1, a one-node pool lost its only node until the new one joined. A pool with passed-through GPUs keeps 1: its replacement VM may need the very GPU the old node holds, and would wait for it - forever. CAPI refuses maxSurge and maxUnavailable both at 0, and - maxReplicas may be 0, hence the floor of 1. */}} + forever. A Proxmox template's GPU is invisible here, hence the + maxUnavailable override. CAPI refuses maxSurge and maxUnavailable both + at 0, and maxReplicas may be 0, hence the floor of 1. */}} + {{- $maxUnavailable := ternary 1 0 (not (empty $group.gpus)) }} + {{- if not (kindIs "invalid" $.Values.maxUnavailable) }} + {{- $maxUnavailable = $.Values.maxUnavailable | int }} + {{- end }} strategy: rollingUpdate: maxSurge: {{ max 1 ($group.maxReplicas | int) }} - maxUnavailable: {{ ternary 1 0 (not (empty $group.gpus)) }} + maxUnavailable: {{ $maxUnavailable }} type: RollingUpdate selector: matchLabels: diff --git a/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml b/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml index ae44bb03d..210afb068 100644 --- a/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml @@ -82,3 +82,35 @@ tests: - equal: path: spec.strategy.rollingUpdate.maxSurge value: 1 + + # A Proxmox template that passes a GPU through leaves gpus empty, so the + # chart cannot tell; the pool says so itself. + - it: lets a pool set maxUnavailable explicitly + set: + maxUnavailable: 1 + documentSelector: + path: kind + value: MachineDeployment + asserts: + - equal: + path: spec.strategy.rollingUpdate.maxUnavailable + value: 1 + + - it: honours an explicit 0 on a GPU pool + set: + maxUnavailable: 0 + gpus: + - name: nvidia.com/AD102GL_L40S + documentSelector: + path: kind + value: MachineDeployment + asserts: + - equal: + path: spec.strategy.rollingUpdate.maxUnavailable + value: 0 + + - it: refuses a negative maxUnavailable (schema minimum 0) + set: + maxUnavailable: -1 + asserts: + - failedTemplate: {} diff --git a/packages/apps/kubernetes-nodes/values.schema.json b/packages/apps/kubernetes-nodes/values.schema.json index 0e99096d5..9fbd9e7cd 100644 --- a/packages/apps/kubernetes-nodes/values.schema.json +++ b/packages/apps/kubernetes-nodes/values.schema.json @@ -146,6 +146,11 @@ } } }, + "maxUnavailable": { + "description": "Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.", + "type": "integer", + "minimum": 0 + }, "kernelModules": { "description": "Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.", "type": "array", diff --git a/packages/apps/kubernetes-nodes/values.yaml b/packages/apps/kubernetes-nodes/values.yaml index c930b0433..f7a6d0fec 100644 --- a/packages/apps/kubernetes-nodes/values.yaml +++ b/packages/apps/kubernetes-nodes/values.yaml @@ -77,6 +77,9 @@ resources: {} ## @param {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). gpus: [] +## @param {*int} [maxUnavailable] - Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. +## @minimum 0 + ## @param {[]KernelModule} [kernelModules] - Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. ## @param {Kubelet} [kubelet] - Kubelet resource reservations for this pool. diff --git a/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml b/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml index dffa00956..739d5480a 100644 --- a/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml +++ b/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml @@ -21,7 +21,7 @@ spec: singular: kubernetesnodes plural: kubernetesnodeses openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool name (the part after `-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `\"010\"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `\"010\"` reserves what `\"10\"` does.","type":"object","default":{},"properties":{"cpu":{"description":"Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"podCpuLimit":{"description":"CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"podCpuRequest":{"description":"CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"kernelModules":{"description":"Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Module name as `modprobe` takes it, e.g. `nvidia_uvm`.","type":"string"},"parameters":{"description":"Module parameters, each as a bare `key=value` string.","type":"array","items":{"type":"string"}}}}},"kubelet":{"description":"Kubelet resource reservations for this pool.","type":"object","default":{},"properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.","type":"boolean","default":false},"maxUnhealthy":{"description":"Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer (\"0\", \"1\", ...) or a percentage (\"0%\", \"50%\"); bare numeric strings are rejected, so the safer default is a percentage. Drop to \"0%\" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At \"50%\" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.","type":"string","default":"50%"},"nodeStartupTimeout":{"description":"Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.","type":"string","default":"20m"},"readyUnknownTimeout":{"description":"How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"5m"},"readyFalseTimeout":{"description":"How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"10m"},"maxNodeProvisionTime":{"description":"How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted. Keep in sync with the parent kubernetes chart.","type":"string","default":"30m"},"version":{"description":"Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`.","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"osImage":{"description":"Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool.","type":"object","default":{},"properties":{"builtin":{"description":"Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled.","type":"object","properties":{"schematicID":{"description":"Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version of the golden to clone. Defaults to `talos.version`.","type":"string"}}},"factory":{"description":"Import from an Image Factory / mirror (default).","type":"object","properties":{"imageFactoryURL":{"description":"Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.","type":"string"},"schematicID":{"description":"Talos schematic ID. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version. Defaults to `talos.version`.","type":"string"}}}}},"talos":{"description":"Talos worker image configuration. Keep in sync with the parent cluster's `talos`.","type":"object","default":{},"required":["imageFactoryURL","installerRepository","registryMirrors","schematicID","version"],"properties":{"imageFactoryCA":{"description":"Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.","type":"string","default":""},"imageFactoryURL":{"description":"Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.","type":"string","default":"https://factory.talos.dev"},"installerRepository":{"description":"OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.","type":"string","default":"factory.talos.dev/installer"},"registryMirrors":{"description":"Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true},"schematicID":{"description":"Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID.","type":"string","default":"ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"},"version":{"description":"Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.","type":"string","default":"v1.13.6"}}},"substrate":{"description":"Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment's template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["network"],"properties":{"dnsServers":{"description":"Nameservers written into the worker machineconfig. On kubevirt the workers use the management cluster's CoreDNS, which they reach over the pod network; an off-cluster Proxmox worker cannot, so it needs reachable resolvers of its own. Required when `substrate` is `proxmox`. Keep in sync with the parent kubernetes chart's `proxmox.dnsServers`.","type":"array","default":[],"items":{"type":"string"}},"full":{"description":"Full clone rather than linked. Default is a linked clone: it costs kilobytes at creation instead of the whole disk, because a ZFS-backed linked clone writes only its own increment (measured: 8K against 10.2G for the same worker full-cloned onto a `sparse 0` pool). The cost is that the clone holds the template's base snapshot, so the template cannot be rotated while any linked clone still references it — which is what the consolidation CronJob resolves 24h after creation. Set true for a pool that must be independent of the template from the first second.","type":"boolean","default":false},"network":{"description":"NIC configuration.","type":"object","default":{},"required":["bridge"],"properties":{"bridge":{"description":"Proxmox bridge the VM NIC attaches to (e.g. `vmbr0`). Required when `substrate` is `proxmox`.","type":"string","default":""},"mtu":{"description":"NIC MTU. Omitted when unset.","type":"integer"},"vlan":{"description":"L2 VLAN tag. Omitted when unset.","type":"integer"}}},"pool":{"description":"Proxmox resource pool the worker VMs are added to. Set the same pool on every node pool of a tenant cluster and grant the tenant's tokens their privileges on `/pool/`: that is the one ACL path that follows every VM capmox creates, replacements included. The pool must exist on the hypervisor; capmox does not create it. Empty adds the workers to no pool.","type":"string","default":""},"storage":{"description":"Proxmox storage for the cloned disk; needs `full: true`. A target storage is a full-clone parameter — a linked clone always lives on the template's storage, and Proxmox refuses the pair — so the render refuses it too. Empty keeps the template's storage.","type":"string","default":""},"templateTags":{"description":"Tags identifying the Talos VM template to clone. capmox matches a template when its tag set is EQUAL to this list, not when it contains it, so every tag on the template must be listed here — a missing one yields `found 0 VM templates with tags ...` while the tags are plainly present. The template has to carry this pool's `talos.schematicID` at `talos.version` in the nocloud flavour, with the Proxmox guest agent enabled for the `qemu-guest-agent` extension of the default schematic; `hack/proxmox-talos-template.sh` builds it from these values and prints the tags to list here. Do not build it from the platform release's `nocloud-amd64.raw.xz`: that is the bare-metal node image, whose `zfs` extension keeps a VM from finishing its boot sequence and reboots it at the Talos boot timeout. Required when `substrate` is `proxmox`.","type":"array","default":[],"items":{"type":"string"}}}}}} + {"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool name (the part after `-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `\"010\"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `\"010\"` reserves what `\"10\"` does.","type":"object","default":{},"properties":{"cpu":{"description":"Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"podCpuLimit":{"description":"CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"podCpuRequest":{"description":"CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"maxUnavailable":{"description":"Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.","type":"integer","minimum":0},"kernelModules":{"description":"Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Module name as `modprobe` takes it, e.g. `nvidia_uvm`.","type":"string"},"parameters":{"description":"Module parameters, each as a bare `key=value` string.","type":"array","items":{"type":"string"}}}}},"kubelet":{"description":"Kubelet resource reservations for this pool.","type":"object","default":{},"properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.","type":"boolean","default":false},"maxUnhealthy":{"description":"Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer (\"0\", \"1\", ...) or a percentage (\"0%\", \"50%\"); bare numeric strings are rejected, so the safer default is a percentage. Drop to \"0%\" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At \"50%\" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.","type":"string","default":"50%"},"nodeStartupTimeout":{"description":"Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.","type":"string","default":"20m"},"readyUnknownTimeout":{"description":"How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"5m"},"readyFalseTimeout":{"description":"How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"10m"},"maxNodeProvisionTime":{"description":"How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted. Keep in sync with the parent kubernetes chart.","type":"string","default":"30m"},"version":{"description":"Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`.","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"osImage":{"description":"Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool.","type":"object","default":{},"properties":{"builtin":{"description":"Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled.","type":"object","properties":{"schematicID":{"description":"Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version of the golden to clone. Defaults to `talos.version`.","type":"string"}}},"factory":{"description":"Import from an Image Factory / mirror (default).","type":"object","properties":{"imageFactoryURL":{"description":"Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.","type":"string"},"schematicID":{"description":"Talos schematic ID. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version. Defaults to `talos.version`.","type":"string"}}}}},"talos":{"description":"Talos worker image configuration. Keep in sync with the parent cluster's `talos`.","type":"object","default":{},"required":["imageFactoryURL","installerRepository","registryMirrors","schematicID","version"],"properties":{"imageFactoryCA":{"description":"Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.","type":"string","default":""},"imageFactoryURL":{"description":"Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.","type":"string","default":"https://factory.talos.dev"},"installerRepository":{"description":"OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.","type":"string","default":"factory.talos.dev/installer"},"registryMirrors":{"description":"Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true},"schematicID":{"description":"Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID.","type":"string","default":"ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"},"version":{"description":"Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.","type":"string","default":"v1.13.6"}}},"substrate":{"description":"Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment's template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["network"],"properties":{"dnsServers":{"description":"Nameservers written into the worker machineconfig. On kubevirt the workers use the management cluster's CoreDNS, which they reach over the pod network; an off-cluster Proxmox worker cannot, so it needs reachable resolvers of its own. Required when `substrate` is `proxmox`. Keep in sync with the parent kubernetes chart's `proxmox.dnsServers`.","type":"array","default":[],"items":{"type":"string"}},"full":{"description":"Full clone rather than linked. Default is a linked clone: it costs kilobytes at creation instead of the whole disk, because a ZFS-backed linked clone writes only its own increment (measured: 8K against 10.2G for the same worker full-cloned onto a `sparse 0` pool). The cost is that the clone holds the template's base snapshot, so the template cannot be rotated while any linked clone still references it — which is what the consolidation CronJob resolves 24h after creation. Set true for a pool that must be independent of the template from the first second.","type":"boolean","default":false},"network":{"description":"NIC configuration.","type":"object","default":{},"required":["bridge"],"properties":{"bridge":{"description":"Proxmox bridge the VM NIC attaches to (e.g. `vmbr0`). Required when `substrate` is `proxmox`.","type":"string","default":""},"mtu":{"description":"NIC MTU. Omitted when unset.","type":"integer"},"vlan":{"description":"L2 VLAN tag. Omitted when unset.","type":"integer"}}},"pool":{"description":"Proxmox resource pool the worker VMs are added to. Set the same pool on every node pool of a tenant cluster and grant the tenant's tokens their privileges on `/pool/`: that is the one ACL path that follows every VM capmox creates, replacements included. The pool must exist on the hypervisor; capmox does not create it. Empty adds the workers to no pool.","type":"string","default":""},"storage":{"description":"Proxmox storage for the cloned disk; needs `full: true`. A target storage is a full-clone parameter — a linked clone always lives on the template's storage, and Proxmox refuses the pair — so the render refuses it too. Empty keeps the template's storage.","type":"string","default":""},"templateTags":{"description":"Tags identifying the Talos VM template to clone. capmox matches a template when its tag set is EQUAL to this list, not when it contains it, so every tag on the template must be listed here — a missing one yields `found 0 VM templates with tags ...` while the tags are plainly present. The template has to carry this pool's `talos.schematicID` at `talos.version` in the nocloud flavour, with the Proxmox guest agent enabled for the `qemu-guest-agent` extension of the default schematic; `hack/proxmox-talos-template.sh` builds it from these values and prints the tags to list here. Do not build it from the platform release's `nocloud-amd64.raw.xz`: that is the bare-metal node image, whose `zfs` extension keeps a VM from finishing its boot sequence and reboots it at the Talos boot timeout. Required when `substrate` is `proxmox`.","type":"array","default":[],"items":{"type":"string"}}}}}} release: labels: sharding.fluxcd.io/key: tenants From 2541aec62550e9b5cee12ec6e15cc5b58bc5adc9 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 2 Nov 2026 11:32:58 +0100 Subject: [PATCH 043/518] fix(redis): keep the Sentinel PDB hook Job name within 63 characters The API server copies a Job's name into its pods' job-name label, which takes 63 characters at most, and a release name may take 53, so -sentinel-pdb could be refused and leave the stale PDB in place. The Job name is truncated; the ServiceAccount, Role and RoleBinding keep the full name. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../redis/templates/hooks/sentinel-pdb.yaml | 5 ++- .../redis/tests/sentinel_pdb_hook_test.yaml | 36 +++++++++++++++++++ 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/packages/apps/redis/templates/hooks/sentinel-pdb.yaml b/packages/apps/redis/templates/hooks/sentinel-pdb.yaml index e29613c8a..1649caa79 100644 --- a/packages/apps/redis/templates/hooks/sentinel-pdb.yaml +++ b/packages/apps/redis/templates/hooks/sentinel-pdb.yaml @@ -57,7 +57,10 @@ subjects: apiVersion: batch/v1 kind: Job metadata: - name: {{ $name }} + {{- /* The API server copies a Job's name into its pods' job-name label, + which takes 63 characters at most, while a release name may take 53. + The objects above have no such limit and keep the full name. */}} + name: {{ $name | trunc 63 | trimSuffix "-" }} labels: app.kubernetes.io/instance: {{ .Release.Name }} annotations: diff --git a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml index 71ef30e23..bc6aea706 100644 --- a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml +++ b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml @@ -76,3 +76,39 @@ tests: asserts: - hasDocuments: count: 0 + + # The pods' job-name label takes 63 characters at most; a 53-character + # release name plus -sentinel-pdb would pass it and Kubernetes would refuse + # the Job, leaving the stale PDB in place. + - it: keeps the Job name within 63 characters for the longest release name + release: + name: redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + upgrade: true + set: + replicas: 1 + documentSelector: + path: kind + value: Job + asserts: + - matchRegex: + path: metadata.name + pattern: ^[a-z0-9-]{1,63}$ + - matchRegex: + path: metadata.name + pattern: '[a-z0-9]$' + - equal: + path: spec.template.spec.containers[0].command[4] + value: rfs-redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + + - it: keeps the full Job name for a short release name + release: + upgrade: true + set: + replicas: 1 + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.name + value: my-redis-sentinel-pdb From ed40d8332306d7fde6583dbf55b0de36f820d2a1 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 2 Nov 2026 11:33:40 +0100 Subject: [PATCH 044/518] docs(kubernetes): say a worker rollout needs room for one extra VM With maxUnavailable 0 a pool short of quota or capacity for the replacement VM stalls its rollout instead of shrinking. The field description, which also feeds the API and dashboard documentation, now says so and names maxUnavailable: 1 as the way out. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- api/apps/v1alpha1/kubernetesnodes/types.go | 2 +- packages/apps/kubernetes-nodes/README.md | 2 +- packages/apps/kubernetes-nodes/values.schema.json | 2 +- packages/apps/kubernetes-nodes/values.yaml | 2 +- .../system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/api/apps/v1alpha1/kubernetesnodes/types.go b/api/apps/v1alpha1/kubernetesnodes/types.go index c5c6dc15b..81cca0bf9 100644 --- a/api/apps/v1alpha1/kubernetesnodes/types.go +++ b/api/apps/v1alpha1/kubernetesnodes/types.go @@ -51,7 +51,7 @@ type ConfigSpec struct { // List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). // +kubebuilder:default:={} Gpus []GPU `json:"gpus,omitempty"` - // Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. + // Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. // +kubebuilder:validation:Minimum=0 MaxUnavailable *int `json:"maxUnavailable,omitempty"` // Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. diff --git a/packages/apps/kubernetes-nodes/README.md b/packages/apps/kubernetes-nodes/README.md index dd2872b93..348b21268 100644 --- a/packages/apps/kubernetes-nodes/README.md +++ b/packages/apps/kubernetes-nodes/README.md @@ -43,7 +43,7 @@ the parent cluster `kubernetes-` in the same namespace, plus a | `podCpuRequest` | CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. | `quantity` | `""` | | `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` | | `gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` | -| `maxUnavailable` | Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. | `*int` | `null` | +| `maxUnavailable` | Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. | `*int` | `null` | | `kernelModules` | Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. | `[]object` | `[]` | | `kernelModules[i].name` | Module name as `modprobe` takes it, e.g. `nvidia_uvm`. | `string` | `""` | | `kernelModules[i].parameters` | Module parameters, each as a bare `key=value` string. | `[]string` | `[]` | diff --git a/packages/apps/kubernetes-nodes/values.schema.json b/packages/apps/kubernetes-nodes/values.schema.json index 9fbd9e7cd..53de10b95 100644 --- a/packages/apps/kubernetes-nodes/values.schema.json +++ b/packages/apps/kubernetes-nodes/values.schema.json @@ -147,7 +147,7 @@ } }, "maxUnavailable": { - "description": "Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.", + "description": "Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.", "type": "integer", "minimum": 0 }, diff --git a/packages/apps/kubernetes-nodes/values.yaml b/packages/apps/kubernetes-nodes/values.yaml index f7a6d0fec..f5d6c7dcf 100644 --- a/packages/apps/kubernetes-nodes/values.yaml +++ b/packages/apps/kubernetes-nodes/values.yaml @@ -77,7 +77,7 @@ resources: {} ## @param {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). gpus: [] -## @param {*int} [maxUnavailable] - Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. +## @param {*int} [maxUnavailable] - Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. ## @minimum 0 ## @param {[]KernelModule} [kernelModules] - Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. diff --git a/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml b/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml index 739d5480a..0e5a196fd 100644 --- a/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml +++ b/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml @@ -21,7 +21,7 @@ spec: singular: kubernetesnodes plural: kubernetesnodeses openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool name (the part after `-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `\"010\"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `\"010\"` reserves what `\"10\"` does.","type":"object","default":{},"properties":{"cpu":{"description":"Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"podCpuLimit":{"description":"CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"podCpuRequest":{"description":"CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"maxUnavailable":{"description":"Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.","type":"integer","minimum":0},"kernelModules":{"description":"Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Module name as `modprobe` takes it, e.g. `nvidia_uvm`.","type":"string"},"parameters":{"description":"Module parameters, each as a bare `key=value` string.","type":"array","items":{"type":"string"}}}}},"kubelet":{"description":"Kubelet resource reservations for this pool.","type":"object","default":{},"properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.","type":"boolean","default":false},"maxUnhealthy":{"description":"Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer (\"0\", \"1\", ...) or a percentage (\"0%\", \"50%\"); bare numeric strings are rejected, so the safer default is a percentage. Drop to \"0%\" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At \"50%\" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.","type":"string","default":"50%"},"nodeStartupTimeout":{"description":"Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.","type":"string","default":"20m"},"readyUnknownTimeout":{"description":"How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"5m"},"readyFalseTimeout":{"description":"How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"10m"},"maxNodeProvisionTime":{"description":"How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted. Keep in sync with the parent kubernetes chart.","type":"string","default":"30m"},"version":{"description":"Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`.","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"osImage":{"description":"Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool.","type":"object","default":{},"properties":{"builtin":{"description":"Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled.","type":"object","properties":{"schematicID":{"description":"Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version of the golden to clone. Defaults to `talos.version`.","type":"string"}}},"factory":{"description":"Import from an Image Factory / mirror (default).","type":"object","properties":{"imageFactoryURL":{"description":"Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.","type":"string"},"schematicID":{"description":"Talos schematic ID. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version. Defaults to `talos.version`.","type":"string"}}}}},"talos":{"description":"Talos worker image configuration. Keep in sync with the parent cluster's `talos`.","type":"object","default":{},"required":["imageFactoryURL","installerRepository","registryMirrors","schematicID","version"],"properties":{"imageFactoryCA":{"description":"Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.","type":"string","default":""},"imageFactoryURL":{"description":"Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.","type":"string","default":"https://factory.talos.dev"},"installerRepository":{"description":"OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.","type":"string","default":"factory.talos.dev/installer"},"registryMirrors":{"description":"Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true},"schematicID":{"description":"Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID.","type":"string","default":"ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"},"version":{"description":"Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.","type":"string","default":"v1.13.6"}}},"substrate":{"description":"Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment's template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["network"],"properties":{"dnsServers":{"description":"Nameservers written into the worker machineconfig. On kubevirt the workers use the management cluster's CoreDNS, which they reach over the pod network; an off-cluster Proxmox worker cannot, so it needs reachable resolvers of its own. Required when `substrate` is `proxmox`. Keep in sync with the parent kubernetes chart's `proxmox.dnsServers`.","type":"array","default":[],"items":{"type":"string"}},"full":{"description":"Full clone rather than linked. Default is a linked clone: it costs kilobytes at creation instead of the whole disk, because a ZFS-backed linked clone writes only its own increment (measured: 8K against 10.2G for the same worker full-cloned onto a `sparse 0` pool). The cost is that the clone holds the template's base snapshot, so the template cannot be rotated while any linked clone still references it — which is what the consolidation CronJob resolves 24h after creation. Set true for a pool that must be independent of the template from the first second.","type":"boolean","default":false},"network":{"description":"NIC configuration.","type":"object","default":{},"required":["bridge"],"properties":{"bridge":{"description":"Proxmox bridge the VM NIC attaches to (e.g. `vmbr0`). Required when `substrate` is `proxmox`.","type":"string","default":""},"mtu":{"description":"NIC MTU. Omitted when unset.","type":"integer"},"vlan":{"description":"L2 VLAN tag. Omitted when unset.","type":"integer"}}},"pool":{"description":"Proxmox resource pool the worker VMs are added to. Set the same pool on every node pool of a tenant cluster and grant the tenant's tokens their privileges on `/pool/`: that is the one ACL path that follows every VM capmox creates, replacements included. The pool must exist on the hypervisor; capmox does not create it. Empty adds the workers to no pool.","type":"string","default":""},"storage":{"description":"Proxmox storage for the cloned disk; needs `full: true`. A target storage is a full-clone parameter — a linked clone always lives on the template's storage, and Proxmox refuses the pair — so the render refuses it too. Empty keeps the template's storage.","type":"string","default":""},"templateTags":{"description":"Tags identifying the Talos VM template to clone. capmox matches a template when its tag set is EQUAL to this list, not when it contains it, so every tag on the template must be listed here — a missing one yields `found 0 VM templates with tags ...` while the tags are plainly present. The template has to carry this pool's `talos.schematicID` at `talos.version` in the nocloud flavour, with the Proxmox guest agent enabled for the `qemu-guest-agent` extension of the default schematic; `hack/proxmox-talos-template.sh` builds it from these values and prints the tags to list here. Do not build it from the platform release's `nocloud-amd64.raw.xz`: that is the bare-metal node image, whose `zfs` extension keeps a VM from finishing its boot sequence and reboots it at the Talos boot timeout. Required when `substrate` is `proxmox`.","type":"array","default":[],"items":{"type":"string"}}}}}} + {"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool name (the part after `-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `\"010\"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `\"010\"` reserves what `\"10\"` does.","type":"object","default":{},"properties":{"cpu":{"description":"Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"podCpuLimit":{"description":"CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"podCpuRequest":{"description":"CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"maxUnavailable":{"description":"Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.","type":"integer","minimum":0},"kernelModules":{"description":"Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Module name as `modprobe` takes it, e.g. `nvidia_uvm`.","type":"string"},"parameters":{"description":"Module parameters, each as a bare `key=value` string.","type":"array","items":{"type":"string"}}}}},"kubelet":{"description":"Kubelet resource reservations for this pool.","type":"object","default":{},"properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.","type":"boolean","default":false},"maxUnhealthy":{"description":"Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer (\"0\", \"1\", ...) or a percentage (\"0%\", \"50%\"); bare numeric strings are rejected, so the safer default is a percentage. Drop to \"0%\" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At \"50%\" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.","type":"string","default":"50%"},"nodeStartupTimeout":{"description":"Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.","type":"string","default":"20m"},"readyUnknownTimeout":{"description":"How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"5m"},"readyFalseTimeout":{"description":"How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"10m"},"maxNodeProvisionTime":{"description":"How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted. Keep in sync with the parent kubernetes chart.","type":"string","default":"30m"},"version":{"description":"Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`.","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"osImage":{"description":"Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool.","type":"object","default":{},"properties":{"builtin":{"description":"Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled.","type":"object","properties":{"schematicID":{"description":"Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version of the golden to clone. Defaults to `talos.version`.","type":"string"}}},"factory":{"description":"Import from an Image Factory / mirror (default).","type":"object","properties":{"imageFactoryURL":{"description":"Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.","type":"string"},"schematicID":{"description":"Talos schematic ID. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version. Defaults to `talos.version`.","type":"string"}}}}},"talos":{"description":"Talos worker image configuration. Keep in sync with the parent cluster's `talos`.","type":"object","default":{},"required":["imageFactoryURL","installerRepository","registryMirrors","schematicID","version"],"properties":{"imageFactoryCA":{"description":"Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.","type":"string","default":""},"imageFactoryURL":{"description":"Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.","type":"string","default":"https://factory.talos.dev"},"installerRepository":{"description":"OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.","type":"string","default":"factory.talos.dev/installer"},"registryMirrors":{"description":"Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true},"schematicID":{"description":"Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID.","type":"string","default":"ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"},"version":{"description":"Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.","type":"string","default":"v1.13.6"}}},"substrate":{"description":"Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment's template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["network"],"properties":{"dnsServers":{"description":"Nameservers written into the worker machineconfig. On kubevirt the workers use the management cluster's CoreDNS, which they reach over the pod network; an off-cluster Proxmox worker cannot, so it needs reachable resolvers of its own. Required when `substrate` is `proxmox`. Keep in sync with the parent kubernetes chart's `proxmox.dnsServers`.","type":"array","default":[],"items":{"type":"string"}},"full":{"description":"Full clone rather than linked. Default is a linked clone: it costs kilobytes at creation instead of the whole disk, because a ZFS-backed linked clone writes only its own increment (measured: 8K against 10.2G for the same worker full-cloned onto a `sparse 0` pool). The cost is that the clone holds the template's base snapshot, so the template cannot be rotated while any linked clone still references it — which is what the consolidation CronJob resolves 24h after creation. Set true for a pool that must be independent of the template from the first second.","type":"boolean","default":false},"network":{"description":"NIC configuration.","type":"object","default":{},"required":["bridge"],"properties":{"bridge":{"description":"Proxmox bridge the VM NIC attaches to (e.g. `vmbr0`). Required when `substrate` is `proxmox`.","type":"string","default":""},"mtu":{"description":"NIC MTU. Omitted when unset.","type":"integer"},"vlan":{"description":"L2 VLAN tag. Omitted when unset.","type":"integer"}}},"pool":{"description":"Proxmox resource pool the worker VMs are added to. Set the same pool on every node pool of a tenant cluster and grant the tenant's tokens their privileges on `/pool/`: that is the one ACL path that follows every VM capmox creates, replacements included. The pool must exist on the hypervisor; capmox does not create it. Empty adds the workers to no pool.","type":"string","default":""},"storage":{"description":"Proxmox storage for the cloned disk; needs `full: true`. A target storage is a full-clone parameter — a linked clone always lives on the template's storage, and Proxmox refuses the pair — so the render refuses it too. Empty keeps the template's storage.","type":"string","default":""},"templateTags":{"description":"Tags identifying the Talos VM template to clone. capmox matches a template when its tag set is EQUAL to this list, not when it contains it, so every tag on the template must be listed here — a missing one yields `found 0 VM templates with tags ...` while the tags are plainly present. The template has to carry this pool's `talos.schematicID` at `talos.version` in the nocloud flavour, with the Proxmox guest agent enabled for the `qemu-guest-agent` extension of the default schematic; `hack/proxmox-talos-template.sh` builds it from these values and prints the tags to list here. Do not build it from the platform release's `nocloud-amd64.raw.xz`: that is the bare-metal node image, whose `zfs` extension keeps a VM from finishing its boot sequence and reboots it at the Talos boot timeout. Required when `substrate` is `proxmox`.","type":"array","default":[],"items":{"type":"string"}}}}}} release: labels: sharding.fluxcd.io/key: tenants From 54695fc83b3328a5401710b9b9d5debbbd05809c Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 3 Nov 2026 03:47:58 +0300 Subject: [PATCH 045/518] fix(api): read pool usage under the quota key the controller counts Hierarchical quota admission subtracts the parent pool's current usage from what a child tenant may still claim, and reads that usage from ResourceQuota.status.used. It looked every non-raw key up under limits., but the ResourceQuota pod evaluator reports usage under limits.* only for cpu, memory and ephemeral-storage. Extended resources and hugepages are counted under requests.* alone, so their usage always read as zero and a child could be granted budget the parent pool had already consumed. Read usage from requests. for every resource the evaluator counts only on the request side. flatten applies no allocation ratio there, so that key holds the declared amount just as limits. does for cpu, memory and ephemeral-storage. Fixes: e3f09bab1bcc ("fix(apiserver): subtract parent pool usage in hierarchical tenant quota admission") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- pkg/registry/apps/application/quota.go | 44 ++++++++++++++------- pkg/registry/apps/application/quota_test.go | 29 ++++++++++++-- 2 files changed, 54 insertions(+), 19 deletions(-) diff --git a/pkg/registry/apps/application/quota.go b/pkg/registry/apps/application/quota.go index eb6f20622..5376139c0 100644 --- a/pkg/registry/apps/application/quota.go +++ b/pkg/registry/apps/application/quota.go @@ -233,7 +233,7 @@ func (r *REST) validateTenantResourceQuotas(ctx context.Context, app *appsv1alph } parentLimit := parentQuota[res] allocated := siblings[res] // zero value when no sibling bounds this resource - used := poolUsed[renderedLimitKey(res)] + used := poolUsed[usageKey(res)] remaining := parentLimit.DeepCopy() remaining.Sub(allocated) remaining.Sub(used) @@ -285,25 +285,39 @@ var rawQuotaKeys = map[string]struct{}{ "resourcequotas": {}, } -// renderedLimitKey maps a shorthand resourceQuotas key (the vocabulary admission -// works in) to the rendered ResourceQuota key that carries its limit-side -// budget, mirroring cozy-lib.resources.flatten. The mapping is allocation-ratio -// independent: ratios only scale the requests.* side that flatten derives from -// the same shorthand value, so the limit-side key always holds the operator's -// declared amount. This lets admission read a tenant's current usage from its -// rendered ResourceQuota.status.used without replicating the chart's ratio math. +// limitCountedKeys are the shorthand keys whose usage the ResourceQuota pod +// evaluator reports under limits.*. For every other resource it counts only +// requests.*, so limits. or limits.hugepages-* never carries +// usage even though the chart renders it. +var limitCountedKeys = map[string]struct{}{ + "cpu": {}, + "memory": {}, + "ephemeral-storage": {}, +} + +// usageKey maps a shorthand resourceQuotas key (the vocabulary admission works +// in) to the rendered ResourceQuota key whose status.used carries its usage. +// cozy-lib.resources.flatten renders a shorthand key both as limits. and as +// requests.; usage is read from whichever of the two the quota controller +// counts. For cpu, memory and ephemeral-storage that is the limit side, which +// holds the operator's declared amount whatever allocation ratio scales the +// request side. Every other resource is counted only on the request side, where +// flatten applies no ratio, so requests. holds the declared amount too. +// This lets admission read a tenant's current usage from its rendered +// ResourceQuota.status.used without replicating the chart's ratio math. // -// cpu, memory, ephemeral-storage, custom resources -> "limits." +// cpu, memory, ephemeral-storage -> "limits." // storage -> "requests.storage" (no hard limits.storage) // pods, services, services.loadbalancers, ... -> "" (verbatim) -func renderedLimitKey(raw string) string { - if raw == "storage" { - return "requests.storage" - } +// extended resources, hugepages-, ... -> "requests." +func usageKey(raw string) string { if _, ok := rawQuotaKeys[raw]; ok { return raw } - return "limits." + raw + if _, ok := limitCountedKeys[raw]; ok { + return "limits." + raw + } + return "requests." + raw } // parentNamespaceOf returns the namespace owned by the parent of the tenant that @@ -341,7 +355,7 @@ func (r *REST) poolRootOf(ns string, bounded map[string]bool) string { // created/updated). Usage is read from each member's ResourceQuota.status.used, // taking a per-resource max within a namespace to avoid double-counting the // chart-rendered and controller-allocated quotas, and is keyed in the rendered -// quota key space (see renderedLimitKey). Boundedness is decided by declared +// quota key space (see usageKey). Boundedness is decided by declared // resourceQuotas so the result does not depend on whether Flux has rendered a // freshly-created tenant's quota yet. func (r *REST) parentPoolUsage(ctx context.Context, parentNS, excludeNS string) (map[string]resource.Quantity, error) { diff --git a/pkg/registry/apps/application/quota_test.go b/pkg/registry/apps/application/quota_test.go index f6e5d3054..3ea2b7a2b 100644 --- a/pkg/registry/apps/application/quota_test.go +++ b/pkg/registry/apps/application/quota_test.go @@ -136,19 +136,20 @@ func usageQuota(t *testing.T, namespace, name string, used map[string]string) *c } } -func TestRenderedLimitKey(t *testing.T) { +func TestUsageKey(t *testing.T) { cases := map[string]string{ "cpu": "limits.cpu", "memory": "limits.memory", "ephemeral-storage": "limits.ephemeral-storage", - "devices.com/nvidia": "limits.devices.com/nvidia", + "devices.com/nvidia": "requests.devices.com/nvidia", + "hugepages-2Mi": "requests.hugepages-2Mi", "storage": "requests.storage", "pods": "pods", "services.loadbalancers": "services.loadbalancers", } for raw, want := range cases { - if got := renderedLimitKey(raw); got != want { - t.Errorf("renderedLimitKey(%q) = %q, want %q", raw, got, want) + if got := usageKey(raw); got != want { + t.Errorf("usageKey(%q) = %q, want %q", raw, got, want) } } } @@ -339,6 +340,26 @@ func TestValidateTenantResourceQuotas_PoolUsage(t *testing.T) { } }) + t.Run("parent usage of an extended resource shrinks remaining", func(t *testing.T) { + parent := tenantHelmRelease(t, "foo", "tenant-root", map[string]string{"example.com/device": "2"}) + used := usageQuota(t, "tenant-foo", "tenant-quota", map[string]string{"requests.example.com/device": "2"}) + r := newTenantREST(t, parent, used) + + if errs := r.validateTenantResourceQuotas(context.Background(), childApplication(t, "bar", "tenant-foo", map[string]string{"example.com/device": "1"})); len(errs) == 0 { + t.Fatalf("child example.com/device=1 should be denied when the parent pool already uses 2 of 2") + } + }) + + t.Run("parent usage of an extended resource leaves the rest grantable", func(t *testing.T) { + parent := tenantHelmRelease(t, "foo", "tenant-root", map[string]string{"example.com/device": "2"}) + used := usageQuota(t, "tenant-foo", "tenant-quota", map[string]string{"requests.example.com/device": "1"}) + r := newTenantREST(t, parent, used) + + if errs := r.validateTenantResourceQuotas(context.Background(), childApplication(t, "bar", "tenant-foo", map[string]string{"example.com/device": "1"})); len(errs) > 0 { + t.Fatalf("child example.com/device=1 should be allowed when the parent pool uses 1 of 2, got: %v", errs) + } + }) + // No rendered usage yet (freshly-created parent, Flux has not rendered the // quota): usage degrades to zero and behaviour matches the pre-usage gate. t.Run("absent usage degrades to budget-only check", func(t *testing.T) { From 7de70ac2fc64a03698f131c883474da73945620a Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:11:15 +0700 Subject: [PATCH 046/518] feat(monitoring-agents): export PVC labels from kube-state-metrics The tenant metrics API scopes a query to one application by looking up its pods and PVCs through their apps.cozystack.io/application.* lineage labels. kube-state-metrics emits kube_persistentvolumeclaim_labels only for resources on --metric-labels-allowlist, so without PVCs there the PVC half of that lookup always comes back empty and volume series of an application fall out of its scope. All labels are allowed, as for pods and deployments: kube-state-metrics emits one series per PVC whatever keys are listed, and PVCs are far fewer than pods, which already export every label. Assisted-by: LLM Signed-off-by: ZverGuy --- .../tests/kube_state_metrics_labels_test.yaml | 15 +++++++++++++++ packages/system/monitoring-agents/values.yaml | 14 ++++++++++---- 2 files changed, 25 insertions(+), 4 deletions(-) create mode 100644 packages/system/monitoring-agents/tests/kube_state_metrics_labels_test.yaml diff --git a/packages/system/monitoring-agents/tests/kube_state_metrics_labels_test.yaml b/packages/system/monitoring-agents/tests/kube_state_metrics_labels_test.yaml new file mode 100644 index 000000000..578371a1e --- /dev/null +++ b/packages/system/monitoring-agents/tests/kube_state_metrics_labels_test.yaml @@ -0,0 +1,15 @@ +suite: kube-state-metrics exports the labels of pods, deployments and PVCs +# The tenant metrics API scopes a query to one application by looking up the +# application's pods and PVCs in kube_pod_labels and +# kube_persistentvolumeclaim_labels by their apps.cozystack.io/application.* +# lineage labels. kube-state-metrics emits no kube_*_labels series at all for +# a resource missing from this allowlist, so dropping persistentvolumeclaims +# silently empties the PVC branch of that lookup. +templates: + - charts/kube-state-metrics/templates/deployment.yaml +tests: + - it: allowlists all labels of pods, deployments and persistentvolumeclaims + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --metric-labels-allowlist=pods=[*],deployments=[*],persistentvolumeclaims=[*] diff --git a/packages/system/monitoring-agents/values.yaml b/packages/system/monitoring-agents/values.yaml index a9c22cb14..84c39ce44 100644 --- a/packages/system/monitoring-agents/values.yaml +++ b/packages/system/monitoring-agents/values.yaml @@ -6,9 +6,10 @@ kube-state-metrics: # 322 pods and roughly 25.6k series, which is the smallest cluster we run. The # request sits above the 176Mi mean rather than at the peak, and the limit # carries wide headroom on purpose: this component's memory scales with the - # number of objects in the cluster, and --metric-labels-allowlist below widens - # every pod and deployment series further, so a cluster larger than the one - # measured will land well above the peak seen here. + # number of objects in the cluster, and --metric-labels-allowlist below copies + # every label of each pod, deployment and PVC onto its kube_*_labels series, + # so a cluster larger than the one measured will land well above the peak + # seen here. # 2Gi is 6.7x that peak, so scaling linearly with object count it stays a # ceiling rather than a bound up to roughly 2100 pods and 170k series. # @@ -27,7 +28,12 @@ kube-state-metrics: limits: memory: 2Gi extraArgs: - - --metric-labels-allowlist=pods=[*],deployments=[*] + # PVCs are listed so the tenant metrics API can find an application's PVCs + # in kube_persistentvolumeclaim_labels by their apps.cozystack.io/application.* + # lineage labels. kube-state-metrics emits that metric only for resources on + # this list, one series per PVC; naming only the lineage keys would emit the + # same number of series. + - --metric-labels-allowlist=pods=[*],deployments=[*],persistentvolumeclaims=[*] rbac: extraRules: - apiGroups: From f98ee76d2db3ab3acea796ce981374461e82caaa Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:16:56 +0700 Subject: [PATCH 047/518] feat(monitoring): stamp application identity on cluster-wide scrapes Scoping a metrics query to one application needs the application's identity on the series themselves, and it is only a boundary if the scraped pod cannot choose it. A default scrape class on the cluster-wide vmagent copies the pod's lineage labels into apps_cozystack_io_application_{group,kind,name} at scrape time. The target always carries all three labels, with a placeholder when the pod has no lineage, so an exporter's own copy always collides, is renamed to exported_* under honorLabels false, and is dropped. Without the placeholder a pod lacking lineage could publish any identity unchallenged. The placeholder is removed again before storage, so series of pods without lineage look as they did before. Scrapes with honorLabels true keep the exporter's value; none of them scrapes a tenant-controlled exporter. Assisted-by: LLM Signed-off-by: ZverGuy --- .../monitoring-agents/templates/vmagent.yaml | 25 +++++++ .../vmagent_application_identity_test.yaml | 67 +++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml diff --git a/packages/system/monitoring-agents/templates/vmagent.yaml b/packages/system/monitoring-agents/templates/vmagent.yaml index 00ab63c10..f0b66bd59 100644 --- a/packages/system/monitoring-agents/templates/vmagent.yaml +++ b/packages/system/monitoring-agents/templates/vmagent.yaml @@ -20,6 +20,31 @@ spec: scrapeInterval: 30s selectAllByDefault: true + # The target always carries all three identity labels, "-" when the pod + # has no lineage ("-" is not a valid Kubernetes label value, so it never + # collides with a real one). That makes every exporter-supplied copy + # collide and be renamed to exported_* (honorLabels false), where it is + # dropped; the placeholder itself is removed before storage. + scrapeClasses: + - name: application-identity + default: true + relabelConfigs: + {{- range list "group" "kind" "name" }} + - targetLabel: apps_cozystack_io_application_{{ . }} + replacement: "-" + - sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_{{ . }}] + regex: (.+) + targetLabel: apps_cozystack_io_application_{{ . }} + {{- end }} + metricRelabelConfigs: + - action: labeldrop + regex: (exported_)+apps_cozystack_io_application_(group|kind|name) + {{- range list "group" "kind" "name" }} + - sourceLabels: [apps_cozystack_io_application_{{ . }}] + regex: "-" + targetLabel: apps_cozystack_io_application_{{ . }} + replacement: "" + {{- end }} additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional.yaml diff --git a/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml b/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml new file mode 100644 index 000000000..601f1c926 --- /dev/null +++ b/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml @@ -0,0 +1,67 @@ +suite: cluster-wide vmagent stamps application identity on exporter series +templates: + - templates/vmagent.yaml +release: + name: monitoring-agents + namespace: cozy-monitoring + +# The tenant-resource-metrics proxy scopes a query to one application by +# apps_cozystack_io_application_{group,kind,name}. The labels are only a +# boundary if the scraped pod cannot choose them: the target always carries +# all three (a placeholder when the pod has no lineage), so with +# honorLabels false an exporter's own copy is renamed to exported_* and +# dropped. The same rules live in the tenant vmagent of the monitoring +# package and are pinned there by an identical suite. +tests: + - it: every scrape gets the application identity class unless it names another + asserts: + - equal: + path: spec.scrapeClasses[0].name + value: application-identity + - equal: + path: spec.scrapeClasses[0].default + value: true + - lengthEqual: + path: spec.scrapeClasses + count: 1 + + - it: the identity comes from the scraped pod's lineage labels, with a placeholder when the pod has none + asserts: + - equal: + path: spec.scrapeClasses[0].relabelConfigs + value: + - targetLabel: apps_cozystack_io_application_group + replacement: "-" + - sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_group] + regex: (.+) + targetLabel: apps_cozystack_io_application_group + - targetLabel: apps_cozystack_io_application_kind + replacement: "-" + - sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_kind] + regex: (.+) + targetLabel: apps_cozystack_io_application_kind + - targetLabel: apps_cozystack_io_application_name + replacement: "-" + - sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_name] + regex: (.+) + targetLabel: apps_cozystack_io_application_name + + - it: an exporter-supplied identity is dropped and the placeholder never reaches storage + asserts: + - equal: + path: spec.scrapeClasses[0].metricRelabelConfigs + value: + - action: labeldrop + regex: (exported_)+apps_cozystack_io_application_(group|kind|name) + - sourceLabels: [apps_cozystack_io_application_group] + regex: "-" + targetLabel: apps_cozystack_io_application_group + replacement: "" + - sourceLabels: [apps_cozystack_io_application_kind] + regex: "-" + targetLabel: apps_cozystack_io_application_kind + replacement: "" + - sourceLabels: [apps_cozystack_io_application_name] + regex: "-" + targetLabel: apps_cozystack_io_application_name + replacement: "" From 14f815a5cc543ba35eeec1d407cc07a3b5b7fea1 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:16:56 +0700 Subject: [PATCH 048/518] feat(monitoring): stamp application identity on tenant vmagent scrapes Application pods in a tenant's namespaces are also scraped by the tenant's own vmagent, and its series need the same identity labels and the same protection against an exporter choosing them, or the two stores would disagree about which application a series belongs to. The class is identical to the cluster-wide one in monitoring-agents. Assisted-by: LLM Signed-off-by: ZverGuy --- .../monitoring/templates/vm/vmagent.yaml | 25 ++++++++ .../vmagent_application_identity_test.yaml | 62 +++++++++++++++++++ 2 files changed, 87 insertions(+) create mode 100644 packages/system/monitoring/tests/vmagent_application_identity_test.yaml diff --git a/packages/system/monitoring/templates/vm/vmagent.yaml b/packages/system/monitoring/templates/vm/vmagent.yaml index 98bff1e14..f65cd287e 100644 --- a/packages/system/monitoring/templates/vm/vmagent.yaml +++ b/packages/system/monitoring/templates/vm/vmagent.yaml @@ -24,6 +24,31 @@ spec: scrapeInterval: 30s selectAllByDefault: false + # The target always carries all three identity labels, "-" when the pod + # has no lineage ("-" is not a valid Kubernetes label value, so it never + # collides with a real one). That makes every exporter-supplied copy + # collide and be renamed to exported_* (honorLabels false), where it is + # dropped; the placeholder itself is removed before storage. + scrapeClasses: + - name: application-identity + default: true + relabelConfigs: + {{- range list "group" "kind" "name" }} + - targetLabel: apps_cozystack_io_application_{{ . }} + replacement: "-" + - sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_{{ . }}] + regex: (.+) + targetLabel: apps_cozystack_io_application_{{ . }} + {{- end }} + metricRelabelConfigs: + - action: labeldrop + regex: (exported_)+apps_cozystack_io_application_(group|kind|name) + {{- range list "group" "kind" "name" }} + - sourceLabels: [apps_cozystack_io_application_{{ . }}] + regex: "-" + targetLabel: apps_cozystack_io_application_{{ . }} + replacement: "" + {{- end }} {{- with .Values.vmagent.inlineScrapeConfig }} inlineScrapeConfig: | {{- . | nindent 4 }} diff --git a/packages/system/monitoring/tests/vmagent_application_identity_test.yaml b/packages/system/monitoring/tests/vmagent_application_identity_test.yaml new file mode 100644 index 000000000..f36105f83 --- /dev/null +++ b/packages/system/monitoring/tests/vmagent_application_identity_test.yaml @@ -0,0 +1,62 @@ +suite: tenant vmagent stamps application identity on exporter series +templates: + - templates/vm/vmagent.yaml +release: + name: monitoring + namespace: tenant-foo + +# Same rules as the cluster-wide vmagent in monitoring-agents, pinned there +# by an identical suite; see it for why the placeholder and the drop exist. +tests: + - it: every scrape gets the application identity class unless it names another + asserts: + - equal: + path: spec.scrapeClasses[0].name + value: application-identity + - equal: + path: spec.scrapeClasses[0].default + value: true + - lengthEqual: + path: spec.scrapeClasses + count: 1 + + - it: the identity comes from the scraped pod's lineage labels, with a placeholder when the pod has none + asserts: + - equal: + path: spec.scrapeClasses[0].relabelConfigs + value: + - targetLabel: apps_cozystack_io_application_group + replacement: "-" + - sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_group] + regex: (.+) + targetLabel: apps_cozystack_io_application_group + - targetLabel: apps_cozystack_io_application_kind + replacement: "-" + - sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_kind] + regex: (.+) + targetLabel: apps_cozystack_io_application_kind + - targetLabel: apps_cozystack_io_application_name + replacement: "-" + - sourceLabels: [__meta_kubernetes_pod_label_apps_cozystack_io_application_name] + regex: (.+) + targetLabel: apps_cozystack_io_application_name + + - it: an exporter-supplied identity is dropped and the placeholder never reaches storage + asserts: + - equal: + path: spec.scrapeClasses[0].metricRelabelConfigs + value: + - action: labeldrop + regex: (exported_)+apps_cozystack_io_application_(group|kind|name) + - sourceLabels: [apps_cozystack_io_application_group] + regex: "-" + targetLabel: apps_cozystack_io_application_group + replacement: "" + - sourceLabels: [apps_cozystack_io_application_kind] + regex: "-" + targetLabel: apps_cozystack_io_application_kind + replacement: "" + - sourceLabels: [apps_cozystack_io_application_name] + regex: "-" + targetLabel: apps_cozystack_io_application_name + replacement: "" From e5f17148368f6187fd36c7fbd0348ed033d98630 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:12:02 +0700 Subject: [PATCH 049/518] feat(monitoring): keep application identity on container log lines A reader that scopes logs to one application needs the owning application on every line, and the lineage webhook already puts it on the pod as apps.cozystack.io/application.{group,kind,name}. Fluent Bit fetched those labels and then dropped them with the rest of the kubernetes map, so the store could only be scoped by namespace. Keep the three labels as apps_cozystack_io_application_{group,kind,name}, the names the metrics side uses, and make them stream fields on every container-log output so the scope is an indexed stream filter. A pod without the labels produces no such fields, and VictoriaLogs builds the stream from the fields present, so existing streams keep their identity; a labelled pod starts a new stream once. Assisted-by: LLM Signed-off-by: ZverGuy --- .../fluent-bit_application_identity_test.yaml | 67 +++++++++++++++++++ packages/system/monitoring-agents/values.yaml | 20 +++++- 2 files changed, 85 insertions(+), 2 deletions(-) create mode 100644 packages/system/monitoring-agents/tests/fluent-bit_application_identity_test.yaml diff --git a/packages/system/monitoring-agents/tests/fluent-bit_application_identity_test.yaml b/packages/system/monitoring-agents/tests/fluent-bit_application_identity_test.yaml new file mode 100644 index 000000000..d1815eb1b --- /dev/null +++ b/packages/system/monitoring-agents/tests/fluent-bit_application_identity_test.yaml @@ -0,0 +1,67 @@ +suite: fluent-bit keeps the application identity of a container log line + +# The lineage webhook labels a pod with apps.cozystack.io/application.{group, +# kind,name}. A container log line keeps them as top-level +# apps_cozystack_io_application_{group,kind,name}, the names the metrics side +# uses, and they are stream fields so a reader can scope to one application +# with extra_stream_filters. The kubernetes map, labels included, is otherwise +# dropped by Remove_wildcard kubernetes_, so the three keys have to be lifted +# and renamed out of the kubernetes_ prefix before that filter runs. +templates: + - charts/fluent-bit/templates/configmap.yaml +release: + name: monitoring-agents + namespace: cozy-monitoring +set: + global: + target: tenant-root +kubernetesProvider: + scheme: + "v1/Namespace": + gvr: + version: v1 + resource: namespaces + namespaced: false + objects: + - kind: Namespace + apiVersion: v1 + metadata: + name: tenant-a + labels: + namespace.cozystack.io/monitoring: tenant-a +tests: + - it: lifts the pod labels after the kubernetes map is lifted + asserts: + - matchRegex: + path: data["fluent-bit.conf"] + pattern: '(?s)Operation lift\n\s+Nested_under kubernetes\n.*Name nest\n\s+Match kube\.\*\n\s+Operation lift\n\s+Nested_under kubernetes_labels\n\s+Add_prefix kubernetes_labels_\n' + + - it: renames the three lineage labels out of the kubernetes_ prefix before Remove_wildcard + asserts: + - matchRegex: + path: data["fluent-bit.conf"] + pattern: '(?s)Nested_under kubernetes_labels\n.*Rename kubernetes_labels_apps\.cozystack\.io/application\.group apps_cozystack_io_application_group\n\s+Rename kubernetes_labels_apps\.cozystack\.io/application\.kind apps_cozystack_io_application_kind\n\s+Rename kubernetes_labels_apps\.cozystack\.io/application\.name apps_cozystack_io_application_name\n.*Remove_wildcard kubernetes_\n' + + - it: makes the lineage keys stream fields on the default container-log output + asserts: + - matchRegex: + path: data["fluent-bit.conf"] + pattern: 'Match kube\.\*\n\s+Host vlinsert-generic\.tenant-root\.svc\n\s+port 9481\n\s+compress gzip\n\s+uri /insert/jsonline\?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&' + + - it: makes the lineage keys stream fields on a routed tenant's container-log output + asserts: + - matchRegex: + path: data["fluent-bit.conf"] + pattern: 'Match tenant\.tenant-a\.\*\n\s+Host vlinsert-generic\.tenant-a\.svc\n\s+port 9481\n\s+compress gzip\n\s+uri /insert/jsonline\?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&' + + - it: leaves the events and audit outputs without lineage stream fields + asserts: + - matchRegex: + path: data["fluent-bit.conf"] + pattern: 'uri /insert/jsonline\?_stream_fields=log_source,reason,metadata_namespace,metadata_name&' + - matchRegex: + path: data["fluent-bit.conf"] + pattern: 'uri /insert/jsonline\?_stream_fields=log_source,stage,user_username,verb,requestUri&' + - notMatchRegex: + path: data["fluent-bit.conf"] + pattern: 'Match (events\.\*|audit\.\*|\*)\n[^\[]*Nested_under kubernetes_labels' diff --git a/packages/system/monitoring-agents/values.yaml b/packages/system/monitoring-agents/values.yaml index a9c22cb14..186828f70 100644 --- a/packages/system/monitoring-agents/values.yaml +++ b/packages/system/monitoring-agents/values.yaml @@ -405,7 +405,7 @@ fluent-bit: Host vlinsert-generic.{{ .Values.global.target }}.svc port 9481 compress gzip - uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&_msg_field=log&_time_field=date + uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&_msg_field=log&_time_field=date format json_lines json_date_format iso8601 header AccountID 0 @@ -439,7 +439,7 @@ fluent-bit: Host vlinsert-generic.{{ $target }}.svc port 9481 compress gzip - uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&_msg_field=log&_time_field=date + uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&_msg_field=log&_time_field=date format json_lines json_date_format iso8601 header AccountID 0 @@ -461,6 +461,19 @@ fluent-bit: Operation lift Nested_under kubernetes Add_prefix kubernetes_ + {{- /* + The pod's lineage labels name the application it belongs to. lift + cannot pick keys, so every label comes up; all but these three keep + the kubernetes_ prefix and fall to Remove_wildcard below. An absent + label stays absent, and VictoriaLogs builds _stream from the stream + fields that are present, so an unlabelled pod keeps its stream. + */}} + [FILTER] + Name nest + Match kube.* + Operation lift + Nested_under kubernetes_labels + Add_prefix kubernetes_labels_ [FILTER] Name modify Match kube.* @@ -501,6 +514,9 @@ fluent-bit: Rename kubernetes_pod_name keep_kubernetes_pod_name Rename kubernetes_namespace_name keep_kubernetes_namespace_name Rename kubernetes_container_name keep_kubernetes_container_name + Rename kubernetes_labels_apps.cozystack.io/application.group apps_cozystack_io_application_group + Rename kubernetes_labels_apps.cozystack.io/application.kind apps_cozystack_io_application_kind + Rename kubernetes_labels_apps.cozystack.io/application.name apps_cozystack_io_application_name [FILTER] Name modify Match kube.* From 95dc5911981adf1ad37aca6c949f33c90708d2a2 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:12:02 +0700 Subject: [PATCH 050/518] test(kubernetes): keep application identity out of guest log streams Guest pods are tenant-controlled, so their lineage labels are not a trustworthy application identity, and the platform must not promote them to the stream fields the parent scopes application logs by. The guest release replaces the monitoring-agents filters and outputs wholesale, which is what keeps the parent's label lift out of guests; pin that so a later change to the guest overrides cannot bring it in. This only covers what the platform ships. A guest admin can still edit the guest's fluent-bit or write to the parent's vlinsert directly; that needs a write-path control, which the logs proposal leaves open. Assisted-by: LLM Signed-off-by: ZverGuy --- ...ring_agents_application_identity_test.yaml | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 packages/apps/kubernetes/tests/monitoring_agents_application_identity_test.yaml diff --git a/packages/apps/kubernetes/tests/monitoring_agents_application_identity_test.yaml b/packages/apps/kubernetes/tests/monitoring_agents_application_identity_test.yaml new file mode 100644 index 000000000..9fbc1c45b --- /dev/null +++ b/packages/apps/kubernetes/tests/monitoring_agents_application_identity_test.yaml @@ -0,0 +1,38 @@ +suite: guest fluent-bit never ships application identity stream fields + +templates: + - templates/helmreleases/monitoring-agents.yaml + +values: + - values/common.yaml + +release: + name: kubernetes-test + namespace: tenant-test + +# Pods in a guest cluster are tenant-controlled, so their +# apps.cozystack.io/application.* labels are not a trustworthy application +# identity, and the platform must not promote them to the stream fields the +# parent scopes application logs by. The guest release replaces the +# monitoring-agents filters and outputs, so it is the guest overrides that keep +# the parent's label lift out. This does not stop a guest admin who edits the +# guest's own fluent-bit or writes to the parent's vlinsert directly; that is +# the write-path control the logs proposal leaves open. + +tests: + - it: does not lift pod labels or declare lineage stream fields in the guest + set: + addons.monitoringAgents.enabled: true + _namespace.monitoring: tenant-test + asserts: + - isKind: + of: HelmRelease + - notMatchRegex: + path: spec.values.fluent-bit.config.filters + pattern: 'kubernetes_labels|apps_cozystack_io_application_' + - notMatchRegex: + path: spec.values.fluent-bit.config.outputs + pattern: 'apps_cozystack_io_application_' + - matchRegex: + path: spec.values.fluent-bit.config.outputs + pattern: '_stream_fields=stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&' From 0e5557d0baef802125b88d438d741055da5ee408 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:26:04 +0700 Subject: [PATCH 051/518] fix(ingress): keep application identity off ingress-nginx series The vmagent scrape class stamps the scraped pod's application identity on its series. ingress-nginx series describe the Ingress objects the controller serves, in whatever namespace they live, so the controller's own identity would attribute another tenant's traffic to the Ingress application. And because these scrapes honor exporter labels, an exporter-supplied identity would win over the class. Dropping the labels leaves these series to the pod and namespace scopes. Assisted-by: LLM Signed-off-by: ZverGuy --- .../extra/ingress/templates/nginx-scrape.yaml | 10 +++++++++ .../tests/application_identity_test.yaml | 21 +++++++++++++++++++ 2 files changed, 31 insertions(+) create mode 100644 packages/extra/ingress/tests/application_identity_test.yaml diff --git a/packages/extra/ingress/templates/nginx-scrape.yaml b/packages/extra/ingress/templates/nginx-scrape.yaml index d04c5b651..780f5e3a9 100644 --- a/packages/extra/ingress/templates/nginx-scrape.yaml +++ b/packages/extra/ingress/templates/nginx-scrape.yaml @@ -9,6 +9,10 @@ spec: matchNames: - {{ .Release.Namespace }} podMetricsEndpoints: + # The series describe the Ingress objects served, in any namespace, not + # this controller, so the application identity the vmagent scrape class + # stamps from the controller pod would misattribute them; and with + # honorLabels an exporter-supplied identity would win over the class. - port: metrics honorLabels: true interval: 30s @@ -19,6 +23,9 @@ spec: targetLabel: node - targetLabel: tier replacement: cluster + metricRelabelConfigs: + - action: labeldrop + regex: apps_cozystack_io_application_(group|kind|name) selector: matchLabels: app.kubernetes.io/component: controller @@ -46,6 +53,9 @@ spec: targetLabel: controller - targetLabel: tier replacement: cluster + metricRelabelConfigs: + - action: labeldrop + regex: apps_cozystack_io_application_(group|kind|name) selector: matchLabels: app.kubernetes.io/component: controller diff --git a/packages/extra/ingress/tests/application_identity_test.yaml b/packages/extra/ingress/tests/application_identity_test.yaml new file mode 100644 index 000000000..e1be5ca4f --- /dev/null +++ b/packages/extra/ingress/tests/application_identity_test.yaml @@ -0,0 +1,21 @@ +suite: ingress-nginx series do not carry an application identity +templates: + - templates/nginx-scrape.yaml +release: + name: ingress + namespace: tenant-root + +# The vmagent scrape class stamps the scraped pod's application identity on +# every series. ingress-nginx series describe the Ingress objects it serves, +# in whatever namespace they live (honorLabels keeps their namespace), so +# the controller's identity would misattribute them, and with honorLabels +# true an exporter-supplied identity would win over the class. Both +# endpoints drop the identity instead. +tests: + - it: drops the identity on every endpoint + asserts: + - equal: + path: spec.podMetricsEndpoints[0].metricRelabelConfigs + value: + - action: labeldrop + regex: apps_cozystack_io_application_(group|kind|name) From dca95eb753fa31271eb7b3cbbe3a0312da138846 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:26:04 +0700 Subject: [PATCH 052/518] docs(monitoring): state when the identity scrape class can be overridden The exported_* rename only protects scrapes that do not honor exporter labels, and a scrape object's own relabel rules run after the class, so both can undo it. Saying so next to the rules keeps the next scrape author from assuming the protection is unconditional. Assisted-by: LLM Signed-off-by: ZverGuy --- packages/system/monitoring-agents/templates/vmagent.yaml | 8 +++++--- packages/system/monitoring/templates/vm/vmagent.yaml | 8 +++++--- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/packages/system/monitoring-agents/templates/vmagent.yaml b/packages/system/monitoring-agents/templates/vmagent.yaml index f0b66bd59..29349f17e 100644 --- a/packages/system/monitoring-agents/templates/vmagent.yaml +++ b/packages/system/monitoring-agents/templates/vmagent.yaml @@ -22,9 +22,11 @@ spec: selectAllByDefault: true # The target always carries all three identity labels, "-" when the pod # has no lineage ("-" is not a valid Kubernetes label value, so it never - # collides with a real one). That makes every exporter-supplied copy - # collide and be renamed to exported_* (honorLabels false), where it is - # dropped; the placeholder itself is removed before storage. + # collides with a real one). Under honorLabels false every exporter-supplied + # copy then collides, is renamed to exported_* and dropped. Under + # honorLabels true the exporter's copy wins, so such a scrape must drop + # these labels itself. A scrape object's own relabel rules run after these + # and must leave the labels alone. The placeholder is removed before storage. scrapeClasses: - name: application-identity default: true diff --git a/packages/system/monitoring/templates/vm/vmagent.yaml b/packages/system/monitoring/templates/vm/vmagent.yaml index f65cd287e..fa4c4fc93 100644 --- a/packages/system/monitoring/templates/vm/vmagent.yaml +++ b/packages/system/monitoring/templates/vm/vmagent.yaml @@ -26,9 +26,11 @@ spec: selectAllByDefault: false # The target always carries all three identity labels, "-" when the pod # has no lineage ("-" is not a valid Kubernetes label value, so it never - # collides with a real one). That makes every exporter-supplied copy - # collide and be renamed to exported_* (honorLabels false), where it is - # dropped; the placeholder itself is removed before storage. + # collides with a real one). Under honorLabels false every exporter-supplied + # copy then collides, is renamed to exported_* and dropped. Under + # honorLabels true the exporter's copy wins, so such a scrape must drop + # these labels itself. A scrape object's own relabel rules run after these + # and must leave the labels alone. The placeholder is removed before storage. scrapeClasses: - name: application-identity default: true From 5a6086c7a1756499f10fd869e20e7bd3c14a5e17 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:30:32 +0700 Subject: [PATCH 053/518] test(monitoring): pin the endpoint count the ingress identity test relies on The ingress assertion reads only the first endpoint of each scrape, so a second endpoint added without the label drop would go unnoticed. Pinning the count makes such an addition fail the suite. Assisted-by: LLM Signed-off-by: ZverGuy --- packages/extra/ingress/tests/application_identity_test.yaml | 3 +++ .../tests/vmagent_application_identity_test.yaml | 2 +- .../monitoring/tests/vmagent_application_identity_test.yaml | 3 ++- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/packages/extra/ingress/tests/application_identity_test.yaml b/packages/extra/ingress/tests/application_identity_test.yaml index e1be5ca4f..0e53a2756 100644 --- a/packages/extra/ingress/tests/application_identity_test.yaml +++ b/packages/extra/ingress/tests/application_identity_test.yaml @@ -14,6 +14,9 @@ release: tests: - it: drops the identity on every endpoint asserts: + - lengthEqual: + path: spec.podMetricsEndpoints + count: 1 - equal: path: spec.podMetricsEndpoints[0].metricRelabelConfigs value: diff --git a/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml b/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml index 601f1c926..b59f9e3ee 100644 --- a/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml +++ b/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml @@ -11,7 +11,7 @@ release: # all three (a placeholder when the pod has no lineage), so with # honorLabels false an exporter's own copy is renamed to exported_* and # dropped. The same rules live in the tenant vmagent of the monitoring -# package and are pinned there by an identical suite. +# package and are pinned there by the same assertions. tests: - it: every scrape gets the application identity class unless it names another asserts: diff --git a/packages/system/monitoring/tests/vmagent_application_identity_test.yaml b/packages/system/monitoring/tests/vmagent_application_identity_test.yaml index f36105f83..da14330de 100644 --- a/packages/system/monitoring/tests/vmagent_application_identity_test.yaml +++ b/packages/system/monitoring/tests/vmagent_application_identity_test.yaml @@ -6,7 +6,8 @@ release: namespace: tenant-foo # Same rules as the cluster-wide vmagent in monitoring-agents, pinned there -# by an identical suite; see it for why the placeholder and the drop exist. +# by the same assertions; see that suite for why the placeholder and the +# drop exist. tests: - it: every scrape gets the application identity class unless it names another asserts: From e96ceedd7ebd6a8e138e8b363ef36a2ffbf7e20f Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:25:58 +0700 Subject: [PATCH 054/518] feat(dashboard): add a console client for the observability proxy The observability proxy serves metrics and logs under Kubernetes-shaped paths but answers in native Prometheus JSON and VictoriaLogs NDJSON, so the typed K8sClient methods cannot read it. A raw fetch on the client keeps the same credentials and 401 handling the rest of the console uses, and the proxy module maps each documented status to its own error kind so a cluster without the proxy (404) can be told apart from a denied request or a store that is down. Assisted-by: LLM Signed-off-by: ZverGuy --- .../__tests__/k8s-client/client.raw.test.ts | 47 ++++ .../src/lib/observability/proxy.test.ts | 179 +++++++++++++++ .../console/src/lib/observability/proxy.ts | 203 ++++++++++++++++++ .../console/packages/k8s-client/src/client.ts | 18 ++ 4 files changed, 447 insertions(+) create mode 100644 packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.raw.test.ts create mode 100644 packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts create mode 100644 packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts diff --git a/packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.raw.test.ts b/packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.raw.test.ts new file mode 100644 index 000000000..ab6c056f5 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.raw.test.ts @@ -0,0 +1,47 @@ +import { describe, it, expect, vi, afterEach } from "vitest" +import { K8sClient } from "@cozystack/k8s-client" + +afterEach(() => { + vi.unstubAllGlobals() + vi.restoreAllMocks() +}) + +describe("K8sClient.raw", () => { + it("returns the response untouched, including a non-2xx one", async () => { + const res = new Response("not found", { status: 404 }) + vi.stubGlobal("fetch", vi.fn(async () => res)) + const client = new K8sClient() + + await expect(client.raw("/x")).resolves.toBe(res) + }) + + it("prefixes the base URL and attaches the bearer token", async () => { + const fetchMock = vi.fn(async () => new Response("{}")) + vi.stubGlobal("fetch", fetchMock) + const client = new K8sClient({ baseUrl: "/base", getToken: async () => "t0k" }) + + await client.raw("/apis/x", { headers: { Accept: "application/json" } }) + + const [url, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] + expect(url).toBe("/base/apis/x") + expect(init.headers).toEqual({ Accept: "application/json", Authorization: "Bearer t0k" }) + }) + + it("routes a 401 through the client's unauthorized handler", async () => { + const onUnauthorized = vi.fn() + vi.stubGlobal("fetch", vi.fn(async () => new Response("", { status: 401 }))) + const client = new K8sClient({ onUnauthorized }) + + const res = await client.raw("/x") + + expect(res.status).toBe(401) + expect(onUnauthorized).toHaveBeenCalledTimes(1) + }) + + it("does not swallow a network failure", async () => { + vi.stubGlobal("fetch", vi.fn(async () => { throw new TypeError("offline") })) + const client = new K8sClient() + + await expect(client.raw("/x")).rejects.toThrow("offline") + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts new file mode 100644 index 000000000..f06835ff7 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts @@ -0,0 +1,179 @@ +import { describe, it, expect, vi } from "vitest" +import { K8sClient } from "@cozystack/k8s-client" +import { + ProxyError, + errorKindForStatus, + fieldValues, + logsRoot, + metricsRoot, + queryLogs, + queryRange, + tailLogs, +} from "./proxy.ts" + +const app = { namespace: "tenant-a", plural: "postgreses", name: "db" } +const ns = { namespace: "tenant-a" } + +function clientReturning(...responses: Response[]) { + const client = new K8sClient({ onUnauthorized: () => {} }) + const raw = vi.spyOn(client, "raw") + for (const r of responses) raw.mockResolvedValueOnce(r) + return { client, raw } +} + +function calledUrl(raw: ReturnType["raw"], call = 0): URL { + return new URL(raw.mock.calls[call][0], "http://x") +} + +describe("roots", () => { + it("builds the application roots under apps.cozystack.io", () => { + expect(metricsRoot(app)).toBe("/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/metrics/") + expect(logsRoot(app)).toBe("/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/log/") + }) + + it("builds the namespace roots under cozystack.io", () => { + expect(metricsRoot(ns)).toBe("/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/") + expect(logsRoot(ns)).toBe("/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/") + }) +}) + +describe("errorKindForStatus", () => { + it.each([ + [400, "bad-request"], + [401, "unauthorized"], + [403, "forbidden"], + [404, "not-available"], + [422, "limit"], + [429, "rate-limited"], + [502, "store-unreachable"], + [503, "unavailable"], + [504, "timeout"], + [500, "unknown"], + ])("maps %i to %s", (status, kind) => { + expect(errorKindForStatus(status)).toBe(kind) + }) +}) + +describe("queryRange", () => { + it("GETs query_range under the root and returns the matrix", async () => { + const body = { status: "success", data: { resultType: "matrix", result: [{ metric: { pod: "p" }, values: [[1, "2"]] }] } } + const { client, raw } = clientReturning(Response.json(body)) + + const result = await queryRange(client, metricsRoot(app), { query: "up", start: 10, end: 20, step: 5 }) + + const url = calledUrl(raw) + expect(url.pathname).toBe(`${metricsRoot(app)}api/v1/query_range`) + expect(Object.fromEntries(url.searchParams)).toEqual({ query: "up", start: "10", end: "20", step: "5" }) + expect(result).toEqual(body.data.result) + }) + + it("throws a ProxyError carrying the Prometheus error text", async () => { + const { client } = clientReturning( + Response.json({ status: "error", error: "range too long" }, { status: 422 }), + ) + + const err = await queryRange(client, metricsRoot(app), { query: "up", start: 0, end: 1, step: 1 }).catch((e) => e) + + expect(err).toBeInstanceOf(ProxyError) + expect(err).toMatchObject({ kind: "limit", status: 422, message: "range too long" }) + }) + + it("reports a fetch failure as a network error", async () => { + const client = new K8sClient() + vi.spyOn(client, "raw").mockRejectedValue(new TypeError("offline")) + + const err = await queryRange(client, metricsRoot(app), { query: "up", start: 0, end: 1, step: 1 }).catch((e) => e) + + expect(err).toMatchObject({ kind: "network" }) + }) +}) + +describe("queryLogs", () => { + it("parses NDJSON lines and sorts them by _time ascending", async () => { + const ndjson = [ + JSON.stringify({ _time: "2026-11-04T10:00:02Z", _msg: "b" }), + "", + JSON.stringify({ _time: "2026-11-04T10:00:01Z", _msg: "a" }), + ].join("\n") + const { client, raw } = clientReturning(new Response(ndjson)) + + const entries = await queryLogs(client, logsRoot(ns), { query: "*", start: "s", end: "e", limit: 100 }) + + expect(calledUrl(raw).pathname).toBe(`${logsRoot(ns)}select/logsql/query`) + expect(Object.fromEntries(calledUrl(raw).searchParams)).toEqual({ query: "*", start: "s", end: "e", limit: "100" }) + expect(entries.map((e) => e._msg)).toEqual(["a", "b"]) + }) + + it("keeps a plain-text error body as the message", async () => { + const { client } = clientReturning(new Response("cannot parse query", { status: 400 })) + + const err = await queryLogs(client, logsRoot(ns), { query: "(", start: "s", end: "e", limit: 1 }).catch((e) => e) + + expect(err).toMatchObject({ kind: "bad-request", message: "cannot parse query" }) + }) +}) + +describe("fieldValues", () => { + it("uses stream_field_values for stream fields and returns the values", async () => { + const { client, raw } = clientReturning( + Response.json({ values: [{ value: "pod-b", hits: 1 }, { value: "pod-a", hits: 9 }] }), + ) + + const values = await fieldValues(client, logsRoot(app), { + field: "kubernetes_pod_name", + stream: true, + query: "*", + start: "s", + end: "e", + }) + + expect(calledUrl(raw).pathname).toBe(`${logsRoot(app)}select/logsql/stream_field_values`) + expect(calledUrl(raw).searchParams.get("field")).toBe("kubernetes_pod_name") + expect(values).toEqual(["pod-a", "pod-b"]) + }) + + it("uses field_values for other fields", async () => { + const { client, raw } = clientReturning(Response.json({ values: [] })) + + await fieldValues(client, logsRoot(app), { field: "level", stream: false, query: "*", start: "s", end: "e" }) + + expect(calledUrl(raw).pathname).toBe(`${logsRoot(app)}select/logsql/field_values`) + }) +}) + +describe("tailLogs", () => { + it("delivers every line as it arrives, including one split across chunks", async () => { + const enc = new TextEncoder() + const chunks = ['{"_time":"t1","_msg":"a"}\n{"_ti', 'me":"t2","_msg":"b"}\n'] + const body = new ReadableStream({ + start(c) { + for (const ch of chunks) c.enqueue(enc.encode(ch)) + c.close() + }, + }) + const { client, raw } = clientReturning(new Response(body)) + const seen: string[] = [] + + await tailLogs(client, logsRoot(ns), { query: "*", since: "t0" }, (e) => seen.push(e._msg)) + + expect(calledUrl(raw).pathname).toBe(`${logsRoot(ns)}select/logsql/tail`) + expect(calledUrl(raw).searchParams.get("since")).toBe("t0") + expect(seen).toEqual(["a", "b"]) + }) + + it("omits since on the first connect", async () => { + const { client, raw } = clientReturning(new Response("")) + + await tailLogs(client, logsRoot(ns), { query: "*" }, () => {}) + + expect(calledUrl(raw).searchParams.has("since")).toBe(false) + }) + + it("throws a ProxyError when the tail is refused", async () => { + const { client } = clientReturning(new Response("", { status: 403 })) + + const err = await tailLogs(client, logsRoot(ns), { query: "*" }, () => {}).catch((e) => e) + + expect(err).toMatchObject({ kind: "forbidden" }) + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts new file mode 100644 index 000000000..a257825c9 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts @@ -0,0 +1,203 @@ +import type { K8sClient } from "@cozystack/k8s-client" + +export type ObservabilityScope = + | { namespace: string } + | { namespace: string; plural: string; name: string } + +function appBase(scope: ObservabilityScope): string | null { + if (!("plural" in scope)) return null + return `/apis/apps.cozystack.io/v1alpha1/namespaces/${scope.namespace}/${scope.plural}/${scope.name}` +} + +export function metricsRoot(scope: ObservabilityScope): string { + const app = appBase(scope) + return app ? `${app}/metrics/` : `/apis/cozystack.io/v1alpha1/namespaces/${scope.namespace}/metrics/` +} + +export function logsRoot(scope: ObservabilityScope): string { + const app = appBase(scope) + return app ? `${app}/log/` : `/apis/cozystack.io/v1alpha1/namespaces/${scope.namespace}/logs/` +} + +export type ProxyErrorKind = + | "bad-request" + | "unauthorized" + | "forbidden" + | "not-available" + | "limit" + | "rate-limited" + | "store-unreachable" + | "unavailable" + | "timeout" + | "network" + | "unknown" + +const KIND_BY_STATUS: Record = { + 400: "bad-request", + 401: "unauthorized", + 403: "forbidden", + 404: "not-available", + 422: "limit", + 429: "rate-limited", + 502: "store-unreachable", + 503: "unavailable", + 504: "timeout", +} + +export function errorKindForStatus(status: number): ProxyErrorKind { + return KIND_BY_STATUS[status] ?? "unknown" +} + +export class ProxyError extends Error { + kind: ProxyErrorKind + status: number + + constructor(kind: ProxyErrorKind, status: number, message: string) { + super(message) + this.name = "ProxyError" + this.kind = kind + this.status = status + } +} + +// Prometheus answers {"error": "..."}, VictoriaLogs and nginx answer plain text. +function errorMessage(text: string, status: number): string { + try { + const parsed: unknown = JSON.parse(text) + if (parsed && typeof parsed === "object" && "error" in parsed && typeof parsed.error === "string") { + return parsed.error + } + } catch { + // not JSON + } + const trimmed = text.trim() + return trimmed && trimmed.length < 300 && !trimmed.startsWith("<") ? trimmed : `Server returned ${status}` +} + +async function proxyGet( + client: K8sClient, + path: string, + params: Record, + signal?: AbortSignal, +): Promise { + const search = new URLSearchParams() + for (const [k, v] of Object.entries(params)) if (v !== undefined) search.set(k, String(v)) + let res: Response + try { + res = await client.raw(`${path}?${search}`, { signal }) + } catch (err) { + if (signal?.aborted) throw err + throw new ProxyError("network", 0, "Unable to connect to the server") + } + if (!res.ok) { + const text = await res.text().catch(() => "") + throw new ProxyError(errorKindForStatus(res.status), res.status, errorMessage(text, res.status)) + } + return res +} + +export interface PromSeries { + metric: Record + values: [number, string][] +} + +export interface RangeParams { + query: string + start: number + end: number + step: number +} + +export async function queryRange( + client: K8sClient, + root: string, + params: RangeParams, + signal?: AbortSignal, +): Promise { + const res = await proxyGet(client, `${root}api/v1/query_range`, { ...params }, signal) + const body = (await res.json()) as { data?: { result?: PromSeries[] } } + return body.data?.result ?? [] +} + +export interface LogEntry { + _time: string + _msg: string + _stream_id?: string + [field: string]: string | undefined +} + +function parseLine(line: string): LogEntry | null { + if (!line.trim()) return null + try { + const value: unknown = JSON.parse(line) + if (value && typeof value === "object" && "_time" in value) return value as LogEntry + } catch { + // a torn or non-JSON line carries nothing to show + } + return null +} + +export interface LogsQueryParams { + query: string + start: string + end: string + limit: number +} + +export async function queryLogs( + client: K8sClient, + root: string, + params: LogsQueryParams, + signal?: AbortSignal, +): Promise { + const res = await proxyGet(client, `${root}select/logsql/query`, { ...params }, signal) + const entries = (await res.text()).split("\n").map(parseLine).filter((e) => e !== null) + return entries.sort((a, b) => (a._time < b._time ? -1 : a._time > b._time ? 1 : 0)) +} + +export interface FieldValuesParams { + field: string + stream: boolean + query: string + start: string + end: string +} + +export async function fieldValues( + client: K8sClient, + root: string, + { stream, ...params }: FieldValuesParams, + signal?: AbortSignal, +): Promise { + const endpoint = stream ? "stream_field_values" : "field_values" + const res = await proxyGet(client, `${root}select/logsql/${endpoint}`, { ...params }, signal) + const body = (await res.json()) as { values?: { value: string }[] } + return (body.values ?? []).map((v) => v.value).sort() +} + +export async function tailLogs( + client: K8sClient, + root: string, + params: { query: string; since?: string }, + onEntry: (entry: LogEntry) => void, + signal?: AbortSignal, +): Promise { + const res = await proxyGet(client, `${root}select/logsql/tail`, params, signal) + if (!res.body) return + const reader = res.body.getReader() + const decoder = new TextDecoder() + let buffer = "" + for (;;) { + const { done, value } = await reader.read() + if (done) break + buffer += decoder.decode(value, { stream: true }) + const lines = buffer.split("\n") + buffer = lines.pop() ?? "" + for (const line of lines) { + const entry = parseLine(line) + if (entry) onEntry(entry) + } + } + const last = parseLine(buffer) + if (last) onEntry(last) +} diff --git a/packages/system/dashboard/images/console/packages/k8s-client/src/client.ts b/packages/system/dashboard/images/console/packages/k8s-client/src/client.ts index 12f50e0fd..04c528760 100644 --- a/packages/system/dashboard/images/console/packages/k8s-client/src/client.ts +++ b/packages/system/dashboard/images/console/packages/k8s-client/src/client.ts @@ -241,6 +241,24 @@ export class K8sClient { }) } + /** + * Fetch a same-origin path with the client's credentials and 401 handling, + * leaving status and body to the caller. For endpoints that do not speak + * Kubernetes JSON, such as the observability proxy's streamed NDJSON. + */ + async raw(path: string, init?: RequestInit): Promise { + const headers: Record = { + ...(init?.headers as Record), + } + if (this.getToken) { + headers["Authorization"] = `Bearer ${await this.getToken()}` + } + const res = await fetch(`${this.baseUrl}${path}`, { ...init, headers }) + if (res.status === 401) this.handleUnauthorized() + else if (res.ok) this.unauthorizedHandled = false + return res + } + getApiGroups(): Promise { return this.request("/apis") } From 063e56ede57a481b2db223a37641e601a2d51a05 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:30:03 +0700 Subject: [PATCH 055/518] feat(dashboard): show resource consumption on application pages Tenants had no way to see what their applications consume. A Metrics tab on every application detail page (except VM disks, which run no containers) charts CPU, memory and network for a selectable range through the observability proxy's application root, which injects the scope server-side, so the queries stay kind-agnostic and no per-kind UI is needed. Charts use uPlot: about 25 KB gzipped with no dependencies, built for time series, and loaded only with the Metrics tab so other pages do not pay for it. A cluster without the proxy answers 404 and the tab says metrics are not available there instead of showing an error. Assisted-by: LLM Signed-off-by: ZverGuy --- .../images/console/apps/console/package.json | 3 +- .../observability/MetricsView.test.tsx | 88 +++++++++++++ .../components/observability/MetricsView.tsx | 72 +++++++++++ .../observability/ProxyNotice.test.tsx | 43 +++++++ .../components/observability/ProxyNotice.tsx | 49 +++++++ .../components/observability/RangePicker.tsx | 30 +++++ .../observability/TimeSeriesChart.test.tsx | 46 +++++++ .../observability/TimeSeriesChart.tsx | 51 ++++++++ .../src/lib/observability/metrics.test.ts | 107 ++++++++++++++++ .../console/src/lib/observability/metrics.ts | 121 ++++++++++++++++++ .../src/lib/observability/proxy.test.ts | 10 ++ .../console/src/lib/observability/proxy.ts | 8 ++ .../detail/ApplicationDetailPage.test.tsx | 28 ++++ .../routes/detail/ApplicationDetailPage.tsx | 12 ++ .../console/src/routes/detail/MetricsTab.tsx | 15 +++ .../dashboard/images/console/pnpm-lock.yaml | 8 ++ 16 files changed, 690 insertions(+), 1 deletion(-) create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.test.ts create mode 100644 packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts create mode 100644 packages/system/dashboard/images/console/apps/console/src/routes/detail/MetricsTab.tsx diff --git a/packages/system/dashboard/images/console/apps/console/package.json b/packages/system/dashboard/images/console/apps/console/package.json index fcddb7023..254195f28 100644 --- a/packages/system/dashboard/images/console/apps/console/package.json +++ b/packages/system/dashboard/images/console/apps/console/package.json @@ -31,7 +31,8 @@ "react": "^19.2.4", "react-dom": "^19.2.4", "react-router": "^7.6.1", - "simple-icons": "^16.22.0" + "simple-icons": "^16.22.0", + "uplot": "^1.6.32" }, "devDependencies": { "@tailwindcss/vite": "^4.2.2", diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx new file mode 100644 index 000000000..f09e15201 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx @@ -0,0 +1,88 @@ +import { describe, it, expect, vi } from "vitest" +import { screen, waitFor } from "@testing-library/react" +import userEvent from "@testing-library/user-event" +import { K8sClient } from "@cozystack/k8s-client" +import { renderWithK8sProvider } from "../../test-utils/render.tsx" +import type { ChartData } from "../../lib/observability/metrics.ts" + +vi.mock("./TimeSeriesChart.tsx", () => ({ + TimeSeriesChart: ({ data }: { data: ChartData }) => ( +
{data.series.map((s) => s.label).join(",")}
+ ), +})) + +const { MetricsView } = await import("./MetricsView.tsx") + +const scope = { namespace: "tenant-a", plural: "postgreses", name: "db" } + +function matrix(pod: string) { + return Response.json({ + status: "success", + data: { resultType: "matrix", result: [{ metric: { pod }, values: [[1, "1"]] }] }, + }) +} + +function clientWith(handler: (url: URL) => Response) { + const client = new K8sClient() + const raw = vi.spyOn(client, "raw").mockImplementation(async (path) => handler(new URL(path, "http://x"))) + return { client, raw } +} + +describe("MetricsView", () => { + it("draws CPU, memory and network from the application root", async () => { + const { client, raw } = clientWith(() => matrix("db-0")) + + renderWithK8sProvider(, { client }) + + await waitFor(() => expect(screen.getAllByTestId("chart")).toHaveLength(3)) + expect(screen.getByText("CPU (cores)")).toBeInTheDocument() + expect(screen.getByText("Memory (working set)")).toBeInTheDocument() + expect(screen.getByText("Network")).toBeInTheDocument() + for (const [path] of raw.mock.calls) { + expect(path.startsWith("/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/metrics/api/v1/query_range?")).toBe(true) + } + }) + + it("requeries with a coarser step when a longer range is picked", async () => { + const { client, raw } = clientWith(() => matrix("db-0")) + renderWithK8sProvider(, { client }) + await waitFor(() => expect(raw).toHaveBeenCalled()) + + await userEvent.click(screen.getByRole("button", { name: "7d" })) + + await waitFor(() => + expect(raw.mock.calls.some(([p]) => new URL(p, "http://x").searchParams.get("step") === "2520")).toBe(true), + ) + }) + + it("shows one quiet notice instead of three errors when the proxy is absent", async () => { + const { client } = clientWith(() => new Response("404 page not found", { status: 404 })) + + renderWithK8sProvider(, { client }) + + expect(await screen.findByText("Metrics are not available on this cluster.")).toBeInTheDocument() + expect(screen.getAllByRole("status")).toHaveLength(1) + expect(screen.queryByText("CPU (cores)")).toBeNull() + }) + + it("keeps the other panels when one query hits a limit", async () => { + const { client } = clientWith((url) => + url.searchParams.get("query")?.includes("memory") + ? Response.json({ status: "error", error: "too many series" }, { status: 422 }) + : matrix("db-0"), + ) + + renderWithK8sProvider(, { client }) + + expect(await screen.findByText("too many series. Choose a shorter range.")).toBeInTheDocument() + await waitFor(() => expect(screen.getAllByTestId("chart")).toHaveLength(2)) + }) + + it("says so when a panel has no data", async () => { + const { client } = clientWith(() => Response.json({ status: "success", data: { resultType: "matrix", result: [] } })) + + renderWithK8sProvider(, { client }) + + await waitFor(() => expect(screen.getAllByText("No data for this range.")).toHaveLength(3)) + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx new file mode 100644 index 000000000..35c8e8417 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx @@ -0,0 +1,72 @@ +import { useState } from "react" +import { useQueries } from "@tanstack/react-query" +import { useK8sClient } from "@cozystack/k8s-client" +import { Section, Spinner } from "@cozystack/ui" +import { + METRIC_PANELS, + RANGES, + rangeWindow, + toChartData, + type RangeKey, +} from "../../lib/observability/metrics.ts" +import { isScopeWide, metricsRoot, queryRange, type ObservabilityScope } from "../../lib/observability/proxy.ts" +import { ProxyNotice } from "./ProxyNotice.tsx" +import { TimeSeriesChart } from "./TimeSeriesChart.tsx" +import { RangePicker } from "./RangePicker.tsx" + +const REFRESH_MS = 60_000 + +export function MetricsView({ scope }: { scope: ObservabilityScope }) { + const client = useK8sClient() + const root = metricsRoot(scope) + const [range, setRange] = useState("1h") + + const results = useQueries({ + queries: METRIC_PANELS.map((panel) => ({ + queryKey: ["observability", "metrics", root, panel.id, range], + queryFn: async ({ signal }: { signal: AbortSignal }) => { + const window = rangeWindow(range, Math.floor(Date.now() / 1000)) + const series = await Promise.all( + panel.queries(window).map(async (q) => ({ + label: q.label, + series: await queryRange(client, root, { query: q.expr, ...window }, signal), + })), + ) + return toChartData(series) + }, + retry: false, + refetchInterval: REFRESH_MS, + })), + }) + + const wide = results.find((r) => isScopeWide(r.error))?.error + + return ( +
+
+

Resource consumption

+ +
+ {wide ? ( + + ) : ( + METRIC_PANELS.map((panel, i) => { + const { data, error, isLoading } = results[i] + return ( +
+ {error ? ( + + ) : isLoading || !data ? ( + + ) : data.series.length === 0 ? ( +

No data for this range.

+ ) : ( + + )} +
+ ) + }) + )} +
+ ) +} diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx new file mode 100644 index 000000000..fd001bcae --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx @@ -0,0 +1,43 @@ +import { describe, it, expect } from "vitest" +import { render, screen } from "@testing-library/react" +import { ProxyError } from "../../lib/observability/proxy.ts" +import { ProxyNotice } from "./ProxyNotice.tsx" + +describe("ProxyNotice", () => { + it("renders a 404 as not available on this cluster, not as an error", () => { + render() + + expect(screen.getByRole("status")).toHaveTextContent("Metrics are not available on this cluster.") + expect(screen.queryByRole("alert")).toBeNull() + }) + + it("renders a 502 as a quiet notice about the store", () => { + render() + + expect(screen.getByRole("status")).toHaveTextContent("The logs store is unreachable right now.") + }) + + it.each([ + ["forbidden", 403, "You do not have permission to read metrics here."], + ["limit", 422, "too many points. Choose a shorter range."], + ["rate-limited", 429, "Too many requests. Try again in a moment."], + ["timeout", 504, "The store did not answer in time. Choose a shorter range or a narrower filter."], + ["unavailable", 503, "The service is temporarily unavailable. Try again in a moment."], + ] as const)("explains %s", (kind, status, text) => { + render() + + expect(screen.getByRole("alert")).toHaveTextContent(text) + }) + + it("shows the server message for a rejected query", () => { + render() + + expect(screen.getByRole("alert")).toHaveTextContent("cannot parse query") + }) + + it("falls back to the message of an unexpected error", () => { + render() + + expect(screen.getByRole("alert")).toHaveTextContent("boom") + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx new file mode 100644 index 000000000..a60b831b0 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx @@ -0,0 +1,49 @@ +import { ProxyError } from "../../lib/observability/proxy.ts" + +interface ProxyNoticeProps { + what: "Metrics" | "Logs" + error: unknown +} + +function describe(what: ProxyNoticeProps["what"], error: unknown): { quiet: boolean; text: string } { + const lower = what.toLowerCase() + if (!(error instanceof ProxyError)) { + return { quiet: false, text: error instanceof Error ? error.message : String(error) } + } + switch (error.kind) { + case "not-available": + return { quiet: true, text: `${what} are not available on this cluster.` } + case "store-unreachable": + return { quiet: true, text: `The ${lower} store is unreachable right now.` } + case "unauthorized": + return { quiet: false, text: "Your session has expired. Sign in again." } + case "forbidden": + return { quiet: false, text: `You do not have permission to read ${lower} here.` } + case "limit": + return { quiet: false, text: `${error.message}. Choose a shorter range.` } + case "rate-limited": + return { quiet: false, text: "Too many requests. Try again in a moment." } + case "timeout": + return { quiet: false, text: "The store did not answer in time. Choose a shorter range or a narrower filter." } + case "unavailable": + return { quiet: false, text: "The service is temporarily unavailable. Try again in a moment." } + default: + return { quiet: false, text: error.message } + } +} + +export function ProxyNotice({ what, error }: ProxyNoticeProps) { + const { quiet, text } = describe(what, error) + if (quiet) { + return ( +

+ {text} +

+ ) + } + return ( +

+ {text} +

+ ) +} diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.tsx new file mode 100644 index 000000000..22030c706 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.tsx @@ -0,0 +1,30 @@ +import { cn } from "@cozystack/ui" + +export function RangePicker({ + options, + value, + onChange, +}: { + options: readonly K[] + value: K + onChange: (value: K) => void +}) { + return ( +
+ {options.map((key) => ( + + ))} +
+ ) +} diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx new file mode 100644 index 000000000..57cf6fb99 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx @@ -0,0 +1,46 @@ +import { describe, it, expect, vi, beforeEach } from "vitest" +import { render } from "@testing-library/react" + +const created: { opts: { series: { label?: string }[] }; data: unknown[]; destroy: ReturnType }[] = [] + +vi.mock("uplot", () => ({ + default: class { + destroy = vi.fn() + constructor(opts: { series: { label?: string }[] }, data: unknown[]) { + created.push({ opts, data, destroy: this.destroy }) + } + setSize() {} + }, +})) + +const { TimeSeriesChart } = await import("./TimeSeriesChart.tsx") + +beforeEach(() => { + created.length = 0 +}) + +describe("TimeSeriesChart", () => { + const data = { + timestamps: [1, 2], + series: [ + { label: "db-0", values: [1, 2] }, + { label: "db-1", values: [null, 3] }, + ], + } + + it("hands uPlot the timestamps first and one labelled series per input", () => { + render() + + expect(created).toHaveLength(1) + expect(created[0].data).toEqual([[1, 2], [1, 2], [null, 3]]) + expect(created[0].opts.series.slice(1).map((s) => s.label)).toEqual(["db-0", "db-1"]) + }) + + it("destroys the chart on unmount", () => { + const { unmount } = render() + + unmount() + + expect(created[0].destroy).toHaveBeenCalled() + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx new file mode 100644 index 000000000..6b596e17f --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx @@ -0,0 +1,51 @@ +import { useEffect, useRef } from "react" +import uPlot from "uplot" +import "uplot/dist/uPlot.min.css" +import type { ChartData } from "../../lib/observability/metrics.ts" + +const COLORS = ["#2563eb", "#16a34a", "#d97706", "#9333ea", "#dc2626", "#0891b2", "#4b5563", "#db2777"] +const HEIGHT = 200 + +interface TimeSeriesChartProps { + data: ChartData + format: (value: number) => string +} + +export function TimeSeriesChart({ data, format }: TimeSeriesChartProps) { + const ref = useRef(null) + + useEffect(() => { + const el = ref.current + if (!el) return + const fmt = (v: number | null) => (v === null ? "–" : format(v)) + const chart = new uPlot( + { + width: el.clientWidth || 600, + height: HEIGHT, + scales: { x: { time: true } }, + legend: { live: true }, + series: [ + {}, + ...data.series.map((s, i) => ({ + label: s.label, + stroke: COLORS[i % COLORS.length], + width: 1.5, + value: (_: uPlot, v: number | null) => fmt(v), + })), + ], + axes: [{}, { size: 70, values: (_: uPlot, splits: number[]) => splits.map(format) }], + }, + [data.timestamps, ...data.series.map((s) => s.values)], + el, + ) + if (typeof ResizeObserver === "undefined") return () => chart.destroy() + const observer = new ResizeObserver(() => chart.setSize({ width: el.clientWidth, height: HEIGHT })) + observer.observe(el) + return () => { + observer.disconnect() + chart.destroy() + } + }, [data, format]) + + return
+} diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.test.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.test.ts new file mode 100644 index 000000000..0803940bf --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.test.ts @@ -0,0 +1,107 @@ +import { describe, it, expect } from "vitest" +import { + METRIC_PANELS, + formatBytes, + formatCores, + rangeWindow, + toChartData, +} from "./metrics.ts" + +describe("rangeWindow", () => { + it("ends now and spans the range in about 240 points", () => { + const w = rangeWindow("6h", 1_000_000) + + expect(w.end).toBe(1_000_000) + expect(w.start).toBe(1_000_000 - 6 * 3600) + expect(w.step).toBe(90) + }) + + it("never steps finer than 15s", () => { + expect(rangeWindow("1h", 0).step).toBe(15) + }) +}) + +describe("METRIC_PANELS", () => { + const window = { start: 0, end: 3600, step: 15 } + + it("rates CPU over at least five minutes so sparse scrapes still draw", () => { + const [query] = METRIC_PANELS.find((p) => p.id === "cpu")!.queries(window) + + expect(query.expr).toBe( + 'sum by (pod) (rate(container_cpu_usage_seconds_total{container!="",container!="POD"}[300s]))', + ) + }) + + it("widens the rate window with the step on long ranges", () => { + const [query] = METRIC_PANELS.find((p) => p.id === "cpu")!.queries({ start: 0, end: 1, step: 2520 }) + + expect(query.expr).toContain("[5040s]") + }) + + it("reads memory as working set per pod", () => { + const [query] = METRIC_PANELS.find((p) => p.id === "memory")!.queries(window) + + expect(query.expr).toBe('sum by (pod) (container_memory_working_set_bytes{container!="",container!="POD"})') + }) + + it("draws network receive and transmit as two named series", () => { + const queries = METRIC_PANELS.find((p) => p.id === "network")!.queries(window) + + expect(queries.map((q) => q.label)).toEqual(["receive", "transmit"]) + expect(queries[0].expr).toBe("sum(rate(container_network_receive_bytes_total[300s]))") + expect(queries[1].expr).toBe("sum(rate(container_network_transmit_bytes_total[300s]))") + }) + + it("names no application kind in any query", () => { + for (const panel of METRIC_PANELS) { + for (const q of panel.queries(window)) expect(q.expr).not.toMatch(/apps_cozystack_io|namespace=/) + } + }) +}) + +describe("toChartData", () => { + it("aligns series on the union of timestamps and leaves gaps as null", () => { + const data = toChartData([ + { label: "a", series: [{ metric: {}, values: [[10, "1"], [20, "2"]] }] }, + { label: "b", series: [{ metric: {}, values: [[20, "5"], [30, "6"]] }] }, + ]) + + expect(data.timestamps).toEqual([10, 20, 30]) + expect(data.series).toEqual([ + { label: "a", values: [1, 2, null] }, + { label: "b", values: [null, 5, 6] }, + ]) + }) + + it("labels each series by its pod, falling back to the query label", () => { + const data = toChartData([ + { + label: "cpu", + series: [ + { metric: { pod: "db-1" }, values: [[1, "0.5"]] }, + { metric: {}, values: [[1, "0.1"]] }, + ], + }, + ]) + + expect(data.series.map((s) => s.label)).toEqual(["db-1", "cpu"]) + }) + + it("drops NaN samples instead of drawing them as zero", () => { + const data = toChartData([{ label: "x", series: [{ metric: {}, values: [[1, "NaN"]] }] }]) + + expect(data.series[0].values).toEqual([null]) + }) +}) + +describe("formatters", () => { + it("formats CPU in millicores below one core", () => { + expect(formatCores(0.25)).toBe("250m") + expect(formatCores(2)).toBe("2") + }) + + it("formats bytes with binary units", () => { + expect(formatBytes(512)).toBe("512 B") + expect(formatBytes(3 * 1024 * 1024)).toBe("3.0 MiB") + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts new file mode 100644 index 000000000..b23a51e27 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts @@ -0,0 +1,121 @@ +import type { PromSeries } from "./proxy.ts" + +export const RANGES = { + "1h": 3600, + "6h": 6 * 3600, + "24h": 24 * 3600, + "7d": 7 * 24 * 3600, +} as const + +export type RangeKey = keyof typeof RANGES + +export interface RangeWindow { + start: number + end: number + step: number +} + +const POINTS = 240 +const MIN_STEP = 15 + +export function rangeWindow(range: RangeKey, nowSeconds: number): RangeWindow { + const span = RANGES[range] + return { start: nowSeconds - span, end: nowSeconds, step: Math.max(MIN_STEP, Math.round(span / POINTS)) } +} + +// A rate needs several scrapes inside its window, and on long ranges the window +// must also cover the step or samples between points are skipped. +function rateWindow(step: number): string { + return `${Math.max(300, step * 2)}s` +} + +const CONTAINERS = 'container!="",container!="POD"' + +export interface MetricQuery { + label: string + expr: string +} + +export interface MetricPanel { + id: "cpu" | "memory" | "network" + title: string + format: (value: number) => string + queries: (window: RangeWindow) => MetricQuery[] +} + +export function formatCores(value: number): string { + if (value === 0) return "0" + if (Math.abs(value) < 1) return `${Math.round(value * 1000)}m` + return `${Number(value.toFixed(2))}` +} + +const BYTE_UNITS = ["B", "KiB", "MiB", "GiB", "TiB"] + +export function formatBytes(value: number): string { + let v = value + let unit = 0 + while (Math.abs(v) >= 1024 && unit < BYTE_UNITS.length - 1) { + v /= 1024 + unit++ + } + return unit === 0 ? `${Math.round(v)} B` : `${v.toFixed(1)} ${BYTE_UNITS[unit]}` +} + +// The proxy injects the namespace or application scope, so these stay kind-agnostic. +export const METRIC_PANELS: MetricPanel[] = [ + { + id: "cpu", + title: "CPU (cores)", + format: formatCores, + queries: ({ step }) => [ + { + label: "cpu", + expr: `sum by (pod) (rate(container_cpu_usage_seconds_total{${CONTAINERS}}[${rateWindow(step)}]))`, + }, + ], + }, + { + id: "memory", + title: "Memory (working set)", + format: formatBytes, + queries: () => [ + { label: "memory", expr: `sum by (pod) (container_memory_working_set_bytes{${CONTAINERS}})` }, + ], + }, + { + id: "network", + title: "Network", + format: (v) => `${formatBytes(v)}/s`, + queries: ({ step }) => [ + { label: "receive", expr: `sum(rate(container_network_receive_bytes_total[${rateWindow(step)}]))` }, + { label: "transmit", expr: `sum(rate(container_network_transmit_bytes_total[${rateWindow(step)}]))` }, + ], + }, +] + +export interface ChartSeries { + label: string + values: (number | null)[] +} + +export interface ChartData { + timestamps: number[] + series: ChartSeries[] +} + +export function toChartData(results: { label: string; series: PromSeries[] }[]): ChartData { + const flat = results.flatMap(({ label, series }) => + series.map((s) => ({ label: s.metric.pod ?? label, points: new Map(s.values.map(([t, v]) => [t, Number(v)])) })), + ) + const timestamps = [...new Set(flat.flatMap((s) => [...s.points.keys()]))].sort((a, b) => a - b) + return { + timestamps, + series: flat.map(({ label, points }) => ({ + label, + values: timestamps.map((t) => { + const v = points.get(t) + return v === undefined || Number.isNaN(v) ? null : v + }), + })), + } +} diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts index f06835ff7..f8cf44656 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts @@ -4,6 +4,7 @@ import { ProxyError, errorKindForStatus, fieldValues, + isScopeWide, logsRoot, metricsRoot, queryLogs, @@ -177,3 +178,12 @@ describe("tailLogs", () => { expect(err).toMatchObject({ kind: "forbidden" }) }) }) + +describe("isScopeWide", () => { + it("is true for answers that do not depend on the query", () => { + expect(isScopeWide(new ProxyError("not-available", 404, ""))).toBe(true) + expect(isScopeWide(new ProxyError("forbidden", 403, ""))).toBe(true) + expect(isScopeWide(new ProxyError("limit", 422, ""))).toBe(false) + expect(isScopeWide(new Error("x"))).toBe(false) + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts index a257825c9..40b451b0b 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts @@ -60,6 +60,14 @@ export class ProxyError extends Error { } } +// These answers hold for every query in the scope, so one notice replaces the whole view. +export function isScopeWide(error: unknown): boolean { + return ( + error instanceof ProxyError && + (error.kind === "not-available" || error.kind === "forbidden" || error.kind === "unauthorized") + ) +} + // Prometheus answers {"error": "..."}, VictoriaLogs and nginx answer plain text. function errorMessage(text: string, status: number): string { try { diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.test.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.test.tsx index a015a98bf..40b923f62 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.test.tsx @@ -59,6 +59,7 @@ vi.mock("./EventsTab.tsx", () => ({ EventsTab: () => null })) vi.mock("./VncTab.tsx", () => ({ VncTab: () => null })) vi.mock("./SerialTab.tsx", () => ({ SerialTab: () => null })) vi.mock("./VMPowerControls.tsx", () => ({ VMPowerControls: () => null })) +vi.mock("./MetricsTab.tsx", () => ({ MetricsTab: () => null })) const { ApplicationDetailPage } = await import("./ApplicationDetailPage.tsx") @@ -111,11 +112,38 @@ describe("ApplicationDetailPage tabs for a virtual machine", () => { "VNC", "Workloads", "Services", + "Metrics", "Events", ]) }) }) +describe("ApplicationDetailPage observability tabs", () => { + function showKind(kind: string, plural: string) { + h.appKind = kind + h.appPlural = plural + h.get = { + data: { kind, metadata: { name: "demo", namespace: "tenant-test" } }, + isLoading: false, + error: undefined, + } + h.configMaps = [] + h.configMapsError = undefined + h.tabLabels = [] + renderPage() + } + + it("offers Metrics on any application kind, ahead of Events", () => { + showKind("Postgres", "postgreses") + expect(h.tabLabels.slice(-3)).toEqual(["Secrets", "Metrics", "Events"]) + }) + + it("leaves Metrics off a VM disk, which runs no containers", () => { + showKind("VMDisk", "vmdisks") + expect(h.tabLabels).not.toContain("Metrics") + }) +}) + describe("ApplicationDetailPage configuration", () => { it("offers ConfigMaps declared by the application's resource map", () => { h.appKind = "FoundationDB" diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx index 55fa684b8..4520244d5 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx @@ -37,6 +37,8 @@ import { VncTab } from "./VncTab.tsx" // fetched only when a VM's console is opened. Every dashboard page reaches this // file, and most of them never touch a VM. const SerialTab = lazy(() => import("./SerialTab.tsx").then((m) => ({ default: m.SerialTab }))) +// Same for the charting library behind the Metrics tab. +const MetricsTab = lazy(() => import("./MetricsTab.tsx").then((m) => ({ default: m.MetricsTab }))) import { VMPowerControls } from "./VMPowerControls.tsx" import { useResourceBasePath } from "../../lib/portal.ts" import { useResourcePresence } from "./use-resource-presence.ts" @@ -143,6 +145,7 @@ export function ApplicationDetailPage() { { to: `${base}/vnc`, label: "VNC", end: false }, { to: `${base}/workloads`, label: "Workloads", end: false }, { to: `${base}/services`, label: "Services", end: false }, + { to: `${base}/metrics`, label: "Metrics", end: false }, { to: `${base}/events`, label: "Events", end: false }, ) } else { @@ -163,6 +166,7 @@ export function ApplicationDetailPage() { } tabs.push( { to: `${base}/secrets`, label: "Secrets", end: false }, + { to: `${base}/metrics`, label: "Metrics", end: false }, { to: `${base}/events`, label: "Events", end: false }, ) } @@ -262,6 +266,14 @@ export function ApplicationDetailPage() { } /> } /> + }> + + + } + />
diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/detail/MetricsTab.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/detail/MetricsTab.tsx new file mode 100644 index 000000000..a928e633d --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/routes/detail/MetricsTab.tsx @@ -0,0 +1,15 @@ +import type { ApplicationDefinition, ApplicationInstance } from "@cozystack/types" +import { useTenantContext } from "../../lib/tenant-context.tsx" +import { MetricsView } from "../../components/observability/MetricsView.tsx" + +interface MetricsTabProps { + ad: ApplicationDefinition + instance: ApplicationInstance +} + +export function MetricsTab({ ad, instance }: MetricsTabProps) { + const { tenantNamespace } = useTenantContext() + const plural = ad.spec?.application.plural + if (!tenantNamespace || !plural) return null + return +} diff --git a/packages/system/dashboard/images/console/pnpm-lock.yaml b/packages/system/dashboard/images/console/pnpm-lock.yaml index 9d90b0c5c..4661c444a 100644 --- a/packages/system/dashboard/images/console/pnpm-lock.yaml +++ b/packages/system/dashboard/images/console/pnpm-lock.yaml @@ -92,6 +92,9 @@ importers: simple-icons: specifier: ^16.22.0 version: 16.22.0 + uplot: + specifier: ^1.6.32 + version: 1.6.32 devDependencies: '@tailwindcss/vite': specifier: ^4.2.2 @@ -1773,6 +1776,9 @@ packages: peerDependencies: browserslist: '>= 4.21.0' + uplot@1.6.32: + resolution: {integrity: sha512-KIMVnG68zvu5XXUbC4LQEPnhwOxBuLyW1AHtpm6IKTXImkbLgkMy+jabjLgSLMasNuGGzQm/ep3tOkyTxpiQIw==} + uri-js@4.4.1: resolution: {integrity: sha512-7rKUyy33Q1yc98pQ1DAmLtwX109F7TIfWlW1Ydo8Wl1ii1SeHieeh0HHfPeL2fMXK6z0s8ecKs9frCuLJvndBg==} @@ -3405,6 +3411,8 @@ snapshots: escalade: 3.2.0 picocolors: 1.1.1 + uplot@1.6.32: {} + uri-js@4.4.1: dependencies: punycode: 2.3.1 From b5a0ab8de6db7e68a6f4396f01becd8cb98b1dff Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:34:01 +0700 Subject: [PATCH 056/518] feat(dashboard): add a log viewer for the observability proxy Tenants could not read their applications' logs without Grafana. The viewer reads VictoriaLogs' native responses through a proxy root: a time-ranged LogsQL query, pod and container pickers fed by the stream field values of the same scope, and a follow mode over the streamed tail. The picked pod and container wrap the filter part of the query rather than being appended to it, because AND binds tighter than OR in LogsQL and the user's pipes must still run last. A dropped tail resumes from the last line's _time; the proxy may replay lines around that point, so the client drops replayed lines it already holds by stream and time, which VictoriaLogs does not guarantee unique and is therefore applied only at the resume boundary. Follow keeps the newest 1000 lines so a chatty stream cannot grow the page without bound. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/LogsView.test.tsx | 111 +++++++++++++ .../src/components/observability/LogsView.tsx | 152 ++++++++++++++++++ .../src/lib/observability/logs.test.ts | 65 ++++++++ .../console/src/lib/observability/logs.ts | 73 +++++++++ .../console/src/lib/observability/proxy.ts | 3 +- .../lib/observability/use-log-tail.test.tsx | 65 ++++++++ .../src/lib/observability/use-log-tail.ts | 95 +++++++++++ 7 files changed, 563 insertions(+), 1 deletion(-) create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts create mode 100644 packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts create mode 100644 packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx new file mode 100644 index 000000000..693adf70a --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -0,0 +1,111 @@ +import { describe, it, expect, vi } from "vitest" +import { screen, waitFor } from "@testing-library/react" +import userEvent from "@testing-library/user-event" +import { K8sClient } from "@cozystack/k8s-client" +import { renderWithK8sProvider } from "../../test-utils/render.tsx" +import { LogsView } from "./LogsView.tsx" + +const app = { namespace: "tenant-a", plural: "postgreses", name: "db" } +const ROOT = "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/log/" + +const line = (time: string, msg: string, pod = "db-0") => + JSON.stringify({ _time: time, _msg: msg, _stream_id: "s", kubernetes_pod_name: pod, kubernetes_container_name: "postgres" }) + +function clientWith(handler: (endpoint: string, params: URLSearchParams) => Response | Promise) { + const client = new K8sClient() + const raw = vi.spyOn(client, "raw").mockImplementation(async (path) => { + const url = new URL(path, "http://x") + return handler(url.pathname, url.searchParams) + }) + return { client, raw } +} + +const values = (...v: string[]) => Response.json({ values: v.map((value) => ({ value, hits: 1 })) }) + +function defaultHandler(endpoint: string, params?: URLSearchParams): Response | Promise { + if (endpoint.endsWith("stream_field_values")) { + return params?.get("field") === "kubernetes_pod_name" ? values("db-0", "db-1") : values("postgres") + } + if (endpoint.endsWith("query")) return new Response(`${line("2026-11-04T10:00:01Z", "ready to accept connections")}\n`) + return new Promise(() => {}) +} + +describe("LogsView", () => { + it("shows recent lines from the application root", async () => { + const { client, raw } = clientWith(defaultHandler) + + renderWithK8sProvider(, { client }) + + expect(await screen.findByText("ready to accept connections")).toBeInTheDocument() + const call = raw.mock.calls.find(([p]) => p.includes("select/logsql/query?"))! + expect(call[0].startsWith(`${ROOT}select/logsql/query?`)).toBe(true) + const params = new URL(call[0], "http://x").searchParams + expect(params.get("query")).toBe("*") + expect(params.get("limit")).toBe("500") + }) + + it("narrows to the picked pod with a stream filter", async () => { + const { client, raw } = clientWith(defaultHandler) + renderWithK8sProvider(, { client }) + await screen.findByRole("option", { name: "db-1" }) + + await userEvent.selectOptions(screen.getByLabelText("Pod"), "db-1") + + await waitFor(() => + expect( + raw.mock.calls.some(([p]) => new URL(p, "http://x").searchParams.get("query") === '{kubernetes_pod_name="db-1"} (*)'), + ).toBe(true), + ) + }) + + it("applies the LogsQL filter on submit", async () => { + const { client, raw } = clientWith(defaultHandler) + renderWithK8sProvider(, { client }) + + await userEvent.type(screen.getByLabelText("LogsQL filter"), "error{Enter}") + + await waitFor(() => + expect(raw.mock.calls.some(([p]) => p.includes("select/logsql/query?") && p.includes("query=error"))).toBe(true), + ) + }) + + it("follows the live tail when Follow is pressed", async () => { + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("tail") ? new Response(`${line("2026-11-04T10:00:05Z", "checkpoint complete")}\n`) : defaultHandler(endpoint, params), + ) + renderWithK8sProvider(, { client }) + + await userEvent.click(screen.getByRole("button", { name: "Follow" })) + + expect(await screen.findByText("checkpoint complete")).toBeInTheDocument() + expect(screen.getByRole("button", { name: "Follow" })).toHaveAttribute("aria-pressed", "true") + }) + + it("renders a cluster without the proxy as not available", async () => { + const { client } = clientWith(() => new Response("404 page not found", { status: 404 })) + + renderWithK8sProvider(, { client }) + + expect(await screen.findByText("Logs are not available on this cluster.")).toBeInTheDocument() + expect(screen.queryByLabelText("LogsQL filter")).toBeNull() + }) + + it("keeps the controls and explains a rejected query", async () => { + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("query") ? new Response("cannot parse query", { status: 400 }) : defaultHandler(endpoint, params), + ) + + renderWithK8sProvider(, { client }) + + expect(await screen.findByText("cannot parse query")).toBeInTheDocument() + expect(screen.getByLabelText("LogsQL filter")).toBeInTheDocument() + }) + + it("names the source pod and container of each line", async () => { + const { client } = clientWith(defaultHandler) + + renderWithK8sProvider(, { client }) + + expect(await screen.findByText("db-0/postgres")).toBeInTheDocument() + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx new file mode 100644 index 000000000..03235cda3 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -0,0 +1,152 @@ +import { useEffect, useRef, useState } from "react" +import { useQuery } from "@tanstack/react-query" +import { useK8sClient } from "@cozystack/k8s-client" +import { Button, Spinner } from "@cozystack/ui" +import { + CONTAINER_FIELD, + LOG_RANGES, + POD_FIELD, + buildLogsQuery, + type LogRangeKey, +} from "../../lib/observability/logs.ts" +import { + fieldValues, + isScopeWide, + logsRoot, + queryLogs, + type LogEntry, + type ObservabilityScope, +} from "../../lib/observability/proxy.ts" +import { useLogTail } from "../../lib/observability/use-log-tail.ts" +import { ProxyNotice } from "./ProxyNotice.tsx" +import { RangePicker } from "./RangePicker.tsx" + +const HISTORY_LIMIT = 500 +const FOLLOW_CAP = 1000 + +function windowFor(range: LogRangeKey) { + const end = Date.now() + return { start: new Date(end - LOG_RANGES[range] * 1000).toISOString(), end: new Date(end).toISOString() } +} + +function Picker({ label, value, options, onChange }: { label: string; value: string; options: string[]; onChange: (v: string) => void }) { + return ( + + ) +} + +function LogLines({ entries, follow }: { entries: LogEntry[]; follow: boolean }) { + const ref = useRef(null) + useEffect(() => { + if (follow && ref.current) ref.current.scrollTop = ref.current.scrollHeight + }, [entries, follow]) + if (entries.length === 0) return

No log lines for this range and filter.

+ return ( +
    + {entries.map((e, i) => ( +
  1. + {new Date(e._time).toLocaleTimeString()}{" "} + {`${e[POD_FIELD] ?? ""}/${e[CONTAINER_FIELD] ?? ""}`}{" "} + {e._msg} +
  2. + ))} +
+ ) +} + +export function LogsView({ scope }: { scope: ObservabilityScope }) { + const client = useK8sClient() + const root = logsRoot(scope) + const [range, setRange] = useState("1h") + const [draft, setDraft] = useState("") + const [filter, setFilter] = useState("") + const [pod, setPod] = useState("") + const [container, setContainer] = useState("") + const [follow, setFollow] = useState(false) + const query = buildLogsQuery({ filter, pod, container }) + + const pickerQuery = (field: string, narrowTo?: string) => ({ + queryKey: ["observability", "logs-fields", root, field, range, narrowTo ?? ""], + queryFn: ({ signal }: { signal: AbortSignal }) => + fieldValues(client, root, { + field, + stream: true, + query: buildLogsQuery({ filter: "", pod: narrowTo }), + ...windowFor(range), + }, signal), + retry: false, + }) + const pods = useQuery(pickerQuery(POD_FIELD)) + const containers = useQuery(pickerQuery(CONTAINER_FIELD, pod || undefined)) + + const history = useQuery({ + queryKey: ["observability", "logs", root, query, range], + queryFn: ({ signal }) => queryLogs(client, root, { query, limit: HISTORY_LIMIT, ...windowFor(range) }, signal), + enabled: !follow, + retry: false, + }) + const tail = useLogTail(root, query, follow, { cap: FOLLOW_CAP }) + + const wide = [pods.error, history.error, tail.error].find(isScopeWide) + if (wide) { + return ( +
+ +
+ ) + } + + const error = follow ? tail.error : history.error + const entries = follow ? tail.entries : (history.data ?? []) + + return ( +
+
+
{ + e.preventDefault() + setFilter(draft) + }} + > + setDraft(e.target.value)} + placeholder='LogsQL filter, e.g. error or "connection refused"' + className="w-full rounded-md border border-slate-200 px-2.5 py-1.5 font-mono text-xs" + /> +
+ { setPod(v); setContainer("") }} /> + + {!follow && } + +
+ {follow && ( +

+ {tail.state === "live" ? "Live" : tail.state === "reconnecting" ? "Reconnecting…" : "Connecting…"} + {` · newest ${FOLLOW_CAP} lines kept`} +

+ )} + {error ? : null} + {!follow && history.isLoading ? : } +
+ ) +} diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts new file mode 100644 index 000000000..25d26a23a --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts @@ -0,0 +1,65 @@ +import { describe, it, expect } from "vitest" +import { buildLogsQuery, compareTime, mergeTail } from "./logs.ts" + +describe("buildLogsQuery", () => { + it("passes the filter through when nothing is picked", () => { + expect(buildLogsQuery({ filter: "error or warn" })).toBe("error or warn") + expect(buildLogsQuery({ filter: " " })).toBe("*") + }) + + it("ANDs the picked pod and container as a stream filter around the whole filter", () => { + expect(buildLogsQuery({ filter: "error or warn", pod: "db-0", container: "postgres" })).toBe( + '{kubernetes_pod_name="db-0",kubernetes_container_name="postgres"} (error or warn)', + ) + }) + + it("keeps the user's pipes after the scoped filter", () => { + expect(buildLogsQuery({ filter: 'a or "x|y" | fields _msg', pod: "db-0" })).toBe( + '{kubernetes_pod_name="db-0"} (a or "x|y") | fields _msg', + ) + }) + + it("escapes quotes in picked values", () => { + expect(buildLogsQuery({ filter: "", pod: 'a"b' })).toBe('{kubernetes_pod_name="a\\"b"} (*)') + }) +}) + +describe("compareTime", () => { + it("orders RFC3339 times with differing fraction lengths", () => { + expect(compareTime("2026-11-04T10:00:01Z", "2026-11-04T10:00:01.1Z")).toBeLessThan(0) + expect(compareTime("2026-11-04T10:00:01.5Z", "2026-11-04T10:00:01.123Z")).toBeGreaterThan(0) + expect(compareTime("2026-11-04T10:00:01.10Z", "2026-11-04T10:00:01.1Z")).toBe(0) + }) +}) + +const e = (time: string, msg: string, stream = "s1") => ({ _time: time, _msg: msg, _stream_id: stream }) + +describe("mergeTail", () => { + it("appends and keeps only the newest lines over the cap", () => { + const out = mergeTail([e("t1", "a"), e("t2", "b")], [e("t3", "c")], 2) + + expect(out.map((x) => x._msg)).toEqual(["b", "c"]) + }) + + it("drops replayed lines at or before the resume point", () => { + const buffer = [e("2026-01-01T00:00:01Z", "a"), e("2026-01-01T00:00:02Z", "b")] + + const out = mergeTail(buffer, [e("2026-01-01T00:00:02Z", "b"), e("2026-01-01T00:00:03Z", "c")], 10, "2026-01-01T00:00:02Z") + + expect(out.map((x) => x._msg)).toEqual(["a", "b", "c"]) + }) + + it("keeps a line at the resume point from another stream", () => { + const buffer = [e("2026-01-01T00:00:02Z", "b", "s1")] + + const out = mergeTail(buffer, [e("2026-01-01T00:00:02Z", "other", "s2")], 10, "2026-01-01T00:00:02Z") + + expect(out).toHaveLength(2) + }) + + it("does not deduplicate outside a resume", () => { + const out = mergeTail([e("t1", "a")], [e("t1", "a")], 10) + + expect(out).toHaveLength(2) + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts new file mode 100644 index 000000000..a6255e541 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts @@ -0,0 +1,73 @@ +import type { LogEntry } from "./proxy.ts" + +export const LOG_RANGES = { + "15m": 15 * 60, + "1h": 3600, + "6h": 6 * 3600, + "24h": 24 * 3600, +} as const + +export type LogRangeKey = keyof typeof LOG_RANGES + +export const POD_FIELD = "kubernetes_pod_name" +export const CONTAINER_FIELD = "kubernetes_container_name" + +function quote(value: string): string { + return `"${value.replace(/\\/g, "\\\\").replace(/"/g, '\\"')}"` +} + +function splitAtFirstPipe(query: string): [string, string] { + let quoteChar = "" + for (let i = 0; i < query.length; i++) { + const c = query[i] + if (quoteChar) { + if (c === "\\") i++ + else if (c === quoteChar) quoteChar = "" + } else if (c === '"' || c === "'" || c === "`") { + quoteChar = c + } else if (c === "|") { + return [query.slice(0, i), query.slice(i + 1)] + } + } + return [query, ""] +} + +// The picker narrows the filter part only: AND binds tighter than OR in +// LogsQL, and the user's pipes must still run after the narrowed filter. +export function buildLogsQuery({ filter, pod, container }: { filter: string; pod?: string; container?: string }): string { + const trimmed = filter.trim() || "*" + const matchers = [ + pod ? `${POD_FIELD}=${quote(pod)}` : "", + container ? `${CONTAINER_FIELD}=${quote(container)}` : "", + ].filter(Boolean) + if (matchers.length === 0) return trimmed + const [head, pipes] = splitAtFirstPipe(trimmed) + const scoped = `{${matchers.join(",")}} (${head.trim() || "*"})` + return pipes.trim() ? `${scoped} | ${pipes.trim()}` : scoped +} + +// VictoriaLogs trims trailing zeros from the fraction, so the strings do not sort lexically. +function padFraction(time: string): string { + const m = /^([^.Z]+)(?:\.(\d+))?Z$/.exec(time) + return m ? `${m[1]}.${(m[2] ?? "").padEnd(9, "0")}Z` : time +} + +export function compareTime(a: string, b: string): number { + const x = padFraction(a) + const y = padFraction(b) + return x < y ? -1 : x > y ? 1 : 0 +} + +const entryKey = (e: LogEntry) => `${e._stream_id ?? ""}\u0000${e._time}` + +// (_stream_id, _time) is not unique in VictoriaLogs, so dedup applies only to +// lines a resumed tail replays at or before the resume point. +export function mergeTail(buffer: LogEntry[], incoming: LogEntry[], cap: number, resumedFrom?: string): LogEntry[] { + let fresh = incoming + if (resumedFrom) { + const seen = new Set(buffer.map(entryKey)) + fresh = incoming.filter((e) => compareTime(e._time, resumedFrom) > 0 || !seen.has(entryKey(e))) + } + const merged = buffer.concat(fresh) + return merged.length > cap ? merged.slice(merged.length - cap) : merged +} diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts index 40b451b0b..345d49815 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts @@ -1,4 +1,5 @@ import type { K8sClient } from "@cozystack/k8s-client" +import { compareTime } from "./logs.ts" export type ObservabilityScope = | { namespace: string } @@ -160,7 +161,7 @@ export async function queryLogs( ): Promise { const res = await proxyGet(client, `${root}select/logsql/query`, { ...params }, signal) const entries = (await res.text()).split("\n").map(parseLine).filter((e) => e !== null) - return entries.sort((a, b) => (a._time < b._time ? -1 : a._time > b._time ? 1 : 0)) + return entries.sort((a, b) => compareTime(a._time, b._time)) } export interface FieldValuesParams { diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx new file mode 100644 index 000000000..5c420314a --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx @@ -0,0 +1,65 @@ +import { describe, it, expect, vi } from "vitest" +import { renderHook, waitFor } from "@testing-library/react" +import type { ReactNode } from "react" +import { K8sClient, K8sProvider } from "@cozystack/k8s-client" +import { useLogTail } from "./use-log-tail.ts" + +const line = (time: string, msg: string) => JSON.stringify({ _time: time, _msg: msg, _stream_id: "s" }) + +function ndjson(...lines: string[]): Response { + return new Response(lines.map((l) => `${l}\n`).join("")) +} + +function setup(...responses: Response[]) { + const client = new K8sClient() + const raw = vi.spyOn(client, "raw") + for (const r of responses) raw.mockResolvedValueOnce(r) + raw.mockImplementation(() => new Promise(() => {})) + const wrapper = ({ children }: { children: ReactNode }) => {children} + return { raw, wrapper } +} + +const T1 = "2026-11-04T10:00:01Z" +const T2 = "2026-11-04T10:00:02Z" +const T3 = "2026-11-04T10:00:03Z" + +describe("useLogTail", () => { + it("streams lines and resumes after a disconnect from the last _time without duplicates", async () => { + const { raw, wrapper } = setup(ndjson(line(T1, "a"), line(T2, "b")), ndjson(line(T2, "b"), line(T3, "c"))) + + const { result } = renderHook(() => useLogTail("/root/", "*", true, { cap: 100, retryDelayMs: 1 }), { wrapper }) + + await waitFor(() => expect(result.current.entries.map((e) => e._msg)).toEqual(["a", "b", "c"])) + expect(new URL(raw.mock.calls[0][0], "http://x").searchParams.has("since")).toBe(false) + expect(new URL(raw.mock.calls[1][0], "http://x").searchParams.get("since")).toBe(T2) + }) + + it("stops and reports an error the proxy will repeat on every retry", async () => { + const { raw, wrapper } = setup(new Response("", { status: 403 })) + + const { result } = renderHook(() => useLogTail("/root/", "*", true, { cap: 100, retryDelayMs: 1 }), { wrapper }) + + await waitFor(() => expect(result.current.state).toBe("stopped")) + expect(result.current.error).toMatchObject({ kind: "forbidden" }) + expect(raw).toHaveBeenCalledTimes(1) + }) + + it("does not connect while disabled", () => { + const { raw, wrapper } = setup() + + renderHook(() => useLogTail("/root/", "*", false), { wrapper }) + + expect(raw).not.toHaveBeenCalled() + }) + + it("aborts the stream on unmount", async () => { + const { raw, wrapper } = setup() + const { unmount } = renderHook(() => useLogTail("/root/", "*", true), { wrapper }) + await waitFor(() => expect(raw).toHaveBeenCalled()) + + unmount() + + const init = raw.mock.calls[0][1] as RequestInit + expect(init.signal?.aborted).toBe(true) + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts new file mode 100644 index 000000000..cb9222f48 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts @@ -0,0 +1,95 @@ +import { useEffect, useState } from "react" +import { useK8sClient } from "@cozystack/k8s-client" +import { mergeTail } from "./logs.ts" +import { ProxyError, tailLogs, type LogEntry } from "./proxy.ts" + +export type TailState = "connecting" | "live" | "reconnecting" | "stopped" + +interface TailSnapshot { + key: string + entries: LogEntry[] + state: TailState + error: unknown +} + +const FLUSH_MS = 100 +const MAX_RETRY_MS = 30_000 +const FATAL: ReadonlySet = new Set(["bad-request", "unauthorized", "forbidden", "not-available", "limit"]) + +const sleep = (ms: number, signal: AbortSignal) => + new Promise((resolve) => { + const t = setTimeout(resolve, ms) + signal.addEventListener("abort", () => { + clearTimeout(t) + resolve() + }) + }) + +export function useLogTail( + root: string, + query: string, + enabled: boolean, + { cap = 1000, retryDelayMs = 2000 }: { cap?: number; retryDelayMs?: number } = {}, +): Omit { + const client = useK8sClient() + const key = `${root}\n${query}\n${enabled}` + const [snap, setSnap] = useState({ key, entries: [], state: "connecting", error: null }) + + useEffect(() => { + if (!enabled) return + const controller = new AbortController() + const { signal } = controller + let pending: LogEntry[] = [] + let timer: ReturnType | undefined + let resumedFrom: string | undefined + let lastTime: string | undefined + + const update = (patch: Partial, batch: LogEntry[] = [], from = resumedFrom) => + setSnap((prev) => { + const base = prev.key === key ? prev : { key, entries: [], state: "connecting" as const, error: null } + return { ...base, ...patch, entries: batch.length ? mergeTail(base.entries, batch, cap, from) : base.entries } + }) + const flush = () => { + clearTimeout(timer) + timer = undefined + if (pending.length === 0 || signal.aborted) return + const batch = pending + pending = [] + update({ state: "live", error: null }, batch) + } + + const run = async () => { + for (let attempt = 0; !signal.aborted; attempt++) { + resumedFrom = lastTime + try { + await tailLogs(client, root, { query, since: lastTime }, (entry) => { + attempt = 0 + lastTime = entry._time + pending.push(entry) + timer ??= setTimeout(flush, FLUSH_MS) + }, signal) + flush() + } catch (err) { + flush() + if (signal.aborted) return + if (err instanceof ProxyError && FATAL.has(err.kind)) { + update({ state: "stopped", error: err }) + return + } + update({ error: err }) + } + if (signal.aborted) return + update({ state: "reconnecting" }) + await sleep(Math.min(retryDelayMs * 2 ** attempt, MAX_RETRY_MS), signal) + } + } + void run() + return () => { + controller.abort() + clearTimeout(timer) + } + }, [client, root, query, enabled, key, cap, retryDelayMs]) + + const current = snap.key === key ? snap : { key, entries: [], state: "connecting" as const, error: null } + return { entries: current.entries, state: current.state, error: current.error } +} From f9ed2afad749646bb5c1de0a56b4e8d17e773bc1 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:34:01 +0700 Subject: [PATCH 057/518] feat(dashboard): show logs on application pages and per tenant A Logs tab on application detail pages reads the application root of the observability proxy, and a Logs page under a new Observability sidebar group reads the tenant namespace root, so lines from before the application identity labels existed stay reachable there. Assisted-by: LLM Signed-off-by: ZverGuy --- .../apps/console/src/routes/ConsolePage.tsx | 2 ++ .../apps/console/src/routes/LogsPage.test.tsx | 20 +++++++++++++++++++ .../apps/console/src/routes/LogsPage.tsx | 15 ++++++++++++++ .../detail/ApplicationDetailPage.test.tsx | 9 ++++++--- .../routes/detail/ApplicationDetailPage.tsx | 4 ++++ .../console/src/routes/detail/LogsTab.tsx | 15 ++++++++++++++ .../src/routes/sidebar-sections.test.tsx | 10 ++++++++++ .../console/src/routes/sidebar-sections.tsx | 8 +++++++- 8 files changed, 79 insertions(+), 4 deletions(-) create mode 100644 packages/system/dashboard/images/console/apps/console/src/routes/LogsPage.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/routes/LogsPage.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/routes/detail/LogsTab.tsx diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx index 37caae92f..5e204e84f 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx @@ -15,6 +15,7 @@ import { MigrationSourceEditPage } from "./MigrationSourceEditPage.tsx" import { MigrationTaskCreatePage } from "./MigrationTaskCreatePage.tsx" import { MigrationTaskDetailPage } from "./MigrationTaskDetailPage.tsx" import { ApplicationOrderPage } from "./ApplicationOrderPage.tsx" +import { LogsPage } from "./LogsPage.tsx" export function ConsolePage() { return ( @@ -96,6 +97,7 @@ export function ConsolePage() { path="migration/vmimporttasks/:name" element={} /> + } /> } /> } /> } /> diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/LogsPage.test.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/LogsPage.test.tsx new file mode 100644 index 000000000..1f4263cf1 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/routes/LogsPage.test.tsx @@ -0,0 +1,20 @@ +import { describe, it, expect, vi } from "vitest" +import { render, screen } from "@testing-library/react" + +vi.mock("../lib/tenant-context.tsx", () => ({ + useTenantContext: () => ({ tenantNamespace: "tenant-a" }), +})) +vi.mock("../components/observability/LogsView.tsx", () => ({ + LogsView: ({ scope }: { scope: object }) =>
{JSON.stringify(scope)}
, +})) + +const { LogsPage } = await import("./LogsPage.tsx") + +describe("LogsPage", () => { + it("reads logs from the selected tenant's namespace root", () => { + render() + + expect(screen.getByRole("heading", { name: "Logs" })).toBeInTheDocument() + expect(JSON.parse(screen.getByTestId("scope").textContent ?? "")).toEqual({ namespace: "tenant-a" }) + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/LogsPage.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/LogsPage.tsx new file mode 100644 index 000000000..7612dfbb2 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/routes/LogsPage.tsx @@ -0,0 +1,15 @@ +import { useTenantContext } from "../lib/tenant-context.tsx" +import { LogsView } from "../components/observability/LogsView.tsx" + +export function LogsPage() { + const { tenantNamespace } = useTenantContext() + return ( +
+
+

Logs

+

Every application in {tenantNamespace ?? "this tenant"}

+
+ {tenantNamespace && } +
+ ) +} diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.test.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.test.tsx index 40b923f62..a0c3a6b84 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.test.tsx @@ -60,6 +60,7 @@ vi.mock("./VncTab.tsx", () => ({ VncTab: () => null })) vi.mock("./SerialTab.tsx", () => ({ SerialTab: () => null })) vi.mock("./VMPowerControls.tsx", () => ({ VMPowerControls: () => null })) vi.mock("./MetricsTab.tsx", () => ({ MetricsTab: () => null })) +vi.mock("./LogsTab.tsx", () => ({ LogsTab: () => null })) const { ApplicationDetailPage } = await import("./ApplicationDetailPage.tsx") @@ -113,6 +114,7 @@ describe("ApplicationDetailPage tabs for a virtual machine", () => { "Workloads", "Services", "Metrics", + "Logs", "Events", ]) }) @@ -133,14 +135,15 @@ describe("ApplicationDetailPage observability tabs", () => { renderPage() } - it("offers Metrics on any application kind, ahead of Events", () => { + it("offers Metrics and Logs on any application kind, ahead of Events", () => { showKind("Postgres", "postgreses") - expect(h.tabLabels.slice(-3)).toEqual(["Secrets", "Metrics", "Events"]) + expect(h.tabLabels.slice(-4)).toEqual(["Secrets", "Metrics", "Logs", "Events"]) }) - it("leaves Metrics off a VM disk, which runs no containers", () => { + it("leaves Metrics and Logs off a VM disk, which runs no containers", () => { showKind("VMDisk", "vmdisks") expect(h.tabLabels).not.toContain("Metrics") + expect(h.tabLabels).not.toContain("Logs") }) }) diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx index 4520244d5..ca499aac1 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx @@ -44,6 +44,7 @@ import { useResourceBasePath } from "../../lib/portal.ts" import { useResourcePresence } from "./use-resource-presence.ts" import { useApplicationConfigMaps } from "./use-app-configmaps.ts" import { ConfigMapsTab } from "./ConfigMapsTab.tsx" +import { LogsTab } from "./LogsTab.tsx" export function ApplicationDetailPage() { const { plural, name } = useParams<{ plural: string; name: string }>() @@ -146,6 +147,7 @@ export function ApplicationDetailPage() { { to: `${base}/workloads`, label: "Workloads", end: false }, { to: `${base}/services`, label: "Services", end: false }, { to: `${base}/metrics`, label: "Metrics", end: false }, + { to: `${base}/logs`, label: "Logs", end: false }, { to: `${base}/events`, label: "Events", end: false }, ) } else { @@ -167,6 +169,7 @@ export function ApplicationDetailPage() { tabs.push( { to: `${base}/secrets`, label: "Secrets", end: false }, { to: `${base}/metrics`, label: "Metrics", end: false }, + { to: `${base}/logs`, label: "Logs", end: false }, { to: `${base}/events`, label: "Events", end: false }, ) } @@ -274,6 +277,7 @@ export function ApplicationDetailPage() { } /> + } /> diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/detail/LogsTab.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/detail/LogsTab.tsx new file mode 100644 index 000000000..4ac5f878d --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/routes/detail/LogsTab.tsx @@ -0,0 +1,15 @@ +import type { ApplicationDefinition, ApplicationInstance } from "@cozystack/types" +import { useTenantContext } from "../../lib/tenant-context.tsx" +import { LogsView } from "../../components/observability/LogsView.tsx" + +interface LogsTabProps { + ad: ApplicationDefinition + instance: ApplicationInstance +} + +export function LogsTab({ ad, instance }: LogsTabProps) { + const { tenantNamespace } = useTenantContext() + const plural = ad.spec?.application.plural + if (!tenantNamespace || !plural) return null + return +} diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx index efb69dc7b..da40081bc 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx @@ -110,6 +110,16 @@ describe("useConsoleSidebarSections — admin areas moved out", () => { expect(findItem(result.current, "Imports")?.to).toBe("/console/migration/vmimporttasks") expect(result.current.some((s) => s.title === "Migration")).toBe(true) }) + + it("offers the tenant's namespace logs in an Observability group", async () => { + const client = makeClient({ nodes: true, backupclasses: true }) + const { result } = renderHook(() => useConsoleSidebarSections(), { + wrapper: makeWrapper(client), + }) + await waitFor(() => expect(result.current.length).toBeGreaterThan(0)) + expect(findItem(result.current, "Logs")?.to).toBe("/console/logs") + expect(result.current.find((s) => s.title === "Observability")?.items).toHaveLength(1) + }) }) function appDef(name: string, kind: string, category: string, module = false) { diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx index d5d100813..4cec3a94c 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx @@ -11,6 +11,7 @@ import { Layers, Network, Package, + ScrollText, Plug, Server, ToyBrick, @@ -118,7 +119,12 @@ export function useConsoleSidebarSections(): SidebarSection[] { ], } - return [...categorySections, backupsSection, migrationSection] + const observabilitySection: SidebarSection = { + title: "Observability", + items: [{ label: "Logs", to: "/console/logs", icon: ScrollText }], + } + + return [...categorySections, backupsSection, migrationSection, observabilitySection] }, [grouped]) } From c5b57bffd173ea5e4bb371edb75a13a60f549d96 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:39:27 +0700 Subject: [PATCH 058/518] fix(dashboard): open log history at the newest line A historical query lists lines oldest first, so the viewer opened at the top of a 500-line window while the line a user looks for is usually the latest. The chart's value axis also clipped wide labels such as "476.8 MiB". Assisted-by: LLM Signed-off-by: ZverGuy --- .../console/src/components/observability/LogsView.tsx | 8 ++++---- .../src/components/observability/TimeSeriesChart.tsx | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index 03235cda3..707b037b9 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -50,11 +50,11 @@ function Picker({ label, value, options, onChange }: { label: string; value: str ) } -function LogLines({ entries, follow }: { entries: LogEntry[]; follow: boolean }) { +function LogLines({ entries }: { entries: LogEntry[] }) { const ref = useRef(null) useEffect(() => { - if (follow && ref.current) ref.current.scrollTop = ref.current.scrollHeight - }, [entries, follow]) + if (ref.current) ref.current.scrollTop = ref.current.scrollHeight + }, [entries]) if (entries.length === 0) return

No log lines for this range and filter.

return (
    @@ -146,7 +146,7 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) {

    )} {error ? : null} - {!follow && history.isLoading ? : } + {!follow && history.isLoading ? : } ) } diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx index 6b596e17f..683df2dff 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx @@ -33,7 +33,7 @@ export function TimeSeriesChart({ data, format }: TimeSeriesChartProps) { value: (_: uPlot, v: number | null) => fmt(v), })), ], - axes: [{}, { size: 70, values: (_: uPlot, splits: number[]) => splits.map(format) }], + axes: [{}, { size: 84, values: (_: uPlot, splits: number[]) => splits.map(format) }], }, [data.timestamps, ...data.series.map((s) => s.values)], el, From 97dcf4c4f02d78dc2dcd4ec862cdaa77631de4f3 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:45:42 +0700 Subject: [PATCH 059/518] fix(dashboard): treat an apiserver answer on a proxy path as no proxy Until the console nginx routes the proxy roots, those paths reach kube-apiserver, which RBAC-denies the namespace root with 403 for a tenant, so the Logs page told every tenant they lacked permission. The proxy itself answers in plain text, so a Kubernetes Status body is a reliable sign that the path did not reach it. Assisted-by: LLM Signed-off-by: ZverGuy --- .../components/observability/LogsView.test.tsx | 10 ++++++++++ .../console/src/lib/observability/proxy.test.ts | 17 +++++++++++++++++ .../apps/console/src/lib/observability/proxy.ts | 14 +++++++++++++- 3 files changed, 40 insertions(+), 1 deletion(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index 693adf70a..fae3f6a4b 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -90,6 +90,16 @@ describe("LogsView", () => { expect(screen.queryByLabelText("LogsQL filter")).toBeNull() }) + it("renders the tenant logs page as not available when kube-apiserver answers instead of the proxy", async () => { + const { client } = clientWith(() => + Response.json({ kind: "Status", apiVersion: "v1", status: "Failure", reason: "Forbidden", code: 403 }, { status: 403 }), + ) + + renderWithK8sProvider(, { client }) + + expect(await screen.findByText("Logs are not available on this cluster.")).toBeInTheDocument() + }) + it("keeps the controls and explains a rejected query", async () => { const { client } = clientWith((endpoint, params) => endpoint.endsWith("query") ? new Response("cannot parse query", { status: 400 }) : defaultHandler(endpoint, params), diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts index f8cf44656..69a8c7c8c 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts @@ -79,6 +79,23 @@ describe("queryRange", () => { expect(err).toMatchObject({ kind: "limit", status: 422, message: "range too long" }) }) + it("treats a Kubernetes Status 403 as a cluster without the proxy route", async () => { + const status = { kind: "Status", apiVersion: "v1", status: "Failure", reason: "Forbidden", code: 403, message: "logs.cozystack.io \"select\" is forbidden" } + const { client } = clientReturning(Response.json(status, { status: 403 })) + + const err = await queryRange(client, metricsRoot(ns), { query: "up", start: 0, end: 1, step: 1 }).catch((e) => e) + + expect(err).toMatchObject({ kind: "not-available", status: 403 }) + }) + + it("keeps the proxy's own plain-text 403 as forbidden", async () => { + const { client } = clientReturning(new Response("forbidden\n", { status: 403 })) + + const err = await queryRange(client, metricsRoot(ns), { query: "up", start: 0, end: 1, step: 1 }).catch((e) => e) + + expect(err).toMatchObject({ kind: "forbidden" }) + }) + it("reports a fetch failure as a network error", async () => { const client = new K8sClient() vi.spyOn(client, "raw").mockRejectedValue(new TypeError("offline")) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts index 345d49815..c9fce3d4f 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts @@ -69,6 +69,17 @@ export function isScopeWide(error: unknown): boolean { ) } +// The proxy answers in plain text; a Kubernetes Status body means nginx sent +// the path to kube-apiserver, which RBAC-denies or 404s a route it does not serve. +function isKubernetesStatus(text: string): boolean { + try { + const parsed: unknown = JSON.parse(text) + return !!parsed && typeof parsed === "object" && "kind" in parsed && parsed.kind === "Status" + } catch { + return false + } +} + // Prometheus answers {"error": "..."}, VictoriaLogs and nginx answer plain text. function errorMessage(text: string, status: number): string { try { @@ -100,7 +111,8 @@ async function proxyGet( } if (!res.ok) { const text = await res.text().catch(() => "") - throw new ProxyError(errorKindForStatus(res.status), res.status, errorMessage(text, res.status)) + const kind = isKubernetesStatus(text) ? "not-available" : errorKindForStatus(res.status) + throw new ProxyError(kind, res.status, errorMessage(text, res.status)) } return res } From 4b44971d125cce2c2fd06457c68f23cf1645ac87 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:47:13 +0700 Subject: [PATCH 060/518] fix(dashboard): start each log follow run with its own buffer Turning Follow off and on again reused the previous run's buffer, so new lines were appended after a silent gap and, on a quick toggle, duplicated. A follow run is now identified by a session value, so the buffer starts empty each time. A resume now uses the newest _time seen rather than the last line received, so an out-of-order line cannot move since backwards, and the backoff sleep no longer leaves an abort listener per retry. Assisted-by: LLM Signed-off-by: ZverGuy --- .../lib/observability/use-log-tail.test.tsx | 31 ++++++++++++++++--- .../src/lib/observability/use-log-tail.ts | 23 ++++++++------ 2 files changed, 41 insertions(+), 13 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx index 5c420314a..46df47970 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx @@ -27,7 +27,7 @@ describe("useLogTail", () => { it("streams lines and resumes after a disconnect from the last _time without duplicates", async () => { const { raw, wrapper } = setup(ndjson(line(T1, "a"), line(T2, "b")), ndjson(line(T2, "b"), line(T3, "c"))) - const { result } = renderHook(() => useLogTail("/root/", "*", true, { cap: 100, retryDelayMs: 1 }), { wrapper }) + const { result } = renderHook(() => useLogTail("/root/", "*", 1, { cap: 100, retryDelayMs: 1 }), { wrapper }) await waitFor(() => expect(result.current.entries.map((e) => e._msg)).toEqual(["a", "b", "c"])) expect(new URL(raw.mock.calls[0][0], "http://x").searchParams.has("since")).toBe(false) @@ -37,24 +37,47 @@ describe("useLogTail", () => { it("stops and reports an error the proxy will repeat on every retry", async () => { const { raw, wrapper } = setup(new Response("", { status: 403 })) - const { result } = renderHook(() => useLogTail("/root/", "*", true, { cap: 100, retryDelayMs: 1 }), { wrapper }) + const { result } = renderHook(() => useLogTail("/root/", "*", 1, { cap: 100, retryDelayMs: 1 }), { wrapper }) await waitFor(() => expect(result.current.state).toBe("stopped")) expect(result.current.error).toMatchObject({ kind: "forbidden" }) expect(raw).toHaveBeenCalledTimes(1) }) + it("starts a new follow session with an empty buffer", async () => { + const { wrapper } = setup(ndjson(line(T1, "first")), ndjson(line(T3, "second"))) + const { result, rerender } = renderHook( + ({ session }: { session: number | null }) => useLogTail("/root/", "*", session, { retryDelayMs: 60_000 }), + { wrapper, initialProps: { session: 1 as number | null } }, + ) + await waitFor(() => expect(result.current.entries.map((e) => e._msg)).toEqual(["first"])) + + rerender({ session: null }) + rerender({ session: 2 }) + + await waitFor(() => expect(result.current.entries.map((e) => e._msg)).toEqual(["second"])) + }) + + it("resumes from the newest line seen, not the last one received", async () => { + const { raw, wrapper } = setup(ndjson(line(T3, "c"), line(T1, "a")), ndjson()) + + renderHook(() => useLogTail("/root/", "*", 1, { retryDelayMs: 1 }), { wrapper }) + + await waitFor(() => expect(raw.mock.calls.length).toBeGreaterThan(1)) + expect(new URL(raw.mock.calls[1][0], "http://x").searchParams.get("since")).toBe(T3) + }) + it("does not connect while disabled", () => { const { raw, wrapper } = setup() - renderHook(() => useLogTail("/root/", "*", false), { wrapper }) + renderHook(() => useLogTail("/root/", "*", null), { wrapper }) expect(raw).not.toHaveBeenCalled() }) it("aborts the stream on unmount", async () => { const { raw, wrapper } = setup() - const { unmount } = renderHook(() => useLogTail("/root/", "*", true), { wrapper }) + const { unmount } = renderHook(() => useLogTail("/root/", "*", 1), { wrapper }) await waitFor(() => expect(raw).toHaveBeenCalled()) unmount() diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts index cb9222f48..ddc835821 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts @@ -1,6 +1,6 @@ import { useEffect, useState } from "react" import { useK8sClient } from "@cozystack/k8s-client" -import { mergeTail } from "./logs.ts" +import { compareTime, mergeTail } from "./logs.ts" import { ProxyError, tailLogs, type LogEntry } from "./proxy.ts" export type TailState = "connecting" | "live" | "reconnecting" | "stopped" @@ -18,25 +18,30 @@ const FATAL: ReadonlySet = new Set(["bad-request", "unauthorized", "forb const sleep = (ms: number, signal: AbortSignal) => new Promise((resolve) => { - const t = setTimeout(resolve, ms) - signal.addEventListener("abort", () => { + const onAbort = () => { clearTimeout(t) resolve() - }) + } + const t = setTimeout(() => { + signal.removeEventListener("abort", onAbort) + resolve() + }, ms) + signal.addEventListener("abort", onAbort, { once: true }) }) +// Each non-null session is one follow run with its own buffer; null stops following. export function useLogTail( root: string, query: string, - enabled: boolean, + session: number | null, { cap = 1000, retryDelayMs = 2000 }: { cap?: number; retryDelayMs?: number } = {}, ): Omit { const client = useK8sClient() - const key = `${root}\n${query}\n${enabled}` + const key = `${root}\n${query}\n${session}` const [snap, setSnap] = useState({ key, entries: [], state: "connecting", error: null }) useEffect(() => { - if (!enabled) return + if (session === null) return const controller = new AbortController() const { signal } = controller let pending: LogEntry[] = [] @@ -64,7 +69,7 @@ export function useLogTail( try { await tailLogs(client, root, { query, since: lastTime }, (entry) => { attempt = 0 - lastTime = entry._time + if (!lastTime || compareTime(entry._time, lastTime) > 0) lastTime = entry._time pending.push(entry) timer ??= setTimeout(flush, FLUSH_MS) }, signal) @@ -88,7 +93,7 @@ export function useLogTail( controller.abort() clearTimeout(timer) } - }, [client, root, query, enabled, key, cap, retryDelayMs]) + }, [client, root, query, session, key, cap, retryDelayMs]) const current = snap.key === key ? snap : { key, entries: [], state: "connecting" as const, error: null } return { entries: current.entries, state: current.state, error: current.error } From 290490769eacf8906256290d3078bba02e4e9d04 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:47:13 +0700 Subject: [PATCH 061/518] fix(dashboard): let the reader scroll up while logs stream The viewer jumped to the newest line on every 100 ms flush, so a line could not be read on a busy application. It now follows the bottom only while the reader is already there, and says when a refused tail has stopped instead of showing it as connecting. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/LogLines.test.tsx | 34 +++++++++++++++++++ .../src/components/observability/LogLines.tsx | 34 +++++++++++++++++++ .../observability/LogsView.test.tsx | 12 +++++++ .../src/components/observability/LogsView.tsx | 33 +++++------------- 4 files changed, 88 insertions(+), 25 deletions(-) create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx new file mode 100644 index 000000000..775c5fcc6 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx @@ -0,0 +1,34 @@ +import { describe, it, expect } from "vitest" +import { fireEvent, render, screen } from "@testing-library/react" +import { LogLines } from "./LogLines.tsx" + +const entry = (n: number) => ({ _time: `2026-11-04T10:00:0${n}Z`, _msg: `line ${n}`, _stream_id: "s" }) + +function sized(list: HTMLElement, scrollHeight: number) { + Object.defineProperty(list, "scrollHeight", { configurable: true, value: scrollHeight }) + Object.defineProperty(list, "clientHeight", { configurable: true, value: 100 }) +} + +describe("LogLines", () => { + it("keeps following the newest line while the reader is at the bottom", () => { + const { rerender } = render() + const list = screen.getByRole("list") + sized(list, 500) + + rerender() + + expect(list.scrollTop).toBe(500) + }) + + it("leaves the position alone once the reader scrolls up", () => { + const { rerender } = render() + const list = screen.getByRole("list") + sized(list, 500) + list.scrollTop = 120 + fireEvent.scroll(list) + + rerender() + + expect(list.scrollTop).toBe(120) + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx new file mode 100644 index 000000000..a6f4b5861 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx @@ -0,0 +1,34 @@ +import { useEffect, useRef } from "react" +import { CONTAINER_FIELD, POD_FIELD } from "../../lib/observability/logs.ts" +import type { LogEntry } from "../../lib/observability/proxy.ts" + +const BOTTOM_SLACK_PX = 24 + +export function LogLines({ entries }: { entries: LogEntry[] }) { + const ref = useRef(null) + const pinned = useRef(true) + + useEffect(() => { + if (ref.current && pinned.current) ref.current.scrollTop = ref.current.scrollHeight + }, [entries]) + + if (entries.length === 0) return

    No log lines for this range and filter.

    + return ( +
      { + const el = e.currentTarget + pinned.current = el.scrollHeight - el.scrollTop - el.clientHeight <= BOTTOM_SLACK_PX + }} + className="max-h-[60vh] overflow-auto rounded-md bg-slate-950 p-3 font-mono text-xs leading-5 text-slate-100" + > + {entries.map((e, i) => ( +
    1. + {new Date(e._time).toLocaleTimeString()}{" "} + {`${e[POD_FIELD] ?? ""}/${e[CONTAINER_FIELD] ?? ""}`}{" "} + {e._msg} +
    2. + ))} +
    + ) +} diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index fae3f6a4b..526efeb23 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -81,6 +81,18 @@ describe("LogsView", () => { expect(screen.getByRole("button", { name: "Follow" })).toHaveAttribute("aria-pressed", "true") }) + it("says the tail stopped when the proxy refuses the query", async () => { + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("tail") ? new Response("pipe not allowed in tail", { status: 400 }) : defaultHandler(endpoint, params), + ) + renderWithK8sProvider(, { client }) + + await userEvent.click(screen.getByRole("button", { name: "Follow" })) + + expect(await screen.findByText(/Stopped/)).toBeInTheDocument() + expect(screen.getByText("pipe not allowed in tail")).toBeInTheDocument() + }) + it("renders a cluster without the proxy as not available", async () => { const { client } = clientWith(() => new Response("404 page not found", { status: 404 })) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index 707b037b9..f21f2eb86 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -1,4 +1,4 @@ -import { useEffect, useRef, useState } from "react" +import { useState } from "react" import { useQuery } from "@tanstack/react-query" import { useK8sClient } from "@cozystack/k8s-client" import { Button, Spinner } from "@cozystack/ui" @@ -14,15 +14,16 @@ import { isScopeWide, logsRoot, queryLogs, - type LogEntry, type ObservabilityScope, } from "../../lib/observability/proxy.ts" import { useLogTail } from "../../lib/observability/use-log-tail.ts" import { ProxyNotice } from "./ProxyNotice.tsx" import { RangePicker } from "./RangePicker.tsx" +import { LogLines } from "./LogLines.tsx" const HISTORY_LIMIT = 500 const FOLLOW_CAP = 1000 +const TAIL_STATE_LABEL = { connecting: "Connecting…", live: "Live", reconnecting: "Reconnecting…", stopped: "Stopped" } function windowFor(range: LogRangeKey) { const end = Date.now() @@ -50,25 +51,6 @@ function Picker({ label, value, options, onChange }: { label: string; value: str ) } -function LogLines({ entries }: { entries: LogEntry[] }) { - const ref = useRef(null) - useEffect(() => { - if (ref.current) ref.current.scrollTop = ref.current.scrollHeight - }, [entries]) - if (entries.length === 0) return

    No log lines for this range and filter.

    - return ( -
      - {entries.map((e, i) => ( -
    1. - {new Date(e._time).toLocaleTimeString()}{" "} - {`${e[POD_FIELD] ?? ""}/${e[CONTAINER_FIELD] ?? ""}`}{" "} - {e._msg} -
    2. - ))} -
    - ) -} - export function LogsView({ scope }: { scope: ObservabilityScope }) { const client = useK8sClient() const root = logsRoot(scope) @@ -77,7 +59,8 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) { const [filter, setFilter] = useState("") const [pod, setPod] = useState("") const [container, setContainer] = useState("") - const [follow, setFollow] = useState(false) + const [followRun, setFollowRun] = useState(null) + const follow = followRun !== null const query = buildLogsQuery({ filter, pod, container }) const pickerQuery = (field: string, narrowTo?: string) => ({ @@ -100,7 +83,7 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) { enabled: !follow, retry: false, }) - const tail = useLogTail(root, query, follow, { cap: FOLLOW_CAP }) + const tail = useLogTail(root, query, followRun, { cap: FOLLOW_CAP }) const wide = [pods.error, history.error, tail.error].find(isScopeWide) if (wide) { @@ -135,13 +118,13 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) { { setPod(v); setContainer("") }} /> {!follow && } - {follow && (

    - {tail.state === "live" ? "Live" : tail.state === "reconnecting" ? "Reconnecting…" : "Connecting…"} + {TAIL_STATE_LABEL[tail.state]} {` · newest ${FOLLOW_CAP} lines kept`}

    )} From b05323f6bee913256c3e4ae6eedc764689ec508c Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:47:22 +0700 Subject: [PATCH 062/518] docs(dashboard): note the observability proxy client in the console guide The next session needs to know why one module bypasses the typed Kubernetes client and where the charting dependency is loaded. Assisted-by: LLM Signed-off-by: ZverGuy --- packages/system/dashboard/images/console/CLAUDE.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/packages/system/dashboard/images/console/CLAUDE.md b/packages/system/dashboard/images/console/CLAUDE.md index 2acf80504..96419d118 100644 --- a/packages/system/dashboard/images/console/CLAUDE.md +++ b/packages/system/dashboard/images/console/CLAUDE.md @@ -32,6 +32,8 @@ Implications: - **RJSF** (`@rjsf/core` + `validator-ajv8`) for application config forms. - **Vitest + jsdom + @testing-library/react** for tests. - **Monaco** for YAML view, **noVNC** for the VM console tab. +- **uPlot** for the Metrics tab charts, lazy-loaded with that tab. +- Metrics and logs come from the observability proxy, not kube-apiserver: `lib/observability/proxy.ts` reads its native Prometheus JSON and VictoriaLogs NDJSON through `K8sClient.raw`, under the paths nginx routes to it ahead of `/apis`. ## Dev loop From 84ed3aaefd34159549534dd8ff6f7e0e3111c3c9 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:50:33 +0700 Subject: [PATCH 063/518] fix(dashboard): follow the newest line again after the log list clears A reader who scrolled up stayed unpinned across a new follow run or a new filter, so the fresh list sat at its top while lines streamed in below. An empty list now means the reader is back at the bottom. Assisted-by: LLM Signed-off-by: ZverGuy --- .../components/observability/LogLines.test.tsx | 16 ++++++++++++++++ .../src/components/observability/LogLines.tsx | 3 ++- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx index 775c5fcc6..7110fe464 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx @@ -31,4 +31,20 @@ describe("LogLines", () => { expect(list.scrollTop).toBe(120) }) + + it("follows again once the list is cleared for a new run", () => { + const { rerender } = render() + const first = screen.getByRole("list") + sized(first, 500) + first.scrollTop = 120 + fireEvent.scroll(first) + + rerender() + rerender() + const second = screen.getByRole("list") + sized(second, 300) + rerender() + + expect(second.scrollTop).toBe(300) + }) }) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx index a6f4b5861..452faac72 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx @@ -9,7 +9,8 @@ export function LogLines({ entries }: { entries: LogEntry[] }) { const pinned = useRef(true) useEffect(() => { - if (ref.current && pinned.current) ref.current.scrollTop = ref.current.scrollHeight + if (entries.length === 0) pinned.current = true + else if (ref.current && pinned.current) ref.current.scrollTop = ref.current.scrollHeight }, [entries]) if (entries.length === 0) return

    No log lines for this range and filter.

    From 1b4ce7433e1acbb7d5b1d5ec6a4634fe2cd1b730 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:50:33 +0700 Subject: [PATCH 064/518] fix(dashboard): keep retrying a log tail refused for load The proxy answers a tail over its rate or concurrent-tail cap with a 4xx, and a reconnect can arrive before the previous tail's slot is freed. Treating that as final left Follow stopped for good, while the existing backoff already waits long enough for the cap to clear. Assisted-by: LLM Signed-off-by: ZverGuy --- .../console/src/lib/observability/use-log-tail.test.tsx | 8 ++++++++ .../apps/console/src/lib/observability/use-log-tail.ts | 3 ++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx index 46df47970..794f30216 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx @@ -67,6 +67,14 @@ describe("useLogTail", () => { expect(new URL(raw.mock.calls[1][0], "http://x").searchParams.get("since")).toBe(T3) }) + it("retries a tail refused for load instead of stopping", async () => { + const { wrapper } = setup(new Response("too many concurrent requests", { status: 422 }), ndjson(line(T1, "a"))) + + const { result } = renderHook(() => useLogTail("/root/", "*", 1, { retryDelayMs: 1 }), { wrapper }) + + await waitFor(() => expect(result.current.entries.map((e) => e._msg)).toEqual(["a"])) + }) + it("does not connect while disabled", () => { const { raw, wrapper } = setup() diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts index ddc835821..292f93e9e 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts @@ -14,7 +14,8 @@ interface TailSnapshot { const FLUSH_MS = 100 const MAX_RETRY_MS = 30_000 -const FATAL: ReadonlySet = new Set(["bad-request", "unauthorized", "forbidden", "not-available", "limit"]) +// A 422 or 429 on tail is load (rate or concurrent-tail caps), which backoff outlasts. +const FATAL: ReadonlySet = new Set(["bad-request", "unauthorized", "forbidden", "not-available"]) const sleep = (ms: number, signal: AbortSignal) => new Promise((resolve) => { From ebdd74ce4e010cc347d1ddae312b2c20dfeb39fa Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:53:23 +0700 Subject: [PATCH 065/518] fix(dashboard): name the proxy's real load status for a refused tail The proxy caps rate and concurrent tails with 429, so the retry reasoning and its test now say so. Leaving Follow also remounts the log list, so the history it falls back to opens at the newest line even when the reader had scrolled up while following. Assisted-by: LLM Signed-off-by: ZverGuy --- .../apps/console/src/components/observability/LogsView.tsx | 2 +- .../apps/console/src/lib/observability/use-log-tail.test.tsx | 2 +- .../console/apps/console/src/lib/observability/use-log-tail.ts | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index f21f2eb86..a27ca41c5 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -129,7 +129,7 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) {

    )} {error ? : null} - {!follow && history.isLoading ? : } + {!follow && history.isLoading ? : } ) } diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx index 794f30216..0c2306613 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.test.tsx @@ -68,7 +68,7 @@ describe("useLogTail", () => { }) it("retries a tail refused for load instead of stopping", async () => { - const { wrapper } = setup(new Response("too many concurrent requests", { status: 422 }), ndjson(line(T1, "a"))) + const { wrapper } = setup(new Response("too many concurrent requests", { status: 429 }), ndjson(line(T1, "a"))) const { result } = renderHook(() => useLogTail("/root/", "*", 1, { retryDelayMs: 1 }), { wrapper }) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts index 292f93e9e..53cbaf856 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts @@ -14,7 +14,8 @@ interface TailSnapshot { const FLUSH_MS = 100 const MAX_RETRY_MS = 30_000 -// A 422 or 429 on tail is load (rate or concurrent-tail caps), which backoff outlasts. +// Load refusals retry: the proxy caps rate and concurrent tails with 429 (422 in +// its first builds), and a reconnect can race the previous tail's freed slot. const FATAL: ReadonlySet = new Set(["bad-request", "unauthorized", "forbidden", "not-available"]) const sleep = (ms: number, signal: AbortSignal) => From 2b823ab79fa178f22b338ad654962bc9c334a30e Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:53:50 +0700 Subject: [PATCH 066/518] fix(dashboard): rate over at least 5m and four steps Long metric ranges are served by the longterm store, which keeps one sample per 5m, so a rate window narrower than that can hold no samples and the chart comes back empty. Four steps also keep every sample between two points inside some window. Assisted-by: LLM Signed-off-by: ZverGuy --- .../src/lib/observability/metrics.test.ts | 17 +++++++++++++---- .../console/src/lib/observability/metrics.ts | 6 +++--- 2 files changed, 16 insertions(+), 7 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.test.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.test.ts index 0803940bf..0598c0c7f 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.test.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.test.ts @@ -32,10 +32,19 @@ describe("METRIC_PANELS", () => { ) }) - it("widens the rate window with the step on long ranges", () => { - const [query] = METRIC_PANELS.find((p) => p.id === "cpu")!.queries({ start: 0, end: 1, step: 2520 }) - - expect(query.expr).toContain("[5040s]") + // Long ranges are served by the longterm store, deduplicated at 5m, where a + // narrower window can hold no samples and the rate comes back empty. + it.each([ + [15, "300s"], + [75, "300s"], + [90, "360s"], + [2520, "10080s"], + ])("rates over max(5m, 4 x step) for step %is", (step, window) => { + for (const panel of METRIC_PANELS) { + for (const q of panel.queries({ start: 0, end: 1, step })) { + if (q.expr.includes("rate(")) expect(q.expr).toContain(`[${window}]`) + } + } }) it("reads memory as working set per pod", () => { diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts index b23a51e27..d7182b8d6 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts @@ -23,10 +23,10 @@ export function rangeWindow(range: RangeKey, nowSeconds: number): RangeWindow { return { start: nowSeconds - span, end: nowSeconds, step: Math.max(MIN_STEP, Math.round(span / POINTS)) } } -// A rate needs several scrapes inside its window, and on long ranges the window -// must also cover the step or samples between points are skipped. +// The longterm store keeps one sample per 5m, so a narrower window can be empty; +// 4 x step also keeps samples between points from being skipped. function rateWindow(step: number): string { - return `${Math.max(300, step * 2)}s` + return `${Math.max(300, step * 4)}s` } const CONTAINERS = 'container!="",container!="POD"' From 41765174c09650dfaab59152abfe54daa4f2899d Mon Sep 17 00:00:00 2001 From: Marian Koreniuk Date: Tue, 3 Nov 2026 21:00:09 +0100 Subject: [PATCH 067/518] docs: add CageForge to ADOPTERS.md Assisted-by: LLM Signed-off-by: Marian Koreniuk --- ADOPTERS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/ADOPTERS.md b/ADOPTERS.md index d6cacdc55..936bfe296 100644 --- a/ADOPTERS.md +++ b/ADOPTERS.md @@ -33,3 +33,4 @@ This list is sorted in chronological order, based on the submission date. | [Hidora](https://hikube.cloud) | @matthieu-robin | 2025-09-17 | Hidora is a Swiss cloud provider delivering managed services and infrastructure solutions through datacenters located in Switzerland, ensuring data sovereignty and reliability. Its sovereign cloud platform, Hikube, is designed to run workloads with high availability across multiple datacenters, providing enterprises with a secure and scalable foundation for their applications based on Cozystack. | | [QOSI](https://qosi.kz) | @tabu-a | 2025-10-04 | QOSI is a non-profit organization driving open-source adoption and digital sovereignty across Kazakhstan and Central Asia. We use Cozystack as a platform for deploying sovereign, GPU-enabled clouds and educational environments under the National AI Program. Our goal is to accelerate the region’s transition toward open, self-hosted cloud-native technologies | | [Cloupard](https://cloupard.kz/) | @serjiott | 2025-12-18 | Cloupard is a public cloud provider offering IaaS and PaaS services via datacenters in Kazakhstan and Uzbekistan. Uses CozyStack on bare metal to extend its managed PaaS offerings. | +| [CageForge](https://cageforge.com/) | @themoriarti | 2026-11-03 | CageForge is a short-cycle R&D company providing independent expertise in cloud native, Cozystack and security for customers in finance, telecom, hosting and cloud providers. We run Cozystack both as a bare-metal installation and on Proxmox VE through the Proxmox integration, and contribute the Proxmox-related work back upstream. | From e79946e79c595d4973f4f0062c9322575e2bbc4a Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 10:08:59 +0500 Subject: [PATCH 068/518] fix(platform): remove the dead registries value and patch-containerd Secret Nothing reads the Secret: the kubeadm certs.d mount it fed is gone and the Talos worker config never took it. The value stayed in the Package spec as a knob that accepts mirrors and does nothing, which is worse than no knob for an operator in an air-gapped setup. Mirrors for tenant workers are a per-pool field of the KubernetesNodes chart. A Package that still sets registries keeps installing: the platform chart has no values schema, so the key is ignored. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .../templates/containerd-registry-secret.yaml | 35 ------------------- packages/core/platform/values.yaml | 31 ---------------- 2 files changed, 66 deletions(-) delete mode 100644 packages/core/platform/templates/containerd-registry-secret.yaml diff --git a/packages/core/platform/templates/containerd-registry-secret.yaml b/packages/core/platform/templates/containerd-registry-secret.yaml deleted file mode 100644 index 6e9fb6408..000000000 --- a/packages/core/platform/templates/containerd-registry-secret.yaml +++ /dev/null @@ -1,35 +0,0 @@ -{{- if .Values.registries.mirrors }} -apiVersion: v1 -kind: Secret -metadata: - name: patch-containerd - namespace: cozy-system -type: Opaque -stringData: -{{- range $registry, $mirror := .Values.registries.mirrors }} -{{- if $mirror.endpoints }} - {{ $registry }}.toml: | - server = "https://{{ $registry }}" -{{- range $endpoint := $mirror.endpoints }} - [host."{{ $endpoint }}"] - capabilities = ["pull", "resolve"] -{{- $endpointConfig := index $.Values.registries.config $endpoint }} -{{- if $endpointConfig }} -{{- if $endpointConfig.tls }} -{{- if $endpointConfig.tls.insecureSkipVerify }} - skip_verify = true -{{- end }} -{{- end }} -{{- if $endpointConfig.auth }} - [host."{{ $endpoint }}".auth] - username = "{{ $endpointConfig.auth.username }}" - password = "{{ $endpointConfig.auth.password }}" -{{- end }} -{{- else }} - skip_verify = true -{{- end }} -{{- end }} -{{- end }} -{{- end }} -{{- end }} - diff --git a/packages/core/platform/values.yaml b/packages/core/platform/values.yaml index f2ce252ed..23ed9a448 100644 --- a/packages/core/platform/values.yaml +++ b/packages/core/platform/values.yaml @@ -768,37 +768,6 @@ scheduling: # loginTheme: acme # loginThemeImage: registry.example.com/acme-keycloak-theme:v1 branding: {} -# Container registry mirrors configuration -# -# Example: -# registries: -# mirrors: -# docker.io: -# endpoints: -# - http://10.0.0.1:8082 -# ghcr.io: -# endpoints: -# - http://10.0.0.1:8083 -# gcr.io: -# endpoints: -# - http://10.0.0.1:8084 -# registry.k8s.io: -# endpoints: -# - http://10.0.0.1:8085 -# quay.io: -# endpoints: -# - http://10.0.0.1:8086 -# cr.fluentbit.io: -# endpoints: -# - http://10.0.0.1:8087 -# docker-registry3.mariadb.com: -# endpoints: -# - http://10.0.0.1:8088 -# config: -# "10.0.0.1:8082": -# tls: -# insecureSkipVerify: true -registries: {} # Resource allocation ratios resources: cpuAllocationRatio: 10 From a2d0e4cb5058092a66cb3173503850c64ba9fab0 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 06:07:07 +0000 Subject: [PATCH 069/518] fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook The hook fed the whole `kubectl get ciliumnodes -o json` to jq as one argument. Cilium copies every Node label onto its CiliumNode, 22 to 30 KiB per node on Talos or with NFD and GPU labels, so the list passes the 128 KiB Linux allows a single argument within six nodes. jq then fails to start with "Argument list too long", the hook Job fails before it moves any node, and the cilium HelmRelease loops on UpgradeFailed with the HelmReleases behind it stuck in DependencyNotReady. The hook renders on every upgrade of cilium in cluster-pool mode, so a cluster that already finished the migration hits the same failure on its next values change. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- hack/cilium-ipam-migration.bats | 27 +++++++++++++++++++ .../system/cilium/files/ipam-migration.sh | 10 ++++--- 2 files changed, 33 insertions(+), 4 deletions(-) diff --git a/hack/cilium-ipam-migration.bats b/hack/cilium-ipam-migration.bats index c54db50ab..53053747a 100644 --- a/hack/cilium-ipam-migration.bats +++ b/hack/cilium-ipam-migration.bats @@ -488,6 +488,33 @@ pod_cidrs_of() { rm -rf "$STATE" } +@test "a CiliumNode list larger than one argument may hold still moves every node and names the pod on an old Ingress IP" { + init_state + add_ip web-0.tenant-c ovn-default 10.244.1.135 + add_ip web-1.tenant-c ovn-default 10.244.4.135 + add_ip db-0.tenant-c ovn-default 10.244.9.9 + # Cilium copies every Node label onto the CiliumNode, which takes six real + # ones past the 128 KiB Linux allows a single argument. One annotation stands + # in for the labels and keeps the list over that limit even printed compact. + for i in 1 2 3 4 5 6; do + add_node "n$i" "10.244.$i.0/24" "10.244.$i.223" "10.244.$i.135" + jq '.metadata.annotations.padding = ("x" * 24576)' "$STATE/ciliumnodes/n$i.json" > "$STATE/node.tmp" + mv "$STATE/node.tmp" "$STATE/ciliumnodes/n$i.json" + done + [ "$(kubectl get ciliumnodes -o json | jq -c . | wc -c)" -gt 131072 ] + "$SCRIPT" > "$STATE/out" 2>&1 + # Each pod is named with the node whose Ingress IP it holds, and a pod on an + # address that is no node's Ingress IP is not named at all. + grep -q "web-0.tenant-c holds 10.244.1.135, the Ingress IP of n1 before the move" "$STATE/out" + grep -q "web-1.tenant-c holds 10.244.4.135, the Ingress IP of n4 before the move" "$STATE/out" + if grep -q "db-0.tenant-c" "$STATE/out"; then + cat "$STATE/out" + false + fi + [ "$(pod_cidrs_of n6)" = '["10.244.248.40/29"]' ] + rm -rf "$STATE" +} + @test "a cluster without kube-ovn skips the exclusion and still moves its nodes" { init_state rm "$STATE/subnet.json" diff --git a/packages/system/cilium/files/ipam-migration.sh b/packages/system/cilium/files/ipam-migration.sh index 002a1bf78..9345c19f6 100755 --- a/packages/system/cilium/files/ipam-migration.sh +++ b/packages/system/cilium/files/ipam-migration.sh @@ -240,10 +240,12 @@ exclude_from_kube_ovn() { fi # A pod that holds a node's current Ingress IP keeps the address after the # move, and that node keeps it in its ipcache as reserved:ingress, so the pod - # stays unreachable from there until it is recreated. - jq -r --argjson cn "$ciliumnodes" ' - ([$cn.items[] | select(.spec.ingress.ipv4) | {key: .spec.ingress.ipv4, value: .metadata.name}] | from_entries) as $ingress - | .[] | select($ingress[.ip]) | "\(.name) \(.ip) \($ingress[.ip])"' <<<"$ipobjects" \ + # stays unreachable from there until it is recreated. The CiliumNode list goes + # in on stdin, because the Node labels Cilium copies onto every CiliumNode take + # it past the 128 KiB a single argument may hold within a handful of nodes. + jq -rn 'input as $cn + | ([$cn.items[] | select(.spec.ingress.ipv4) | {key: .spec.ingress.ipv4, value: .metadata.name}] | from_entries) as $ingress + | input[] | select($ingress[.ip]) | "\(.name) \(.ip) \($ingress[.ip])"' <<<"$ciliumnodes"$'\n'"$ipobjects" \ | while read -r ipname ip node; do if ! in_any "$POOL" "$ip"; then log "$ipname holds $ip, the Ingress IP of $node before the move. Recreate it once the run is done, $node keeps that address as reserved:ingress" From 7b7444feaf5508eaf62767b8c999468558732d1b Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 06:16:36 +0000 Subject: [PATCH 070/518] fix(kubernetes): move the OIDC users out of the bootstrap Job's script The bootstrap Job wrote a ClusterRoleBinding per oidc.users entry into its script, and the script reaches the container as one `sh -c` argument. Linux caps a single argument at 128 KiB, which about 240 users cross at roughly 530 bytes each. The container then cannot start, the post-install and post-upgrade hook fails, and the tenant cluster's HelmRelease stays unready. The values schema puts no bound on oidc.users, so a shared cluster reaches that size with ordinary per-person entries. The bindings and the list of their names now reach the Job through a ConfigMap it mounts, so the script is the same size for any list. Names, labels and roles of the bindings are unchanged, so the bindings already in a tenant cluster are updated in place and none is orphaned. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- hack/kubernetes-oidc-bindings.bats | 187 +++++++++++++++++ .../templates/oidc-rbac-bindings.yaml | 42 ++++ .../kubernetes/templates/oidc-rbac-job.yaml | 57 +++--- packages/apps/kubernetes/tests/oidc_test.yaml | 193 +++++++++++++++++- 4 files changed, 439 insertions(+), 40 deletions(-) create mode 100644 hack/kubernetes-oidc-bindings.bats create mode 100644 packages/apps/kubernetes/templates/oidc-rbac-bindings.yaml diff --git a/hack/kubernetes-oidc-bindings.bats b/hack/kubernetes-oidc-bindings.bats new file mode 100644 index 000000000..f37273802 --- /dev/null +++ b/hack/kubernetes-oidc-bindings.bats @@ -0,0 +1,187 @@ +#!/usr/bin/env bats +# ----------------------------------------------------------------------------- +# packages/apps/kubernetes: the OIDC bootstrap Job must not carry the users[] +# list inside its container command. +# +# The Job's script reaches the container as one `sh -c` argument, and Linux caps +# one exec argument at 128 KiB (MAX_ARG_STRLEN). With a ClusterRoleBinding per +# user written into that script, about 240 users take it past the cap: the +# container cannot start, the post-install/post-upgrade hook fails, and the +# tenant cluster's HelmRelease stays unready. The bindings reach the Job through +# a mounted ConfigMap instead, so the script is the same size for any list. +# +# The last three tests execute the rendered script against a fake kubectl, since +# no assertion on the script's text says what the prune step does with the list: +# a condition forced either way passes every helm-unittest case. +# +# Requires: helm, yq (mikefarah v4+), jq, like hack/admin-kubeconfig-invariant.bats. +# ----------------------------------------------------------------------------- + +load test_helper + +USERS=300 + +# render_users renders the chart in System mode with +# generated users and leaves one JSON document per object in /rendered.json. +render_users() { + jq -n --argjson n "$2" ' + {oidc: {users: [range($n) | {email: "user\(.)@example.org", role: (if . % 7 == 0 then "admin" else "view" end)}]}}' \ + > "$1/users.yaml" + helm template bindings packages/apps/kubernetes \ + --namespace tenant-test \ + --values packages/apps/kubernetes/tests/values/oidc-system.yaml \ + --values "$1/users.yaml" \ + > "$1/rendered.yaml" + yq --output-format=json eval-all '.' "$1/rendered.yaml" > "$1/rendered.json" +} + +# render_job renders the chart in CustomConfig mode and unpacks +# what the bootstrap Job runs and mounts into : script.sh is its `sh -c` +# script, bindings/ holds the two files of the mounted ConfigMap. -j keeps an +# empty value an empty file, as the kubelet writes it. +render_job() { + jq -n --argjson users "$2" ' + {_cluster: {"oidc-enabled": "false"}, + oidc: {mode: "CustomConfig", + customConfig: {config: "apiVersion: apiserver.config.k8s.io/v1beta1\nkind: AuthenticationConfiguration\njwt: []\n"}, + users: $users}}' > "$1/values.yaml" + helm template bindings packages/apps/kubernetes \ + --namespace tenant-test \ + --values packages/apps/kubernetes/tests/values/oidc-system.yaml \ + --values "$1/values.yaml" \ + > "$1/rendered.yaml" + yq --output-format=json eval-all '.' "$1/rendered.yaml" > "$1/rendered.json" + jq -s -j '.[] | select(.kind == "Job" and .metadata.name == "bindings-oidc-bootstrap") + | .spec.template.spec.containers[0].command[2]' "$1/rendered.json" > "$1/script.sh" + mkdir -p "$1/bindings" + jq -s -j '.[] | select(.kind == "ConfigMap" and .metadata.name == "bindings-oidc-bindings") + | .data["crbs.yaml"]' "$1/rendered.json" > "$1/bindings/crbs.yaml" + jq -s -j '.[] | select(.kind == "ConfigMap" and .metadata.name == "bindings-oidc-bindings") + | .data["desired.txt"]' "$1/rendered.json" > "$1/bindings/desired.txt" +} + +# fake_kubectl puts a kubectl first on PATH that logs every call and +# answers the script's list of existing bindings with . +fake_kubectl() { + mkdir -p "$1/bin" + printf '%s' "$2" > "$1/existing" + cat > "$1/bin/kubectl" <<'KEOF' +#!/bin/sh +echo "$*" >> "$KLOG" +case "$*" in + *"get clusterrolebindings"*) cat "$EXISTING" ;; +esac +KEOF + chmod +x "$1/bin/kubectl" +} + +# run_job runs the unpacked script as the container does and leaves its +# status in $rc and its output in /out. Run by bash, which has the pipefail +# the script sets and which is what the unit lane runs under; the container's own +# shell is not exercised. +run_job() { + : > "$1/kubectl.log" + echo kubeconfig > "$1/kubeconfig" + rc=0 + env PATH="$1/bin:$PATH" KLOG="$1/kubectl.log" EXISTING="$1/existing" HOME="$1" \ + KUBECONFIG_TENANT="$1/kubeconfig" RELEASE=bindings RELEASE_NAMESPACE=tenant-test \ + OIDC_MODE=CustomConfig OIDC_BINDINGS_DIR="$1/bindings" \ + bash "$1/script.sh" > "$1/out" 2>&1 || rc=$? +} + +@test "the OIDC bootstrap Job script stays under the exec argument cap and carries no user" { + dir=$(mktemp -d) + render_users "$dir" "$USERS" + jq -s -r '.[] | select(.kind == "Job" and .metadata.name == "bindings-oidc-bootstrap") + | .spec.template.spec.containers[0].command[2]' "$dir/rendered.json" > "$dir/script" + size=$(wc -c < "$dir/script") + [ "$size" -gt 0 ] + [ "$size" -lt 131072 ] + if grep -q -E 'user[0-9]+@example\.org' "$dir/script"; then + echo "the Job script still carries users[] entries" >&2 + false + fi + rm -rf "$dir" +} + +@test "every user keeps one ClusterRoleBinding with its old name and role in the bindings ConfigMap" { + dir=$(mktemp -d) + render_users "$dir" "$USERS" + jq -s -r '.[] | select(.kind == "ConfigMap" and .metadata.name == "bindings-oidc-bindings") + | .data["crbs.yaml"]' "$dir/rendered.json" > "$dir/crbs.yaml" + jq -s -r '.[] | select(.kind == "ConfigMap" and .metadata.name == "bindings-oidc-bindings") + | .data["desired.txt"]' "$dir/rendered.json" > "$dir/desired.txt" + [ -s "$dir/crbs.yaml" ] + yq --output-format=json eval-all '.' "$dir/crbs.yaml" | jq -s '.' > "$dir/crbs.json" + + [ "$(jq 'length' "$dir/crbs.json")" -eq "$USERS" ] + [ "$(jq -r '[.[].subjects[0].name] | unique | length' "$dir/crbs.json")" -eq "$USERS" ] + [ "$(jq -r '[.[] | select(.roleRef.name == "cluster-admin")] | length' "$dir/crbs.json")" -eq $(( (USERS + 6) / 7 )) ] + + # The name is what the prune step and every binding already in a tenant cluster + # go by, so it must not move: cozystack-oidc- plus the first 16 hex digits of + # sha256("-"). + for i in 0 1 6 7; do + email="user$i@example.org" + name="cozystack-oidc-$(printf '%s' "bindings-$email" | sha256sum | cut -c1-16)" + [ "$(jq -r --arg e "$email" '.[] | select(.subjects[0].name == $e) | .metadata.name' "$dir/crbs.json")" = "$name" ] + grep -q -x -F "$name" "$dir/desired.txt" + done + + # The list the prune step compares against is exactly the applied names. + [ "$(jq -r '.[].metadata.name' "$dir/crbs.json" | sort)" = "$(grep . "$dir/desired.txt" | sort)" ] + rm -rf "$dir" +} + +@test "the Job applies the mounted bindings and deletes only the bindings no user has" { + dir=$(mktemp -d) + render_job "$dir" '[{"email":"alice@example.org","role":"admin"},{"email":"bob@example.org","role":"view"}]' + alice="cozystack-oidc-$(printf '%s' 'bindings-alice@example.org' | sha256sum | cut -c1-16)" + bob="cozystack-oidc-$(printf '%s' 'bindings-bob@example.org' | sha256sum | cut -c1-16)" + fake_kubectl "$dir" "$alice cozystack-oidc-000000000000dead $bob cozystack-oidc-000000000000beef" + run_job "$dir" + if [ "$rc" -ne 0 ]; then + cat "$dir/out" + false + fi + grep -q -F -- "apply --filename $dir/bindings/crbs.yaml" "$dir/kubectl.log" + grep -q -F -- "delete clusterrolebinding cozystack-oidc-000000000000dead --ignore-not-found" "$dir/kubectl.log" + grep -q -F -- "delete clusterrolebinding cozystack-oidc-000000000000beef --ignore-not-found" "$dir/kubectl.log" + if grep -F -e "delete clusterrolebinding $alice" -e "delete clusterrolebinding $bob" "$dir/kubectl.log"; then + echo "the Job deleted a binding a user still has" >&2 + false + fi + [ "$(grep -c 'delete clusterrolebinding' "$dir/kubectl.log")" -eq 2 ] + rm -rf "$dir" +} + +@test "a release with no users applies nothing and deletes every binding it left behind" { + dir=$(mktemp -d) + render_job "$dir" '[]' + fake_kubectl "$dir" "cozystack-oidc-000000000000dead cozystack-oidc-000000000000beef" + run_job "$dir" + if [ "$rc" -ne 0 ]; then + cat "$dir/out" + false + fi + if grep -q -F -- ' apply ' "$dir/kubectl.log"; then + echo "the Job ran kubectl apply with no bindings to apply" >&2 + false + fi + [ "$(grep -c 'delete clusterrolebinding' "$dir/kubectl.log")" -eq 2 ] + rm -rf "$dir" +} + +@test "the Job stops before deleting anything when the desired list is not mounted" { + dir=$(mktemp -d) + render_job "$dir" '[{"email":"alice@example.org","role":"admin"}]' + fake_kubectl "$dir" "cozystack-oidc-000000000000dead cozystack-oidc-000000000000beef" + rm "$dir/bindings/desired.txt" + run_job "$dir" + [ "$rc" -ne 0 ] + if grep -q -E 'delete clusterrolebinding| apply ' "$dir/kubectl.log"; then + cat "$dir/kubectl.log" + false + fi + rm -rf "$dir" +} diff --git a/packages/apps/kubernetes/templates/oidc-rbac-bindings.yaml b/packages/apps/kubernetes/templates/oidc-rbac-bindings.yaml new file mode 100644 index 000000000..a8517da4f --- /dev/null +++ b/packages/apps/kubernetes/templates/oidc-rbac-bindings.yaml @@ -0,0 +1,42 @@ +{{- /* + The ClusterRoleBindings the OIDC bootstrap Job (oidc-rbac-job.yaml) applies + inside the tenant cluster, one per `oidc.users` entry, and the list of their + names it prunes against. They travel in a ConfigMap the Job mounts and not as + text inside its script: the script reaches the container as a single `sh -c` + argument, Linux caps one argument at 128 KiB, and a binding per user crosses + that cap at about 240 users, after which the container cannot start. + + A release-owned object, not a hook resource, so it exists before the + post-install and post-upgrade hooks run. It is rendered only where users are + bound; at `mode: None` the Job takes its early branch and the volume is + optional. +*/}} +{{- if ne .Values.oidc.mode "None" }} +{{- $desired := "" }} +{{- $bindings := "" }} +{{- range .Values.oidc.users }} +{{- $name := printf "cozystack-oidc-%s" (printf "%s-%s" $.Release.Name .email | sha256sum | trunc 16) }} +{{- $role := "view" }} +{{- if eq .role "admin" }} +{{- $role = "cluster-admin" }} +{{- end }} +{{- $desired = printf "%s%s\n" $desired $name }} +{{- $binding := dict + "apiVersion" "rbac.authorization.k8s.io/v1" + "kind" "ClusterRoleBinding" + "metadata" (dict "name" $name "labels" (dict "app.kubernetes.io/managed-by" "cozystack-oidc" "app.kubernetes.io/instance" $.Release.Name)) + "roleRef" (dict "apiGroup" "rbac.authorization.k8s.io" "kind" "ClusterRole" "name" $role) + "subjects" (list (dict "kind" "User" "apiGroup" "rbac.authorization.k8s.io" "name" .email)) }} +{{- $bindings = printf "%s---\n%s\n" $bindings (toYaml $binding) }} +{{- end }} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ .Release.Name }}-oidc-bindings + namespace: {{ .Release.Namespace }} + labels: + {{- include "kubernetes.labels" . | nindent 4 }} +data: + {{- dict "crbs.yaml" $bindings "desired.txt" $desired | toYaml | nindent 2 }} +{{- end }} diff --git a/packages/apps/kubernetes/templates/oidc-rbac-job.yaml b/packages/apps/kubernetes/templates/oidc-rbac-job.yaml index 8672ce124..8556d092b 100644 --- a/packages/apps/kubernetes/templates/oidc-rbac-job.yaml +++ b/packages/apps/kubernetes/templates/oidc-rbac-job.yaml @@ -25,7 +25,9 @@ - `mode: System | CustomConfig`: - Apply one ClusterRoleBinding per `users[]` entry inside the tenant cluster (`admin` → `cluster-admin`, `view` → `view`, subject = the - literal email matched against the `email` claim). + literal email matched against the `email` claim). The bindings are + rendered by oidc-rbac-bindings.yaml into a ConfigMap this Job mounts, + not into the script below. - Prune stale CRBs labelled by this release. - `mode: System`: patch the pre-rendered `-oidc-kubeconfig` Secret with a real kubeconfig wired to `kubectl oidc-login` @@ -168,6 +170,8 @@ spec: value: {{ $issuer | quote }} - name: OIDC_SERVER_URL value: {{ printf "https://%s:443" $externalHost | quote }} + - name: OIDC_BINDINGS_DIR + value: /etc/oidc-bindings resources: requests: cpu: 10m @@ -254,46 +258,32 @@ spec: sleep 5 done + # A missing list must stop the Job here: grep on a file that is not + # there exits 2, `! grep` is then true, and the prune below would + # delete every binding of the release. + BINDINGS=$OIDC_BINDINGS_DIR + if [ ! -f "$BINDINGS/desired.txt" ]; then + echo "$BINDINGS/desired.txt is not mounted (ConfigMap $RELEASE-oidc-bindings); refusing to prune against a list that is not there" >&2 + exit 1 + fi + echo "Applying ${OIDC_MODE} ClusterRoleBindings to the tenant cluster..." - DESIRED_FILE=$(mktemp) - {{- range .Values.oidc.users }} - {{- $hash := printf "%s-%s" $.Release.Name .email | sha256sum | trunc 16 }} - {{- $crbName := printf "cozystack-oidc-%s" $hash }} - {{- $role := "view" }} - {{- if eq .role "admin" }} - {{- $role = "cluster-admin" }} - {{- end }} - echo "{{ $crbName }}" >> "$DESIRED_FILE" - cat <<'EOF' | kubectl --kubeconfig "$KUBECONFIG_TENANT" apply --filename - - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: {{ $crbName }} - labels: - app.kubernetes.io/managed-by: cozystack-oidc - app.kubernetes.io/instance: {{ $.Release.Name }} - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: {{ $role }} - subjects: - - kind: User - apiGroup: rbac.authorization.k8s.io - name: {{ .email | quote }} - EOF - {{- end }} + # kubectl apply fails on a file with no objects, and a release with + # no users has none. + if [ -s "$BINDINGS/crbs.yaml" ]; then + kubectl --kubeconfig "$KUBECONFIG_TENANT" apply --filename "$BINDINGS/crbs.yaml" + fi echo "Pruning stale ClusterRoleBindings labelled by this release..." EXISTING=$(kubectl --kubeconfig "$KUBECONFIG_TENANT" get clusterrolebindings \ --selector=app.kubernetes.io/managed-by=cozystack-oidc,app.kubernetes.io/instance=$RELEASE \ --output=jsonpath='{.items[*].metadata.name}') for crb in $EXISTING; do - if ! grep -x -q -F "$crb" "$DESIRED_FILE"; then + if ! grep -x -q -F "$crb" "$BINDINGS/desired.txt"; then echo " removing $crb" kubectl --kubeconfig "$KUBECONFIG_TENANT" delete clusterrolebinding "$crb" --ignore-not-found fi done - rm -f "$DESIRED_FILE" if [ "$OIDC_MODE" = "System" ]; then echo "Patching the OIDC kubeconfig Secret on the management cluster..." @@ -349,6 +339,9 @@ spec: - name: kubeconfig mountPath: /etc/kubernetes/kubeconfig readOnly: true + - name: oidc-bindings + mountPath: /etc/oidc-bindings + readOnly: true - name: tmp mountPath: /tmp volumes: @@ -356,6 +349,10 @@ spec: secret: secretName: {{ .Release.Name }}-admin-kubeconfig optional: true + - name: oidc-bindings + configMap: + name: {{ .Release.Name }}-oidc-bindings + optional: true - name: tmp emptyDir: {} {{- /* diff --git a/packages/apps/kubernetes/tests/oidc_test.yaml b/packages/apps/kubernetes/tests/oidc_test.yaml index 0623063e7..8511e392c 100644 --- a/packages/apps/kubernetes/tests/oidc_test.yaml +++ b/packages/apps/kubernetes/tests/oidc_test.yaml @@ -2561,7 +2561,73 @@ tests: value: "true" documentIndex: 3 - - it: bootstrap Job command builds a CRB for each users[] entry + - it: the bindings ConfigMap holds a CRB for each users[] entry + release: + name: kubernetes-app + namespace: tenant-foo + template: templates/oidc-rbac-bindings.yaml + set: + _cluster.oidc-enabled: "true" + _cluster.root-host: example.org + oidc.mode: System + oidc.users: + - email: alice@example.org + role: admin + - email: bob@example.org + role: view + asserts: + - isKind: + of: ConfigMap + - equal: + path: metadata.name + value: kubernetes-app-oidc-bindings + - matchRegex: + path: data["crbs.yaml"] + pattern: 'name: alice@example.org' + - matchRegex: + path: data["crbs.yaml"] + pattern: 'name: bob@example.org' + - matchRegex: + path: data["crbs.yaml"] + pattern: 'app.kubernetes.io/managed-by: cozystack-oidc' + - matchRegex: + path: data["crbs.yaml"] + pattern: 'app.kubernetes.io/instance: kubernetes-app' + + - it: the binding names and roles stay what the Job script used to render + # The prune step and every binding already in a tenant cluster go by these + # names: cozystack-oidc- plus the first 16 hex digits of + # sha256("-"). A different name leaves the old binding + # behind next to the new one. + release: + name: kubernetes-app + namespace: tenant-foo + template: templates/oidc-rbac-bindings.yaml + set: + _cluster.oidc-enabled: "true" + _cluster.root-host: example.org + oidc.mode: System + oidc.users: + - email: alice@example.org + role: admin + - email: bob@example.org + role: view + asserts: + - equal: + path: data["desired.txt"] + value: "cozystack-oidc-0001a28136ca1a02\ncozystack-oidc-ec8dfb1cbc75e699\n" + - matchRegex: + path: data["crbs.yaml"] + pattern: 'name: cozystack-oidc-0001a28136ca1a02\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: cluster-admin\nsubjects:\n- apiGroup: rbac.authorization.k8s.io\n kind: User\n name: alice@example.org\n' + - matchRegex: + path: data["crbs.yaml"] + pattern: 'name: cozystack-oidc-ec8dfb1cbc75e699\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: view\nsubjects:\n- apiGroup: rbac.authorization.k8s.io\n kind: User\n name: bob@example.org\n' + + - it: the bootstrap Job mounts the bindings ConfigMap and carries no users[] entry in its script + # The script is one `sh -c` argument and Linux caps one argument at 128 KiB: + # a binding per user in it stops the container from starting at about 240 + # users. hack/kubernetes-oidc-bindings.bats renders 300 users to hold the + # size itself. release: name: kubernetes-app namespace: tenant-foo @@ -2575,23 +2641,130 @@ tests: role: admin - email: bob@example.org role: view + asserts: + - contains: + path: spec.template.spec.volumes + content: + name: oidc-bindings + configMap: + name: kubernetes-app-oidc-bindings + optional: true + documentIndex: 3 + - contains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: oidc-bindings + mountPath: /etc/oidc-bindings + readOnly: true + documentIndex: 3 + # The script reads the directory from the environment, which is also how + # hack/kubernetes-oidc-bindings.bats points it at a scratch directory. + - contains: + path: spec.template.spec.containers[0].env + content: + name: OIDC_BINDINGS_DIR + value: /etc/oidc-bindings + documentIndex: 3 + - notMatchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'alice@example\.org' + documentIndex: 3 + - notMatchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'bob@example\.org' + documentIndex: 3 + + - it: the bootstrap Job stops before pruning when the desired list is not mounted + # `! grep -F` on a file that is not there is true, so a missing mount would + # delete every binding of the release. + release: + name: kubernetes-app + namespace: tenant-foo + template: templates/oidc-rbac-job.yaml + set: + _cluster.oidc-enabled: "true" + _cluster.root-host: example.org + oidc.mode: System asserts: - matchRegex: path: spec.template.spec.containers[0].command[2] - pattern: 'name: "alice@example.org"' + pattern: '(?s)if \[ ! -f "\$BINDINGS/desired\.txt" \]; then.*?exit 1\n\s*fi.*Pruning stale ClusterRoleBindings' documentIndex: 3 + + - it: the bootstrap Job applies the bindings file only when it has objects in it + # kubectl apply exits non-zero on a file with no objects, and a release + # with no users has none. + release: + name: kubernetes-app + namespace: tenant-foo + template: templates/oidc-rbac-job.yaml + set: + _cluster.oidc-enabled: "true" + _cluster.root-host: example.org + oidc.mode: System + asserts: - matchRegex: path: spec.template.spec.containers[0].command[2] - pattern: 'name: "bob@example.org"' + pattern: 'if \[ -s "\$BINDINGS/crbs\.yaml" \]; then\n\s+kubectl --kubeconfig "\$KUBECONFIG_TENANT" apply --filename "\$BINDINGS/crbs\.yaml"\n\s*fi' documentIndex: 3 + + - it: a release with no users renders an empty bindings ConfigMap + release: + name: kubernetes-app + namespace: tenant-foo + template: templates/oidc-rbac-bindings.yaml + set: + _cluster.oidc-enabled: "true" + _cluster.root-host: example.org + oidc.mode: System + oidc.users: [] + asserts: + - equal: + path: data["crbs.yaml"] + value: "" + - equal: + path: data["desired.txt"] + value: "" + + - it: CustomConfig mode renders the bindings ConfigMap for its users + release: + name: kubernetes-app + namespace: tenant-foo + template: templates/oidc-rbac-bindings.yaml + set: + _cluster.oidc-enabled: "false" + oidc.mode: CustomConfig + oidc.customConfig.config: | + apiVersion: apiserver.config.k8s.io/v1beta1 + kind: AuthenticationConfiguration + jwt: + - issuer: + url: https://idp.example.com + audiences: [my-cluster] + oidc.users: + - email: alice@example.org + role: admin + asserts: + - isKind: + of: ConfigMap - matchRegex: - path: spec.template.spec.containers[0].command[2] - pattern: 'name: cluster-admin' - documentIndex: 3 - - matchRegex: - path: spec.template.spec.containers[0].command[2] - pattern: 'name: view' - documentIndex: 3 + path: data["crbs.yaml"] + pattern: 'name: alice@example.org' + + - it: mode=None renders no bindings ConfigMap even when users are set + # The Job takes its early branch at mode None and never reads the file. + release: + name: kubernetes-app + namespace: tenant-foo + template: templates/oidc-rbac-bindings.yaml + set: + oidc.mode: None + oidc.users: + - email: alice@example.org + role: admin + asserts: + - hasDocuments: + count: 0 - it: mode=System aborts the Job instead of leaving the OIDC kubeconfig Secret as the install-time placeholder when the tenant CA is missing # A placeholder kubeconfig cannot authenticate, so the release must not go Ready with it. From 47b497194eab5187e09ecbebc1ca862ae229808d Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 11:10:42 +0000 Subject: [PATCH 071/518] fix(backups): restore the ClickHouse archive a Backup was taken as The Altinity strategy named each archive inside its Pod with date -u and recorded nothing on the Backup, so a restore could only guess: it listed the remote archives under the application's prefix and took the newest. A RestoreJob of an older Backup therefore restored whichever backup had run last. The controller now names the archive before it creates the backup Job, from the BackupJob's persisted StartedAt, so every reconcile and every retried Pod of the Job writes the same one. The name is recorded in the Backup's driverMetadata next to the strategy parameters, and a restore renders it into the strategy Pod, which restores exactly that archive and fails, naming it, when remote storage no longer holds it. The name is recorded only when the Job's Pod was given it, so a Job created by an earlier controller, or a strategy that names its own archives, leaves the Backup without one instead of pointing it at an archive that does not exist. Because a retried Pod reuses the name, it waits on an attempt the sidecar is still running instead of being refused with 423, and asks for resume, so a local or partial remote backup an earlier attempt left is completed rather than refused as existing. Backups taken before names were recorded stay restorable: the Pod picks the newest archive whose name is stamped no later than the Backup's takenAt, which is set only after the backup Job finished, and logs a warning that the choice is a best effort. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- hack/altinity-backup-strategy.bats | 209 +++++++++++++++++ .../altinitystrategy_controller.go | 69 +++++- .../altinitystrategy_controller_test.go | 210 +++++++++++++++++- .../templates/strategy-altinity-default.yaml | 57 ++++- 4 files changed, 523 insertions(+), 22 deletions(-) create mode 100644 hack/altinity-backup-strategy.bats diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats new file mode 100644 index 000000000..234ef0b8b --- /dev/null +++ b/hack/altinity-backup-strategy.bats @@ -0,0 +1,209 @@ +#!/usr/bin/env bats +# Behavioural tests for the shipped Altinity (ClickHouse) backup/restore +# strategy script, extracted from the rendered cozy-default-altinity Strategy +# CR and EXECUTED against a stubbed clickhouse-backup HTTP API, with the real +# jq doing the archive selection. Which archive a restore reads back is +# decided by that selection over list/remote, so it is exercised against +# several archives rather than string-matched. +# +# The script runs with `bash`, not `sh`, for the reason given in +# hack/redis-backup-strategy.bats: it opens `set -euo pipefail`, which the +# production image's busybox ash supports and the CI runner's dash rejects. + +load test_helper + +CHART=packages/system/backupstrategy-controller + +# Render the shipped strategy's in-Pod script into $1 and substitute the +# controller-render placeholders the way the driver fills them: $2 mode, $3 +# app (release and source application), $4 the archive name recorded on the +# Backup (empty for a Backup that records none), $5 the Backup name, $6 its +# takenAt in the archive-name layout. An empty render fails early, as does a +# placeholder left unsubstituted. +render_script() { + out=$1; mode=$2; app=$3; archive=$4; backup=$5; takenat=$6 + helm template t "$CHART" -n cozy-backup-controller \ + --set backupStorage.bucketNameOverride=test-bucket 2>/dev/null \ + | yq eval 'select(.kind == "Altinity") | .spec.template.spec.containers[0].args[0]' - \ + | sed -e "s#{{ .Mode }}#${mode}#g" \ + -e "s#{{ .Release.Name }}#${app}#g" \ + -e "s#{{ .Release.Namespace }}#tenant-x#g" \ + -e "s#{{ .Backup.ApplicationRef.Name }}#${app}#g" \ + -e "s#{{ .Backup.Name }}#${backup}#g" \ + -e "s#{{ .ArchiveName | default \"\" }}#${archive}#g" \ + -e "s#{{ .Backup.TakenAt | default \"\" }}#${takenat}#g" > "$out" + [ -s "$out" ] || return 1 + if grep -q '{{' "$out"; then + echo "rendered script still holds a controller placeholder" >&2 + return 1 + fi +} + +# Stub the clickhouse-backup API behind curl, first on PATH. list/remote +# serves $state/remote; actions serves $state/actions plus one success row per +# operation the script has issued, and after its first read swaps in +# $state/actions.next when a test seeds one (an operation that finishes while +# the script waits). Every request is logged to $state/curl.log. sleep is +# stubbed to fail after a few calls, so a script that never sees its +# operation finish fails the test instead of hanging it. +make_stubs() { + bin=$1; state=$2 + mkdir -p "$bin" + : > "$state/remote" + : > "$state/actions" + cat > "$bin/curl" <> "$state/curl.log" +case "\$method \$url" in + "GET "*/backup/status) echo '{}' ;; + "GET "*/backup/list/remote) cat "$state/remote" ;; + "GET "*/backup/actions) + cat "$state/actions" + if [ -f "$state/actions.done" ]; then cat "$state/actions.done"; fi + if [ -f "$state/actions.next" ]; then mv "$state/actions.next" "$state/actions"; fi + ;; + "POST "*/backup/create_remote*) + name=\${url#*name=}; name=\${name%%&*} + printf '{"command":"create_remote --resume %s","status":"success","start":"2099-01-01 00:00:00"}\n' "\$name" >> "$state/actions.done" + ;; + "POST "*/backup/restore_remote/*) + printf '{"command":"restore_remote %s","status":"success","start":"2099-01-01 00:00:00"}\n' "\${url##*/}" >> "$state/actions.done" + ;; + *) echo "unexpected request: \$method \$url" >&2; exit 22 ;; +esac +STUB + cat > "$bin/sleep" </dev/null || echo 0) +echo \$((n + 1)) > "$state/sleeps" +if [ "\$n" -ge 3 ]; then echo "sleep budget spent: the operation never finished" >&2; exit 1; fi +STUB + chmod 0755 "$bin/curl" "$bin/sleep" + export API_USERNAME=u API_PASSWORD=p +} + +# One list/remote row per argument, "name" or "name|desc"; created mirrors +# the name's timestamp, which is the order the pre-fix selection sorted by. +remote_rows() { + for row in "$@"; do + name=${row%%|*}; desc="tar, regular" + if [ "$row" != "$name" ]; then desc=${row#*|}; fi + ts=${name##*-} + printf '{"name":"%s","created":"%s","location":"remote","required":"","desc":"%s"}\n' "$name" "$ts" "$desc" + done +} + +@test "an empty render fails the fixture instead of passing as a no-op" { + tmp=$(mktemp -d) + CHART=$tmp/no-such-chart + if render_script "$tmp/s.sh" restore myapp "" bkp 20260304T050622Z 2>/dev/null; then + echo "FAIL: render_script reported success with no chart" + false + fi +} + +@test "restore reads back the archive the Backup recorded, not the newest one" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp" + render_script "$tmp/s.sh" restore myapp clickhouse-myapp-20260304T050607Z bkp-a 20260304T050622Z + remote_rows clickhouse-myapp-20260304T040000Z clickhouse-myapp-20260304T050607Z \ + clickhouse-myapp-20260304T060000Z clickhouse-myapp-20260304T070000Z > "$tmp/remote" + PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" + grep -qxF 'POST http://chi-clickhouse-myapp-clickhouse-0-0.tenant-x.svc:7171/backup/restore_remote/clickhouse-myapp-20260304T050607Z' "$tmp/curl.log" + [ "$(grep -c '/backup/restore_remote/' "$tmp/curl.log")" -eq 1 ] +} + +@test "restore fails naming the recorded archive when remote storage lacks it" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp" + render_script "$tmp/s.sh" restore myapp clickhouse-myapp-20260304T050607Z bkp-a 20260304T050622Z + remote_rows clickhouse-myapp-20260304T040000Z clickhouse-myapp-20260304T060000Z > "$tmp/remote" + out="" + if out=$(PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" 2>&1); then + echo "FAIL: restore succeeded without the recorded archive: $out" + false + fi + printf '%s\n' "$out" | grep -qF 'archive clickhouse-myapp-20260304T050607Z recorded on Backup bkp-a is not in remote storage' + if grep -qF '/backup/restore_remote/' "$tmp/curl.log"; then + echo "FAIL: restored another archive: $(cat "$tmp/curl.log")" + false + fi +} + +@test "restore of a Backup without a recorded archive takes the newest one stamped by its takenAt, and warns" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp" + render_script "$tmp/s.sh" restore myapp "" bkp-a 20260304T050800Z + # Each decoy sorts after the expected archive and is ruled out by one + # filter: broken, not a bare timestamp after the prefix, or stamped after + # takenAt. + remote_rows clickhouse-myapp-20260304T040000Z clickhouse-myapp-20260304T050607Z \ + "clickhouse-myapp-20260304T050700Z|broken (can't stat metadata.json)" \ + clickhouse-myapp-20260304T050710Z-copy \ + clickhouse-myapp-20260304T060000Z > "$tmp/remote" + out=$(PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" 2>&1) + grep -qxF 'POST http://chi-clickhouse-myapp-clickhouse-0-0.tenant-x.svc:7171/backup/restore_remote/clickhouse-myapp-20260304T050607Z' "$tmp/curl.log" + [ "$(grep -c '/backup/restore_remote/' "$tmp/curl.log")" -eq 1 ] + printf '%s\n' "$out" | grep -qF 'WARNING: Backup bkp-a records no archive name' +} + +@test "restore of a Backup without a recorded archive fails when no archive is stamped by its takenAt" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp" + render_script "$tmp/s.sh" restore myapp "" bkp-a 20260304T050800Z + remote_rows clickhouse-myapp-20260304T060000Z clickhouse-myapp-20260304T070000Z > "$tmp/remote" + out="" + if out=$(PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" 2>&1); then + echo "FAIL: restore succeeded with every archive newer than takenAt: $out" + false + fi + printf '%s\n' "$out" | grep -qF 'no archive of myapp stamped no later than 20260304T050800Z' + if grep -qF '/backup/restore_remote/' "$tmp/curl.log"; then + echo "FAIL: restored an archive newer than takenAt: $(cat "$tmp/curl.log")" + false + fi +} + +@test "backup writes the archive under the name the controller chose" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp" + render_script "$tmp/s.sh" backup myapp clickhouse-myapp-20260304T050607Z "" "" + PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" + grep -qxF 'POST http://chi-clickhouse-myapp-clickhouse-0-0.tenant-x.svc:7171/backup/create_remote?name=clickhouse-myapp-20260304T050607Z&resume=true' "$tmp/curl.log" +} + +@test "a retried backup waits on the attempt the sidecar is still running" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp" + render_script "$tmp/s.sh" backup myapp clickhouse-myapp-20260304T050607Z "" "" + printf '{"command":"create_remote --resume clickhouse-myapp-20260304T050607Z","status":"in progress","start":"2026-03-04 05:06:08"}\n' > "$tmp/actions" + printf '{"command":"create_remote --resume clickhouse-myapp-20260304T050607Z","status":"success","start":"2026-03-04 05:06:08"}\n' > "$tmp/actions.next" + PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" + # The sidecar answers a second operation with 423 while one runs. + if grep -qF 'POST ' "$tmp/curl.log"; then + echo "FAIL: started another operation over the running one: $(cat "$tmp/curl.log")" + false + fi +} + +@test "a retried backup after a failed attempt resumes the same archive" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp" + render_script "$tmp/s.sh" backup myapp clickhouse-myapp-20260304T050607Z "" "" + printf '{"command":"create_remote clickhouse-myapp-20260304T050607Z","status":"error","start":"2026-03-04 05:06:08","error":"S3 PutObject 403"}\n' > "$tmp/actions" + PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" + grep -qxF 'POST http://chi-clickhouse-myapp-clickhouse-0-0.tenant-x.svc:7171/backup/create_remote?name=clickhouse-myapp-20260304T050607Z&resume=true' "$tmp/curl.log" +} + +@test "backup falls back to a timestamped name when the controller supplies none" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp" + render_script "$tmp/s.sh" backup myapp "" "" "" + PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" + grep -qE '^POST http://[^ ]*/backup/create_remote\?name=clickhouse-myapp-[0-9]{8}T[0-9]{6}Z&resume=true$' "$tmp/curl.log" +} diff --git a/internal/backupcontroller/altinitystrategy_controller.go b/internal/backupcontroller/altinitystrategy_controller.go index b96d571f0..f7f60bde8 100644 --- a/internal/backupcontroller/altinitystrategy_controller.go +++ b/internal/backupcontroller/altinitystrategy_controller.go @@ -2,7 +2,9 @@ package backupcontroller import ( + "bytes" "context" + "encoding/json" "fmt" "maps" "strings" @@ -43,6 +45,18 @@ const ( // were in effect at backup time. altinityParamPrefix = "altinity.strategy.backups.cozystack.io/parameter/" + // altinityArchiveMetaKey records, on the Backup, the clickhouse-backup + // archive its Job wrote, as named by altinityArchiveName. A restore reads + // back exactly that archive: the newest one under the release's prefix + // may belong to a later Backup. + altinityArchiveMetaKey = "altinity.strategy.backups.cozystack.io/archive" + + // altinityArchiveTimeLayout is the UTC timestamp that ends an archive + // name. It is fixed-width, so names sharing a prefix sort chronologically + // as plain strings, which is what a restore of a Backup that records no + // archive relies on. + altinityArchiveTimeLayout = "20060102T150405Z" + // Polling cadence for the Altinity Job lifecycle. Mirrors the CNPG / // Velero strategy cadence so behaviour is uniform across drivers. altinityPollInterval = 5 * time.Second @@ -102,14 +116,37 @@ func backupParameters(b *backupsv1alpha1.Backup) map[string]string { return out } +// altinityArchiveName mints the archive name for one BackupJob from its +// persisted StartedAt, so every reconcile and every Pod retry of the Job +// writes the same archive. It keeps the clickhouse-- +// shape, because a Backup that records no archive is restored by the +// timestamp in that shape. +func altinityArchiveName(j *backupsv1alpha1.BackupJob) string { + return fmt.Sprintf("clickhouse-%s-%s", j.Spec.ApplicationRef.Name, j.Status.StartedAt.UTC().Format(altinityArchiveTimeLayout)) +} + +// altinityJobCarriesArchive reports whether the Job's Pod was rendered with +// the archive name. A Job created by a controller that predates archive +// names, or rendered from a strategy that names its own archives, wrote an +// archive under another name, and recording this one would point the Backup +// at an archive that does not exist. +func altinityJobCarriesArchive(job *batchv1.Job, archive string) bool { + spec, err := json.Marshal(job.Spec.Template.Spec) + if err != nil { + return false + } + return bytes.Contains(spec, []byte(archive)) +} + // renderAltinityTemplate runs the strategy's PodTemplateSpec through the // repository's text/template engine with the same context shape as the CNPG // driver: every string field is templated against the application object, -// the release shorthand, the run mode, and the resolved parameters. +// the release shorthand, the run mode, the archive name, and the resolved +// parameters. func renderAltinityTemplate( tmpl corev1.PodTemplateSpec, app map[string]any, - releaseName, releaseNamespace, mode string, + releaseName, releaseNamespace, mode, archiveName string, parameters map[string]string, backup *backupsv1alpha1.Backup, ) (*corev1.PodTemplateSpec, error) { @@ -119,8 +156,9 @@ func renderAltinityTemplate( "Name": releaseName, "Namespace": releaseNamespace, }, - "Mode": mode, - "Parameters": parameters, + "Mode": mode, + "ArchiveName": archiveName, + "Parameters": parameters, } if backup != nil { // Expose ApplicationRef so restore strategies can reference the @@ -143,6 +181,9 @@ func renderAltinityTemplate( "Kind": backup.Spec.ApplicationRef.Kind, "Name": backup.Spec.ApplicationRef.Name, }, + // In the archive-name layout, so a strategy can compare it with + // the timestamp an archive name ends in. + "TakenAt": backup.Spec.TakenAt.UTC().Format(altinityArchiveTimeLayout), } } return template.Template(&tmpl, ctxMap) @@ -209,12 +250,14 @@ func (r *BackupJobReconciler) reconcileAltinity(ctx context.Context, j *backupsv return ctrl.Result{}, err } + archive := altinityArchiveName(j) rendered, err := renderAltinityTemplate( strategy.Spec.Template, app, j.Spec.ApplicationRef.Name, j.Namespace, altinityModeBackup, + archive, resolved.Parameters, nil, ) @@ -239,7 +282,10 @@ func (r *BackupJobReconciler) reconcileAltinity(ctx context.Context, j *backupsv if j.Status.BackupRef != nil { return ctrl.Result{}, nil } - artifact, err := r.createAltinityBackupArtifact(ctx, j, resolved) + if !altinityJobCarriesArchive(batchJob, archive) { + archive = "" + } + artifact, err := r.createAltinityBackupArtifact(ctx, j, resolved, archive) if err != nil { return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to create Backup artifact: %v", err)) } @@ -411,16 +457,20 @@ func jobFailureMessage(j *batchv1.Job) string { } // createAltinityBackupArtifact materialises a Cozystack Backup carrying the -// strategy reference and the BackupClassStrategy parameters in effect at -// backup time. Parameters round-trip via DriverMetadata under -// altinityParamPrefix so a later RestoreJob can re-render the strategy -// template against the same values. +// strategy reference, the archive the Job wrote (when known) and the +// BackupClassStrategy parameters in effect at backup time. Parameters +// round-trip via DriverMetadata under altinityParamPrefix so a later +// RestoreJob can re-render the strategy template against the same values. func (r *BackupJobReconciler) createAltinityBackupArtifact( ctx context.Context, j *backupsv1alpha1.BackupJob, resolved *ResolvedBackupConfig, + archive string, ) (*backupsv1alpha1.Backup, error) { driverMD := map[string]string{} + if archive != "" { + driverMD[altinityArchiveMetaKey] = archive + } for k, v := range resolved.Parameters { driverMD[altinityParamPrefix+k] = v } @@ -616,6 +666,7 @@ func (r *RestoreJobReconciler) reconcileAltinityRestore(ctx context.Context, res targetAppName, targetNamespace, altinityModeRestore, + backup.Spec.DriverMetadata[altinityArchiveMetaKey], backupParameters(backup), backup, ) diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index 68f18fa28..7797842c0 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -4,6 +4,7 @@ package backupcontroller import ( "context" "testing" + "time" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" @@ -697,6 +698,7 @@ func TestBackupParameters_RoundTrip(t *testing.T) { DriverMetadata: map[string]string{ altinityParamPrefix + "bucketName": "b1", altinityParamPrefix + "endpoint": "https://s3", + altinityArchiveMetaKey: "clickhouse-ch-test-20260304T050607Z", "job.batch/name": "ignored", "": "ignored", }, @@ -706,7 +708,213 @@ func TestBackupParameters_RoundTrip(t *testing.T) { if got["bucketName"] != "b1" || got["endpoint"] != "https://s3" { t.Errorf("backupParameters = %v", got) } - if _, ok := got["job.batch/name"]; ok { + if len(got) != 2 { t.Errorf("non-parameter key leaked through: %v", got) } } + +// newAltinityArchiveStrategy renders the archive name and the Backup's +// takenAt where a test can read them back off the Job. +func newAltinityArchiveStrategy(name string) *strategyv1alpha1.Altinity { + s := newAltinityStrategy(name) + s.Spec.Template.Spec.Containers[0].Args = []string{ + "--mode={{ .Mode }}", + "--archive={{ .ArchiveName }}", + "--taken-at={{ .Backup.TakenAt }}", + "--bucket={{ .Parameters.bucketName }}", + } + return s +} + +func TestAltinityArchiveName_FromStartedAtInUTC(t *testing.T) { + startedAt := metav1.NewTime(time.Date(2026, 3, 4, 8, 6, 7, 890000000, time.FixedZone("UTC+3", 3*3600))) + j := &backupsv1alpha1.BackupJob{ + Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: newAltinityClickHouseRef("ch-test")}, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt}, + } + if got, want := altinityArchiveName(j), "clickhouse-ch-test-20260304T050607Z"; got != want { + t.Errorf("altinityArchiveName = %q, want %q", got, want) + } +} + +// TestReconcileAltinity_RecordsTheArchiveTheJobWasGiven drives one BackupJob +// through Job creation and completion in separate reconciles. The archive +// name has to come out of the persisted StartedAt both times: the Pod is +// told it on the first, the Backup records it on the second, and a restore +// later reads back exactly that archive. +func TestReconcileAltinity_RecordsTheArchiveTheJobWasGiven(t *testing.T) { + app := newAltinityClickHouseApp("ch-test", "tenant-test") + strategy := newAltinityArchiveStrategy("clickhouse-strategy") + startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC)) + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + BackupClassName: "clickhouse-backup", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt, Phase: backupsv1alpha1.BackupJobPhaseRunning}, + } + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + Parameters: map[string]string{"bucketName": "ch-bucket"}, + } + r, _, _ := newAltinityTestEnv(t, app, clientfake.NewClientBuilder().WithObjects(backupJob, strategy)) + ctx := context.Background() + const want = "clickhouse-ch-test-20260304T050607Z" + + if _, err := r.reconcileAltinity(ctx, backupJob, resolved); err != nil { + t.Fatalf("reconcileAltinity() creating the Job: %v", err) + } + job := &batchv1.Job{} + if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}, job); err != nil { + t.Fatalf("get backup Job: %v", err) + } + if got := job.Spec.Template.Spec.Containers[0].Args[1]; got != "--archive="+want { + t.Fatalf("backup Pod was rendered with %q, want --archive=%s", got, want) + } + + job.Status.Conditions = []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}} + if err := r.Status().Update(ctx, job); err != nil { + t.Fatalf("mark backup Job complete: %v", err) + } + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), backupJob); err != nil { + t.Fatalf("refresh BackupJob: %v", err) + } + if _, err := r.reconcileAltinity(ctx, backupJob, resolved); err != nil { + t.Fatalf("reconcileAltinity() on the completed Job: %v", err) + } + + created := &backupsv1alpha1.Backup{} + if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: backupJob.Name}, created); err != nil { + t.Fatalf("get Backup: %v", err) + } + if got := created.Spec.DriverMetadata[altinityArchiveMetaKey]; got != want { + t.Errorf("Backup records archive %q, want %q", got, want) + } + if got := created.Spec.DriverMetadata[altinityParamPrefix+"bucketName"]; got != "ch-bucket" { + t.Errorf("expected driverMetadata[parameter/bucketName]=ch-bucket, got %q", got) + } +} + +// TestReconcileAltinity_RecordsNoArchiveTheJobWasNotGiven covers a Job whose +// Pod never carried the driver's archive name: one created before the +// upgrade that introduced the names, or rendered from a strategy that names +// its own archives. Recording the name would send the restore after an +// archive nobody wrote; recording none leaves it the fallback for Backups +// without a name. +func TestReconcileAltinity_RecordsNoArchiveTheJobWasNotGiven(t *testing.T) { + app := newAltinityClickHouseApp("ch-test", "tenant-test") + strategy := newAltinityArchiveStrategy("clickhouse-strategy") + startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC)) + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + BackupClassName: "clickhouse-backup", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt, Phase: backupsv1alpha1.BackupJobPhaseRunning}, + } + completedK8sJob := &batchv1.Job{ + ObjectMeta: metav1.ObjectMeta{Name: jobNameForBackupJob(backupJob), Namespace: backupJob.Namespace}, + Spec: batchv1.JobSpec{ + Template: corev1.PodTemplateSpec{Spec: corev1.PodSpec{ + RestartPolicy: corev1.RestartPolicyNever, + Containers: []corev1.Container{{ + Name: "clickhouse-backup", + Image: "altinity/clickhouse-backup:test", + Args: []string{`target="clickhouse-ch-test-$(date -u +%Y%m%dT%H%M%SZ)"`}, + }}, + }}, + }, + Status: batchv1.JobStatus{ + Conditions: []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}}, + }, + } + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + Parameters: map[string]string{"bucketName": "ch-bucket"}, + } + r, _, _ := newAltinityTestEnv(t, app, + clientfake.NewClientBuilder().WithObjects(backupJob, strategy, completedK8sJob)) + ctx := context.Background() + + if _, err := r.reconcileAltinity(ctx, backupJob, resolved); err != nil { + t.Fatalf("reconcileAltinity() error = %v", err) + } + created := &backupsv1alpha1.Backup{} + if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: backupJob.Name}, created); err != nil { + t.Fatalf("get Backup: %v", err) + } + if got, ok := created.Spec.DriverMetadata[altinityArchiveMetaKey]; ok { + t.Errorf("Backup records archive %q its Job was never given", got) + } + if got := created.Spec.DriverMetadata[altinityParamPrefix+"bucketName"]; got != "ch-bucket" { + t.Errorf("expected driverMetadata[parameter/bucketName]=ch-bucket, got %q", got) + } +} + +func TestReconcileAltinityRestore_RendersTheBackupsArchive(t *testing.T) { + cases := []struct { + name string + archive string // recorded on the Backup; empty for a Backup taken before names were recorded + wantArchive string + }{ + {name: "recorded archive", archive: "clickhouse-ch-test-20260304T050607Z", wantArchive: "--archive=clickhouse-ch-test-20260304T050607Z"}, + {name: "no recorded archive", archive: "", wantArchive: "--archive="}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + targetApp := newAltinityClickHouseApp("ch-test", "tenant-test") + strategy := newAltinityArchiveStrategy("clickhouse-strategy") + driverMD := map[string]string{altinityParamPrefix + "bucketName": "ch-bucket"} + if tc.archive != "" { + driverMD[altinityArchiveMetaKey] = tc.archive + } + now := metav1.Now() + backup := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Name: "ch-backup", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + TakenAt: metav1.NewTime(time.Date(2026, 3, 4, 8, 6, 22, 0, time.FixedZone("UTC+3", 3*3600))), + DriverMetadata: driverMD, + }, + } + restoreJob := &backupsv1alpha1.RestoreJob{ + ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}}, + Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning}, + } + _, rr, _ := newAltinityTestEnv(t, targetApp, + clientfake.NewClientBuilder().WithObjects(strategy, backup, restoreJob)) + ctx := context.Background() + + if _, err := rr.reconcileAltinityRestore(ctx, restoreJob, backup); err != nil { + t.Fatalf("reconcileAltinityRestore: %v", err) + } + job := &batchv1.Job{} + if err := rr.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForRestoreJob(restoreJob)}, job); err != nil { + t.Fatalf("get restore Job: %v", err) + } + args := job.Spec.Template.Spec.Containers[0].Args + want := []string{"--mode=restore", tc.wantArchive, "--taken-at=20260304T050622Z", "--bucket=ch-bucket"} + for i, w := range want { + if i >= len(args) || args[i] != w { + t.Errorf("rendered restore args[%d]: want %q, got %q", i, w, args) + } + } + }) + } +} diff --git a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml index 3397d6d0d..bb9917094 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml @@ -3,8 +3,8 @@ 1. Helm renders this file at chart install time. Anything inside `{{ ... }}` is interpreted by Helm. 2. The strategy controller renders the resulting Strategy CR - per-BackupJob, expanding `{{ .Application | .Release | .Mode }}` - against the live application object. + per-BackupJob, expanding `{{ .Application | .Release | .Mode | + .ArchiveName | .Backup }}` against the live application object. To emit a literal "{{ .Foo }}" that survives Helm, we pass it as the string argument of `printf` — Helm does not interpret inner template delimiters inside printf's quoted strings. Do NOT simplify by inlining @@ -51,23 +51,56 @@ spec: host="chi-clickhouse-{{ printf "{{ .Release.Name }}" }}-clickhouse-0-0.{{ printf "{{ .Release.Namespace }}" }}.svc" api="http://${host}:7171" auth="-u ${API_USERNAME}:${API_PASSWORD}" + # The archive the controller named for this BackupJob and records + # on its Backup. The default renders it empty, rather than as + # text/template's missing-key marker, when a controller that + # predates archive names renders this template during an upgrade. + archive="{{ printf `{{ .ArchiveName | default "" }}` }}" for _ in $(seq 1 60); do if curl -fsS ${auth} --max-time 2 "${api}/backup/status" >/dev/null 2>&1; then break; fi sleep 5 done if [ "{{ printf "{{ .Mode }}" }}" = "backup" ]; then prefix="clickhouse-{{ printf "{{ .Release.Name }}" }}-" - target="${prefix}$(date -u +%Y%m%dT%H%M%SZ)" - echo "create_remote ${target}" - curl -fsS ${auth} -X POST "${api}/backup/create_remote?name=${target}" >/dev/null + # Only that older controller leaves the name empty, and it + # records no name on the Backup either. + target="${archive:-${prefix}$(date -u +%Y%m%dT%H%M%SZ)}" + # A retried Pod gets the same name. Wait on an attempt the + # sidecar is still running, which would answer a new one with + # 423, and ask for resume, so a local or partial remote backup + # an earlier attempt left is resumed rather than refused as + # existing, whatever use_resumable_state the sidecar runs with. + state=$(curl -fsS ${auth} "${api}/backup/actions" \ + | jq -rs --arg suffix "${target}" \ + '[.[] | select((.command // "") | endswith($suffix))] | sort_by(.start) | last | .status // ""') + if [ "${state}" != "in progress" ]; then + echo "create_remote ${target}" + curl -fsS ${auth} -X POST "${api}/backup/create_remote?name=${target}&resume=true" >/dev/null + fi else - prefix="clickhouse-{{ printf "{{ .Backup.ApplicationRef.Name }}" }}-" - target=$(curl -fsS ${auth} "${api}/backup/list/remote" \ - | jq -rs --arg prefix "${prefix}" \ - '[.[] | select((.name // "") | startswith($prefix))] | sort_by(.created // "") | last.name // ""') - if [ -z "${target}" ] || [ "${target}" = "null" ]; then - echo "no remote backup found" >&2 - exit 1 + app="{{ printf "{{ .Backup.ApplicationRef.Name }}" }}" + backup="{{ printf "{{ .Backup.Name }}" }}" + prefix="clickhouse-${app}-" + remote=$(curl -fsS ${auth} "${api}/backup/list/remote") + if [ -n "${archive}" ]; then + if ! printf '%s\n' "${remote}" | jq -es --arg name "${archive}" 'any(.[]; .name == $name)' >/dev/null; then + echo "archive ${archive} recorded on Backup ${backup} is not in remote storage; not restoring any other archive" >&2 + exit 1 + fi + target="${archive}" + else + # takenAt is set after the backup Job finished, so the + # archive that Job wrote is stamped no later than it. + notafter="{{ printf `{{ .Backup.TakenAt | default "" }}` }}" + echo "WARNING: Backup ${backup} records no archive name; restoring the newest archive of ${app} stamped no later than its takenAt (${notafter}), a best-effort choice that may not be the archive this Backup was taken as" >&2 + target=$(printf '%s\n' "${remote}" | jq -rs --arg prefix "${prefix}" --arg notafter "${notafter}" ' + [.[] | select((.desc // "") | startswith("broken") | not) | .name // "" + | select(startswith($prefix) and (ltrimstr($prefix) | test("^[0-9]{8}T[0-9]{6}Z$")) and ltrimstr($prefix) <= $notafter)] + | sort | last // ""') + if [ -z "${target}" ]; then + echo "no archive of ${app} stamped no later than ${notafter} in remote storage" >&2 + exit 1 + fi fi echo "restore_remote ${target}" curl -fsS ${auth} -X POST "${api}/backup/restore_remote/${target}" >/dev/null From 3b1302ba8e8d44a5294b28302a775f805bd40e2a Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 18:45:36 +0700 Subject: [PATCH 072/518] feat(dashboard): add an accessible Select to the console The console has no select primitive: form widgets fall back to the native element, and the shared Dropdown has no listbox roles, no keyboard navigation and no label association. Base UI, already a dependency, ships one; this wraps it the same way dialog.tsx wraps the Base UI dialog so pickers get a labelled, keyboard-driven listbox that matches the console's buttons. Assisted-by: LLM Signed-off-by: ZverGuy --- .../console/src/components/ui/select.test.tsx | 64 +++++++++++ .../apps/console/src/components/ui/select.tsx | 107 ++++++++++++++++++ 2 files changed, 171 insertions(+) create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/ui/select.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/ui/select.tsx diff --git a/packages/system/dashboard/images/console/apps/console/src/components/ui/select.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/ui/select.test.tsx new file mode 100644 index 000000000..fb5b84e6f --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/ui/select.test.tsx @@ -0,0 +1,64 @@ +import { describe, it, expect, vi } from "vitest" +import { render, screen } from "@testing-library/react" +import userEvent from "@testing-library/user-event" +import { SelectItem, SelectLabel, SelectPopup, SelectRoot, SelectTrigger, SelectValue } from "./select.tsx" + +const items = [ + { value: "", label: "All" }, + { value: "db-0", label: "db-0" }, + { value: "db-1", label: "db-1" }, +] + +function PodSelect({ value = "", onValueChange = () => {} }: { value?: string; onValueChange?: (v: string) => void }) { + return ( + onValueChange(v ?? "")}> + Pod + + + + + {items.map((i) => ( + + {i.label} + + ))} + + + ) +} + +describe("Select", () => { + it("names the trigger by its label and shows the chosen item", () => { + render() + + expect(screen.getByRole("combobox", { name: "Pod" })).toHaveTextContent("db-1") + }) + + it("shows the label of an empty-string value", () => { + render() + + expect(screen.getByRole("combobox", { name: "Pod" })).toHaveTextContent("All") + }) + + it("lists the items and reports the one clicked", async () => { + const onValueChange = vi.fn() + render() + + await userEvent.click(screen.getByRole("combobox", { name: "Pod" })) + await userEvent.click(await screen.findByRole("option", { name: "db-1" })) + + expect(onValueChange).toHaveBeenCalledWith("db-1") + }) + + it("can be driven from the keyboard", async () => { + const onValueChange = vi.fn() + render() + + screen.getByRole("combobox", { name: "Pod" }).focus() + await userEvent.keyboard("{ArrowDown}") + expect(await screen.findByRole("listbox")).toBeInTheDocument() + await userEvent.keyboard("{ArrowDown}{Enter}") + + expect(onValueChange).toHaveBeenCalledWith("db-0") + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/ui/select.tsx b/packages/system/dashboard/images/console/apps/console/src/components/ui/select.tsx new file mode 100644 index 000000000..f256dba45 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/ui/select.tsx @@ -0,0 +1,107 @@ +import { Select as SelectPrimitive } from "@base-ui/react/select" +import { Check, ChevronDown } from "lucide-react" +import { cn } from "@cozystack/ui" + +const SelectRoot = SelectPrimitive.Root + +function SelectLabel({ + className, + ...props +}: SelectPrimitive.Label.Props) { + return ( + + ) +} + +function SelectTrigger({ + className, + children, + ...props +}: SelectPrimitive.Trigger.Props) { + return ( + + {children} + + + + + ) +} + +function SelectValue({ + className, + ...props +}: SelectPrimitive.Value.Props) { + return ( + + ) +} + +function SelectPopup({ + className, + children, + ...props +}: SelectPrimitive.Popup.Props) { + return ( + + + + {children} + + + + ) +} + +function SelectItem({ + className, + children, + ...props +}: SelectPrimitive.Item.Props) { + return ( + + + + + {children} + + ) +} + +export { + SelectRoot, + SelectLabel, + SelectTrigger, + SelectValue, + SelectPopup, + SelectItem, +} From f500aa955ac65384b527bf61167e77bf5dd02a3e Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 18:46:15 +0700 Subject: [PATCH 073/518] feat(dashboard): add a text Input matching the console select Every text field in the console restyles a bare input by hand, each a little differently. A toolbar that mixes a text filter with the new Select needs the two at the same height, border and focus ring, so the styling lives in one place next to the Select instead of being copied. Assisted-by: LLM Signed-off-by: ZverGuy --- .../console/src/components/ui/input.test.tsx | 21 +++++++++++++++++++ .../apps/console/src/components/ui/input.tsx | 17 +++++++++++++++ 2 files changed, 38 insertions(+) create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/ui/input.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/ui/input.tsx diff --git a/packages/system/dashboard/images/console/apps/console/src/components/ui/input.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/ui/input.test.tsx new file mode 100644 index 000000000..e65a70a3a --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/ui/input.test.tsx @@ -0,0 +1,21 @@ +import { describe, it, expect, vi } from "vitest" +import { render, screen } from "@testing-library/react" +import userEvent from "@testing-library/user-event" +import { Input } from "./input.tsx" + +describe("Input", () => { + it("is a labelled text box that reports what is typed", async () => { + const onValueChange = vi.fn() + render() + + await userEvent.type(screen.getByRole("textbox", { name: "Filter" }), "x") + + expect(onValueChange).toHaveBeenCalledWith("x", expect.anything()) + }) + + it("keeps the caller's classes next to its own", () => { + render() + + expect(screen.getByRole("textbox", { name: "Filter" })).toHaveClass("font-mono", "rounded-lg") + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/ui/input.tsx b/packages/system/dashboard/images/console/apps/console/src/components/ui/input.tsx new file mode 100644 index 000000000..b079e6b35 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/ui/input.tsx @@ -0,0 +1,17 @@ +import { Input as InputPrimitive } from "@base-ui/react/input" +import { cn } from "@cozystack/ui" + +function Input({ className, ...props }: InputPrimitive.Props) { + return ( + + ) +} + +export { Input } From b88ad09678a234ff351173a1cc0674b9721d795f Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 18:47:25 +0700 Subject: [PATCH 074/518] fix(dashboard): keep the log console on screen when it has no lines With no matching lines, while loading, and while Follow waits for the first line, the dark log console vanished and left a line of small grey text or a bare spinner, so the page jumped and an empty result looked like a broken view. The console frame now keeps a fixed height in every state and says inside it what is going on. The pod and container pickers move to the console Select and the filter to the console Input, so the toolbar is keyboard and screen reader friendly and a long pod name truncates instead of widening it. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/EmptyState.test.tsx | 17 +++++ .../components/observability/EmptyState.tsx | 25 ++++++++ .../observability/LogLines.test.tsx | 35 ++++++++--- .../src/components/observability/LogLines.tsx | 44 +++++++------ .../observability/LogsView.test.tsx | 50 ++++++++++++++- .../src/components/observability/LogsView.tsx | 62 +++++++++++++------ 6 files changed, 184 insertions(+), 49 deletions(-) create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.tsx diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.test.tsx new file mode 100644 index 000000000..8872f4d43 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.test.tsx @@ -0,0 +1,17 @@ +import { describe, it, expect } from "vitest" +import { render, screen } from "@testing-library/react" +import { EmptyState } from "./EmptyState.tsx" + +describe("EmptyState", () => { + it("announces its message as a status", () => { + render() + + expect(screen.getByRole("status")).toHaveTextContent("No data for this range.Pick a longer range.") + }) + + it("marks itself busy while something is loading", () => { + render() + + expect(screen.getByRole("status")).toHaveAttribute("aria-busy", "true") + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.tsx new file mode 100644 index 000000000..0cf58224b --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.tsx @@ -0,0 +1,25 @@ +import type { ReactNode } from "react" +import { cn, Spinner } from "@cozystack/ui" + +interface EmptyStateProps { + title: ReactNode + hint?: ReactNode + busy?: boolean + tone?: "light" | "dark" + className?: string +} + +export function EmptyState({ title, hint, busy, tone = "light", className }: EmptyStateProps) { + const dark = tone === "dark" + return ( +
    + {busy && } +

    {title}

    + {hint &&

    {hint}

    } +
    + ) +} diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx index 7110fe464..dd8c503eb 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx @@ -10,40 +10,59 @@ function sized(list: HTMLElement, scrollHeight: number) { } describe("LogLines", () => { + it("keeps the console frame and shows the placeholder inside it when there are no lines", () => { + render() + + const frame = screen.getByRole("region", { name: "Log lines" }) + expect(frame).toHaveTextContent("Waiting for new log lines…") + expect(screen.queryByRole("list")).toBeNull() + }) + + it("puts the lines in the same frame once they arrive", () => { + const { rerender } = render() + const frame = screen.getByRole("region", { name: "Log lines" }) + + rerender() + + expect(screen.getByRole("region", { name: "Log lines" })).toBe(frame) + expect(frame).toContainElement(screen.getByRole("list")) + expect(frame).not.toHaveTextContent("Waiting") + }) + it("keeps following the newest line while the reader is at the bottom", () => { - const { rerender } = render() + const { rerender } = render() const list = screen.getByRole("list") sized(list, 500) - rerender() + rerender() expect(list.scrollTop).toBe(500) }) it("leaves the position alone once the reader scrolls up", () => { - const { rerender } = render() + const { rerender } = render() const list = screen.getByRole("list") sized(list, 500) list.scrollTop = 120 fireEvent.scroll(list) - rerender() + rerender() expect(list.scrollTop).toBe(120) }) it("follows again once the list is cleared for a new run", () => { - const { rerender } = render() + const { rerender } = render() const first = screen.getByRole("list") sized(first, 500) first.scrollTop = 120 fireEvent.scroll(first) - rerender() - rerender() + rerender() + rerender() const second = screen.getByRole("list") sized(second, 300) - rerender() + rerender() expect(second.scrollTop).toBe(300) }) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx index 452faac72..14b941ab2 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx @@ -1,10 +1,10 @@ -import { useEffect, useRef } from "react" +import { useEffect, useRef, type ReactNode } from "react" import { CONTAINER_FIELD, POD_FIELD } from "../../lib/observability/logs.ts" import type { LogEntry } from "../../lib/observability/proxy.ts" const BOTTOM_SLACK_PX = 24 -export function LogLines({ entries }: { entries: LogEntry[] }) { +export function LogLines({ entries, placeholder }: { entries: LogEntry[]; placeholder: ReactNode }) { const ref = useRef(null) const pinned = useRef(true) @@ -13,23 +13,31 @@ export function LogLines({ entries }: { entries: LogEntry[] }) { else if (ref.current && pinned.current) ref.current.scrollTop = ref.current.scrollHeight }, [entries]) - if (entries.length === 0) return

    No log lines for this range and filter.

    return ( -
      { - const el = e.currentTarget - pinned.current = el.scrollHeight - el.scrollTop - el.clientHeight <= BOTTOM_SLACK_PX - }} - className="max-h-[60vh] overflow-auto rounded-md bg-slate-950 p-3 font-mono text-xs leading-5 text-slate-100" +
      - {entries.map((e, i) => ( -
    1. - {new Date(e._time).toLocaleTimeString()}{" "} - {`${e[POD_FIELD] ?? ""}/${e[CONTAINER_FIELD] ?? ""}`}{" "} - {e._msg} -
    2. - ))} -
    + {entries.length === 0 ? ( + placeholder + ) : ( +
      { + const el = e.currentTarget + pinned.current = el.scrollHeight - el.scrollTop - el.clientHeight <= BOTTOM_SLACK_PX + }} + className="h-full overflow-auto p-3" + > + {entries.map((e, i) => ( +
    1. + {new Date(e._time).toLocaleTimeString()}{" "} + {`${e[POD_FIELD] ?? ""}/${e[CONTAINER_FIELD] ?? ""}`}{" "} + {e._msg} +
    2. + ))} +
    + )} + ) } diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index 526efeb23..31c6fac0e 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -1,5 +1,5 @@ import { describe, it, expect, vi } from "vitest" -import { screen, waitFor } from "@testing-library/react" +import { screen, waitFor, within } from "@testing-library/react" import userEvent from "@testing-library/user-event" import { K8sClient } from "@cozystack/k8s-client" import { renderWithK8sProvider } from "../../test-utils/render.tsx" @@ -47,9 +47,9 @@ describe("LogsView", () => { it("narrows to the picked pod with a stream filter", async () => { const { client, raw } = clientWith(defaultHandler) renderWithK8sProvider(, { client }) - await screen.findByRole("option", { name: "db-1" }) - await userEvent.selectOptions(screen.getByLabelText("Pod"), "db-1") + await userEvent.click(screen.getByRole("combobox", { name: "Pod" })) + await userEvent.click(await screen.findByRole("option", { name: "db-1" })) await waitFor(() => expect( @@ -58,6 +58,50 @@ describe("LogsView", () => { ) }) + it("offers All next to every pod and starts on it", async () => { + const { client } = clientWith(defaultHandler) + renderWithK8sProvider(, { client }) + const pod = screen.getByRole("combobox", { name: "Pod" }) + expect(pod).toHaveTextContent("All") + + await userEvent.click(pod) + + expect(await screen.findByRole("option", { name: "db-1" })).toBeInTheDocument() + expect(screen.getAllByRole("option").map((o) => o.textContent)).toEqual(["All", "db-0", "db-1"]) + }) + + it("keeps the log console and says so inside it when nothing matches", async () => { + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("query") ? new Response("") : defaultHandler(endpoint, params), + ) + + renderWithK8sProvider(, { client }) + + const frame = screen.getByRole("region", { name: "Log lines" }) + expect(await within(frame).findByText("No log lines for this range and filter.")).toBeInTheDocument() + }) + + it("shows loading inside the log console", () => { + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("query") ? new Promise(() => {}) : defaultHandler(endpoint, params), + ) + + renderWithK8sProvider(, { client }) + + const frame = screen.getByRole("region", { name: "Log lines" }) + expect(within(frame).getByRole("status")).toHaveAttribute("aria-busy", "true") + }) + + it("says it is waiting while Follow has no line yet", async () => { + const { client } = clientWith(defaultHandler) + renderWithK8sProvider(, { client }) + + await userEvent.click(screen.getByRole("button", { name: "Follow" })) + + const frame = screen.getByRole("region", { name: "Log lines" }) + expect(within(frame).getByText("Waiting for new log lines…")).toBeInTheDocument() + }) + it("applies the LogsQL filter on submit", async () => { const { client, raw } = clientWith(defaultHandler) renderWithK8sProvider(, { client }) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index a27ca41c5..d2a000010 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -1,7 +1,9 @@ import { useState } from "react" import { useQuery } from "@tanstack/react-query" import { useK8sClient } from "@cozystack/k8s-client" -import { Button, Spinner } from "@cozystack/ui" +import { Button } from "@cozystack/ui" +import { Input } from "../ui/input.tsx" +import { SelectItem, SelectLabel, SelectPopup, SelectRoot, SelectTrigger, SelectValue } from "../ui/select.tsx" import { CONTAINER_FIELD, LOG_RANGES, @@ -20,6 +22,7 @@ import { useLogTail } from "../../lib/observability/use-log-tail.ts" import { ProxyNotice } from "./ProxyNotice.tsx" import { RangePicker } from "./RangePicker.tsx" import { LogLines } from "./LogLines.tsx" +import { EmptyState } from "./EmptyState.tsx" const HISTORY_LIMIT = 500 const FOLLOW_CAP = 1000 @@ -30,27 +33,42 @@ function windowFor(range: LogRangeKey) { return { start: new Date(end - LOG_RANGES[range] * 1000).toISOString(), end: new Date(end).toISOString() } } +const ALL = "" + function Picker({ label, value, options, onChange }: { label: string; value: string; options: string[]; onChange: (v: string) => void }) { + const items = [{ value: ALL, label: "All" }, ...options.map((o) => ({ value: o, label: o }))] return ( - + + ) } +function placeholderFor({ follow, stopped, loading, failed }: { follow: boolean; stopped: boolean; loading: boolean; failed: boolean }) { + if (follow) { + return stopped ? ( + + ) : ( + + ) + } + if (loading) return + if (failed) return + return +} + export function LogsView({ scope }: { scope: ObservabilityScope }) { const client = useK8sClient() const root = logsRoot(scope) @@ -107,12 +125,12 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) { setFilter(draft) }} > - setDraft(e.target.value)} + onValueChange={setDraft} placeholder='LogsQL filter, e.g. error or "connection refused"' - className="w-full rounded-md border border-slate-200 px-2.5 py-1.5 font-mono text-xs" + className="font-mono" /> { setPod(v); setContainer("") }} /> @@ -129,7 +147,11 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) {

    )} {error ? : null} - {!follow && history.isLoading ? : } + ) } From f5ebb43c4bdce908887029568065eb27785ff833 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 18:48:43 +0700 Subject: [PATCH 075/518] fix(dashboard): match the range picker to the log toolbar controls Next to the new Select and Input the range picker was shorter, lighter bordered and had no focus ring, so the toolbar looked uneven and a keyboard user could not see which range had focus. It now shares their height, border and focus ring, and its buttons are grouped under one accessible name. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/RangePicker.test.tsx | 23 +++++++++++++++++++ .../components/observability/RangePicker.tsx | 4 ++-- 2 files changed, 25 insertions(+), 2 deletions(-) create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.test.tsx diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.test.tsx new file mode 100644 index 000000000..9787963b2 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.test.tsx @@ -0,0 +1,23 @@ +import { describe, it, expect, vi } from "vitest" +import { render, screen, within } from "@testing-library/react" +import userEvent from "@testing-library/user-event" +import { RangePicker } from "./RangePicker.tsx" + +describe("RangePicker", () => { + it("groups the ranges under one name and marks the chosen one pressed", () => { + render( {}} />) + + const group = screen.getByRole("group", { name: "Time range" }) + expect(within(group).getByRole("button", { name: "6h" })).toHaveAttribute("aria-pressed", "true") + expect(within(group).getByRole("button", { name: "1h" })).toHaveAttribute("aria-pressed", "false") + }) + + it("reports the range clicked", async () => { + const onChange = vi.fn() + render() + + await userEvent.click(screen.getByRole("button", { name: "1h" })) + + expect(onChange).toHaveBeenCalledWith("1h") + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.tsx index 22030c706..95d245176 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/RangePicker.tsx @@ -10,7 +10,7 @@ export function RangePicker({ onChange: (value: K) => void }) { return ( -
    +
    {options.map((key) => ( @@ -150,7 +153,7 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) {
    ) From 9498c9bff520d73fe200f5043bc7efce9eb901ef Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 19:05:42 +0700 Subject: [PATCH 082/518] fix(dashboard): keep the proxy's own advice on a limit refusal The notice appended "Choose a shorter range." to every 422, so a message that already ended in "use a shorter range" said it twice and one that ended in "raise step" got advice that contradicted it. The console's own advice is now added only when the proxy gave none. Assisted-by: LLM Signed-off-by: ZverGuy --- .../src/components/observability/ProxyNotice.test.tsx | 9 +++++++++ .../console/src/components/observability/ProxyNotice.tsx | 3 ++- 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx index fd001bcae..6e85ced33 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx @@ -29,6 +29,15 @@ describe("ProxyNotice", () => { expect(screen.getByRole("alert")).toHaveTextContent(text) }) + it.each([ + "the application has more than 50 pods in this window; use a shorter range", + "1200 points per series exceed the limit of 1000; raise step", + ])("does not add advice to a limit message that already carries its own: %s", (message) => { + render() + + expect(screen.getByRole("alert").textContent).toBe(`${message}.`) + }) + it("shows the server message for a rejected query", () => { render() diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx index a60b831b0..ee68e4633 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx @@ -20,7 +20,8 @@ function describe(what: ProxyNoticeProps["what"], error: unknown): { quiet: bool case "forbidden": return { quiet: false, text: `You do not have permission to read ${lower} here.` } case "limit": - return { quiet: false, text: `${error.message}. Choose a shorter range.` } + // The proxy puts its own remedy after "; " (e.g. "use a shorter range", "raise step"). + return { quiet: false, text: error.message.includes("; ") ? `${error.message}.` : `${error.message}. Choose a shorter range.` } case "rate-limited": return { quiet: false, text: "Too many requests. Try again in a moment." } case "timeout": From 76441871b8c3ae73f40ecf29fd454a468f1e0573 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 19:06:59 +0700 Subject: [PATCH 083/518] fix(dashboard): keep caller headers of any shape in the k8s client The client spread init.headers as a plain object, so a Headers instance or a list of pairs passed by a caller was silently dropped and the request went out without them. Headers are now built with the Headers constructor, which accepts every shape fetch does. Assisted-by: LLM Signed-off-by: ZverGuy --- .../k8s-client/client.headers.test.ts | 31 +++++++++++++++++++ .../__tests__/k8s-client/client.raw.test.ts | 20 +++++++++++- .../console/packages/k8s-client/src/client.ts | 14 +++------ 3 files changed, 55 insertions(+), 10 deletions(-) create mode 100644 packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.headers.test.ts diff --git a/packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.headers.test.ts b/packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.headers.test.ts new file mode 100644 index 000000000..37985fca5 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.headers.test.ts @@ -0,0 +1,31 @@ +import { describe, it, expect, vi, afterEach } from "vitest" +import { K8sClient } from "@cozystack/k8s-client" + +afterEach(() => { + vi.unstubAllGlobals() +}) + +describe("K8sClient request headers", () => { + it("lets a method's own content type replace the JSON default", async () => { + const fetchMock = vi.fn(async () => new Response("{}")) + vi.stubGlobal("fetch", fetchMock) + const client = new K8sClient({ getToken: async () => "t0k" }) + + await client.patch("apps", "v1", "deployments", "web", { spec: {} }, "default", "merge") + + const [, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] + const headers = new Headers(init.headers) + expect(headers.get("Content-Type")).toBe("application/merge-patch+json") + expect(headers.get("Authorization")).toBe("Bearer t0k") + }) + + it("sends JSON by default", async () => { + const fetchMock = vi.fn(async () => new Response("{}")) + vi.stubGlobal("fetch", fetchMock) + + await new K8sClient().getApiGroups() + + const [, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] + expect(new Headers(init.headers).get("Content-Type")).toBe("application/json") + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.raw.test.ts b/packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.raw.test.ts index ab6c056f5..0ac0d75ae 100644 --- a/packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.raw.test.ts +++ b/packages/system/dashboard/images/console/apps/console/src/__tests__/k8s-client/client.raw.test.ts @@ -24,7 +24,25 @@ describe("K8sClient.raw", () => { const [url, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] expect(url).toBe("/base/apis/x") - expect(init.headers).toEqual({ Accept: "application/json", Authorization: "Bearer t0k" }) + const headers = new Headers(init.headers) + expect(headers.get("Accept")).toBe("application/json") + expect(headers.get("Authorization")).toBe("Bearer t0k") + }) + + it.each([ + ["a Headers instance", new Headers({ Accept: "application/x-ndjson" })], + ["a tuple list", [["Accept", "application/x-ndjson"]] as [string, string][]], + ])("keeps caller headers passed as %s", async (_, given) => { + const fetchMock = vi.fn(async () => new Response("{}")) + vi.stubGlobal("fetch", fetchMock) + const client = new K8sClient({ getToken: async () => "t0k" }) + + await client.raw("/x", { headers: given }) + + const [, init] = fetchMock.mock.calls[0] as unknown as [string, RequestInit] + const headers = new Headers(init.headers) + expect(headers.get("Accept")).toBe("application/x-ndjson") + expect(headers.get("Authorization")).toBe("Bearer t0k") }) it("routes a 401 through the client's unauthorized handler", async () => { diff --git a/packages/system/dashboard/images/console/packages/k8s-client/src/client.ts b/packages/system/dashboard/images/console/packages/k8s-client/src/client.ts index 04c528760..64af1131d 100644 --- a/packages/system/dashboard/images/console/packages/k8s-client/src/client.ts +++ b/packages/system/dashboard/images/console/packages/k8s-client/src/client.ts @@ -50,14 +50,12 @@ export class K8sClient { } private async request(path: string, init?: RequestInit): Promise { - const headers: Record = { - "Content-Type": "application/json", - ...(init?.headers as Record), - } + const headers = new Headers(init?.headers) + if (!headers.has("Content-Type")) headers.set("Content-Type", "application/json") if (this.getToken) { const token = await this.getToken() - headers["Authorization"] = `Bearer ${token}` + headers.set("Authorization", `Bearer ${token}`) } let res: Response @@ -247,11 +245,9 @@ export class K8sClient { * Kubernetes JSON, such as the observability proxy's streamed NDJSON. */ async raw(path: string, init?: RequestInit): Promise { - const headers: Record = { - ...(init?.headers as Record), - } + const headers = new Headers(init?.headers) if (this.getToken) { - headers["Authorization"] = `Bearer ${await this.getToken()}` + headers.set("Authorization", `Bearer ${await this.getToken()}`) } const res = await fetch(`${this.baseUrl}${path}`, { ...init, headers }) if (res.status === 401) this.handleUnauthorized() From 2a473086cf6cd1d245e014ed56ed44f9594038cd Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 19:11:05 +0700 Subject: [PATCH 084/518] fix(dashboard): advise a shorter range unless the proxy already did Any "; " in a 422 message was taken as the proxy's own remedy, but the proxy passes vmselect's 422 bodies through, and their advice (raise the -search.max* flags) is for operators. Those lost the one thing a tenant can do. Only the remedies the proxy itself appends now replace the console's advice. Assisted-by: LLM Signed-off-by: ZverGuy --- .../src/components/observability/ProxyNotice.test.tsx | 7 +++++++ .../src/components/observability/ProxyNotice.tsx | 10 ++++++++-- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx index 6e85ced33..c4b4e89c1 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.test.tsx @@ -38,6 +38,13 @@ describe("ProxyNotice", () => { expect(screen.getByRole("alert").textContent).toBe(`${message}.`) }) + it("still advises a shorter range for a store limit whose own advice is for operators", () => { + const message = "the number of matching timeseries exceeds 300000; either narrow down the search or increase -search.max* command-line flag values" + render() + + expect(screen.getByRole("alert").textContent).toBe(`${message}. Choose a shorter range.`) + }) + it("shows the server message for a rejected query", () => { render() diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx index ee68e4633..1fb30c8ff 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/ProxyNotice.tsx @@ -5,6 +5,10 @@ interface ProxyNoticeProps { error: unknown } +// Remedies the observability proxy itself appends; vmselect's 422 text carries +// advice for operators (raise -search.max* flags), so it still gets ours. +const PROXY_REMEDIES = ["; use a shorter range", "; raise step"] + function describe(what: ProxyNoticeProps["what"], error: unknown): { quiet: boolean; text: string } { const lower = what.toLowerCase() if (!(error instanceof ProxyError)) { @@ -20,8 +24,10 @@ function describe(what: ProxyNoticeProps["what"], error: unknown): { quiet: bool case "forbidden": return { quiet: false, text: `You do not have permission to read ${lower} here.` } case "limit": - // The proxy puts its own remedy after "; " (e.g. "use a shorter range", "raise step"). - return { quiet: false, text: error.message.includes("; ") ? `${error.message}.` : `${error.message}. Choose a shorter range.` } + return { + quiet: false, + text: PROXY_REMEDIES.some((r) => error.message.endsWith(r)) ? `${error.message}.` : `${error.message}. Choose a shorter range.`, + } case "rate-limited": return { quiet: false, text: "Too many requests. Try again in a moment." } case "timeout": From fe6968be7f2530f37738bf767e0cd0a3e54955ff Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 14:45:42 +0300 Subject: [PATCH 085/518] fix(postgres): fail a backup job as soon as its cnpg Backup fails A failed cnpg.io/Backup kept its BackupJob Running for 30 minutes as a "transient" failure, on the premise that CloudNativePG may move a Backup out of the failed phase. It does not: its backup controller returns early on any Backup already failed or completed, so the wait only delayed the verdict and hid it behind a misleading reason. A failed Backup now fails the BackupJob at once with the error CNPG recorded, pointing to the plugin-barman-cloud sidecar log when that error is only the plugin's exit status. A Backup in any other phase keeps the BackupJob waiting as before, now with a WaitingForCNPGBackup condition naming the Backup and its phase instead of no condition at all. It is still not capped: CloudNativePG keeps a healthy Backup pending while another backup of the same Cluster runs or while it restarts one, and a fixed deadline would fail those. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 76 ++++----- .../cnpgstrategy_controller_test.go | 152 +++++++++++++++++- 2 files changed, 188 insertions(+), 40 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 99e6b03f6..27e50f4d8 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -175,13 +175,11 @@ const ( // against an unreachable target reproduces the same FATAL. cnpgRecoveryMaxInspectPods = 5 - // Cap on the wall-clock time a BackupJob can spend observing a - // cnpg.io/Backup stuck in phase=failed before the driver gives up and - // marks the BackupJob Failed. CNPG can transition out of "failed" on - // internal retries (e.g. transient instance-manager restart), so we - // don't fail on the first observation. But without this cap, a - // permanently-broken Backup would pin the BackupJob in Running forever, - // blocking later runs and wedging the Plan-controller queue. + // Cap on the wall-clock time a BackupJob waits for the barman-cloud + // plugin rollout before the cnpg.io/Backup exists. A Backup that exists + // is not capped: CloudNativePG keeps a healthy one pending while another + // backup of the same Cluster runs, or while it restarts one on another + // instance, so no fixed deadline tells those apart from a stuck one. cnpgDefaultBackupDeadline = 30 * time.Minute cnpgClusterHealthyPhase = "Cluster in healthy state" @@ -453,44 +451,46 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp return ctrl.Result{}, nil case cnpgBackupPhaseFailed: - // CNPG can transition the same Backup CR back out of "failed" when it - // retries internally (e.g., after a transient instance-manager - // restart), so we don't fail the BackupJob on the first observation. - // But without a cap a permanently-broken Backup would pin the - // BackupJob in Running forever; once StartedAt + cnpgDefaultBackupDeadline - // has elapsed and the cnpg.io/Backup is still failed, we give up. - if cnpgBackupDeadlineExceeded(j.Status.StartedAt) { - final := message - if final == "" { - final = fmt.Sprintf( - "cnpg.io Backup remained in phase=failed past %s deadline", - cnpgDefaultBackupDeadline) - } - return r.markBackupJobFailed(ctx, j, final) - } - if message == "" { - message = "cnpg.io Backup reported phase=failed; awaiting recovery or retry" - } - apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ - Type: "Ready", - Status: metav1.ConditionFalse, - Reason: "CNPGBackupTransientFailure", - Message: message, - }) - if err := r.Status().Update(ctx, j); err != nil { - return ctrl.Result{}, err - } - return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil + // CloudNativePG does not reconcile a Backup out of a terminal phase, + // so waiting on a failed one cannot change the outcome. + return r.markBackupJobFailed(ctx, j, cnpgBackupFailedMessage(cnpgBackup, message)) default: + waiting := fmt.Sprintf("waiting for cnpg.io/Backup %s/%s, phase %q", + cnpgBackup.Namespace, cnpgBackup.Name, phase) + if apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ + Type: "Ready", + Status: metav1.ConditionFalse, + Reason: "WaitingForCNPGBackup", + Message: waiting, + }) { + if err := r.Status().Update(ctx, j); err != nil { + return ctrl.Result{}, err + } + } return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil } } +// cnpgBackupFailedMessage carries the error CloudNativePG recorded. When the +// plugin run itself failed that is only its exit status, and the storage +// error behind it is logged by the plugin-barman-cloud sidecar. +func cnpgBackupFailedMessage(b *cnpgtypes.Backup, errText string) string { + if errText == "" { + errText = "no error recorded" + } + message := fmt.Sprintf("cnpg.io/Backup %s/%s failed: %s", b.Namespace, b.Name, errText) + if strings.Contains(errText, "exit status") { + message += "; the cause is in the log of the plugin-barman-cloud sidecar of the instance that ran it" + } + return message +} + // cnpgBackupDeadlineExceeded reports whether enough wall-clock time has -// elapsed since the BackupJob started that we should stop polling a stuck -// cnpg.io/Backup and fail the run. Returns false when StartedAt is nil so -// the very first reconcile (which sets StartedAt) does not trip the gate. +// elapsed since the BackupJob started that we should stop waiting for the +// barman-cloud plugin rollout and fail the run. Returns false when +// StartedAt is nil so the very first reconcile (which sets StartedAt) does +// not trip the gate. func cnpgBackupDeadlineExceeded(startedAt *metav1.Time) bool { if startedAt == nil { return false diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 697843d9e..551209fac 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2332,6 +2332,154 @@ func TestReconcileCNPG_StartedAtReturnsEarlyToAvoidStaleResourceVersion(t *testi } } +// CloudNativePG never reconciles a Backup out of a terminal phase (the +// backup controller returns early on failed and completed), so a failed +// Backup fails the BackupJob at once. Any other phase keeps the BackupJob +// waiting past the backup deadline: CNPG keeps a healthy Backup pending +// while another backup of the Cluster runs or while it restarts one. +func TestReconcileCNPG_BackupPhaseOutcome(t *testing.T) { + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + strategyGroup := strategyv1alpha1.GroupVersion.Group + const ns, appName = "tenant", "pg" + clusterName := cnpgClusterNameForApp(appName) + expired := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)) + + cases := []struct { + name string + phase string + errText string + startedAt time.Time + wantPhase backupsv1alpha1.BackupJobPhase + wantMessage []string + notInMessage []string + }{ + {name: "failed fails at once", phase: cnpgBackupPhaseFailed, errText: "rpc error: code = Unknown desc = exit status 1", startedAt: time.Now(), + wantPhase: backupsv1alpha1.BackupJobPhaseFailed, wantMessage: []string{"bj-cnpg", "exit status 1", "plugin-barman-cloud"}}, + {name: "failed without an error text fails at once", phase: cnpgBackupPhaseFailed, startedAt: time.Now(), + wantPhase: backupsv1alpha1.BackupJobPhaseFailed, wantMessage: []string{"bj-cnpg", "no error recorded"}, notInMessage: []string{"plugin-barman-cloud"}}, + {name: "failed before the plugin ran points nowhere else", phase: cnpgBackupPhaseFailed, errText: "cannot backup a hibernated cluster", startedAt: time.Now(), + wantPhase: backupsv1alpha1.BackupJobPhaseFailed, wantMessage: []string{"bj-cnpg", "hibernated"}, notInMessage: []string{"plugin-barman-cloud"}}, + {name: "pending keeps waiting", phase: "pending", startedAt: time.Now(), + wantPhase: backupsv1alpha1.BackupJobPhaseRunning, wantMessage: []string{"bj-cnpg", "pending"}}, + {name: "pending is not capped by the backup deadline", phase: "pending", startedAt: expired, + wantPhase: backupsv1alpha1.BackupJobPhaseRunning, wantMessage: []string{"bj-cnpg", "pending"}}, + {name: "no phase yet keeps waiting", phase: "", startedAt: expired, + wantPhase: backupsv1alpha1.BackupJobPhaseRunning, wantMessage: []string{"bj-cnpg"}}, + {name: "started keeps running", phase: "started", startedAt: expired, + wantPhase: backupsv1alpha1.BackupJobPhaseRunning, wantMessage: []string{"bj-cnpg", "started"}}, + {name: "walArchivingFailing keeps waiting", phase: "walArchivingFailing", startedAt: expired, + wantPhase: backupsv1alpha1.BackupJobPhaseRunning, wantMessage: []string{"bj-cnpg", "walArchivingFailing"}}, + {name: "completed succeeds", phase: cnpgBackupPhaseComplete, startedAt: expired, + wantPhase: backupsv1alpha1.BackupJobPhaseSucceeded}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + startedAt := metav1.NewTime(tc.startedAt) + job := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: ns, Name: "bj"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: appName}, + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt}, + } + strategy := &strategyv1alpha1.CNPG{ + ObjectMeta: metav1.ObjectMeta{Name: "cnpg-strategy"}, + Spec: strategyv1alpha1.CNPGSpec{ + Template: strategyv1alpha1.CNPGTemplate{ + BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://bucket/"}, + }, + }, + } + cluster := &cnpgtypes.Cluster{ + ObjectMeta: metav1.ObjectMeta{Namespace: ns, Name: clusterName, UID: "cluster-uid"}, + } + cnpgBackup := &cnpgtypes.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: ns, Name: "bj-cnpg", Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: job.Name, + backupsv1alpha1.OwningJobNamespaceLabel: job.Namespace, + }}, + Spec: cnpgtypes.BackupSpec{Cluster: cnpgtypes.ClusterReference{Name: clusterName}}, + Status: cnpgtypes.BackupStatus{Phase: tc.phase, Error: tc.errText}, + } + c := newCNPGStrategyTestClient(t, job, strategy, newPostgresApp(appName, ns), cluster, cnpgBackup) + r := &BackupJobReconciler{Client: c, Interface: cnpgDynamicFor(t, cluster)} + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{APIGroup: &strategyGroup, Kind: strategyv1alpha1.CNPGStrategyKind, Name: strategy.Name}, + } + + j := &backupsv1alpha1.BackupJob{} + if err := c.Get(context.Background(), client.ObjectKeyFromObject(job), j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + res, err := r.reconcileCNPG(context.Background(), j, resolved) + if err != nil { + t.Fatalf("reconcileCNPG: %v", err) + } + if j.Status.Phase != tc.wantPhase { + t.Fatalf("BackupJob phase = %q (%s), want %q", j.Status.Phase, j.Status.Message, tc.wantPhase) + } + + message := j.Status.Message + if tc.wantPhase == backupsv1alpha1.BackupJobPhaseRunning { + if res.RequeueAfter == 0 { + t.Errorf("expected a requeue while the cnpg.io/Backup is in progress, got %+v", res) + } + persisted := &backupsv1alpha1.BackupJob{} + if err := c.Get(context.Background(), client.ObjectKeyFromObject(job), persisted); err != nil { + t.Fatalf("get BackupJob after reconcile: %v", err) + } + ready := apimeta.FindStatusCondition(persisted.Status.Conditions, "Ready") + if ready == nil || ready.Status != metav1.ConditionFalse || ready.Reason != "WaitingForCNPGBackup" { + t.Fatalf("expected Ready=False/WaitingForCNPGBackup while waiting, got %+v", ready) + } + message = ready.Message + + // A pass that observes the same phase again must not write the + // BackupJob status: the driver polls every few seconds. + statusWrites := 0 + r.Client = interceptor.NewClient(c.(client.WithWatch), interceptor.Funcs{ + SubResourceUpdate: func(ctx context.Context, cl client.Client, sub string, obj client.Object, opts ...client.SubResourceUpdateOption) error { + statusWrites++ + return cl.SubResource(sub).Update(ctx, obj, opts...) + }, + SubResourcePatch: func(ctx context.Context, cl client.Client, sub string, obj client.Object, patch client.Patch, opts ...client.SubResourcePatchOption) error { + statusWrites++ + return cl.SubResource(sub).Patch(ctx, obj, patch, opts...) + }, + }) + if _, err := r.reconcileCNPG(context.Background(), persisted, resolved); err != nil { + t.Fatalf("second reconcileCNPG: %v", err) + } + if statusWrites != 0 { + t.Errorf("an unchanged phase wrote the BackupJob status %d time(s)", statusWrites) + } + } + for _, want := range tc.wantMessage { + if !strings.Contains(message, want) { + t.Errorf("message %q does not mention %q", message, want) + } + } + for _, unwanted := range tc.notInMessage { + if strings.Contains(message, unwanted) { + t.Errorf("message %q mentions %q", message, unwanted) + } + } + + artifacts := &backupsv1alpha1.BackupList{} + if err := c.List(context.Background(), artifacts, client.InNamespace(ns)); err != nil { + t.Fatalf("list Backups: %v", err) + } + wantArtifacts := 0 + if tc.wantPhase == backupsv1alpha1.BackupJobPhaseSucceeded { + wantArtifacts = 1 + } + if len(artifacts.Items) != wantArtifacts { + t.Errorf("got %d Backup artifacts, want %d", len(artifacts.Items), wantArtifacts) + } + }) + } +} + // cnpgRolloutFixture drives reconcileCNPG against one Postgres app whose // Cluster and instance pods a test swaps between passes, standing in for // CNPG rolling the instances. @@ -2516,8 +2664,8 @@ func TestReconcileCNPG_WaitsForPluginRolloutBeforeBackup(t *testing.T) { if !apimeta.IsStatusConditionTrue(j.Status.Conditions, backupCondPluginRolledOut) { t.Errorf("%s condition = %+v, want True", backupCondPluginRolledOut, apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut)) } - if cond := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); cond != nil { - t.Errorf("Ready condition = %+v left over from the wait while the backup runs", cond) + if cond := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); cond == nil || cond.Reason != "WaitingForCNPGBackup" { + t.Errorf("Ready condition = %+v, want the wait on the cnpg.io/Backup rather than one left over from the rollout", cond) } } From 6e36385e00fcb9cc8b24e19ec8e9a5e11056b383 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Tue, 27 Oct 2026 18:25:19 +0300 Subject: [PATCH 086/518] feat(api): expose HelmRelease failure state on Application status Tenants usually read Applications but not the HelmReleases behind them, and only Ready and Released were copied over. A client could therefore see that a release was failing but not how often it had failed, nor whether helm-controller had stopped retrying it. Copy the Stalled condition, which helm-controller sets once it stops reconciling a release (remediation retries exhausted, rollback target missing, invalid CEL, access denied), and surface the install and upgrade failure counters with the last attempted release action. Stalled is dropped when it was observed for an older generation than the HelmRelease's current one: the Application carries no generation, so after a spec update a client would otherwise read the previous spec's failure as terminal for the new values. With the RetryOnFailure policy the platform generates today, a failed release is retried rather than stalled, so the counters are the signal. That does not hold for every state: a release missing from storage while an install failure count is retained still stalls, and a release on a remediation policy stalls once its retries run out. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- pkg/apis/apps/v1alpha1/types.go | 17 +++ .../apps/v1alpha1/applicationstatus.go | 38 ++++++ pkg/generated/openapi/zz_generated.openapi.go | 21 +++ pkg/registry/apps/application/rest.go | 14 +- .../application/rest_release_failure_test.go | 123 ++++++++++++++++++ 5 files changed, 211 insertions(+), 2 deletions(-) create mode 100644 pkg/registry/apps/application/rest_release_failure_test.go diff --git a/pkg/apis/apps/v1alpha1/types.go b/pkg/apis/apps/v1alpha1/types.go index 15bab146f..13270b0f1 100644 --- a/pkg/apis/apps/v1alpha1/types.go +++ b/pkg/apis/apps/v1alpha1/types.go @@ -55,6 +55,23 @@ type ApplicationStatus struct { // ExternalIPsCount holds the number of LoadBalancer services with assigned external IPs for Tenant applications. // +optional ExternalIPsCount int32 `json:"externalIPsCount,omitempty"` + // InstallFailures is the install failure count of the underlying HelmRelease. + // It counts only attempts that recorded a Helm release, so a failure before + // Helm records one, such as a chart that fails to render, shows only as + // Ready=False; a failure to apply the rendered resources is counted. It + // resets when the desired state (generation, chart version, values) changes, + // and on success only under the RetryOnFailure strategy, so a Ready release + // may carry it. + // +optional + InstallFailures int64 `json:"installFailures,omitempty"` + // UpgradeFailures is the upgrade failure count of the underlying HelmRelease, + // with the same counting and reset rules as InstallFailures. + // +optional + UpgradeFailures int64 `json:"upgradeFailures,omitempty"` + // LastAttemptedReleaseAction is the last release action (install or upgrade) + // the underlying HelmRelease attempted. + // +optional + LastAttemptedReleaseAction string `json:"lastAttemptedReleaseAction,omitempty"` } // SchedulingClass returns the scheduling class requested by this Application. diff --git a/pkg/generated/applyconfiguration/apps/v1alpha1/applicationstatus.go b/pkg/generated/applyconfiguration/apps/v1alpha1/applicationstatus.go index e20e08a51..77db6b0e7 100644 --- a/pkg/generated/applyconfiguration/apps/v1alpha1/applicationstatus.go +++ b/pkg/generated/applyconfiguration/apps/v1alpha1/applicationstatus.go @@ -34,6 +34,20 @@ type ApplicationStatusApplyConfiguration struct { Namespace *string `json:"namespace,omitempty"` // ExternalIPsCount holds the number of LoadBalancer services with assigned external IPs for Tenant applications. ExternalIPsCount *int32 `json:"externalIPsCount,omitempty"` + // InstallFailures is the install failure count of the underlying HelmRelease. + // It counts only attempts that recorded a Helm release, so a failure before + // Helm records one, such as a chart that fails to render, shows only as + // Ready=False; a failure to apply the rendered resources is counted. It + // resets when the desired state (generation, chart version, values) changes, + // and on success only under the RetryOnFailure strategy, so a Ready release + // may carry it. + InstallFailures *int64 `json:"installFailures,omitempty"` + // UpgradeFailures is the upgrade failure count of the underlying HelmRelease, + // with the same counting and reset rules as InstallFailures. + UpgradeFailures *int64 `json:"upgradeFailures,omitempty"` + // LastAttemptedReleaseAction is the last release action (install or upgrade) + // the underlying HelmRelease attempted. + LastAttemptedReleaseAction *string `json:"lastAttemptedReleaseAction,omitempty"` } // ApplicationStatusApplyConfiguration constructs a declarative configuration of the ApplicationStatus type for use with @@ -78,3 +92,27 @@ func (b *ApplicationStatusApplyConfiguration) WithExternalIPsCount(value int32) b.ExternalIPsCount = &value return b } + +// WithInstallFailures sets the InstallFailures field in the declarative configuration to the given value +// and returns the receiver, so that objects can be built by chaining "With" function invocations. +// If called multiple times, the InstallFailures field is set to the value of the last call. +func (b *ApplicationStatusApplyConfiguration) WithInstallFailures(value int64) *ApplicationStatusApplyConfiguration { + b.InstallFailures = &value + return b +} + +// WithUpgradeFailures sets the UpgradeFailures field in the declarative configuration to the given value +// and returns the receiver, so that objects can be built by chaining "With" function invocations. +// If called multiple times, the UpgradeFailures field is set to the value of the last call. +func (b *ApplicationStatusApplyConfiguration) WithUpgradeFailures(value int64) *ApplicationStatusApplyConfiguration { + b.UpgradeFailures = &value + return b +} + +// WithLastAttemptedReleaseAction sets the LastAttemptedReleaseAction field in the declarative configuration to the given value +// and returns the receiver, so that objects can be built by chaining "With" function invocations. +// If called multiple times, the LastAttemptedReleaseAction field is set to the value of the last call. +func (b *ApplicationStatusApplyConfiguration) WithLastAttemptedReleaseAction(value string) *ApplicationStatusApplyConfiguration { + b.LastAttemptedReleaseAction = &value + return b +} diff --git a/pkg/generated/openapi/zz_generated.openapi.go b/pkg/generated/openapi/zz_generated.openapi.go index 1edff5230..3c1e96f75 100644 --- a/pkg/generated/openapi/zz_generated.openapi.go +++ b/pkg/generated/openapi/zz_generated.openapi.go @@ -289,6 +289,27 @@ func schema_pkg_apis_apps_v1alpha1_ApplicationStatus(ref common.ReferenceCallbac Format: "int32", }, }, + "installFailures": { + SchemaProps: spec.SchemaProps{ + Description: "InstallFailures is the install failure count of the underlying HelmRelease. It counts only attempts that recorded a Helm release, so a failure before Helm records one, such as a chart that fails to render, shows only as Ready=False; a failure to apply the rendered resources is counted. It resets when the desired state (generation, chart version, values) changes, and on success only under the RetryOnFailure strategy, so a Ready release may carry it.", + Type: []string{"integer"}, + Format: "int64", + }, + }, + "upgradeFailures": { + SchemaProps: spec.SchemaProps{ + Description: "UpgradeFailures is the upgrade failure count of the underlying HelmRelease, with the same counting and reset rules as InstallFailures.", + Type: []string{"integer"}, + Format: "int64", + }, + }, + "lastAttemptedReleaseAction": { + SchemaProps: spec.SchemaProps{ + Description: "LastAttemptedReleaseAction is the last release action (install or upgrade) the underlying HelmRelease attempted.", + Type: []string{"string"}, + Format: "", + }, + }, }, }, }, diff --git a/pkg/registry/apps/application/rest.go b/pkg/registry/apps/application/rest.go index 322ef6c6d..5f618b4e3 100644 --- a/pkg/registry/apps/application/rest.go +++ b/pkg/registry/apps/application/rest.go @@ -1573,13 +1573,23 @@ func (r *REST) convertHelmReleaseToApplication(ctx context.Context, hr *helmv2.H }, Spec: filteredSpec, Status: appsv1alpha1.ApplicationStatus{ - Version: hr.Status.LastAttemptedRevision, + Version: hr.Status.LastAttemptedRevision, + InstallFailures: hr.Status.InstallFailures, + UpgradeFailures: hr.Status.UpgradeFailures, + LastAttemptedReleaseAction: string(hr.Status.LastAttemptedReleaseAction), }, } var conditions []metav1.Condition for _, hrCondition := range hr.GetConditions() { - if hrCondition.Type == "Ready" || hrCondition.Type == "Released" { + // The Application carries no generation, so a client cannot tell a Stalled + // left over from the previous spec from one about the current spec. Compare + // the condition's own ObservedGeneration, not Status.ObservedGeneration: the + // question is which spec this condition describes. + if hrCondition.Type == "Stalled" && hrCondition.ObservedGeneration < hr.Generation { + continue + } + if hrCondition.Type == "Ready" || hrCondition.Type == "Released" || hrCondition.Type == "Stalled" { conditions = append(conditions, metav1.Condition{ LastTransitionTime: hrCondition.LastTransitionTime, Reason: hrCondition.Reason, diff --git a/pkg/registry/apps/application/rest_release_failure_test.go b/pkg/registry/apps/application/rest_release_failure_test.go new file mode 100644 index 000000000..fc28f1553 --- /dev/null +++ b/pkg/registry/apps/application/rest_release_failure_test.go @@ -0,0 +1,123 @@ +package application + +import ( + "context" + "testing" + + helmv2 "github.com/fluxcd/helm-controller/api/v2" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +func TestConvertReleaseFailure_StalledAndCountersExposed(t *testing.T) { + r := newTestRESTWithSchemes() + hr := makeHelmRelease("postgresql-mydb", "default") + hr.Generation = 2 + hr.Status.Conditions = []metav1.Condition{ + {Type: "Stalled", Status: metav1.ConditionTrue, Reason: "RetriesExceeded", Message: "Failed to install after 1 attempt(s)", ObservedGeneration: 2}, + {Type: "Ready", Status: metav1.ConditionFalse, Reason: "InstallFailed", Message: "install failed", ObservedGeneration: 2}, + {Type: "Released", Status: metav1.ConditionFalse, Reason: "InstallFailed", Message: "install failed", ObservedGeneration: 2}, + } + hr.Status.Failures = 7 + hr.Status.InstallFailures = 1 + hr.Status.UpgradeFailures = 3 + hr.Status.LastAttemptedReleaseAction = helmv2.ReleaseActionInstall + + app, err := r.convertHelmReleaseToApplication(context.TODO(), hr, nil) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + + sc := findCondition(app.GetConditions(), "Stalled") + if sc == nil { + t.Fatal("expected Stalled condition to be copied from the HelmRelease") + } + if sc.Status != metav1.ConditionTrue || sc.Reason != "RetriesExceeded" { + t.Errorf("expected Stalled=True/RetriesExceeded, got %s/%s", sc.Status, sc.Reason) + } + if app.Status.InstallFailures != 1 { + t.Errorf("expected installFailures=1, got %d", app.Status.InstallFailures) + } + if app.Status.UpgradeFailures != 3 { + t.Errorf("expected upgradeFailures=3, got %d", app.Status.UpgradeFailures) + } + if app.Status.LastAttemptedReleaseAction != "install" { + t.Errorf("expected lastAttemptedReleaseAction=install, got %q", app.Status.LastAttemptedReleaseAction) + } +} + +// After a spec update the HelmRelease keeps the Stalled condition of the +// previous generation until helm-controller reconciles it; reported as is, it +// would read as a terminal failure of the new values. +func TestConvertReleaseFailure_StalledFromOlderGenerationOmitted(t *testing.T) { + r := newTestRESTWithSchemes() + hr := makeHelmRelease("postgresql-mydb", "default") + hr.Generation = 2 + hr.Status.Conditions = []metav1.Condition{ + {Type: "Stalled", Status: metav1.ConditionTrue, Reason: "RetriesExceeded", Message: "Failed to install after 1 attempt(s)", ObservedGeneration: 1}, + {Type: "Ready", Status: metav1.ConditionFalse, Reason: "InstallFailed", Message: "install failed", ObservedGeneration: 1}, + } + hr.Status.InstallFailures = 1 + hr.Status.LastAttemptedReleaseAction = helmv2.ReleaseActionInstall + + app, err := r.convertHelmReleaseToApplication(context.TODO(), hr, nil) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + + if findCondition(app.GetConditions(), "Stalled") != nil { + t.Error("expected Stalled observed at an older generation not to be copied") + } + if findCondition(app.GetConditions(), "Ready") == nil { + t.Error("expected Ready to still be copied") + } +} + +// helm-controller clears the counters on success only under the RetryOnFailure +// strategy, so a Ready release can still carry them; they must not be hidden. +func TestConvertReleaseFailure_CountersKeptOnReadyRelease(t *testing.T) { + r := newTestRESTWithSchemes() + hr := makeHelmRelease("postgresql-mydb", "default") + hr.Status.Conditions = []metav1.Condition{ + {Type: "Ready", Status: metav1.ConditionTrue, Reason: "UpgradeSucceeded", Message: "ok"}, + {Type: "Released", Status: metav1.ConditionTrue, Reason: "UpgradeSucceeded", Message: "ok"}, + } + hr.Status.InstallFailures = 2 + hr.Status.UpgradeFailures = 1 + hr.Status.LastAttemptedReleaseAction = helmv2.ReleaseActionUpgrade + + app, err := r.convertHelmReleaseToApplication(context.TODO(), hr, nil) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + + if findCondition(app.GetConditions(), "Stalled") != nil { + t.Error("expected no Stalled condition on a release that does not carry one") + } + if app.Status.InstallFailures != 2 || app.Status.UpgradeFailures != 1 { + t.Errorf("expected install=2 upgrade=1, got install=%d upgrade=%d", app.Status.InstallFailures, app.Status.UpgradeFailures) + } + if app.Status.LastAttemptedReleaseAction != "upgrade" { + t.Errorf("expected lastAttemptedReleaseAction=upgrade, got %q", app.Status.LastAttemptedReleaseAction) + } +} + +func TestConvertReleaseFailure_ReconcilingNotCopied(t *testing.T) { + r := newTestRESTWithSchemes() + hr := makeHelmRelease("postgresql-mydb", "default") + hr.Status.Conditions = []metav1.Condition{ + {Type: "Reconciling", Status: metav1.ConditionTrue, Reason: "Progressing", Message: "Running 'install' action"}, + {Type: "Ready", Status: metav1.ConditionUnknown, Reason: "Progressing", Message: "Running 'install' action"}, + } + + app, err := r.convertHelmReleaseToApplication(context.TODO(), hr, nil) + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + + if findCondition(app.GetConditions(), "Reconciling") != nil { + t.Error("expected Reconciling not to be copied") + } + if findCondition(app.GetConditions(), "Ready") == nil { + t.Error("expected Ready to be copied") + } +} From d4ad5612d35d0272ec8a05ed0f90990102be7340 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Wed, 21 Oct 2026 10:53:05 +0400 Subject: [PATCH 087/518] fix(backups): report sizeBytes for MariaDB backup artifacts Backup.status.artifact.sizeBytes was only ever set by the etcd strategy, so MariaDB backups always arrived without a size. mariadb-operator writes each logical dump as /backup.[.].sql under a prefix shared by every backup of the application, and the URI the driver records is synthetic, so there is no single object to HEAD. On completion the driver now lists the prefix with the tenant credentials the operator Backup was rendered with and takes the size of the one dump whose timestamp falls inside the operator Backup's lifetime. When the bucket cannot be read or the window does not match exactly one dump, the field stays unset. The API doc now states that an absent sizeBytes means "unknown", and the Velero strategy documents why it leaves the field unset: kopia stores data content-addressed across backups, so there is no per-backup stored size. Refs #4316 Assisted-by: LLM Signed-off-by: Andrey Kolkov --- api/backups/v1alpha1/backup_types.go | 4 +- go.mod | 9 + go.sum | 19 ++ .../backupcontroller/backupjob_controller.go | 4 + .../backupcontroller/mariadb_artifact_size.go | 193 +++++++++++++++ .../mariadb_artifact_size_test.go | 234 ++++++++++++++++++ .../mariadbstrategy_controller.go | 5 +- .../velerostrategy_controller.go | 4 +- .../backups.cozystack.io_backups.yaml | 6 +- 9 files changed, 473 insertions(+), 5 deletions(-) create mode 100644 internal/backupcontroller/mariadb_artifact_size.go create mode 100644 internal/backupcontroller/mariadb_artifact_size_test.go diff --git a/api/backups/v1alpha1/backup_types.go b/api/backups/v1alpha1/backup_types.go index fd76ac47e..483bb1fd8 100644 --- a/api/backups/v1alpha1/backup_types.go +++ b/api/backups/v1alpha1/backup_types.go @@ -37,7 +37,9 @@ type BackupArtifact struct { // For example: s3://bucket/prefix/file.tar.gz URI string `json:"uri"` - // SizeBytes is the size of the artifact in bytes, if known. + // SizeBytes is the size of the artifact in bytes, if known. Drivers + // leave it unset rather than writing 0 when they cannot measure the + // stored size, so an absent value means "unknown", not "empty". // +optional SizeBytes int64 `json:"sizeBytes,omitempty"` diff --git a/go.mod b/go.mod index 6109faf15..34352e583 100644 --- a/go.mod +++ b/go.mod @@ -16,6 +16,7 @@ require ( github.com/go-logr/logr v1.4.4 github.com/go-logr/zapr v1.3.0 github.com/go-task/slim-sprig/v3 v3.0.0 + github.com/minio/minio-go/v7 v7.0.98 github.com/onsi/ginkgo/v2 v2.33.0 github.com/onsi/gomega v1.43.1 github.com/prometheus/client_golang v1.24.1 @@ -63,6 +64,7 @@ require ( github.com/fsnotify/fsnotify v1.10.1 // indirect github.com/fxamacker/cbor/v2 v2.9.4 // indirect github.com/gkampitakis/go-snaps v0.5.23 // indirect + github.com/go-ini/ini v1.67.0 // indirect github.com/go-logr/stdr v1.2.2 // indirect github.com/go-openapi/jsonpointer v1.0.1 // indirect github.com/go-openapi/jsonreference v1.0.2 // indirect @@ -93,18 +95,25 @@ require ( github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/klauspost/compress v1.20.0 // indirect + github.com/klauspost/cpuid/v2 v2.3.0 // indirect + github.com/klauspost/crc32 v1.3.0 // indirect github.com/kylelemons/godebug v1.1.0 // indirect + github.com/minio/crc64nvme v1.1.1 // indirect + github.com/minio/md5-simd v1.1.2 // indirect github.com/moby/spdystream v0.5.1 // indirect github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/philhofer/fwd v1.2.0 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/prometheus/client_model v0.6.3 // indirect github.com/prometheus/common v0.71.0 // indirect github.com/prometheus/procfs v0.22.0 // indirect github.com/rogpeppe/go-internal v1.16.0 // indirect + github.com/rs/xid v1.6.0 // indirect github.com/sirupsen/logrus v1.10.2 // indirect github.com/tidwall/match v1.2.0 // indirect + github.com/tinylib/msgp v1.6.1 // indirect github.com/x448/float16 v0.8.4 // indirect go.etcd.io/etcd/api/v3 v3.7.0 // indirect go.etcd.io/etcd/client/pkg/v3 v3.7.0 // indirect diff --git a/go.sum b/go.sum index dea6aaf08..667ca71ee 100644 --- a/go.sum +++ b/go.sum @@ -61,6 +61,8 @@ github.com/gkampitakis/ciinfo v0.3.4 h1:5eBSibVuSMbb/H6Elc0IIEFbkzCJi3lm94n0+U7Z github.com/gkampitakis/ciinfo v0.3.4/go.mod h1:1NIwaOcFChN4fa/B0hEBdAb6npDlFL8Bwx4dfRLRqAo= github.com/gkampitakis/go-snaps v0.5.23 h1:okh5QR48zpUjpWtu65AtqxdCY8huJq+dEDuUzd1PuKg= github.com/gkampitakis/go-snaps v0.5.23/go.mod h1:uy3lVzCCRRsAwYqSocyw5fY8xRLCYEfqoOJNxr8HonM= +github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A= +github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8= github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8= github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= @@ -141,6 +143,11 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA= github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI= +github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= +github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y= +github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0= +github.com/klauspost/crc32 v1.3.0 h1:sSmTt3gUt81RP655XGZPElI0PelVTZ6YwCRnPSupoFM= +github.com/klauspost/crc32 v1.3.0/go.mod h1:D7kQaZhnkX/Y0tstFGf8VUzv2UofNGqCjnC3zdHB0Hw= github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= @@ -151,6 +158,12 @@ github.com/maruel/natural v1.3.0 h1:VsmCsBmEyrR46RomtgHs5hbKADGRVtliHTyCOLFBpsg= github.com/maruel/natural v1.3.0/go.mod h1:v+Rfd79xlw1AgVBjbO0BEQmptqb5HvL/k9GRHB7ZKEg= github.com/mfridman/tparse v0.18.0 h1:wh6dzOKaIwkUGyKgOntDW4liXSo37qg5AXbIhkMV3vE= github.com/mfridman/tparse v0.18.0/go.mod h1:gEvqZTuCgEhPbYk/2lS3Kcxg1GmTxxU7kTC8DvP0i/A= +github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI= +github.com/minio/crc64nvme v1.1.1/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg= +github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= +github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= +github.com/minio/minio-go/v7 v7.0.98 h1:MeAVKjLVz+XJ28zFcuYyImNSAh8Mq725uNW4beRisi0= +github.com/minio/minio-go/v7 v7.0.98/go.mod h1:cY0Y+W7yozf0mdIclrttzo1Iiu7mEf9y7nk2uXqMOvM= github.com/moby/spdystream v0.5.1 h1:9sNYeYZUcci9R6/w7KDaFWEWeV4LStVG78Mpyq/Zm/Y= github.com/moby/spdystream v0.5.1/go.mod h1:xBAYlnt/ay+11ShkdFKNAG7LsyK/tmNBVvVOwrfMgdI= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= @@ -165,6 +178,8 @@ github.com/onsi/ginkgo/v2 v2.33.0 h1:C8gBA6Uc2ZEubiV+SXiu5tZnMTwEmXHgkJwGozKtZf8 github.com/onsi/ginkgo/v2 v2.33.0/go.mod h1:+aXOY+vzZ5mu2iI2HpTZUPmM//oQfsNFX6gU9kNcA44= github.com/onsi/gomega v1.43.1 h1:vGIPFuYrIO6/0Z09s0I0QQQgFchiX4+tb1re3MScJYo= github.com/onsi/gomega v1.43.1/go.mod h1:e/C2HwaZ1DhvjzXXuFhcR7hY7Sh9pl7MmoWKEjzwcdA= +github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM= +github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= @@ -180,6 +195,8 @@ github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs= github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro= github.com/rogpeppe/go-internal v1.16.0 h1:O9DK+vNMDVGLr2BeZqmpLeMjiMNkuXfcqntWbZV6S5g= github.com/rogpeppe/go-internal v1.16.0/go.mod h1:DrUVZyrJU+txYW5/1kwtXQSMFio52ZOxX7yM1VHvnxs= +github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU= +github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0= github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw= github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4= @@ -206,6 +223,8 @@ github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4= github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU= github.com/tidwall/sjson v1.2.5 h1:kLy8mja+1c9jlljvWTlSazM7cKDRfJuR/bOJhcY5NcY= github.com/tidwall/sjson v1.2.5/go.mod h1:Fvgq9kS/6ociJEDnK0Fk1cpYF4FIW6ZF7LAe+6jwd28= +github.com/tinylib/msgp v1.6.1 h1:ESRv8eL3u+DNHUoSAAQRE50Hm162zqAnBoGv9PzScPY= +github.com/tinylib/msgp v1.6.1/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA= github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 h1:6fotK7otjonDflCTK0BCfls4SPy3NcCVb5dqqmbRknE= github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75/go.mod h1:KO6IkyS8Y3j8OdNO85qEYBsRPuteD+YciPomcXdrMnk= github.com/vmware-tanzu/velero v1.18.3 h1:9uwN5yxTWQl9PsF5ljX4xiG+7VIq3I9mxOQ0DyyZ31A= diff --git a/internal/backupcontroller/backupjob_controller.go b/internal/backupcontroller/backupjob_controller.go index c6b743356..f329eed4d 100644 --- a/internal/backupcontroller/backupjob_controller.go +++ b/internal/backupcontroller/backupjob_controller.go @@ -39,6 +39,10 @@ type BackupJobReconciler struct { // re-reads the BackupJob before dispatching to a driver. Wired in // SetupWithManager. APIReader client.Reader + + // newS3ObjectLister overrides the S3 client used to measure artifact + // sizes; nil means a real minio client. Set by unit tests. + newS3ObjectLister func(ctx context.Context, namespace string, s3 *strategyv1alpha1.MariaDBS3Template) (s3ObjectLister, error) } func (r *BackupJobReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go new file mode 100644 index 000000000..b968a98d4 --- /dev/null +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -0,0 +1,193 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "crypto/tls" + "crypto/x509" + "fmt" + "net/http" + "strings" + "time" + + "github.com/minio/minio-go/v7" + "github.com/minio/minio-go/v7/pkg/credentials" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/types" + + strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" + "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" +) + +const ( + mariadbSizeProbeTimeout = 30 * time.Second + + // Slack around the [created, completed] window of the operator Backup + // when matching the timestamp mariadb-operator embeds in the dump file + // name: that timestamp comes from the backup Job pod's clock, not the + // apiserver's, and is truncated to seconds. + mariadbSizeProbeClockSkew = time.Minute +) + +type s3ObjectInfo struct { + Key string + Size int64 +} + +type s3ObjectLister interface { + ListObjects(ctx context.Context, bucket, prefix string) ([]s3ObjectInfo, error) +} + +// mariadbArtifactSizeBytes measures the dump mariadb-operator uploaded for +// mdbBackup. The strategy prefix is shared by every backup of the +// application, so the dump is identified by the timestamp in its file name +// falling inside the operator Backup's lifetime. Returns 0 - leaving +// Artifact.SizeBytes absent - when the object store cannot be read or the +// window does not match exactly one dump: an unknown size must not be +// reported as a measured one. +func (r *BackupJobReconciler) mariadbArtifactSizeBytes( + ctx context.Context, + namespace string, + s3 *strategyv1alpha1.MariaDBS3Template, + mdbBackup *mariadbtypes.Backup, + completedAt time.Time, +) int64 { + logger := getLogger(ctx) + ctx, cancel := context.WithTimeout(ctx, mariadbSizeProbeTimeout) + defer cancel() + + newLister := r.newS3ObjectLister + if newLister == nil { + newLister = r.newMariaDBMinioLister + } + lister, err := newLister(ctx, namespace, s3) + if err != nil { + logger.Debug("cannot build S3 client to measure MariaDB backup size", "backup", mdbBackup.Name, "error", err) + return 0 + } + prefix := mariadbOperatorS3Prefix(s3.Prefix) + objects, err := lister.ListObjects(ctx, s3.Bucket, prefix) + if err != nil { + logger.Debug("cannot list MariaDB backup objects to measure size", "backup", mdbBackup.Name, "error", err) + return 0 + } + + lower := mdbBackup.CreationTimestamp.Add(-mariadbSizeProbeClockSkew) + upper := completedAt.Add(mariadbSizeProbeClockSkew) + var matched []s3ObjectInfo + for _, obj := range objects { + ts, ok := parseMariaDBLogicalBackupFile(strings.TrimPrefix(obj.Key, prefix)) + if !ok || ts.Before(lower) || ts.After(upper) { + continue + } + matched = append(matched, obj) + } + if len(matched) != 1 { + logger.Debug("MariaDB backup dump not attributable to a single object; leaving size unset", + "backup", mdbBackup.Name, "candidates", len(matched)) + return 0 + } + return matched[0].Size +} + +// mariadbOperatorS3Prefix mirrors how mariadb-operator normalises +// spec.storage.s3.prefix before prepending it to object keys. +func mariadbOperatorS3Prefix(prefix string) string { + if prefix == "" || prefix == "/" { + return "" + } + if !strings.HasSuffix(prefix, "/") { + return prefix + "/" + } + return prefix +} + +// parseMariaDBLogicalBackupFile accepts the logical-backup file names +// mariadb-operator writes, backup..sql and +// backup...sql, and returns the embedded timestamp. +func parseMariaDBLogicalBackupFile(name string) (time.Time, bool) { + parts := strings.Split(name, ".") + if len(parts) != 3 && len(parts) != 4 { + return time.Time{}, false + } + if parts[0] != "backup" || parts[len(parts)-1] != "sql" { + return time.Time{}, false + } + ts, err := time.Parse(time.RFC3339, parts[1]) + if err != nil { + return time.Time{}, false + } + return ts, true +} + +type minioObjectLister struct { + client *minio.Client +} + +func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix string) ([]s3ObjectInfo, error) { + var out []s3ObjectInfo + for obj := range l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix}) { + if obj.Err != nil { + return nil, obj.Err + } + out = append(out, s3ObjectInfo{Key: obj.Key, Size: obj.Size}) + } + return out, nil +} + +// newMariaDBMinioLister builds an S3 client from the same Secrets and TLS +// settings the operator Backup was rendered with, so the controller reads +// the bucket with the tenant's credentials rather than its own. +func (r *BackupJobReconciler) newMariaDBMinioLister(ctx context.Context, namespace string, s3 *strategyv1alpha1.MariaDBS3Template) (s3ObjectLister, error) { + accessKey, err := r.mariadbSecretValue(ctx, namespace, s3.AccessKeyIDSecretKeyRef) + if err != nil { + return nil, err + } + secretKey, err := r.mariadbSecretValue(ctx, namespace, s3.SecretAccessKeySecretKeyRef) + if err != nil { + return nil, err + } + var sessionToken string + if s3.SessionTokenSecretKeyRef != nil { + if sessionToken, err = r.mariadbSecretValue(ctx, namespace, *s3.SessionTokenSecretKeyRef); err != nil { + return nil, err + } + } + + opts := &minio.Options{ + Creds: credentials.NewStaticV4(accessKey, secretKey, sessionToken), + Region: s3.Region, + Secure: s3.TLS != nil && s3.TLS.Enabled, + } + if opts.Secure && s3.TLS.CASecretKeyRef != nil { + caPEM, err := r.mariadbSecretValue(ctx, namespace, *s3.TLS.CASecretKeyRef) + if err != nil { + return nil, err + } + pool, err := x509.SystemCertPool() + if err != nil { + pool = x509.NewCertPool() + } + if !pool.AppendCertsFromPEM([]byte(caPEM)) { + return nil, fmt.Errorf("no PEM certificates in Secret %s/%s key %q", namespace, s3.TLS.CASecretKeyRef.Name, s3.TLS.CASecretKeyRef.Key) + } + opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}} + } + client, err := minio.New(s3.Endpoint, opts) + if err != nil { + return nil, err + } + return &minioObjectLister{client: client}, nil +} + +func (r *BackupJobReconciler) mariadbSecretValue(ctx context.Context, namespace string, ref strategyv1alpha1.MariaDBSecretKeySelector) (string, error) { + secret := &corev1.Secret{} + if err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: ref.Name}, secret); err != nil { + return "", err + } + v, ok := secret.Data[ref.Key] + if !ok { + return "", fmt.Errorf("Secret %s/%s has no key %q", namespace, ref.Name, ref.Key) + } + return string(v), nil +} diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go new file mode 100644 index 000000000..f7e450a26 --- /dev/null +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -0,0 +1,234 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" + "github.com/cozystack/cozystack/internal/backupcontroller/mariadbapp" + "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" +) + +type fakeS3Lister struct { + objects []s3ObjectInfo + err error + + bucket, prefix string +} + +func (f *fakeS3Lister) ListObjects(_ context.Context, bucket, prefix string) ([]s3ObjectInfo, error) { + f.bucket, f.prefix = bucket, prefix + return f.objects, f.err +} + +func reconcilerWithLister(l *fakeS3Lister) *BackupJobReconciler { + return &BackupJobReconciler{ + newS3ObjectLister: func(context.Context, string, *strategyv1alpha1.MariaDBS3Template) (s3ObjectLister, error) { + return l, nil + }, + } +} + +func dumpKey(prefix string, ts time.Time, compression string) string { + if compression == "" { + return fmt.Sprintf("%sbackup.%s.sql", prefix, ts.UTC().Format(time.RFC3339)) + } + return fmt.Sprintf("%sbackup.%s.%s.sql", prefix, ts.UTC().Format(time.RFC3339), compression) +} + +func TestParseMariaDBLogicalBackupFile(t *testing.T) { + cases := map[string]bool{ + "backup.2026-10-21T10:00:00Z.sql": true, + "backup.2026-10-21T10:00:00Z.gzip.sql": true, + "backup.2026-10-21T10:00:00+04:00.sql": true, + "backup.2026-10-21T10:00:00Z.sql.tmp": false, + "physicalbackup-20261021100000.xb": false, + "backup.not-a-date.sql": false, + "sub/backup.2026-10-21T10:00:00Z.sql": false, + "backup.2026-10-21T10:00:00Z.a.gzip.sql": false, + "backups.2026-10-21T10:00:00Z.gzip.sql": false, + "backup.2026-10-21T10:00:00Z.gzip.dump": false, + "": false, + } + for name, want := range cases { + if _, got := parseMariaDBLogicalBackupFile(name); got != want { + t.Errorf("parseMariaDBLogicalBackupFile(%q) = %v, want %v", name, got, want) + } + } +} + +func TestMariaDBOperatorS3Prefix(t *testing.T) { + cases := map[string]string{"": "", "/": "", "ns/app": "ns/app/", "ns/app/": "ns/app/"} + for in, want := range cases { + if got := mariadbOperatorS3Prefix(in); got != want { + t.Errorf("mariadbOperatorS3Prefix(%q) = %q, want %q", in, got, want) + } + } +} + +func TestMariaDBArtifactSizeBytes(t *testing.T) { + created := time.Date(2026, 10, 21, 10, 0, 0, 0, time.UTC) + completed := created.Add(3 * time.Minute) + mdbBackup := &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", Name: "op-bk", CreationTimestamp: metav1.NewTime(created), + }} + s3 := newRenderedMariaDBTemplate().Storage.S3 + s3.Prefix = "tenant/db" + const prefix = "tenant/db/" + + t.Run("single dump inside the backup window", func(t *testing.T) { + l := &fakeS3Lister{objects: []s3ObjectInfo{ + {Key: dumpKey(prefix, created.Add(-24*time.Hour), "gzip"), Size: 111}, + {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4242}, + {Key: prefix + "0-backup-target.txt", Size: 7}, + }} + got := reconcilerWithLister(l).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed) + if got != 4242 { + t.Errorf("size = %d, want 4242", got) + } + if l.bucket != "bkt" || l.prefix != prefix { + t.Errorf("listed %q/%q, want bkt/%q", l.bucket, l.prefix, prefix) + } + }) + + t.Run("pod clock skew within tolerance", func(t *testing.T) { + l := &fakeS3Lister{objects: []s3ObjectInfo{ + {Key: dumpKey(prefix, created.Add(-30*time.Second), ""), Size: 9}, + }} + if got := reconcilerWithLister(l).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 9 { + t.Errorf("size = %d, want 9", got) + } + }) + + t.Run("two dumps in the window leave size unset", func(t *testing.T) { + l := &fakeS3Lister{objects: []s3ObjectInfo{ + {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 1}, + {Key: dumpKey(prefix, created.Add(2*time.Minute), "gzip"), Size: 2}, + }} + if got := reconcilerWithLister(l).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + t.Errorf("size = %d, want 0 (ambiguous)", got) + } + }) + + t.Run("no dump in the window leaves size unset", func(t *testing.T) { + l := &fakeS3Lister{objects: []s3ObjectInfo{ + {Key: dumpKey(prefix, completed.Add(time.Hour), "gzip"), Size: 1}, + }} + if got := reconcilerWithLister(l).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + t.Errorf("size = %d, want 0", got) + } + }) + + t.Run("list error leaves size unset", func(t *testing.T) { + l := &fakeS3Lister{err: errors.New("AccessDenied")} + if got := reconcilerWithLister(l).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + t.Errorf("size = %d, want 0", got) + } + }) + + t.Run("missing credentials Secret leaves size unset", func(t *testing.T) { + c := newMariaDBStrategyTestClient(t) + r := &BackupJobReconciler{Client: c, Scheme: c.Scheme()} + if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + t.Errorf("size = %d, want 0", got) + } + }) +} + +// TestMariaDBArtifactSizeBytes_MinioClient drives the real minio client +// against a stub ListObjectsV2 endpoint, so the Secret wiring, prefix and +// response parsing are exercised together. +func TestMariaDBArtifactSizeBytes_MinioClient(t *testing.T) { + created := time.Date(2026, 10, 21, 10, 0, 0, 0, time.UTC) + key := dumpKey("tenant/db/", created.Add(time.Minute), "gzip") + + var gotPath, gotPrefix, gotAuth string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { + gotPath, gotPrefix, gotAuth = req.URL.Path, req.URL.Query().Get("prefix"), req.Header.Get("Authorization") + w.Header().Set("Content-Type", "application/xml") + fmt.Fprintf(w, ` + + bkttenant/db/11000false + %s31337"x"2026-10-21T10:01:00.000Z +`, key) + })) + defer srv.Close() + + creds := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "creds"}, + Data: map[string][]byte{ + "AWS_ACCESS_KEY_ID": []byte("AKID"), + "AWS_SECRET_ACCESS_KEY": []byte("secret"), + }, + } + c := newMariaDBStrategyTestClient(t, creds) + r := &BackupJobReconciler{Client: c, Scheme: c.Scheme()} + + s3 := newRenderedMariaDBTemplate().Storage.S3 + s3.Endpoint = strings.TrimPrefix(srv.URL, "http://") + s3.Prefix = "tenant/db/" + s3.Region = "us-east-1" + mdbBackup := &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", Name: "op-bk", CreationTimestamp: metav1.NewTime(created), + }} + + if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, created.Add(3*time.Minute)); got != 31337 { + t.Fatalf("size = %d, want 31337", got) + } + if gotPath != "/bkt/" || gotPrefix != "tenant/db/" { + t.Errorf("request path=%q prefix=%q, want /bkt/ and tenant/db/", gotPath, gotPrefix) + } + if !strings.Contains(gotAuth, "Credential=AKID/") { + t.Errorf("request not signed with the Secret's access key: %q", gotAuth) + } +} + +func TestCreateMariaDBBackupArtifact_SizeBytes(t *testing.T) { + apps := mariadbapp.GroupName + completed := metav1.NewTime(time.Date(2026, 10, 21, 10, 3, 0, 0, time.UTC)) + mdbBackup := &mariadbtypes.Backup{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", Name: "src-op-bk", + CreationTimestamp: metav1.NewTime(completed.Add(-3 * time.Minute)), + }, + Status: mariadbtypes.BackupStatus{Conditions: []metav1.Condition{{ + Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionTrue, LastTransitionTime: completed, + }}}, + } + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "mariadb-strategy-default"}, + Parameters: map[string]string{}, + } + job := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-size"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, + }, + } + c := newMariaDBStrategyTestClient(t, job) + r := reconcilerWithLister(&fakeS3Lister{objects: []s3ObjectInfo{ + {Key: dumpKey("src/", completed.Add(-time.Minute), "gzip"), Size: 2048}, + }}) + r.Client, r.Scheme = c, c.Scheme() + rendered := newRenderedMariaDBTemplate() + rendered.Storage.S3.Prefix = "src/" + + artefact, err := r.createMariaDBBackupArtifact(context.Background(), job, resolved, mdbBackup, rendered) + if err != nil { + t.Fatalf("createMariaDBBackupArtifact: %v", err) + } + if artefact.Status.Artifact == nil || artefact.Status.Artifact.SizeBytes != 2048 { + t.Fatalf("Artifact = %#v, want SizeBytes 2048", artefact.Status.Artifact) + } +} diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index a5bd77094..84fe0c2df 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -392,7 +392,10 @@ func (r *BackupJobReconciler) createMariaDBBackupArtifact( // operator's PVC ID), so an Artifact{URI: ""} is a worse signal than // no Artifact at all - leave it nil instead. if uri := driverMD[mariadbStorageURIKey]; uri != "" { - status.Artifact = &backupsv1alpha1.BackupArtifact{URI: uri} + status.Artifact = &backupsv1alpha1.BackupArtifact{ + URI: uri, + SizeBytes: r.mariadbArtifactSizeBytes(ctx, j.Namespace, rendered.Storage.S3, mdbBackup, takenAt.Time), + } } backup := &backupsv1alpha1.Backup{ ObjectMeta: metav1.ObjectMeta{ diff --git a/internal/backupcontroller/velerostrategy_controller.go b/internal/backupcontroller/velerostrategy_controller.go index e08b2f818..abedc97be 100644 --- a/internal/backupcontroller/velerostrategy_controller.go +++ b/internal/backupcontroller/velerostrategy_controller.go @@ -558,7 +558,9 @@ func (r *BackupJobReconciler) createBackupResource(ctx context.Context, backupJo veleroBackupNamespaceMetadataKey: veleroBackup.Namespace, } - // Create a basic artifact referencing the Velero backup + // SizeBytes stays unset: data lands in a content-addressed kopia + // repository shared across backups, so there is no per-backup stored + // size, and DataUpload totalBytes counts bytes read, not bytes stored. artifact := &backupsv1alpha1.BackupArtifact{ URI: fmt.Sprintf("velero://%s/%s", veleroBackup.Namespace, veleroBackup.Name), } diff --git a/packages/system/backup-controller/definitions/backups.cozystack.io_backups.yaml b/packages/system/backup-controller/definitions/backups.cozystack.io_backups.yaml index 28ef57fb7..f30d3dbc7 100644 --- a/packages/system/backup-controller/definitions/backups.cozystack.io_backups.yaml +++ b/packages/system/backup-controller/definitions/backups.cozystack.io_backups.yaml @@ -131,8 +131,10 @@ spec: For example: "sha256:". type: string sizeBytes: - description: SizeBytes is the size of the artifact in bytes, if - known. + description: |- + SizeBytes is the size of the artifact in bytes, if known. Drivers + leave it unset rather than writing 0 when they cannot measure the + stored size, so an absent value means "unknown", not "empty". format: int64 type: integer uri: From de60488a2cb9c25500af07ce13dbc467dfd72c98 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Wed, 21 Oct 2026 11:03:59 +0400 Subject: [PATCH 088/518] fix(backups): report sizeBytes for CNPG backup artifacts CNPG backups arrived without Backup.status.artifact.sizeBytes: cnpg.io/Backup status carries no size, and the data lives in barman's object-store layout rather than in a single object. barman-cloud uploads every base backup under //base// (backup.info plus the data and tablespace tarballs, possibly split into parts). On completion the driver now lists that prefix with the credentials and endpoint CA of the rendered barmanObjectStore and records the total. The WAL archive is not counted: it is one stream shared by every base backup of the server. The field stays unset when the backup ID is unknown, the bucket cannot be read, or nothing is stored under the prefix. The S3 client and Secret lookup are now shared with the MariaDB driver. Refs #4316 Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 102 ++++++++ .../backupcontroller/artifact_size_test.go | 49 ++++ .../backupcontroller/backupjob_controller.go | 2 +- .../backupcontroller/cnpg_artifact_size.go | 128 ++++++++++ .../cnpg_artifact_size_test.go | 229 ++++++++++++++++++ .../cnpgstrategy_controller.go | 3 +- .../backupcontroller/mariadb_artifact_size.go | 146 ++++------- .../mariadb_artifact_size_test.go | 49 +--- 8 files changed, 564 insertions(+), 144 deletions(-) create mode 100644 internal/backupcontroller/artifact_size.go create mode 100644 internal/backupcontroller/artifact_size_test.go create mode 100644 internal/backupcontroller/cnpg_artifact_size.go create mode 100644 internal/backupcontroller/cnpg_artifact_size_test.go diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go new file mode 100644 index 000000000..6ca2ad3ea --- /dev/null +++ b/internal/backupcontroller/artifact_size.go @@ -0,0 +1,102 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "fmt" + "net/http" + "time" + + "github.com/minio/minio-go/v7" + "github.com/minio/minio-go/v7/pkg/credentials" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/types" +) + +// artifactSizeProbeTimeout bounds the object-store round trips a driver +// makes to measure Backup.status.artifact.sizeBytes; the probe runs inside +// the reconcile that creates the Backup. +const artifactSizeProbeTimeout = 30 * time.Second + +type s3ObjectInfo struct { + Key string + Size int64 +} + +type s3ObjectLister interface { + ListObjects(ctx context.Context, bucket, prefix string) ([]s3ObjectInfo, error) +} + +// s3Connection is what a driver resolves from its rendered strategy to +// read the bucket with the same identity the backup was written with. +type s3Connection struct { + // Endpoint is host[:port], without a scheme. + Endpoint string + Secure bool + Region string + AccessKey string + SecretKey string + SessionToken string + // CAPEM is trusted on top of the system pool when set. + CAPEM []byte +} + +func (r *BackupJobReconciler) s3Lister(conn s3Connection) (s3ObjectLister, error) { + if r.newS3ObjectLister != nil { + return r.newS3ObjectLister(conn) + } + return newMinioObjectLister(conn) +} + +type minioObjectLister struct { + client *minio.Client +} + +func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { + opts := &minio.Options{ + Creds: credentials.NewStaticV4(conn.AccessKey, conn.SecretKey, conn.SessionToken), + Region: conn.Region, + Secure: conn.Secure, + } + if conn.Secure && len(conn.CAPEM) > 0 { + pool, err := x509.SystemCertPool() + if err != nil { + pool = x509.NewCertPool() + } + if !pool.AppendCertsFromPEM(conn.CAPEM) { + return nil, errors.New("no PEM certificates in the endpoint CA bundle") + } + opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}} + } + client, err := minio.New(conn.Endpoint, opts) + if err != nil { + return nil, err + } + return &minioObjectLister{client: client}, nil +} + +func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix string) ([]s3ObjectInfo, error) { + var out []s3ObjectInfo + for obj := range l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: true}) { + if obj.Err != nil { + return nil, obj.Err + } + out = append(out, s3ObjectInfo{Key: obj.Key, Size: obj.Size}) + } + return out, nil +} + +func (r *BackupJobReconciler) secretValue(ctx context.Context, namespace, name, key string) (string, error) { + secret := &corev1.Secret{} + if err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, secret); err != nil { + return "", err + } + v, ok := secret.Data[key] + if !ok { + return "", fmt.Errorf("Secret %s/%s has no key %q", namespace, name, key) + } + return string(v), nil +} diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go new file mode 100644 index 000000000..6e389cf66 --- /dev/null +++ b/internal/backupcontroller/artifact_size_test.go @@ -0,0 +1,49 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "testing" + + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" +) + +type fakeS3Lister struct { + objects []s3ObjectInfo + err error + + conn s3Connection + bucket, prefix string +} + +func (f *fakeS3Lister) ListObjects(_ context.Context, bucket, prefix string) ([]s3ObjectInfo, error) { + f.bucket, f.prefix = bucket, prefix + return f.objects, f.err +} + +func s3CredsSecret(namespace, name string) *corev1.Secret { + return &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: namespace, Name: name}, + Data: map[string][]byte{ + "AWS_ACCESS_KEY_ID": []byte("AKID"), + "AWS_SECRET_ACCESS_KEY": []byte("secret"), + }, + } +} + +// reconcilerWithLister returns a reconciler whose S3 reads go to l, backed +// by a fake client seeded with objs (typically the credentials Secret). +func reconcilerWithLister(t *testing.T, l *fakeS3Lister, objs ...client.Object) *BackupJobReconciler { + t.Helper() + c := newMariaDBStrategyTestClient(t, objs...) + return &BackupJobReconciler{ + Client: c, + Scheme: c.Scheme(), + newS3ObjectLister: func(conn s3Connection) (s3ObjectLister, error) { + l.conn = conn + return l, nil + }, + } +} diff --git a/internal/backupcontroller/backupjob_controller.go b/internal/backupcontroller/backupjob_controller.go index f329eed4d..b86278649 100644 --- a/internal/backupcontroller/backupjob_controller.go +++ b/internal/backupcontroller/backupjob_controller.go @@ -42,7 +42,7 @@ type BackupJobReconciler struct { // newS3ObjectLister overrides the S3 client used to measure artifact // sizes; nil means a real minio client. Set by unit tests. - newS3ObjectLister func(ctx context.Context, namespace string, s3 *strategyv1alpha1.MariaDBS3Template) (s3ObjectLister, error) + newS3ObjectLister func(conn s3Connection) (s3ObjectLister, error) } func (r *BackupJobReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { diff --git a/internal/backupcontroller/cnpg_artifact_size.go b/internal/backupcontroller/cnpg_artifact_size.go new file mode 100644 index 000000000..bff2d5b39 --- /dev/null +++ b/internal/backupcontroller/cnpg_artifact_size.go @@ -0,0 +1,128 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "errors" + "fmt" + "net/url" + "strings" + + strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" +) + +// cnpgArtifactSizeBytes sums the objects barman-cloud uploaded for one base +// backup, //base//. The WAL archive +// is not counted: it is one continuous stream shared by every base backup of +// the server, so no part of it belongs to a single Backup. Returns 0 - +// leaving Artifact.SizeBytes absent - when the backup ID is unknown, the +// bucket cannot be read, or nothing is stored under the prefix. +func (r *BackupJobReconciler) cnpgArtifactSizeBytes( + ctx context.Context, + namespace string, + store *strategyv1alpha1.BarmanObjectStoreTemplate, + serverName, backupID string, +) int64 { + logger := getLogger(ctx) + if backupID == "" { + return 0 + } + ctx, cancel := context.WithTimeout(ctx, artifactSizeProbeTimeout) + defer cancel() + + bucket, path, err := parseBarmanS3DestinationPath(store.DestinationPath) + if err != nil { + logger.Debug("cannot measure CNPG backup size", "backupID", backupID, "error", err) + return 0 + } + conn, err := r.cnpgS3Connection(ctx, namespace, store) + if err != nil { + logger.Debug("cannot resolve S3 credentials to measure CNPG backup size", "backupID", backupID, "error", err) + return 0 + } + lister, err := r.s3Lister(conn) + if err != nil { + logger.Debug("cannot build S3 client to measure CNPG backup size", "backupID", backupID, "error", err) + return 0 + } + prefix := barmanBaseBackupPrefix(path, serverName, backupID) + objects, err := lister.ListObjects(ctx, bucket, prefix) + if err != nil { + logger.Debug("cannot list CNPG base backup objects to measure size", "backupID", backupID, "error", err) + return 0 + } + var total int64 + for _, obj := range objects { + total += obj.Size + } + if total == 0 { + logger.Debug("no CNPG base backup objects found; leaving size unset", "backupID", backupID, "prefix", prefix) + } + return total +} + +func (r *BackupJobReconciler) cnpgS3Connection(ctx context.Context, namespace string, store *strategyv1alpha1.BarmanObjectStoreTemplate) (s3Connection, error) { + var conn s3Connection + if store.S3Credentials == nil { + return conn, errors.New("barmanObjectStore has no s3Credentials") + } + if store.EndpointURL == "" { + conn.Endpoint, conn.Secure = "s3.amazonaws.com", true + } else { + u, err := url.Parse(store.EndpointURL) + if err != nil || u.Host == "" { + return conn, fmt.Errorf("invalid endpointURL %q", store.EndpointURL) + } + conn.Endpoint, conn.Secure = u.Host, u.Scheme == "https" + } + + creds := store.S3Credentials + accessKeyKey, secretKeyKey := creds.AccessKeyIDKey, creds.SecretAccessKeyKey + if accessKeyKey == "" { + accessKeyKey = defaultS3AccessKeyIDKey + } + if secretKeyKey == "" { + secretKeyKey = defaultS3SecretAccessKeyKey + } + var err error + if conn.AccessKey, err = r.secretValue(ctx, namespace, creds.SecretRef.Name, accessKeyKey); err != nil { + return conn, err + } + if conn.SecretKey, err = r.secretValue(ctx, namespace, creds.SecretRef.Name, secretKeyKey); err != nil { + return conn, err + } + if ca := store.EndpointCA; ca != nil && conn.Secure { + key := ca.Key + if key == "" { + key = defaultEndpointCAKey + } + pem, err := r.secretValue(ctx, namespace, ca.SecretRef.Name, key) + if err != nil { + return conn, err + } + conn.CAPEM = []byte(pem) + } + return conn, nil +} + +// parseBarmanS3DestinationPath splits s3://bucket/path into the bucket and +// the key path barman-cloud prefixes every object with. +func parseBarmanS3DestinationPath(destinationPath string) (bucket, path string, err error) { + u, err := url.Parse(destinationPath) + if err != nil || u.Scheme != "s3" || u.Host == "" { + return "", "", fmt.Errorf("destinationPath %q is not an s3:// URL", destinationPath) + } + return u.Host, strings.Trim(u.Path, "/"), nil +} + +// barmanBaseBackupPrefix mirrors barman-cloud's +// os.path.join(path, server_name, "base", backup_id), with a trailing slash +// so one backup ID never matches another that extends it. +func barmanBaseBackupPrefix(path, serverName, backupID string) string { + parts := make([]string, 0, 4) + if path != "" { + parts = append(parts, path) + } + parts = append(parts, serverName, "base", backupID) + return strings.Join(parts, "/") + "/" +} diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go new file mode 100644 index 000000000..29f09253b --- /dev/null +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -0,0 +1,229 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" + "github.com/cozystack/cozystack/internal/backupcontroller/cnpgtypes" +) + +func newSizedBarmanObjectStore() *strategyv1alpha1.BarmanObjectStoreTemplate { + return &strategyv1alpha1.BarmanObjectStoreTemplate{ + DestinationPath: "s3://bkt/tenant/pg/", + EndpointURL: "https://s3.example:8443", + S3Credentials: &strategyv1alpha1.S3CredentialsTemplate{ + SecretRef: corev1.LocalObjectReference{Name: "creds"}, + }, + } +} + +func TestBarmanBaseBackupPrefix(t *testing.T) { + cases := []struct{ path, want string }{ + {"tenant/pg", "tenant/pg/srv/base/20261021T100000/"}, + {"", "srv/base/20261021T100000/"}, + } + for _, tc := range cases { + if got := barmanBaseBackupPrefix(tc.path, "srv", "20261021T100000"); got != tc.want { + t.Errorf("barmanBaseBackupPrefix(%q) = %q, want %q", tc.path, got, tc.want) + } + } +} + +func TestParseBarmanS3DestinationPath(t *testing.T) { + cases := []struct { + in, bucket, path string + ok bool + }{ + {"s3://bkt/tenant/pg/", "bkt", "tenant/pg", true}, + {"s3://bkt", "bkt", "", true}, + {"s3://bkt/", "bkt", "", true}, + {"gs://bkt/x/", "", "", false}, + {"bkt/x/", "", "", false}, + } + for _, tc := range cases { + bucket, path, err := parseBarmanS3DestinationPath(tc.in) + if (err == nil) != tc.ok || bucket != tc.bucket || path != tc.path { + t.Errorf("parseBarmanS3DestinationPath(%q) = %q, %q, %v", tc.in, bucket, path, err) + } + } +} + +func TestCNPGArtifactSizeBytes(t *testing.T) { + const prefix = "tenant/pg/srv/base/20261021T100000/" + + t.Run("sums every object of the base backup", func(t *testing.T) { + l := &fakeS3Lister{objects: []s3ObjectInfo{ + {Key: prefix + "backup.info", Size: 1}, + {Key: prefix + "data_0000.tar.gz", Size: 1000}, + {Key: prefix + "data_0001.tar.gz", Size: 200}, + {Key: prefix + "16385.tar.gz", Size: 30}, + }} + r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")) + got := r.cnpgArtifactSizeBytes(context.Background(), "tenant", newSizedBarmanObjectStore(), "srv", "20261021T100000") + if got != 1231 { + t.Errorf("size = %d, want 1231", got) + } + if l.bucket != "bkt" || l.prefix != prefix { + t.Errorf("listed %q/%q, want bkt/%q", l.bucket, l.prefix, prefix) + } + if l.conn.Endpoint != "s3.example:8443" || !l.conn.Secure || l.conn.AccessKey != "AKID" || l.conn.SecretKey != "secret" { + t.Errorf("connection = %+v", l.conn) + } + }) + + t.Run("honours custom credential keys and the endpoint CA", func(t *testing.T) { + store := newSizedBarmanObjectStore() + store.S3Credentials.AccessKeyIDKey = "id" + store.S3Credentials.SecretAccessKeyKey = "key" + store.EndpointCA = &strategyv1alpha1.EndpointCARef{SecretRef: corev1.LocalObjectReference{Name: "ca"}} + creds := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "creds"}, + Data: map[string][]byte{"id": []byte("ID2"), "key": []byte("KEY2")}, + } + ca := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "ca"}, + Data: map[string][]byte{"ca.crt": []byte("PEM")}, + } + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: prefix + "data.tar", Size: 5}}} + r := reconcilerWithLister(t, l, creds, ca) + if got := r.cnpgArtifactSizeBytes(context.Background(), "tenant", store, "srv", "20261021T100000"); got != 5 { + t.Errorf("size = %d, want 5", got) + } + if l.conn.AccessKey != "ID2" || l.conn.SecretKey != "KEY2" || string(l.conn.CAPEM) != "PEM" { + t.Errorf("connection = %+v", l.conn) + } + }) + + unset := map[string]struct { + store func() *strategyv1alpha1.BarmanObjectStoreTemplate + backupID string + lister *fakeS3Lister + noSecret bool + }{ + "unknown backup ID": { + store: newSizedBarmanObjectStore, lister: &fakeS3Lister{objects: []s3ObjectInfo{{Key: "x", Size: 1}}}, + }, + "no objects under the prefix": { + store: newSizedBarmanObjectStore, backupID: "id", lister: &fakeS3Lister{}, + }, + "list error": { + store: newSizedBarmanObjectStore, backupID: "id", lister: &fakeS3Lister{err: errors.New("AccessDenied")}, + }, + "missing credentials Secret": { + store: newSizedBarmanObjectStore, backupID: "id", noSecret: true, + lister: &fakeS3Lister{objects: []s3ObjectInfo{{Key: "x", Size: 1}}}, + }, + "no s3Credentials": { + store: func() *strategyv1alpha1.BarmanObjectStoreTemplate { + s := newSizedBarmanObjectStore() + s.S3Credentials = nil + return s + }, + backupID: "id", lister: &fakeS3Lister{objects: []s3ObjectInfo{{Key: "x", Size: 1}}}, + }, + "non-s3 destination": { + store: func() *strategyv1alpha1.BarmanObjectStoreTemplate { + s := newSizedBarmanObjectStore() + s.DestinationPath = "azure://container/x" + return s + }, + backupID: "id", lister: &fakeS3Lister{objects: []s3ObjectInfo{{Key: "x", Size: 1}}}, + }, + } + for name, tc := range unset { + t.Run(name+" leaves size unset", func(t *testing.T) { + var r *BackupJobReconciler + if tc.noSecret { + r = reconcilerWithLister(t, tc.lister) + } else { + r = reconcilerWithLister(t, tc.lister, s3CredsSecret("tenant", "creds")) + } + if got := r.cnpgArtifactSizeBytes(context.Background(), "tenant", tc.store(), "srv", tc.backupID); got != 0 { + t.Errorf("size = %d, want 0", got) + } + }) + } +} + +// TestCNPGArtifactSizeBytes_MinioClient drives the real minio client against +// a stub ListObjectsV2 endpoint, so the endpoint URL parsing, Secret wiring +// and response parsing are exercised together. +func TestCNPGArtifactSizeBytes_MinioClient(t *testing.T) { + const prefix = "tenant/pg/srv/base/20261021T100000/" + var gotPath, gotPrefix, gotAuth string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { + if _, ok := req.URL.Query()["location"]; ok { + w.Header().Set("Content-Type", "application/xml") + fmt.Fprint(w, ``) + return + } + gotPath, gotPrefix, gotAuth = req.URL.Path, req.URL.Query().Get("prefix"), req.Header.Get("Authorization") + w.Header().Set("Content-Type", "application/xml") + fmt.Fprintf(w, ` + + bkt%[1]s21000false + %[1]sbackup.info7"a"2026-10-21T10:01:00.000Z + %[1]sdata.tar.gz4096"b"2026-10-21T10:01:00.000Z +`, prefix) + })) + defer srv.Close() + + c := newMariaDBStrategyTestClient(t, s3CredsSecret("tenant", "creds")) + r := &BackupJobReconciler{Client: c, Scheme: c.Scheme()} + store := newSizedBarmanObjectStore() + store.EndpointURL = srv.URL + + if got := r.cnpgArtifactSizeBytes(context.Background(), "tenant", store, "srv", "20261021T100000"); got != 4103 { + t.Fatalf("size = %d, want 4103", got) + } + if gotPath != "/bkt/" || gotPrefix != prefix { + t.Errorf("request path=%q prefix=%q, want /bkt/ and %q", gotPath, gotPrefix, prefix) + } + if !strings.Contains(gotAuth, "Credential=AKID/") { + t.Errorf("request not signed with the Secret's access key: %q", gotAuth) + } +} + +func TestCreateCNPGBackupArtifact_SizeBytes(t *testing.T) { + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + strategyGroup := strategyv1alpha1.GroupVersion.Group + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-size"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"}, + }, + } + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{APIGroup: &strategyGroup, Kind: strategyv1alpha1.CNPGStrategyKind, Name: "strat"}, + } + cnpgBk := &cnpgtypes.Backup{ + ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, + Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}, + } + rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()} + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: "tenant/pg/srv/base/20261021T100000/data.tar.gz", Size: 777}}} + r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")) + r.Client = newCNPGStrategyTestClient(t, s3CredsSecret("tenant", "creds")) + + got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")) + if err != nil { + t.Fatalf("createCNPGBackupArtifact: %v", err) + } + if got.Status.Artifact == nil || got.Status.Artifact.SizeBytes != 777 { + t.Fatalf("Artifact = %#v, want SizeBytes 777", got.Status.Artifact) + } + if l.prefix != "tenant/pg/srv/base/20261021T100000/" { + t.Errorf("listed prefix %q does not use the effective serverName", l.prefix) + } +} diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 99e6b03f6..3b62ce6e1 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -855,7 +855,8 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact( Status: backupsv1alpha1.BackupStatus{ Phase: backupsv1alpha1.BackupPhaseReady, Artifact: &backupsv1alpha1.BackupArtifact{ - URI: fmt.Sprintf("cnpg://%s/%s", serverName, cnpgBackup.Name), + URI: fmt.Sprintf("cnpg://%s/%s", serverName, cnpgBackup.Name), + SizeBytes: r.cnpgArtifactSizeBytes(ctx, j.Namespace, &rendered.BarmanObjectStore, serverName, cnpgBackup.Status.BackupID), }, UnderlyingResources: underlyingResources, }, diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index b968a98d4..20f68a454 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -3,40 +3,18 @@ package backupcontroller import ( "context" - "crypto/tls" - "crypto/x509" - "fmt" - "net/http" "strings" "time" - "github.com/minio/minio-go/v7" - "github.com/minio/minio-go/v7/pkg/credentials" - corev1 "k8s.io/api/core/v1" - "k8s.io/apimachinery/pkg/types" - strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" ) -const ( - mariadbSizeProbeTimeout = 30 * time.Second - - // Slack around the [created, completed] window of the operator Backup - // when matching the timestamp mariadb-operator embeds in the dump file - // name: that timestamp comes from the backup Job pod's clock, not the - // apiserver's, and is truncated to seconds. - mariadbSizeProbeClockSkew = time.Minute -) - -type s3ObjectInfo struct { - Key string - Size int64 -} - -type s3ObjectLister interface { - ListObjects(ctx context.Context, bucket, prefix string) ([]s3ObjectInfo, error) -} +// Slack around the [created, completed] window of the operator Backup when +// matching the timestamp mariadb-operator embeds in the dump file name: that +// timestamp comes from the backup Job pod's clock, not the apiserver's, and +// is truncated to seconds. +const mariadbSizeProbeClockSkew = time.Minute // mariadbArtifactSizeBytes measures the dump mariadb-operator uploaded for // mdbBackup. The strategy prefix is shared by every backup of the @@ -53,14 +31,15 @@ func (r *BackupJobReconciler) mariadbArtifactSizeBytes( completedAt time.Time, ) int64 { logger := getLogger(ctx) - ctx, cancel := context.WithTimeout(ctx, mariadbSizeProbeTimeout) + ctx, cancel := context.WithTimeout(ctx, artifactSizeProbeTimeout) defer cancel() - newLister := r.newS3ObjectLister - if newLister == nil { - newLister = r.newMariaDBMinioLister + conn, err := r.mariadbS3Connection(ctx, namespace, s3) + if err != nil { + logger.Debug("cannot resolve S3 credentials to measure MariaDB backup size", "backup", mdbBackup.Name, "error", err) + return 0 } - lister, err := newLister(ctx, namespace, s3) + lister, err := r.s3Lister(conn) if err != nil { logger.Debug("cannot build S3 client to measure MariaDB backup size", "backup", mdbBackup.Name, "error", err) return 0 @@ -90,6 +69,37 @@ func (r *BackupJobReconciler) mariadbArtifactSizeBytes( return matched[0].Size } +// mariadbS3Connection resolves the Secrets and TLS settings the operator +// Backup was rendered with, so the bucket is read with the tenant's +// credentials rather than the controller's. +func (r *BackupJobReconciler) mariadbS3Connection(ctx context.Context, namespace string, s3 *strategyv1alpha1.MariaDBS3Template) (s3Connection, error) { + conn := s3Connection{ + Endpoint: s3.Endpoint, + Region: s3.Region, + Secure: s3.TLS != nil && s3.TLS.Enabled, + } + var err error + if conn.AccessKey, err = r.secretValue(ctx, namespace, s3.AccessKeyIDSecretKeyRef.Name, s3.AccessKeyIDSecretKeyRef.Key); err != nil { + return conn, err + } + if conn.SecretKey, err = r.secretValue(ctx, namespace, s3.SecretAccessKeySecretKeyRef.Name, s3.SecretAccessKeySecretKeyRef.Key); err != nil { + return conn, err + } + if ref := s3.SessionTokenSecretKeyRef; ref != nil { + if conn.SessionToken, err = r.secretValue(ctx, namespace, ref.Name, ref.Key); err != nil { + return conn, err + } + } + if conn.Secure && s3.TLS.CASecretKeyRef != nil { + ca, err := r.secretValue(ctx, namespace, s3.TLS.CASecretKeyRef.Name, s3.TLS.CASecretKeyRef.Key) + if err != nil { + return conn, err + } + conn.CAPEM = []byte(ca) + } + return conn, nil +} + // mariadbOperatorS3Prefix mirrors how mariadb-operator normalises // spec.storage.s3.prefix before prepending it to object keys. func mariadbOperatorS3Prefix(prefix string) string { @@ -119,75 +129,3 @@ func parseMariaDBLogicalBackupFile(name string) (time.Time, bool) { } return ts, true } - -type minioObjectLister struct { - client *minio.Client -} - -func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix string) ([]s3ObjectInfo, error) { - var out []s3ObjectInfo - for obj := range l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix}) { - if obj.Err != nil { - return nil, obj.Err - } - out = append(out, s3ObjectInfo{Key: obj.Key, Size: obj.Size}) - } - return out, nil -} - -// newMariaDBMinioLister builds an S3 client from the same Secrets and TLS -// settings the operator Backup was rendered with, so the controller reads -// the bucket with the tenant's credentials rather than its own. -func (r *BackupJobReconciler) newMariaDBMinioLister(ctx context.Context, namespace string, s3 *strategyv1alpha1.MariaDBS3Template) (s3ObjectLister, error) { - accessKey, err := r.mariadbSecretValue(ctx, namespace, s3.AccessKeyIDSecretKeyRef) - if err != nil { - return nil, err - } - secretKey, err := r.mariadbSecretValue(ctx, namespace, s3.SecretAccessKeySecretKeyRef) - if err != nil { - return nil, err - } - var sessionToken string - if s3.SessionTokenSecretKeyRef != nil { - if sessionToken, err = r.mariadbSecretValue(ctx, namespace, *s3.SessionTokenSecretKeyRef); err != nil { - return nil, err - } - } - - opts := &minio.Options{ - Creds: credentials.NewStaticV4(accessKey, secretKey, sessionToken), - Region: s3.Region, - Secure: s3.TLS != nil && s3.TLS.Enabled, - } - if opts.Secure && s3.TLS.CASecretKeyRef != nil { - caPEM, err := r.mariadbSecretValue(ctx, namespace, *s3.TLS.CASecretKeyRef) - if err != nil { - return nil, err - } - pool, err := x509.SystemCertPool() - if err != nil { - pool = x509.NewCertPool() - } - if !pool.AppendCertsFromPEM([]byte(caPEM)) { - return nil, fmt.Errorf("no PEM certificates in Secret %s/%s key %q", namespace, s3.TLS.CASecretKeyRef.Name, s3.TLS.CASecretKeyRef.Key) - } - opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}} - } - client, err := minio.New(s3.Endpoint, opts) - if err != nil { - return nil, err - } - return &minioObjectLister{client: client}, nil -} - -func (r *BackupJobReconciler) mariadbSecretValue(ctx context.Context, namespace string, ref strategyv1alpha1.MariaDBSecretKeySelector) (string, error) { - secret := &corev1.Secret{} - if err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: ref.Name}, secret); err != nil { - return "", err - } - v, ok := secret.Data[ref.Key] - if !ok { - return "", fmt.Errorf("Secret %s/%s has no key %q", namespace, ref.Name, ref.Key) - } - return string(v), nil -} diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index f7e450a26..214c7274c 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -14,32 +14,11 @@ import ( corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" "github.com/cozystack/cozystack/internal/backupcontroller/mariadbapp" "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" ) -type fakeS3Lister struct { - objects []s3ObjectInfo - err error - - bucket, prefix string -} - -func (f *fakeS3Lister) ListObjects(_ context.Context, bucket, prefix string) ([]s3ObjectInfo, error) { - f.bucket, f.prefix = bucket, prefix - return f.objects, f.err -} - -func reconcilerWithLister(l *fakeS3Lister) *BackupJobReconciler { - return &BackupJobReconciler{ - newS3ObjectLister: func(context.Context, string, *strategyv1alpha1.MariaDBS3Template) (s3ObjectLister, error) { - return l, nil - }, - } -} - func dumpKey(prefix string, ts time.Time, compression string) string { if compression == "" { return fmt.Sprintf("%sbackup.%s.sql", prefix, ts.UTC().Format(time.RFC3339)) @@ -93,10 +72,13 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4242}, {Key: prefix + "0-backup-target.txt", Size: 7}, }} - got := reconcilerWithLister(l).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed) + got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed) if got != 4242 { t.Errorf("size = %d, want 4242", got) } + if l.conn.AccessKey != "AKID" || l.conn.Endpoint != "s3.example:9000" { + t.Errorf("connection = %+v, want the Secret's key and the rendered endpoint", l.conn) + } if l.bucket != "bkt" || l.prefix != prefix { t.Errorf("listed %q/%q, want bkt/%q", l.bucket, l.prefix, prefix) } @@ -106,7 +88,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey(prefix, created.Add(-30*time.Second), ""), Size: 9}, }} - if got := reconcilerWithLister(l).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 9 { + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 9 { t.Errorf("size = %d, want 9", got) } }) @@ -116,7 +98,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 1}, {Key: dumpKey(prefix, created.Add(2*time.Minute), "gzip"), Size: 2}, }} - if got := reconcilerWithLister(l).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0 (ambiguous)", got) } }) @@ -125,14 +107,14 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey(prefix, completed.Add(time.Hour), "gzip"), Size: 1}, }} - if got := reconcilerWithLister(l).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0", got) } }) t.Run("list error leaves size unset", func(t *testing.T) { l := &fakeS3Lister{err: errors.New("AccessDenied")} - if got := reconcilerWithLister(l).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0", got) } }) @@ -165,14 +147,7 @@ func TestMariaDBArtifactSizeBytes_MinioClient(t *testing.T) { })) defer srv.Close() - creds := &corev1.Secret{ - ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "creds"}, - Data: map[string][]byte{ - "AWS_ACCESS_KEY_ID": []byte("AKID"), - "AWS_SECRET_ACCESS_KEY": []byte("secret"), - }, - } - c := newMariaDBStrategyTestClient(t, creds) + c := newMariaDBStrategyTestClient(t, s3CredsSecret("tenant", "creds")) r := &BackupJobReconciler{Client: c, Scheme: c.Scheme()} s3 := newRenderedMariaDBTemplate().Storage.S3 @@ -216,11 +191,9 @@ func TestCreateMariaDBBackupArtifact_SizeBytes(t *testing.T) { ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, }, } - c := newMariaDBStrategyTestClient(t, job) - r := reconcilerWithLister(&fakeS3Lister{objects: []s3ObjectInfo{ + r := reconcilerWithLister(t, &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey("src/", completed.Add(-time.Minute), "gzip"), Size: 2048}, - }}) - r.Client, r.Scheme = c, c.Scheme() + }}, job, s3CredsSecret("tenant", "creds")) rendered := newRenderedMariaDBTemplate() rendered.Storage.S3.Prefix = "src/" From f222e52863ff2432329c3971c8a3d1882a6e379d Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Wed, 21 Oct 2026 12:12:11 +0400 Subject: [PATCH 089/518] fix(backups): harden the artifact size probe The probe runs inside the reconcile that creates the Backup, and the BackupJob controller has a single worker, so its timeout delays every other BackupJob while an endpoint hangs. Lower it from 30s to 10s. List the MariaDB prefix non-recursively, as mariadb-operator does. The default MariaDB prefix and the default CNPG destinationPath are the same // path, so a recursive listing of a MariaDB backed up next to a same-named Postgres walked the whole WAL archive and ran out of time. The CNPG base-backup prefix is still listed recursively. Build the custom-CA transport from minio.DefaultTransport so it keeps the proxy-from-environment and dial/TLS timeout settings a bare http.Transport drops, and cover it with a test against a TLS endpoint with a self-signed certificate. Log the reasons the size could not be measured (credentials, client, listing, empty CNPG prefix) at info. The chart runs the controller at info, which hid them before. The expected ambiguity of several MariaDB dumps in the window stays at debug. Skip an endpointCA without a Secret name, as buildBarmanObjectStore does. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 32 +++++---- .../backupcontroller/artifact_size_test.go | 5 +- .../backupcontroller/cnpg_artifact_size.go | 14 ++-- .../cnpg_artifact_size_test.go | 71 ++++++++++++++----- .../backupcontroller/mariadb_artifact_size.go | 11 +-- .../mariadb_artifact_size_test.go | 34 +++++++-- 6 files changed, 119 insertions(+), 48 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 6ca2ad3ea..0c6d52f19 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -3,11 +3,9 @@ package backupcontroller import ( "context" - "crypto/tls" "crypto/x509" "errors" "fmt" - "net/http" "time" "github.com/minio/minio-go/v7" @@ -17,9 +15,11 @@ import ( ) // artifactSizeProbeTimeout bounds the object-store round trips a driver -// makes to measure Backup.status.artifact.sizeBytes; the probe runs inside -// the reconcile that creates the Backup. -const artifactSizeProbeTimeout = 30 * time.Second +// makes to measure Backup.status.artifact.sizeBytes. The probe runs inside +// the reconcile that creates the Backup, and the BackupJob controller has a +// single worker, so an unreachable endpoint stalls every other BackupJob for +// this long. +const artifactSizeProbeTimeout = 10 * time.Second type s3ObjectInfo struct { Key string @@ -27,7 +27,7 @@ type s3ObjectInfo struct { } type s3ObjectLister interface { - ListObjects(ctx context.Context, bucket, prefix string) ([]s3ObjectInfo, error) + ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) } // s3Connection is what a driver resolves from its rendered strategy to @@ -62,14 +62,22 @@ func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { Secure: conn.Secure, } if conn.Secure && len(conn.CAPEM) > 0 { - pool, err := x509.SystemCertPool() + // Start from minio's transport to keep its proxy and timeout settings. + tr, err := minio.DefaultTransport(true) if err != nil { - pool = x509.NewCertPool() + return nil, err + } + pool := tr.TLSClientConfig.RootCAs + if pool == nil { + if pool, err = x509.SystemCertPool(); err != nil { + pool = x509.NewCertPool() + } } if !pool.AppendCertsFromPEM(conn.CAPEM) { return nil, errors.New("no PEM certificates in the endpoint CA bundle") } - opts.Transport = &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12}} + tr.TLSClientConfig.RootCAs = pool + opts.Transport = tr } client, err := minio.New(conn.Endpoint, opts) if err != nil { @@ -78,9 +86,9 @@ func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { return &minioObjectLister{client: client}, nil } -func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix string) ([]s3ObjectInfo, error) { +func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) { var out []s3ObjectInfo - for obj := range l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: true}) { + for obj := range l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: recursive}) { if obj.Err != nil { return nil, obj.Err } @@ -96,7 +104,7 @@ func (r *BackupJobReconciler) secretValue(ctx context.Context, namespace, name, } v, ok := secret.Data[key] if !ok { - return "", fmt.Errorf("Secret %s/%s has no key %q", namespace, name, key) + return "", fmt.Errorf("secret %s/%s has no key %q", namespace, name, key) } return string(v), nil } diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go index 6e389cf66..c0a957852 100644 --- a/internal/backupcontroller/artifact_size_test.go +++ b/internal/backupcontroller/artifact_size_test.go @@ -16,10 +16,11 @@ type fakeS3Lister struct { conn s3Connection bucket, prefix string + recursive bool } -func (f *fakeS3Lister) ListObjects(_ context.Context, bucket, prefix string) ([]s3ObjectInfo, error) { - f.bucket, f.prefix = bucket, prefix +func (f *fakeS3Lister) ListObjects(_ context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) { + f.bucket, f.prefix, f.recursive = bucket, prefix, recursive return f.objects, f.err } diff --git a/internal/backupcontroller/cnpg_artifact_size.go b/internal/backupcontroller/cnpg_artifact_size.go index bff2d5b39..61b00d5e9 100644 --- a/internal/backupcontroller/cnpg_artifact_size.go +++ b/internal/backupcontroller/cnpg_artifact_size.go @@ -32,23 +32,23 @@ func (r *BackupJobReconciler) cnpgArtifactSizeBytes( bucket, path, err := parseBarmanS3DestinationPath(store.DestinationPath) if err != nil { - logger.Debug("cannot measure CNPG backup size", "backupID", backupID, "error", err) + logger.Info("cannot measure CNPG backup size", "backupID", backupID, "error", err) return 0 } conn, err := r.cnpgS3Connection(ctx, namespace, store) if err != nil { - logger.Debug("cannot resolve S3 credentials to measure CNPG backup size", "backupID", backupID, "error", err) + logger.Info("cannot resolve S3 credentials to measure CNPG backup size", "backupID", backupID, "error", err) return 0 } lister, err := r.s3Lister(conn) if err != nil { - logger.Debug("cannot build S3 client to measure CNPG backup size", "backupID", backupID, "error", err) + logger.Info("cannot build S3 client to measure CNPG backup size", "backupID", backupID, "error", err) return 0 } prefix := barmanBaseBackupPrefix(path, serverName, backupID) - objects, err := lister.ListObjects(ctx, bucket, prefix) + objects, err := lister.ListObjects(ctx, bucket, prefix, true) if err != nil { - logger.Debug("cannot list CNPG base backup objects to measure size", "backupID", backupID, "error", err) + logger.Info("cannot list CNPG base backup objects to measure size", "backupID", backupID, "error", err) return 0 } var total int64 @@ -56,7 +56,7 @@ func (r *BackupJobReconciler) cnpgArtifactSizeBytes( total += obj.Size } if total == 0 { - logger.Debug("no CNPG base backup objects found; leaving size unset", "backupID", backupID, "prefix", prefix) + logger.Info("no CNPG base backup objects found; leaving size unset", "backupID", backupID, "prefix", prefix) } return total } @@ -91,7 +91,7 @@ func (r *BackupJobReconciler) cnpgS3Connection(ctx context.Context, namespace st if conn.SecretKey, err = r.secretValue(ctx, namespace, creds.SecretRef.Name, secretKeyKey); err != nil { return conn, err } - if ca := store.EndpointCA; ca != nil && conn.Secure { + if ca := store.EndpointCA; ca != nil && ca.SecretRef.Name != "" && conn.Secure { key := ca.Key if key == "" { key = defaultEndpointCAKey diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index 29f09253b..81d850f64 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -3,6 +3,7 @@ package backupcontroller import ( "context" + "encoding/pem" "errors" "fmt" "net/http" @@ -74,8 +75,8 @@ func TestCNPGArtifactSizeBytes(t *testing.T) { if got != 1231 { t.Errorf("size = %d, want 1231", got) } - if l.bucket != "bkt" || l.prefix != prefix { - t.Errorf("listed %q/%q, want bkt/%q", l.bucket, l.prefix, prefix) + if l.bucket != "bkt" || l.prefix != prefix || !l.recursive { + t.Errorf("listed %q/%q recursive=%v, want bkt/%q recursive", l.bucket, l.prefix, l.recursive, prefix) } if l.conn.Endpoint != "s3.example:8443" || !l.conn.Secure || l.conn.AccessKey != "AKID" || l.conn.SecretKey != "secret" { t.Errorf("connection = %+v", l.conn) @@ -161,21 +162,10 @@ func TestCNPGArtifactSizeBytes(t *testing.T) { // and response parsing are exercised together. func TestCNPGArtifactSizeBytes_MinioClient(t *testing.T) { const prefix = "tenant/pg/srv/base/20261021T100000/" - var gotPath, gotPrefix, gotAuth string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { - if _, ok := req.URL.Query()["location"]; ok { - w.Header().Set("Content-Type", "application/xml") - fmt.Fprint(w, ``) - return - } + var gotPath, gotPrefix, gotDelimiter, gotAuth string + srv := httptest.NewServer(cnpgListStub(prefix, func(req *http.Request) { gotPath, gotPrefix, gotAuth = req.URL.Path, req.URL.Query().Get("prefix"), req.Header.Get("Authorization") - w.Header().Set("Content-Type", "application/xml") - fmt.Fprintf(w, ` - - bkt%[1]s21000false - %[1]sbackup.info7"a"2026-10-21T10:01:00.000Z - %[1]sdata.tar.gz4096"b"2026-10-21T10:01:00.000Z -`, prefix) + gotDelimiter = req.URL.Query().Get("delimiter") })) defer srv.Close() @@ -187,14 +177,59 @@ func TestCNPGArtifactSizeBytes_MinioClient(t *testing.T) { if got := r.cnpgArtifactSizeBytes(context.Background(), "tenant", store, "srv", "20261021T100000"); got != 4103 { t.Fatalf("size = %d, want 4103", got) } - if gotPath != "/bkt/" || gotPrefix != prefix { - t.Errorf("request path=%q prefix=%q, want /bkt/ and %q", gotPath, gotPrefix, prefix) + if gotPath != "/bkt/" || gotPrefix != prefix || gotDelimiter != "" { + t.Errorf("request path=%q prefix=%q delimiter=%q, want /bkt/, %q and a recursive listing", gotPath, gotPrefix, gotDelimiter, prefix) } if !strings.Contains(gotAuth, "Credential=AKID/") { t.Errorf("request not signed with the Secret's access key: %q", gotAuth) } } +func cnpgListStub(prefix string, seen func(*http.Request)) http.HandlerFunc { + return func(w http.ResponseWriter, req *http.Request) { + w.Header().Set("Content-Type", "application/xml") + if _, ok := req.URL.Query()["location"]; ok { + fmt.Fprint(w, ``) + return + } + if seen != nil { + seen(req) + } + fmt.Fprintf(w, ` + + bkt%[1]s21000false + %[1]sbackup.info7"a"2026-10-21T10:01:00.000Z + %[1]sdata.tar.gz4096"b"2026-10-21T10:01:00.000Z +`, prefix) + } +} + +// TestCNPGArtifactSizeBytes_EndpointCA serves the listing over TLS with a +// self-signed certificate: the probe must trust it only through endpointCA. +func TestCNPGArtifactSizeBytes_EndpointCA(t *testing.T) { + const prefix = "tenant/pg/srv/base/20261021T100000/" + srv := httptest.NewTLSServer(cnpgListStub(prefix, nil)) + defer srv.Close() + caPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: srv.Certificate().Raw}) + + ca := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "s3-ca"}, + Data: map[string][]byte{"ca.crt": caPEM}, + } + c := newMariaDBStrategyTestClient(t, s3CredsSecret("tenant", "creds"), ca) + r := &BackupJobReconciler{Client: c, Scheme: c.Scheme()} + store := newSizedBarmanObjectStore() + store.EndpointURL = srv.URL + + if got := r.cnpgArtifactSizeBytes(context.Background(), "tenant", store, "srv", "20261021T100000"); got != 0 { + t.Errorf("without endpointCA: size = %d, want 0 (untrusted certificate)", got) + } + store.EndpointCA = &strategyv1alpha1.EndpointCARef{SecretRef: corev1.LocalObjectReference{Name: "s3-ca"}} + if got := r.cnpgArtifactSizeBytes(context.Background(), "tenant", store, "srv", "20261021T100000"); got != 4103 { + t.Errorf("with endpointCA: size = %d, want 4103", got) + } +} + func TestCreateCNPGBackupArtifact_SizeBytes(t *testing.T) { apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup strategyGroup := strategyv1alpha1.GroupVersion.Group diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index 20f68a454..b13e7ff1f 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -36,18 +36,21 @@ func (r *BackupJobReconciler) mariadbArtifactSizeBytes( conn, err := r.mariadbS3Connection(ctx, namespace, s3) if err != nil { - logger.Debug("cannot resolve S3 credentials to measure MariaDB backup size", "backup", mdbBackup.Name, "error", err) + logger.Info("cannot resolve S3 credentials to measure MariaDB backup size", "backup", mdbBackup.Name, "error", err) return 0 } lister, err := r.s3Lister(conn) if err != nil { - logger.Debug("cannot build S3 client to measure MariaDB backup size", "backup", mdbBackup.Name, "error", err) + logger.Info("cannot build S3 client to measure MariaDB backup size", "backup", mdbBackup.Name, "error", err) return 0 } + // Non-recursive, as mariadb-operator lists it: the default prefix is also + // the default CNPG destinationPath, so an app of each kind with the same + // name shares it and a recursive listing would walk the WAL archive. prefix := mariadbOperatorS3Prefix(s3.Prefix) - objects, err := lister.ListObjects(ctx, s3.Bucket, prefix) + objects, err := lister.ListObjects(ctx, s3.Bucket, prefix, false) if err != nil { - logger.Debug("cannot list MariaDB backup objects to measure size", "backup", mdbBackup.Name, "error", err) + logger.Info("cannot list MariaDB backup objects to measure size", "backup", mdbBackup.Name, "error", err) return 0 } diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index 214c7274c..0260cb14e 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -14,6 +14,7 @@ import ( corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" "github.com/cozystack/cozystack/internal/backupcontroller/mariadbapp" "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" @@ -79,8 +80,29 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { if l.conn.AccessKey != "AKID" || l.conn.Endpoint != "s3.example:9000" { t.Errorf("connection = %+v, want the Secret's key and the rendered endpoint", l.conn) } - if l.bucket != "bkt" || l.prefix != prefix { - t.Errorf("listed %q/%q, want bkt/%q", l.bucket, l.prefix, prefix) + if l.bucket != "bkt" || l.prefix != prefix || l.recursive { + t.Errorf("listed %q/%q recursive=%v, want bkt/%q non-recursive", l.bucket, l.prefix, l.recursive, prefix) + } + }) + + t.Run("reads the endpoint CA from its Secret", func(t *testing.T) { + tlsS3 := newRenderedMariaDBTemplate().Storage.S3 + tlsS3.Prefix = "tenant/db" + tlsS3.TLS = &strategyv1alpha1.MariaDBS3TLS{ + Enabled: true, + CASecretKeyRef: &strategyv1alpha1.MariaDBSecretKeySelector{Name: "ca", Key: "ca.crt"}, + } + ca := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "ca"}, + Data: map[string][]byte{"ca.crt": []byte("PEM")}, + } + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 3}}} + r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds"), ca) + if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", tlsS3, mdbBackup, completed); got != 3 { + t.Errorf("size = %d, want 3", got) + } + if !l.conn.Secure || string(l.conn.CAPEM) != "PEM" { + t.Errorf("connection = %+v, want TLS with the Secret's CA", l.conn) } }) @@ -135,14 +157,16 @@ func TestMariaDBArtifactSizeBytes_MinioClient(t *testing.T) { created := time.Date(2026, 10, 21, 10, 0, 0, 0, time.UTC) key := dumpKey("tenant/db/", created.Add(time.Minute), "gzip") - var gotPath, gotPrefix, gotAuth string + var gotPath, gotPrefix, gotDelimiter, gotAuth string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { gotPath, gotPrefix, gotAuth = req.URL.Path, req.URL.Query().Get("prefix"), req.Header.Get("Authorization") + gotDelimiter = req.URL.Query().Get("delimiter") w.Header().Set("Content-Type", "application/xml") fmt.Fprintf(w, ` bkttenant/db/11000false %s31337"x"2026-10-21T10:01:00.000Z + tenant/db/srv/ `, key) })) defer srv.Close() @@ -161,8 +185,8 @@ func TestMariaDBArtifactSizeBytes_MinioClient(t *testing.T) { if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, created.Add(3*time.Minute)); got != 31337 { t.Fatalf("size = %d, want 31337", got) } - if gotPath != "/bkt/" || gotPrefix != "tenant/db/" { - t.Errorf("request path=%q prefix=%q, want /bkt/ and tenant/db/", gotPath, gotPrefix) + if gotPath != "/bkt/" || gotPrefix != "tenant/db/" || gotDelimiter != "/" { + t.Errorf("request path=%q prefix=%q delimiter=%q, want /bkt/, tenant/db/ and a non-recursive listing", gotPath, gotPrefix, gotDelimiter) } if !strings.Contains(gotAuth, "Credential=AKID/") { t.Errorf("request not signed with the Secret's access key: %q", gotAuth) From 038f35376ab269d75e18cb6e56450bd7e616b06b Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Wed, 21 Oct 2026 12:44:23 +0400 Subject: [PATCH 090/518] fix(backups): do not report a partial listing or re-probe on retry minio-go ends a listing whose context is cancelled between pages by closing the channel without an error item. When the probe timeout expired between pages of a large CNPG base backup, the pages already read were summed and reported as the measured size. Check the context once the listing channel closes and leave the size unset instead. The probe ran before the Backup was created, so a reconcile that retried after the Backup already existed probed the bucket again, held the single BackupJob worker for up to the timeout, and then discarded the result in favour of the existing object. Return an existing Backup before probing. Log the ambiguous MariaDB case (several dumps in the window) at info like the other reasons the size stays unset, and cover the session-token credential path with a test. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 30 ++++++++++- .../backupcontroller/artifact_size_test.go | 32 ++++++++++++ .../cnpg_artifact_size_test.go | 25 ++++++++- .../cnpgstrategy_controller.go | 3 ++ .../backupcontroller/mariadb_artifact_size.go | 2 +- .../mariadb_artifact_size_test.go | 51 +++++++++++++++++++ .../mariadbstrategy_controller.go | 3 ++ 7 files changed, 143 insertions(+), 3 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 0c6d52f19..6d65284a7 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -11,7 +11,10 @@ import ( "github.com/minio/minio-go/v7" "github.com/minio/minio-go/v7/pkg/credentials" corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/types" + + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" ) // artifactSizeProbeTimeout bounds the object-store round trips a driver @@ -87,16 +90,41 @@ func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { } func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) { + return collectObjects(ctx, l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: recursive})) +} + +// collectObjects drains a minio listing. minio-go ends a listing whose +// context is cancelled between pages by closing the channel without an +// error item, so a cancelled context is checked once the channel closes: +// otherwise a timeout would pass a partial listing off as complete. +func collectObjects(ctx context.Context, objects <-chan minio.ObjectInfo) ([]s3ObjectInfo, error) { var out []s3ObjectInfo - for obj := range l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: recursive}) { + for obj := range objects { if obj.Err != nil { return nil, obj.Err } out = append(out, s3ObjectInfo{Key: obj.Key, Size: obj.Size}) } + if err := ctx.Err(); err != nil { + return nil, err + } return out, nil } +// existingBackup returns the Backup a previous reconcile already created +// for the job, so a retry skips the size probe whose result it would +// discard. Returns (nil, nil) when there is none. +func (r *BackupJobReconciler) existingBackup(ctx context.Context, namespace, name string) (*backupsv1alpha1.Backup, error) { + existing := &backupsv1alpha1.Backup{} + if err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, existing); err != nil { + if apierrors.IsNotFound(err) { + return nil, nil + } + return nil, err + } + return existing, nil +} + func (r *BackupJobReconciler) secretValue(ctx context.Context, namespace, name, key string) (string, error) { secret := &corev1.Secret{} if err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, secret); err != nil { diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go index c0a957852..cfe95d16d 100644 --- a/internal/backupcontroller/artifact_size_test.go +++ b/internal/backupcontroller/artifact_size_test.go @@ -3,8 +3,11 @@ package backupcontroller import ( "context" + "errors" "testing" + "github.com/minio/minio-go/v7" + corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client" @@ -17,9 +20,11 @@ type fakeS3Lister struct { conn s3Connection bucket, prefix string recursive bool + calls int } func (f *fakeS3Lister) ListObjects(_ context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) { + f.calls++ f.bucket, f.prefix, f.recursive = bucket, prefix, recursive return f.objects, f.err } @@ -48,3 +53,30 @@ func reconcilerWithLister(t *testing.T, l *fakeS3Lister, objs ...client.Object) }, } } + +func TestCollectObjects(t *testing.T) { + listing := func() <-chan minio.ObjectInfo { + ch := make(chan minio.ObjectInfo, 1) + ch <- minio.ObjectInfo{Key: "a", Size: 1} + close(ch) + return ch + } + + got, err := collectObjects(context.Background(), listing()) + if err != nil || len(got) != 1 || got[0].Size != 1 { + t.Fatalf("collectObjects = %v, %v; want one object", got, err) + } + + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if got, err := collectObjects(ctx, listing()); !errors.Is(err, context.Canceled) { + t.Errorf("listing closed after cancellation: got %v, %v; want context.Canceled", got, err) + } + + errCh := make(chan minio.ObjectInfo, 1) + errCh <- minio.ObjectInfo{Err: errors.New("AccessDenied")} + close(errCh) + if _, err := collectObjects(context.Background(), errCh); err == nil { + t.Error("listing error was not returned") + } +} diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index 81d850f64..2db8bc084 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -248,7 +248,7 @@ func TestCreateCNPGBackupArtifact_SizeBytes(t *testing.T) { } rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()} l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: "tenant/pg/srv/base/20261021T100000/data.tar.gz", Size: 777}}} - r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")) + r := reconcilerWithLister(t, l) r.Client = newCNPGStrategyTestClient(t, s3CredsSecret("tenant", "creds")) got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")) @@ -262,3 +262,26 @@ func TestCreateCNPGBackupArtifact_SizeBytes(t *testing.T) { t.Errorf("listed prefix %q does not use the effective serverName", l.prefix) } } + +func TestCreateCNPGBackupArtifact_ExistingBackupSkipsProbe(t *testing.T) { + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-retry"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"}, + }, + } + existing := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-retry"}} + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: "x", Size: 1}}} + r := reconcilerWithLister(t, l) + r.Client = newCNPGStrategyTestClient(t, existing, s3CredsSecret("tenant", "creds")) + cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}} + rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()} + + if _, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")); err != nil { + t.Fatalf("createCNPGBackupArtifact: %v", err) + } + if l.calls != 0 { + t.Errorf("S3 listed %d times for a Backup that already exists", l.calls) + } +} diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 3b62ce6e1..50e9dd8ab 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -815,6 +815,9 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact( rendered *strategyv1alpha1.CNPGTemplate, sourceApp *postgresapp.Postgres, ) (*backupsv1alpha1.Backup, error) { + if existing, err := r.existingBackup(ctx, j.Namespace, j.Name); err != nil || existing != nil { + return existing, err + } takenAt := metav1.Now() if cnpgBackup.Status.StartedAt != nil && !cnpgBackup.Status.StartedAt.IsZero() { takenAt = *cnpgBackup.Status.StartedAt diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index b13e7ff1f..86d31694a 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -65,7 +65,7 @@ func (r *BackupJobReconciler) mariadbArtifactSizeBytes( matched = append(matched, obj) } if len(matched) != 1 { - logger.Debug("MariaDB backup dump not attributable to a single object; leaving size unset", + logger.Info("MariaDB backup dump not attributable to a single object; leaving size unset", "backup", mdbBackup.Name, "candidates", len(matched)) return 0 } diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index 0260cb14e..d068d9898 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -106,6 +106,24 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { } }) + t.Run("reads the session token from its Secret", func(t *testing.T) { + stsS3 := newRenderedMariaDBTemplate().Storage.S3 + stsS3.Prefix = "tenant/db" + stsS3.SessionTokenSecretKeyRef = &strategyv1alpha1.MariaDBSecretKeySelector{Name: "sts", Key: "token"} + sts := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "sts"}, + Data: map[string][]byte{"token": []byte("TOKEN")}, + } + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4}}} + r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds"), sts) + if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", stsS3, mdbBackup, completed); got != 4 { + t.Errorf("size = %d, want 4", got) + } + if l.conn.SessionToken != "TOKEN" { + t.Errorf("session token = %q, want the Secret's", l.conn.SessionToken) + } + }) + t.Run("pod clock skew within tolerance", func(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey(prefix, created.Add(-30*time.Second), ""), Size: 9}, @@ -229,3 +247,36 @@ func TestCreateMariaDBBackupArtifact_SizeBytes(t *testing.T) { t.Fatalf("Artifact = %#v, want SizeBytes 2048", artefact.Status.Artifact) } } + +func TestCreateMariaDBBackupArtifact_ExistingBackupSkipsProbe(t *testing.T) { + apps := mariadbapp.GroupName + job := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-retry"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, + }, + } + existing := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-retry"}, + Status: backupsv1alpha1.BackupStatus{Artifact: &backupsv1alpha1.BackupArtifact{URI: "s3://bkt/src/op-bk", SizeBytes: 99}}, + } + l := &fakeS3Lister{} + r := reconcilerWithLister(t, l, job, existing, s3CredsSecret("tenant", "creds")) + rendered := newRenderedMariaDBTemplate() + rendered.Storage.S3.Prefix = "src/" + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "mariadb-strategy-default"}, + Parameters: map[string]string{}, + } + + got, err := r.createMariaDBBackupArtifact(context.Background(), job, resolved, &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "op-bk"}}, rendered) + if err != nil { + t.Fatalf("createMariaDBBackupArtifact: %v", err) + } + if l.calls != 0 { + t.Errorf("S3 listed %d times for a Backup that already exists", l.calls) + } + if got.Status.Artifact == nil || got.Status.Artifact.SizeBytes != 99 { + t.Errorf("Artifact = %#v, want the existing one", got.Status.Artifact) + } +} diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 84fe0c2df..bdffc7e7c 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -365,6 +365,9 @@ func (r *BackupJobReconciler) createMariaDBBackupArtifact( mdbBackup *mariadbtypes.Backup, rendered *strategyv1alpha1.MariaDBTemplate, ) (*backupsv1alpha1.Backup, error) { + if existing, err := r.existingBackup(ctx, j.Namespace, j.Name); err != nil || existing != nil { + return existing, err + } takenAt := metav1.Now() if cond := apimeta.FindStatusCondition(mdbBackup.Status.Conditions, mariadbtypes.ConditionTypeComplete); cond != nil && !cond.LastTransitionTime.IsZero() { takenAt = cond.LastTransitionTime From 3b86f578035006b392b672d90d8662644d910991 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Wed, 21 Oct 2026 20:12:38 +0400 Subject: [PATCH 091/518] fix(backups): refuse to probe an S3 endpoint behind a path prefix An endpoint such as https://s3.example.org/prefix cannot be addressed by minio-go, which has no path-prefix option, and the prefix cannot be added below the client either, because the SigV4 signature covers the request path. The CNPG probe was worse than failing: it kept only the host of endpointURL and listed the bucket at a location barman never writes to. Reject such endpoints with an explicit error, in the CNPG probe and in the shared client, so the size is left unset instead of measured somewhere else. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 9 ++++++++- internal/backupcontroller/artifact_size_test.go | 9 +++++++++ internal/backupcontroller/cnpg_artifact_size.go | 5 +++++ internal/backupcontroller/cnpg_artifact_size_test.go | 8 ++++++++ 4 files changed, 30 insertions(+), 1 deletion(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 6d65284a7..7a5fec8f8 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -6,6 +6,7 @@ import ( "crypto/x509" "errors" "fmt" + "strings" "time" "github.com/minio/minio-go/v7" @@ -36,7 +37,7 @@ type s3ObjectLister interface { // s3Connection is what a driver resolves from its rendered strategy to // read the bucket with the same identity the backup was written with. type s3Connection struct { - // Endpoint is host[:port], without a scheme. + // Endpoint is host[:port], without a scheme or a path. Endpoint string Secure bool Region string @@ -59,6 +60,12 @@ type minioObjectLister struct { } func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { + // An endpoint behind a path prefix cannot be addressed: minio-go has no + // such option, and the SigV4 signature covers the request path, so the + // prefix cannot be added below the client either. + if strings.Contains(conn.Endpoint, "/") { + return nil, fmt.Errorf("endpoint %q has a path, which the S3 client cannot address", conn.Endpoint) + } opts := &minio.Options{ Creds: credentials.NewStaticV4(conn.AccessKey, conn.SecretKey, conn.SessionToken), Region: conn.Region, diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go index cfe95d16d..5b636a44b 100644 --- a/internal/backupcontroller/artifact_size_test.go +++ b/internal/backupcontroller/artifact_size_test.go @@ -80,3 +80,12 @@ func TestCollectObjects(t *testing.T) { t.Error("listing error was not returned") } } + +func TestNewMinioObjectLister_RejectsEndpointPath(t *testing.T) { + if _, err := newMinioObjectLister(s3Connection{Endpoint: "s3.example.org/prefix", Secure: true}); err == nil { + t.Error("an endpoint with a path was accepted") + } + if _, err := newMinioObjectLister(s3Connection{Endpoint: "s3.example.org:9000", Secure: true}); err != nil { + t.Errorf("host:port endpoint rejected: %v", err) + } +} diff --git a/internal/backupcontroller/cnpg_artifact_size.go b/internal/backupcontroller/cnpg_artifact_size.go index 61b00d5e9..c7d91bdb2 100644 --- a/internal/backupcontroller/cnpg_artifact_size.go +++ b/internal/backupcontroller/cnpg_artifact_size.go @@ -73,6 +73,11 @@ func (r *BackupJobReconciler) cnpgS3Connection(ctx context.Context, namespace st if err != nil || u.Host == "" { return conn, fmt.Errorf("invalid endpointURL %q", store.EndpointURL) } + // Dropping the path would probe the bucket at another location than + // the one barman writes to. + if strings.Trim(u.Path, "/") != "" { + return conn, fmt.Errorf("endpointURL %q has a path, which the S3 client cannot address", store.EndpointURL) + } conn.Endpoint, conn.Secure = u.Host, u.Scheme == "https" } diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index 2db8bc084..de3960261 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -141,6 +141,14 @@ func TestCNPGArtifactSizeBytes(t *testing.T) { }, backupID: "id", lister: &fakeS3Lister{objects: []s3ObjectInfo{{Key: "x", Size: 1}}}, }, + "endpointURL with a path": { + store: func() *strategyv1alpha1.BarmanObjectStoreTemplate { + s := newSizedBarmanObjectStore() + s.EndpointURL = "https://s3.example:8443/prefix" + return s + }, + backupID: "20261021T100000", lister: &fakeS3Lister{objects: []s3ObjectInfo{{Key: "x", Size: 1}}}, + }, } for name, tc := range unset { t.Run(name+" leaves size unset", func(t *testing.T) { From a3d854739d17602575ed6cb2ffa1a885c0892d01 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Thu, 22 Oct 2026 17:30:32 +0400 Subject: [PATCH 092/518] fix(backups): attribute MariaDB dumps on two clocks and read retries uncached The MariaDB probe identified this run's dump by the timestamp in its file name, which comes from the backup pod's clock. With that clock skewed past the slack, this run's dump falls outside the window while a neighbouring run's dump can be the single match left, and its size is reported as this backup's. A dump now has to fall inside the window by both its name and its LastModified, which the object store stamps, so a single skewed clock leaves the size unset instead. The name parser also accepts the layout later mariadb-operator releases write for compressed dumps, backup..sql., and checks the compression token in either layout, so an operator upgrade does not silently stop the measurement. The retry check and the AlreadyExists refetch read the Backup through the cache. A retry usually follows a Create made moments earlier; a cache that does not show it yet turned the refetch into NotFound and failed a backup that had succeeded. Read it from the API server instead. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 13 ++++--- .../backupcontroller/artifact_size_test.go | 17 +++++++++ .../cnpg_artifact_size_test.go | 13 +++++++ .../cnpgstrategy_controller.go | 2 +- .../backupcontroller/mariadb_artifact_size.go | 37 ++++++++++++++----- .../mariadb_artifact_size_test.go | 36 +++++++++++++----- .../mariadbstrategy_controller.go | 2 +- 7 files changed, 94 insertions(+), 26 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 7a5fec8f8..517fe07a6 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -26,8 +26,9 @@ import ( const artifactSizeProbeTimeout = 10 * time.Second type s3ObjectInfo struct { - Key string - Size int64 + Key string + Size int64 + LastModified time.Time } type s3ObjectLister interface { @@ -110,7 +111,7 @@ func collectObjects(ctx context.Context, objects <-chan minio.ObjectInfo) ([]s3O if obj.Err != nil { return nil, obj.Err } - out = append(out, s3ObjectInfo{Key: obj.Key, Size: obj.Size}) + out = append(out, s3ObjectInfo{Key: obj.Key, Size: obj.Size, LastModified: obj.LastModified}) } if err := ctx.Err(); err != nil { return nil, err @@ -120,10 +121,12 @@ func collectObjects(ctx context.Context, objects <-chan minio.ObjectInfo) ([]s3O // existingBackup returns the Backup a previous reconcile already created // for the job, so a retry skips the size probe whose result it would -// discard. Returns (nil, nil) when there is none. +// discard. Returns (nil, nil) when there is none. It reads past the cache: +// a retry typically follows a Create made moments earlier, which the cache +// may not show yet. func (r *BackupJobReconciler) existingBackup(ctx context.Context, namespace, name string) (*backupsv1alpha1.Backup, error) { existing := &backupsv1alpha1.Backup{} - if err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, existing); err != nil { + if err := r.apiReader().Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, existing); err != nil { if apierrors.IsNotFound(err) { return nil, nil } diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go index 5b636a44b..f24079f8b 100644 --- a/internal/backupcontroller/artifact_size_test.go +++ b/internal/backupcontroller/artifact_size_test.go @@ -11,6 +11,8 @@ import ( corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client" + + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" ) type fakeS3Lister struct { @@ -89,3 +91,18 @@ func TestNewMinioObjectLister_RejectsEndpointPath(t *testing.T) { t.Errorf("host:port endpoint rejected: %v", err) } } + +// TestExistingBackup_ReadsPastTheCache pins that the retry check sees a +// Backup the cache does not show yet: the previous reconcile created it +// moments earlier. +func TestExistingBackup_ReadsPastTheCache(t *testing.T) { + backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj"}} + r := &BackupJobReconciler{ + Client: newMariaDBStrategyTestClient(t), + APIReader: newMariaDBStrategyTestClient(t, backup), + } + got, err := r.existingBackup(context.Background(), "tenant", "bj") + if err != nil || got == nil { + t.Fatalf("existingBackup = %v, %v; want the Backup the API server holds", got, err) + } +} diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index de3960261..13b647bdd 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -83,6 +83,19 @@ func TestCNPGArtifactSizeBytes(t *testing.T) { } }) + t.Run("an empty endpointURL means AWS", func(t *testing.T) { + store := newSizedBarmanObjectStore() + store.EndpointURL = "" + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: prefix + "data.tar", Size: 5}}} + r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")) + if got := r.cnpgArtifactSizeBytes(context.Background(), "tenant", store, "srv", "20261021T100000"); got != 5 { + t.Errorf("size = %d, want 5", got) + } + if l.conn.Endpoint != "s3.amazonaws.com" || !l.conn.Secure { + t.Errorf("connection = %+v, want s3.amazonaws.com over TLS", l.conn) + } + }) + t.Run("honours custom credential keys and the endpoint CA", func(t *testing.T) { store := newSizedBarmanObjectStore() store.S3Credentials.AccessKeyIDKey = "id" diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 50e9dd8ab..5dabfa66c 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -876,7 +876,7 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact( return nil, err } existing := &backupsv1alpha1.Backup{} - if getErr := r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: backup.Name}, existing); getErr != nil { + if getErr := r.apiReader().Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: backup.Name}, existing); getErr != nil { return nil, getErr } return existing, nil diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index 86d31694a..273eb8105 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -11,15 +11,17 @@ import ( ) // Slack around the [created, completed] window of the operator Backup when -// matching the timestamp mariadb-operator embeds in the dump file name: that -// timestamp comes from the backup Job pod's clock, not the apiserver's, and -// is truncated to seconds. +// matching a dump: the timestamp in its file name comes from the backup Job +// pod's clock and is truncated to seconds, and its LastModified comes from +// the object store's clock, neither of which is the apiserver's. const mariadbSizeProbeClockSkew = time.Minute // mariadbArtifactSizeBytes measures the dump mariadb-operator uploaded for // mdbBackup. The strategy prefix is shared by every backup of the -// application, so the dump is identified by the timestamp in its file name -// falling inside the operator Backup's lifetime. Returns 0 - leaving +// application, so the dump is identified by both the timestamp in its file +// name and its LastModified falling inside the operator Backup's lifetime: +// the two come from different clocks, so a skewed pod clock alone cannot +// pull a neighbouring run's dump into the window. Returns 0 - leaving // Artifact.SizeBytes absent - when the object store cannot be read or the // window does not match exactly one dump: an unknown size must not be // reported as a measured one. @@ -59,7 +61,7 @@ func (r *BackupJobReconciler) mariadbArtifactSizeBytes( var matched []s3ObjectInfo for _, obj := range objects { ts, ok := parseMariaDBLogicalBackupFile(strings.TrimPrefix(obj.Key, prefix)) - if !ok || ts.Before(lower) || ts.After(upper) { + if !ok || !inWindow(ts, lower, upper) || !inWindow(obj.LastModified, lower, upper) { continue } matched = append(matched, obj) @@ -115,15 +117,30 @@ func mariadbOperatorS3Prefix(prefix string) string { return prefix } +func inWindow(t, lower, upper time.Time) bool { + return !t.Before(lower) && !t.After(upper) +} + +// Compression tokens mariadb-operator puts in a dump file name: the +// algorithm before .sql in v25.10 (backup..gzip.sql), the extension +// after it in later releases (backup..sql.gz). +var ( + mariadbDumpAlgorithms = map[string]bool{"none": true, "bzip2": true, "gzip": true, "zstd": true} + mariadbDumpExtensions = map[string]bool{"bz2": true, "gz": true, "zst": true} +) + // parseMariaDBLogicalBackupFile accepts the logical-backup file names -// mariadb-operator writes, backup..sql and -// backup...sql, and returns the embedded timestamp. +// mariadb-operator writes and returns the embedded timestamp. func parseMariaDBLogicalBackupFile(name string) (time.Time, bool) { parts := strings.Split(name, ".") - if len(parts) != 3 && len(parts) != 4 { + if parts[0] != "backup" { return time.Time{}, false } - if parts[0] != "backup" || parts[len(parts)-1] != "sql" { + switch { + case len(parts) == 3 && parts[2] == "sql": + case len(parts) == 4 && parts[2] == "sql" && mariadbDumpExtensions[parts[3]]: + case len(parts) == 4 && parts[3] == "sql" && mariadbDumpAlgorithms[parts[2]]: + default: return time.Time{}, false } ts, err := time.Parse(time.RFC3339, parts[1]) diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index d068d9898..6f9df56e7 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -33,6 +33,11 @@ func TestParseMariaDBLogicalBackupFile(t *testing.T) { "backup.2026-10-21T10:00:00Z.gzip.sql": true, "backup.2026-10-21T10:00:00+04:00.sql": true, "backup.2026-10-21T10:00:00Z.sql.tmp": false, + "backup.2026-10-21T10:00:00Z.sql.gz": true, + "backup.2026-10-21T10:00:00Z.sql.zst": true, + "backup.2026-10-21T10:00:00Z.zstd.sql": true, + "backup.2026-10-21T10:00:00Z.xz.sql": false, + "backup.2026-10-21T10:00:00Z.sql.gz.tmp": false, "physicalbackup-20261021100000.xb": false, "backup.not-a-date.sql": false, "sub/backup.2026-10-21T10:00:00Z.sql": false, @@ -69,8 +74,8 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { t.Run("single dump inside the backup window", func(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ - {Key: dumpKey(prefix, created.Add(-24*time.Hour), "gzip"), Size: 111}, - {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4242}, + {Key: dumpKey(prefix, created.Add(-24*time.Hour), "gzip"), Size: 111, LastModified: created.Add(-24 * time.Hour)}, + {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4242, LastModified: created.Add(time.Minute)}, {Key: prefix + "0-backup-target.txt", Size: 7}, }} got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed) @@ -96,7 +101,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "ca"}, Data: map[string][]byte{"ca.crt": []byte("PEM")}, } - l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 3}}} + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 3, LastModified: created.Add(time.Minute)}}} r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds"), ca) if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", tlsS3, mdbBackup, completed); got != 3 { t.Errorf("size = %d, want 3", got) @@ -114,7 +119,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "sts"}, Data: map[string][]byte{"token": []byte("TOKEN")}, } - l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4}}} + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4, LastModified: created.Add(time.Minute)}}} r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds"), sts) if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", stsS3, mdbBackup, completed); got != 4 { t.Errorf("size = %d, want 4", got) @@ -126,17 +131,30 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { t.Run("pod clock skew within tolerance", func(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ - {Key: dumpKey(prefix, created.Add(-30*time.Second), ""), Size: 9}, + {Key: dumpKey(prefix, created.Add(-30*time.Second), ""), Size: 9, LastModified: created.Add(time.Minute)}, }} if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 9 { t.Errorf("size = %d, want 9", got) } }) + t.Run("a skewed pod clock does not attribute a neighbouring run's dump", func(t *testing.T) { + // The pod clock runs 5m behind. This run's dump is named outside the + // window; the previous run's dump, uploaded before this run started, + // is named inside it. + l := &fakeS3Lister{objects: []s3ObjectInfo{ + {Key: dumpKey(prefix, created.Add(-4*time.Minute), "gzip"), Size: 7, LastModified: created.Add(time.Minute)}, + {Key: dumpKey(prefix, created.Add(30*time.Second), "gzip"), Size: 666, LastModified: created.Add(-4 * time.Minute)}, + }} + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + t.Errorf("size = %d, want 0: no dump is inside the window on both clocks", got) + } + }) + t.Run("two dumps in the window leave size unset", func(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ - {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 1}, - {Key: dumpKey(prefix, created.Add(2*time.Minute), "gzip"), Size: 2}, + {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 1, LastModified: created.Add(time.Minute)}, + {Key: dumpKey(prefix, created.Add(2*time.Minute), "gzip"), Size: 2, LastModified: created.Add(2 * time.Minute)}, }} if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0 (ambiguous)", got) @@ -145,7 +163,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { t.Run("no dump in the window leaves size unset", func(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ - {Key: dumpKey(prefix, completed.Add(time.Hour), "gzip"), Size: 1}, + {Key: dumpKey(prefix, completed.Add(time.Hour), "gzip"), Size: 1, LastModified: completed.Add(time.Hour)}, }} if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0", got) @@ -234,7 +252,7 @@ func TestCreateMariaDBBackupArtifact_SizeBytes(t *testing.T) { }, } r := reconcilerWithLister(t, &fakeS3Lister{objects: []s3ObjectInfo{ - {Key: dumpKey("src/", completed.Add(-time.Minute), "gzip"), Size: 2048}, + {Key: dumpKey("src/", completed.Add(-time.Minute), "gzip"), Size: 2048, LastModified: completed.Add(-time.Minute)}, }}, job, s3CredsSecret("tenant", "creds")) rendered := newRenderedMariaDBTemplate() rendered.Storage.S3.Prefix = "src/" diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index bdffc7e7c..f6abed32d 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -421,7 +421,7 @@ func (r *BackupJobReconciler) createMariaDBBackupArtifact( return nil, err } existing := &backupsv1alpha1.Backup{} - if getErr := r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: backup.Name}, existing); getErr != nil { + if getErr := r.apiReader().Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: backup.Name}, existing); getErr != nil { return nil, getErr } return existing, nil From 2b2176357761ac32a1215f19eeaf4e5cbde224a5 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Thu, 22 Oct 2026 18:18:51 +0400 Subject: [PATCH 093/518] fix(backups): do not fail a backup on a transient retry-check read The check for a Backup an earlier reconcile already created returned its read error, and the drivers turn any error from that path into a terminally Failed BackupJob. An apiserver timeout right after the operator backup completed therefore lost a backup whose data was safely in S3. The check only saves a probe, and the Create that follows already handles an existing Backup, so log the error and go on to Create. A trailing slash on the S3 endpoint is not a path prefix; the operators accept it, so do not refuse it. Pin the uncached AlreadyExists refetch, the upper slack of the MariaDB window and the probe timeout with tests. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 30 ++++--- .../backupcontroller/artifact_size_test.go | 85 +++++++++++++++++-- .../cnpg_artifact_size_test.go | 28 ++++++ .../cnpgstrategy_controller.go | 4 +- .../mariadb_artifact_size_test.go | 61 +++++++++++++ .../mariadbstrategy_controller.go | 4 +- 6 files changed, 190 insertions(+), 22 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 517fe07a6..182ea0b58 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -22,8 +22,8 @@ import ( // makes to measure Backup.status.artifact.sizeBytes. The probe runs inside // the reconcile that creates the Backup, and the BackupJob controller has a // single worker, so an unreachable endpoint stalls every other BackupJob for -// this long. -const artifactSizeProbeTimeout = 10 * time.Second +// this long. A variable so tests can shorten it. +var artifactSizeProbeTimeout = 10 * time.Second type s3ObjectInfo struct { Key string @@ -63,8 +63,10 @@ type minioObjectLister struct { func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { // An endpoint behind a path prefix cannot be addressed: minio-go has no // such option, and the SigV4 signature covers the request path, so the - // prefix cannot be added below the client either. - if strings.Contains(conn.Endpoint, "/") { + // prefix cannot be added below the client either. A bare trailing slash + // is not a prefix, and the operators accept it. + endpoint := strings.TrimSuffix(conn.Endpoint, "/") + if strings.Contains(endpoint, "/") { return nil, fmt.Errorf("endpoint %q has a path, which the S3 client cannot address", conn.Endpoint) } opts := &minio.Options{ @@ -90,7 +92,7 @@ func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { tr.TLSClientConfig.RootCAs = pool opts.Transport = tr } - client, err := minio.New(conn.Endpoint, opts) + client, err := minio.New(endpoint, opts) if err != nil { return nil, err } @@ -121,18 +123,20 @@ func collectObjects(ctx context.Context, objects <-chan minio.ObjectInfo) ([]s3O // existingBackup returns the Backup a previous reconcile already created // for the job, so a retry skips the size probe whose result it would -// discard. Returns (nil, nil) when there is none. It reads past the cache: -// a retry typically follows a Create made moments earlier, which the cache -// may not show yet. -func (r *BackupJobReconciler) existingBackup(ctx context.Context, namespace, name string) (*backupsv1alpha1.Backup, error) { +// discard. It reads past the cache: a retry typically follows a Create made +// moments earlier, which the cache may not show yet. It returns nil both +// when there is no Backup and when the read fails: it only saves a probe, +// and the Create that follows handles an existing Backup on its own, so a +// transient read error must not fail a backup that succeeded. +func (r *BackupJobReconciler) existingBackup(ctx context.Context, namespace, name string) *backupsv1alpha1.Backup { existing := &backupsv1alpha1.Backup{} if err := r.apiReader().Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, existing); err != nil { - if apierrors.IsNotFound(err) { - return nil, nil + if !apierrors.IsNotFound(err) { + getLogger(ctx).Info("cannot check for an existing Backup; creating it", "backup", name, "error", err) } - return nil, err + return nil } - return existing, nil + return existing } func (r *BackupJobReconciler) secretValue(ctx context.Context, namespace, name, key string) (string, error) { diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go index f24079f8b..0e66968b5 100644 --- a/internal/backupcontroller/artifact_size_test.go +++ b/internal/backupcontroller/artifact_size_test.go @@ -5,12 +5,15 @@ import ( "context" "errors" "testing" + "time" "github.com/minio/minio-go/v7" corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" ) @@ -87,8 +90,10 @@ func TestNewMinioObjectLister_RejectsEndpointPath(t *testing.T) { if _, err := newMinioObjectLister(s3Connection{Endpoint: "s3.example.org/prefix", Secure: true}); err == nil { t.Error("an endpoint with a path was accepted") } - if _, err := newMinioObjectLister(s3Connection{Endpoint: "s3.example.org:9000", Secure: true}); err != nil { - t.Errorf("host:port endpoint rejected: %v", err) + for _, ok := range []string{"s3.example.org:9000", "s3.example.org/"} { + if _, err := newMinioObjectLister(s3Connection{Endpoint: ok, Secure: true}); err != nil { + t.Errorf("endpoint %q rejected: %v", ok, err) + } } } @@ -101,8 +106,78 @@ func TestExistingBackup_ReadsPastTheCache(t *testing.T) { Client: newMariaDBStrategyTestClient(t), APIReader: newMariaDBStrategyTestClient(t, backup), } - got, err := r.existingBackup(context.Background(), "tenant", "bj") - if err != nil || got == nil { - t.Fatalf("existingBackup = %v, %v; want the Backup the API server holds", got, err) + if got := r.existingBackup(context.Background(), "tenant", "bj"); got == nil { + t.Fatal("existingBackup = nil; want the Backup the API server holds") + } +} + +// staleCache wraps a fake client so that Get reports NotFound for Backups, +// the way an informer cache does right after another writer created one. +func staleCache(t *testing.T, c client.Client) client.Client { + t.Helper() + return interceptor.NewClient(c.(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*backupsv1alpha1.Backup); ok { + return apierrors.NewNotFound(backupsv1alpha1.GroupVersion.WithResource("backups").GroupResource(), key.Name) + } + return cl.Get(ctx, key, obj, opts...) + }, + }) +} + +// notFoundOnce wraps a fake client whose first Backup Get reports NotFound: +// the retry check runs before the Backup exists, the refetch after Create +// collides with it. +func notFoundOnce(t *testing.T, c client.Client) client.Client { + t.Helper() + missed := false + return interceptor.NewClient(c.(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*backupsv1alpha1.Backup); ok && !missed { + missed = true + return apierrors.NewNotFound(backupsv1alpha1.GroupVersion.WithResource("backups").GroupResource(), key.Name) + } + return cl.Get(ctx, key, obj, opts...) + }, + }) +} + +func TestExistingBackup_ReadErrorIsNotFatal(t *testing.T) { + failing := interceptor.NewClient(newMariaDBStrategyTestClient(t).(client.WithWatch), interceptor.Funcs{ + Get: func(context.Context, client.WithWatch, client.ObjectKey, client.Object, ...client.GetOption) error { + return errors.New("etcdserver: request timed out") + }, + }) + r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t), APIReader: failing} + if got := r.existingBackup(context.Background(), "tenant", "bj"); got != nil { + t.Errorf("existingBackup = %#v, want nil on a read error", got) + } +} + +// blockingLister answers nothing until the probe's context is done, the way +// a black-holed endpoint does. +type blockingLister struct{} + +func (blockingLister) ListObjects(ctx context.Context, _, _ string, _ bool) ([]s3ObjectInfo, error) { + <-ctx.Done() + return nil, ctx.Err() +} + +// withShortProbeTimeout shortens the probe timeout and fails the test if fn +// is not back well before the default timeout would have expired. +func withShortProbeTimeout(t *testing.T, fn func()) { + t.Helper() + saved := artifactSizeProbeTimeout + artifactSizeProbeTimeout = 50 * time.Millisecond + defer func() { artifactSizeProbeTimeout = saved }() + done := make(chan struct{}) + go func() { + fn() + close(done) + }() + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("the probe did not give up at its timeout") } } diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index 13b647bdd..70bb2a485 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -83,6 +83,16 @@ func TestCNPGArtifactSizeBytes(t *testing.T) { } }) + t.Run("a hanging endpoint is given up at the probe timeout", func(t *testing.T) { + r := reconcilerWithLister(t, &fakeS3Lister{}, s3CredsSecret("tenant", "creds")) + r.newS3ObjectLister = func(s3Connection) (s3ObjectLister, error) { return blockingLister{}, nil } + withShortProbeTimeout(t, func() { + if got := r.cnpgArtifactSizeBytes(context.Background(), "tenant", newSizedBarmanObjectStore(), "srv", "20261021T100000"); got != 0 { + t.Errorf("size = %d, want 0", got) + } + }) + }) + t.Run("an empty endpointURL means AWS", func(t *testing.T) { store := newSizedBarmanObjectStore() store.EndpointURL = "" @@ -306,3 +316,21 @@ func TestCreateCNPGBackupArtifact_ExistingBackupSkipsProbe(t *testing.T) { t.Errorf("S3 listed %d times for a Backup that already exists", l.calls) } } + +func TestCreateCNPGBackupArtifact_AlreadyExistsRefetchReadsPastTheCache(t *testing.T) { + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-race"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"}, + }, + } + backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-race"}} + store := newCNPGStrategyTestClient(t, backup) + r := &BackupJobReconciler{Client: staleCache(t, store), APIReader: notFoundOnce(t, store)} + rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://b/"}} + got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}}, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")) + if err != nil || got == nil || got.Name != "bj-race" { + t.Fatalf("createCNPGBackupArtifact = %v, %v; want the Backup that already exists", got, err) + } +} diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 5dabfa66c..712dd8713 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -815,8 +815,8 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact( rendered *strategyv1alpha1.CNPGTemplate, sourceApp *postgresapp.Postgres, ) (*backupsv1alpha1.Backup, error) { - if existing, err := r.existingBackup(ctx, j.Namespace, j.Name); err != nil || existing != nil { - return existing, err + if existing := r.existingBackup(ctx, j.Namespace, j.Name); existing != nil { + return existing, nil } takenAt := metav1.Now() if cnpgBackup.Status.StartedAt != nil && !cnpgBackup.Status.StartedAt.IsZero() { diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index 6f9df56e7..91f555bbd 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -13,6 +13,8 @@ import ( corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" @@ -151,6 +153,25 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { } }) + t.Run("a dump uploaded just after Complete is inside the slack", func(t *testing.T) { + l := &fakeS3Lister{objects: []s3ObjectInfo{ + {Key: dumpKey(prefix, completed.Add(30*time.Second), "gzip"), Size: 5, LastModified: completed.Add(30 * time.Second)}, + }} + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 5 { + t.Errorf("size = %d, want 5", got) + } + }) + + t.Run("a hanging endpoint is given up at the probe timeout", func(t *testing.T) { + r := reconcilerWithLister(t, &fakeS3Lister{}, s3CredsSecret("tenant", "creds")) + r.newS3ObjectLister = func(s3Connection) (s3ObjectLister, error) { return blockingLister{}, nil } + withShortProbeTimeout(t, func() { + if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + t.Errorf("size = %d, want 0", got) + } + }) + }) + t.Run("two dumps in the window leave size unset", func(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 1, LastModified: created.Add(time.Minute)}, @@ -298,3 +319,43 @@ func TestCreateMariaDBBackupArtifact_ExistingBackupSkipsProbe(t *testing.T) { t.Errorf("Artifact = %#v, want the existing one", got.Status.Artifact) } } + +func TestCreateMariaDBBackupArtifact_TransientRetryReadStillCreates(t *testing.T) { + apps := mariadbapp.GroupName + job := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-transient"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, + }, + } + r := reconcilerWithLister(t, &fakeS3Lister{}, job) + r.APIReader = interceptor.NewClient(newMariaDBStrategyTestClient(t).(client.WithWatch), interceptor.Funcs{ + Get: func(context.Context, client.WithWatch, client.ObjectKey, client.Object, ...client.GetOption) error { + return errors.New("etcdserver: request timed out") + }, + }) + got, err := r.createMariaDBBackupArtifact(context.Background(), job, &ResolvedBackupConfig{}, &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "op"}}, newRenderedMariaDBTemplate()) + if err != nil || got == nil || got.Name != "bj-transient" { + t.Fatalf("createMariaDBBackupArtifact = %v, %v; a transient read must not fail the backup", got, err) + } + if err := r.Get(context.Background(), client.ObjectKeyFromObject(got), &backupsv1alpha1.Backup{}); err != nil { + t.Errorf("the Backup was not created: %v", err) + } +} + +func TestCreateMariaDBBackupArtifact_AlreadyExistsRefetchReadsPastTheCache(t *testing.T) { + apps := mariadbapp.GroupName + job := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-race"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, + }, + } + backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-race"}} + store := newMariaDBStrategyTestClient(t, job, backup) + r := &BackupJobReconciler{Client: staleCache(t, store), APIReader: notFoundOnce(t, store), Scheme: store.Scheme()} + got, err := r.createMariaDBBackupArtifact(context.Background(), job, &ResolvedBackupConfig{}, &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "op"}}, &strategyv1alpha1.MariaDBTemplate{}) + if err != nil || got == nil || got.Name != "bj-race" { + t.Fatalf("createMariaDBBackupArtifact = %v, %v; want the Backup that already exists", got, err) + } +} diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index f6abed32d..8db41fb7f 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -365,8 +365,8 @@ func (r *BackupJobReconciler) createMariaDBBackupArtifact( mdbBackup *mariadbtypes.Backup, rendered *strategyv1alpha1.MariaDBTemplate, ) (*backupsv1alpha1.Backup, error) { - if existing, err := r.existingBackup(ctx, j.Namespace, j.Name); err != nil || existing != nil { - return existing, err + if existing := r.existingBackup(ctx, j.Namespace, j.Name); existing != nil { + return existing, nil } takenAt := metav1.Now() if cond := apimeta.FindStatusCondition(mdbBackup.Status.Conditions, mariadbtypes.ConditionTypeComplete); cond != nil && !cond.LastTransitionTime.IsZero() { From 523d74a5033cedb15158c5c3e3673c4e79d98fb7 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Thu, 22 Oct 2026 18:41:03 +0400 Subject: [PATCH 094/518] fix(backups): retry instead of failing when an existing Backup cannot be read When Create collides with a Backup an earlier reconcile already made and the read that follows fails, the drivers failed the BackupJob, although the backup and its Backup both exist. Return a distinct error instead, which the drivers hand back to the controller so the reconcile is retried with backoff and then finds the Backup. minio.New already refuses an endpoint with a path prefix and accepts a trailing slash, so drop the duplicate check in the shared client; the CNPG probe keeps its own check because it parses the URL itself and would otherwise drop the path before minio sees it. Say in the sizeBytes API doc what the value covers, such as the base backup without the shared WAL archive for CNPG, and that strategies which cannot attribute stored bytes to one backup leave it unset. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- api/backups/v1alpha1/backup_types.go | 11 ++-- internal/backupcontroller/artifact_size.go | 31 +++++++---- .../backupcontroller/artifact_size_test.go | 52 +++++++++++++++---- .../cnpg_artifact_size_test.go | 18 +++++++ .../cnpgstrategy_controller.go | 4 +- .../mariadb_artifact_size_test.go | 17 ++++++ .../mariadbstrategy_controller.go | 4 +- .../backups.cozystack.io_backups.yaml | 11 ++-- 8 files changed, 116 insertions(+), 32 deletions(-) diff --git a/api/backups/v1alpha1/backup_types.go b/api/backups/v1alpha1/backup_types.go index 483bb1fd8..541488202 100644 --- a/api/backups/v1alpha1/backup_types.go +++ b/api/backups/v1alpha1/backup_types.go @@ -37,9 +37,14 @@ type BackupArtifact struct { // For example: s3://bucket/prefix/file.tar.gz URI string `json:"uri"` - // SizeBytes is the size of the artifact in bytes, if known. Drivers - // leave it unset rather than writing 0 when they cannot measure the - // stored size, so an absent value means "unknown", not "empty". + // SizeBytes is the stored size in bytes of the files this backup owns, + // if known: the dump of a MariaDB backup, for example, or the base backup + // of a CNPG one, which leaves out the WAL archive the server's backups + // share and a point-in-time restore also reads. Strategies that cannot + // attribute stored bytes to a single backup, such as Velero's, whose + // repository deduplicates across backups, leave it unset. Drivers never + // write 0 for a size they cannot measure, so an absent value means + // "unknown", not "empty". // +optional SizeBytes int64 `json:"sizeBytes,omitempty"` diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 182ea0b58..5addefd5a 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -6,7 +6,6 @@ import ( "crypto/x509" "errors" "fmt" - "strings" "time" "github.com/minio/minio-go/v7" @@ -14,6 +13,7 @@ import ( corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/types" + ctrl "sigs.k8s.io/controller-runtime" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" ) @@ -38,7 +38,7 @@ type s3ObjectLister interface { // s3Connection is what a driver resolves from its rendered strategy to // read the bucket with the same identity the backup was written with. type s3Connection struct { - // Endpoint is host[:port], without a scheme or a path. + // Endpoint is host[:port], without a scheme. Endpoint string Secure bool Region string @@ -61,14 +61,6 @@ type minioObjectLister struct { } func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { - // An endpoint behind a path prefix cannot be addressed: minio-go has no - // such option, and the SigV4 signature covers the request path, so the - // prefix cannot be added below the client either. A bare trailing slash - // is not a prefix, and the operators accept it. - endpoint := strings.TrimSuffix(conn.Endpoint, "/") - if strings.Contains(endpoint, "/") { - return nil, fmt.Errorf("endpoint %q has a path, which the S3 client cannot address", conn.Endpoint) - } opts := &minio.Options{ Creds: credentials.NewStaticV4(conn.AccessKey, conn.SecretKey, conn.SessionToken), Region: conn.Region, @@ -92,7 +84,9 @@ func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { tr.TLSClientConfig.RootCAs = pool opts.Transport = tr } - client, err := minio.New(endpoint, opts) + // minio.New refuses an endpoint with a path prefix, which it cannot + // address, and accepts a bare trailing slash. + client, err := minio.New(conn.Endpoint, opts) if err != nil { return nil, err } @@ -121,6 +115,21 @@ func collectObjects(ctx context.Context, objects <-chan minio.ObjectInfo) ([]s3O return out, nil } +// errBackupNotReadBack reports that Create collided with an existing Backup +// that could then not be read. The backup itself succeeded, so callers +// retry the reconcile instead of failing the BackupJob. +var errBackupNotReadBack = errors.New("the Backup already exists but could not be read back") + +// failBackupArtifact handles an error from creating the Backup of a +// completed backup: errBackupNotReadBack is retried through the controller's +// backoff, anything else fails the BackupJob. +func (r *BackupJobReconciler) failBackupArtifact(ctx context.Context, j *backupsv1alpha1.BackupJob, err error) (ctrl.Result, error) { + if errors.Is(err, errBackupNotReadBack) { + return ctrl.Result{}, err + } + return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to create Backup artifact: %v", err)) +} + // existingBackup returns the Backup a previous reconcile already created // for the job, so a retry skips the size probe whose result it would // discard. It reads past the cache: a retry typically follows a Create made diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go index 0e66968b5..69112cd9d 100644 --- a/internal/backupcontroller/artifact_size_test.go +++ b/internal/backupcontroller/artifact_size_test.go @@ -4,6 +4,7 @@ package backupcontroller import ( "context" "errors" + "fmt" "testing" "time" @@ -86,17 +87,6 @@ func TestCollectObjects(t *testing.T) { } } -func TestNewMinioObjectLister_RejectsEndpointPath(t *testing.T) { - if _, err := newMinioObjectLister(s3Connection{Endpoint: "s3.example.org/prefix", Secure: true}); err == nil { - t.Error("an endpoint with a path was accepted") - } - for _, ok := range []string{"s3.example.org:9000", "s3.example.org/"} { - if _, err := newMinioObjectLister(s3Connection{Endpoint: ok, Secure: true}); err != nil { - t.Errorf("endpoint %q rejected: %v", ok, err) - } - } -} - // TestExistingBackup_ReadsPastTheCache pins that the retry check sees a // Backup the cache does not show yet: the previous reconcile created it // moments earlier. @@ -181,3 +171,43 @@ func withShortProbeTimeout(t *testing.T, fn func()) { t.Fatal("the probe did not give up at its timeout") } } + +// failingReads wraps a fake client whose every Get fails, like an apiserver +// that times out. +func failingReads(t *testing.T, c client.Client) client.Client { + t.Helper() + return interceptor.NewClient(c.(client.WithWatch), interceptor.Funcs{ + Get: func(context.Context, client.WithWatch, client.ObjectKey, client.Object, ...client.GetOption) error { + return errors.New("etcdserver: request timed out") + }, + }) +} + +func TestFailBackupArtifact(t *testing.T) { + newJob := func() *backupsv1alpha1.BackupJob { + return &backupsv1alpha1.BackupJob{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj"}} + } + + t.Run("a Backup that exists but cannot be read is retried", func(t *testing.T) { + j := newJob() + r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t, j)} + _, err := r.failBackupArtifact(context.Background(), j, fmt.Errorf("%w: timeout", errBackupNotReadBack)) + if !errors.Is(err, errBackupNotReadBack) { + t.Errorf("err = %v, want it returned so the reconcile is retried", err) + } + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Error("the BackupJob was failed although its Backup exists") + } + }) + + t.Run("any other error fails the BackupJob", func(t *testing.T) { + j := newJob() + r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t, j)} + if _, err := r.failBackupArtifact(context.Background(), j, errors.New("admission webhook denied")); err != nil { + t.Fatalf("failBackupArtifact: %v", err) + } + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed { + t.Errorf("phase = %q, want Failed", j.Status.Phase) + } + }) +} diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index 70bb2a485..f6ea74b6f 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -334,3 +334,21 @@ func TestCreateCNPGBackupArtifact_AlreadyExistsRefetchReadsPastTheCache(t *testi t.Fatalf("createCNPGBackupArtifact = %v, %v; want the Backup that already exists", got, err) } } + +func TestCreateCNPGBackupArtifact_UnreadableExistingBackupIsRetried(t *testing.T) { + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-unreadable"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"}, + }, + } + backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-unreadable"}} + store := newCNPGStrategyTestClient(t, backup) + r := &BackupJobReconciler{Client: staleCache(t, store), APIReader: failingReads(t, store)} + rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://b/"}} + _, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}}, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")) + if !errors.Is(err, errBackupNotReadBack) { + t.Errorf("err = %v, want errBackupNotReadBack", err) + } +} diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 712dd8713..ed994ed22 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -435,7 +435,7 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } artifact, err := r.createCNPGBackupArtifact(ctx, j, resolved, cnpgBackup, clusterName, serverName, attachment.flavor, rendered, app) if err != nil { - return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to create Backup artifact: %v", err)) + return r.failBackupArtifact(ctx, j, err) } now := metav1.Now() j.Status.BackupRef = &corev1.LocalObjectReference{Name: artifact.Name} @@ -877,7 +877,7 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact( } existing := &backupsv1alpha1.Backup{} if getErr := r.apiReader().Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: backup.Name}, existing); getErr != nil { - return nil, getErr + return nil, fmt.Errorf("%w: %v", errBackupNotReadBack, getErr) } return existing, nil } diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index 91f555bbd..fd3ec30d4 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -359,3 +359,20 @@ func TestCreateMariaDBBackupArtifact_AlreadyExistsRefetchReadsPastTheCache(t *te t.Fatalf("createMariaDBBackupArtifact = %v, %v; want the Backup that already exists", got, err) } } + +func TestCreateMariaDBBackupArtifact_UnreadableExistingBackupIsRetried(t *testing.T) { + apps := mariadbapp.GroupName + job := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-unreadable"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, + }, + } + backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-unreadable"}} + store := newMariaDBStrategyTestClient(t, job, backup) + r := &BackupJobReconciler{Client: staleCache(t, store), APIReader: failingReads(t, store), Scheme: store.Scheme()} + _, err := r.createMariaDBBackupArtifact(context.Background(), job, &ResolvedBackupConfig{}, &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "op"}}, &strategyv1alpha1.MariaDBTemplate{}) + if !errors.Is(err, errBackupNotReadBack) { + t.Errorf("err = %v, want errBackupNotReadBack", err) + } +} diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 8db41fb7f..4c920dcaa 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -214,7 +214,7 @@ func (r *BackupJobReconciler) reconcileMariaDB(ctx context.Context, j *backupsv1 } artifact, err := r.createMariaDBBackupArtifact(ctx, j, resolved, mdbBackup, rendered) if err != nil { - return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to create Backup artifact: %v", err)) + return r.failBackupArtifact(ctx, j, err) } now := metav1.Now() j.Status.BackupRef = &corev1.LocalObjectReference{Name: artifact.Name} @@ -422,7 +422,7 @@ func (r *BackupJobReconciler) createMariaDBBackupArtifact( } existing := &backupsv1alpha1.Backup{} if getErr := r.apiReader().Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: backup.Name}, existing); getErr != nil { - return nil, getErr + return nil, fmt.Errorf("%w: %v", errBackupNotReadBack, getErr) } return existing, nil } diff --git a/packages/system/backup-controller/definitions/backups.cozystack.io_backups.yaml b/packages/system/backup-controller/definitions/backups.cozystack.io_backups.yaml index f30d3dbc7..dfa25dbc5 100644 --- a/packages/system/backup-controller/definitions/backups.cozystack.io_backups.yaml +++ b/packages/system/backup-controller/definitions/backups.cozystack.io_backups.yaml @@ -132,9 +132,14 @@ spec: type: string sizeBytes: description: |- - SizeBytes is the size of the artifact in bytes, if known. Drivers - leave it unset rather than writing 0 when they cannot measure the - stored size, so an absent value means "unknown", not "empty". + SizeBytes is the stored size in bytes of the files this backup owns, + if known: the dump of a MariaDB backup, for example, or the base backup + of a CNPG one, which leaves out the WAL archive the server's backups + share and a point-in-time restore also reads. Strategies that cannot + attribute stored bytes to a single backup, such as Velero's, whose + repository deduplicates across backups, leave it unset. Drivers never + write 0 for a size they cannot measure, so an absent value means + "unknown", not "empty". format: int64 type: integer uri: From 080df1b2f204377b24da31ad399df21b63179a07 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Thu, 22 Oct 2026 19:38:00 +0400 Subject: [PATCH 095/518] fix(backups): surface a Backup that cannot be read back and skip its probe A Backup that exists but cannot be read back is retried through the controller's backoff, which left the BackupJob Running with nothing on it saying why. Set Ready=False with reason BackupNotReadBack before handing the error back. When the check for an existing Backup itself fails, the Backup may well exist, so a size probe would only be discarded, and a read that keeps failing reran it, up to the probe timeout on the single BackupJob worker, on every retry. Skip the probe in that case and create the Backup without a size. Drive both behaviours through the MariaDB and CNPG reconciles in tests. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 36 ++++++---- .../backupcontroller/artifact_size_test.go | 20 ++++-- .../cnpg_artifact_size_test.go | 65 +++++++++++++++++++ .../cnpgstrategy_controller.go | 9 ++- .../mariadb_artifact_size_test.go | 65 +++++++++++++++++-- .../mariadbstrategy_controller.go | 9 +-- 6 files changed, 175 insertions(+), 29 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 5addefd5a..534019184 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -12,6 +12,8 @@ import ( "github.com/minio/minio-go/v7/pkg/credentials" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" + apimeta "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" ctrl "sigs.k8s.io/controller-runtime" @@ -122,9 +124,19 @@ var errBackupNotReadBack = errors.New("the Backup already exists but could not b // failBackupArtifact handles an error from creating the Backup of a // completed backup: errBackupNotReadBack is retried through the controller's -// backoff, anything else fails the BackupJob. +// backoff, with Ready=False telling why the BackupJob is still Running; +// anything else fails the BackupJob. func (r *BackupJobReconciler) failBackupArtifact(ctx context.Context, j *backupsv1alpha1.BackupJob, err error) (ctrl.Result, error) { if errors.Is(err, errBackupNotReadBack) { + apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ + Type: "Ready", + Status: metav1.ConditionFalse, + Reason: "BackupNotReadBack", + Message: err.Error(), + }) + if uerr := r.Status().Update(ctx, j); uerr != nil { + getLogger(ctx).Info("cannot record that the Backup could not be read back", "backupjob", j.Name, "error", uerr) + } return ctrl.Result{}, err } return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to create Backup artifact: %v", err)) @@ -133,19 +145,21 @@ func (r *BackupJobReconciler) failBackupArtifact(ctx context.Context, j *backups // existingBackup returns the Backup a previous reconcile already created // for the job, so a retry skips the size probe whose result it would // discard. It reads past the cache: a retry typically follows a Create made -// moments earlier, which the cache may not show yet. It returns nil both -// when there is no Backup and when the read fails: it only saves a probe, -// and the Create that follows handles an existing Backup on its own, so a -// transient read error must not fail a backup that succeeded. -func (r *BackupJobReconciler) existingBackup(ctx context.Context, namespace, name string) *backupsv1alpha1.Backup { - existing := &backupsv1alpha1.Backup{} +// moments earlier, which the cache may not show yet. A failed read is not +// fatal, because the Create that follows handles an existing Backup on its +// own; probe is false then, since the Backup may well exist and a probe +// would only be discarded, while a read that keeps failing would rerun it on +// every retry. +func (r *BackupJobReconciler) existingBackup(ctx context.Context, namespace, name string) (existing *backupsv1alpha1.Backup, probe bool) { + existing = &backupsv1alpha1.Backup{} if err := r.apiReader().Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, existing); err != nil { - if !apierrors.IsNotFound(err) { - getLogger(ctx).Info("cannot check for an existing Backup; creating it", "backup", name, "error", err) + if apierrors.IsNotFound(err) { + return nil, true } - return nil + getLogger(ctx).Info("cannot check for an existing Backup; creating it without measuring its size", "backup", name, "error", err) + return nil, false } - return existing + return existing, false } func (r *BackupJobReconciler) secretValue(ctx context.Context, namespace, name, key string) (string, error) { diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go index 69112cd9d..523d56469 100644 --- a/internal/backupcontroller/artifact_size_test.go +++ b/internal/backupcontroller/artifact_size_test.go @@ -12,6 +12,7 @@ import ( corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" + apimeta "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/interceptor" @@ -96,9 +97,13 @@ func TestExistingBackup_ReadsPastTheCache(t *testing.T) { Client: newMariaDBStrategyTestClient(t), APIReader: newMariaDBStrategyTestClient(t, backup), } - if got := r.existingBackup(context.Background(), "tenant", "bj"); got == nil { + if got, _ := r.existingBackup(context.Background(), "tenant", "bj"); got == nil { t.Fatal("existingBackup = nil; want the Backup the API server holds") } + r.APIReader = newMariaDBStrategyTestClient(t) + if got, probe := r.existingBackup(context.Background(), "tenant", "bj"); got != nil || !probe { + t.Errorf("existingBackup = %v, probe=%v; want nil and a probe when there is no Backup", got, probe) + } } // staleCache wraps a fake client so that Get reports NotFound for Backups, @@ -139,8 +144,8 @@ func TestExistingBackup_ReadErrorIsNotFatal(t *testing.T) { }, }) r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t), APIReader: failing} - if got := r.existingBackup(context.Background(), "tenant", "bj"); got != nil { - t.Errorf("existingBackup = %#v, want nil on a read error", got) + if got, probe := r.existingBackup(context.Background(), "tenant", "bj"); got != nil || probe { + t.Errorf("existingBackup = %#v, probe=%v; want nil and no probe on a read error", got, probe) } } @@ -195,9 +200,16 @@ func TestFailBackupArtifact(t *testing.T) { if !errors.Is(err, errBackupNotReadBack) { t.Errorf("err = %v, want it returned so the reconcile is retried", err) } - if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + persisted := &backupsv1alpha1.BackupJob{} + if err := r.Get(context.Background(), client.ObjectKeyFromObject(j), persisted); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if persisted.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { t.Error("the BackupJob was failed although its Backup exists") } + if cond := apimeta.FindStatusCondition(persisted.Status.Conditions, "Ready"); cond == nil || cond.Status != metav1.ConditionFalse || cond.Reason != "BackupNotReadBack" { + t.Errorf("Ready condition = %+v, want False/BackupNotReadBack", cond) + } }) t.Run("any other error fails the BackupJob", func(t *testing.T) { diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index f6ea74b6f..7b3852317 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -10,9 +10,11 @@ import ( "net/http/httptest" "strings" "testing" + "time" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" @@ -352,3 +354,66 @@ func TestCreateCNPGBackupArtifact_UnreadableExistingBackupIsRetried(t *testing.T t.Errorf("err = %v, want errBackupNotReadBack", err) } } + +// TestReconcileCNPG_UnreadableBackupIsRetried drives a completed backup +// through the reconcile: when the Backup exists but cannot be read back, +// the reconcile is retried and the BackupJob is not failed. +func TestReconcileCNPG_UnreadableBackupIsRetried(t *testing.T) { + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Minute)), + newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)), + newRolloutInstancePod(time.Now().Add(-time.Hour))) + _, backups := f.pass() + if len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups after the first pass, want 1", len(backups)) + } + ctx := context.Background() + completed := backups[0].DeepCopy() + completed.Status.Phase = cnpgBackupPhaseComplete + completed.Status.BackupID = "20261021T100000" + if err := f.c.Update(ctx, completed); err != nil { + t.Fatalf("complete cnpg.io/Backup: %v", err) + } + if err := f.c.Create(ctx, &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: rolloutNS, Name: "bj"}}); err != nil { + t.Fatalf("seed Backup: %v", err) + } + f.r.APIReader = failingReads(t, f.c) + + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(ctx, client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := f.r.reconcileCNPG(ctx, j, f.resolved); !errors.Is(err, errBackupNotReadBack) { + t.Fatalf("reconcileCNPG err = %v, want errBackupNotReadBack so the reconcile is retried", err) + } + persisted := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(ctx, client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, persisted); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if persisted.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Error("the BackupJob was failed although its Backup exists") + } +} + +func TestCreateCNPGBackupArtifact_FailedExistenceCheckSkipsProbe(t *testing.T) { + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-transient"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"}, + }, + } + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: "x", Size: 1}}} + r := reconcilerWithLister(t, l) + r.Client = newCNPGStrategyTestClient(t, s3CredsSecret("tenant", "creds")) + r.APIReader = failingReads(t, newCNPGStrategyTestClient(t)) + cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}} + rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()} + + got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")) + if err != nil || got == nil { + t.Fatalf("createCNPGBackupArtifact = %v, %v; a transient read must not fail the backup", got, err) + } + if l.calls != 0 { + t.Errorf("S3 probed %d times after the existence check failed", l.calls) + } +} diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index ed994ed22..819efc10f 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -815,7 +815,8 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact( rendered *strategyv1alpha1.CNPGTemplate, sourceApp *postgresapp.Postgres, ) (*backupsv1alpha1.Backup, error) { - if existing := r.existingBackup(ctx, j.Namespace, j.Name); existing != nil { + existing, probe := r.existingBackup(ctx, j.Namespace, j.Name) + if existing != nil { return existing, nil } takenAt := metav1.Now() @@ -858,12 +859,14 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact( Status: backupsv1alpha1.BackupStatus{ Phase: backupsv1alpha1.BackupPhaseReady, Artifact: &backupsv1alpha1.BackupArtifact{ - URI: fmt.Sprintf("cnpg://%s/%s", serverName, cnpgBackup.Name), - SizeBytes: r.cnpgArtifactSizeBytes(ctx, j.Namespace, &rendered.BarmanObjectStore, serverName, cnpgBackup.Status.BackupID), + URI: fmt.Sprintf("cnpg://%s/%s", serverName, cnpgBackup.Name), }, UnderlyingResources: underlyingResources, }, } + if probe { + backup.Status.Artifact.SizeBytes = r.cnpgArtifactSizeBytes(ctx, j.Namespace, &rendered.BarmanObjectStore, serverName, cnpgBackup.Status.BackupID) + } if j.Spec.PlanRef != nil { backup.Spec.PlanRef = j.Spec.PlanRef } diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index fd3ec30d4..b49bcae66 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -13,8 +13,8 @@ import ( corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/tools/record" "sigs.k8s.io/controller-runtime/pkg/client" - "sigs.k8s.io/controller-runtime/pkg/client/interceptor" strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" @@ -328,12 +328,9 @@ func TestCreateMariaDBBackupArtifact_TransientRetryReadStillCreates(t *testing.T ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, }, } - r := reconcilerWithLister(t, &fakeS3Lister{}, job) - r.APIReader = interceptor.NewClient(newMariaDBStrategyTestClient(t).(client.WithWatch), interceptor.Funcs{ - Get: func(context.Context, client.WithWatch, client.ObjectKey, client.Object, ...client.GetOption) error { - return errors.New("etcdserver: request timed out") - }, - }) + l := &fakeS3Lister{} + r := reconcilerWithLister(t, l, job, s3CredsSecret("tenant", "creds")) + r.APIReader = failingReads(t, newMariaDBStrategyTestClient(t)) got, err := r.createMariaDBBackupArtifact(context.Background(), job, &ResolvedBackupConfig{}, &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "op"}}, newRenderedMariaDBTemplate()) if err != nil || got == nil || got.Name != "bj-transient" { t.Fatalf("createMariaDBBackupArtifact = %v, %v; a transient read must not fail the backup", got, err) @@ -341,6 +338,9 @@ func TestCreateMariaDBBackupArtifact_TransientRetryReadStillCreates(t *testing.T if err := r.Get(context.Background(), client.ObjectKeyFromObject(got), &backupsv1alpha1.Backup{}); err != nil { t.Errorf("the Backup was not created: %v", err) } + if l.calls != 0 { + t.Errorf("S3 probed %d times after the existence check failed", l.calls) + } } func TestCreateMariaDBBackupArtifact_AlreadyExistsRefetchReadsPastTheCache(t *testing.T) { @@ -376,3 +376,54 @@ func TestCreateMariaDBBackupArtifact_UnreadableExistingBackupIsRetried(t *testin t.Errorf("err = %v, want errBackupNotReadBack", err) } } + +// TestReconcileMariaDB_UnreadableBackupIsRetried drives a completed backup +// through the reconcile: when the Backup exists but cannot be read back, +// the reconcile is retried and the BackupJob is not failed. +func TestReconcileMariaDB_UnreadableBackupIsRetried(t *testing.T) { + strategyGroup := strategyv1alpha1.GroupVersion.Group + now := metav1.Now() + bj := newMariaDBBackupJob("bj-unreadable", "tenant") + bj.Status.StartedAt = &now + bj.Status.Phase = backupsv1alpha1.BackupJobPhaseRunning + strategy := &strategyv1alpha1.MariaDB{ + ObjectMeta: metav1.ObjectMeta{Name: "mdb-strategy"}, + Spec: strategyv1alpha1.MariaDBSpec{Template: *newRenderedMariaDBTemplate()}, + } + operatorMariaDB := &mariadbtypes.MariaDB{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: mariadbNameForApp("mariadb-src")}} + operatorBackup := &mariadbtypes.Backup{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", Name: "op-bk", + Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: bj.Name, + backupsv1alpha1.OwningJobNamespaceLabel: bj.Namespace, + }, + }, + Status: mariadbtypes.BackupStatus{Conditions: []metav1.Condition{{ + Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionTrue, LastTransitionTime: now, + }}}, + } + backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: bj.Name}} + store := newMariaDBStrategyTestClient(t, bj, strategy, newMariaDBApp("mariadb-src", "tenant"), operatorMariaDB, operatorBackup, backup) + r := &BackupJobReconciler{ + Client: store, + APIReader: failingReads(t, store), + Scheme: store.Scheme(), + Recorder: record.NewFakeRecorder(10), + } + resolved := &ResolvedBackupConfig{StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: &strategyGroup, Kind: strategyv1alpha1.MariaDBStrategyKind, Name: strategy.Name, + }} + + _, err := r.reconcileMariaDB(context.Background(), bj, resolved) + if !errors.Is(err, errBackupNotReadBack) { + t.Fatalf("reconcileMariaDB err = %v, want errBackupNotReadBack so the reconcile is retried", err) + } + persisted := &backupsv1alpha1.BackupJob{} + if err := store.Get(context.Background(), client.ObjectKeyFromObject(bj), persisted); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if persisted.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Error("the BackupJob was failed although its Backup exists") + } +} diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 4c920dcaa..749d63c9c 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -365,7 +365,8 @@ func (r *BackupJobReconciler) createMariaDBBackupArtifact( mdbBackup *mariadbtypes.Backup, rendered *strategyv1alpha1.MariaDBTemplate, ) (*backupsv1alpha1.Backup, error) { - if existing := r.existingBackup(ctx, j.Namespace, j.Name); existing != nil { + existing, probe := r.existingBackup(ctx, j.Namespace, j.Name) + if existing != nil { return existing, nil } takenAt := metav1.Now() @@ -395,9 +396,9 @@ func (r *BackupJobReconciler) createMariaDBBackupArtifact( // operator's PVC ID), so an Artifact{URI: ""} is a worse signal than // no Artifact at all - leave it nil instead. if uri := driverMD[mariadbStorageURIKey]; uri != "" { - status.Artifact = &backupsv1alpha1.BackupArtifact{ - URI: uri, - SizeBytes: r.mariadbArtifactSizeBytes(ctx, j.Namespace, rendered.Storage.S3, mdbBackup, takenAt.Time), + status.Artifact = &backupsv1alpha1.BackupArtifact{URI: uri} + if probe { + status.Artifact.SizeBytes = r.mariadbArtifactSizeBytes(ctx, j.Namespace, rendered.Storage.S3, mdbBackup, takenAt.Time) } } backup := &backupsv1alpha1.Backup{ From 92fa97969016ca92a4b123bcd4c11c710f6cfd32 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Thu, 22 Oct 2026 20:08:29 +0400 Subject: [PATCH 096/518] fix(backups): trust only the endpoint CA where the backup writer does With an endpoint CA the probe trusted the system roots plus that CA. barman-cloud gets endpointCA as AWS_CA_BUNDLE, which replaces the default trust store, so a CNPG probe could accept an endpoint the backup itself would have refused. Let the connection say whether the CA adds to the system roots or replaces them, and replace them for CNPG. mariadb-operator appends its CA to the system pool, so MariaDB keeps that. Pin in reconcile tests that a Create the apiserver rejects still fails the BackupJob, so only the read-back of an existing Backup is retried. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 38 ++++++++---- .../backupcontroller/artifact_size_test.go | 62 +++++++++++++++++++ .../backupcontroller/cnpg_artifact_size.go | 4 +- .../cnpg_artifact_size_test.go | 36 ++++++++++- .../mariadb_artifact_size_test.go | 48 ++++++++++++++ 5 files changed, 175 insertions(+), 13 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 534019184..1a822699c 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -47,8 +47,11 @@ type s3Connection struct { AccessKey string SecretKey string SessionToken string - // CAPEM is trusted on top of the system pool when set. - CAPEM []byte + // CAPEM is an endpoint CA bundle. It is trusted on top of the system + // roots, or instead of them when CAOnly is set: the probe must trust + // what the writer of the backup trusted, no more. + CAPEM []byte + CAOnly bool } func (r *BackupJobReconciler) s3Lister(conn s3Connection) (s3ObjectLister, error) { @@ -74,16 +77,9 @@ func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { if err != nil { return nil, err } - pool := tr.TLSClientConfig.RootCAs - if pool == nil { - if pool, err = x509.SystemCertPool(); err != nil { - pool = x509.NewCertPool() - } + if tr.TLSClientConfig.RootCAs, err = endpointTrustPool(conn, tr.TLSClientConfig.RootCAs); err != nil { + return nil, err } - if !pool.AppendCertsFromPEM(conn.CAPEM) { - return nil, errors.New("no PEM certificates in the endpoint CA bundle") - } - tr.TLSClientConfig.RootCAs = pool opts.Transport = tr } // minio.New refuses an endpoint with a path prefix, which it cannot @@ -95,6 +91,26 @@ func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { return &minioObjectLister{client: client}, nil } +// endpointTrustPool returns the roots to verify the endpoint against: the +// CA bundle on top of system, or the CA bundle alone with CAOnly. A nil +// system means the platform's roots. +func endpointTrustPool(conn s3Connection, system *x509.CertPool) (*x509.CertPool, error) { + pool := system + switch { + case conn.CAOnly: + pool = x509.NewCertPool() + case pool == nil: + var err error + if pool, err = x509.SystemCertPool(); err != nil { + pool = x509.NewCertPool() + } + } + if !pool.AppendCertsFromPEM(conn.CAPEM) { + return nil, errors.New("no PEM certificates in the endpoint CA bundle") + } + return pool, nil +} + func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) { return collectObjects(ctx, l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: recursive})) } diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go index 523d56469..3d9c35b23 100644 --- a/internal/backupcontroller/artifact_size_test.go +++ b/internal/backupcontroller/artifact_size_test.go @@ -3,8 +3,17 @@ package backupcontroller import ( "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" "errors" "fmt" + "math/big" + "net/http" + "net/http/httptest" "testing" "time" @@ -223,3 +232,56 @@ func TestFailBackupArtifact(t *testing.T) { } }) } + +// TestEndpointTrustPool_CAOnly tells apart "the CA on top of the system +// roots" and "the CA alone". The system pool holds the endpoint's +// certificate, and the configured CA is an unrelated one, so the endpoint +// must stop being trusted only when CAOnly is set. +func TestEndpointTrustPool_CAOnly(t *testing.T) { + endpoint := httptest.NewTLSServer(http.NotFoundHandler()) + defer endpoint.Close() + system := x509.NewCertPool() + system.AddCert(endpoint.Certificate()) + + // Every httptest server shares one certificate, so the unrelated CA is + // minted here. + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + tmpl := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "unrelated CA"}, NotBefore: time.Now(), NotAfter: time.Now().Add(time.Hour), IsCA: true, BasicConstraintsValid: true} + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + if err != nil { + t.Fatal(err) + } + unrelatedCA := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}) + + verifies := func(caOnly bool) bool { + pool, err := endpointTrustPool(s3Connection{CAPEM: unrelatedCA, CAOnly: caOnly}, system.Clone()) + if err != nil { + t.Fatalf("endpointTrustPool: %v", err) + } + _, err = endpoint.Certificate().Verify(x509.VerifyOptions{Roots: pool}) + return err == nil + } + if !verifies(false) { + t.Error("CA on top of the system roots: the endpoint was not trusted") + } + if verifies(true) { + t.Error("CA alone: the endpoint was trusted through the system roots") + } +} + +// forbidBackupCreate wraps a fake client whose Create of a Backup is +// rejected, the way an admission webhook rejects one. +func forbidBackupCreate(t *testing.T, c client.Client) client.Client { + t.Helper() + return interceptor.NewClient(c.(client.WithWatch), interceptor.Funcs{ + Create: func(ctx context.Context, cl client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + if _, ok := obj.(*backupsv1alpha1.Backup); ok { + return apierrors.NewForbidden(backupsv1alpha1.GroupVersion.WithResource("backups").GroupResource(), obj.GetName(), errors.New("denied by policy")) + } + return cl.Create(ctx, obj, opts...) + }, + }) +} diff --git a/internal/backupcontroller/cnpg_artifact_size.go b/internal/backupcontroller/cnpg_artifact_size.go index c7d91bdb2..0756e2bb9 100644 --- a/internal/backupcontroller/cnpg_artifact_size.go +++ b/internal/backupcontroller/cnpg_artifact_size.go @@ -105,7 +105,9 @@ func (r *BackupJobReconciler) cnpgS3Connection(ctx context.Context, namespace st if err != nil { return conn, err } - conn.CAPEM = []byte(pem) + // barman-cloud gets endpointCA as AWS_CA_BUNDLE, which replaces the + // default trust store rather than adding to it. + conn.CAPEM, conn.CAOnly = []byte(pem), true } return conn, nil } diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index 7b3852317..85e55e8c2 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -126,7 +126,7 @@ func TestCNPGArtifactSizeBytes(t *testing.T) { if got := r.cnpgArtifactSizeBytes(context.Background(), "tenant", store, "srv", "20261021T100000"); got != 5 { t.Errorf("size = %d, want 5", got) } - if l.conn.AccessKey != "ID2" || l.conn.SecretKey != "KEY2" || string(l.conn.CAPEM) != "PEM" { + if l.conn.AccessKey != "ID2" || l.conn.SecretKey != "KEY2" || string(l.conn.CAPEM) != "PEM" || !l.conn.CAOnly { t.Errorf("connection = %+v", l.conn) } }) @@ -417,3 +417,37 @@ func TestCreateCNPGBackupArtifact_FailedExistenceCheckSkipsProbe(t *testing.T) { t.Errorf("S3 probed %d times after the existence check failed", l.calls) } } + +// TestReconcileCNPG_RejectedCreateFailsTheJob pins that only the read-back +// is retried: a Create the apiserver rejects still fails the BackupJob. +func TestReconcileCNPG_RejectedCreateFailsTheJob(t *testing.T) { + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Minute)), + newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)), + newRolloutInstancePod(time.Now().Add(-time.Hour))) + _, backups := f.pass() + if len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups after the first pass, want 1", len(backups)) + } + ctx := context.Background() + completed := backups[0].DeepCopy() + completed.Status.Phase = cnpgBackupPhaseComplete + if err := f.c.Update(ctx, completed); err != nil { + t.Fatalf("complete cnpg.io/Backup: %v", err) + } + f.r.Client = forbidBackupCreate(t, f.c) + + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(ctx, client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := f.r.reconcileCNPG(ctx, j, f.resolved); err != nil { + t.Fatalf("reconcileCNPG err = %v, want the BackupJob failed rather than retried", err) + } + persisted := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(ctx, client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, persisted); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if persisted.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed { + t.Errorf("phase = %q, want Failed for a rejected Create", persisted.Status.Phase) + } +} diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index b49bcae66..084abe31c 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -427,3 +427,51 @@ func TestReconcileMariaDB_UnreadableBackupIsRetried(t *testing.T) { t.Error("the BackupJob was failed although its Backup exists") } } + +// TestReconcileMariaDB_RejectedCreateFailsTheJob pins that only the +// read-back is retried: a Create the apiserver rejects still fails the +// BackupJob. +func TestReconcileMariaDB_RejectedCreateFailsTheJob(t *testing.T) { + strategyGroup := strategyv1alpha1.GroupVersion.Group + now := metav1.Now() + bj := newMariaDBBackupJob("bj-rejected", "tenant") + bj.Status.StartedAt = &now + bj.Status.Phase = backupsv1alpha1.BackupJobPhaseRunning + strategy := &strategyv1alpha1.MariaDB{ + ObjectMeta: metav1.ObjectMeta{Name: "mdb-strategy"}, + Spec: strategyv1alpha1.MariaDBSpec{Template: *newRenderedMariaDBTemplate()}, + } + operatorMariaDB := &mariadbtypes.MariaDB{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: mariadbNameForApp("mariadb-src")}} + operatorBackup := &mariadbtypes.Backup{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", Name: "op-bk", + Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: bj.Name, + backupsv1alpha1.OwningJobNamespaceLabel: bj.Namespace, + }, + }, + Status: mariadbtypes.BackupStatus{Conditions: []metav1.Condition{{ + Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionTrue, LastTransitionTime: now, + }}}, + } + store := newMariaDBStrategyTestClient(t, bj, strategy, newMariaDBApp("mariadb-src", "tenant"), operatorMariaDB, operatorBackup) + r := &BackupJobReconciler{ + Client: forbidBackupCreate(t, store), + Scheme: store.Scheme(), + Recorder: record.NewFakeRecorder(10), + } + resolved := &ResolvedBackupConfig{StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: &strategyGroup, Kind: strategyv1alpha1.MariaDBStrategyKind, Name: strategy.Name, + }} + + if _, err := r.reconcileMariaDB(context.Background(), bj, resolved); err != nil { + t.Fatalf("reconcileMariaDB err = %v, want the BackupJob failed rather than retried", err) + } + persisted := &backupsv1alpha1.BackupJob{} + if err := store.Get(context.Background(), client.ObjectKeyFromObject(bj), persisted); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if persisted.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed { + t.Errorf("phase = %q, want Failed for a rejected Create", persisted.Status.Phase) + } +} From c140354977cda456d63c55d8cf3f12c0f6b6f33c Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:13:54 +0000 Subject: [PATCH 097/518] test(backups): build operator Backups with the operator's success shape mariadb-operator marks a Backup Complete=True for a failed Job as well as a successful one; only the reason tells them apart (JobComplete with message Success, or JobFailed with message Failed). The fixtures that stand for a finished backup set Complete=True and nothing else, so they describe a state the operator never produces and would read as a failure to a driver that checks the reason. Give them the success shape the operator writes. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- .../mariadb_artifact_size_test.go | 25 ++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index 084abe31c..d4bf91a1d 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -29,6 +29,19 @@ func dumpKey(prefix string, ts time.Time, compression string) string { return fmt.Sprintf("%sbackup.%s.%s.sql", prefix, ts.UTC().Format(time.RFC3339), compression) } +// mariadbBackupSucceeded is the Complete condition mariadb-operator sets on a +// Backup whose Job succeeded. A failed Job also gets Status=True, with +// Reason=JobFailed, so a fixture without the reason is not a success. +func mariadbBackupSucceeded(at metav1.Time) metav1.Condition { + return metav1.Condition{ + Type: mariadbtypes.ConditionTypeComplete, + Status: metav1.ConditionTrue, + Reason: mariadbtypes.ConditionReasonJobComplete, + Message: "Success", + LastTransitionTime: at, + } +} + func TestParseMariaDBLogicalBackupFile(t *testing.T) { cases := map[string]bool{ "backup.2026-10-21T10:00:00Z.sql": true, @@ -258,9 +271,7 @@ func TestCreateMariaDBBackupArtifact_SizeBytes(t *testing.T) { Namespace: "tenant", Name: "src-op-bk", CreationTimestamp: metav1.NewTime(completed.Add(-3 * time.Minute)), }, - Status: mariadbtypes.BackupStatus{Conditions: []metav1.Condition{{ - Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionTrue, LastTransitionTime: completed, - }}}, + Status: mariadbtypes.BackupStatus{Conditions: []metav1.Condition{mariadbBackupSucceeded(completed)}}, } resolved := &ResolvedBackupConfig{ StrategyRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "mariadb-strategy-default"}, @@ -399,9 +410,7 @@ func TestReconcileMariaDB_UnreadableBackupIsRetried(t *testing.T) { backupsv1alpha1.OwningJobNamespaceLabel: bj.Namespace, }, }, - Status: mariadbtypes.BackupStatus{Conditions: []metav1.Condition{{ - Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionTrue, LastTransitionTime: now, - }}}, + Status: mariadbtypes.BackupStatus{Conditions: []metav1.Condition{mariadbBackupSucceeded(now)}}, } backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: bj.Name}} store := newMariaDBStrategyTestClient(t, bj, strategy, newMariaDBApp("mariadb-src", "tenant"), operatorMariaDB, operatorBackup, backup) @@ -450,9 +459,7 @@ func TestReconcileMariaDB_RejectedCreateFailsTheJob(t *testing.T) { backupsv1alpha1.OwningJobNamespaceLabel: bj.Namespace, }, }, - Status: mariadbtypes.BackupStatus{Conditions: []metav1.Condition{{ - Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionTrue, LastTransitionTime: now, - }}}, + Status: mariadbtypes.BackupStatus{Conditions: []metav1.Condition{mariadbBackupSucceeded(now)}}, } store := newMariaDBStrategyTestClient(t, bj, strategy, newMariaDBApp("mariadb-src", "tenant"), operatorMariaDB, operatorBackup) r := &BackupJobReconciler{ From 21f51a2e74c3cb3cabb639d192f7953e0028837f Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:16:15 +0000 Subject: [PATCH 098/518] refactor(backups): return the matched MariaDB dump, not only its size A MariaDB restore has to name the dump it restores from: without a target time the operator restores the newest dump under the prefix, which is not necessarily the one of the Backup being restored. The size probe already finds that dump, but it returns only its size, folds a missing, an ambiguous and an unreadable listing into the same 0, and lives on BackupJobReconciler, which the restore reconciler is not. Split the probe in two: findMariaDBDump reads the bucket and returns the matched object (key, file name, the timestamp in its name, size and LastModified) or an error that tells "not found" from "ambiguous"; the size path keeps its timeout and reads only the size. The Secret lookup and the S3 client move onto an objectStore value that any reconciler can build from its own client, and BackupJobReconciler keeps one-line wrappers so the CNPG probe is unchanged. sizeBytes is reported exactly as before. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- internal/backupcontroller/artifact_size.go | 48 ++++++-- .../backupcontroller/mariadb_artifact_size.go | 104 ++++++++++++------ .../mariadb_artifact_size_test.go | 32 ++++++ 3 files changed, 140 insertions(+), 44 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 1a822699c..063e165d6 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -16,6 +16,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" ) @@ -54,13 +55,44 @@ type s3Connection struct { CAOnly bool } -func (r *BackupJobReconciler) s3Lister(conn s3Connection) (s3ObjectLister, error) { - if r.newS3ObjectLister != nil { - return r.newS3ObjectLister(conn) +// objectStore reads back what a backup wrote: the Secrets it was written +// with and the bucket itself. It is a value rather than reconciler methods +// so that the restore path, which has a reconciler of its own, can read +// the same objects. +type objectStore struct { + // reader must reach Secrets; the manager's client reads them uncached. + reader client.Reader + // newLister overrides the S3 client; nil means a real minio client. + newLister func(conn s3Connection) (s3ObjectLister, error) +} + +func (s objectStore) lister(conn s3Connection) (s3ObjectLister, error) { + if s.newLister != nil { + return s.newLister(conn) } return newMinioObjectLister(conn) } +func (s objectStore) secretValue(ctx context.Context, namespace, name, key string) (string, error) { + secret := &corev1.Secret{} + if err := s.reader.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, secret); err != nil { + return "", err + } + v, ok := secret.Data[key] + if !ok { + return "", fmt.Errorf("secret %s/%s has no key %q", namespace, name, key) + } + return string(v), nil +} + +func (r *BackupJobReconciler) objectStore() objectStore { + return objectStore{reader: r.Client, newLister: r.newS3ObjectLister} +} + +func (r *BackupJobReconciler) s3Lister(conn s3Connection) (s3ObjectLister, error) { + return r.objectStore().lister(conn) +} + type minioObjectLister struct { client *minio.Client } @@ -179,13 +211,5 @@ func (r *BackupJobReconciler) existingBackup(ctx context.Context, namespace, nam } func (r *BackupJobReconciler) secretValue(ctx context.Context, namespace, name, key string) (string, error) { - secret := &corev1.Secret{} - if err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, secret); err != nil { - return "", err - } - v, ok := secret.Data[key] - if !ok { - return "", fmt.Errorf("secret %s/%s has no key %q", namespace, name, key) - } - return string(v), nil + return r.objectStore().secretValue(ctx, namespace, name, key) } diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index 273eb8105..bf9dedcd9 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -3,6 +3,8 @@ package backupcontroller import ( "context" + "errors" + "fmt" "strings" "time" @@ -16,15 +18,28 @@ import ( // the object store's clock, neither of which is the apiserver's. const mariadbSizeProbeClockSkew = time.Minute +var ( + errMariaDBDumpNotFound = errors.New("no dump in the bucket belongs to the operator Backup") + errMariaDBDumpAmbiguous = errors.New("the dump of the operator Backup cannot be told apart from another run's") +) + +// mariadbDump is a logical dump mariadb-operator wrote to the bucket. +type mariadbDump struct { + // Key is the object key; Name is the file name, Key without the prefix. + Key string + Name string + // Timestamp is the one in the file name, taken by the backup Job pod's + // clock as the dump started. + Timestamp time.Time + Size int64 + // LastModified is the object store's clock as the upload finished. + LastModified time.Time +} + // mariadbArtifactSizeBytes measures the dump mariadb-operator uploaded for -// mdbBackup. The strategy prefix is shared by every backup of the -// application, so the dump is identified by both the timestamp in its file -// name and its LastModified falling inside the operator Backup's lifetime: -// the two come from different clocks, so a skewed pod clock alone cannot -// pull a neighbouring run's dump into the window. Returns 0 - leaving -// Artifact.SizeBytes absent - when the object store cannot be read or the -// window does not match exactly one dump: an unknown size must not be -// reported as a measured one. +// mdbBackup. Returns 0 - leaving Artifact.SizeBytes absent - when the dump +// cannot be found or told apart: an unknown size must not be reported as a +// measured one. func (r *BackupJobReconciler) mariadbArtifactSizeBytes( ctx context.Context, namespace string, @@ -32,19 +47,33 @@ func (r *BackupJobReconciler) mariadbArtifactSizeBytes( mdbBackup *mariadbtypes.Backup, completedAt time.Time, ) int64 { - logger := getLogger(ctx) ctx, cancel := context.WithTimeout(ctx, artifactSizeProbeTimeout) defer cancel() - - conn, err := r.mariadbS3Connection(ctx, namespace, s3) + dump, err := findMariaDBDump(ctx, r.objectStore(), namespace, s3, mdbBackup, completedAt) if err != nil { - logger.Info("cannot resolve S3 credentials to measure MariaDB backup size", "backup", mdbBackup.Name, "error", err) + getLogger(ctx).Info("MariaDB backup size not measured; leaving it unset", "backup", mdbBackup.Name, "error", err) return 0 } - lister, err := r.s3Lister(conn) + return dump.Size +} + +// findMariaDBDump lists the bucket the operator Backup wrote to and returns +// the one dump that belongs to it, as matchMariaDBDump decides. +func findMariaDBDump( + ctx context.Context, + store objectStore, + namespace string, + s3 *strategyv1alpha1.MariaDBS3Template, + mdbBackup *mariadbtypes.Backup, + completedAt time.Time, +) (mariadbDump, error) { + conn, err := mariadbS3Connection(ctx, store, namespace, s3) if err != nil { - logger.Info("cannot build S3 client to measure MariaDB backup size", "backup", mdbBackup.Name, "error", err) - return 0 + return mariadbDump{}, fmt.Errorf("resolve S3 credentials: %w", err) + } + lister, err := store.lister(conn) + if err != nil { + return mariadbDump{}, fmt.Errorf("build S3 client: %w", err) } // Non-recursive, as mariadb-operator lists it: the default prefix is also // the default CNPG destinationPath, so an app of each kind with the same @@ -52,51 +81,62 @@ func (r *BackupJobReconciler) mariadbArtifactSizeBytes( prefix := mariadbOperatorS3Prefix(s3.Prefix) objects, err := lister.ListObjects(ctx, s3.Bucket, prefix, false) if err != nil { - logger.Info("cannot list MariaDB backup objects to measure size", "backup", mdbBackup.Name, "error", err) - return 0 + return mariadbDump{}, fmt.Errorf("list s3://%s/%s: %w", s3.Bucket, prefix, err) } + return matchMariaDBDump(objects, prefix, mdbBackup.CreationTimestamp.Time, completedAt) +} - lower := mdbBackup.CreationTimestamp.Add(-mariadbSizeProbeClockSkew) - upper := completedAt.Add(mariadbSizeProbeClockSkew) - var matched []s3ObjectInfo +// matchMariaDBDump picks the dump of the run that lived from created to +// completed. The prefix is shared by every backup of the application, so +// the dump is identified by both the timestamp in its file name and its +// LastModified falling inside that lifetime: the two come from different +// clocks, so a skewed pod clock alone cannot pull a neighbouring run's dump +// into the window. +func matchMariaDBDump(objects []s3ObjectInfo, prefix string, created, completed time.Time) (mariadbDump, error) { + lower := created.Add(-mariadbSizeProbeClockSkew) + upper := completed.Add(mariadbSizeProbeClockSkew) + var matched []mariadbDump for _, obj := range objects { - ts, ok := parseMariaDBLogicalBackupFile(strings.TrimPrefix(obj.Key, prefix)) + name := strings.TrimPrefix(obj.Key, prefix) + ts, ok := parseMariaDBLogicalBackupFile(name) if !ok || !inWindow(ts, lower, upper) || !inWindow(obj.LastModified, lower, upper) { continue } - matched = append(matched, obj) + matched = append(matched, mariadbDump{Key: obj.Key, Name: name, Timestamp: ts, Size: obj.Size, LastModified: obj.LastModified}) } - if len(matched) != 1 { - logger.Info("MariaDB backup dump not attributable to a single object; leaving size unset", - "backup", mdbBackup.Name, "candidates", len(matched)) - return 0 + switch len(matched) { + case 0: + return mariadbDump{}, errMariaDBDumpNotFound + case 1: + return matched[0], nil + default: + return mariadbDump{}, fmt.Errorf("%w: %d dumps fall in its lifetime", errMariaDBDumpAmbiguous, len(matched)) } - return matched[0].Size } // mariadbS3Connection resolves the Secrets and TLS settings the operator // Backup was rendered with, so the bucket is read with the tenant's // credentials rather than the controller's. -func (r *BackupJobReconciler) mariadbS3Connection(ctx context.Context, namespace string, s3 *strategyv1alpha1.MariaDBS3Template) (s3Connection, error) { +func mariadbS3Connection(ctx context.Context, store objectStore, namespace string, s3 *strategyv1alpha1.MariaDBS3Template) (s3Connection, error) { conn := s3Connection{ Endpoint: s3.Endpoint, Region: s3.Region, Secure: s3.TLS != nil && s3.TLS.Enabled, } var err error - if conn.AccessKey, err = r.secretValue(ctx, namespace, s3.AccessKeyIDSecretKeyRef.Name, s3.AccessKeyIDSecretKeyRef.Key); err != nil { + if conn.AccessKey, err = store.secretValue(ctx, namespace, s3.AccessKeyIDSecretKeyRef.Name, s3.AccessKeyIDSecretKeyRef.Key); err != nil { return conn, err } - if conn.SecretKey, err = r.secretValue(ctx, namespace, s3.SecretAccessKeySecretKeyRef.Name, s3.SecretAccessKeySecretKeyRef.Key); err != nil { + if conn.SecretKey, err = store.secretValue(ctx, namespace, s3.SecretAccessKeySecretKeyRef.Name, s3.SecretAccessKeySecretKeyRef.Key); err != nil { return conn, err } if ref := s3.SessionTokenSecretKeyRef; ref != nil { - if conn.SessionToken, err = r.secretValue(ctx, namespace, ref.Name, ref.Key); err != nil { + if conn.SessionToken, err = store.secretValue(ctx, namespace, ref.Name, ref.Key); err != nil { return conn, err } } if conn.Secure && s3.TLS.CASecretKeyRef != nil { - ca, err := r.secretValue(ctx, namespace, s3.TLS.CASecretKeyRef.Name, s3.TLS.CASecretKeyRef.Key) + ca, err := store.secretValue(ctx, namespace, s3.TLS.CASecretKeyRef.Name, s3.TLS.CASecretKeyRef.Key) if err != nil { return conn, err } diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index d4bf91a1d..c28d0ad4e 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -220,6 +220,38 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { }) } +// TestMatchMariaDBDump pins what a restore reads off a match: the object +// itself, and errors it can tell apart. +func TestMatchMariaDBDump(t *testing.T) { + created := time.Date(2026, 10, 21, 10, 0, 0, 0, time.UTC) + completed := created.Add(3 * time.Minute) + const prefix = "tenant/db/" + own := s3ObjectInfo{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4242, LastModified: created.Add(2 * time.Minute)} + + got, err := matchMariaDBDump([]s3ObjectInfo{own}, prefix, created, completed) + if err != nil { + t.Fatalf("matchMariaDBDump: %v", err) + } + want := mariadbDump{ + Key: own.Key, + Name: strings.TrimPrefix(own.Key, prefix), + Timestamp: created.Add(time.Minute), + Size: 4242, + LastModified: own.LastModified, + } + if got != want { + t.Errorf("matchMariaDBDump = %+v, want %+v", got, want) + } + + if _, err := matchMariaDBDump(nil, prefix, created, completed); !errors.Is(err, errMariaDBDumpNotFound) { + t.Errorf("no dump: err = %v, want errMariaDBDumpNotFound", err) + } + second := s3ObjectInfo{Key: dumpKey(prefix, created.Add(2*time.Minute), "gzip"), Size: 1, LastModified: created.Add(2 * time.Minute)} + if _, err := matchMariaDBDump([]s3ObjectInfo{own, second}, prefix, created, completed); !errors.Is(err, errMariaDBDumpAmbiguous) { + t.Errorf("two dumps: err = %v, want errMariaDBDumpAmbiguous", err) + } +} + // TestMariaDBArtifactSizeBytes_MinioClient drives the real minio client // against a stub ListObjectsV2 endpoint, so the Secret wiring, prefix and // response parsing are exercised together. From 560c44f71eba87c6bb08bb119e7572abdc9bfc69 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:18:39 +0000 Subject: [PATCH 099/518] fix(backups): read a MariaDB dump from where its operator Backup wrote it The size probe took the bucket, endpoint, prefix, credentials and TLS settings from the strategy as it renders on the reconcile that creates the Backup, which is not necessarily what it rendered when the operator Backup was created: an existing operator Backup is reused untouched, and a backup can run for up to the driver's 30-minute deadline. An admin who edits the strategy in between sends the probe to the new location, where it measures the wrong prefix or nothing at all. The operator Backup's own spec.storage.s3 is where the dump went, so the probe reads that. It is also the only source a restore can use, since it targets that operator Backup long after the strategy may have changed. An operator Backup that does not store to S3 is refused before anything is listed. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- .../backupcontroller/mariadb_artifact_size.go | 43 +++--- .../mariadb_artifact_size_test.go | 139 ++++++++++++++---- .../mariadbstrategy_controller.go | 2 +- 3 files changed, 131 insertions(+), 53 deletions(-) diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index bf9dedcd9..d10f1c4a5 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -8,7 +8,6 @@ import ( "strings" "time" - strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" ) @@ -40,16 +39,10 @@ type mariadbDump struct { // mdbBackup. Returns 0 - leaving Artifact.SizeBytes absent - when the dump // cannot be found or told apart: an unknown size must not be reported as a // measured one. -func (r *BackupJobReconciler) mariadbArtifactSizeBytes( - ctx context.Context, - namespace string, - s3 *strategyv1alpha1.MariaDBS3Template, - mdbBackup *mariadbtypes.Backup, - completedAt time.Time, -) int64 { +func (r *BackupJobReconciler) mariadbArtifactSizeBytes(ctx context.Context, mdbBackup *mariadbtypes.Backup, completedAt time.Time) int64 { ctx, cancel := context.WithTimeout(ctx, artifactSizeProbeTimeout) defer cancel() - dump, err := findMariaDBDump(ctx, r.objectStore(), namespace, s3, mdbBackup, completedAt) + dump, err := findMariaDBDump(ctx, r.objectStore(), mdbBackup, completedAt) if err != nil { getLogger(ctx).Info("MariaDB backup size not measured; leaving it unset", "backup", mdbBackup.Name, "error", err) return 0 @@ -57,17 +50,17 @@ func (r *BackupJobReconciler) mariadbArtifactSizeBytes( return dump.Size } -// findMariaDBDump lists the bucket the operator Backup wrote to and returns -// the one dump that belongs to it, as matchMariaDBDump decides. -func findMariaDBDump( - ctx context.Context, - store objectStore, - namespace string, - s3 *strategyv1alpha1.MariaDBS3Template, - mdbBackup *mariadbtypes.Backup, - completedAt time.Time, -) (mariadbDump, error) { - conn, err := mariadbS3Connection(ctx, store, namespace, s3) +// findMariaDBDump lists the bucket mdbBackup wrote to and returns the one +// dump that belongs to it, as matchMariaDBDump decides. The location and +// credentials come from mdbBackup's own spec: the strategy it was rendered +// from can have been edited since, and pointing at another bucket now says +// nothing about where this dump went. +func findMariaDBDump(ctx context.Context, store objectStore, mdbBackup *mariadbtypes.Backup, completedAt time.Time) (mariadbDump, error) { + s3 := mdbBackup.Spec.Storage.S3 + if s3 == nil { + return mariadbDump{}, errors.New("the operator Backup does not store to S3") + } + conn, err := mariadbS3Connection(ctx, store, mdbBackup.Namespace, s3) if err != nil { return mariadbDump{}, fmt.Errorf("resolve S3 credentials: %w", err) } @@ -114,17 +107,17 @@ func matchMariaDBDump(objects []s3ObjectInfo, prefix string, created, completed } } -// mariadbS3Connection resolves the Secrets and TLS settings the operator -// Backup was rendered with, so the bucket is read with the tenant's -// credentials rather than the controller's. -func mariadbS3Connection(ctx context.Context, store objectStore, namespace string, s3 *strategyv1alpha1.MariaDBS3Template) (s3Connection, error) { +// mariadbS3Connection resolves the Secrets and TLS settings of the operator +// Backup's storage, so the bucket is read with the tenant's credentials +// rather than the controller's. +func mariadbS3Connection(ctx context.Context, store objectStore, namespace string, s3 *mariadbtypes.S3Storage) (s3Connection, error) { conn := s3Connection{ Endpoint: s3.Endpoint, Region: s3.Region, Secure: s3.TLS != nil && s3.TLS.Enabled, } var err error - if conn.AccessKey, err = store.secretValue(ctx, namespace, s3.AccessKeyIDSecretKeyRef.Name, s3.AccessKeyIDSecretKeyRef.Key); err != nil { + if conn.AccessKey, err = store.secretValue(ctx, namespace, s3.AccessKeyIdSecretKeyRef.Name, s3.AccessKeyIdSecretKeyRef.Key); err != nil { return conn, err } if conn.SecretKey, err = store.secretValue(ctx, namespace, s3.SecretAccessKeySecretKeyRef.Name, s3.SecretAccessKeySecretKeyRef.Key); err != nil { diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index c28d0ad4e..719678892 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -77,14 +77,26 @@ func TestMariaDBOperatorS3Prefix(t *testing.T) { } } +// operatorBackupStoringTo returns the operator Backup a strategy with s3 +// storage renders, created at created in namespace tenant. +func operatorBackupStoringTo(t *testing.T, s3 *strategyv1alpha1.MariaDBS3Template, created time.Time) *mariadbtypes.Backup { + t.Helper() + storage, err := buildMariaDBBackupStorage(strategyv1alpha1.MariaDBStorageTemplate{S3: s3}) + if err != nil { + t.Fatal(err) + } + return &mariadbtypes.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "op-bk", CreationTimestamp: metav1.NewTime(created)}, + Spec: mariadbtypes.BackupSpec{Storage: storage}, + } +} + func TestMariaDBArtifactSizeBytes(t *testing.T) { created := time.Date(2026, 10, 21, 10, 0, 0, 0, time.UTC) completed := created.Add(3 * time.Minute) - mdbBackup := &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{ - Namespace: "tenant", Name: "op-bk", CreationTimestamp: metav1.NewTime(created), - }} s3 := newRenderedMariaDBTemplate().Storage.S3 s3.Prefix = "tenant/db" + mdbBackup := operatorBackupStoringTo(t, s3, created) const prefix = "tenant/db/" t.Run("single dump inside the backup window", func(t *testing.T) { @@ -93,7 +105,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4242, LastModified: created.Add(time.Minute)}, {Key: prefix + "0-backup-target.txt", Size: 7}, }} - got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed) + got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed) if got != 4242 { t.Errorf("size = %d, want 4242", got) } @@ -118,7 +130,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { } l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 3, LastModified: created.Add(time.Minute)}}} r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds"), ca) - if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", tlsS3, mdbBackup, completed); got != 3 { + if got := r.mariadbArtifactSizeBytes(context.Background(), operatorBackupStoringTo(t, tlsS3, created), completed); got != 3 { t.Errorf("size = %d, want 3", got) } if !l.conn.Secure || string(l.conn.CAPEM) != "PEM" { @@ -136,7 +148,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { } l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4, LastModified: created.Add(time.Minute)}}} r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds"), sts) - if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", stsS3, mdbBackup, completed); got != 4 { + if got := r.mariadbArtifactSizeBytes(context.Background(), operatorBackupStoringTo(t, stsS3, created), completed); got != 4 { t.Errorf("size = %d, want 4", got) } if l.conn.SessionToken != "TOKEN" { @@ -148,7 +160,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey(prefix, created.Add(-30*time.Second), ""), Size: 9, LastModified: created.Add(time.Minute)}, }} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 9 { + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 9 { t.Errorf("size = %d, want 9", got) } }) @@ -161,7 +173,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { {Key: dumpKey(prefix, created.Add(-4*time.Minute), "gzip"), Size: 7, LastModified: created.Add(time.Minute)}, {Key: dumpKey(prefix, created.Add(30*time.Second), "gzip"), Size: 666, LastModified: created.Add(-4 * time.Minute)}, }} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0: no dump is inside the window on both clocks", got) } }) @@ -170,7 +182,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey(prefix, completed.Add(30*time.Second), "gzip"), Size: 5, LastModified: completed.Add(30 * time.Second)}, }} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 5 { + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 5 { t.Errorf("size = %d, want 5", got) } }) @@ -179,7 +191,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { r := reconcilerWithLister(t, &fakeS3Lister{}, s3CredsSecret("tenant", "creds")) r.newS3ObjectLister = func(s3Connection) (s3ObjectLister, error) { return blockingLister{}, nil } withShortProbeTimeout(t, func() { - if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + if got := r.mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0", got) } }) @@ -190,7 +202,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 1, LastModified: created.Add(time.Minute)}, {Key: dumpKey(prefix, created.Add(2*time.Minute), "gzip"), Size: 2, LastModified: created.Add(2 * time.Minute)}, }} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0 (ambiguous)", got) } }) @@ -199,14 +211,14 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey(prefix, completed.Add(time.Hour), "gzip"), Size: 1, LastModified: completed.Add(time.Hour)}, }} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0", got) } }) t.Run("list error leaves size unset", func(t *testing.T) { l := &fakeS3Lister{err: errors.New("AccessDenied")} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0", got) } }) @@ -214,7 +226,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { t.Run("missing credentials Secret leaves size unset", func(t *testing.T) { c := newMariaDBStrategyTestClient(t) r := &BackupJobReconciler{Client: c, Scheme: c.Scheme()} - if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, completed); got != 0 { + if got := r.mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { t.Errorf("size = %d, want 0", got) } }) @@ -252,6 +264,45 @@ func TestMatchMariaDBDump(t *testing.T) { } } +// TestFindMariaDBDump pins that a dump is read with nothing but a client, +// as a reconciler other than BackupJobReconciler has, and that an operator +// Backup that does not store to S3 is refused without a listing. +func TestFindMariaDBDump(t *testing.T) { + created := time.Date(2026, 10, 21, 10, 0, 0, 0, time.UTC) + completed := created.Add(3 * time.Minute) + s3 := newRenderedMariaDBTemplate().Storage.S3 + s3.Prefix = "tenant/db" + l := &fakeS3Lister{objects: []s3ObjectInfo{ + {Key: dumpKey("tenant/db/", created.Add(time.Minute), "gzip"), Size: 4242, LastModified: created.Add(time.Minute)}, + }} + store := objectStore{ + reader: newMariaDBStrategyTestClient(t, s3CredsSecret("tenant", "creds")), + newLister: func(conn s3Connection) (s3ObjectLister, error) { + l.conn = conn + return l, nil + }, + } + + got, err := findMariaDBDump(context.Background(), store, operatorBackupStoringTo(t, s3, created), completed) + if err != nil || got.Size != 4242 { + t.Fatalf("findMariaDBDump = %+v, %v; want the 4242-byte dump", got, err) + } + if l.conn.AccessKey != "AKID" { + t.Errorf("connection = %+v, want the Secret's key", l.conn) + } + + pvc := &mariadbtypes.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "op-pvc", CreationTimestamp: metav1.NewTime(created)}, + Spec: mariadbtypes.BackupSpec{Storage: mariadbtypes.BackupStorage{PersistentVolumeClaim: &corev1.PersistentVolumeClaimSpec{}}}, + } + if _, err := findMariaDBDump(context.Background(), store, pvc, completed); err == nil { + t.Error("findMariaDBDump on a PVC-backed operator Backup returned no error") + } + if l.calls != 1 { + t.Errorf("S3 listed %d times, want once: a PVC-backed Backup has nothing to list", l.calls) + } +} + // TestMariaDBArtifactSizeBytes_MinioClient drives the real minio client // against a stub ListObjectsV2 endpoint, so the Secret wiring, prefix and // response parsing are exercised together. @@ -280,11 +331,8 @@ func TestMariaDBArtifactSizeBytes_MinioClient(t *testing.T) { s3.Endpoint = strings.TrimPrefix(srv.URL, "http://") s3.Prefix = "tenant/db/" s3.Region = "us-east-1" - mdbBackup := &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{ - Namespace: "tenant", Name: "op-bk", CreationTimestamp: metav1.NewTime(created), - }} - if got := r.mariadbArtifactSizeBytes(context.Background(), "tenant", s3, mdbBackup, created.Add(3*time.Minute)); got != 31337 { + if got := r.mariadbArtifactSizeBytes(context.Background(), operatorBackupStoringTo(t, s3, created), created.Add(3*time.Minute)); got != 31337 { t.Fatalf("size = %d, want 31337", got) } if gotPath != "/bkt/" || gotPrefix != "tenant/db/" || gotDelimiter != "/" { @@ -298,13 +346,11 @@ func TestMariaDBArtifactSizeBytes_MinioClient(t *testing.T) { func TestCreateMariaDBBackupArtifact_SizeBytes(t *testing.T) { apps := mariadbapp.GroupName completed := metav1.NewTime(time.Date(2026, 10, 21, 10, 3, 0, 0, time.UTC)) - mdbBackup := &mariadbtypes.Backup{ - ObjectMeta: metav1.ObjectMeta{ - Namespace: "tenant", Name: "src-op-bk", - CreationTimestamp: metav1.NewTime(completed.Add(-3 * time.Minute)), - }, - Status: mariadbtypes.BackupStatus{Conditions: []metav1.Condition{mariadbBackupSucceeded(completed)}}, - } + rendered := newRenderedMariaDBTemplate() + rendered.Storage.S3.Prefix = "src/" + mdbBackup := operatorBackupStoringTo(t, rendered.Storage.S3, completed.Add(-3*time.Minute)) + mdbBackup.Name = "src-op-bk" + mdbBackup.Status.Conditions = []metav1.Condition{mariadbBackupSucceeded(completed)} resolved := &ResolvedBackupConfig{ StrategyRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "mariadb-strategy-default"}, Parameters: map[string]string{}, @@ -318,8 +364,6 @@ func TestCreateMariaDBBackupArtifact_SizeBytes(t *testing.T) { r := reconcilerWithLister(t, &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey("src/", completed.Add(-time.Minute), "gzip"), Size: 2048, LastModified: completed.Add(-time.Minute)}, }}, job, s3CredsSecret("tenant", "creds")) - rendered := newRenderedMariaDBTemplate() - rendered.Storage.S3.Prefix = "src/" artefact, err := r.createMariaDBBackupArtifact(context.Background(), job, resolved, mdbBackup, rendered) if err != nil { @@ -330,6 +374,47 @@ func TestCreateMariaDBBackupArtifact_SizeBytes(t *testing.T) { } } +// TestCreateMariaDBBackupArtifact_StrategyEditedMidBackup pins that the +// probe reads where the operator Backup wrote its dump, not where the +// strategy, edited while the backup ran, now points. +func TestCreateMariaDBBackupArtifact_StrategyEditedMidBackup(t *testing.T) { + apps := mariadbapp.GroupName + completed := metav1.NewTime(time.Date(2026, 10, 21, 10, 3, 0, 0, time.UTC)) + original := newRenderedMariaDBTemplate().Storage.S3 + original.Prefix = "original/" + mdbBackup := operatorBackupStoringTo(t, original, completed.Add(-3*time.Minute)) + mdbBackup.Status.Conditions = []metav1.Condition{mariadbBackupSucceeded(completed)} + edited := newRenderedMariaDBTemplate() + edited.Storage.S3.Bucket = "replacement-bkt" + edited.Storage.S3.Endpoint = "replacement.example:9000" + edited.Storage.S3.Prefix = "replacement/" + job := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-edited"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, + }, + } + l := &fakeS3Lister{objects: []s3ObjectInfo{ + {Key: dumpKey("original/", completed.Add(-time.Minute), "gzip"), Size: 2048, LastModified: completed.Add(-time.Minute)}, + }} + r := reconcilerWithLister(t, l, job, s3CredsSecret("tenant", "creds")) + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "mariadb-strategy-default"}, + Parameters: map[string]string{}, + } + + artefact, err := r.createMariaDBBackupArtifact(context.Background(), job, resolved, mdbBackup, edited) + if err != nil { + t.Fatalf("createMariaDBBackupArtifact: %v", err) + } + if l.bucket != "bkt" || l.prefix != "original/" || l.conn.Endpoint != "s3.example:9000" { + t.Errorf("listed %s/%s at %s, want bkt/original/ at s3.example:9000, where the operator Backup wrote", l.bucket, l.prefix, l.conn.Endpoint) + } + if artefact.Status.Artifact == nil || artefact.Status.Artifact.SizeBytes != 2048 { + t.Errorf("Artifact = %#v, want SizeBytes 2048", artefact.Status.Artifact) + } +} + func TestCreateMariaDBBackupArtifact_ExistingBackupSkipsProbe(t *testing.T) { apps := mariadbapp.GroupName job := &backupsv1alpha1.BackupJob{ diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 749d63c9c..4d88ab8a1 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -398,7 +398,7 @@ func (r *BackupJobReconciler) createMariaDBBackupArtifact( if uri := driverMD[mariadbStorageURIKey]; uri != "" { status.Artifact = &backupsv1alpha1.BackupArtifact{URI: uri} if probe { - status.Artifact.SizeBytes = r.mariadbArtifactSizeBytes(ctx, j.Namespace, rendered.Storage.S3, mdbBackup, takenAt.Time) + status.Artifact.SizeBytes = r.mariadbArtifactSizeBytes(ctx, mdbBackup, takenAt.Time) } } backup := &backupsv1alpha1.Backup{ From 59f0013b1ff205a62387dafff80349b10e8f660b Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:21:55 +0000 Subject: [PATCH 100/518] fix(backups): attribute a MariaDB dump only when no other run can own it Every backup of a MariaDB application writes its dump under one shared prefix, so the probe took the dump whose file-name timestamp (the backup pod's clock) and LastModified (the object store's clock) both fall in the operator Backup's lifetime, with a minute of slack for skew. That keeps a neighbour's dump out only while this run's own dump is in the window. With the pod clock more than the slack behind, the own dump is named before the window opens, and a previous run that finished within the minute before this one was created becomes the single match: a 123-byte dump was reported as 666 bytes. A pod clock ahead or an object store clock behind does the same with the next run, and an own dump that is gone by the time it is looked up, as it can be for a restore long after the backup, leaves a neighbour as the single match with no skew at all. A dump is now attributed only when ownership is proven: - it is the one dump that either clock places in the window. A dump that one clock places there and the other does not may be this run's own with a skewed clock, so it leaves any other match unproven, the way a second dump in the window already did, and is not taken itself; - both of its clocks place it in the window; - neither clock places it in the window of another operator Backup that writes under the same bucket and prefix and has completed. This rejects a neighbour's dump when this run's own is missing, which the clocks alone cannot tell. Operator Backups still running are not counted as neighbours: their window has no end, and one stuck short of Complete (a Job whose pod never starts stays JobRunning for good) would otherwise keep every later run of the application from being sized. A dump such a run writes inside this run's window still competes with this run's own. Narrower slack does not prove ownership: a neighbour that finished within the narrower slack still matches, and a smaller skew pushes the own dump out. The Job's start and completion times are stamped by the control plane, like the operator Backup's own, so they narrow the window by seconds without removing the dependence on the two clocks a dump carries. What is left: a neighbour's dump is taken only when this run's own is missing from the listing or misplaced on both clocks and the neighbour's operator Backup no longer exists or has not completed. Every other unproven case leaves the size unset, as two runs that overlap or follow each other within the slack already did. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- internal/backupcontroller/artifact_size.go | 3 +- .../backupcontroller/mariadb_artifact_size.go | 115 ++++++++++--- .../mariadb_artifact_size_test.go | 155 +++++++++++++++++- 3 files changed, 250 insertions(+), 23 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 063e165d6..941b00323 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -60,7 +60,8 @@ type s3Connection struct { // so that the restore path, which has a reconciler of its own, can read // the same objects. type objectStore struct { - // reader must reach Secrets; the manager's client reads them uncached. + // reader reads Secrets, which the manager's client fetches uncached, + // and lists operator Backups, which it serves from the cache. reader client.Reader // newLister overrides the S3 client; nil means a real minio client. newLister func(conn s3Connection) (s3ObjectLister, error) diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index d10f1c4a5..6397325ca 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -8,6 +8,10 @@ import ( "strings" "time" + apimeta "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" ) @@ -60,6 +64,10 @@ func findMariaDBDump(ctx context.Context, store objectStore, mdbBackup *mariadbt if s3 == nil { return mariadbDump{}, errors.New("the operator Backup does not store to S3") } + others, err := finishedMariaDBRuns(ctx, store.reader, mdbBackup) + if err != nil { + return mariadbDump{}, fmt.Errorf("list operator Backups: %w", err) + } conn, err := mariadbS3Connection(ctx, store, mdbBackup.Namespace, s3) if err != nil { return mariadbDump{}, fmt.Errorf("resolve S3 credentials: %w", err) @@ -76,35 +84,104 @@ func findMariaDBDump(ctx context.Context, store objectStore, mdbBackup *mariadbt if err != nil { return mariadbDump{}, fmt.Errorf("list s3://%s/%s: %w", s3.Bucket, prefix, err) } - return matchMariaDBDump(objects, prefix, mdbBackup.CreationTimestamp.Time, completedAt) + own := newMariaDBRun(mdbBackup, mdbBackup.CreationTimestamp.Time, completedAt) + return matchMariaDBDump(objects, prefix, own, others) } -// matchMariaDBDump picks the dump of the run that lived from created to -// completed. The prefix is shared by every backup of the application, so -// the dump is identified by both the timestamp in its file name and its -// LastModified falling inside that lifetime: the two come from different -// clocks, so a skewed pod clock alone cannot pull a neighbouring run's dump -// into the window. -func matchMariaDBDump(objects []s3ObjectInfo, prefix string, created, completed time.Time) (mariadbDump, error) { - lower := created.Add(-mariadbSizeProbeClockSkew) - upper := completed.Add(mariadbSizeProbeClockSkew) - var matched []mariadbDump +// mariadbRun is the span in which an operator Backup can have written its +// dump: its lifetime, widened by mariadbSizeProbeClockSkew on each side. +type mariadbRun struct { + name string + lower, upper time.Time +} + +func newMariaDBRun(b *mariadbtypes.Backup, created, completed time.Time) mariadbRun { + return mariadbRun{ + name: b.Namespace + "/" + b.Name, + lower: created.Add(-mariadbSizeProbeClockSkew), + upper: completed.Add(mariadbSizeProbeClockSkew), + } +} + +func (r mariadbRun) placesOnBothClocks(d mariadbDump) bool { + return inWindow(d.Timestamp, r.lower, r.upper) && inWindow(d.LastModified, r.lower, r.upper) +} + +func (r mariadbRun) placesOnEitherClock(d mariadbDump) bool { + return inWindow(d.Timestamp, r.lower, r.upper) || inWindow(d.LastModified, r.lower, r.upper) +} + +// matchMariaDBDump picks the dump of run own among objects, which share a +// prefix with every other run of the application. A dump carries two +// clocks, the backup pod's in its file name and the object store's in its +// LastModified, and either can be skewed. A dump is taken only when it is +// the one dump either clock places in own's window, both clocks place it +// there, and neither places it in the window of another finished run: a +// dump in the window on one clock only may be this run's own, misplaced by +// a skewed clock, which leaves any other candidate unproven. A neighbour's +// dump is still taken if this run's own is missing from the listing, or +// misplaced on both clocks, and the neighbour's operator Backup is gone or +// has not completed. +func matchMariaDBDump(objects []s3ObjectInfo, prefix string, own mariadbRun, others []mariadbRun) (mariadbDump, error) { + var near []mariadbDump for _, obj := range objects { name := strings.TrimPrefix(obj.Key, prefix) ts, ok := parseMariaDBLogicalBackupFile(name) - if !ok || !inWindow(ts, lower, upper) || !inWindow(obj.LastModified, lower, upper) { + if !ok { continue } - matched = append(matched, mariadbDump{Key: obj.Key, Name: name, Timestamp: ts, Size: obj.Size, LastModified: obj.LastModified}) + d := mariadbDump{Key: obj.Key, Name: name, Timestamp: ts, Size: obj.Size, LastModified: obj.LastModified} + if own.placesOnEitherClock(d) { + near = append(near, d) + } } - switch len(matched) { - case 0: + if len(near) == 0 { return mariadbDump{}, errMariaDBDumpNotFound - case 1: - return matched[0], nil - default: - return mariadbDump{}, fmt.Errorf("%w: %d dumps fall in its lifetime", errMariaDBDumpAmbiguous, len(matched)) } + if len(near) > 1 { + return mariadbDump{}, fmt.Errorf("%w: %d dumps fall in its lifetime on at least one clock", errMariaDBDumpAmbiguous, len(near)) + } + d := near[0] + if !own.placesOnBothClocks(d) { + return mariadbDump{}, fmt.Errorf("%w: %s falls in its lifetime on one clock only", errMariaDBDumpAmbiguous, d.Name) + } + for _, other := range others { + if other.placesOnEitherClock(d) { + return mariadbDump{}, fmt.Errorf("%w: %s also falls in the lifetime of operator Backup %s", errMariaDBDumpAmbiguous, d.Name, other.name) + } + } + return d, nil +} + +// finishedMariaDBRuns returns the runs of the other operator Backups that +// write under the same bucket and prefix as b and have completed, whether +// their Job succeeded or failed. One still running is left out: its window +// has no end yet, and an operator Backup stuck short of Complete, as one +// whose Job pod never starts is, would otherwise contest every later run of +// the application for good. The endpoint is not compared, since one has +// several spellings, and a false neighbour only leaves a size unset. +func finishedMariaDBRuns(ctx context.Context, reader client.Reader, b *mariadbtypes.Backup) ([]mariadbRun, error) { + list := &mariadbtypes.BackupList{} + if err := reader.List(ctx, list); err != nil { + return nil, err + } + var runs []mariadbRun + for i := range list.Items { + other := &list.Items[i] + if other.Namespace == b.Namespace && other.Name == b.Name { + continue + } + s3 := other.Spec.Storage.S3 + if s3 == nil || s3.Bucket != b.Spec.Storage.S3.Bucket || mariadbOperatorS3Prefix(s3.Prefix) != mariadbOperatorS3Prefix(b.Spec.Storage.S3.Prefix) { + continue + } + cond := apimeta.FindStatusCondition(other.Status.Conditions, mariadbtypes.ConditionTypeComplete) + if cond == nil || cond.Status != metav1.ConditionTrue { + continue + } + runs = append(runs, newMariaDBRun(other, other.CreationTimestamp.Time, cond.LastTransitionTime.Time)) + } + return runs, nil } // mariadbS3Connection resolves the Secrets and TLS settings of the operator diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index 719678892..9d649c15c 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -239,8 +239,9 @@ func TestMatchMariaDBDump(t *testing.T) { completed := created.Add(3 * time.Minute) const prefix = "tenant/db/" own := s3ObjectInfo{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4242, LastModified: created.Add(2 * time.Minute)} + run := newMariaDBRun(&mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "op-bk"}}, created, completed) - got, err := matchMariaDBDump([]s3ObjectInfo{own}, prefix, created, completed) + got, err := matchMariaDBDump([]s3ObjectInfo{own}, prefix, run, nil) if err != nil { t.Fatalf("matchMariaDBDump: %v", err) } @@ -255,11 +256,11 @@ func TestMatchMariaDBDump(t *testing.T) { t.Errorf("matchMariaDBDump = %+v, want %+v", got, want) } - if _, err := matchMariaDBDump(nil, prefix, created, completed); !errors.Is(err, errMariaDBDumpNotFound) { + if _, err := matchMariaDBDump(nil, prefix, run, nil); !errors.Is(err, errMariaDBDumpNotFound) { t.Errorf("no dump: err = %v, want errMariaDBDumpNotFound", err) } second := s3ObjectInfo{Key: dumpKey(prefix, created.Add(2*time.Minute), "gzip"), Size: 1, LastModified: created.Add(2 * time.Minute)} - if _, err := matchMariaDBDump([]s3ObjectInfo{own, second}, prefix, created, completed); !errors.Is(err, errMariaDBDumpAmbiguous) { + if _, err := matchMariaDBDump([]s3ObjectInfo{own, second}, prefix, run, nil); !errors.Is(err, errMariaDBDumpAmbiguous) { t.Errorf("two dumps: err = %v, want errMariaDBDumpAmbiguous", err) } } @@ -303,6 +304,154 @@ func TestFindMariaDBDump(t *testing.T) { } } +// TestFindMariaDBDump_Ownership drives runs of one application that share +// a prefix. This run lives from t0 to t0+40s, so its window with the +// one-minute slack is [t0-1m, t0+1m40s]; its own dump is 123 bytes and a +// neighbour's is 666 or 777. A neighbour's dump must never be returned. +func TestFindMariaDBDump_Ownership(t *testing.T) { + t0 := time.Date(2026, 10, 21, 10, 0, 0, 0, time.UTC) + s3 := newRenderedMariaDBTemplate().Storage.S3 + s3.Prefix = "tenant/db" + const prefix = "tenant/db/" + run := func(name string, created, completed time.Time) *mariadbtypes.Backup { + b := operatorBackupStoringTo(t, s3, created) + b.Name = name + b.Status.Conditions = []metav1.Condition{mariadbBackupSucceeded(metav1.NewTime(completed))} + return b + } + dump := func(named, uploaded time.Time, size int64) s3ObjectInfo { + return s3ObjectInfo{Key: dumpKey(prefix, named, "gzip"), Size: size, LastModified: uploaded} + } + own := run("op-bk", t0, t0.Add(40*time.Second)) + find := func(t *testing.T, objects []s3ObjectInfo, operatorBackups ...client.Object) (mariadbDump, error) { + t.Helper() + store := objectStore{ + reader: newMariaDBStrategyTestClient(t, append(operatorBackups, own, s3CredsSecret("tenant", "creds"))...), + newLister: func(s3Connection) (s3ObjectLister, error) { + return &fakeS3Lister{objects: objects}, nil + }, + } + return findMariaDBDump(context.Background(), store, own, t0.Add(40*time.Second)) + } + wantAmbiguous := func(t *testing.T, got mariadbDump, err error) { + t.Helper() + if !errors.Is(err, errMariaDBDumpAmbiguous) { + t.Errorf("findMariaDBDump = %+v, %v; want errMariaDBDumpAmbiguous", got, err) + } + } + wantOwn := func(t *testing.T, got mariadbDump, err error) { + t.Helper() + if err != nil || got.Size != 123 { + t.Errorf("findMariaDBDump = %+v, %v; want this run's 123-byte dump", got, err) + } + } + + t.Run("a pod clock behind does not hand this run the previous run's dump", func(t *testing.T) { + // The previous run finished 20s before this one was created and its + // operator Backup is gone. This run's pod clock is 5m behind, so its + // own dump is named outside the window and uploaded inside it. + got, err := find(t, []s3ObjectInfo{ + dump(t0.Add(-30*time.Second), t0.Add(-20*time.Second), 666), + dump(t0.Add(5*time.Second-5*time.Minute), t0.Add(30*time.Second), 123), + }) + wantAmbiguous(t, got, err) + }) + + t.Run("a pod clock ahead does not hand this run the next run's dump", func(t *testing.T) { + // The next run started 5s after this one completed and has uploaded + // its dump by the time this one is probed; it has not completed yet. + next := operatorBackupStoringTo(t, s3, t0.Add(45*time.Second)) + next.Name = "op-next" + next.Status.Conditions = []metav1.Condition{{ + Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionFalse, + Reason: mariadbtypes.ConditionReasonJobRunning, Message: "Running", LastTransitionTime: metav1.NewTime(t0.Add(45 * time.Second)), + }} + got, err := find(t, []s3ObjectInfo{ + dump(t0.Add(5*time.Second+5*time.Minute), t0.Add(30*time.Second), 123), + dump(t0.Add(50*time.Second), t0.Add(80*time.Second), 777), + }, next) + wantAmbiguous(t, got, err) + }) + + t.Run("an object store clock behind does not hand this run the next run's dump", func(t *testing.T) { + // The store clock is 2m behind: this run's upload is stamped before + // the window opens, the next run's is pulled back into it. + got, err := find(t, []s3ObjectInfo{ + dump(t0.Add(5*time.Second), t0.Add(30*time.Second-2*time.Minute), 123), + dump(t0.Add(50*time.Second), t0.Add(3*time.Minute-2*time.Minute), 777), + }) + wantAmbiguous(t, got, err) + }) + + t.Run("a dump another finished run wrote is not taken when this run's own is gone", func(t *testing.T) { + // This run's dump was pruned; the next run's falls in this run's + // window on both clocks, and in its own. + next := run("op-next", t0.Add(45*time.Second), t0.Add(85*time.Second)) + got, err := find(t, []s3ObjectInfo{ + dump(t0.Add(50*time.Second), t0.Add(75*time.Second), 777), + }, next) + wantAmbiguous(t, got, err) + }) + + t.Run("a dump the window holds on one clock only is not taken", func(t *testing.T) { + // This run's dump is gone; the previous run's operator Backup is too, + // and its dump, started 2m before this run, finished uploading 20s + // before it. + got, err := find(t, []s3ObjectInfo{ + dump(t0.Add(-2*time.Minute), t0.Add(-20*time.Second), 666), + }) + if err == nil { + t.Errorf("findMariaDBDump = %+v; want no dump attributed", got) + } + }) + + t.Run("a run still in progress does not contest this run's dump", func(t *testing.T) { + // A concurrent run created 30s before this one has not uploaded yet. + running := operatorBackupStoringTo(t, s3, t0.Add(-30*time.Second)) + running.Name = "op-running" + running.Status.Conditions = []metav1.Condition{{ + Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionFalse, + Reason: mariadbtypes.ConditionReasonJobRunning, Message: "Running", LastTransitionTime: metav1.NewTime(t0.Add(-30 * time.Second)), + }} + got, err := find(t, []s3ObjectInfo{dump(t0.Add(5*time.Second), t0.Add(30*time.Second), 123)}, running) + wantOwn(t, got, err) + }) + + t.Run("runs minutes apart each keep their own dump", func(t *testing.T) { + previous := run("op-previous", t0.Add(-4*time.Minute), t0.Add(-3*time.Minute)) + next := run("op-next", t0.Add(3*time.Minute), t0.Add(4*time.Minute)) + got, err := find(t, []s3ObjectInfo{ + dump(t0.Add(-230*time.Second), t0.Add(-190*time.Second), 666), + dump(t0.Add(5*time.Second), t0.Add(30*time.Second), 123), + dump(t0.Add(185*time.Second), t0.Add(220*time.Second), 777), + }, previous, next) + wantOwn(t, got, err) + }) + + t.Run("a run that writes elsewhere does not contest this run's dump", func(t *testing.T) { + other := newRenderedMariaDBTemplate().Storage.S3 + other.Prefix = "tenant/other" + elsewhere := operatorBackupStoringTo(t, other, t0) + elsewhere.Name = "op-elsewhere" + elsewhere.Status.Conditions = []metav1.Condition{mariadbBackupSucceeded(metav1.NewTime(t0.Add(40 * time.Second)))} + got, err := find(t, []s3ObjectInfo{dump(t0.Add(5*time.Second), t0.Add(30*time.Second), 123)}, elsewhere) + wantOwn(t, got, err) + }) + + t.Run("a run stuck short of Complete does not contest later runs", func(t *testing.T) { + // A backup Job whose pod never starts leaves its operator Backup + // JobRunning for good. + stuck := operatorBackupStoringTo(t, s3, t0.Add(-3*time.Hour)) + stuck.Name = "op-stuck" + stuck.Status.Conditions = []metav1.Condition{{ + Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionFalse, + Reason: mariadbtypes.ConditionReasonJobRunning, Message: "Running", LastTransitionTime: metav1.NewTime(t0.Add(-3 * time.Hour)), + }} + got, err := find(t, []s3ObjectInfo{dump(t0.Add(5*time.Second), t0.Add(30*time.Second), 123)}, stuck) + wantOwn(t, got, err) + }) +} + // TestMariaDBArtifactSizeBytes_MinioClient drives the real minio client // against a stub ListObjectsV2 endpoint, so the Secret wiring, prefix and // response parsing are exercised together. From c9ce4c480f6a589edfaa13539990f18ff661b28f Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 17:12:31 +0300 Subject: [PATCH 101/518] fix(clickhouse): let the backup sidecar pick up credentials projected after it started With backup.useSystemBucket the clickhouse-backup sidecar took its S3 settings from optional secretKeyRefs on cozy-backups-creds. The platform projects that Secret into the namespace at the first BackupJob, usually after the ClickHouse pod started, and a container's environment is fixed at start, so the sidecar ran without credentials and every backup and restore failed until the pod was restarted by hand. Rotated keys stayed stale the same way. The projector now also writes the s3 section of a clickhouse-backup config into the Secret, and the chart mounts that key, optionally, as the sidecar's config file instead of setting S3 environment variables. clickhouse-backup reloads its config file in each API handler that reaches S3 and kubelet rewrites a mounted Secret in place, so projected and rotated credentials reach the running sidecar. The flat keys stay for the Redis strategy and for ClickHouse releases still rendered by the previous chart. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backupcontroller/credentials_projector.go | 75 +++++++++++---- .../credentials_projector_test.go | 68 +++++++++++++ .../apps/clickhouse/templates/clickhouse.yaml | 95 +++++++++---------- .../apps/clickhouse/tests/backup_test.yaml | 77 ++++++++------- 4 files changed, 211 insertions(+), 104 deletions(-) diff --git a/internal/backupcontroller/credentials_projector.go b/internal/backupcontroller/credentials_projector.go index 4e0c3b75a..eb34d22bc 100644 --- a/internal/backupcontroller/credentials_projector.go +++ b/internal/backupcontroller/credentials_projector.go @@ -6,6 +6,7 @@ import ( "errors" "fmt" "net/url" + "strconv" "strings" "time" @@ -15,6 +16,7 @@ import ( "k8s.io/apimachinery/pkg/types" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + "sigs.k8s.io/yaml" ) // CredentialsProjectionRequeue is the backoff between reconcile attempts @@ -73,9 +75,8 @@ func IsTransient(err error) bool { // BackupCredentialsConfig describes the platform-managed source Secret // produced by the cozy-backups Bucket, the per-tenant projection target // Secret name, and the bucket coordinates (endpoint used to format the -// FDB blob_credentials account-host key, region carried verbatim for -// drivers like clickhouse-backup that wire it through env, forcePathStyle -// projected as a flat key so the chart-emitted ClickHouse sidecar can read +// FDB blob_credentials account-host key, region carried verbatim, +// forcePathStyle projected so the chart-emitted ClickHouse sidecar can read // a platform value it cannot see directly). Values are wired from // environment variables in main() and the deployment chart populates them // from .Values.backupStorage. @@ -87,9 +88,9 @@ type BackupCredentialsConfig struct { Region string // ForcePathStyle is the string "true"/"false" from // backupStorage.forcePathStyle, projected into the credentials Secret - // under the forcePathStyle key. The app charts cannot read - // backupStorage.* directly, so the ClickHouse sidecar consumes it via - // an optional secretKeyRef as S3_FORCE_PATH_STYLE. + // under the forcePathStyle key and into clickhouse-backup.yml. The app + // charts cannot read backupStorage.* directly, so the ClickHouse + // sidecar takes it from there. ForcePathStyle string } @@ -107,10 +108,11 @@ func (c BackupCredentialsConfig) IsEnabled() bool { // the data so every default Strategy CR can consume the same Secret: // // - AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY — CNPG, MariaDB, Etcd. -// - accessKey / secretKey + bucketName / endpoint / region — ClickHouse -// sidecar. +// - accessKey / secretKey + bucketName / endpoint / region — Redis +// strategy, and ClickHouse sidecars rendered before clickhouse-backup.yml. // - cloud — Velero credentials file ([default] section). // - blob_credentials.json — FoundationDB backup_agent. +// - clickhouse-backup.yml — ClickHouse sidecar, mounted as its config file. // // The source Secret is read in two formats: // @@ -156,8 +158,8 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr if creds.endpoint == "" { // Scheme-strip the env-var fallback to match the source-Secret path // (parseSourceSecret strips there). The projected `endpoint` key is - // consumed verbatim as S3_ENDPOINT by the ClickHouse sidecar, which - // expects a bare host:port — see https://cozystack.io/docs/v1/operations/services/backup-classes/. + // a bare host:port and each consumer adds its own scheme — see + // https://cozystack.io/docs/v1/operations/services/backup-classes/. creds.endpoint = stripScheme(cfg.Endpoint) } if creds.endpoint == "" { @@ -177,6 +179,13 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr } cloud := buildVeleroCredentialsFile(creds.accessKey, creds.secretKey) + clickHouseConfig, err := buildClickHouseBackupConfig(creds, cfg.ForcePathStyle) + if err != nil { + return &ProjectionError{ + Reason: ReasonSourceMalformed, + Message: fmt.Sprintf("build %s: %v", clickHouseBackupConfigKey, err), + } + } blob, err := buildFDBBlobCredentials(creds.endpoint, creds.accessKey, creds.secretKey) if err != nil { return &ProjectionError{ @@ -233,24 +242,25 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr } target.Data["AWS_ACCESS_KEY_ID"] = []byte(creds.accessKey) target.Data["AWS_SECRET_ACCESS_KEY"] = []byte(creds.secretKey) - // Mirror under the legacy bucket-controller key names so - // chart-emitted sidecars (clickhouse-backup) can consume the - // same Secret without a separate field map. + // Mirror under the legacy bucket-controller key names so the Redis + // strategy and older ClickHouse charts can consume the same Secret + // without a separate field map. target.Data["accessKey"] = []byte(creds.accessKey) target.Data["secretKey"] = []byte(creds.secretKey) target.Data["cloud"] = []byte(cloud) target.Data["blob_credentials.json"] = blob + target.Data[clickHouseBackupConfigKey] = clickHouseConfig // Always overwrite endpoint/bucket/region (delete if empty in // source) so a re-projection cannot leave stale values lingering - // after the bucket-controller rotates the source. Consumers - // (ClickHouse sidecar via secretKeyRef) must see exactly what the - // source carries, not a half-stale Secret. + // after the bucket-controller rotates the source. Consumers must + // see exactly what the source carries, not a half-stale Secret. setOrDelete(target.Data, "endpoint", creds.endpoint) setOrDelete(target.Data, "bucketName", creds.bucket) setOrDelete(target.Data, "region", creds.region) // forcePathStyle comes from the platform config (backupStorage. - // forcePathStyle), not the source Secret: the ClickHouse sidecar - // reads it as S3_FORCE_PATH_STYLE because the app chart cannot see + // forcePathStyle), not the source Secret: ClickHouse sidecars + // rendered before clickhouse-backup.yml read it as + // S3_FORCE_PATH_STYLE because the app chart cannot see // backupStorage.* directly. setOrDelete(target.Data, "forcePathStyle", cfg.ForcePathStyle) return nil @@ -414,6 +424,35 @@ func buildFDBBlobCredentials(endpoint, accessKey, secretKey string) ([]byte, err return json.Marshal(doc) } +// clickHouseBackupConfigKey holds the s3 section of a clickhouse-backup +// config. The ClickHouse chart mounts it as the sidecar's config file +// instead of passing the values as environment: clickhouse-backup reloads +// the file on every API call and kubelet rewrites a mounted Secret in +// place, so credentials projected after the pod started, or rotated since, +// reach a running sidecar. +const clickHouseBackupConfigKey = "clickhouse-backup.yml" + +// buildClickHouseBackupConfig renders that file. The projected endpoint is +// a bare host, and clickhouse-backup's S3 client rejects one without a +// scheme. +func buildClickHouseBackupConfig(creds parsedCreds, forcePathStyle string) ([]byte, error) { + s3 := map[string]any{ + "bucket": creds.bucket, + "endpoint": "https://" + creds.endpoint, + "region": creds.region, + "access_key": creds.accessKey, + "secret_key": creds.secretKey, + } + if forcePathStyle != "" { + pathStyle, err := strconv.ParseBool(forcePathStyle) + if err != nil { + return nil, fmt.Errorf("forcePathStyle %q is not a boolean", forcePathStyle) + } + s3["force_path_style"] = pathStyle + } + return yaml.Marshal(map[string]any{"s3": s3}) +} + // stripScheme returns the endpoint stripped of any leading scheme prefix // (http:// or https://) and preserves the path portion if present. The // resulting form is what `host[:port][/path]` callers (MariaDB strategy, diff --git a/internal/backupcontroller/credentials_projector_test.go b/internal/backupcontroller/credentials_projector_test.go index 2b4f0bb1c..c4b154dfa 100644 --- a/internal/backupcontroller/credentials_projector_test.go +++ b/internal/backupcontroller/credentials_projector_test.go @@ -13,6 +13,7 @@ import ( "k8s.io/apimachinery/pkg/types" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/yaml" ) func newFakeClient(objs ...client.Object) client.Client { @@ -664,3 +665,70 @@ func TestProject_HappyPath_HTTPEndpoint(t *testing.T) { t.Errorf("blob account host not stripped: %+v", blob.Accounts) } } + +// The ClickHouse sidecar reads its S3 settings from this key as a mounted +// file, which kubelet refreshes in place: clickhouse-backup reloads its +// config file on every API call, while its environment is fixed at start. +func TestProject_WritesClickHouseBackupConfig(t *testing.T) { + cases := []struct { + name string + forcePathStyle string + wantPathStyle *bool + }{ + {name: "path style on", forcePathStyle: "true", wantPathStyle: ptrBool(true)}, + {name: "path style off", forcePathStyle: "false", wantPathStyle: ptrBool(false)}, + {name: "path style unset", forcePathStyle: ""}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + c := newFakeClient(flatSourceSecret()) + ctx := context.Background() + cfg := defaultCfg() + cfg.ForcePathStyle = tc.forcePathStyle + if err := ProjectBackupCredentials(ctx, c, cfg, "tenant-acme"); err != nil { + t.Fatalf("unexpected error: %v", err) + } + got := &corev1.Secret{} + if err := c.Get(ctx, types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}, got); err != nil { + t.Fatalf("projected Secret not found: %v", err) + } + var doc struct { + S3 map[string]any `json:"s3"` + } + if err := yaml.Unmarshal(got.Data[clickHouseBackupConfigKey], &doc); err != nil { + t.Fatalf("%s is not YAML: %v\n%s", clickHouseBackupConfigKey, err, got.Data[clickHouseBackupConfigKey]) + } + for key, want := range map[string]string{ + "bucket": "cozy-backups", + "endpoint": "https://seaweedfs-s3.tenant-root.svc:8333", + "region": "us-east-1", + "access_key": "AK", + "secret_key": "SK", + } { + if doc.S3[key] != want { + t.Errorf("s3.%s = %v, want %q", key, doc.S3[key], want) + } + } + pathStyle, set := doc.S3["force_path_style"] + switch { + case tc.wantPathStyle == nil && set: + t.Errorf("s3.force_path_style = %v, want it left to the clickhouse-backup default", pathStyle) + case tc.wantPathStyle != nil && pathStyle != *tc.wantPathStyle: + t.Errorf("s3.force_path_style = %v, want %v", pathStyle, *tc.wantPathStyle) + } + }) + } +} + +func TestProject_RejectsMalformedForcePathStyle(t *testing.T) { + c := newFakeClient(flatSourceSecret()) + cfg := defaultCfg() + cfg.ForcePathStyle = "yes please" + err := ProjectBackupCredentials(context.Background(), c, cfg, "tenant-acme") + var perr *ProjectionError + if !errors.As(err, &perr) || perr.Reason != ReasonSourceMalformed { + t.Fatalf("err = %v, want a %s ProjectionError", err, ReasonSourceMalformed) + } +} + +func ptrBool(b bool) *bool { return &b } diff --git a/packages/apps/clickhouse/templates/clickhouse.yaml b/packages/apps/clickhouse/templates/clickhouse.yaml index 827b5d579..d8791878e 100644 --- a/packages/apps/clickhouse/templates/clickhouse.yaml +++ b/packages/apps/clickhouse/templates/clickhouse.yaml @@ -225,15 +225,18 @@ spec: - name: REMOTE_STORAGE value: s3 {{- /* + On the useSystemBucket flow every S3 setting, the + platform's backupStorage.forcePathStyle included, comes + from the clickhouse-backup.yml key the platform projects + into cozy-backups-creds, mounted below as the config file. + clickhouse-backup reloads that file on every API call and + kubelet rewrites a mounted Secret in place, so credentials + projected after the Pod started, or rotated since, reach + the running sidecar; environment would be read only at + start and would override the file, so none is set here. + S3_FORCE_PATH_STYLE selects path-style vs vhost-style S3 - addressing: SeaweedFS (the platform default) needs - path-style, while AWS public S3 needs vhost-style in many - regions. On the useSystemBucket flow the value is the - platform's backupStorage.forcePathStyle, which this app - chart cannot read directly, so it is projected into - cozy-backups-creds and consumed via an optional - secretKeyRef (optional: true so the Pod starts before the - first projection). On the legacy paths (own Secret via + addressing. On the legacy paths (own Secret via s3CredentialsSecret.name, or a configured endpoint) we cannot peek inside the Secret at render time, so we keep the historical path-style default — AWS users in that mode @@ -241,12 +244,8 @@ spec: clickhouse-backup config. */}} {{- if .Values.backup.useSystemBucket }} - - name: S3_FORCE_PATH_STYLE - valueFrom: - secretKeyRef: - name: {{ $sidecarSecretName | quote }} - key: forcePathStyle - optional: true + - name: CLICKHOUSE_BACKUP_CONFIG + value: /etc/clickhouse-backup-s3/config.yml {{- else if or (not (eq (default "" .Values.backup.s3CredentialsSecret.name) "")) .Values.backup.endpoint }} - name: S3_FORCE_PATH_STYLE value: "true" @@ -282,44 +281,18 @@ spec: */}} - name: S3_PATH value: {{ $s3Path | quote }} - {{- /* - optional: true in the useSystemBucket path so the Pod - starts even before the platform projector materialises - cozy-backups-creds in this namespace (projection runs at - first BackupJob time; the sidecar surfaces backup errors - only when its HTTP API is exercised). Without this gate - every fresh ClickHouse release with useSystemBucket=true - would hang in CreateContainerConfigError until a - BackupJob fires. Legacy path keeps the strict reference - shape — its Secret is chart-emitted and present at - install time. - */}} - {{- $secretOptional := .Values.backup.useSystemBucket }} + {{- if not .Values.backup.useSystemBucket }} - name: S3_BUCKET - valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $bucketKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } } - {{- /* - clickhouse-backup's aws-sdk-go-v2 client rejects a scheme-less - S3_ENDPOINT ("was not a valid URI"). On the useSystemBucket flow - the projected cozy-backups-creds.endpoint is a bare host, so - prepend https:// (the platform's CNPG endpoint helper does the - same). Legacy / BYO-Secret endpoints are tenant-supplied with - their own scheme and are consumed verbatim. - */}} - {{- if .Values.backup.useSystemBucket }} - - name: S3_ENDPOINT_HOST - valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $endpointKey | quote }}, optional: true } } - - name: S3_ENDPOINT - value: "https://$(S3_ENDPOINT_HOST)" - {{- else }} + valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $bucketKey | quote }} } } - name: S3_ENDPOINT valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $endpointKey | quote }} } } - {{- end }} - name: S3_REGION - valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $regionKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } } + valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $regionKey | quote }} } } - name: S3_ACCESS_KEY - valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $accessKeyKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } } + valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $accessKeyKey | quote }} } } - name: S3_SECRET_KEY - valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $secretKeyKey | quote }}{{ if $secretOptional }}, optional: true{{ end }} } } + valueFrom: { secretKeyRef: { name: {{ $sidecarSecretName | quote }}, key: {{ $secretKeyKey | quote }} } } + {{- end }} {{- if $endpointCAName }} # Trust the mounted private-CA bundle for the S3 endpoint (see # the `backup.endpointCA` field in values.yaml). SSL_CERT_DIR, @@ -343,6 +316,11 @@ spec: - name: backup-s3-ca mountPath: /etc/ssl/cozy-s3-ca readOnly: true +{{- end }} +{{- if .Values.backup.useSystemBucket }} + - name: backup-s3-config + mountPath: /etc/clickhouse-backup-s3 + readOnly: true {{- end }} securityContext: # clickhouse-server stores data under /var/lib/clickhouse with @@ -366,13 +344,14 @@ spec: runAsGroup: 0 seccompProfile: type: RuntimeDefault -{{- if $endpointCAName }} - # Private-CA bundle for the backup sidecar, projected as ca.crt for - # SSL_CERT_DIR above. Not optional on purpose: a missing Secret holds - # the whole ClickHouse Pod in ContainerCreating (naming the Secret in - # `kubectl describe`) rather than mounting an empty trust dir and - # failing later inside clickhouse-backup with an opaque x509 error. +{{- if or $endpointCAName .Values.backup.useSystemBucket }} volumes: +{{- if $endpointCAName }} + # Private-CA bundle for the backup sidecar, projected as ca.crt for + # SSL_CERT_DIR above. Not optional on purpose: a missing Secret holds + # the whole ClickHouse Pod in ContainerCreating (naming the Secret in + # `kubectl describe`) rather than mounting an empty trust dir and + # failing later inside clickhouse-backup with an opaque x509 error. - name: backup-s3-ca secret: secretName: {{ $endpointCAName | quote }} @@ -380,6 +359,18 @@ spec: - key: {{ $endpointCAKey | quote }} path: ca.crt {{- end }} +{{- if .Values.backup.useSystemBucket }} + # Optional: the platform projects cozy-backups-creds into this + # namespace at the first BackupJob, after the Pod has started. + - name: backup-s3-config + secret: + secretName: {{ $sidecarSecretName | quote }} + optional: true + items: + - key: clickhouse-backup.yml + path: config.yml +{{- end }} +{{- end }} {{- end }} serviceTemplates: - name: svc-template diff --git a/packages/apps/clickhouse/tests/backup_test.yaml b/packages/apps/clickhouse/tests/backup_test.yaml index 66eda4dc8..7448e02a3 100644 --- a/packages/apps/clickhouse/tests/backup_test.yaml +++ b/packages/apps/clickhouse/tests/backup_test.yaml @@ -2,10 +2,12 @@ suite: backup sidecar S3 wiring (useSystemBucket vs legacy, path-override guard) # Locks in the clickhouse-backup sidecar S3 wiring this PR introduced on the # ClickHouseInstallation PodTemplate: -# - useSystemBucket=true points every S3_* env at the platform-projected -# cozy-backups-creds Secret with optional:true (the Pod starts before the -# first projection) and scopes S3_PATH to /; the -# chart-managed -backup-s3 / -backup Secrets are skipped; +# - useSystemBucket=true mounts the clickhouse-backup.yml key of the +# platform-projected cozy-backups-creds Secret as the sidecar's config +# file with optional:true (the Pod starts before the first projection), +# sets no other S3 setting in the environment, and scopes S3_PATH to +# /; the chart-managed -backup-s3 / +# -backup Secrets are skipped; # - the legacy flow keeps the chart-emitted -backup-s3 Secret and a # strict (non-optional) secretKeyRef; # - s3PathOverride + useSystemBucket fails fast at render time. @@ -25,7 +27,7 @@ set: _cluster: {} tests: - - it: "useSystemBucket=true: sidecar reads cozy-backups-creds (optional) and scopes S3_PATH" + - it: "useSystemBucket=true: sidecar reads its S3 settings from the projected config file and scopes S3_PATH" template: templates/clickhouse.yaml documentSelector: path: kind @@ -45,28 +47,29 @@ tests: - equal: path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_PATH")].value value: tenant-test/ch-test + # clickhouse-backup reloads this file on every API call and kubelet + # rewrites the mounted Secret in place, so credentials projected after + # the Pod started, or rotated since, reach the running sidecar. - equal: - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")].valueFrom.secretKeyRef.name - value: cozy-backups-creds + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="CLICKHOUSE_BACKUP_CONFIG")].value + value: /etc/clickhouse-backup-s3/config.yml - equal: - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")].valueFrom.secretKeyRef.optional + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].volumeMounts[?(@.name=="backup-s3-config")].mountPath + value: /etc/clickhouse-backup-s3 + - equal: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].volumeMounts[?(@.name=="backup-s3-config")].readOnly value: true - equal: - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_SECRET_KEY")].valueFrom.secretKeyRef.name - value: cozy-backups-creds - # The projected endpoint is a bare host; clickhouse-backup needs a URI, so - # the chart prepends https:// via env-var expansion on this flow. - - equal: - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT_HOST")].valueFrom.secretKeyRef.name - value: cozy-backups-creds - - equal: - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT_HOST")].valueFrom.secretKeyRef.key - value: endpoint - - equal: - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT")].value - value: https://$(S3_ENDPOINT_HOST) + path: spec.templates.podTemplates[0].spec.volumes[?(@.name=="backup-s3-config")].secret + value: + secretName: cozy-backups-creds + # The Pod starts before the first projection. + optional: true + items: + - key: clickhouse-backup.yml + path: config.yml - - it: "useSystemBucket=true: S3_FORCE_PATH_STYLE is sourced from cozy-backups-creds, not hardcoded" + - it: "useSystemBucket=true: no S3 setting is fixed in the sidecar environment" template: templates/clickhouse.yaml documentSelector: path: kind @@ -76,18 +79,21 @@ tests: enabled: true useSystemBucket: true asserts: - - equal: - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].valueFrom.secretKeyRef.name - value: cozy-backups-creds - - equal: - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].valueFrom.secretKeyRef.key - value: forcePathStyle - - equal: - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].valueFrom.secretKeyRef.optional - value: true + # Environment overrides the config file and is read only at start. - notExists: - # must no longer be the hardcoded literal value: "true" - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")].value + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")] + - notExists: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT")] + - notExists: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ENDPOINT_HOST")] + - notExists: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_REGION")] + - notExists: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_ACCESS_KEY")] + - notExists: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_SECRET_KEY")] + - notExists: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_FORCE_PATH_STYLE")] - it: "useSystemBucket=true: chart-managed backup Secrets are skipped" set: @@ -240,7 +246,10 @@ tests: path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="SSL_CERT_DIR")].value value: /etc/ssl/cozy-s3-ca:/etc/ssl/certs - equal: - path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_BUCKET")].valueFrom.secretKeyRef.name + path: spec.templates.podTemplates[0].spec.volumes[?(@.name=="backup-s3-ca")].secret.secretName + value: my-s3-ca + - equal: + path: spec.templates.podTemplates[0].spec.volumes[?(@.name=="backup-s3-config")].secret.secretName value: cozy-backups-creds # backup.enabled=false renders no sidecar at all, so the CA must not leak a From f5f50d2a77179c823ad6d6ab1f1e61d9c49a359c Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 17:15:10 +0300 Subject: [PATCH 102/518] fix(clickhouse): hold a strategy job until projected credentials reach the sidecar The clickhouse-backup sidecar now reads the projected credentials from a mounted Secret, which kubelet rewrites only after its sync period, about 75 seconds after the change in practice. The first BackupJob projects the Secret and started the strategy Job a second later, so its attempts raced the volume update and the backup could still fail on a sidecar without credentials. The projector records when the projected content last changed, and the ClickHouse driver holds back creating a backup or restore Job until two minutes have passed since, with a WaitingForCredentials condition. An unchanged re-projection keeps the time, so only the first projection and a rotation cause a wait, and a Job already running is never held. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../altinitystrategy_controller.go | 74 +++++++ .../altinitystrategy_controller_test.go | 186 ++++++++++++++++++ .../backupcontroller/credentials_projector.go | 15 ++ .../credentials_projector_test.go | 56 ++++++ 4 files changed, 331 insertions(+) diff --git a/internal/backupcontroller/altinitystrategy_controller.go b/internal/backupcontroller/altinitystrategy_controller.go index b96d571f0..a99801ac6 100644 --- a/internal/backupcontroller/altinitystrategy_controller.go +++ b/internal/backupcontroller/altinitystrategy_controller.go @@ -71,6 +71,56 @@ func validateAltinityApplicationRef(ref corev1.TypedLocalObjectReference) error return nil } +// altinityCredentialsPropagation bounds how long kubelet takes to rewrite +// the mounted cozy-backups-creds in the ClickHouse Pod after it changes: +// its sync period (a minute by default) plus the Secret cache. About 75s was +// measured; the margin covers the rest. +const altinityCredentialsPropagation = 2 * time.Minute + +// credentialsPropagationWait returns how long to hold back creating the +// strategy Job named jobName in namespace, so that the clickhouse-backup +// sidecar, which reads the projected credentials from a mounted file, sees +// their latest content. A Job that already exists, or a namespace without +// projected credentials, does not wait. +func credentialsPropagationWait(ctx context.Context, c client.Client, secretName, namespace, jobName string) (time.Duration, error) { + if secretName == "" { + return 0, nil + } + if err := c.Get(ctx, types.NamespacedName{Namespace: namespace, Name: jobName}, &batchv1.Job{}); err == nil { + return 0, nil + } else if !apierrors.IsNotFound(err) { + return 0, err + } + secret := &corev1.Secret{} + if err := c.Get(ctx, types.NamespacedName{Namespace: namespace, Name: secretName}, secret); err != nil { + if apierrors.IsNotFound(err) { + return 0, nil + } + return 0, err + } + raw := secret.Annotations[credentialsUpdatedAtAnnotation] + if raw == "" { + return 0, nil + } + updatedAt, err := time.Parse(time.RFC3339, raw) + if err != nil { + return 0, fmt.Errorf("parse %s on Secret %s/%s: %w", credentialsUpdatedAtAnnotation, namespace, secretName, err) + } + return time.Until(updatedAt.Add(altinityCredentialsPropagation)), nil +} + +// credentialsWaitCondition is the Ready condition of a job held back by +// credentialsPropagationWait. +func credentialsWaitCondition(secretName string, wait time.Duration) metav1.Condition { + return metav1.Condition{ + Type: "Ready", + Status: metav1.ConditionFalse, + Reason: "WaitingForCredentials", + Message: fmt.Sprintf("waiting %s for the updated Secret %s to reach the ClickHouse backup sidecar", + wait.Round(time.Second), secretName), + } +} + // jobNameForBackupJob returns the deterministic batch/v1.Job name used for // the backup pod that drives a BackupJob. Idempotency relies on this being // pure: a retried reconcile must hit the same Job name. @@ -222,6 +272,18 @@ func (r *BackupJobReconciler) reconcileAltinity(ctx context.Context, j *backupsv return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to template Altinity strategy: %v", err)) } + wait, err := credentialsPropagationWait(ctx, r.Client, r.CredentialsConfig.TargetSecretName, j.Namespace, jobNameForBackupJob(j)) + if err != nil { + return ctrl.Result{}, err + } + if wait > 0 { + apimeta.SetStatusCondition(&j.Status.Conditions, credentialsWaitCondition(r.CredentialsConfig.TargetSecretName, wait)) + if err := r.Status().Update(ctx, j); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: wait}, nil + } + batchJob, err := r.ensureAltinityJob(ctx, j, j.Namespace, jobNameForBackupJob(j), altinityModeBackup, map[string]string{ @@ -623,6 +685,18 @@ func (r *RestoreJobReconciler) reconcileAltinityRestore(ctx context.Context, res return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("failed to template Altinity strategy: %v", err)) } + wait, err := credentialsPropagationWait(ctx, r.Client, r.CredentialsConfig.TargetSecretName, targetNamespace, jobNameForRestoreJob(restoreJob)) + if err != nil { + return ctrl.Result{}, err + } + if wait > 0 { + apimeta.SetStatusCondition(&restoreJob.Status.Conditions, credentialsWaitCondition(r.CredentialsConfig.TargetSecretName, wait)) + if err := r.Status().Update(ctx, restoreJob); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: wait}, nil + } + batchJob, err := r.ensureAltinityRestoreJob(ctx, restoreJob, targetNamespace, jobNameForRestoreJob(restoreJob), altinityModeRestore, map[string]string{ diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index 68f18fa28..75564538b 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -3,7 +3,9 @@ package backupcontroller import ( "context" + "errors" "testing" + "time" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" @@ -17,6 +19,7 @@ import ( "k8s.io/client-go/tools/record" "sigs.k8s.io/controller-runtime/pkg/client" clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" @@ -710,3 +713,186 @@ func TestBackupParameters_RoundTrip(t *testing.T) { t.Errorf("non-parameter key leaked through: %v", got) } } + +func projectedCredentials(namespace string, updatedAt time.Time) *corev1.Secret { + return &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: "cozy-backups-creds", + Annotations: map[string]string{credentialsUpdatedAtAnnotation: updatedAt.UTC().Format(time.RFC3339)}, + }, + } +} + +// The ClickHouse sidecar reads the projected credentials from a mounted +// file that kubelet rewrites only some time after the Secret changes, so a +// strategy Job started right after a projection would reach a sidecar with +// no or stale credentials. +func TestReconcileAltinity_WaitsForProjectedCredentialsToReachTheSidecar(t *testing.T) { + cases := []struct { + name string + secret *corev1.Secret + jobExists bool + wantWait bool + }{ + {name: "credentials changed just now", secret: projectedCredentials("tenant-test", time.Now()), wantWait: true}, + {name: "credentials changed long ago", secret: projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))}, + {name: "no projected credentials", secret: nil}, + {name: "the Job already runs", secret: projectedCredentials("tenant-test", time.Now()), jobExists: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + now := metav1.Now() + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + BackupClassName: "clickhouse-backup", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now}, + } + objs := []client.Object{backupJob, newAltinityStrategy("clickhouse-strategy")} + if tc.secret != nil { + objs = append(objs, tc.secret) + } + if tc.jobExists { + objs = append(objs, &batchv1.Job{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}}) + } + r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), clientfake.NewClientBuilder().WithObjects(objs...)) + r.CredentialsConfig.TargetSecretName = "cozy-backups-creds" + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + } + ctx := context.Background() + j := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + res, err := r.reconcileAltinity(ctx, j, resolved) + if err != nil { + t.Fatalf("reconcileAltinity: %v", err) + } + jobs := &batchv1.JobList{} + if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil { + t.Fatalf("list Jobs: %v", err) + } + ready := meta.FindStatusCondition(j.Status.Conditions, "Ready") + if tc.wantWait { + if len(jobs.Items) != 0 { + t.Errorf("a strategy Job was created %d time(s) before the credentials could reach the sidecar", len(jobs.Items)) + } + if res.RequeueAfter <= 0 || res.RequeueAfter > altinityCredentialsPropagation { + t.Errorf("RequeueAfter = %v, want within (0, %v]", res.RequeueAfter, altinityCredentialsPropagation) + } + if ready == nil || ready.Reason != "WaitingForCredentials" { + t.Errorf("Ready = %+v, want WaitingForCredentials", ready) + } + return + } + if len(jobs.Items) != 1 { + t.Errorf("got %d strategy Jobs, want 1", len(jobs.Items)) + } + if ready != nil && ready.Reason == "WaitingForCredentials" { + t.Errorf("Ready = %+v, the job should not wait", ready) + } + }) + } +} + +func TestReconcileAltinityRestore_WaitsForProjectedCredentialsToReachTheSidecar(t *testing.T) { + now := metav1.Now() + backup := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Name: "ch-backup", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + TakenAt: now, + }, + } + restoreJob := &backupsv1alpha1.RestoreJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-rj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}}, + Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning}, + } + _, rr, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), + clientfake.NewClientBuilder().WithObjects(newAltinityStrategy("clickhouse-strategy"), backup, restoreJob, + projectedCredentials("tenant-test", time.Now()))) + rr.CredentialsConfig.TargetSecretName = "cozy-backups-creds" + ctx := context.Background() + rj := &backupsv1alpha1.RestoreJob{} + if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), rj); err != nil { + t.Fatalf("get RestoreJob: %v", err) + } + res, err := rr.reconcileAltinityRestore(ctx, rj, backup) + if err != nil { + t.Fatalf("reconcileAltinityRestore: %v", err) + } + jobs := &batchv1.JobList{} + if err := rr.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil { + t.Fatalf("list Jobs: %v", err) + } + if len(jobs.Items) != 0 { + t.Errorf("a restore Job was created before the credentials could reach the sidecar") + } + if res.RequeueAfter <= 0 { + t.Errorf("RequeueAfter = %v, want a wait", res.RequeueAfter) + } + if ready := meta.FindStatusCondition(rj.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForCredentials" { + t.Errorf("Ready = %+v, want WaitingForCredentials", ready) + } +} + +// Without credentials projection configured there is no Secret to wait +// for; client-go refuses a Get with an empty name, so it must not be asked. +func TestReconcileAltinity_NoProjectionConfiguredDoesNotWait(t *testing.T) { + now := metav1.Now() + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + BackupClassName: "clickhouse-backup", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now}, + } + builder := clientfake.NewClientBuilder(). + WithObjects(backupJob, newAltinityStrategy("clickhouse-strategy")). + WithInterceptorFuncs(interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if key.Name == "" { + return errors.New("resource name may not be empty") + } + return cl.Get(ctx, key, obj, opts...) + }, + }) + r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), builder) + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + } + ctx := context.Background() + j := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := r.reconcileAltinity(ctx, j, resolved); err != nil { + t.Fatalf("reconcileAltinity: %v", err) + } + jobs := &batchv1.JobList{} + if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil { + t.Fatalf("list Jobs: %v", err) + } + if len(jobs.Items) != 1 { + t.Fatalf("got %d strategy Jobs, want 1", len(jobs.Items)) + } +} diff --git a/internal/backupcontroller/credentials_projector.go b/internal/backupcontroller/credentials_projector.go index eb34d22bc..99d4e182e 100644 --- a/internal/backupcontroller/credentials_projector.go +++ b/internal/backupcontroller/credentials_projector.go @@ -1,10 +1,12 @@ package backupcontroller import ( + "bytes" "context" "encoding/json" "errors" "fmt" + "maps" "net/url" "strconv" "strings" @@ -240,6 +242,7 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr if target.Data == nil { target.Data = map[string][]byte{} } + before := maps.Clone(target.Data) target.Data["AWS_ACCESS_KEY_ID"] = []byte(creds.accessKey) target.Data["AWS_SECRET_ACCESS_KEY"] = []byte(creds.secretKey) // Mirror under the legacy bucket-controller key names so the Redis @@ -263,6 +266,12 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr // S3_FORCE_PATH_STYLE because the app chart cannot see // backupStorage.* directly. setOrDelete(target.Data, "forcePathStyle", cfg.ForcePathStyle) + if !maps.EqualFunc(before, target.Data, bytes.Equal) { + if target.Annotations == nil { + target.Annotations = map[string]string{} + } + target.Annotations[credentialsUpdatedAtAnnotation] = time.Now().UTC().Format(time.RFC3339) + } return nil }); err != nil { // If the mutator refused to overwrite an unowned Secret, surface @@ -424,6 +433,12 @@ func buildFDBBlobCredentials(endpoint, accessKey, secretKey string) ([]byte, err return json.Marshal(doc) } +// credentialsUpdatedAtAnnotation records when the projected content last +// changed. A consumer that mounts the Secret as a volume sees a change only +// once kubelet rewrites the volume, so it uses this to tell how fresh its +// copy can be. +const credentialsUpdatedAtAnnotation = "backups.cozystack.io/credentials-updated-at" + // clickHouseBackupConfigKey holds the s3 section of a clickhouse-backup // config. The ClickHouse chart mounts it as the sidecar's config file // instead of passing the values as environment: clickhouse-backup reloads diff --git a/internal/backupcontroller/credentials_projector_test.go b/internal/backupcontroller/credentials_projector_test.go index c4b154dfa..208ba253c 100644 --- a/internal/backupcontroller/credentials_projector_test.go +++ b/internal/backupcontroller/credentials_projector_test.go @@ -6,6 +6,7 @@ import ( "errors" "fmt" "testing" + "time" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -732,3 +733,58 @@ func TestProject_RejectsMalformedForcePathStyle(t *testing.T) { } func ptrBool(b bool) *bool { return &b } + +// Consumers that read the Secret through a mounted volume need to know when +// its content last changed; an unchanged re-projection must keep the time. +func TestProject_StampsWhenContentChanges(t *testing.T) { + src := flatSourceSecret() + c := newFakeClient(src) + ctx := context.Background() + key := types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"} + stamp := func() string { + got := &corev1.Secret{} + if err := c.Get(ctx, key, got); err != nil { + t.Fatalf("projected Secret not found: %v", err) + } + return got.Annotations[credentialsUpdatedAtAnnotation] + } + + if err := ProjectBackupCredentials(ctx, c, defaultCfg(), "tenant-acme"); err != nil { + t.Fatalf("first projection: %v", err) + } + first := stamp() + at, err := time.Parse(time.RFC3339, first) + if err != nil || time.Since(at) > time.Minute { + t.Fatalf("first projection stamp %q is not a recent RFC3339 time: %v", first, err) + } + + old := &corev1.Secret{} + if err := c.Get(ctx, key, old); err != nil { + t.Fatalf("get projected Secret: %v", err) + } + old.Annotations[credentialsUpdatedAtAnnotation] = "2000-01-01T00:00:00Z" + if err := c.Update(ctx, old); err != nil { + t.Fatalf("age the stamp: %v", err) + } + if err := ProjectBackupCredentials(ctx, c, defaultCfg(), "tenant-acme"); err != nil { + t.Fatalf("unchanged re-projection: %v", err) + } + if got := stamp(); got != "2000-01-01T00:00:00Z" { + t.Fatalf("an unchanged re-projection moved the stamp to %q", got) + } + + rotated := &corev1.Secret{} + if err := c.Get(ctx, types.NamespacedName{Namespace: src.Namespace, Name: src.Name}, rotated); err != nil { + t.Fatalf("get source Secret: %v", err) + } + rotated.Data["secretKey"] = []byte("SK2") + if err := c.Update(ctx, rotated); err != nil { + t.Fatalf("rotate source Secret: %v", err) + } + if err := ProjectBackupCredentials(ctx, c, defaultCfg(), "tenant-acme"); err != nil { + t.Fatalf("rotated re-projection: %v", err) + } + if got := stamp(); got == "2000-01-01T00:00:00Z" { + t.Fatalf("a rotation did not move the stamp") + } +} From 35a4f7c5d93eac727354631d5de5c0fc2bb3b294 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:40:11 +0000 Subject: [PATCH 103/518] docs(release): split the release document into one page per topic docs/release.md had grown into one page that mixed the release model, the CI pipeline, changelogs, signatures, nightlies and the fires seen at release time. A reader looking for one of them read all of it, and an edit to one part reflowed the rest. It now indexes one page per topic under docs/release/. The pages already describe how fixes reach a release line through the backport batch, which the following commits add. Workflows, scripts and tests that named docs/release.md now point at the page that holds the text they mean. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .github/workflows/changelog-generate.yaml | 16 +- .github/workflows/cut-prerelease.yaml | 2 +- .github/workflows/tags.yaml | 4 +- docs/agents/overview.md | 4 +- docs/release.md | 708 +--------------------- docs/release/backport-mechanics.md | 37 ++ docs/release/changelogs.md | 107 ++++ docs/release/ci-failures.md | 16 + docs/release/ci-pipeline.md | 107 ++++ docs/release/cleanup.md | 6 + docs/release/introduction.md | 11 + docs/release/nightly-builds.md | 21 + docs/release/patch-releases.md | 82 +++ docs/release/pre-release-checklist.md | 14 + docs/release/regular-releases.md | 78 +++ docs/release/release-candidates.md | 18 + docs/release/release-time-fires.md | 35 ++ docs/release/signatures-and-sbom.md | 29 + docs/release/splitting-bundle-prs.md | 10 + docs/release/stable-tags.md | 7 + docs/release/tag-immutability.md | 18 + hack/ci-tool-installs-contract.bats | 5 +- hack/release-changelog-contract.bats | 38 +- hack/sign-release-assets.bats | 4 +- hack/sign-release-assets.sh | 7 +- 25 files changed, 663 insertions(+), 721 deletions(-) create mode 100644 docs/release/backport-mechanics.md create mode 100644 docs/release/changelogs.md create mode 100644 docs/release/ci-failures.md create mode 100644 docs/release/ci-pipeline.md create mode 100644 docs/release/cleanup.md create mode 100644 docs/release/introduction.md create mode 100644 docs/release/nightly-builds.md create mode 100644 docs/release/patch-releases.md create mode 100644 docs/release/pre-release-checklist.md create mode 100644 docs/release/regular-releases.md create mode 100644 docs/release/release-candidates.md create mode 100644 docs/release/release-time-fires.md create mode 100644 docs/release/signatures-and-sbom.md create mode 100644 docs/release/splitting-bundle-prs.md create mode 100644 docs/release/stable-tags.md create mode 100644 docs/release/tag-immutability.md diff --git a/.github/workflows/changelog-generate.yaml b/.github/workflows/changelog-generate.yaml index 10452fb85..ff67b4f7e 100644 --- a/.github/workflows/changelog-generate.yaml +++ b/.github/workflows/changelog-generate.yaml @@ -154,6 +154,7 @@ jobs: sparse-checkout: | hack docs/agents + docs/release persist-credentials: false # The agent reads docs/agents/changelog.md from the WORKING TREE, which is @@ -163,18 +164,17 @@ jobs: # detached rc-tag checkout is exactly that, so a COMPLIANT agent would # silently decline to generate on every promotion of an existing rc. # Overlay the current instructions so the agent reads today's contract. - # changelog.md sends the agent to the checks in docs/release.md and to - # hack/validate-changelog.sh, so those are overlaid too: the rc tag's - # copies predate any check added since. docs/release.md reaches - # .release-tooling only because the sparse checkout above is in cone - # mode (the action's default), which carries the files of every parent - # directory of docs/agents. + # changelog.md sends the agent to the checks in docs/release/changelogs.md + # and to hack/validate-changelog.sh, so those are overlaid too: the rc + # tag's copies predate any check added since, and an rc cut before the + # release docs were split into docs/release/ has no such directory. - name: Refresh agent instructions run: | cp .release-tooling/docs/agents/changelog.md docs/agents/changelog.md - cp .release-tooling/docs/release.md docs/release.md + mkdir -p docs/release + cp .release-tooling/docs/release/changelogs.md docs/release/changelogs.md cp .release-tooling/hack/validate-changelog.sh hack/validate-changelog.sh - echo "Refreshed docs/agents/changelog.md, docs/release.md and hack/validate-changelog.sh from ${GITHUB_REF_NAME}." + echo "Refreshed docs/agents/changelog.md, docs/release/changelogs.md and hack/validate-changelog.sh from ${GITHUB_REF_NAME}." # Skip-priority + validate-then-fall-through: use a pre-existing changelog # only if it is USABLE, so an invalid higher-priority source can never diff --git a/.github/workflows/cut-prerelease.yaml b/.github/workflows/cut-prerelease.yaml index e740c2978..bfd159d94 100644 --- a/.github/workflows/cut-prerelease.yaml +++ b/.github/workflows/cut-prerelease.yaml @@ -6,7 +6,7 @@ name: Cut Pre-release Tag # This is the single entry point for creating a pre-release. Stable vX.Y.Z tags # are never cut here (or by hand) — they are created write-once by # pull-requests-release.yaml finalize at a promote PR's merge commit. See -# docs/release.md and #2677. +# docs/release/stable-tags.md and #2677. # # Dispatch from the branch the tag belongs to: `main` for a new minor's # pre-release (vX.Y.0-rc.N), or `release-X.Y` for a patch line (vX.Y.Z-rc.N). diff --git a/.github/workflows/tags.yaml b/.github/workflows/tags.yaml index 8c123fdb1..f35ca8265 100644 --- a/.github/workflows/tags.yaml +++ b/.github/workflows/tags.yaml @@ -556,8 +556,8 @@ jobs: tar -xzf "$syft_dir/syft.tar.gz" -C "$syft_dir" syft rm -f "$syft_dir/syft.tar.gz" echo "$syft_dir" >> "$GITHUB_PATH" - # cosign is the version docs/release.md asks users for, as in - # promote-rc.yaml. Bump with it. + # cosign is the version docs/release/signatures-and-sbom.md asks users + # for, as in promote-rc.yaml. Bump with it. - name: Set up cosign if: needs.build-amd64.outputs.release_exists == 'false' uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 diff --git a/docs/agents/overview.md b/docs/agents/overview.md index a4e47b23b..9ab67c41d 100644 --- a/docs/agents/overview.md +++ b/docs/agents/overview.md @@ -96,7 +96,7 @@ Build environment variables: - `LOAD=1 PUSH=0` — load images locally instead of pushing. Builds for the host architecture only, because the classic docker image store cannot load a multi-platform index. - `PLATFORM` — the buildx platforms. Defaults to `linux/amd64,linux/arm64` for anything but `LOAD=1`, so a push from either architecture publishes an index both can run. Needs a `docker-container` builder (`BUILDER`) or a docker daemon on the containerd image store, with QEMU registered for the non-native half. Compiling builder stages run natively on `$BUILDPLATFORM` and cross-compile for the target, but every `RUN` step of a stage that runs on the target platform (package installs, `setcap`, install scripts, patch steps) still runs emulated. `packages/core/testing` pins `linux/amd64`, because it carries an x86 KVM sandbox. `packages/core/talos` builds the Talos installer and release assets for the arches in `PLATFORM` with the upstream imager, which needs no QEMU for either; its matchbox image carries the amd64 and arm64 kernel and initramfs whatever `PLATFORM` says, because it network-boots machines of either arch. - `PUSHED_TAGS_LOG` — a file the `image-tags` macro appends every pushed `:` to, one per line. Unset by default; the release build sets it so `hack/stitch-multiarch.sh` can move every tag it pushed, versioned ones included, onto the multi-arch index. The append happens at recipe expansion, so `make -n` writes it too. -- `CACHE_TAG` — the tag of the mode=max build cache each image reads (and writes under `WRITE_CACHE=1`). Defaults to `buildcache`. Apart from the Talos installer and matchbox in the release job, CI builds one architecture per job, and builds arm64 images only for pre-release tags and in a nightly build of main: the images are built again on native arm64 runners under `-arm64` with `CACHE_TAG=buildcache-arm64`, which only the nightly writes. The talos and testing packages get no arm64 leg: the release job builds the Talos installer and matchbox for both arches itself, and testing is amd64 only. For a pre-release, `hack/stitch-multiarch.sh` then joins each pair into one index, `hack/verify-multiarch.sh` fails the rc on any pinned image that is not an amd64+arm64 index, and the packages artifact is republished pinned on the index digests; see `docs/release.md`. The stitch skips any image whose digest is also pinned in a file it does not rewrite, which then fails that gate; the kamaji control-plane provider's gzipped manifest is the one such copy it does rewrite (see `docs/agents/image-refs.md`). +- `CACHE_TAG` — the tag of the mode=max build cache each image reads (and writes under `WRITE_CACHE=1`). Defaults to `buildcache`. Apart from the Talos installer and matchbox in the release job, CI builds one architecture per job, and builds arm64 images only for pre-release tags and in a nightly build of main: the images are built again on native arm64 runners under `-arm64` with `CACHE_TAG=buildcache-arm64`, which only the nightly writes. The talos and testing packages get no arm64 leg: the release job builds the Talos installer and matchbox for both arches itself, and testing is amd64 only. For a pre-release, `hack/stitch-multiarch.sh` then joins each pair into one index, `hack/verify-multiarch.sh` fails the rc on any pinned image that is not an amd64+arm64 index, and the packages artifact is republished pinned on the index digests; see `docs/release/ci-pipeline.md`. The stitch skips any image whose digest is also pinned in a file it does not rewrite, which then fails that gate; the kamaji control-plane provider's gzipped manifest is the one such copy it does rewrite (see `docs/agents/image-refs.md`). ### Values schema generation @@ -150,7 +150,7 @@ Product documentation lives in [`cozystack/website`](https://github.com/cozystac | --- | --- | | `packages///README.md` | Anything scoped to a single package. Prose goes **above** `## Parameters`; that section is `cozyvalues-gen` output and is never hand-edited. The website's `make update-apps` / `update-vms` / `update-networking` / `update-k8s` / `update-services` targets append the whole README onto a published page, but only for the packages named in the `APPS`, `VMS`, `NETWORKING`, `K8S` and `SERVICES` lists of the website `Makefile`, and only from `packages/apps/` and `packages/extra/`: nothing under `packages/system/` is ever imported, and neither is a package that is not on those lists. The sync runs when a release is promoted or tagged, or by hand in the website repository, not when a pull request merges here. Use this route only for a package that is on the lists; for any other the prose reaches no reader, so put it on the website instead. | | `cozystack/website`, under `content/en/docs/` | Operator- and tenant-facing content that spans packages or is task-shaped: guides, how-tos, troubleshooting runbooks, reference tables. Which version directory a page goes into is set by the website's own [`CONTRIBUTING.md`](https://github.com/cozystack/website/blob/main/CONTRIBUTING.md); follow that file rather than a copy of its rules here. | -| `docs/` | Documentation that a chart template, migration, script or workflow references by repo-relative path, plus this repository's own process docs (`docs/agents/`, `docs/changelogs/`, `docs/release.md`). | +| `docs/` | Documentation that a chart template, migration, script or workflow references by repo-relative path, plus this repository's own process docs (`docs/agents/`, `docs/changelogs/`, `docs/release.md` and the pages under `docs/release/`). | | `api///DESIGN.md`, `pkg/apis//DESIGN.md` | Design of an API group, beside the types it describes. | | [`cozystack/community`](https://github.com/cozystack/community) | Design proposals and RFCs. Not this repository. | diff --git a/docs/release.md b/docs/release.md index c8620dbe9..175bd37fc 100644 --- a/docs/release.md +++ b/docs/release.md @@ -1,6 +1,6 @@ # Release Workflow -This document is both the process design (what the release model looks like) and the operational playbook (what to actually do, what breaks, what to verify). Companion to [`agents/changelog.md`](./agents/changelog.md), which is the canonical changelog process. +The release documentation is both the process design (what the release model looks like) and the operational playbook (what to actually do, what breaks, what to verify), split into one page per topic under [`release/`](./release/). Companion to [`agents/changelog.md`](./agents/changelog.md), which is the canonical changelog process. ## When to use this guide @@ -15,692 +15,31 @@ You are about to: If you are about to generate a changelog only, jump straight to [`agents/changelog.md`](./agents/changelog.md). -## Introduction - -Cozystack uses a staged release process to ensure stability and flexibility during development. - -There are three types of releases: - -- **Release Candidates (RC)** – Preview versions (e.g., `v1.2.0-rc.1`) used for final testing and validation. -- **Regular Releases** – Final versions (e.g., `v1.2.0`) that are feature-complete and thoroughly tested. -- **Patch Releases** – Bugfix-only updates (e.g., `v1.2.1`) made after a stable release, based on a dedicated release branch. - -All three are matched by `tags.yaml`'s regex `^v\d+\.\d+\.\d+(-(alpha|beta|rc)\.\d+)?$`. The tag push is the trigger for the whole pipeline. **Pre-release tags (`-alpha` / `-beta` / `-rc`) are cut with the [`Cut Pre-release Tag`](../.github/workflows/cut-prerelease.yaml) workflow** (manual dispatch from `main` or `release-X.Y`) — it pushes the tag as the CI app at the dispatch branch's tip, so GitHub fills `base_ref` (`tags.yaml`'s `Get base branch` step refuses tags pushed without a base) and the push triggers `tags.yaml` (a tag pushed with the default `GITHUB_TOKEN` would not). Stable `vX.Y.Z` tags are never pushed by hand at all — they are cut write-once by the promote flow at a PR merge commit. Only `-rc.N` pre-releases feed the promotion flow ([`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) accepts `vX.Y.Z-rc.N` alone); `-alpha.N` / `-beta.N` tags are built and published for download and validated on demand with the [`e2e-tag.yaml`](../.github/workflows/e2e-tag.yaml) button, but are never promoted to a stable release. - -## Release Candidates - -Release candidates are Cozystack versions that introduce new features and are published before a stable release. Their purpose is to help validate stability before finalizing a new feature release. - -Release candidates are given numbers `vX.Y.0-rc.N`, for example, `v1.2.0-rc.1`. The first one is cut from `main`, once all major features for the upcoming release have merged and the release enters its testing phase. - -**Cutting the first rc freezes the line.** [`cut-prerelease.yaml`](../.github/workflows/cut-prerelease.yaml) creates `release-X.Y` at the tagged commit, and from that point the release's content is closed: - -- Every later cut for that line — `rc.2`, `rc.3`, an `alpha`/`beta`, or a patch-line rc — must be dispatched from `release-X.Y`. A dispatch from `main` is refused. -- Fixes reach the release only by cherry-pick or backport onto `release-X.Y` (see [Backports](#backports)). -- `main` reopens immediately for the next minor, so feature work never has to wait for the release to ship. - -This is what makes an rc mean something. Previously `release-X.Y` was created only when the stable release merged, so `rc.2` was cut from `main`'s tip and silently absorbed everything that had landed since `rc.1` — the shipped release could contain code that no rc had ever validated. - -**Un-freezing is manual and deliberate.** If an `rc.1` turns out to be unusable and the line has to be re-cut from `main`, delete `release-X.Y` by hand first; the refuse step blocks every other route back to a `main` dispatch, which is the point of it. Nothing automates the deletion, and nothing should — throwing away a frozen line is a decision worth making explicitly. - -Each RC contributes to a cumulative set of release notes that will be finalized when `vX.Y.0` is released. After testing, if no critical issues remain, the last good rc is **promoted** to the regular release `vX.Y.0` — the exact rc images retagged by digest, never rebuilt and never tagged from a later commit (see [Regular Releases](#regular-releases)). Promotion cuts the write-once `vX.Y.0` tag at the promote PR's merge commit, fast-forwards the `release-X.Y` maintenance branch the freeze already created, and opens the way for patch releases. - -## Regular Releases - -A regular release `vX.Y.0` is **promoted from a release-candidate that already passed e2e** — never tagged by hand and never rebuilt. The bytes shipped as `vX.Y.0` are the exact `vX.Y.0-rc.N` images, retagged by digest. We'll use `v1.2.0` as an example. - -```mermaid -gitGraph - commit id: "feature" - commit id: "feature 2" - commit id: "feature 3" tag: "v1.2.0-rc.1" -``` - -A regular release sequence runs as follows: - -1. Cut the first release candidate (`v1.2.0-rc.1`) from the last good commit on `main`: run the [`Cut Pre-release Tag`](../.github/workflows/cut-prerelease.yaml) workflow (manual dispatch **from `main`**) with the tag. It tags `main`'s HEAD as the CI app, then **creates `release-1.2` at that commit, freezing the line**; `tags.yaml` then builds the rc images, publishes the rc prerelease, pushes the digest-vendored `release-1.2.0-rc.N` staging branch, and runs its mandatory `rc-e2e` job by calling [`e2e-tag.yaml`](../.github/workflows/e2e-tag.yaml) against that published tag. -2. Watch the rc's mandatory full E2E and wait for it to pass, then complete the other checks in [Pre-release verification checklist](#pre-release-verification-checklist). If the latest `tags.yaml` run cannot provide a green `rc-e2e`, a successful manual dispatch of [`e2e-tag.yaml`](../.github/workflows/e2e-tag.yaml) for the same tag is equivalent gate evidence. `main` is already open for the next minor, so anything landing there is **not** in this release: fixes for `1.2` are cherry-picked onto `release-1.2`, and a later rc is cut by dispatching the workflow **from `release-1.2`**. **Optional:** once the notes are worth reviewing, press the `Generate RC Changelog` button ([`changelog-rc.yaml`](../.github/workflows/changelog-rc.yaml)) to generate `docs/changelogs/v1.2.0.md` now and bank it on the `release-1.2.0-rc.N` staging branch — so there is time to review and hand-edit it before promotion. Skipping the button changes nothing; promotion generates the changelog itself. - - ```mermaid - gitGraph - commit id: "feature" - commit id: "feature 2" - commit id: "feature 3" tag: "v1.2.0-rc.1" - branch release-1.2 - checkout main - commit id: "feature 4 (next minor)" - checkout release-1.2 - commit id: "cherry-picked fix" tag: "v1.2.0-rc.2" - ``` - -3. Once the checks are green, run the [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) workflow (manual dispatch **from `main`**, the only branch it is dispatched from, whichever line the rc belongs to) with that rc tag. Its `parse` job verifies E2E before any staging: it accepts a successful `rc-e2e` job in the latest `tags.yaml` push run for the tag, or any retained successful manual `e2e-tag.yaml` run whose expanded E2E job name targets the tag. The emergency `skip_e2e_gate` input bypasses this check, logs a loud warning, and marks the promote PR with ⚠️. Before its first registry write, promotion also proves that the selected `main` or `release-X.Y` target already contains the candidate-aware PR gate, finalizer and trusted verifier; backport this pipeline to an existing maintenance branch before promoting its next patch. Promotion **creates no stable name before merge**: dispatch does write to the registry — it publishes one `promotion-*` packages candidate, and that tag is real and publicly listable — but it creates no `vX.Y.Z` tag, moves no `latest`, and publishes no stable installer, so nothing a consumer resolves changes until the PR merges. It: - 1. Rewrites the rc version substring to `1.2.0` and commits that exact tree, serializes its `packages/` directory under a run-specific temporary OCI tag, then adds a second commit pinning the candidate's immutable digest in the installer before pushing `release-1.2.0`. Container digests remain exactly the rc's; the two commits give the OCI revision an honest source SHA without pretending the artifact can contain its own digest. - 2. Drafts the `v1.2.0` release and uploads the (restamped) assets. - 3. Generates `docs/changelogs/v1.2.0.md` (in a parallel job, detached at the rc tag) — or, if you pressed the rc-time `Generate RC Changelog` button, copies the file already banked on the `release-1.2.0-rc.N` staging branch instead of regenerating — and commits it onto the `release-1.2.0` branch, so the changelog is reviewed as part of the promotion and finalize can publish the release with it already in place. Non-blocking: if generation fails the PR still opens, carrying a ⚠️ line that says the changelog is missing. - 4. Opens the `cozystack/website` "update managed apps reference" PR (`update-docs-v1.2.0` branch), generating the docs from the `release-1.2.0` staging branch via `FETCH_REF`. Non-blocking, and **do not merge that PR until `v1.2.0` is published** — merging early breaks the live site's version routing (see [Phase 3](#phase-3--update-website-docs-now-the-backstop)). - 5. Opens the `chore(release): promote v1.2.0-rc.N -> v1.2.0` PR into `release-1.2`, labelled `release` only. The PR body records whether E2E was verified or bypassed, plus ✅/⚠️ changelog and website-docs lines. E2E does not run on this promote PR by default because the immutable rc already passed the mandatory full suite; a maintainer may add `full-e2e` to run it again. The stable retag itself is deferred to the merge (step 5), so an abandoned promotion leaves no stable-named images and cannot wedge a re-promotion; retention protects the temporary candidate for as long as this release PR remains open, then may remove it after 30 days. - - The PR targets the maintenance branch, not `main`: `promote-rc.yaml` prefers `release-X.Y` whenever it exists, and since the rc freeze it always does. The digest-vendored `Prepare release` commit therefore stays on the release line instead of landing on `main`. - - ```mermaid - gitGraph - commit id: "feature 3" tag: "v1.2.0-rc.1" - branch release-1.2 - checkout release-1.2 - commit id: "cherry-picked fix" tag: "v1.2.0-rc.2" - branch release-1.2.0 - checkout release-1.2.0 - commit id: "Prepare release v1.2.0" - commit id: "Pin promoted packages artifact" - checkout release-1.2 - merge release-1.2.0 id: "Pull Request" - ``` - -4. Maintainer reviews the PR and the draft release against the [Phase 4 checklist](#phase-4--review-and-merge-the-promote-pr): confirm the E2E-verification line is ✅ (or explicitly accept its ⚠️ bypass), the changelog line is ✅, and note the website-docs line — the `cozystack/website` docs PR that promotion parked, which stays unmerged for now. E2E does not run on this promote PR by default. Then merge — **do not squash-merge**, the stable tag must attach to a real merge commit. GitHub removes the merged branch `release-1.2.0` (the repo has auto-delete-on-merge enabled). -5. CI workflow triggers on merge (this is where every consumer-visible or write-once side effect happens, after the rc E2E gate and PR review): - 1. Pulls the pinned packages candidate by digest and verifies that it matches the merged `packages/` tree apart from the artifact's impossible self-reference, carries no rc image tags, and preserves the original rc artifact's container `repo@digest` set. - 2. Creates the tag `v1.2.0` at the newly created merge commit — write-once. The tag is published here for the first time, never moved. - 3. Cuts the write-once `api/apps/v1alpha1/v1.2.0` Go-module tag at the same commit. - 4. Ensures the `release-1.2` maintenance branch exists at the tag commit. Since the rc freeze created it, this is normally a no-op fast-forward: the merge commit is already its tip. - 5. Publishes the release page (`draft` → `latest`), using `docs/changelogs/v1.2.0.md` from the merge commit as the release body. - 6. Retags the verified packages candidate and rc container images by digest to `v1.2.0` (and `:latest` only when `v1.2.0` is the newest published stable), and publishes the stable `cozy-installer` chart — no container rebuild. -6. Once the release is published, merge the parked `cozystack/website` docs PR (opened back at promote time, step 3.iv) so the live site's version routing flips to `v1.2.0` — its versioned docs and `openapi.json` are on GitHub now. The maintainer can then announce the release to the community. - -```mermaid -gitGraph - commit id: "feature 2" - commit id: "feature 3" tag: "v1.2.0-rc.1" - branch release-1.2 - checkout main - commit id: "feature 4 (next minor)" - checkout release-1.2 - commit id: "cherry-picked fix" tag: "v1.2.0-rc.2" - branch release-1.2.0 - checkout release-1.2.0 - commit id: "Prepare release v1.2.0" - checkout release-1.2 - merge release-1.2.0 id: "Release v1.2.0" tag: "v1.2.0" -``` - -Note that `main` never receives the release commit. It forked away at the freeze and carries the next minor's work. - -## Patch Releases - -A patch release now follows exactly the same shape as the regular release above — by the time `vX.Y.0` ships, everything is already happening on `release-X.Y`. The only difference is where the line starts: - -* `release-X.Y` already exists, created by the freeze at `vX.Y.0-rc.1`, so there is no branch to create. -* Patch commits are cherry-picked onto it, and the rc is cut by dispatching **from `release-X.Y`** — the same dispatch used for `rc.2` of the original release. - - -Let's assume that we've released `v1.2.0` and that development is ongoing. -We have introduced a couple of new features and some fixes to features that we have released -in `v1.2.0`. - -Once problems were found and fixed, a patch release is due. - -```mermaid -gitGraph - commit id: "feature 3" tag: "v1.2.0-rc.1" - branch release-1.2 - checkout release-1.2 - commit id: "Release v1.2.0" tag: "v1.2.0" - checkout main - commit id: "feature 4" - commit id: "patch 1" - commit id: "feature 5" - commit id: "patch 2" -``` - - -1. The maintainer cherry-picks patch commits from `main` onto the existing `release-1.2` branch. These must be only patches to features that were present in version `v1.2.0`. - - Cherry-picking can be done as soon as each patch is merged into `main`, or directly before the release. - - The `backport` label automates this: [`backport.yaml`](../.github/workflows/backport.yaml) targets the newest existing `release-X.Y` branch, which during a freeze window is the line being stabilised rather than the last published one. - - ```mermaid - gitGraph - commit id: "feature 3" tag: "v1.2.0-rc.1" - branch release-1.2 - checkout release-1.2 - commit id: "Release v1.2.0" tag: "v1.2.0" - checkout main - commit id: "feature 4" - commit id: "patch 1" - commit id: "feature 5" - commit id: "patch 2" - checkout release-1.2 - cherry-pick id: "patch 1" - cherry-pick id: "patch 2" - ``` - - When all relevant patch commits are cherry-picked, the branch is ready for release. Build with `go build ./cmd/backport-audit`, then confirm with `./backport-audit release-1.2` — it exits non-zero while anything labeled for the line is still missing from it, and prints the URLs. See [Cherry-pick triage before a patch](#cherry-pick-triage-before-a-patch). - -2. The maintainer cuts a release candidate (`v1.2.1-rc.N`) via the [`Cut Pre-release Tag`](../.github/workflows/cut-prerelease.yaml) workflow (manual dispatch **from `release-1.2`**), which tags that branch's `HEAD`. CI builds and publishes the rc, pushes the `release-1.2.1-rc.N` staging branch, and runs the mandatory full `rc-e2e` job against the published tag. -3. Watch `rc-e2e` and wait for it to pass, then complete the other release checks. A successful manual `e2e-tag.yaml` run for `v1.2.1-rc.N` is also valid evidence if the latest `tags.yaml` run does not carry a green `rc-e2e`. **Optional:** press the `Generate RC Changelog` button ([`changelog-rc.yaml`](../.github/workflows/changelog-rc.yaml)) to generate `docs/changelogs/v1.2.1.md` at rc time and bank it on the rc staging branch; promotion then reuses it instead of regenerating. -4. Once the checks are green, run [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) (manual dispatch **from `main`**, as for every promotion, not from `release-1.2`) with that rc tag. It runs the same five jobs as a regular promotion: the dispatch first gates on the latest tag-push `rc-e2e` or a successful tag-correlated manual E2E run (`skip_e2e_gate` overrides it), verifies that `release-1.2` already carries the candidate-aware pipeline, then commits and publishes the rewritten packages tree under a run-specific temporary `promotion-*` tag, adds a second commit pinning the candidate on the `release-1.2.1` branch, drafts the `v1.2.1` release with the restamped assets, generates or picks up `docs/changelogs/v1.2.1.md` and commits it onto that branch, opens the parked `cozystack/website` docs PR (updating the existing `v1.2/` directory in place, since a patch does not promote `next/`), and opens the promote PR into `release-1.2` with only the `release` label. The promote PR body carries the same ✅/⚠️ E2E, changelog, and website-docs lines and runs no E2E by default; add `full-e2e` manually to force a repeat. As with a regular release every stable name is deferred to merge; retention protects the temporary candidate while this PR is open and, after it merges, while `release-1.2` still pins it — until finalize retags it stable. - - ```mermaid - gitGraph - commit id: "feature 3" tag: "v1.2.0-rc.1" - branch release-1.2 - checkout release-1.2 - commit id: "Release v1.2.0" tag: "v1.2.0" - checkout main - commit id: "feature 4" - commit id: "patch 1" - commit id: "feature 5" - commit id: "patch 2" - checkout release-1.2 - cherry-pick id: "patch 1" - cherry-pick id: "patch 2" tag: "v1.2.1-rc.1" - branch release-1.2.1 - commit id: "Prepare release v1.2.1" - checkout release-1.2 - merge release-1.2.1 id: "Pull request" - ``` - - Finally, when release is confirmed, the release sequence goes on. - -5. Maintainer reviews the PR against the [Phase 4 checklist](#phase-4--review-and-merge-the-promote-pr), confirms the E2E-verification and changelog lines and notes the parked website-docs line, and merges it — **do not squash-merge**. GitHub removes the merged branch `release-1.2.1` (auto-delete-on-merge is enabled). -6. CI workflow triggers on merge (all consumer-visible or write-once side effects, after the rc E2E gate and PR review): - 1. Verifies the pinned packages candidate against the merged tree and the original rc artifact's container digests. - 2. Creates the tag `v1.2.1` at the newly created merge commit — write-once. The tag is published here for the first time, never moved. - 3. Cuts the write-once `api/apps/v1alpha1/v1.2.1` Go-module tag at the same commit. - 4. Publishes the release page (`draft` → `latest`; `latest` moves only if `v1.2.1` is the newest published stable). - 5. Retags the verified packages candidate and rc container images by digest to `v1.2.1` and publishes the stable `cozy-installer` chart — no container rebuild. -7. Once the release is published, merge the parked `cozystack/website` docs PR, then announce the release to the community. - -## What CI does during the release process - -The numbered process above is implemented by seven workflows, including the reusable/manual [`e2e-tag.yaml`](../.github/workflows/e2e-tag.yaml), plus the reusable [`changelog-generate.yaml`](../.github/workflows/changelog-generate.yaml) that two of them share to produce the changelog. Knowing which job does what makes the failure modes much easier to diagnose. - -1. [`cut-prerelease.yaml`](../.github/workflows/cut-prerelease.yaml) — manual dispatch (from `main` or `release-X.Y`): the sole entry point for creating a pre-release tag. Validates the `-alpha`/`-beta`/`-rc` tag, then pushes it (write-once, at the branch tip) as the CI app so `tags.yaml` fires. Cutting the first `vX.Y.0-rc.N` also creates `release-X.Y`, freezing the line; afterwards a `vX.Y.*` dispatch from `main` is refused. Stable tags are never created here. -2. [`tags.yaml`](../.github/workflows/tags.yaml) — fires on any pre-release tag push: `build-amd64` and `build-arm64` build the pre-release side by side, then `prepare-release` publishes it and pushes its digest-pinned staging branch, then — **only for `-rc.N` tags** — the mandatory `rc-e2e` job calls `e2e-tag.yaml` in the same run (`-alpha.N` / `-beta.N` skip it and are validated on demand with `e2e-tag.yaml`'s manual button). `generate-changelog` and `update-website-docs` also live here but are **backstops only** — both are normally produced at promotion time (see 4); on a stable tag push each self-skips or no-ops when the promote flow already did the work. These stable-tag backstops only fire because finalize pushes the stable tag with the CI app token (a `GITHUB_TOKEN` push triggers no run — see 5). -3. [`e2e-tag.yaml`](../.github/workflows/e2e-tag.yaml) — reusable from `tags.yaml::rc-e2e` and manually dispatchable for a published tag. It checks that the tag's release carries its nocloud disk, resolves the digest-pinned tree and runs the full install, OpenAPI, and Chainsaw suite on Talos containers. A tag cut before the container lane existed has to be dispatched from its own ref. Its dispatch run title is the constant workflow name, so the promote gate correlates a manual run through the successful tag-bearing E2E job name rather than the title. -4. [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) — manual dispatch, from `main` only (the signature over a stable release's assets names that ref, see [Release signatures and SBOM](#release-signatures-and-sbom)). Five jobs: `parse` validates the rc tag and requires green full-E2E evidence before any staging; `promote` rewrites and commits the rc tags, publishes that packages tree as a run-specific temporary reproducible candidate, commits its digest pin in the `release-X.Y.Z` tree, and drafts the stable release; `changelog` calls the reusable `changelog-generate.yaml` **in parallel** to produce `docs/changelogs/vX.Y.Z.md` (regenerating it, or copying an rc-time changelog already committed to the `release-X.Y.Z-rc.N` staging branch by `changelog-rc.yaml` — see 7); `website-docs` (needs `parse`, `promote`) opens the `cozystack/website` docs PR from the staging branch via `FETCH_REF` (see [Phase 3](#phase-3--update-website-docs-now-the-backstop)); `open-pr` joins them, commits the changelog onto the staging branch, and opens the `release-X.Y.Z` promote PR with the `release` label but no automatic `full-e2e`. So the changelog and website docs are reviewed as part of the promotion, the already-checked rc suite is not repeated by default, and the PR body carries ✅ verified or ⚠️ bypassed plus ✅/⚠️ changelog and website-docs lines. Both `changelog` and `website-docs` are non-blocking. No stable registry name is created at dispatch. -5. [`.github/workflows/pull-requests.yaml`](../.github/workflows/pull-requests.yaml) verifies a promotion candidate against the prospective merge tree and original rc artifact before the required `E2E Tests` status can turn green. [`pull-requests-release.yaml`](../.github/workflows/pull-requests-release.yaml) fires after merge and repeats that verification before any stable tag, then cuts the write-once stable + Go-module tags, publishes the release **with the merged changelog as its body**, retags the candidate and rc images to stable by digest (`:latest` gated on newest-stable), and publishes the stable chart. -6. [`update-releasenotes.yaml`](../.github/workflows/update-releasenotes.yaml) — fires on pushes to `main` that touch `docs/changelogs/v*.md`; syncs that content into the corresponding GitHub Release body. No longer the primary path (finalize sets the body directly, which is race-free and works for maintenance lines too) — this now covers later edits to a published changelog and manual re-syncs via `workflow_dispatch`. -7. [`changelog-rc.yaml`](../.github/workflows/changelog-rc.yaml) — manual dispatch, **optional**: generate the changelog at rc time instead of waiting for promotion, while there is still time to review it. It calls the same reusable [`changelog-generate.yaml`](../.github/workflows/changelog-generate.yaml) and commits the result to the `release-X.Y.Z-rc.N` staging branch as `docs/changelogs/vX.Y.Z.md`; the promote-time `changelog` job then copies that file instead of running the AI again. Skipping this workflow changes nothing — promote generates the changelog itself, exactly as before. - -### Phase 1 — `build-amd64`, `build-arm64` and `prepare-release` (hard gate) - -On an rc tag push: builds images for both architectures, commits digest pins, creates the draft release, and pushes the `release-X.Y.Z-rc.N` staging branch. `build-amd64` holds the release-exists no-op and the rejection of a hand-pushed stable tag, and builds the amd64 images and the release assets (its `make build` also pushes the packages artifact and the installer chart, pinned on amd64 digests); `prepare-release` stitches, gates, republishes those two from the repinned tree, and publishes the release. (The stable promote PR is opened later by `promote-rc.yaml`, not here. When the stable tag itself is pushed by finalize, a draft already exists, so this job is a no-op — see [Phase 5](#phase-5--pull-requests-releaseyaml-finalize).) - -The commit (`Prepare release vX.Y.Z`, authored by `cozystack-ci[bot]`) is **digest pins and image tags only**: - -- `packages/core/{installer,platform,testing}/values.yaml` -- `packages/system/*/values.yaml` (cozystack-api, dashboard, kamaji, linstor, kubevirt-csi-node, etc.) -- `packages/apps/kubernetes/images/{kubevirt-csi-driver,ubuntu-container-disk-*}.tag` -- `packages/system/dashboard/templates/configmap.yaml` (the `$tenantText` value) - -Things that look surprising in the diff but are normal: - -- **Zero-diff packages**: when buildx fully caches and upstream base images haven't moved, the digest is byte-identical to the previous release and won't appear. Verify the build step actually ran — don't assume "no diff = nothing built." -- **Sudden churn on `ubuntu-container-disk-*` tags**: `cloud-images.ubuntu.com/noble/current/` is a moving target; these often rebuild even without code changes. -- **Switched registry**: if a self-built image moved from `docker.io` to `ghcr.io`, the registry portion of the digest changes — that's a deliberate move, not a regression. -- **`targetVersion` is NOT touched by the release PR.** Platform migration `targetVersion` is bumped earlier in a feature commit (e.g. `[platform] Bump migration targetVersion to 39 for migration 38`). The release PR only re-pins the `platform-migrations` image digest. - -#### Multi-arch images - -Every pre-release is published for amd64 and arm64. `tags.yaml::build-arm64` builds the images on a native arm64 runner and pushes each as `-arm64`, while `build-amd64` builds amd64, the Talos installer and matchbox for both arches, and the assets on x86 at the same time. `build-amd64` hands its stamped tree, the log of the tags it pushed and the assets to `prepare-release` as workflow artifacts; a missing one fails the upload or the download rather than publishing a partial release. `prepare-release` waits for both builds and runs only if both succeeded: when either fails, the rc is not prepared at all, so a release is never published amd64 only. It applies the stamped tree and runs [`hack/stitch-multiarch.sh`](../hack/stitch-multiarch.sh), which joins each image with its arm64 twin into one index, moves onto it every tag the build pushed for that image (the rc tag and, under `PUBLISH_VERSIONED=1`, its component version, which the build records in `PUSHED_TAGS_LOG`), and repins the tree on the index digests, then [`hack/verify-multiarch.sh`](../hack/verify-multiarch.sh), then republishes the packages artifact and the installer chart from the repinned tree. A stable tag skips `build-arm64` and builds nothing: promotion copies the rc's indexes with every platform, so the stable release is exactly as multi-arch as its rc. Because the two builds run in parallel, an rc that is not prepared (its arm64 build failed, or `prepare-release` failed before the stitch) still leaves behind what the amd64 build pushed: amd64-only images under `:` and under the component-version tags, and the `cozystack-packages:` artifact and the installer chart, both pinned on those amd64 digests. A component tag an earlier rc had moved to an index then points at an amd64-only image again, until a run that succeeds stitches it, a re-run of the same run included. - -`verify-multiarch.sh` is a hard gate. It resolves every ref the tree pins, third-party images included, and fails naming each one that is not an index carrying both `linux/amd64` and `linux/arm64`, or that it cannot resolve at all. The fix is to make the image multi-arch (build it for both, or pin a multi-arch digest of the same version); an image that cannot exist for arm64 goes in [`hack/multiarch-allowlist`](../hack/multiarch-allowlist) by repository, with the reason on the same line. - -That gate sees only what the tree pins by digest. An image a vendored chart renders from its own defaults, or one an operator launches at runtime, is invisible to it. `rc-e2e` closes most of that gap: before tearing the sandbox down, [`e2e-tag.yaml`](../.github/workflows/e2e-tag.yaml) reads every image the three Talos nodes pulled from their image stores (`hack/collect-images.sh`), each by the digest the pull resolved to, and runs `verify-multiarch.sh --refs-file` on that list with the same allowlist. A pre-release fails on any single-arch image, on any ref it cannot read or that is still rate-limited after the retries, and on a list the collection could not produce; a stable tag's run only warns, since it ships its rc's images. A tag whose tree predates the audit (it has no `hack/verify-multiarch.sh`) only warns and passes, so an rc cut before the audit existed can still be re-validated. The list and the audit output are in the run's `image-list-` artifact. Neither check covers a package that e2e never installs and whose images the tree does not pin by digest: an arm64 break there shows up only on an arm64 cluster. - -[`build-arm64-nightly.yaml`](../.github/workflows/build-arm64-nightly.yaml) builds `main` for arm64 every day, so an arm64 build break shows up there before the next rc. Its first step prints `verify-multiarch.sh --report` for `main`'s committed tree: `main` pins amd64 images, so its first-party refs are all listed and are expected to be, since the rc's stitch turns them into indexes. A third-party ref that an in-tree rebuild stamps over at build time, as the Harbor components and shadowbox are, is listed too and goes away on the rc. The lines that matter are the third-party refs no build replaces and refs that cannot be resolved; each of those fails the next rc until it is fixed or allowlisted. matchbox is listed too and does not matter: `build-amd64` rebuilds it as a two-arch index. The stitch names the images it skips in its own summary: an image with no `-arm64` twin, one whose digest is also pinned in a file the rewrite does not reach, and a pair that is not one amd64 and one arm64 image. matchbox (already an index) and the e2e sandbox (never built for arm64, and allowlisted) are skipped on every rc and pass the gate; any other skipped image fails it. - -### RC E2E — mandatory check before promotion - -For an rc, `prepare-release` must finish before `tags.yaml::rc-e2e` calls the published-tag lane, so the full suite tests the digest-pinned staging tree that promotion will reuse, and the release it runs against already carries its disk. `promote-rc.yaml::parse` then checks the latest `tags.yaml` push run whose `head_branch` is the rc tag and accepts it only when the reusable E2E job succeeded. GitHub exposes called-workflow job names as `E2E Release Candidate / E2E vX.Y.Z-rc.N (full suite)`, which the gate matches by the exact tag-bearing suffix. - -The alternate evidence path scans retained successful `workflow_dispatch` runs of `e2e-tag.yaml` and accepts one whose direct E2E job is named `E2E vX.Y.Z-rc.N (full suite)`. The workflow's run title is always `E2E Release Tag` and therefore cannot correlate the input tag; artifact names also carry the tag but expire independently, so the job name is the strongest available correlator. If Actions retention or manual deletion removes the run/job metadata, or that job is renamed without updating the matcher, the gate fails closed and tells the maintainer to run the E2E Release Tag button again. `skip_e2e_gate=true` is an emergency override, not alternate evidence: it writes a loud warning and a ⚠️ PR-body line. - -### Phase 2 — the changelog (generated at rc time or promotion, not here) - -The changelog for `vX.Y.Z` is produced by [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml)'s `changelog` job (a thin call to the reusable [`changelog-generate.yaml`](../.github/workflows/changelog-generate.yaml)) and committed onto the `release-X.Y.Z` staging branch by `open-pr`, so it rides the promote PR and is on the base branch the moment the release publishes. Finalize then uses that file verbatim as the GitHub Release body. Generating it here — on the stable tag push — would be too late by construction: that tag is created by finalize, which has already published the release. It can also be produced earlier, at rc time, via the optional [`changelog-rc.yaml`](../.github/workflows/changelog-rc.yaml) button — which runs the same reusable core and commits the file to the `release-X.Y.Z-rc.N` staging branch, from which the promote-time `changelog` job copies it (validated) instead of regenerating. - -The `changelog` job runs **in parallel** with `promote`, detached at the rc tag (the "promotion" configuration in §2 of [`agents/changelog.md`](./agents/changelog.md)), using a separate **read-only** GitHub App token for the AI step so the model cannot mutate the repo even with `--allow-all-tools`. Every write — the commit onto the staging branch and the PR — happens in `open-pr` under a write-scoped token. - -Generation is deliberately **non-blocking**: a Copilot outage or exhausted token quota must never block a release. Only the AI step carries `continue-on-error`, and `open-pr` is gated on `promote` alone (`if: !cancelled() && needs.promote.result == 'success'`) so the promote PR opens regardless. When the changelog is missing the PR body says so explicitly — that ⚠️ line is the signal to act on. - -Recovery, in order of preference: - -1. Commit `docs/changelogs/vX.Y.Z.md` to the `release-X.Y.Z` branch by hand (follow [`agents/changelog.md`](./agents/changelog.md)). `promote` preserves it across a re-dispatch, and `open-pr` will not overwrite it with generated output. -2. Re-dispatch `promote-rc.yaml` — generation runs again; anything already on the branch wins. -3. Merge without one and let `tags.yaml::generate-changelog` (the backstop) open a changelog PR against `main` after the fact. The release is published with the draft's body until that merges. - -Known failure modes: - -- Copilot quota exhausted (`COPILOT_GITHUB_TOKEN` 402). Refill or rotate, then re-dispatch. -- AI step timeout (hard 30-min cap). -- Output truncated or whitespace-only — caught by the `Verify changelog` step (`hack/validate-changelog.sh`), which accepts either header convention (`# Cozystack vX.Y.Z` for a minor or `# vX.Y.Z ()` for a patch), and requires the leading release-link comment pointing at `releases/tag/vX.Y.Z`, a `compare/...vX.Y.Z` link ending in this version, and at least one `## ` section — deliberately with no line-count floor, since a complete short patch changelog (v1.5.1 ships in 19 lines) must not be rejected as a fragment. It downgrades to the missing-changelog path rather than shipping a fragment as release notes. - -**`tags.yaml::generate-changelog` is the backstop.** It self-skips when the changelog is already on `main`. It earns its keep in two cases: generation failed during promotion, or the promote PR targeted `release-X.Y` so the file never reached `main`. Since the rc freeze the second case is **every** release, minor and patch alike — the promote PR always targets the release line (see [Regular Releases](#regular-releases)), which makes this backstop the only route by which a published changelog reaches `main`. In that case it **ports the reviewed file from the tag commit verbatim** rather than regenerating — a second AI pass there would spend quota and then overwrite already-published, already-reviewed release notes. - -### Phase 3 — `update-website-docs` (now the backstop) - -The `cozystack/website` "update managed apps reference" PR is normally opened **earlier, at promote time**, by [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml)'s `website-docs` job — not here. That job fetches the README content from the `release-X.Y.Z` staging branch `promote` just pushed (via the website Makefile's `FETCH_REF` variable), so the docs are ready to review alongside the promote PR while the stable tag is still unborn. It refreshes the trunk version pins from the staging ref before promoting `next/` → `vX.Y/`, so a new minor's docs inherit fresh Talos/cozystack pins instead of a stale `next.yaml`. It is **non-blocking**: if it fails (or trips its `FETCH_REF`-support guard against a website checkout that predates that Makefile variable), the promote PR opens with a ⚠️ website-docs line and this tag-time job backfills the same branch. - -**Merge-timing contract:** the website PR must NOT be merged before the release is published (finalize). Merging it early flips the site's latest-version pointer to a version whose versioned docs and `openapi.json` are not on GitHub yet, so every version-pin and API link for the new version 404s. Both the website PR body and the promote PR body say so; park the website PR until the GitHub release for `vX.Y.Z` is live, then merge. - -This tag-time `update-website-docs` job stays as the **backstop**. It runs with `if: !cancelled() && needs.prepare-release.result == 'success'` (surviving a failed/skipped changelog phase, stopping on cancel), decides whether to promote `next/` → `vX.Y/`, runs `make update-all`, stages `content hugo.yaml data/versions`, and force-pushes the same `update-docs-vX.Y.Z` branch. On the normal path — where `promote-rc.yaml::website-docs` already did the work — it no-ops into "No changes to commit". At tag time the stable tag exists, so its `BRANCH=RELEASE_TAG` fetch resolves without needing `FETCH_REF`. For it to fire at all, finalize must push the stable tag with the **CI app token** (see [Phase 5](#phase-5--pull-requests-releaseyaml-finalize)) — a tag pushed with the default `GITHUB_TOKEN` triggers no workflow run. - -If anyone changes the website Makefile to write somewhere else (e.g. `static/`, `i18n/`), the `git add` list must grow in **both** jobs or the PR silently drops files. - -### Phase 4 — review and merge the promote PR - -Reviewer checklist for the `chore(release): promote vX.Y.Z-rc.N -> vX.Y.Z` PR: - -- [ ] The PR body says `✅ RC full e2e was verified`; if it says the gate was bypassed, obtain explicit maintainer acceptance or add `full-e2e` and wait for that opt-in run. -- [ ] `Verify release packages candidate` is green. It pulls the candidate and original rc artifact by digest, compares the candidate to GitHub's prospective merge tree, and feeds the required `E2E Tests` status even when full E2E is not repeated. Do not edit `packages/` on this PR after generation; if the guard reports drift, re-dispatch promotion to create and pin a new run-specific candidate. Note what a green here does *not* cover: GitHub fires no `pull_request` event when the base branch advances, so a `packages/` change merged to the base afterwards is never re-verified on the PR. Finalize repeats the check against the real merge commit and catches that case before any stable name exists — but it catches it after the promote PR has already merged, so a `packages/` change landing on the base while a promote PR is open is worth holding back. -- [ ] The PR body's **website-docs line** is present: if ✅, the `cozystack/website` docs PR is open and **parked** — do NOT merge it until `vX.Y.Z` is published (see [Phase 3](#phase-3--update-website-docs-now-the-backstop)); if ⚠️, the tag-time backstop will backfill it after finalize. -- [ ] Diff is digest pins + image tags only — nothing else. -- [ ] No accidental config drift (a value file you don't recognize). -- [ ] If any chart bumped, check the new digest pulls — `crane manifest ghcr.io/cozystack/cozystack/@` should return. -- [ ] Build artifacts on the draft release page exist and match the expected count. -- [ ] The PR includes `docs/changelogs/vX.Y.Z.md`, it has content, and the entries match the commit range (see [Changelog pre-publish verification](#changelog-pre-publish-verification)). If the PR body carries the ⚠️ "No changelog in this PR" line, resolve that before merging — finalize publishes this file as the release body, unless the PR also carries `docs/releasenotes/vX.Y.Z.md` (see [Phase 6](#phase-6--update-releasenotesyaml-sync-github-release-body)). - -### Phase 5 — `pull-requests-release.yaml` (finalize) - -Fires on merge of a PR that is merged, carries the `release` label, and is authored by `cozystack-ci[bot]` (the last guard closes the "name a branch `release-X.Y.Z`, get it labelled, merge it" hole). Head branch must match `release-X.Y.Z[-suffix]`. This is where **every consumer-visible or write-once side effect of a promotion happens** — after the rc's mandatory E2E was checked at dispatch and the PR merged, making promotion transactional at stable names. Steps: - -1. **Verify the packages candidate before creating any stable tag.** [`hack/verify-promoted-packages.sh`](../hack/verify-promoted-packages.sh) pulls the candidate by the digest committed in installer values, proves it matches the merged `packages/` tree after normalizing only the impossible self-reference, rejects surviving rc image tags, then follows the candidate's embedded pre-push `platformSourceRef` back to the original rc artifact and compares normalized container `repo@digest` sets. Any package edit after candidate publication or any changed container digest aborts finalize while all stable names are still absent. -2. **Create the tag at the merge commit** (write-once). The merge commit of `Prepare release vX.Y.Z` did not exist before the PR opened, so there is nothing to move — the tag is created here for the first time. A pre-existing tag at a different commit fails the step loudly rather than being force-moved (see [Tag immutability](#tag-immutability)). The checkout uses `persist-credentials: false` so this push authenticates as the **CI app** (via `git remote set-url`), not the default `GITHUB_TOKEN` — a `GITHUB_TOKEN` push creates no workflow run, so a persisted-credential push would silently skip `tags.yaml` entirely and its stable-tag backstops (`generate-changelog`, `update-website-docs`) would never fire. -3. **Cut the `api/apps/v1alpha1/vX.Y.Z` Go-module tag** (write-once, stable only) at the same commit, so Go consumers of `api/apps/v1alpha1` get the release. Moved here from `tags.yaml`, whose `prepare-release` body is skipped for a promoted stable (the draft already exists). -4. **Ensure the maintenance branch `release-X.Y` exists** at the tag commit. Since the rc freeze creates it, this is normally a no-op fast-forward — the merge commit is already the branch tip. Still created if missing; updated fast-forward-only — a non-fast-forward update warns and is left for a maintainer rather than being force-updated. -5. **Publish the draft release**, with `docs/releasenotes/vX.Y.Z.md`, or without it `docs/changelogs/vX.Y.Z.md`, from the merge commit as its body (absent or whitespace-only files warn and leave the draft's body in place, rather than publishing blank notes, and a body over 100000 bytes is published as a link, see [Phase 6](#phase-6--update-releasenotesyaml-sync-github-release-body)). `make_latest` is computed against published-non-prerelease tags: prereleases stay `false`; tags older than the current max stay `false` (and the current max is force-restored to `latest` if necessary, so an older patch tag cut after a newer minor won't downgrade `latest`). -6. **Retag the verified packages candidate and rc container images to stable** by digest (`hack/promote-retag.sh`, no container rebuild) and **publish the stable `cozy-installer` chart**. `:latest` (on both the images and the chart) moves only when this release's `make_latest` was `true` — the same decision as step 5, so the release's `latest` and the images' `:latest` never disagree. The chart is packaged with `platformVersion` stamped into its default values so the documented `helm --install --version X.Y.Z` path reports the stable version. - -### Phase 6 — `update-releasenotes.yaml` (sync GitHub Release body) - -Fires on pushes to `main` that touch `docs/changelogs/v*.md`. Reads each `docs/changelogs/vX.Y.Z.md` and PATCHes the matching GitHub Release's `body` if it differs. Since Phase 5 now sets the body directly from the merge commit, this is no longer on the critical path — it covers later **edits** to a published changelog, the backstop PR landing a changelog that reached `main` after the fact, and manual re-syncs via `workflow_dispatch`. It only watches `main`, which is why finalize (not this job) is what populates a release promoted onto `release-X.Y` — since the freeze, that is every release rather than only a patch. - -The release body is `docs/releasenotes/vX.Y.Z.md` when it exists and is not blank, otherwise the changelog, which stays the full record either way, and a push to `main` touching either file fires this sync. Finalize sets the body after the stable tags exist, where a body GitHub refuses would strand the release, so at publish time a body over 100000 bytes is replaced by a one-line link to `docs/changelogs/vX.Y.Z.md` on the tag, with a warning in the run's log. This sync never writes that link: it warns and leaves such a release's body as it is. The `tags.yaml` backstop ports only the changelog to `main`, so release notes have to be added to `main` as well, otherwise the sync falls back to the changelog and replaces the release notes with it whenever the changelog fits. - -## Stable tags come from rc promotion - -A stable `vX.Y.Z` is created only by **promoting an existing release-candidate** that already passed e2e — never by rebuilding and never by a cron. - -- [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) is triggered manually once an rc has gone green. It rewrites the rc version substring in the vendored image tags (the container digests stay the rc's), commits and publishes the resulting declarative packages tree under a unique run-specific `promotion-*` tag, commits that digest pin, and opens the `release-X.Y.Z` staging PR. The PR gate verifies the candidate against the prospective merge and original rc artifact; finalize repeats that check before creating the write-once stable tag at the merge commit, then retags the verified packages artifact and rc's already-built, e2e-passed container digests to the stable tag **by digest** (no container rebuild — see [`hack/promote-retag.sh`](../hack/promote-retag.sh)) and publishes the release. An abandoned promotion leaves no stable-named bytes; its temporary candidate is retention-managed. - -So a commit on a supported `release-X.Y` line ships when a maintainer promotes the next rc for that line — not automatically within 24h. Pushing a stable `vX.Y.Z` tag by hand is **not** a supported path: `tags.yaml` fails fast on a stable tag that has no pre-existing draft ("stable tags come from promote-rc.yaml"), and even so the finalize step would refuse to move a pre-existing tag. Pre-releases (`vX.Y.Z-rc.N`, `-alpha.N`, `-beta.N`) are cut with the [`Cut Pre-release Tag`](../.github/workflows/cut-prerelease.yaml) workflow, which pushes the tag as the CI app; [`tags.yaml`](../.github/workflows/tags.yaml) then fires on that push, builds it, and publishes the pre-release. Cutting via the workflow (rather than a manual `git push`) is what lets repo admins lock `v*` tag creation to the CI app — see the tag-protection note below. - -## Release signatures and SBOM - -A release cut from a tree that has [`hack/sign-release-assets.sh`](../hack/sign-release-assets.sh) carries three files beside its other assets, which [`hack/upload-assets.sh`](../hack/upload-assets.sh) has that script write before it uploads anything. Releases cut before it have none of them, and neither does a maintenance line until the script is backported to it (see below). - -- `cozystack-source-sbom.spdx.json` is an SPDX SBOM, made with syft, of the committed source tree (the Go modules, npm lockfiles, GitHub Actions references and other manifests syft reads in it) and of the `cozypkg` binaries in the `cozypkg-*.tar.gz` assets. It does not describe what runs in the cluster: the container images carry their own SBOMs, see [SBOM of the container images](#sbom-of-the-container-images), and the Talos installer image, ISO, disk, kernel and initramfs images have none. Its name says what it covers, so that `cozystack-sbom` stays free for an SBOM of the release as a whole. -- `cozystack-checksums.txt` holds the SHA-256 of every other asset and of the SBOM. -- `cozystack-checksums.txt.sigstore.json` is a Sigstore bundle with a keyless cosign signature over the checksums. Its certificate names the workflow that published the release: `tags.yaml` for a pre-release, `promote-rc.yaml` for a stable one. - -One signature covers every asset through the checksums. To check a download, verify the checksums, then the file against them, with cosign v3.1.3 or later (v2.6.5 or later on the v2 line): - -```sh -cosign verify-blob cozystack-checksums.txt \ - --bundle cozystack-checksums.txt.sigstore.json \ - --certificate-identity-regexp '^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)' \ - --certificate-oidc-issuer https://token.actions.githubusercontent.com && - sha256sum --check --ignore-missing cozystack-checksums.txt -``` - -Older cosign accepts a legacy-format bundle that carries a bare public key in place of a certificate and skips the identity and issuer checks for it ([GHSA-fx35-mq7g-6g98](https://github.com/sigstore/cosign/security/advisories/GHSA-fx35-mq7g-6g98)), so a bundle swapped in next to the download, by a mirror or a proxy, passes there. The bundles this repository writes are in the new format, but cosign picks the format from the file, and `--new-bundle-format` does not stop that on the v3 line (v3.1.2 still accepts such a bundle with it), so the cosign version is what protects the check. - -The regexp pins the ref each workflow runs from: `tags.yaml` on a `v*` tag push, and `promote-rc.yaml` dispatched from `main`, the only branch it is dispatched from. A passing check proves that the checksums were signed by one of those runs in `cozystack/cozystack`. A `promote-rc.yaml` dispatched from another branch fails this check in its own upload step, with no asset uploaded, and an edited `promote-rc.yaml` has to be merged into `main` before it can sign. Until the `v*` [tag rulesets](#repo-side-enforcement-tag-protection-rulesets) restrict tag creation to the CI app, though, anyone with write access can push a `v*` tag whose tree carries an edited `tags.yaml`, and its signature passes. - -The script runs the same `cosign verify-blob` before it returns, so a signature this check would reject fails the job before the first upload, and the release is not published with assets but without a signature. Both jobs that call `make upload_assets`, `tags.yaml::prepare-release` and `promote-rc.yaml::promote`, hold `id-token: write` for the signing identity and install syft and cosign v3.1.3, the version users are asked for above. They install syft from its release tarball, checked against a SHA-256 committed in the workflow, and not with `anchore/sbom-action/download-syft`, which runs an install script fetched at run time in a job that holds the release credentials; [`hack/ci-tool-installs-contract.bats`](../hack/ci-tool-installs-contract.bats) keeps it that way. A stable release gets its own three files rather than its rc's: promotion restamps several assets, so the checksums are recomputed and signed again. - -`promote-rc.yaml` runs the `hack/upload-assets.sh` of the rc tree, not the dispatch ref's. A stable release is therefore signed only when its rc was tagged from a tree that has `hack/sign-release-assets.sh`, and a maintenance line starts carrying signatures once that script is backported to its `release-X.Y` branch. - -### SBOM of the container images - -The container images a release pins by digest each carry an SPDX SBOM of the packages in them, which BuildKit attaches at build time as an attestation (`SBOM=1` in `tags.yaml`, scanner and flags in [`hack/common-envs.mk`](../hack/common-envs.mk)). The attestation is an entry of the image's index, so the digest the release pins names both, and promotion copies it with the image. Read it with `docker buildx imagetools inspect @ --format '{{ json .SBOM }}'`, which prints one document per platform. The attestations are not signed. A release cut before `SBOM=1` was set has none, and neither has the Talos installer image, which the imager builds and `skopeo` pushes, nor the third-party images `packages/` vendors by digest. - -## Nightly builds - -A nightly is an **installable copy of `main` on GHCR — not a rebuild and not a release**. [`build-main.yaml`](../.github/workflows/build-main.yaml) already builds every push to `main` into the CI registry (OCIR); the nightly promotes that build to the public release registry (GHCR) and proves it installs. - -[`nightly.yaml`](../.github/workflows/nightly.yaml) runs daily (gated by the `NIGHTLY_ENABLED` repo variable) as three jobs; build-disk and e2e both start once mirror has finished: - -1. **mirror** — resolve the exact commit `cozystack-packages:main` was built from, then [`hack/nightly-mirror.sh`](../hack/nightly-mirror.sh) copies every cozystack-owned component image OCIR→GHCR **by digest** (bit-for-bit, no rebuild) and re-publishes the rewritten `cozystack-packages` artifact and the `cozy-installer` chart to GHCR, tagged `0.0.0-nightly...` plus a floating `nightly`. -2. **build-disk** — assemble the Talos `nocloud` disk from the upstream siderolabs imager (the profile references only `ghcr.io/siderolabs/*`, so this needs no cozystack image and no rebuild) and publish it as a GHCR OCI artifact (`cozystack-nocloud`) so a nightly is installable on real hardware. The nightly e2e does not boot it. -3. **e2e** — stage the published GHCR closure (rewritten tree + the `cozy-installer` chart pinned to the GHCR packages artifact) and run the **full** app suite on Talos containers, which do not boot the disk. A nightly has no diff, so Test Impact Analysis does not apply. - -No GitHub release and no `api/apps/v1alpha1/*` Go-module tag are created for a nightly. [`retention.yaml`](../.github/workflows/retention.yaml) prunes old GHCR nightly versions (keeps the newest 14 per package the current nightly publishes, named from the published `cozystack-packages:nightly` tree by [`hack/retention-packages.sh`](../hack/retention-packages.sh), except packages linked to another repository, which are skipped with a warning; the floating `nightly`, release tags and untagged versions are never touched) and abandoned packages candidates. A candidate version is eligible only when all of these hold: it carries at least one tag and **every** tag on that manifest matches the strict `promotion-vX.Y.Z-from-vX.Y.Z-rc.N-run--` form, so a manifest that has been retagged stable is excluded; it was last updated more than 30 days ago; its digest is not pinned by `packages/core/installer/values.yaml` at the head of an **open** pull request that is same-repository, authored by `cozystack-ci[bot]`, on a `release-X.Y.Z` branch and carrying the `release` label; and its digest is not pinned by that same file at the tip of `main` or of any `release-X.Y` branch — which is what keeps a candidate safe between the promote PR merging and finalize retagging it. Any such PR head whose pin is not exactly one `digest=sha256:<64 hex>` stops the candidate sweep for that run and fails the job, without affecting the nightly sweep. - -Install a nightly: - -```bash -helm upgrade --install cozystack \ - oci://ghcr.io/cozystack/cozystack/cozy-installer --version 0.0.0-nightly.20260626.15992304129.1 -# ...or --version nightly for the latest -``` - -The matching Talos node image is `ghcr.io/cozystack/cozystack/cozystack-nocloud:0.0.0-nightly.20260626.15992304129.1` (pull with `oras`). - -## Backports - -### The backport bot - -[`backport.yaml`](../.github/workflows/backport.yaml) wraps [`korthout/backport-action`](https://github.com/korthout/backport-action) and fires on a merged `main`-targeted PR carrying one of: - -| Label | Target branch | -|-------|---------------| -| `kind/backport` | the newest existing `release-X.Y` branch | -| `kind/backport-previous` | the second-newest existing `release-X.Y` branch | - -Resolution is dynamic at run time, and it reads the branches themselves: the job lists the repository's branches, keeps the ones matching `release-.`, and sorts them numerically descending — so `release-1.10` ranks above `release-1.9`, which a lexicographic sort gets backwards. `backport` takes the first, `backport-previous` the second, so both name a branch that exists by construction; asking for `backport-previous` when only one line exists fails the job rather than inventing a target. Nothing is derived arithmetically — an earlier version computed the previous line as `Y-1`, which named a non-existent branch whenever a minor was skipped. - -**During a freeze window the newest line is the one being stabilised, not the last published one.** `release-X.Y` is created when the first `vX.Y.0-rc.1` is cut, before `vX.Y.0` is published, so from the freeze until the release `backport` targets the release being stabilised and `backport-previous` targets the last published stable. That is the intent: the frozen branch is the only way into the upcoming release, which is when backports matter most. The trade-off is that for the length of the window the line one step further back has no label pointing at it, so a fix that has to reach it before the new release publishes must be cherry-picked by hand. - -The bot creates a backport PR with title `[Backport release-X.Y] `. When this PR merges, the title prefix used to re-trigger the bot through `pr-labeler.yaml`, which auto-applied `backport` to any `[Backport release-X.Y]`-titled PR. Combined with the org-level `dosubot` re-applying the label, this caused recursive backports. - -The fix (PR #2584): both job `if:` blocks gate on `github.event.pull_request.base.ref == 'main'`. Backport PRs target `release-X.Y`, so they cannot satisfy this — architectural protection, regardless of which bot relabels them. - -### When the bot fails - -If the run triggered by the merge dies without opening a backport PR, look for a run of this workflow that has not finished before retrying — `gh run list --workflow backport.yaml --branch ` — and clear it, because the retry queues behind it rather than replacing it. Do not filter by status: a run whose jobs are waiting for a runner, or which is held behind the concurrency group, is not `in_progress`, and filtering it out is how you conclude nothing is running. `--branch` narrows the list but does not identify the PR, since head branch names are not unique across forks and `gh run list` has no PR filter, so confirm which PR a run belongs to before cancelling it or you cancel someone else's backport. If cancelling does not clear the run it holds the group until GitHub retires it and the retry keeps queueing; that case has not been exercised here. A run whose jobs have all finished can still sit `in_progress`, which looks the same from the PR as a run that died. Then remove whichever backport label the PR already carries and add it back: a label event is the only other trigger, and it qualifies on the label it carries, so re-applying it is what starts a fresh attempt. - -Conflicting cherry-picks produce a draft PR via `conflict_resolution: draft_commit_conflicts`. Look for the bot's comment with the merge-conflict diff. You either: - -- Resolve in the draft branch and undraft, or -- Drop the bot's branch and cherry-pick manually: - -```bash -git checkout release-X.Y -git cherry-pick -x -s -# resolve conflicts -git commit -s -git push origin release-X.Y # or push to a new branch and open a PR -``` - -To find the bot's failed comments across a batch of PRs: - -```bash -for n in $(gh pr list --search "label:kind/backport label:kind/backport-previous merged:>=2026-01-01" --json number --jq '.[].number'); do - echo "=== #$n ===" - gh pr view $n --json comments --jq '.comments[] | select(.author.login == "github-actions" or (.author.login | contains("backport"))) | .body' | head -20 -done -``` - -Before [PR #3155](https://github.com/cozystack/cozystack/pull/3155), `conflict_resolution` was passed outside `experimental`, so a conflicting cherry-pick used the `fail` fallback and left only a "Backport failed" comment on the source PR. No draft backport PR was opened. When [`backport-audit`](../cmd/backport-audit/README.md) reports `MISSING`, check those comments and the workflow logs to establish why the change did not arrive. - -### Cherry-pick triage before a patch - -A patch release includes bugfixes for code that shipped in the corresponding minor `vX.Y.0`. Use: - -```bash -# 1. Audit what was labeled for this line and has NOT landed on it. -# Exits non-zero when anything is outstanding; see cmd/backport-audit/README.md. -go build ./cmd/backport-audit -./backport-audit release-X.Y - -# 2. List commits on main since the release branch diverged that are NOT yet on release-X.Y -git merge-base origin/main origin/release-X.Y -git log ..origin/main --grep="(#" --oneline - -# 3. Open PRs that may need labeling before the cut -gh pr list --state open --base main --label kind/bug -``` - -Step 1 covers the currently labeled work: [`backport-audit`](../cmd/backport-audit/README.md) assigns each request to the release line selected at merge time, then checks whether its commits reached the branch. Incomplete delivery, open or dropped backports, missing changes and duplicate open claims fail the gate. Late-added labels can disagree with the bot's target, and branch-opening dates are estimated from rc tags; see the audit's documented limits before deciding what to backport. Steps 2 and 3 cover changes that were never labeled. - -**Include rule:** - -- `kind/bug` — especially destructive (data loss, crash-loop, OOM, eviction). -- `area/security` / CVE. -- Narrow chart fixes whose blast radius is contained (e.g. one operator's resources). -- Patch-line-policy-matching dependency bumps (e.g. cilium 1.X.Y → 1.X.(Y+1) on a `release-A.B` shipping cilium 1.X — never a minor bump, only a patch within the line the branch already ships). - -**Skip rule:** - -- `kind/feature` — even with `lgtm`. A patch release is not a delivery vehicle for features. -- CI-only changes (`area/ci`, test infrastructure). -- Docs-only changes (those land via `update-website-docs` automatically). -- Large multi-package dependency churn (Dependabot Go-deps bumps). -- PRs whose conventional-commit type is `!` (breaking) or that mix `feat` with `fix`. - -**Borderline cases:** - -- Large refactors with `lgtm` AND an obvious bug payload — prefer extracting the bug fix into a narrower commit and backporting only that. -- PRs labeled both `kind/bug` and `kind/feature` — split them. - -### Backporting code that touches monotonic counters - -Migrations are the canonical case. `packages/core/platform/images/migrations/run-migrations.sh` is a linear counter — it runs `seq $CURRENT_VERSION $((TARGET_VERSION - 1))` and stops if `CURRENT >= TARGET`. This creates a forward-upgrade trap. Walking through it with `release-1.2` as the maintenance branch: - -- `main` has `targetVersion=10` and migrations 1–9 on disk. -- The `release-1.2` branch was cut when `targetVersion=5` (migrations 1–4 on disk). -- You backport a new migration to `release-1.2` at slot 5, and naively bump `targetVersion` 5 → 6 to enable it. -- Customer running 1.2.x runs the new migration, stamps `cozystack-version=6` in-cluster. -- On upgrade to 1.3 (main's `targetVersion=10`), the runner walks `seq 6 9` and runs main's migrations 6–9 — **main's own migration at slot 5 never runs.** The backport claimed that slot on `release-1.2`; main's slot 5 holds an unrelated migration that gets silently skipped. - -Mitigations, in order of preference: - -1. **Don't bump `targetVersion` in the backport.** Just ship the migration file. Prior precedent in the repo: an earlier maintenance branch's ACME backport did exactly this — the migration shipped but ran on no clusters until a later commit raised the counter. Pair this with relaxing the chart-side `{{ fail }}` guard on the maintenance branch (chart-side coalesce) so the new value works without the migration having executed. -2. **If you must bump,** add an idempotent duplicate of the now-skipped main migration at a higher slot. Bounded cost (one extra file per affected migration) and survives the skip window. -3. **Audit cross-branch counter state before deciding.** Compare migration numbering across `main`, `release-X.Y`, and `release-X.(Y-1)`. If a backport would create a skip window on any forward-upgrade path, choose option 1. - -This rule generalizes to any monotonic-counter state: schema versions, feature-flag generations, anything that gates "have I run X yet?" with a single integer. - -## Changelogs - -### Where the canonical process lives - -[`agents/changelog.md`](./agents/changelog.md) is the source of truth. Read it end-to-end before generating; do not infer the process from past commits or memory. The CI runs it under Copilot via the reusable `changelog-generate.yaml` — called by `promote-rc.yaml::changelog` at promote time and by the optional `changelog-rc.yaml` at rc time — and, as a backstop, `tags.yaml::generate-changelog`; locally the same prompt is replayable by following the doc directly. - -### Common changelog failure modes - -These are mistakes that have shipped in real changelogs. Verify against each before merging the changelog PR. - -**1. Commits from outside the release range.** Caused by running `git log ..HEAD` from `main` while generating a patch changelog — `HEAD` on `main` contains everything that landed since the tag plus backports merged to `release-X.Y` after the tag was cut. Use `git log ..` explicitly. The doc was updated to require this after v1.3.1's changelog shipped with 14 entries for what was actually a 1-commit release. - -**2. Original + backport listed as two separate entries.** Each backport must coalesce with its original into a single entry of the form `(in #orig, backport #bp)`. To verify after generation: - -```bash -# Every entry that mentions "backport" should also reference the original PR number -grep -E 'backport #[0-9]+' docs/changelogs/v.md | grep -v '#[0-9]*, backport' -``` - -Output should be empty. - -**3. Wrong PR author.** The squash-merge commit's author is whoever clicked "Merge" — not the person who wrote the code. **Always** resolve via `gh pr view --json author --jq .author.login`, never `git log --format=%an`. This bites hardest for website-repo entries where the same merger handles most PRs. - -**4. Superseding patch, stale changelog.** Tags are write-once, so a critical fix after `vX.Y.Z` ships as the next patch `vX.Y.(Z+1)` rather than moving the tag. If a changelog PR already exists for a tag you are superseding, the changelog work transfers to the new patch. Compare `git log ..` against what the existing `changelog-vX.Y.Z` branch already documents; only add/remove the deltas. If the only new commits are CI-internal or a revert of a feature that never reached a stable tag, no changelog edit is needed. - -**5. PR numbers swapped inside prose.** The entry-format validator checks bullet entries but ignores Feature Highlights paragraphs and Upgrade Notes. Both have caused wrong PR references in shipped changelogs. Verify every `#NNNN` in prose with `gh pr view `. - -**6. Hallucinated entries.** Every entry must correspond to a commit in `git log ..`. Verify by extracting PR numbers from the file and grep-searching the commit range: - -```bash -grep -oE '#[0-9]+' docs/changelogs/v.md | sort -u | while read pr; do - n=${pr#\#} - if ! git log .. --grep="#${n}" --oneline | grep -q .; then - echo "PR $pr in changelog but not in commit range" - fi -done -``` - -**7. Bot accounts as human contributors.** `cozystack-ci[bot]`, `github-actions`, `dependabot`, `renovate`, and any `app/*` login must NOT appear in the `## Contributors` list. They legitimately appear in **per-entry attribution** (e.g. a Renovate-authored PR) — but not in the human roll-call. - -**8. Title duplicated as description.** `* **fix(foo): X**: fix(foo): X (...)` means you wrote the conventional-commit subject twice and never wrote a user-facing description. The brief and the detail must say different things — the detail explains what the change means for users. - -### Changelog pre-publish verification - -Run these checks before merging the changelog PR. Catches problems in roughly the order they occur: - -```bash -PREV=v1.2.0; NEW=v1.2.1 - -# A. Every chart bump is mentioned somewhere in the changelog -git diff --name-only $PREV..$NEW -- 'packages/*/charts/*/Chart.yaml' 'packages/*/*/charts/*/Chart.yaml' | while read f; do - old=$(git show "$PREV:$f" 2>/dev/null | yq '.version // ""') - new=$(git show "$NEW:$f" 2>/dev/null | yq '.version // ""') - if [ "$old" != "$new" ]; then - component=$(basename "$(dirname "$f")") - if ! grep -qi "$component" docs/changelogs/$NEW.md; then - echo "MISSING: $component $old -> $new ($f)" - fi - fi -done - -# B. Out-of-Chart.yaml pins (Talos, tenant K8s, vendored images) -for path in packages/core/talos/images/talos/profiles/installer.yaml packages/core/installer/values.yaml packages/apps/kubernetes/values.yaml; do - git diff $PREV..$NEW -- "$path" | head -40 -done -git diff --name-only $PREV..$NEW -- 'images/*/Dockerfile' - -# C. Every cited #NNNN exists and is a merged PR (not an issue) -grep -oE '#[0-9]+' docs/changelogs/$NEW.md | sort -u | sed 's/#//' | xargs -P 20 -I{} sh -c ' - state=$(gh api /repos/cozystack/cozystack/issues/{} --jq ".pull_request.merged_at // \"ISSUE\"" 2>/dev/null) - echo "#{} $state" -' | grep -E 'ISSUE|null|^#[0-9]+ $' - -# D. URL liveness -grep -oE 'https://[^)" ]+' docs/changelogs/$NEW.md | sort -u | xargs -P 10 -I{} sh -c ' - code=$(curl -s -o /dev/null -w "%{http_code}" {}) - if [ "$code" != "200" ]; then echo "$code {}"; fi -' - -# E. Website PR authors are PR authors, not commit authors -grep -oE 'cozystack/website#[0-9]+' docs/changelogs/$NEW.md | sort -u | sed 's|cozystack/website#||' | while read n; do - pr_author=$(gh pr view $n --repo cozystack/website --json author --jq .author.login) - echo "website#$n -> @$pr_author" -done - -# F. Working tree clean except for the changelog -git status --porcelain | grep -v "docs/changelogs/$NEW.md" -# Should output nothing. -``` - -Ship criterion: every chart bump surfaced, no issue-cited-as-PR, no 404 URLs, every website entry attributed via `gh pr view` (not commit author), `git status` shows only the changelog. - -### Cozystack-specific URL quirks - -`cozystack.io` paths are versioned: `/docs/operations/foo/` and `/docs/next/foo/` both 404. Only `/docs/vX.Y/foo/` and the unversioned canonical `/docs/foo/` serve content. AI-generated changelogs invent the unversioned form. Verify URLs with the `curl` loop above. - -### Platform-component coverage - -For a minor release (`vX.Y.0`), changelog entries should surface what an upstream chart bump actually brought in — features, breaking changes, security fixes. PR commit messages usually don't capture this; the upstream `CHANGELOG.md` or GitHub Release notes do. For each bumped `Chart.yaml`: - -1. Read the `sources:` field — typically a GitHub URL. -2. Fetch release notes strictly inside `old < x <= new` via `gh release view --repo / --json body,name`. -3. Summarize in 2–5 user-impact bullets per component (new features, breaking changes, CVEs, important deprecations). Don't reproduce upstream notes verbatim; link to them. -4. If a PR-level entry already mentions the bump, enrich that entry with bullets — don't duplicate the entry. -5. If 3+ bumps have no PR-level mention, introduce a `## Platform Components` section. - -If you can't find upstream notes for a bumped component, list it with version-only info and no bullets. Hallucinated bullets are worse than missing detail. - -## Release-time fires — what's gone wrong recently - -These are real regressions that escaped to users. The pattern in each: code passed CI, shipped in a stable release, broke customers. Read them as a working list of pre-release verification gaps. - -### v1.3.0 — cert-manager ingressClassName regression (fix #2562) - -**Bug:** A commit migrated ACME HTTP-01 to use the modern `acme.cert-manager.io/http01-ingress-ingressclassname` annotation. The matching ingress-shim code that reads that annotation was added by cert-manager upstream PR #8244 and was **never backported** to the `release-1.19` branch — but cozystack 1.3.0 shipped cert-manager v1.19.3. The annotation was silently dropped at runtime and every solver Ingress fell back to the ClusterIssuer's default class `tenant-root`. Let's Encrypt validation broke for every tenant whose `publishing.ingressName != tenant-root`. - -**Blast radius:** v1.3.0, v1.3.1, and v1.1.7 (the migration was cherry-picked to `release-1.1`). - -**Why CI missed it:** the migration commit was self-contained and `helm template` rendered cleanly. There was no E2E that creates a tenant with a non-default ingress class and watches a real Let's Encrypt cert issue. Verification had been done from the docs side ("the API has been available since cert-manager 1.12"), not against the shipped cert-manager source. - -**Would have caught it:** non-root-tenant E2E with LE-staging cert issuance, asserting the solver Ingress lands on the tenant's class. - -### v1.4.0 — kubevirt-instancetypes upgrade failure (fix #2612) - -**Bug:** `packages/system/kubevirt-instancetypes/Makefile` contained `sed -i '/persistent: true/d' templates/preferences.yaml`, which stripped the only child of `preferredTPM`, leaving `preferredTPM:` (null) in six Windows preferences. KubeVirt v1.6.x silently accepted it. The v1.6.3 → v1.8.2 operator bump (2026-04-27) hardened the CRD's OpenAPI schema and started rejecting null for an object-typed field. `helm upgrade` then failed on every cluster with `kubevirt-instancetypes` enabled. - -**Why CI missed it:** the kubevirt-operator bump and the inert `preferences.yaml` were two weeks apart in commit history. CI only tests fresh installs against the new CRDs; it doesn't run `helm upgrade` from prior versions. Schema regression on a vendored asset that hadn't been touched in 18 months. - -**Would have caught it:** `helm template | kubectl apply --dry-run=server -f -` against the freshly-installed CRDs of every bumped operator. Or an upgrade-from-N-1 E2E lane. - -### v1.4.0 — Flux 2.7 → 2.8 readiness deadlock (#2602) - -**Bug:** helm-controller v1.5.0 (shipped in Flux 2.8) changes the default wait strategy to kstatus polling, which polls **every applied resource including child HelmReleases**. The `packages/apps/kubernetes` umbrella has 19 child HRs that each `dependsOn: parent` (with a `lookup` guard so the dependency only activates after the parent exists). Pre-2.8: parent went Ready as soon as Helm returned. Post-2.8: parent waits for children, children wait for parent. Deadlock. - -**Why CI missed it:** the cycle only manifests on the second reconcile (parent exists, so the `lookup` activates the child's `dependsOn`). Fresh-install E2E dodges this — the parent-already-exists branch isn't taken on the first apply. - -**Would have caught it:** apply-mutate-reapply E2E (any chart bump in the kubernetes umbrella triggers a second reconcile). Plus an explicit PR-template requirement: for any flux/helm-controller bump, paste the upstream breaking-changes section into the PR body. - -### Patterns these share - -1. **Fresh-install CI is insufficient.** All three regressions passed fresh-install lanes. -2. **Upstream breaking-defaults bumps are the common shape.** kubevirt CRD hardening, helm-controller default wait strategy, cert-manager API surface drift. Pre-merge reading of upstream changelogs would have caught two of three. -3. **One-engineer-bitten-once is the only safety net for live verification.** The cert-manager bump PR included an explicit live-cluster verification step — because the same engineer had been bitten the prior week. There is no checklist requiring it. - -## Pre-release verification checklist - -For RCs and final releases, run this before merging the release PR. Each item is here because something has shipped without it and broken users. - -- [ ] **The published rc passed mandatory full E2E.** Confirm the promote PR says `✅ RC full e2e was verified`; the normal promote PR itself runs no E2E. Add `full-e2e` manually only when a repeat is required, and treat a ⚠️ bypass line as an explicit release-risk decision. -- [ ] **Upgrade from previous patch** (`vX.Y.(Z-1)`) on a real cluster. `helm upgrade` succeeds, all HelmReleases reach Ready, no CRD schema rejections. -- [ ] **Upgrade from previous minor** (`v(X-1).Y.0`) for RC and minor releases. Same assertions. -- [ ] **At least one tenant with `publishing.ingressName != tenant-root`** and a real Let's Encrypt-staging cert issuance. -- [ ] **Apply-mutate-reapply** on the kubernetes umbrella HelmRelease (catches second-reconcile bugs). -- [ ] **For every CRD-source bump** (kubevirt, cert-manager, flux, cilium, kamaji): `helm template | kubectl apply --dry-run=server` against the freshly-installed CRDs. -- [ ] **For every flux / helm-controller / kubevirt-operator / cert-manager bump:** paste the upstream breaking-changes section of the bump's CHANGELOG into the PR body before merge. -- [ ] **After cutting the tag,** watch `pull-requests-release.yaml::Finalize Release`. If it fails on `Draft release for v... not found`, undraft the release manually and file the workflow regression — this has been a known recurring failure since the `Publish draft release` step lost its explicit `github-token` in refactor `66a756b6`. -- [ ] **The promote PR carries `docs/changelogs/vX.Y.Z.md` and it was verified** per [Changelog pre-publish verification](#changelog-pre-publish-verification). Finalize publishes this file as the release body, so verifying it before merge is verifying the release notes. -- [ ] **The `cozystack/website` docs PR stays parked until publish.** Promotion opens it early; merging it before the GitHub release for `vX.Y.Z` is live 404s the new version's docs and API links. Merge it right after finalize publishes the release (regular-release step 6, patch step 7). - -## CI failures release engineers commonly hit - -| Symptom | Class | Diagnosis | Block release? | -|---------|-------|-----------|----------------| -| `Draft release for vX.Y.Z not found` on every release-* merge | Workflow regression — `Publish draft release` lost its `github-token` in a past refactor; default `GITHUB_TOKEN` cannot list drafts | At `pull-requests-release.yaml:163`. Undraft manually, file the fix | No | -| Changelog Copilot job returns 402 | Token quota — `COPILOT_GITHUB_TOKEN` premium-request quota empty | Both changelog entry points hit it: the optional rc-time `changelog-rc` dispatch and the promote-time `changelog` job (both call the reusable `changelog-generate.yaml`), plus the `tags.yaml` backstop | No — `continue-on-error: true` on the AI step, generate changelog by hand following [`agents/changelog.md`](./agents/changelog.md) | -| `Prepare Release` skipped and the run red, `Build arm64 images` failed | The rc's arm64 build failed | Read the `build-arm64` log; `build-arm64-nightly.yaml` usually failed the same way first. Fix on the line, cut the next rc | Yes — an rc is not published amd64 only | -| `Stitch multi-arch indexes and republish` fails listing refs | `verify-multiarch.sh` found refs that are not amd64+arm64 indexes, or that it could not check | Each listed ref names its reason. Single-platform manifest or index without one architecture: make the image multi-arch, or allowlist its repository with a reason if it cannot be. Manifest unreadable: correct the ref or digest; the allowlist does not excuse it. Rate-limited or unavailable after 3 tries: re-run once the throttle clears | Yes | -| E2E `kubernetes-test` fails with `CSINode does not contain driver csi.kubevirt.io` | Test ordering bug | Tenant's `kubernetes-${test_name}-csi` HelmRelease still installing when NFS PVC is created. Fix: wait for `csinode/` to advertise the driver before creating the PVC | Yes — flaky and masks real CSI bugs | -| `tenant-root` HR in perpetual upgrade-rollback loop after OIDC patch | Real bug at the intersection of #2602's readiness change and the cozystack reconcile cadence | Look for `rate: Wait(n=1) would exceed context deadline` in operator logs; multiple tenant child HRs stuck `InProgress` | Yes | -| `cozy-dashboard` ImagePullBackOff from `cozystack-ui:latest` | Registry flake | Known noisy on some OCI mirrors; not currently in the prepull set | No — retry | -| PR CI: `docker push` 409 on `:latest` for the same image across two PRs | Concurrent manifest race on a shared floating tag | The `:latest` tag was previously written by every `make image` regardless of PR. Resolved by the image-tag refactor (`IMAGE_TAG=pr--` in PR builds, `IMAGE_TAG=` only at release time) | No — `gh run rerun --failed` | - -**Heuristic:** workflow-token and Copilot-quota issues are out-of-band — release the tag, fix the workflow in a follow-up PR. Test ordering bugs and tenant-root reconcile storms are real and should block. - -If you find yourself doing the same manual fixup on two consecutive releases (e.g. "undraft the release"), open a workflow-regression issue. Workflow bugs with known manual workarounds rot silently for months. - -## Tag immutability - -Published tags are **write-once** — once a `vX.Y.Z` or rc tag is pushed it is never moved or deleted. (Nightlies are not git tags at all — they are GHCR OCI tags, see [Nightly builds](#nightly-builds).) Moving a tag silently poisons the Go module proxy / pkg.go.dev cache for `api/apps/v1alpha1/vX.Y.Z` and drifts SBOM/provenance toolchains, so the release flow is built so a move is impossible by construction: - -| File | Tag / branch handling | -|------|------------------------| -| [`pull-requests-release.yaml`](../.github/workflows/pull-requests-release.yaml) | Verifies the pinned packages candidate before creating a stable name. Creates `vX.Y.Z` at the PR merge commit **write-once** (create if absent, no-op if unchanged, fail loudly if it would move — the merge commit is new, so there is nothing to move). Cuts the `api/apps/v1alpha1/` Go-submodule tag write-once at the same commit, **stable only** (never rc/beta/alpha). The `release-X.Y` maintenance branch is fast-forward-only. Retags the verified packages artifact and rc container digests to the stable image tag (also write-once at the image level). | -| [`tags.yaml`](../.github/workflows/tags.yaml) | The `release-X.Y.Z-rc.N` staging branch is a mutable staging ref (compare-before-force: skipped when unchanged, force+log only when it genuinely moves). Fails fast on a stable `vX.Y.Z` tag with no pre-existing draft — stable tags come from `promote-rc.yaml`, never a hand push. | -| [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) | Mutates no stable tags. Before staging, verifies green full E2E from the latest rc tag-push run or a tag-correlated manual published-tag run (unless loudly bypassed). Commits and publishes the rewritten packages tree under a run-specific temporary `promotion-*` tag, adds a second commit pinning its digest on the `release-X.Y.Z` branch, generates the changelog there, and opens the promote PR with `release` but no automatic `full-e2e`; the PR gate verifies the candidate and finalize repeats that verification before the stable tag and image retag. The staging branch is force-pushed on re-dispatch, but an existing `docs/changelogs/vX.Y.Z.md` is preserved across the rebuild. | - -This is the immutable-tag + rc-promotion model from [#2677](https://github.com/cozystack/cozystack/issues/2677): stable `vX.Y.Z` is created only by promoting an existing rc, rc tags are write-once, and `api/apps/v1alpha1/vX.Y.Z` is created only on a stable release. The old nightly `auto-release.yaml` (which delete-recreated auto-bumped patch tags) has been removed. rc and stable tags accrete permanently; the only churning artifacts are GHCR nightlies and abandoned temporary promotion candidates, which [`retention.yaml`](../.github/workflows/retention.yaml) prunes (see [Nightly builds](#nightly-builds)) — GHCR storage is the cost vector to watch. - -### Repo-side enforcement (tag-protection rulesets) - -The guarantees above are enforced in the workflows. Repo admins back them with two GitHub **tag rulesets** targeting `v*`: - -- **Immutability** — *Restrict updates* + *Restrict deletions*, **no bypass**. No actor (not even the CI app) can move or delete a shipped `v*` tag. Safe to enable at any time: nothing in the release flow ever moves or deletes a `v*` tag, it only creates new ones. -- **Creation control** — *Restrict creations* with the **CI app as the sole bypass actor**, plus *Limit branches/tags updated in a single push* = 1. This turns "don't hand-push tags" from convention into hard enforcement: every `v*` tag — pre-release **and** stable — can then be created only by CI. Enable this **only after** [`cut-prerelease.yaml`](../.github/workflows/cut-prerelease.yaml) is in use, since that workflow is what makes CI the sole creator of pre-release tags; enabling it earlier would lock maintainers out of cutting rc/alpha/beta. (The Go-module tag `api/apps/v1alpha1/*` gets an equivalent pair of rulesets.) - -## Splitting a release-blocking bundle PR - -Sometimes the work that has to land before a release is a 40-commit grab bag (CI stabilization, dependency bumps, chart fixes). Splitting it makes it reviewable and survivable. The strategy that has worked: - -1. **Verify what's already on `main`.** Use `git merge-base --is-ancestor` and subject grep to drop commits that have already landed. -2. **Find existing open PRs that cover the same commits.** Force-push to the existing branch instead of opening a duplicate — preserves review threads. -3. **Identify the "bottleneck" PR.** The one whose changes make previously-silent misconfigurations into hard errors (typically an operator-version bump). Everything else can fan out in parallel. -4. **Squash iterative same-author bug-on-bug fixes** inside each split. Preserve authorship via `--author` on the squashed commit. -5. **Mark the keystone PR as draft** and link in the body to the dependency PRs. Reviewers know it intentionally fails CI until the chain lands. -6. **Beware of concurrent registry pushes.** When fanning out 5+ PR CI runs simultaneously, multiple `make image` jobs can collide on shared floating tags. The `IMAGE_TAG=pr--` refactor solved this; if you see `Conflicted with another upload of the same manifest`, `gh run rerun --failed`. - -## Cleanup after release - -- The `release-X.Y.Z` branch is deleted by GitHub when its PR merges. The `release-X.Y` maintenance branch is created by `cut-prerelease.yaml` at the rc freeze (create-only, never force-moved) and fast-forwarded afterwards by `pull-requests-release.yaml::Ensure maintenance branch`. -- The draft release is published by the same workflow. -- The release body is set by finalize from `docs/changelogs/vX.Y.Z.md` at publication. Afterwards, `update-releasenotes.yaml` re-syncs that file into the GitHub Release body on any push to `main` that touches it — so edits to a published changelog land on the release page the next time `main` moves. -- Local cleanup: remove your worktree (`git worktree remove`) and prune merged release branches. +## Pages + +- [Introduction](./release/introduction.md): the three release types and the tag push that drives the pipeline. +- [Release Candidates](./release/release-candidates.md): what an rc is, and how cutting the first one freezes `release-X.Y`. +- [Regular Releases](./release/regular-releases.md): promoting an rc to `vX.Y.0`, step by step. +- [Patch Releases](./release/patch-releases.md): the same flow for `vX.Y.Z` on an existing `release-X.Y` line. +- [What CI does during the release process](./release/ci-pipeline.md): which workflow does what, phase by phase, including multi-arch builds and the rc E2E gate. +- [Stable tags come from rc promotion](./release/stable-tags.md): why a stable tag only ever comes from promoting an rc. +- [Release signatures and SBOM](./release/signatures-and-sbom.md): the source SBOM, checksums and cosign signature on each release, the SBOM each release image carries, and how to verify and read them. +- [Nightly builds](./release/nightly-builds.md): how a nightly mirrors `main` to GHCR, and how to install one. +- [Backports](./release/backports.md): the policy that decides what goes into a patch release, and how a contributor gets a fix there. +- [Backport mechanics](./release/backport-mechanics.md): cherry-pick triage before a patch, and the monotonic-counter trap. +- [Backport tasks](./release/backport-tasks/README.md): resolving a batch conflict, deferring the original a failed batch CI comes from, and checking a pull request title, with the check each has to pass. +- [Changelogs](./release/changelogs.md): changelog failure modes and the pre-publish checks. +- [Release-time fires](./release/release-time-fires.md): regressions that escaped to users, and the verification gaps they expose. +- [Pre-release verification checklist](./release/pre-release-checklist.md): what to verify before merging a release PR. +- [CI failures release engineers commonly hit](./release/ci-failures.md): known failures and whether each one blocks a release. +- [Tag immutability](./release/tag-immutability.md): write-once tags and the rulesets that enforce them. +- [Splitting a release-blocking bundle PR](./release/splitting-bundle-prs.md): how to break up a grab-bag PR that blocks a release. +- [Cleanup after release](./release/cleanup.md): what is left to clean up once a release is out. ## See also - [`agents/changelog.md`](./agents/changelog.md) — canonical changelog generation process. -- [`agents/contributing.md`](./agents/contributing.md) — commit/PR conventions, backport label semantics. +- [`agents/contributing.md`](./agents/contributing.md) — commit/PR conventions. - [`agents/releasing.md`](./agents/releasing.md) — pointer file for AI agents handling release tasks. - [`.github/workflows/cut-prerelease.yaml`](../.github/workflows/cut-prerelease.yaml) — the sole entry point for cutting a pre-release tag as the CI app. - [`.github/workflows/tags.yaml`](../.github/workflows/tags.yaml) — tag-push pipeline (rc build + mandatory `rc-e2e`; stable-tag changelog/website backstops). @@ -714,5 +53,6 @@ Sometimes the work that has to land before a release is a 40-commit grab bag (CI - [`agents/image-refs.md`](./agents/image-refs.md) — where image references live, what their tags mean, and the invariants promotion and mirroring rely on. - [`hack/lib/image-refs.sh`](../hack/lib/image-refs.sh) — the single enumeration of ref-bearing files, shared by the promote, retag and mirror tooling. - [`.github/workflows/retention.yaml`](../.github/workflows/retention.yaml) — GHCR nightly and abandoned promotion-candidate pruning. -- [`.github/workflows/backport.yaml`](../.github/workflows/backport.yaml) — automatic cherry-pick bot. +- [`cmd/backport-batch`](../cmd/backport-batch/README.md) — the backport batch that applies the [backport policy](./release/backports.md). +- [`.github/workflows/backport-batch.yaml`](../.github/workflows/backport-batch.yaml) — runs the backport batch on Monday and Thursday for each line it runs for: the newest released lines, and a line in its rc window. - [`.github/workflows/update-releasenotes.yaml`](../.github/workflows/update-releasenotes.yaml) — sync changelog → GitHub Release body. diff --git a/docs/release/backport-mechanics.md b/docs/release/backport-mechanics.md new file mode 100644 index 000000000..f713d20aa --- /dev/null +++ b/docs/release/backport-mechanics.md @@ -0,0 +1,37 @@ +## Backport mechanics + +Under the [backport policy](./backports.md) fixes reach a release line through the batch, and nobody labels pull requests for backport. This page covers the mechanics around it: cherry-pick triage before a patch, and the trap that keeps migrations out of patch releases. + +### Cherry-pick triage before a patch + +A patch release includes bugfixes for code that shipped in the corresponding minor `vX.Y.0`. What goes in is decided by the rules in [Backports](./backports.md), not per pull request, and each batch pull request says what it picked, what it deferred and what it skipped under which rule. What is left to check before a cut: + +```bash +# Exits 1 and prints the URLs while a pull request the rules take is not on +# the line, including one only on the open batch PR, which the cut would not +# carry; see cmd/backport-batch/README.md. +go build ./cmd/backport-batch +./backport-batch check --line release-X.Y --cut +``` + +[`backport-batch check`](../../cmd/backport-batch/README.md) applies the rules to every pull request merged into `main` since the line branched, and looks for each one they take on the line and then on the open batch pull request: by picks whose `-x` lines name its merge commit or every commit of its own, by a pick that would change nothing, and, on commits from before the policy's batch-only day, by what their text says, such as `Backport of #N`. With `--cut` a change found only on the open batch pull request fails the check too, since a cut tags the line and would not carry it: merge the batch pull request first. A conflict the batch deferred, or an original deferred to the next batch, prints as a warning without failing the check. A change that reached the line by hand counts like any other, by what reached the line. + +### Backporting code that touches monotonic counters + +The batch never takes a change like this: rule 4 in [Backports](./backports.md) holds migrations, `targetVersion`, sizeable API changes and CRDs for the next minor. What follows is why, and what a maintainer has to do when a migration must reach a release line anyway. + +Migrations are the canonical case. `packages/core/platform/images/migrations/run-migrations.sh` is a linear counter — it runs `seq $CURRENT_VERSION $((TARGET_VERSION - 1))` and stops if `CURRENT >= TARGET`. This creates a forward-upgrade trap. Walking through it with `release-1.2` as the maintenance branch: + +- `main` has `targetVersion=10` and migrations 1–9 on disk. +- The `release-1.2` branch was cut when `targetVersion=5` (migrations 1–4 on disk). +- You backport a new migration to `release-1.2` at slot 5, and naively bump `targetVersion` 5 → 6 to enable it. +- Customer running 1.2.x runs the new migration, stamps `cozystack-version=6` in-cluster. +- On upgrade to 1.3 (main's `targetVersion=10`), the runner walks `seq 6 9` and runs main's migrations 6–9 — **main's own migration at slot 5 never runs.** The backport claimed that slot on `release-1.2`; main's slot 5 holds an unrelated migration that gets silently skipped. + +Mitigations, in order of preference: + +1. **Don't bump `targetVersion` in the backport.** Just ship the migration file. Prior precedent in the repo: an earlier maintenance branch's ACME backport did exactly this — the migration shipped but ran on no clusters until a later commit raised the counter. Pair this with relaxing the chart-side `{{ fail }}` guard on the maintenance branch (chart-side coalesce) so the new value works without the migration having executed. +2. **If you must bump,** add an idempotent duplicate of the now-skipped main migration at a higher slot. Bounded cost (one extra file per affected migration) and survives the skip window. +3. **Audit cross-branch counter state before deciding.** Compare migration numbering across `main`, `release-X.Y`, and `release-X.(Y-1)`. If a backport would create a skip window on any forward-upgrade path, choose option 1. + +This rule generalizes to any monotonic-counter state: schema versions, feature-flag generations, anything that gates "have I run X yet?" with a single integer. diff --git a/docs/release/changelogs.md b/docs/release/changelogs.md new file mode 100644 index 000000000..f26ac56c5 --- /dev/null +++ b/docs/release/changelogs.md @@ -0,0 +1,107 @@ +## Changelogs + +### Where the canonical process lives + +[`agents/changelog.md`](../agents/changelog.md) is the source of truth. Read it end-to-end before generating; do not infer the process from past commits or memory. The CI runs it under Copilot via the reusable `changelog-generate.yaml` — called by `promote-rc.yaml::changelog` at promote time and by the optional `changelog-rc.yaml` at rc time — and, as a backstop, `tags.yaml::generate-changelog`; locally the same prompt is replayable by following the doc directly. + +### Common changelog failure modes + +These are mistakes that have shipped in real changelogs. Verify against each before merging the changelog PR. + +**1. Commits from outside the release range.** Caused by running `git log ..HEAD` from `main` while generating a patch changelog — `HEAD` on `main` contains everything that landed since the tag plus backports merged to `release-X.Y` after the tag was cut. Use `git log ..` explicitly. The doc was updated to require this after v1.3.1's changelog shipped with 14 entries for what was actually a 1-commit release. + +**2. Original + backport listed as two separate entries.** Each backport must coalesce with its original into a single entry of the form `(in #orig, backport #bp)`. To verify after generation: + +```bash +# Every entry that mentions "backport" should also reference the original PR number +grep -E 'backport #[0-9]+' docs/changelogs/v.md | grep -v '#[0-9]*, backport' +``` + +Output should be empty. + +**3. Wrong PR author.** The squash-merge commit's author is whoever clicked "Merge" — not the person who wrote the code. **Always** resolve via `gh pr view --json author --jq .author.login`, never `git log --format=%an`. This bites hardest for website-repo entries where the same merger handles most PRs. + +**4. Superseding patch, stale changelog.** Tags are write-once, so a critical fix after `vX.Y.Z` ships as the next patch `vX.Y.(Z+1)` rather than moving the tag. If a changelog PR already exists for a tag you are superseding, the changelog work transfers to the new patch. Compare `git log ..` against what the existing `changelog-vX.Y.Z` branch already documents; only add/remove the deltas. If the only new commits are CI-internal or a revert of a feature that never reached a stable tag, no changelog edit is needed. + +**5. PR numbers swapped inside prose.** The entry-format validator checks bullet entries but ignores Feature Highlights paragraphs and Upgrade Notes. Both have caused wrong PR references in shipped changelogs. Verify every `#NNNN` in prose with `gh pr view `. + +**6. Hallucinated entries.** Every entry must correspond to a commit in `git log ..`. Verify by extracting PR numbers from the file and grep-searching the commit range: + +```bash +grep -oE '#[0-9]+' docs/changelogs/v.md | sort -u | while read pr; do + n=${pr#\#} + if ! git log .. --grep="#${n}" --oneline | grep -q .; then + echo "PR $pr in changelog but not in commit range" + fi +done +``` + +**7. Bot accounts as human contributors.** `cozystack-ci[bot]`, `github-actions`, `dependabot`, `renovate`, and any `app/*` login must NOT appear in the `## Contributors` list. They legitimately appear in **per-entry attribution** (e.g. a Renovate-authored PR) — but not in the human roll-call. + +**8. Title duplicated as description.** `* **fix(foo): X**: fix(foo): X (...)` means you wrote the conventional-commit subject twice and never wrote a user-facing description. The brief and the detail must say different things — the detail explains what the change means for users. + +### Changelog pre-publish verification + +Run these checks before merging the changelog PR. Catches problems in roughly the order they occur: + +```bash +PREV=v1.2.0; NEW=v1.2.1 + +# A. Every chart bump is mentioned somewhere in the changelog +git diff --name-only $PREV..$NEW -- 'packages/*/charts/*/Chart.yaml' 'packages/*/*/charts/*/Chart.yaml' | while read f; do + old=$(git show "$PREV:$f" 2>/dev/null | yq '.version // ""') + new=$(git show "$NEW:$f" 2>/dev/null | yq '.version // ""') + if [ "$old" != "$new" ]; then + component=$(basename "$(dirname "$f")") + if ! grep -qi "$component" docs/changelogs/$NEW.md; then + echo "MISSING: $component $old -> $new ($f)" + fi + fi +done + +# B. Out-of-Chart.yaml pins (Talos, tenant K8s, vendored images) +for path in packages/core/talos/images/talos/profiles/installer.yaml packages/core/installer/values.yaml packages/apps/kubernetes/values.yaml; do + git diff $PREV..$NEW -- "$path" | head -40 +done +git diff --name-only $PREV..$NEW -- 'images/*/Dockerfile' + +# C. Every cited #NNNN exists and is a merged PR (not an issue) +grep -oE '#[0-9]+' docs/changelogs/$NEW.md | sort -u | sed 's/#//' | xargs -P 20 -I{} sh -c ' + state=$(gh api /repos/cozystack/cozystack/issues/{} --jq ".pull_request.merged_at // \"ISSUE\"" 2>/dev/null) + echo "#{} $state" +' | grep -E 'ISSUE|null|^#[0-9]+ $' + +# D. URL liveness +grep -oE 'https://[^)" ]+' docs/changelogs/$NEW.md | sort -u | xargs -P 10 -I{} sh -c ' + code=$(curl -s -o /dev/null -w "%{http_code}" {}) + if [ "$code" != "200" ]; then echo "$code {}"; fi +' + +# E. Website PR authors are PR authors, not commit authors +grep -oE 'cozystack/website#[0-9]+' docs/changelogs/$NEW.md | sort -u | sed 's|cozystack/website#||' | while read n; do + pr_author=$(gh pr view $n --repo cozystack/website --json author --jq .author.login) + echo "website#$n -> @$pr_author" +done + +# F. Working tree clean except for the changelog +git status --porcelain | grep -v "docs/changelogs/$NEW.md" +# Should output nothing. +``` + +Ship criterion: every chart bump surfaced, no issue-cited-as-PR, no 404 URLs, every website entry attributed via `gh pr view` (not commit author), `git status` shows only the changelog. + +### Cozystack-specific URL quirks + +`cozystack.io` paths are versioned: `/docs/operations/foo/` and `/docs/next/foo/` both 404. Only `/docs/vX.Y/foo/` and the unversioned canonical `/docs/foo/` serve content. AI-generated changelogs invent the unversioned form. Verify URLs with the `curl` loop above. + +### Platform-component coverage + +For a minor release (`vX.Y.0`), changelog entries should surface what an upstream chart bump actually brought in — features, breaking changes, security fixes. PR commit messages usually don't capture this; the upstream `CHANGELOG.md` or GitHub Release notes do. For each bumped `Chart.yaml`: + +1. Read the `sources:` field — typically a GitHub URL. +2. Fetch release notes strictly inside `old < x <= new` via `gh release view --repo / --json body,name`. +3. Summarize in 2–5 user-impact bullets per component (new features, breaking changes, CVEs, important deprecations). Don't reproduce upstream notes verbatim; link to them. +4. If a PR-level entry already mentions the bump, enrich that entry with bullets — don't duplicate the entry. +5. If 3+ bumps have no PR-level mention, introduce a `## Platform Components` section. + +If you can't find upstream notes for a bumped component, list it with version-only info and no bullets. Hallucinated bullets are worse than missing detail. diff --git a/docs/release/ci-failures.md b/docs/release/ci-failures.md new file mode 100644 index 000000000..9beabe02c --- /dev/null +++ b/docs/release/ci-failures.md @@ -0,0 +1,16 @@ +## CI failures release engineers commonly hit + +| Symptom | Class | Diagnosis | Block release? | +|---------|-------|-----------|----------------| +| `Draft release for vX.Y.Z not found` on every release-* merge | Workflow regression — `Publish draft release` lost its `github-token` in a past refactor; default `GITHUB_TOKEN` cannot list drafts | At `pull-requests-release.yaml:163`. Undraft manually, file the fix | No | +| Changelog Copilot job returns 402 | Token quota — `COPILOT_GITHUB_TOKEN` premium-request quota empty | Both changelog entry points hit it: the optional rc-time `changelog-rc` dispatch and the promote-time `changelog` job (both call the reusable `changelog-generate.yaml`), plus the `tags.yaml` backstop | No — `continue-on-error: true` on the AI step, generate changelog by hand following [`agents/changelog.md`](../agents/changelog.md) | +| `Prepare Release` skipped and the run red, `Build arm64 images` failed | The rc's arm64 build failed | Read the `build-arm64` log; `build-arm64-nightly.yaml` usually failed the same way first. Fix on the line, cut the next rc | Yes — an rc is not published amd64 only | +| `Stitch multi-arch indexes and republish` fails listing refs | `verify-multiarch.sh` found refs that are not amd64+arm64 indexes, or that it could not check | Each listed ref names its reason. Single-platform manifest or index without one architecture: make the image multi-arch, or allowlist its repository with a reason if it cannot be. Manifest unreadable: correct the ref or digest; the allowlist does not excuse it. Rate-limited or unavailable after 3 tries: re-run once the throttle clears | Yes | +| E2E `kubernetes-test` fails with `CSINode does not contain driver csi.kubevirt.io` | Test ordering bug | Tenant's `kubernetes-${test_name}-csi` HelmRelease still installing when NFS PVC is created. Fix: wait for `csinode/` to advertise the driver before creating the PVC | Yes — flaky and masks real CSI bugs | +| `tenant-root` HR in perpetual upgrade-rollback loop after OIDC patch | Real bug at the intersection of #2602's readiness change and the cozystack reconcile cadence | Look for `rate: Wait(n=1) would exceed context deadline` in operator logs; multiple tenant child HRs stuck `InProgress` | Yes | +| `cozy-dashboard` ImagePullBackOff from `cozystack-ui:latest` | Registry flake | Known noisy on some OCI mirrors; not currently in the prepull set | No — retry | +| PR CI: `docker push` 409 on `:latest` for the same image across two PRs | Concurrent manifest race on a shared floating tag | The `:latest` tag was previously written by every `make image` regardless of PR. Resolved by the image-tag refactor (`IMAGE_TAG=pr--` in PR builds, `IMAGE_TAG=` only at release time) | No — `gh run rerun --failed` | + +**Heuristic:** workflow-token and Copilot-quota issues are out-of-band — release the tag, fix the workflow in a follow-up PR. Test ordering bugs and tenant-root reconcile storms are real and should block. + +If you find yourself doing the same manual fixup on two consecutive releases (e.g. "undraft the release"), open a workflow-regression issue. Workflow bugs with known manual workarounds rot silently for months. diff --git a/docs/release/ci-pipeline.md b/docs/release/ci-pipeline.md new file mode 100644 index 000000000..73f105bb7 --- /dev/null +++ b/docs/release/ci-pipeline.md @@ -0,0 +1,107 @@ +## What CI does during the release process + +The numbered process of a [regular release](./regular-releases.md#regular-releases) is implemented by seven workflows, including the reusable/manual [`e2e-tag.yaml`](../../.github/workflows/e2e-tag.yaml), plus the reusable [`changelog-generate.yaml`](../../.github/workflows/changelog-generate.yaml) that two of them share to produce the changelog. Knowing which job does what makes the failure modes much easier to diagnose. + +1. [`cut-prerelease.yaml`](../../.github/workflows/cut-prerelease.yaml) — manual dispatch (from `main` or `release-X.Y`): the sole entry point for creating a pre-release tag. Validates the `-alpha`/`-beta`/`-rc` tag, then pushes it (write-once, at the branch tip) as the CI app so `tags.yaml` fires. Cutting the first `vX.Y.0-rc.N` also creates `release-X.Y`, freezing the line; afterwards a `vX.Y.*` dispatch from `main` is refused. Stable tags are never created here. +2. [`tags.yaml`](../../.github/workflows/tags.yaml) — fires on any pre-release tag push: `build-amd64` and `build-arm64` build the pre-release side by side, then `prepare-release` publishes it and pushes its digest-pinned staging branch, then — **only for `-rc.N` tags** — the mandatory `rc-e2e` job calls `e2e-tag.yaml` in the same run (`-alpha.N` / `-beta.N` skip it and are validated on demand with `e2e-tag.yaml`'s manual button). `generate-changelog` and `update-website-docs` also live here but are **backstops only** — both are normally produced at promotion time (see 4); on a stable tag push each self-skips or no-ops when the promote flow already did the work. These stable-tag backstops only fire because finalize pushes the stable tag with the CI app token (a `GITHUB_TOKEN` push triggers no run — see 5). +3. [`e2e-tag.yaml`](../../.github/workflows/e2e-tag.yaml) — reusable from `tags.yaml::rc-e2e` and manually dispatchable for a published tag. It checks that the tag's release carries its nocloud disk, resolves the digest-pinned tree and runs the full install, OpenAPI, and Chainsaw suite on Talos containers. A tag cut before the container lane existed has to be dispatched from its own ref. Its dispatch run title is the constant workflow name, so the promote gate correlates a manual run through the successful tag-bearing E2E job name rather than the title. +4. [`promote-rc.yaml`](../../.github/workflows/promote-rc.yaml) — manual dispatch, from `main` only (the signature over a stable release's assets names that ref, see [Release signatures and SBOM](./signatures-and-sbom.md#release-signatures-and-sbom)). Five jobs: `parse` validates the rc tag and requires green full-E2E evidence before any staging; `promote` rewrites and commits the rc tags, publishes that packages tree as a run-specific temporary reproducible candidate, commits its digest pin in the `release-X.Y.Z` tree, and drafts the stable release; `changelog` calls the reusable `changelog-generate.yaml` **in parallel** to produce `docs/changelogs/vX.Y.Z.md` (regenerating it, or copying an rc-time changelog already committed to the `release-X.Y.Z-rc.N` staging branch by `changelog-rc.yaml` — see 7); `website-docs` (needs `parse`, `promote`) opens the `cozystack/website` docs PR from the staging branch via `FETCH_REF` (see [Phase 3](#phase-3--update-website-docs-now-the-backstop)); `open-pr` joins them, commits the changelog onto the staging branch, and opens the `release-X.Y.Z` promote PR with the `release` label but no automatic `full-e2e`. So the changelog and website docs are reviewed as part of the promotion, the already-checked rc suite is not repeated by default, and the PR body carries ✅ verified or ⚠️ bypassed plus ✅/⚠️ changelog and website-docs lines. Both `changelog` and `website-docs` are non-blocking. No stable registry name is created at dispatch. +5. [`.github/workflows/pull-requests.yaml`](../../.github/workflows/pull-requests.yaml) verifies a promotion candidate against the prospective merge tree and original rc artifact before the required `E2E Tests` status can turn green. [`pull-requests-release.yaml`](../../.github/workflows/pull-requests-release.yaml) fires after merge and repeats that verification before any stable tag, then cuts the write-once stable + Go-module tags, publishes the release **with the merged changelog as its body**, retags the candidate and rc images to stable by digest (`:latest` gated on newest-stable), and publishes the stable chart. +6. [`update-releasenotes.yaml`](../../.github/workflows/update-releasenotes.yaml) — fires on pushes to `main` that touch `docs/changelogs/v*.md`; syncs that content into the corresponding GitHub Release body. No longer the primary path (finalize sets the body directly, which is race-free and works for maintenance lines too) — this now covers later edits to a published changelog and manual re-syncs via `workflow_dispatch`. +7. [`changelog-rc.yaml`](../../.github/workflows/changelog-rc.yaml) — manual dispatch, **optional**: generate the changelog at rc time instead of waiting for promotion, while there is still time to review it. It calls the same reusable [`changelog-generate.yaml`](../../.github/workflows/changelog-generate.yaml) and commits the result to the `release-X.Y.Z-rc.N` staging branch as `docs/changelogs/vX.Y.Z.md`; the promote-time `changelog` job then copies that file instead of running the AI again. Skipping this workflow changes nothing — promote generates the changelog itself, exactly as before. + +### Phase 1 — `build-amd64`, `build-arm64` and `prepare-release` (hard gate) + +On an rc tag push: builds images for both architectures, commits digest pins, creates the draft release, and pushes the `release-X.Y.Z-rc.N` staging branch. `build-amd64` holds the release-exists no-op and the rejection of a hand-pushed stable tag, and builds the amd64 images and the release assets (its `make build` also pushes the packages artifact and the installer chart, pinned on amd64 digests); `prepare-release` stitches, gates, republishes those two from the repinned tree, and publishes the release. (The stable promote PR is opened later by `promote-rc.yaml`, not here. When the stable tag itself is pushed by finalize, a draft already exists, so this job is a no-op — see [Phase 5](#phase-5--pull-requests-releaseyaml-finalize).) + +The commit (`Prepare release vX.Y.Z`, authored by `cozystack-ci[bot]`) is **digest pins and image tags only**: + +- `packages/core/{installer,platform,testing}/values.yaml` +- `packages/system/*/values.yaml` (cozystack-api, dashboard, kamaji, linstor, kubevirt-csi-node, etc.) +- `packages/apps/kubernetes/images/{kubevirt-csi-driver,ubuntu-container-disk-*}.tag` +- `packages/system/dashboard/templates/configmap.yaml` (the `$tenantText` value) + +Things that look surprising in the diff but are normal: + +- **Zero-diff packages**: when buildx fully caches and upstream base images haven't moved, the digest is byte-identical to the previous release and won't appear. Verify the build step actually ran — don't assume "no diff = nothing built." +- **Sudden churn on `ubuntu-container-disk-*` tags**: `cloud-images.ubuntu.com/noble/current/` is a moving target; these often rebuild even without code changes. +- **Switched registry**: if a self-built image moved from `docker.io` to `ghcr.io`, the registry portion of the digest changes — that's a deliberate move, not a regression. +- **`targetVersion` is NOT touched by the release PR.** Platform migration `targetVersion` is bumped earlier in a feature commit (e.g. `[platform] Bump migration targetVersion to 39 for migration 38`). The release PR only re-pins the `platform-migrations` image digest. + +#### Multi-arch images + +Every pre-release is published for amd64 and arm64. `tags.yaml::build-arm64` builds the images on a native arm64 runner and pushes each as `-arm64`, while `build-amd64` builds amd64, the Talos installer and matchbox for both arches, and the assets on x86 at the same time. `build-amd64` hands its stamped tree, the log of the tags it pushed and the assets to `prepare-release` as workflow artifacts; a missing one fails the upload or the download rather than publishing a partial release. `prepare-release` waits for both builds and runs only if both succeeded: when either fails, the rc is not prepared at all, so a release is never published amd64 only. It applies the stamped tree and runs [`hack/stitch-multiarch.sh`](../../hack/stitch-multiarch.sh), which joins each image with its arm64 twin into one index, moves onto it every tag the build pushed for that image (the rc tag and, under `PUBLISH_VERSIONED=1`, its component version, which the build records in `PUSHED_TAGS_LOG`), and repins the tree on the index digests, then [`hack/verify-multiarch.sh`](../../hack/verify-multiarch.sh), then republishes the packages artifact and the installer chart from the repinned tree. A stable tag skips `build-arm64` and builds nothing: promotion copies the rc's indexes with every platform, so the stable release is exactly as multi-arch as its rc. Because the two builds run in parallel, an rc that is not prepared (its arm64 build failed, or `prepare-release` failed before the stitch) still leaves behind what the amd64 build pushed: amd64-only images under `:` and under the component-version tags, and the `cozystack-packages:` artifact and the installer chart, both pinned on those amd64 digests. A component tag an earlier rc had moved to an index then points at an amd64-only image again, until a run that succeeds stitches it, a re-run of the same run included. + +`verify-multiarch.sh` is a hard gate. It resolves every ref the tree pins, third-party images included, and fails naming each one that is not an index carrying both `linux/amd64` and `linux/arm64`, or that it cannot resolve at all. The fix is to make the image multi-arch (build it for both, or pin a multi-arch digest of the same version); an image that cannot exist for arm64 goes in [`hack/multiarch-allowlist`](../../hack/multiarch-allowlist) by repository, with the reason on the same line. + +That gate sees only what the tree pins by digest. An image a vendored chart renders from its own defaults, or one an operator launches at runtime, is invisible to it. `rc-e2e` closes most of that gap: before tearing the sandbox down, [`e2e-tag.yaml`](../../.github/workflows/e2e-tag.yaml) reads every image the three Talos nodes pulled from their image stores (`hack/collect-images.sh`), each by the digest the pull resolved to, and runs `verify-multiarch.sh --refs-file` on that list with the same allowlist. A pre-release fails on any single-arch image, on any ref it cannot read or that is still rate-limited after the retries, and on a list the collection could not produce; a stable tag's run only warns, since it ships its rc's images. A tag whose tree predates the audit (it has no `hack/verify-multiarch.sh`) only warns and passes, so an rc cut before the audit existed can still be re-validated. The list and the audit output are in the run's `image-list-` artifact. Neither check covers a package that e2e never installs and whose images the tree does not pin by digest: an arm64 break there shows up only on an arm64 cluster. + +[`build-arm64-nightly.yaml`](../../.github/workflows/build-arm64-nightly.yaml) builds `main` for arm64 every day, so an arm64 build break shows up there before the next rc. Its first step prints `verify-multiarch.sh --report` for `main`'s committed tree: `main` pins amd64 images, so its first-party refs are all listed and are expected to be, since the rc's stitch turns them into indexes. A third-party ref that an in-tree rebuild stamps over at build time, as the Harbor components and shadowbox are, is listed too and goes away on the rc. The lines that matter are the third-party refs no build replaces and refs that cannot be resolved; each of those fails the next rc until it is fixed or allowlisted. matchbox is listed too and does not matter: `build-amd64` rebuilds it as a two-arch index. The stitch names the images it skips in its own summary: an image with no `-arm64` twin, one whose digest is also pinned in a file the rewrite does not reach, and a pair that is not one amd64 and one arm64 image. matchbox (already an index) and the e2e sandbox (never built for arm64, and allowlisted) are skipped on every rc and pass the gate; any other skipped image fails it. + +### RC E2E — mandatory check before promotion + +For an rc, `prepare-release` must finish before `tags.yaml::rc-e2e` calls the published-tag lane, so the full suite tests the digest-pinned staging tree that promotion will reuse, and the release it runs against already carries its disk. `promote-rc.yaml::parse` then checks the latest `tags.yaml` push run whose `head_branch` is the rc tag and accepts it only when the reusable E2E job succeeded. GitHub exposes called-workflow job names as `E2E Release Candidate / E2E vX.Y.Z-rc.N (full suite)`, which the gate matches by the exact tag-bearing suffix. + +The alternate evidence path scans retained successful `workflow_dispatch` runs of `e2e-tag.yaml` and accepts one whose direct E2E job is named `E2E vX.Y.Z-rc.N (full suite)`. The workflow's run title is always `E2E Release Tag` and therefore cannot correlate the input tag; artifact names also carry the tag but expire independently, so the job name is the strongest available correlator. If Actions retention or manual deletion removes the run/job metadata, or that job is renamed without updating the matcher, the gate fails closed and tells the maintainer to run the E2E Release Tag button again. `skip_e2e_gate=true` is an emergency override, not alternate evidence: it writes a loud warning and a ⚠️ PR-body line. + +### Phase 2 — the changelog (generated at rc time or promotion, not here) + +The changelog for `vX.Y.Z` is produced by [`promote-rc.yaml`](../../.github/workflows/promote-rc.yaml)'s `changelog` job (a thin call to the reusable [`changelog-generate.yaml`](../../.github/workflows/changelog-generate.yaml)) and committed onto the `release-X.Y.Z` staging branch by `open-pr`, so it rides the promote PR and is on the base branch the moment the release publishes. Finalize then uses that file verbatim as the GitHub Release body. Generating it here — on the stable tag push — would be too late by construction: that tag is created by finalize, which has already published the release. It can also be produced earlier, at rc time, via the optional [`changelog-rc.yaml`](../../.github/workflows/changelog-rc.yaml) button — which runs the same reusable core and commits the file to the `release-X.Y.Z-rc.N` staging branch, from which the promote-time `changelog` job copies it (validated) instead of regenerating. + +The `changelog` job runs **in parallel** with `promote`, detached at the rc tag (the "promotion" configuration in §2 of [`agents/changelog.md`](../agents/changelog.md)), using a separate **read-only** GitHub App token for the AI step so the model cannot mutate the repo even with `--allow-all-tools`. Every write — the commit onto the staging branch and the PR — happens in `open-pr` under a write-scoped token. + +Generation is deliberately **non-blocking**: a Copilot outage or exhausted token quota must never block a release. Only the AI step carries `continue-on-error`, and `open-pr` is gated on `promote` alone (`if: !cancelled() && needs.promote.result == 'success'`) so the promote PR opens regardless. When the changelog is missing the PR body says so explicitly — that ⚠️ line is the signal to act on. + +Recovery, in order of preference: + +1. Commit `docs/changelogs/vX.Y.Z.md` to the `release-X.Y.Z` branch by hand (follow [`agents/changelog.md`](../agents/changelog.md)). `promote` preserves it across a re-dispatch, and `open-pr` will not overwrite it with generated output. +2. Re-dispatch `promote-rc.yaml` — generation runs again; anything already on the branch wins. +3. Merge without one and let `tags.yaml::generate-changelog` (the backstop) open a changelog PR against `main` after the fact. The release is published with the draft's body until that merges. + +Known failure modes: + +- Copilot quota exhausted (`COPILOT_GITHUB_TOKEN` 402). Refill or rotate, then re-dispatch. +- AI step timeout (hard 30-min cap). +- Output truncated or whitespace-only — caught by the `Verify changelog` step (`hack/validate-changelog.sh`), which accepts either header convention (`# Cozystack vX.Y.Z` for a minor or `# vX.Y.Z ()` for a patch), and requires the leading release-link comment pointing at `releases/tag/vX.Y.Z`, a `compare/...vX.Y.Z` link ending in this version, and at least one `## ` section — deliberately with no line-count floor, since a complete short patch changelog (v1.5.1 ships in 19 lines) must not be rejected as a fragment. It downgrades to the missing-changelog path rather than shipping a fragment as release notes. + +**`tags.yaml::generate-changelog` is the backstop.** It self-skips when the changelog is already on `main`. It earns its keep in two cases: generation failed during promotion, or the promote PR targeted `release-X.Y` so the file never reached `main`. Since the rc freeze the second case is **every** release, minor and patch alike — the promote PR always targets the release line (see [Regular Releases](./regular-releases.md#regular-releases)), which makes this backstop the only route by which a published changelog reaches `main`. In that case it **ports the reviewed file from the tag commit verbatim** rather than regenerating — a second AI pass there would spend quota and then overwrite already-published, already-reviewed release notes. + +### Phase 3 — `update-website-docs` (now the backstop) + +The `cozystack/website` "update managed apps reference" PR is normally opened **earlier, at promote time**, by [`promote-rc.yaml`](../../.github/workflows/promote-rc.yaml)'s `website-docs` job — not here. That job fetches the README content from the `release-X.Y.Z` staging branch `promote` just pushed (via the website Makefile's `FETCH_REF` variable), so the docs are ready to review alongside the promote PR while the stable tag is still unborn. It refreshes the trunk version pins from the staging ref before promoting `next/` → `vX.Y/`, so a new minor's docs inherit fresh Talos/cozystack pins instead of a stale `next.yaml`. It is **non-blocking**: if it fails (or trips its `FETCH_REF`-support guard against a website checkout that predates that Makefile variable), the promote PR opens with a ⚠️ website-docs line and this tag-time job backfills the same branch. + +**Merge-timing contract:** the website PR must NOT be merged before the release is published (finalize). Merging it early flips the site's latest-version pointer to a version whose versioned docs and `openapi.json` are not on GitHub yet, so every version-pin and API link for the new version 404s. Both the website PR body and the promote PR body say so; park the website PR until the GitHub release for `vX.Y.Z` is live, then merge. + +This tag-time `update-website-docs` job stays as the **backstop**. It runs with `if: !cancelled() && needs.prepare-release.result == 'success' && !contains(github.ref_name, '-')` (surviving a failed/skipped changelog phase, stopping on cancel, and skipping every pre-release tag), decides whether to promote `next/` → `vX.Y/`, runs `make update-all`, stages `content hugo.yaml data/versions`, and force-pushes the same `update-docs-vX.Y.Z` branch. On the normal path — where `promote-rc.yaml::website-docs` already did the work — it no-ops into "No changes to commit". At tag time the stable tag exists, so its `BRANCH=RELEASE_TAG` fetch resolves without needing `FETCH_REF`. For it to fire at all, finalize must push the stable tag with the **CI app token** (see [Phase 5](#phase-5--pull-requests-releaseyaml-finalize)) — a tag pushed with the default `GITHUB_TOKEN` triggers no workflow run. + +If anyone changes the website Makefile to write somewhere else (e.g. `static/`, `i18n/`), the `git add` list must grow in **both** jobs or the PR silently drops files. + +### Phase 4 — review and merge the promote PR + +Reviewer checklist for the `chore(release): promote vX.Y.Z-rc.N -> vX.Y.Z` PR: + +- [ ] The PR body says `✅ RC full e2e was verified`; if it says the gate was bypassed, obtain explicit maintainer acceptance or add `full-e2e` and wait for that opt-in run. +- [ ] `Verify release packages candidate` is green. It pulls the candidate and original rc artifact by digest, compares the candidate to GitHub's prospective merge tree, and feeds the required `E2E Tests` status even when full E2E is not repeated. Do not edit `packages/` on this PR after generation; if the guard reports drift, re-dispatch promotion to create and pin a new run-specific candidate. Note what a green here does *not* cover: GitHub fires no `pull_request` event when the base branch advances, so a `packages/` change merged to the base afterwards is never re-verified on the PR. Finalize repeats the check against the real merge commit and catches that case before any stable name exists — but it catches it after the promote PR has already merged, so a `packages/` change landing on the base while a promote PR is open is worth holding back. +- [ ] The PR body's **website-docs line** is present: if ✅, the `cozystack/website` docs PR is open and **parked** — do NOT merge it until `vX.Y.Z` is published (see [Phase 3](#phase-3--update-website-docs-now-the-backstop)); if ⚠️, the tag-time backstop will backfill it after finalize. +- [ ] Diff is digest pins + image tags only — nothing else. +- [ ] No accidental config drift (a value file you don't recognize). +- [ ] If any chart bumped, check the new digest pulls — `crane manifest ghcr.io/cozystack/cozystack/@` should return. +- [ ] Build artifacts on the draft release page exist and match the expected count. +- [ ] The PR includes `docs/changelogs/vX.Y.Z.md`, it has content, and the entries match the commit range (see [Changelog pre-publish verification](./changelogs.md#changelog-pre-publish-verification)). If the PR body carries the ⚠️ "No changelog in this PR" line, resolve that before merging — finalize publishes this file as the release body, unless the PR also carries `docs/releasenotes/vX.Y.Z.md` (see [Phase 6](#phase-6--update-releasenotesyaml-sync-github-release-body)). + +### Phase 5 — `pull-requests-release.yaml` (finalize) + +Fires on merge of a PR that is merged, carries the `release` label, and is authored by `cozystack-ci[bot]` (the last guard closes the "name a branch `release-X.Y.Z`, get it labelled, merge it" hole). Head branch must match `release-X.Y.Z[-suffix]`. This is where **every consumer-visible or write-once side effect of a promotion happens** — after the rc's mandatory E2E was checked at dispatch and the PR merged, making promotion transactional at stable names. Steps: + +1. **Verify the packages candidate before creating any stable tag.** [`hack/verify-promoted-packages.sh`](../../hack/verify-promoted-packages.sh) pulls the candidate by the digest committed in installer values, proves it matches the merged `packages/` tree after normalizing only the impossible self-reference, rejects surviving rc image tags, then follows the candidate's embedded pre-push `platformSourceRef` back to the original rc artifact and compares normalized container `repo@digest` sets. Any package edit after candidate publication or any changed container digest aborts finalize while all stable names are still absent. +2. **Create the tag at the merge commit** (write-once). The merge commit of `Prepare release vX.Y.Z` did not exist before the PR opened, so there is nothing to move — the tag is created here for the first time. A pre-existing tag at a different commit fails the step loudly rather than being force-moved (see [Tag immutability](./tag-immutability.md#tag-immutability)). The checkout uses `persist-credentials: false` so this push authenticates as the **CI app** (via `git remote set-url`), not the default `GITHUB_TOKEN` — a `GITHUB_TOKEN` push creates no workflow run, so a persisted-credential push would silently skip `tags.yaml` entirely and its stable-tag backstops (`generate-changelog`, `update-website-docs`) would never fire. +3. **Cut the `api/apps/v1alpha1/vX.Y.Z` Go-module tag** (write-once, stable only) at the same commit, so Go consumers of `api/apps/v1alpha1` get the release. Moved here from `tags.yaml`, whose `prepare-release` body is skipped for a promoted stable (the draft already exists). +4. **Ensure the maintenance branch `release-X.Y` exists** at the tag commit. Since the rc freeze creates it, this is normally a no-op fast-forward — the merge commit is already the branch tip. Still created if missing; updated fast-forward-only — a non-fast-forward update warns and is left for a maintainer rather than being force-updated. +5. **Publish the draft release**, with `docs/releasenotes/vX.Y.Z.md`, or without it `docs/changelogs/vX.Y.Z.md`, from the merge commit as its body (absent or whitespace-only files warn and leave the draft's body in place, rather than publishing blank notes, and a body over 100000 bytes is published as a link, see [Phase 6](#phase-6--update-releasenotesyaml-sync-github-release-body)). `make_latest` is computed against published-non-prerelease tags: prereleases stay `false`; tags older than the current max stay `false` (and the current max is force-restored to `latest` if necessary, so an older patch tag cut after a newer minor won't downgrade `latest`). +6. **Retag the verified packages candidate and rc container images to stable** by digest (`hack/promote-retag.sh`, no container rebuild) and **publish the stable `cozy-installer` chart**. `:latest` (on both the images and the chart) moves only when this release's `make_latest` was `true` — the same decision as step 5, so the release's `latest` and the images' `:latest` never disagree. The chart is packaged with `platformVersion` stamped into its default values so the documented `helm --install --version X.Y.Z` path reports the stable version. + +### Phase 6 — `update-releasenotes.yaml` (sync GitHub Release body) + +Fires on pushes to `main` that touch `docs/changelogs/v*.md`. Reads each `docs/changelogs/vX.Y.Z.md` and PATCHes the matching GitHub Release's `body` if it differs. Since Phase 5 now sets the body directly from the merge commit, this is no longer on the critical path — it covers later **edits** to a published changelog, the backstop PR landing a changelog that reached `main` after the fact, and manual re-syncs via `workflow_dispatch`. It only watches `main`, which is why finalize (not this job) is what populates a release promoted onto `release-X.Y` — since the freeze, that is every release rather than only a patch. + +The release body is `docs/releasenotes/vX.Y.Z.md` when it exists and is not blank, otherwise the changelog, which stays the full record either way, and a push to `main` touching either file fires this sync. Finalize sets the body after the stable tags exist, where a body GitHub refuses would strand the release, so at publish time a body over 100000 bytes is replaced by a one-line link to `docs/changelogs/vX.Y.Z.md` on the tag, with a warning in the run's log. This sync never writes that link: it warns and leaves such a release's body as it is. The `tags.yaml` backstop ports only the changelog to `main`, so release notes have to be added to `main` as well, otherwise the sync falls back to the changelog and replaces the release notes with it whenever the changelog fits. diff --git a/docs/release/cleanup.md b/docs/release/cleanup.md new file mode 100644 index 000000000..b192c5af5 --- /dev/null +++ b/docs/release/cleanup.md @@ -0,0 +1,6 @@ +## Cleanup after release + +- The `release-X.Y.Z` branch is deleted by GitHub when its PR merges. The `release-X.Y` maintenance branch is created by `cut-prerelease.yaml` at the rc freeze (create-only, never force-moved) and fast-forwarded afterwards by `pull-requests-release.yaml::Ensure maintenance branch`. +- The draft release is published by the same workflow. +- The release body is set by finalize from `docs/changelogs/vX.Y.Z.md` at publication. Afterwards, `update-releasenotes.yaml` re-syncs that file into the GitHub Release body on any push to `main` that touches it — so edits to a published changelog land on the release page the next time `main` moves. +- Local cleanup: remove your worktree (`git worktree remove`) and prune merged release branches. diff --git a/docs/release/introduction.md b/docs/release/introduction.md new file mode 100644 index 000000000..78c0e1781 --- /dev/null +++ b/docs/release/introduction.md @@ -0,0 +1,11 @@ +## Introduction + +Cozystack uses a staged release process to ensure stability and flexibility during development. + +There are three types of releases: + +- **Release Candidates (RC)** – Preview versions (e.g., `v1.2.0-rc.1`) used for final testing and validation. +- **Regular Releases** – Final versions (e.g., `v1.2.0`) that are feature-complete and thoroughly tested. +- **Patch Releases** – Bugfix-only updates (e.g., `v1.2.1`) made after a stable release, based on a dedicated release branch. + +All three are matched by `tags.yaml`'s regex `^v\d+\.\d+\.\d+(-(alpha|beta|rc)\.\d+)?$`. The tag push is the trigger for the whole pipeline. **Pre-release tags (`-alpha` / `-beta` / `-rc`) are cut with the [`Cut Pre-release Tag`](../../.github/workflows/cut-prerelease.yaml) workflow** (manual dispatch from `main` or `release-X.Y`) — it pushes the tag as the CI app at the dispatch branch's tip, so GitHub fills `base_ref` (`tags.yaml`'s `Get base branch` step refuses tags pushed without a base) and the push triggers `tags.yaml` (a tag pushed with the default `GITHUB_TOKEN` would not). Stable `vX.Y.Z` tags are never pushed by hand at all — they are cut write-once by the promote flow at a PR merge commit. Only `-rc.N` pre-releases feed the promotion flow ([`promote-rc.yaml`](../../.github/workflows/promote-rc.yaml) accepts `vX.Y.Z-rc.N` alone); `-alpha.N` / `-beta.N` tags are built and published for download and validated on demand with the [`e2e-tag.yaml`](../../.github/workflows/e2e-tag.yaml) button, but are never promoted to a stable release. diff --git a/docs/release/nightly-builds.md b/docs/release/nightly-builds.md new file mode 100644 index 000000000..df83de7f2 --- /dev/null +++ b/docs/release/nightly-builds.md @@ -0,0 +1,21 @@ +## Nightly builds + +A nightly is an **installable copy of `main` on GHCR — not a rebuild and not a release**. [`build-main.yaml`](../../.github/workflows/build-main.yaml) already builds every push to `main` into the CI registry (OCIR); the nightly promotes that build to the public release registry (GHCR) and proves it installs. + +[`nightly.yaml`](../../.github/workflows/nightly.yaml) runs daily (gated by the `NIGHTLY_ENABLED` repo variable) as three jobs; build-disk and e2e both start once mirror has finished: + +1. **mirror** — resolve the exact commit `cozystack-packages:main` was built from, then [`hack/nightly-mirror.sh`](../../hack/nightly-mirror.sh) copies every cozystack-owned component image OCIR→GHCR **by digest** (bit-for-bit, no rebuild) and re-publishes the rewritten `cozystack-packages` artifact and the `cozy-installer` chart to GHCR, tagged `0.0.0-nightly...` plus a floating `nightly`. +2. **build-disk** — assemble the Talos `nocloud` disk from the upstream siderolabs imager (the profile references only `ghcr.io/siderolabs/*`, so this needs no cozystack image and no rebuild) and publish it as a GHCR OCI artifact (`cozystack-nocloud`) so a nightly is installable on real hardware. The nightly e2e does not boot it. +3. **e2e** — stage the published GHCR closure (rewritten tree + the `cozy-installer` chart pinned to the GHCR packages artifact) and run the **full** app suite on Talos containers, which do not boot the disk. A nightly has no diff, so Test Impact Analysis does not apply. + +No GitHub release and no `api/apps/v1alpha1/*` Go-module tag are created for a nightly. [`retention.yaml`](../../.github/workflows/retention.yaml) prunes old GHCR nightly versions (keeps the newest 14 per package the current nightly publishes, named from the published `cozystack-packages:nightly` tree by [`hack/retention-packages.sh`](../../hack/retention-packages.sh), except packages linked to another repository, which are skipped with a warning; the floating `nightly`, release tags and untagged versions are never touched) and abandoned packages candidates. A candidate version is eligible only when all of these hold: it carries at least one tag and **every** tag on that manifest matches the strict `promotion-vX.Y.Z-from-vX.Y.Z-rc.N-run--` form, so a manifest that has been retagged stable is excluded; it was last updated more than 30 days ago; its digest is not pinned by `packages/core/installer/values.yaml` at the head of an **open** pull request that is same-repository, authored by `cozystack-ci[bot]`, on a `release-X.Y.Z` branch and carrying the `release` label; and its digest is not pinned by that same file at the tip of `main` or of any `release-X.Y` branch — which is what keeps a candidate safe between the promote PR merging and finalize retagging it. Any such PR head whose pin is not exactly one `digest=sha256:<64 hex>` stops the candidate sweep for that run and fails the job, without affecting the nightly sweep. + +Install a nightly: + +```bash +helm upgrade --install cozystack \ + oci://ghcr.io/cozystack/cozystack/cozy-installer --version 0.0.0-nightly.20260626.15992304129.1 +# ...or --version nightly for the latest +``` + +The matching Talos node image is `ghcr.io/cozystack/cozystack/cozystack-nocloud:0.0.0-nightly.20260626.15992304129.1` (pull with `oras`). diff --git a/docs/release/patch-releases.md b/docs/release/patch-releases.md new file mode 100644 index 000000000..9f23e23d6 --- /dev/null +++ b/docs/release/patch-releases.md @@ -0,0 +1,82 @@ +## Patch Releases + +A patch release now follows exactly the same shape as a [regular release](./regular-releases.md#regular-releases) — by the time `vX.Y.0` ships, everything is already happening on `release-X.Y`. The only difference is where the line starts: + +* `release-X.Y` already exists, created by the freeze at `vX.Y.0-rc.1`, so there is no branch to create. +* Patch commits are cherry-picked onto it, and the rc is cut by dispatching **from `release-X.Y`** — the same dispatch used for `rc.2` of the original release. + + +Let's assume that we've released `v1.2.0` and that development is ongoing. We have introduced a couple of new features and some fixes to features that we have released in `v1.2.0`. + +Once problems were found and fixed, a patch release is due. + +```mermaid +gitGraph + commit id: "feature 3" tag: "v1.2.0-rc.1" + branch release-1.2 + checkout release-1.2 + commit id: "Release v1.2.0" tag: "v1.2.0" + checkout main + commit id: "feature 4" + commit id: "patch 1" + commit id: "feature 5" + commit id: "patch 2" +``` + + +1. Fixes reach the existing `release-1.2` branch from `main` through the backport batch, which cherry-picks what the [backport policy](./backports.md) admits and nothing else. + + The batch runs twice a week, so a fix merged into `main` reaches the line with the next batch that can pick it cleanly, without anyone asking for it. + + ```mermaid + gitGraph + commit id: "feature 3" tag: "v1.2.0-rc.1" + branch release-1.2 + checkout release-1.2 + commit id: "Release v1.2.0" tag: "v1.2.0" + checkout main + commit id: "feature 4" + commit id: "patch 1" + commit id: "feature 5" + commit id: "patch 2" + checkout release-1.2 + cherry-pick id: "patch 1" + cherry-pick id: "patch 2" + ``` + + Merge the open batch pull request first. Before the cut, `./backport-batch check --line release-1.2 --cut` (build it with `go build ./cmd/backport-batch`) has to exit 0: it exits 1 while a fix the rules take is not on the line, including one that is only on the open batch pull request, which the cut would not carry, and prints the URLs; it prints what the batch deferred as warnings. See [Cherry-pick triage before a patch](./backport-mechanics.md#cherry-pick-triage-before-a-patch). + +2. The maintainer cuts a release candidate (`v1.2.1-rc.N`) via the [`Cut Pre-release Tag`](../../.github/workflows/cut-prerelease.yaml) workflow (manual dispatch **from `release-1.2`**), which tags that branch's `HEAD`. CI builds and publishes the rc, pushes the `release-1.2.1-rc.N` staging branch, and runs the mandatory full `rc-e2e` job against the published tag. +3. Watch `rc-e2e` and wait for it to pass, then complete the other release checks. A successful manual `e2e-tag.yaml` run for `v1.2.1-rc.N` is also valid evidence if the latest `tags.yaml` run does not carry a green `rc-e2e`. **Optional:** press the `Generate RC Changelog` button ([`changelog-rc.yaml`](../../.github/workflows/changelog-rc.yaml)) to generate `docs/changelogs/v1.2.1.md` at rc time and bank it on the rc staging branch; promotion then reuses it instead of regenerating. +4. Once the checks are green, run [`promote-rc.yaml`](../../.github/workflows/promote-rc.yaml) (manual dispatch **from `main`**, as for every promotion, not from `release-1.2`) with that rc tag. It runs the same five jobs as a regular promotion: the dispatch first gates on the latest tag-push `rc-e2e` or a successful tag-correlated manual E2E run (`skip_e2e_gate` overrides it), verifies that `release-1.2` already carries the candidate-aware pipeline, then commits and publishes the rewritten packages tree under a run-specific temporary `promotion-*` tag, adds a second commit pinning the candidate on the `release-1.2.1` branch, drafts the `v1.2.1` release with the restamped assets, generates or picks up `docs/changelogs/v1.2.1.md` and commits it onto that branch, opens the parked `cozystack/website` docs PR (updating the existing `v1.2/` directory in place, since a patch does not promote `next/`), and opens the promote PR into `release-1.2` with only the `release` label. The promote PR body carries the same ✅/⚠️ E2E, changelog, and website-docs lines and runs no E2E by default; add `full-e2e` manually to force a repeat. As with a regular release every stable name is deferred to merge; retention protects the temporary candidate while this PR is open and, after it merges, while `release-1.2` still pins it — until finalize retags it stable. + + ```mermaid + gitGraph + commit id: "feature 3" tag: "v1.2.0-rc.1" + branch release-1.2 + checkout release-1.2 + commit id: "Release v1.2.0" tag: "v1.2.0" + checkout main + commit id: "feature 4" + commit id: "patch 1" + commit id: "feature 5" + commit id: "patch 2" + checkout release-1.2 + cherry-pick id: "patch 1" + cherry-pick id: "patch 2" tag: "v1.2.1-rc.1" + branch release-1.2.1 + commit id: "Prepare release v1.2.1" + checkout release-1.2 + merge release-1.2.1 id: "Pull request" + ``` + + Finally, when release is confirmed, the release sequence goes on. + +5. Maintainer reviews the PR against the [Phase 4 checklist](./ci-pipeline.md#phase-4--review-and-merge-the-promote-pr), confirms the E2E-verification and changelog lines and notes the parked website-docs line, and merges it — **do not squash-merge**. GitHub removes the merged branch `release-1.2.1` (auto-delete-on-merge is enabled). +6. CI workflow triggers on merge (all consumer-visible or write-once side effects, after the rc E2E gate and PR review): + 1. Verifies the pinned packages candidate against the merged tree and the original rc artifact's container digests. + 2. Creates the tag `v1.2.1` at the newly created merge commit — write-once. The tag is published here for the first time, never moved. + 3. Cuts the write-once `api/apps/v1alpha1/v1.2.1` Go-module tag at the same commit. + 4. Publishes the release page (`draft` → `latest`; `latest` moves only if `v1.2.1` is the newest published stable). + 5. Retags the verified packages candidate and rc container images by digest to `v1.2.1` and publishes the stable `cozy-installer` chart — no container rebuild. +7. Once the release is published, merge the parked `cozystack/website` docs PR, then announce the release to the community. diff --git a/docs/release/pre-release-checklist.md b/docs/release/pre-release-checklist.md new file mode 100644 index 000000000..8e6257304 --- /dev/null +++ b/docs/release/pre-release-checklist.md @@ -0,0 +1,14 @@ +## Pre-release verification checklist + +For RCs and final releases, run this before merging the release PR. Each item is here because something has shipped without it and broken users. + +- [ ] **The published rc passed mandatory full E2E.** Confirm the promote PR says `✅ RC full e2e was verified`; the normal promote PR itself runs no E2E. Add `full-e2e` manually only when a repeat is required, and treat a ⚠️ bypass line as an explicit release-risk decision. +- [ ] **Upgrade from previous patch** (`vX.Y.(Z-1)`) on a real cluster. `helm upgrade` succeeds, all HelmReleases reach Ready, no CRD schema rejections. +- [ ] **Upgrade from previous minor** (`v(X-1).Y.0`) for RC and minor releases. Same assertions. +- [ ] **At least one tenant with `publishing.ingressName != tenant-root`** and a real Let's Encrypt-staging cert issuance. +- [ ] **Apply-mutate-reapply** on the kubernetes umbrella HelmRelease (catches second-reconcile bugs). +- [ ] **For every CRD-source bump** (kubevirt, cert-manager, flux, cilium, kamaji): `helm template | kubectl apply --dry-run=server` against the freshly-installed CRDs. +- [ ] **For every flux / helm-controller / kubevirt-operator / cert-manager bump:** paste the upstream breaking-changes section of the bump's CHANGELOG into the PR body before merge. +- [ ] **After cutting the tag,** watch `pull-requests-release.yaml::Finalize Release`. If it fails on `Draft release for v... not found`, undraft the release manually and file the workflow regression — this has been a known recurring failure since the `Publish draft release` step lost its explicit `github-token` in refactor `66a756b6`. +- [ ] **The promote PR carries `docs/changelogs/vX.Y.Z.md` and it was verified** per [Changelog pre-publish verification](./changelogs.md#changelog-pre-publish-verification). Finalize publishes this file as the release body, so verifying it before merge is verifying the release notes. +- [ ] **The `cozystack/website` docs PR stays parked until publish.** Promotion opens it early; merging it before the GitHub release for `vX.Y.Z` is live 404s the new version's docs and API links. Merge it right after finalize publishes the release (regular-release step 6, patch step 7). diff --git a/docs/release/regular-releases.md b/docs/release/regular-releases.md new file mode 100644 index 000000000..cdb007025 --- /dev/null +++ b/docs/release/regular-releases.md @@ -0,0 +1,78 @@ +## Regular Releases + +A regular release `vX.Y.0` is **promoted from a release-candidate that already passed e2e** — never tagged by hand and never rebuilt. The bytes shipped as `vX.Y.0` are the exact `vX.Y.0-rc.N` images, retagged by digest. We'll use `v1.2.0` as an example. + +```mermaid +gitGraph + commit id: "feature" + commit id: "feature 2" + commit id: "feature 3" tag: "v1.2.0-rc.1" +``` + +A regular release sequence runs as follows: + +1. Cut the first release candidate (`v1.2.0-rc.1`) from the last good commit on `main`: run the [`Cut Pre-release Tag`](../../.github/workflows/cut-prerelease.yaml) workflow (manual dispatch **from `main`**) with the tag. It tags `main`'s HEAD as the CI app, then **creates `release-1.2` at that commit, freezing the line**; `tags.yaml` then builds the rc images, publishes the rc prerelease, pushes the digest-vendored `release-1.2.0-rc.N` staging branch, and runs its mandatory `rc-e2e` job by calling [`e2e-tag.yaml`](../../.github/workflows/e2e-tag.yaml) against that published tag. +2. Watch the rc's mandatory full E2E and wait for it to pass, then complete the other checks in [Pre-release verification checklist](./pre-release-checklist.md#pre-release-verification-checklist). If the latest `tags.yaml` run cannot provide a green `rc-e2e`, a successful manual dispatch of [`e2e-tag.yaml`](../../.github/workflows/e2e-tag.yaml) for the same tag is equivalent gate evidence. `main` is already open for the next minor, so anything landing there is **not** in this release until it reaches `release-1.2`: a fix merged into `main` gets there through the [backport batch](./backports.md#which-lines-the-batch-runs-for), which runs for `release-1.2` from the freeze on, and a fix with no counterpart on `main` through a maintainer's pull request against `release-1.2`. A later rc is cut by dispatching the workflow **from `release-1.2`**, and only once `./backport-batch check --line release-1.2 --cut` exits 0: merge the open batch pull request first, so that the rc carries every fix the policy takes. **Optional:** once the notes are worth reviewing, press the `Generate RC Changelog` button ([`changelog-rc.yaml`](../../.github/workflows/changelog-rc.yaml)) to generate `docs/changelogs/v1.2.0.md` now and bank it on the `release-1.2.0-rc.N` staging branch — so there is time to review and hand-edit it before promotion. Skipping the button changes nothing; promotion generates the changelog itself. + + ```mermaid + gitGraph + commit id: "feature" + commit id: "feature 2" + commit id: "feature 3" tag: "v1.2.0-rc.1" + branch release-1.2 + checkout main + commit id: "feature 4 (next minor)" + checkout release-1.2 + commit id: "backported fix" tag: "v1.2.0-rc.2" + ``` + +3. Once the checks are green, run the [`promote-rc.yaml`](../../.github/workflows/promote-rc.yaml) workflow (manual dispatch **from `main`**, the only branch it is dispatched from, whichever line the rc belongs to) with that rc tag. Its `parse` job verifies E2E before any staging: it accepts a successful `rc-e2e` job in the latest `tags.yaml` push run for the tag, or any retained successful manual `e2e-tag.yaml` run whose expanded E2E job name targets the tag. The emergency `skip_e2e_gate` input bypasses this check, logs a loud warning, and marks the promote PR with ⚠️. Before its first registry write, promotion also proves that the selected `main` or `release-X.Y` target already contains the candidate-aware PR gate, finalizer and trusted verifier. A maintenance branch that lacks them gets them before its next patch is promoted, through the batch with `backport/include` on the pull request that brought them to `main`, or through a maintainer's pull request against the line. Promotion **creates no stable name before merge**: dispatch does write to the registry — it publishes one `promotion-*` packages candidate, and that tag is real and publicly listable — but it creates no `vX.Y.Z` tag, moves no `latest`, and publishes no stable installer, so nothing a consumer resolves changes until the PR merges. It: + 1. Rewrites the rc version substring to `1.2.0` and commits that exact tree, serializes its `packages/` directory under a run-specific temporary OCI tag, then adds a second commit pinning the candidate's immutable digest in the installer before pushing `release-1.2.0`. Container digests remain exactly the rc's; the two commits give the OCI revision an honest source SHA without pretending the artifact can contain its own digest. + 2. Drafts the `v1.2.0` release and uploads the (restamped) assets. + 3. Generates `docs/changelogs/v1.2.0.md` (in a parallel job, detached at the rc tag) — or, if you pressed the rc-time `Generate RC Changelog` button, copies the file already banked on the `release-1.2.0-rc.N` staging branch instead of regenerating — and commits it onto the `release-1.2.0` branch, so the changelog is reviewed as part of the promotion and finalize can publish the release with it already in place. Non-blocking: if generation fails the PR still opens, carrying a ⚠️ line that says the changelog is missing. + 4. Opens the `cozystack/website` "update managed apps reference" PR (`update-docs-v1.2.0` branch), generating the docs from the `release-1.2.0` staging branch via `FETCH_REF`. Non-blocking, and **do not merge that PR until `v1.2.0` is published** — merging early breaks the live site's version routing (see [Phase 3](./ci-pipeline.md#phase-3--update-website-docs-now-the-backstop)). + 5. Opens the `chore(release): promote v1.2.0-rc.N -> v1.2.0` PR into `release-1.2`, labelled `release` only. The PR body records whether E2E was verified or bypassed, plus ✅/⚠️ changelog and website-docs lines. E2E does not run on this promote PR by default because the immutable rc already passed the mandatory full suite; a maintainer may add `full-e2e` to run it again. The stable retag itself is deferred to the merge (step 5), so an abandoned promotion leaves no stable-named images and cannot wedge a re-promotion; retention protects the temporary candidate for as long as this release PR remains open, then may remove it after 30 days. + + The PR targets the maintenance branch, not `main`: `promote-rc.yaml` prefers `release-X.Y` whenever it exists, and since the rc freeze it always does. The digest-vendored `Prepare release` commit therefore stays on the release line instead of landing on `main`. + + ```mermaid + gitGraph + commit id: "feature 3" tag: "v1.2.0-rc.1" + branch release-1.2 + checkout release-1.2 + commit id: "backported fix" tag: "v1.2.0-rc.2" + branch release-1.2.0 + checkout release-1.2.0 + commit id: "Prepare release v1.2.0" + commit id: "Pin promoted packages artifact" + checkout release-1.2 + merge release-1.2.0 id: "Pull Request" + ``` + +4. Maintainer reviews the PR and the draft release against the [Phase 4 checklist](./ci-pipeline.md#phase-4--review-and-merge-the-promote-pr): confirm the E2E-verification line is ✅ (or explicitly accept its ⚠️ bypass), the changelog line is ✅, and note the website-docs line — the `cozystack/website` docs PR that promotion parked, which stays unmerged for now. E2E does not run on this promote PR by default. Then merge — **do not squash-merge**, the stable tag must attach to a real merge commit. GitHub removes the merged branch `release-1.2.0` (the repo has auto-delete-on-merge enabled). +5. CI workflow triggers on merge (this is where every consumer-visible or write-once side effect happens, after the rc E2E gate and PR review): + 1. Pulls the pinned packages candidate by digest and verifies that it matches the merged `packages/` tree apart from the artifact's impossible self-reference, carries no rc image tags, and preserves the original rc artifact's container `repo@digest` set. + 2. Creates the tag `v1.2.0` at the newly created merge commit — write-once. The tag is published here for the first time, never moved. + 3. Cuts the write-once `api/apps/v1alpha1/v1.2.0` Go-module tag at the same commit. + 4. Ensures the `release-1.2` maintenance branch exists at the tag commit. Since the rc freeze created it, this is normally a no-op fast-forward: the merge commit is already its tip. + 5. Publishes the release page (`draft` → `latest`), using `docs/changelogs/v1.2.0.md` from the merge commit as the release body. + 6. Retags the verified packages candidate and rc container images by digest to `v1.2.0` (and `:latest` only when `v1.2.0` is the newest published stable), and publishes the stable `cozy-installer` chart — no container rebuild. +6. Once the release is published, merge the parked `cozystack/website` docs PR (opened back at promote time, step 3.iv) so the live site's version routing flips to `v1.2.0` — its versioned docs and `openapi.json` are on GitHub now. The maintainer can then announce the release to the community. + +```mermaid +gitGraph + commit id: "feature 2" + commit id: "feature 3" tag: "v1.2.0-rc.1" + branch release-1.2 + checkout main + commit id: "feature 4 (next minor)" + checkout release-1.2 + commit id: "backported fix" tag: "v1.2.0-rc.2" + branch release-1.2.0 + checkout release-1.2.0 + commit id: "Prepare release v1.2.0" + checkout release-1.2 + merge release-1.2.0 id: "Release v1.2.0" tag: "v1.2.0" +``` + +Note that `main` never receives the release commit. It forked away at the freeze and carries the next minor's work. diff --git a/docs/release/release-candidates.md b/docs/release/release-candidates.md new file mode 100644 index 000000000..55f7eec61 --- /dev/null +++ b/docs/release/release-candidates.md @@ -0,0 +1,18 @@ +## Release Candidates + +Release candidates are Cozystack versions that introduce new features and are published before a stable release. Their purpose is to help validate stability before finalizing a new feature release. + +Release candidates are given numbers `vX.Y.0-rc.N`, for example, `v1.2.0-rc.1`. The first one is cut from `main`, once all major features for the upcoming release have merged and the release enters its testing phase. + +**Cutting the first rc freezes the line.** [`cut-prerelease.yaml`](../../.github/workflows/cut-prerelease.yaml) creates `release-X.Y` at the tagged commit, and from that point the release's content is closed: + +- Every later cut for that line — `rc.2`, `rc.3`, an `alpha`/`beta`, or a patch-line rc — must be dispatched from `release-X.Y`. A dispatch from `main` is refused. +- Fixes reach the release only through the backport batch onto `release-X.Y`, or a maintainer's pull request against it (see [Backports](./backports.md)). The batch runs for the line from its next scheduled run on, [while it has not shipped](./backports.md#which-lines-the-batch-runs-for): it takes the fixes merged into `main` since the freeze, and an older line takes each of them only once this line's batch that carries it has merged. +- A later rc is cut only once `backport-batch check --line release-X.Y --cut` exits 0, so that it carries every fix the batch takes: the open batch pull request merges first. +- `main` reopens immediately for the next minor, so feature work never has to wait for the release to ship. + +This is what makes an rc mean something. Previously `release-X.Y` was created only when the stable release merged, so `rc.2` was cut from `main`'s tip and silently absorbed everything that had landed since `rc.1` — the shipped release could contain code that no rc had ever validated. + +**Un-freezing is manual and deliberate.** If an `rc.1` turns out to be unusable and the line has to be re-cut from `main`, delete `release-X.Y` by hand first; the refuse step blocks every other route back to a `main` dispatch, which is the point of it. Nothing automates the deletion, and nothing should — throwing away a frozen line is a decision worth making explicitly. + +Each RC contributes to a cumulative set of release notes that will be finalized when `vX.Y.0` is released. After testing, if no critical issues remain, the last good rc is **promoted** to the regular release `vX.Y.0` — the exact rc images retagged by digest, never rebuilt and never tagged from a later commit (see [Regular Releases](./regular-releases.md#regular-releases)). Promotion cuts the write-once `vX.Y.0` tag at the promote PR's merge commit, fast-forwards the `release-X.Y` maintenance branch the freeze already created, and opens the way for patch releases. diff --git a/docs/release/release-time-fires.md b/docs/release/release-time-fires.md new file mode 100644 index 000000000..f58fd48b7 --- /dev/null +++ b/docs/release/release-time-fires.md @@ -0,0 +1,35 @@ +## Release-time fires — what's gone wrong recently + +These are real regressions that escaped to users. The pattern in each: code passed CI, shipped in a stable release, broke customers. Read them as a working list of pre-release verification gaps. + +### v1.3.0 — cert-manager ingressClassName regression (fix #2562) + +**Bug:** A commit migrated ACME HTTP-01 to use the modern `acme.cert-manager.io/http01-ingress-ingressclassname` annotation. The matching ingress-shim code that reads that annotation was added by cert-manager upstream PR #8244 and was **never backported** to the `release-1.19` branch — but cozystack 1.3.0 shipped cert-manager v1.19.3. The annotation was silently dropped at runtime and every solver Ingress fell back to the ClusterIssuer's default class `tenant-root`. Let's Encrypt validation broke for every tenant whose `publishing.ingressName != tenant-root`. + +**Blast radius:** v1.3.0, v1.3.1, and v1.1.7 (the migration was cherry-picked to `release-1.1`). + +**Why CI missed it:** the migration commit was self-contained and `helm template` rendered cleanly. There was no E2E that creates a tenant with a non-default ingress class and watches a real Let's Encrypt cert issue. Verification had been done from the docs side ("the API has been available since cert-manager 1.12"), not against the shipped cert-manager source. + +**Would have caught it:** non-root-tenant E2E with LE-staging cert issuance, asserting the solver Ingress lands on the tenant's class. + +### v1.4.0 — kubevirt-instancetypes upgrade failure (fix #2612) + +**Bug:** `packages/system/kubevirt-instancetypes/Makefile` contained `sed -i '/persistent: true/d' templates/preferences.yaml`, which stripped the only child of `preferredTPM`, leaving `preferredTPM:` (null) in six Windows preferences. KubeVirt v1.6.x silently accepted it. The v1.6.3 → v1.8.2 operator bump (2026-04-27) hardened the CRD's OpenAPI schema and started rejecting null for an object-typed field. `helm upgrade` then failed on every cluster with `kubevirt-instancetypes` enabled. + +**Why CI missed it:** the kubevirt-operator bump and the inert `preferences.yaml` were two weeks apart in commit history. CI only tests fresh installs against the new CRDs; it doesn't run `helm upgrade` from prior versions. Schema regression on a vendored asset that hadn't been touched in 18 months. + +**Would have caught it:** `helm template | kubectl apply --dry-run=server -f -` against the freshly-installed CRDs of every bumped operator. Or an upgrade-from-N-1 E2E lane. + +### v1.4.0 — Flux 2.7 → 2.8 readiness deadlock (#2602) + +**Bug:** helm-controller v1.5.0 (shipped in Flux 2.8) changes the default wait strategy to kstatus polling, which polls **every applied resource including child HelmReleases**. The `packages/apps/kubernetes` umbrella has 19 child HRs that each `dependsOn: parent` (with a `lookup` guard so the dependency only activates after the parent exists). Pre-2.8: parent went Ready as soon as Helm returned. Post-2.8: parent waits for children, children wait for parent. Deadlock. + +**Why CI missed it:** the cycle only manifests on the second reconcile (parent exists, so the `lookup` activates the child's `dependsOn`). Fresh-install E2E dodges this — the parent-already-exists branch isn't taken on the first apply. + +**Would have caught it:** apply-mutate-reapply E2E (any chart bump in the kubernetes umbrella triggers a second reconcile). Plus an explicit PR-template requirement: for any flux/helm-controller bump, paste the upstream breaking-changes section into the PR body. + +### Patterns these share + +1. **Fresh-install CI is insufficient.** All three regressions passed fresh-install lanes. +2. **Upstream breaking-defaults bumps are the common shape.** kubevirt CRD hardening, helm-controller default wait strategy, cert-manager API surface drift. Pre-merge reading of upstream changelogs would have caught two of three. +3. **One-engineer-bitten-once is the only safety net for live verification.** The cert-manager bump PR included an explicit live-cluster verification step — because the same engineer had been bitten the prior week. There is no checklist requiring it. diff --git a/docs/release/signatures-and-sbom.md b/docs/release/signatures-and-sbom.md new file mode 100644 index 000000000..8aa47051e --- /dev/null +++ b/docs/release/signatures-and-sbom.md @@ -0,0 +1,29 @@ +## Release signatures and SBOM + +A release cut from a tree that has [`hack/sign-release-assets.sh`](../../hack/sign-release-assets.sh) carries three files beside its other assets, which [`hack/upload-assets.sh`](../../hack/upload-assets.sh) has that script write before it uploads anything. Releases cut before it have none of them, and neither does a maintenance line until the script is backported to it (see below). + +- `cozystack-source-sbom.spdx.json` is an SPDX SBOM, made with syft, of the committed source tree (the Go modules, npm lockfiles, GitHub Actions references and other manifests syft reads in it) and of the `cozypkg` binaries in the `cozypkg-*.tar.gz` assets. It does not describe what runs in the cluster: the container images carry their own SBOMs, see [SBOM of the container images](#sbom-of-the-container-images), and the Talos installer image, ISO, disk, kernel and initramfs images have none. Its name says what it covers, so that `cozystack-sbom` stays free for an SBOM of the release as a whole. +- `cozystack-checksums.txt` holds the SHA-256 of every other asset and of the SBOM. +- `cozystack-checksums.txt.sigstore.json` is a Sigstore bundle with a keyless cosign signature over the checksums. Its certificate names the workflow that published the release: `tags.yaml` for a pre-release, `promote-rc.yaml` for a stable one. + +One signature covers every asset through the checksums. To check a download, verify the checksums, then the file against them, with cosign v3.1.3 or later (v2.6.5 or later on the v2 line): + +```sh +cosign verify-blob cozystack-checksums.txt \ + --bundle cozystack-checksums.txt.sigstore.json \ + --certificate-identity-regexp '^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com && + sha256sum --check --ignore-missing cozystack-checksums.txt +``` + +Older cosign accepts a legacy-format bundle that carries a bare public key in place of a certificate and skips the identity and issuer checks for it ([GHSA-fx35-mq7g-6g98](https://github.com/sigstore/cosign/security/advisories/GHSA-fx35-mq7g-6g98)), so a bundle swapped in next to the download, by a mirror or a proxy, passes there. The bundles this repository writes are in the new format, but cosign picks the format from the file, and `--new-bundle-format` does not stop that on the v3 line (v3.1.2 still accepts such a bundle with it), so the cosign version is what protects the check. + +The regexp pins the ref each workflow runs from: `tags.yaml` on a `v*` tag push, and `promote-rc.yaml` dispatched from `main`, the only branch it is dispatched from. A passing check proves that the checksums were signed by one of those runs in `cozystack/cozystack`. A `promote-rc.yaml` dispatched from another branch fails this check in its own upload step, with no asset uploaded, and an edited `promote-rc.yaml` has to be merged into `main` before it can sign. Until the `v*` [tag rulesets](./tag-immutability.md#repo-side-enforcement-tag-protection-rulesets) restrict tag creation to the CI app, though, anyone with write access can push a `v*` tag whose tree carries an edited `tags.yaml`, and its signature passes. + +The script runs the same `cosign verify-blob` before it returns, so a signature this check would reject fails the job before the first upload, and the release is not published with assets but without a signature. Both jobs that call `make upload_assets`, `tags.yaml::prepare-release` and `promote-rc.yaml::promote`, hold `id-token: write` for the signing identity and install syft and cosign v3.1.3, the version users are asked for above. They install syft from its release tarball, checked against a SHA-256 committed in the workflow, and not with `anchore/sbom-action/download-syft`, which runs an install script fetched at run time in a job that holds the release credentials; [`hack/ci-tool-installs-contract.bats`](../../hack/ci-tool-installs-contract.bats) keeps it that way. A stable release gets its own three files rather than its rc's: promotion restamps several assets, so the checksums are recomputed and signed again. + +`promote-rc.yaml` runs the `hack/upload-assets.sh` of the rc tree, not the dispatch ref's. A stable release is therefore signed only when its rc was tagged from a tree that has `hack/sign-release-assets.sh`, and a maintenance line starts carrying signatures once that script is backported to its `release-X.Y` branch. + +### SBOM of the container images + +The container images a release pins by digest each carry an SPDX SBOM of the packages in them, which BuildKit attaches at build time as an attestation (`SBOM=1` in `tags.yaml`, scanner and flags in [`hack/common-envs.mk`](../../hack/common-envs.mk)). The attestation is an entry of the image's index, so the digest the release pins names both, and promotion copies it with the image. Read it with `docker buildx imagetools inspect @ --format '{{ json .SBOM }}'`, which prints one document per platform. The attestations are not signed. A release cut before `SBOM=1` was set has none, and neither has the Talos installer image, which the imager builds and `skopeo` pushes, nor the third-party images `packages/` vendors by digest. diff --git a/docs/release/splitting-bundle-prs.md b/docs/release/splitting-bundle-prs.md new file mode 100644 index 000000000..ace989a88 --- /dev/null +++ b/docs/release/splitting-bundle-prs.md @@ -0,0 +1,10 @@ +## Splitting a release-blocking bundle PR + +Sometimes the work that has to land before a release is a 40-commit grab bag (CI stabilization, dependency bumps, chart fixes). Splitting it makes it reviewable and survivable. The strategy that has worked: + +1. **Verify what's already on `main`.** Use `git merge-base --is-ancestor` and subject grep to drop commits that have already landed. +2. **Find existing open PRs that cover the same commits.** Force-push to the existing branch instead of opening a duplicate — preserves review threads. +3. **Identify the "bottleneck" PR.** The one whose changes make previously-silent misconfigurations into hard errors (typically an operator-version bump). Everything else can fan out in parallel. +4. **Squash iterative same-author bug-on-bug fixes** inside each split. Preserve authorship via `--author` on the squashed commit. +5. **Mark the keystone PR as draft** and link in the body to the dependency PRs. Reviewers know it intentionally fails CI until the chain lands. +6. **Beware of concurrent registry pushes.** When fanning out 5+ PR CI runs simultaneously, multiple `make image` jobs can collide on shared floating tags. The `IMAGE_TAG=pr--` refactor solved this; if you see `Conflicted with another upload of the same manifest`, `gh run rerun --failed`. diff --git a/docs/release/stable-tags.md b/docs/release/stable-tags.md new file mode 100644 index 000000000..af908d046 --- /dev/null +++ b/docs/release/stable-tags.md @@ -0,0 +1,7 @@ +## Stable tags come from rc promotion + +A stable `vX.Y.Z` is created only by **promoting an existing release-candidate** that already passed e2e — never by rebuilding and never by a cron. + +- [`promote-rc.yaml`](../../.github/workflows/promote-rc.yaml) is triggered manually once an rc has gone green. It rewrites the rc version substring in the vendored image tags (the container digests stay the rc's), commits and publishes the resulting declarative packages tree under a unique run-specific `promotion-*` tag, commits that digest pin, and opens the `release-X.Y.Z` staging PR. The PR gate verifies the candidate against the prospective merge and original rc artifact; finalize repeats that check before creating the write-once stable tag at the merge commit, then retags the verified packages artifact and rc's already-built, e2e-passed container digests to the stable tag **by digest** (no container rebuild — see [`hack/promote-retag.sh`](../../hack/promote-retag.sh)) and publishes the release. An abandoned promotion leaves no stable-named bytes; its temporary candidate is retention-managed. + +So a commit on a supported `release-X.Y` line ships when a maintainer promotes the next rc for that line — not automatically within 24h. Pushing a stable `vX.Y.Z` tag by hand is **not** a supported path: `tags.yaml` fails fast on a stable tag that has no pre-existing draft ("stable tags come from promote-rc.yaml"), and even so the finalize step would refuse to move a pre-existing tag. Pre-releases (`vX.Y.Z-rc.N`, `-alpha.N`, `-beta.N`) are cut with the [`Cut Pre-release Tag`](../../.github/workflows/cut-prerelease.yaml) workflow, which pushes the tag as the CI app; [`tags.yaml`](../../.github/workflows/tags.yaml) then fires on that push, builds it, and publishes the pre-release. Cutting via the workflow (rather than a manual `git push`) is what lets repo admins lock `v*` tag creation to the CI app — see the tag-protection note below. diff --git a/docs/release/tag-immutability.md b/docs/release/tag-immutability.md new file mode 100644 index 000000000..fe7418279 --- /dev/null +++ b/docs/release/tag-immutability.md @@ -0,0 +1,18 @@ +## Tag immutability + +Published tags are **write-once** — once a `vX.Y.Z` or rc tag is pushed it is never moved or deleted. (Nightlies are not git tags at all — they are GHCR OCI tags, see [Nightly builds](./nightly-builds.md#nightly-builds).) Moving a tag silently poisons the Go module proxy / pkg.go.dev cache for `api/apps/v1alpha1/vX.Y.Z` and drifts SBOM/provenance toolchains, so the release flow is built so a move is impossible by construction: + +| File | Tag / branch handling | +|------|------------------------| +| [`pull-requests-release.yaml`](../../.github/workflows/pull-requests-release.yaml) | Verifies the pinned packages candidate before creating a stable name. Creates `vX.Y.Z` at the PR merge commit **write-once** (create if absent, no-op if unchanged, fail loudly if it would move — the merge commit is new, so there is nothing to move). Cuts the `api/apps/v1alpha1/` Go-submodule tag write-once at the same commit, **stable only** (never rc/beta/alpha). The `release-X.Y` maintenance branch is fast-forward-only. Retags the verified packages artifact and rc container digests to the stable image tag (also write-once at the image level). | +| [`tags.yaml`](../../.github/workflows/tags.yaml) | The `release-X.Y.Z-rc.N` staging branch is a mutable staging ref (compare-before-force: skipped when unchanged, force+log only when it genuinely moves). Fails fast on a stable `vX.Y.Z` tag with no pre-existing draft — stable tags come from `promote-rc.yaml`, never a hand push. | +| [`promote-rc.yaml`](../../.github/workflows/promote-rc.yaml) | Mutates no stable tags. Before staging, verifies green full E2E from the latest rc tag-push run or a tag-correlated manual published-tag run (unless loudly bypassed). Commits and publishes the rewritten packages tree under a run-specific temporary `promotion-*` tag, adds a second commit pinning its digest on the `release-X.Y.Z` branch, generates the changelog there, and opens the promote PR with `release` but no automatic `full-e2e`; the PR gate verifies the candidate and finalize repeats that verification before the stable tag and image retag. The staging branch is force-pushed on re-dispatch, but an existing `docs/changelogs/vX.Y.Z.md` is preserved across the rebuild. | + +This is the immutable-tag + rc-promotion model from [#2677](https://github.com/cozystack/cozystack/issues/2677): stable `vX.Y.Z` is created only by promoting an existing rc, rc tags are write-once, and `api/apps/v1alpha1/vX.Y.Z` is created only on a stable release. The old nightly `auto-release.yaml` (which delete-recreated auto-bumped patch tags) has been removed. rc and stable tags accrete permanently; the only churning artifacts are GHCR nightlies and abandoned temporary promotion candidates, which [`retention.yaml`](../../.github/workflows/retention.yaml) prunes (see [Nightly builds](./nightly-builds.md#nightly-builds)) — GHCR storage is the cost vector to watch. + +### Repo-side enforcement (tag-protection rulesets) + +The guarantees above are enforced in the workflows. Repo admins back them with two GitHub **tag rulesets** targeting `v*`: + +- **Immutability** — *Restrict updates* + *Restrict deletions*, **no bypass**. No actor (not even the CI app) can move or delete a shipped `v*` tag. Safe to enable at any time: nothing in the release flow ever moves or deletes a `v*` tag, it only creates new ones. +- **Creation control** — *Restrict creations* with the **CI app as the sole bypass actor**, plus *Limit branches/tags updated in a single push* = 1. This turns "don't hand-push tags" from convention into hard enforcement: every `v*` tag — pre-release **and** stable — can then be created only by CI. Enable this **only after** [`cut-prerelease.yaml`](../../.github/workflows/cut-prerelease.yaml) is in use, since that workflow is what makes CI the sole creator of pre-release tags; enabling it earlier would lock maintainers out of cutting rc/alpha/beta. (The Go-module tag `api/apps/v1alpha1/*` gets an equivalent pair of rulesets.) diff --git a/hack/ci-tool-installs-contract.bats b/hack/ci-tool-installs-contract.bats index 009911249..739f72394 100755 --- a/hack/ci-tool-installs-contract.bats +++ b/hack/ci-tool-installs-contract.bats @@ -111,8 +111,9 @@ one_value_of() { @test "every use of the cosign installer is SHA-pinned at one cosign release" { # Without cosign-release the action installs its own default, which moves # with a digest bump of the action. The release jobs verify their signature - # with the cosign docs/release.md asks users for, so the version is named in - # each step, and all steps agree on it and on the action's commit. + # with the cosign docs/release/signatures-and-sbom.md asks users for, so the + # version is named in each step, and all steps agree on it and on the + # action's commit. uses="$(workflow_code 'uses: sigstore/cosign-installer@')" [ -n "$uses" ] bad="$(printf '%s\n' "$uses" | grep --invert-match --extended-regexp 'installer@[0-9a-f]{40} # v[0-9.]+$' || true)" diff --git a/hack/release-changelog-contract.bats b/hack/release-changelog-contract.bats index 44d0925e9..5938b77f2 100644 --- a/hack/release-changelog-contract.bats +++ b/hack/release-changelog-contract.bats @@ -570,10 +570,11 @@ code_lines() { grep -vE '^[[:space:]]*#'; } exit 1 } - # changelog.md sends the agent to run the docs/release.md checks and the - # validator on its own output, by their working-tree paths. Without these two - # copies it runs the rc tag's versions, which predate every check added since. - for f in docs/release.md hack/validate-changelog.sh; do + # changelog.md sends the agent to run the docs/release/changelogs.md checks + # and the validator on its own output, by their working-tree paths. Without + # these two copies it runs the rc tag's versions, which predate every check + # added since. + for f in docs/release/changelogs.md hack/validate-changelog.sh; do printf '%s\n' "$overlay" | code_lines | grep -qF "cp .release-tooling/$f $f" || { echo "The refresh step does not overlay $f from the tooling checkout, so the" >&2 echo "agent following docs/agents/changelog.md section 9 runs the rc tag's copy." >&2 @@ -581,23 +582,26 @@ code_lines() { grep -vE '^[[:space:]]*#'; } } done - # docs/release.md sits under neither sparse-checkout path. It reaches - # .release-tooling only because cone mode also checks out the files of every - # parent directory of a listed one; without it the copy above fails the job. + # An rc tagged before the release docs were split has no docs/release/ + # directory, and cp does not create one. + printf '%s\n' "$overlay" | code_lines | grep -qE '^[[:space:]]+mkdir -p docs/release$' || { + echo "The refresh step does not create docs/release before copying into it, so" >&2 + echo "the copy fails on every rc tagged before the release docs were split." >&2 + exit 1 + } + + # The tooling checkout has to list both directories the copies read from. tooling="$(printf '%s\n' "$block" | awk ' /^ - name: Checkout release tooling from the dispatch ref$/ { inside = 1; next } /^ - name: / { inside = 0 } inside' | code_lines)" - printf '%s\n' "$tooling" | grep -qE '^[[:space:]]+docs/agents$' || { - echo "The tooling checkout no longer lists docs/agents, which is what carries" >&2 - echo "docs/release.md into .release-tooling." >&2 - exit 1 - } - if printf '%s\n' "$tooling" | grep -qE 'sparse-checkout-cone-mode:[[:space:]]*false'; then - echo "The tooling checkout turned cone mode off, which drops docs/release.md" >&2 - echo "from .release-tooling and fails the refresh step's copy." >&2 - exit 1 - fi + for d in docs/agents docs/release; do + printf '%s\n' "$tooling" | grep -qE "^[[:space:]]+$d\$" || { + echo "The tooling checkout no longer lists $d, so .release-tooling lacks the" >&2 + echo "file the refresh step copies from it." >&2 + exit 1 + } + done # Every job that runs release tooling against an older checkout must source it # from the dispatch ref. A script fetched by a script that is itself missing diff --git a/hack/sign-release-assets.bats b/hack/sign-release-assets.bats index 9cfd25aca..0dfa06763 100644 --- a/hack/sign-release-assets.bats +++ b/hack/sign-release-assets.bats @@ -161,13 +161,13 @@ cleanup() { cleanup } -@test "docs/release.md tells users to verify the identity the script verifies" { +@test "docs/release/signatures-and-sbom.md tells users to verify the identity the script verifies" { regexp="$(sed -n "s/^IDENTITY_REGEXP='\(.*\)'\$/\1/p" "$SCRIPT")" if [ -z "$regexp" ]; then echo "IDENTITY_REGEXP not found in $SCRIPT" >&2 exit 1 fi - grep -qF -- "--certificate-identity-regexp '$regexp'" "$REPO_ROOT/docs/release.md" + grep -qF -- "--certificate-identity-regexp '$regexp'" "$REPO_ROOT/docs/release/signatures-and-sbom.md" } @test "fails when the signature does not verify" { diff --git a/hack/sign-release-assets.sh b/hack/sign-release-assets.sh index e8120feed..6549ee4f9 100755 --- a/hack/sign-release-assets.sh +++ b/hack/sign-release-assets.sh @@ -26,8 +26,9 @@ # copies along with its other assets, and a stable release gets its own. # # The bundle is verified before the script returns, with the same certificate -# identity users are told to check in docs/release.md, so a release whose -# signature that check would reject fails here instead of being uploaded. +# identity users are told to check in docs/release/signatures-and-sbom.md, so a +# release whose signature that check would reject fails here instead of being +# uploaded. # # Needs syft and cosign on PATH, and `id-token: write` in the job. set -euo pipefail @@ -39,7 +40,7 @@ BUNDLE=cozystack-checksums.txt.sigstore.json # The workflows that publish releases, at the refs they run from: tags.yaml for a # pre-release, on a v* tag push, and promote-rc.yaml for a stable one, dispatched # from main only. A run from any other ref signs, but fails the check below. -# docs/release.md shows users the same pattern. +# docs/release/signatures-and-sbom.md shows users the same pattern. IDENTITY_REGEXP='^https://github\.com/cozystack/cozystack/\.github/workflows/(tags\.yaml@refs/tags/v|promote-rc\.yaml@refs/heads/main$)' OIDC_ISSUER=https://token.actions.githubusercontent.com From a042f335900b0a2340d27d3a235bb7ea2748c0ec Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 15:49:35 +0000 Subject: [PATCH 104/518] fix(mariadb): record the dump a Backup was taken as Every run of a MariaDB application writes its dump under the same storage prefix, and the Backup did not say which of those dumps is its own. A restore had nothing to give mariadb-operator but the operator Backup, which resolves to the newest dump under the prefix. The Backup now records the dump's file name and the timestamp in that name in driverMetadata, taken from the match the size probe already makes. Nothing is recorded when that match cannot prove the dump belongs to the operator Backup, so a Backup never names another run's dump. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- .../backupcontroller/mariadb_artifact_size.go | 16 ++-- .../mariadb_artifact_size_test.go | 95 ++++++++++++++++--- .../mariadbstrategy_controller.go | 14 ++- 3 files changed, 104 insertions(+), 21 deletions(-) diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index 6397325ca..950303d78 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -39,19 +39,19 @@ type mariadbDump struct { LastModified time.Time } -// mariadbArtifactSizeBytes measures the dump mariadb-operator uploaded for -// mdbBackup. Returns 0 - leaving Artifact.SizeBytes absent - when the dump -// cannot be found or told apart: an unknown size must not be reported as a -// measured one. -func (r *BackupJobReconciler) mariadbArtifactSizeBytes(ctx context.Context, mdbBackup *mariadbtypes.Backup, completedAt time.Time) int64 { +// attributedMariaDBDump returns the dump mariadb-operator uploaded for +// mdbBackup. ok is false when the dump cannot be found or told apart, and +// the Backup then reports neither a size nor a dump: an unknown dump must +// not be reported as a measured one. +func (r *BackupJobReconciler) attributedMariaDBDump(ctx context.Context, mdbBackup *mariadbtypes.Backup, completedAt time.Time) (dump mariadbDump, ok bool) { ctx, cancel := context.WithTimeout(ctx, artifactSizeProbeTimeout) defer cancel() dump, err := findMariaDBDump(ctx, r.objectStore(), mdbBackup, completedAt) if err != nil { - getLogger(ctx).Info("MariaDB backup size not measured; leaving it unset", "backup", mdbBackup.Name, "error", err) - return 0 + getLogger(ctx).Info("MariaDB dump not attributed; leaving its size and name unset", "backup", mdbBackup.Name, "error", err) + return mariadbDump{}, false } - return dump.Size + return dump, true } // findMariaDBDump lists the bucket mdbBackup wrote to and returns the one diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index 9d649c15c..80e9e4617 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -29,6 +29,14 @@ func dumpKey(prefix string, ts time.Time, compression string) string { return fmt.Sprintf("%sbackup.%s.%s.sql", prefix, ts.UTC().Format(time.RFC3339), compression) } +// probedSize is the size a Backup reports for a probe result. +func probedSize(dump mariadbDump, attributed bool) int64 { + if !attributed { + return 0 + } + return dump.Size +} + // mariadbBackupSucceeded is the Complete condition mariadb-operator sets on a // Backup whose Job succeeded. A failed Job also gets Status=True, with // Reason=JobFailed, so a fixture without the reason is not a success. @@ -105,7 +113,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4242, LastModified: created.Add(time.Minute)}, {Key: prefix + "0-backup-target.txt", Size: 7}, }} - got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed) + got := probedSize(reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).attributedMariaDBDump(context.Background(), mdbBackup, completed)) if got != 4242 { t.Errorf("size = %d, want 4242", got) } @@ -130,7 +138,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { } l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 3, LastModified: created.Add(time.Minute)}}} r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds"), ca) - if got := r.mariadbArtifactSizeBytes(context.Background(), operatorBackupStoringTo(t, tlsS3, created), completed); got != 3 { + if got := probedSize(r.attributedMariaDBDump(context.Background(), operatorBackupStoringTo(t, tlsS3, created), completed)); got != 3 { t.Errorf("size = %d, want 3", got) } if !l.conn.Secure || string(l.conn.CAPEM) != "PEM" { @@ -148,7 +156,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { } l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 4, LastModified: created.Add(time.Minute)}}} r := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds"), sts) - if got := r.mariadbArtifactSizeBytes(context.Background(), operatorBackupStoringTo(t, stsS3, created), completed); got != 4 { + if got := probedSize(r.attributedMariaDBDump(context.Background(), operatorBackupStoringTo(t, stsS3, created), completed)); got != 4 { t.Errorf("size = %d, want 4", got) } if l.conn.SessionToken != "TOKEN" { @@ -160,7 +168,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey(prefix, created.Add(-30*time.Second), ""), Size: 9, LastModified: created.Add(time.Minute)}, }} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 9 { + if got := probedSize(reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).attributedMariaDBDump(context.Background(), mdbBackup, completed)); got != 9 { t.Errorf("size = %d, want 9", got) } }) @@ -173,7 +181,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { {Key: dumpKey(prefix, created.Add(-4*time.Minute), "gzip"), Size: 7, LastModified: created.Add(time.Minute)}, {Key: dumpKey(prefix, created.Add(30*time.Second), "gzip"), Size: 666, LastModified: created.Add(-4 * time.Minute)}, }} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { + if got := probedSize(reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).attributedMariaDBDump(context.Background(), mdbBackup, completed)); got != 0 { t.Errorf("size = %d, want 0: no dump is inside the window on both clocks", got) } }) @@ -182,7 +190,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey(prefix, completed.Add(30*time.Second), "gzip"), Size: 5, LastModified: completed.Add(30 * time.Second)}, }} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 5 { + if got := probedSize(reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).attributedMariaDBDump(context.Background(), mdbBackup, completed)); got != 5 { t.Errorf("size = %d, want 5", got) } }) @@ -191,7 +199,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { r := reconcilerWithLister(t, &fakeS3Lister{}, s3CredsSecret("tenant", "creds")) r.newS3ObjectLister = func(s3Connection) (s3ObjectLister, error) { return blockingLister{}, nil } withShortProbeTimeout(t, func() { - if got := r.mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { + if got := probedSize(r.attributedMariaDBDump(context.Background(), mdbBackup, completed)); got != 0 { t.Errorf("size = %d, want 0", got) } }) @@ -202,7 +210,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { {Key: dumpKey(prefix, created.Add(time.Minute), "gzip"), Size: 1, LastModified: created.Add(time.Minute)}, {Key: dumpKey(prefix, created.Add(2*time.Minute), "gzip"), Size: 2, LastModified: created.Add(2 * time.Minute)}, }} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { + if got := probedSize(reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).attributedMariaDBDump(context.Background(), mdbBackup, completed)); got != 0 { t.Errorf("size = %d, want 0 (ambiguous)", got) } }) @@ -211,14 +219,14 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { l := &fakeS3Lister{objects: []s3ObjectInfo{ {Key: dumpKey(prefix, completed.Add(time.Hour), "gzip"), Size: 1, LastModified: completed.Add(time.Hour)}, }} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { + if got := probedSize(reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).attributedMariaDBDump(context.Background(), mdbBackup, completed)); got != 0 { t.Errorf("size = %d, want 0", got) } }) t.Run("list error leaves size unset", func(t *testing.T) { l := &fakeS3Lister{err: errors.New("AccessDenied")} - if got := reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { + if got := probedSize(reconcilerWithLister(t, l, s3CredsSecret("tenant", "creds")).attributedMariaDBDump(context.Background(), mdbBackup, completed)); got != 0 { t.Errorf("size = %d, want 0", got) } }) @@ -226,7 +234,7 @@ func TestMariaDBArtifactSizeBytes(t *testing.T) { t.Run("missing credentials Secret leaves size unset", func(t *testing.T) { c := newMariaDBStrategyTestClient(t) r := &BackupJobReconciler{Client: c, Scheme: c.Scheme()} - if got := r.mariadbArtifactSizeBytes(context.Background(), mdbBackup, completed); got != 0 { + if got := probedSize(r.attributedMariaDBDump(context.Background(), mdbBackup, completed)); got != 0 { t.Errorf("size = %d, want 0", got) } }) @@ -481,7 +489,7 @@ func TestMariaDBArtifactSizeBytes_MinioClient(t *testing.T) { s3.Prefix = "tenant/db/" s3.Region = "us-east-1" - if got := r.mariadbArtifactSizeBytes(context.Background(), operatorBackupStoringTo(t, s3, created), created.Add(3*time.Minute)); got != 31337 { + if got := probedSize(r.attributedMariaDBDump(context.Background(), operatorBackupStoringTo(t, s3, created), created.Add(3*time.Minute))); got != 31337 { t.Fatalf("size = %d, want 31337", got) } if gotPath != "/bkt/" || gotPrefix != "tenant/db/" || gotDelimiter != "/" { @@ -523,6 +531,69 @@ func TestCreateMariaDBBackupArtifact_SizeBytes(t *testing.T) { } } +// TestCreateMariaDBBackupArtifact_RecordsDump pins that the Backup names +// its dump only when the probe attributed one to the operator Backup. +func TestCreateMariaDBBackupArtifact_RecordsDump(t *testing.T) { + apps := mariadbapp.GroupName + created := time.Date(2026, 10, 21, 10, 0, 0, 0, time.UTC) + completed := metav1.NewTime(created.Add(3 * time.Minute)) + rendered := newRenderedMariaDBTemplate() + rendered.Storage.S3.Prefix = "src/" + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "mariadb-strategy-default"}, + Parameters: map[string]string{}, + } + own := s3ObjectInfo{Key: "src/backup.2026-10-21T10:01:00Z.gzip.sql", Size: 2048, LastModified: created.Add(2 * time.Minute)} + second := s3ObjectInfo{Key: "src/backup.2026-10-21T10:02:00Z.gzip.sql", Size: 1, LastModified: created.Add(2 * time.Minute)} + offset := s3ObjectInfo{Key: "src/backup.2026-10-21T14:01:00+04:00.sql", Size: 7, LastModified: created.Add(2 * time.Minute)} + + cases := []struct { + name string + lister *fakeS3Lister + wantFile, wantTime string + }{ + {name: "an attributed dump is recorded", lister: &fakeS3Lister{objects: []s3ObjectInfo{own}}, + wantFile: "backup.2026-10-21T10:01:00Z.gzip.sql", wantTime: "2026-10-21T10:01:00Z"}, + {name: "a dump named with an offset is timed in UTC", lister: &fakeS3Lister{objects: []s3ObjectInfo{offset}}, + wantFile: "backup.2026-10-21T14:01:00+04:00.sql", wantTime: "2026-10-21T10:01:00Z"}, + {name: "two dumps in the lifetime record none", lister: &fakeS3Lister{objects: []s3ObjectInfo{own, second}}}, + {name: "no dump in the lifetime records none", lister: &fakeS3Lister{}}, + {name: "a failed listing records none", lister: &fakeS3Lister{err: errors.New("AccessDenied")}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + mdbBackup := operatorBackupStoringTo(t, rendered.Storage.S3, created) + mdbBackup.Status.Conditions = []metav1.Condition{mariadbBackupSucceeded(completed)} + job := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-dump"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, + }, + } + r := reconcilerWithLister(t, tc.lister, job, s3CredsSecret("tenant", "creds")) + if _, err := r.createMariaDBBackupArtifact(context.Background(), job, resolved, mdbBackup, rendered); err != nil { + t.Fatalf("createMariaDBBackupArtifact: %v", err) + } + persisted := &backupsv1alpha1.Backup{} + if err := r.Get(context.Background(), client.ObjectKey{Namespace: "tenant", Name: "bj-dump"}, persisted); err != nil { + t.Fatalf("get Backup: %v", err) + } + md := persisted.Spec.DriverMetadata + file, hasFile := md[mariadbDumpFileKey] + at, hasTime := md[mariadbDumpTimeKey] + if tc.wantFile == "" { + if hasFile || hasTime { + t.Errorf("driverMetadata = %v; want no dump recorded", md) + } + return + } + if file != tc.wantFile || at != tc.wantTime { + t.Errorf("recorded dump %q at %q; want %q at %q", file, at, tc.wantFile, tc.wantTime) + } + }) + } +} + // TestCreateMariaDBBackupArtifact_StrategyEditedMidBackup pins that the // probe reads where the operator Backup wrote its dump, not where the // strategy, edited while the backup ran, now points. diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 4d88ab8a1..50c871db3 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -47,6 +47,14 @@ const ( mariadbBackupNamespaceKey = "k8s.mariadb.com/backup-namespace" mariadbStorageURIKey = "k8s.mariadb.com/storage-uri" + // The dump the operator Backup wrote: its file name under the storage + // prefix, and the RFC3339 timestamp in that name. Every run of an + // application writes under the same prefix, so only this name tells its + // dump from the others. Recorded only when findMariaDBDump proved the + // dump belongs to the operator Backup. + mariadbDumpFileKey = "k8s.mariadb.com/dump-file" + mariadbDumpTimeKey = "k8s.mariadb.com/dump-time" + // Polling cadence for the operator Backup/Restore lifecycle. mariadbPollInterval = 5 * time.Second @@ -398,7 +406,11 @@ func (r *BackupJobReconciler) createMariaDBBackupArtifact( if uri := driverMD[mariadbStorageURIKey]; uri != "" { status.Artifact = &backupsv1alpha1.BackupArtifact{URI: uri} if probe { - status.Artifact.SizeBytes = r.mariadbArtifactSizeBytes(ctx, mdbBackup, takenAt.Time) + if dump, ok := r.attributedMariaDBDump(ctx, mdbBackup, takenAt.Time); ok { + status.Artifact.SizeBytes = dump.Size + driverMD[mariadbDumpFileKey] = dump.Name + driverMD[mariadbDumpTimeKey] = dump.Timestamp.UTC().Format(time.RFC3339) + } } } backup := &backupsv1alpha1.Backup{ From 21d1e9f02b4dafeefab3acece372cc8f077b7701 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 15:53:38 +0000 Subject: [PATCH 105/518] fix(mariadb): restore the dump a Backup records A RestoreJob created a mariadb-operator Restore that named only the operator Backup. The operator then lists the prefix that Backup wrote to and restores the dump nearest to the moment the restore started: the application's newest dump, not the one the Backup was taken as. When the Backup records its dump, the Restore now carries the timestamp in that dump's name as targetRecoveryTime. mariadb-operator returns the dump whose timestamp equals the target before it looks for the nearest one, and the nearest one is exactly what it would restore if the recorded dump were gone. So the controller first checks, with its own bounded call, that the dump is still in the operator Backup's storage. A dump that is gone fails the RestoreJob, naming the dump, instead of restoring a neighbour in its place. A check that cannot be made keeps the RestoreJob waiting with Ready=False until the restore deadline, then fails it with the cause: the Backup could record its dump only because the controller reached that storage when it was taken, so an unreachable storage is not a reason to guess. The dump is chosen once, as the operator Restore is created; a Restore that already exists is followed as before. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- internal/backupcontroller/artifact_size.go | 4 + .../backupcontroller/mariadb_artifact_size.go | 52 ++- .../mariadb_restore_dump_test.go | 304 ++++++++++++++++++ .../mariadbstrategy_controller.go | 111 ++++++- .../mariadbstrategy_controller_test.go | 19 +- .../backupcontroller/restorejob_controller.go | 5 + 6 files changed, 467 insertions(+), 28 deletions(-) create mode 100644 internal/backupcontroller/mariadb_restore_dump_test.go diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 941b00323..9197f07be 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -90,6 +90,10 @@ func (r *BackupJobReconciler) objectStore() objectStore { return objectStore{reader: r.Client, newLister: r.newS3ObjectLister} } +func (r *RestoreJobReconciler) objectStore() objectStore { + return objectStore{reader: r.Client, newLister: r.newS3ObjectLister} +} + func (r *BackupJobReconciler) s3Lister(conn s3Connection) (s3ObjectLister, error) { return r.objectStore().lister(conn) } diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index 950303d78..a61ec4a43 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -60,22 +60,14 @@ func (r *BackupJobReconciler) attributedMariaDBDump(ctx context.Context, mdbBack // from can have been edited since, and pointing at another bucket now says // nothing about where this dump went. func findMariaDBDump(ctx context.Context, store objectStore, mdbBackup *mariadbtypes.Backup, completedAt time.Time) (mariadbDump, error) { - s3 := mdbBackup.Spec.Storage.S3 - if s3 == nil { - return mariadbDump{}, errors.New("the operator Backup does not store to S3") + lister, s3, err := mariadbStorage(ctx, store, mdbBackup) + if err != nil { + return mariadbDump{}, err } others, err := finishedMariaDBRuns(ctx, store.reader, mdbBackup) if err != nil { return mariadbDump{}, fmt.Errorf("list operator Backups: %w", err) } - conn, err := mariadbS3Connection(ctx, store, mdbBackup.Namespace, s3) - if err != nil { - return mariadbDump{}, fmt.Errorf("resolve S3 credentials: %w", err) - } - lister, err := store.lister(conn) - if err != nil { - return mariadbDump{}, fmt.Errorf("build S3 client: %w", err) - } // Non-recursive, as mariadb-operator lists it: the default prefix is also // the default CNPG destinationPath, so an app of each kind with the same // name shares it and a recursive listing would walk the WAL archive. @@ -88,6 +80,44 @@ func findMariaDBDump(ctx context.Context, store objectStore, mdbBackup *mariadbt return matchMariaDBDump(objects, prefix, own, others) } +// mariadbDumpExists reports whether the dump file name is still in the +// storage mdbBackup wrote to. +func mariadbDumpExists(ctx context.Context, store objectStore, mdbBackup *mariadbtypes.Backup, name string) (bool, error) { + lister, s3, err := mariadbStorage(ctx, store, mdbBackup) + if err != nil { + return false, err + } + key := mariadbOperatorS3Prefix(s3.Prefix) + name + objects, err := lister.ListObjects(ctx, s3.Bucket, key, false) + if err != nil { + return false, fmt.Errorf("list s3://%s/%s: %w", s3.Bucket, key, err) + } + for _, obj := range objects { + if obj.Key == key { + return true, nil + } + } + return false, nil +} + +// mariadbStorage opens the bucket mdbBackup wrote to, with the credentials +// its own spec names. +func mariadbStorage(ctx context.Context, store objectStore, mdbBackup *mariadbtypes.Backup) (s3ObjectLister, *mariadbtypes.S3Storage, error) { + s3 := mdbBackup.Spec.Storage.S3 + if s3 == nil { + return nil, nil, errors.New("the operator Backup does not store to S3") + } + conn, err := mariadbS3Connection(ctx, store, mdbBackup.Namespace, s3) + if err != nil { + return nil, nil, fmt.Errorf("resolve S3 credentials: %w", err) + } + lister, err := store.lister(conn) + if err != nil { + return nil, nil, fmt.Errorf("build S3 client: %w", err) + } + return lister, s3, nil +} + // mariadbRun is the span in which an operator Backup can have written its // dump: its lifetime, widened by mariadbSizeProbeClockSkew on each side. type mariadbRun struct { diff --git a/internal/backupcontroller/mariadb_restore_dump_test.go b/internal/backupcontroller/mariadb_restore_dump_test.go new file mode 100644 index 000000000..0375def33 --- /dev/null +++ b/internal/backupcontroller/mariadb_restore_dump_test.go @@ -0,0 +1,304 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "errors" + "strings" + "testing" + "time" + + corev1 "k8s.io/api/core/v1" + apimeta "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/tools/record" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" + "github.com/cozystack/cozystack/internal/backupcontroller/mariadbapp" + "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" +) + +// The source run of the restore fixture: operator Backup op-bk of +// application src, alive from mdbRestoreT0 to mdbRestoreT0+40s, writing +// under tenant/src/ with mdbRestoreDump as its dump. +var mdbRestoreT0 = time.Date(2026, 10, 21, 10, 0, 0, 0, time.UTC) + +const ( + mdbRestorePrefix = "tenant/src/" + mdbRestoreDump = "backup.2026-10-21T10:00:05Z.gzip.sql" + mdbRestoreTime = "2026-10-21T10:00:05Z" + mdbNewerDump = "backup.2026-10-21T11:00:05Z.gzip.sql" +) + +type mariadbRestoreFixture struct { + c client.Client + r *RestoreJobReconciler + rj *backupsv1alpha1.RestoreJob + backup *backupsv1alpha1.Backup + src *mariadbtypes.Backup + lister *fakeS3Lister + events *record.FakeRecorder +} + +// newMariaDBRestoreFixture seeds a RestoreJob of Backup src-bk, whose +// driverMetadata is driverMD plus the operator Backup's name, into the +// existing application target. extra objects are seeded too. +func newMariaDBRestoreFixture(t *testing.T, driverMD map[string]string, lister *fakeS3Lister, extra ...client.Object) *mariadbRestoreFixture { + t.Helper() + apps := mariadbapp.GroupName + s3 := newRenderedMariaDBTemplate().Storage.S3 + s3.Prefix = "tenant/src" + src := operatorBackupStoringTo(t, s3, mdbRestoreT0) + src.Status.Conditions = []metav1.Condition{mariadbBackupSucceeded(metav1.NewTime(mdbRestoreT0.Add(40 * time.Second)))} + + md := map[string]string{mariadbBackupNameKey: src.Name} + for k, v := range driverMD { + md[k] = v + } + backup := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "src-bk"}, + Spec: backupsv1alpha1.BackupSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, + TakenAt: metav1.NewTime(mdbRestoreT0.Add(40 * time.Second)), + DriverMetadata: md, + }, + } + started := metav1.Now() + rj := &backupsv1alpha1.RestoreJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "rj"}, + Spec: backupsv1alpha1.RestoreJobSpec{ + BackupRef: corev1.LocalObjectReference{Name: backup.Name}, + TargetApplicationRef: &corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "target", APIGroup: &apps}, + }, + Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &started, Phase: backupsv1alpha1.RestoreJobPhaseRunning}, + } + target := &mariadbtypes.MariaDB{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: mariadbNameForApp("target")}} + objs := append([]client.Object{src, backup, rj, target}, extra...) + c := newMariaDBStrategyTestClient(t, objs...) + fetched := &backupsv1alpha1.RestoreJob{} + if err := c.Get(context.Background(), client.ObjectKeyFromObject(rj), fetched); err != nil { + t.Fatalf("get RestoreJob: %v", err) + } + events := record.NewFakeRecorder(10) + return &mariadbRestoreFixture{ + c: c, + rj: fetched, + backup: backup, + src: src, + lister: lister, + events: events, + r: &RestoreJobReconciler{ + Client: c, + Recorder: events, + newS3ObjectLister: func(conn s3Connection) (s3ObjectLister, error) { + lister.conn = conn + return lister, nil + }, + }, + } +} + +func (f *mariadbRestoreFixture) reconcile(t *testing.T) ctrl.Result { + t.Helper() + res, err := f.r.reconcileMariaDBRestore(context.Background(), f.rj, f.backup) + if err != nil { + t.Fatalf("reconcileMariaDBRestore: %v", err) + } + return res +} + +func (f *mariadbRestoreFixture) restores(t *testing.T) []mariadbtypes.Restore { + t.Helper() + list := &mariadbtypes.RestoreList{} + if err := f.c.List(context.Background(), list, client.InNamespace("tenant")); err != nil { + t.Fatalf("list operator Restores: %v", err) + } + return list.Items +} + +// wantFailed asserts the RestoreJob failed with a message holding every +// fragment, and that no operator Restore was created. +func (f *mariadbRestoreFixture) wantFailed(t *testing.T, fragments ...string) { + t.Helper() + if f.rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("phase = %q, want Failed", f.rj.Status.Phase) + } + for _, s := range fragments { + if !strings.Contains(f.rj.Status.Message, s) { + t.Errorf("message %q does not contain %q", f.rj.Status.Message, s) + } + } + if got := f.restores(t); len(got) != 0 { + t.Errorf("created %d operator Restores; want none", len(got)) + } +} + +// wantWaiting asserts the RestoreJob is kept waiting on the dump check +// with Ready=False saying why, and that no operator Restore was created. +func (f *mariadbRestoreFixture) wantWaiting(t *testing.T, res ctrl.Result, fragments ...string) { + t.Helper() + if f.rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("RestoreJob failed: %s", f.rj.Status.Message) + } + if res.RequeueAfter != mariadbDumpCheckRetryInterval { + t.Errorf("RequeueAfter = %s, want %s", res.RequeueAfter, mariadbDumpCheckRetryInterval) + } + persisted := &backupsv1alpha1.RestoreJob{} + if err := f.c.Get(context.Background(), client.ObjectKeyFromObject(f.rj), persisted); err != nil { + t.Fatalf("get RestoreJob: %v", err) + } + cond := apimeta.FindStatusCondition(persisted.Status.Conditions, "Ready") + if cond == nil || cond.Status != metav1.ConditionFalse || cond.Reason != "DumpCheckPending" { + t.Fatalf("Ready = %+v, want False/DumpCheckPending", cond) + } + for _, s := range fragments { + if !strings.Contains(cond.Message, s) { + t.Errorf("Ready message %q does not contain %q", cond.Message, s) + } + } + if got := f.restores(t); len(got) != 0 { + t.Errorf("created %d operator Restores; want none", len(got)) + } +} + +// wantRestore asserts exactly one operator Restore of op-bk into target, +// asking for targetTime. +func (f *mariadbRestoreFixture) wantRestore(t *testing.T, targetTime string) { + t.Helper() + got := f.restores(t) + if len(got) != 1 { + t.Fatalf("created %d operator Restores; want one", len(got)) + } + spec := got[0].Spec + if spec.BackupRef == nil || spec.BackupRef.Name != f.src.Name || spec.MariaDBRef.Name != mariadbNameForApp("target") { + t.Errorf("Restore spec = %+v, want backupRef %s into %s", spec, f.src.Name, mariadbNameForApp("target")) + } + if spec.TargetRecoveryTime != targetTime { + t.Errorf("targetRecoveryTime = %q, want %q", spec.TargetRecoveryTime, targetTime) + } +} + +func recordedDump(name string) map[string]string { + return map[string]string{mariadbDumpFileKey: name, mariadbDumpTimeKey: mdbRestoreTime} +} + +func TestReconcileMariaDBRestore_RecordedDump(t *testing.T) { + creds := s3CredsSecret("tenant", "creds") + + t.Run("restores the recorded dump by its own timestamp", func(t *testing.T) { + l := &fakeS3Lister{objects: []s3ObjectInfo{ + {Key: mdbRestorePrefix + mdbRestoreDump, LastModified: mdbRestoreT0.Add(20 * time.Second)}, + {Key: mdbRestorePrefix + mdbNewerDump, LastModified: mdbRestoreT0.Add(time.Hour)}, + }} + f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), l, creds) + f.reconcile(t) + f.wantRestore(t, mdbRestoreTime) + if l.bucket != "bkt" || l.prefix != mdbRestorePrefix+mdbRestoreDump || l.conn.AccessKey != "AKID" { + t.Errorf("checked %s/%s as %q, want bkt/%s%s with the operator Backup's credentials", l.bucket, l.prefix, l.conn.AccessKey, mdbRestorePrefix, mdbRestoreDump) + } + }) + + t.Run("a recorded dump gone from storage fails the RestoreJob", func(t *testing.T) { + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: mdbRestorePrefix + mdbNewerDump, LastModified: mdbRestoreT0.Add(time.Hour)}}} + f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), l, creds) + f.reconcile(t) + f.wantFailed(t, mdbRestoreDump, "no longer exists in storage", "s3://bkt/"+mdbRestorePrefix) + }) + + t.Run("a storage that cannot be read keeps the RestoreJob waiting", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), &fakeS3Lister{err: errors.New("Access Denied.")}, creds) + f.wantWaiting(t, f.reconcile(t), mdbRestoreDump, "Access Denied.") + }) + + t.Run("missing credentials keep the RestoreJob waiting", func(t *testing.T) { + l := &fakeS3Lister{} + f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), l) + f.wantWaiting(t, f.reconcile(t), mdbRestoreDump, "resolve S3 credentials") + if l.calls != 0 { + t.Errorf("S3 listed %d times without credentials", l.calls) + } + }) + + t.Run("a hanging endpoint is given up at the check timeout", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), &fakeS3Lister{}, creds) + f.r.newS3ObjectLister = func(s3Connection) (s3ObjectLister, error) { return blockingLister{}, nil } + var res ctrl.Result + var err error + withShortDumpCheckTimeout(t, func() { res, err = f.r.reconcileMariaDBRestore(context.Background(), f.rj, f.backup) }) + if err != nil { + t.Fatalf("reconcileMariaDBRestore: %v", err) + } + f.wantWaiting(t, res, mdbRestoreDump, context.DeadlineExceeded.Error()) + }) + + t.Run("a storage still unreadable at the deadline fails the RestoreJob with the cause", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), &fakeS3Lister{err: errors.New("Access Denied.")}, creds) + long := metav1.NewTime(time.Now().Add(-mariadbDefaultRestoreDeadline - time.Minute)) + f.rj.Status.StartedAt = &long + f.reconcile(t) + f.wantFailed(t, mdbRestoreDump, "Access Denied.", "gave up after") + }) + + t.Run("a recorded name that is not a dump fails the RestoreJob unchecked", func(t *testing.T) { + l := &fakeS3Lister{} + f := newMariaDBRestoreFixture(t, recordedDump("dump.sql"), l, creds) + f.reconcile(t) + f.wantFailed(t, `"dump.sql"`) + if l.calls != 0 { + t.Errorf("S3 listed %d times for a name that is not a dump", l.calls) + } + }) + + t.Run("an operator Restore created earlier is followed without a check", func(t *testing.T) { + l := &fakeS3Lister{err: errors.New("Access Denied.")} + existing := &mariadbtypes.Restore{ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", Name: "rj-abcde", + Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: "rj", + backupsv1alpha1.OwningJobNamespaceLabel: "tenant", + }, + }} + f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), l, creds, existing) + res := f.reconcile(t) + if l.calls != 0 { + t.Errorf("S3 listed %d times for a RestoreJob whose operator Restore exists", l.calls) + } + if f.rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed || res.RequeueAfter != mariadbPollInterval { + t.Errorf("phase %q, requeue %s; want the existing Restore polled", f.rj.Status.Phase, res.RequeueAfter) + } + if got := f.restores(t); len(got) != 1 { + t.Errorf("%d operator Restores; want only the existing one", len(got)) + } + }) +} + +func TestReconcileMariaDBRestore_NoRecordedDump(t *testing.T) { + l := &fakeS3Lister{err: errors.New("Access Denied.")} + f := newMariaDBRestoreFixture(t, nil, l, s3CredsSecret("tenant", "creds")) + f.reconcile(t) + f.wantRestore(t, "") + if l.calls != 0 { + t.Errorf("S3 listed %d times for a Backup that records no dump", l.calls) + } +} + +// withShortDumpCheckTimeout shortens the restore's dump check timeout and +// fails the test if fn is not back well before the default would expire. +func withShortDumpCheckTimeout(t *testing.T, fn func()) { + t.Helper() + saved := mariadbRestoreDumpCheckTimeout + mariadbRestoreDumpCheckTimeout = 50 * time.Millisecond + defer func() { mariadbRestoreDumpCheckTimeout = saved }() + done := make(chan struct{}) + go func() { + fn() + close(done) + }() + select { + case <-done: + case <-time.After(2 * time.Second): + t.Fatal("the dump check did not give up at its timeout") + } +} diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 50c871db3..86b42d33f 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -4,6 +4,7 @@ package backupcontroller import ( "context" "encoding/json" + "errors" "fmt" "time" @@ -50,8 +51,9 @@ const ( // The dump the operator Backup wrote: its file name under the storage // prefix, and the RFC3339 timestamp in that name. Every run of an // application writes under the same prefix, so only this name tells its - // dump from the others. Recorded only when findMariaDBDump proved the - // dump belongs to the operator Backup. + // dump from the others, and a restore asks for exactly that dump. + // Recorded only when findMariaDBDump proved the dump belongs to the + // operator Backup. mariadbDumpFileKey = "k8s.mariadb.com/dump-file" mariadbDumpTimeKey = "k8s.mariadb.com/dump-time" @@ -68,6 +70,12 @@ const ( // to terminate. Tenants override via spec.options.restoreTimeoutSeconds. mariadbDefaultRestoreDeadline = 30 * time.Minute + // Retry cadence while the dump a restore asks for cannot be checked. + // Longer than mariadbPollInterval because each attempt can hold the + // RestoreJob controller's single worker for + // mariadbRestoreDumpCheckTimeout. + mariadbDumpCheckRetryInterval = 30 * time.Second + // mariadbBackupSnapshotKind is the Kind stamped onto the snapshot // persisted in Backup.status.underlyingResources. The same shape carries // the storage descriptor and the rendered strategy parameters used at @@ -81,6 +89,12 @@ const ( // existing API surface. var mariadbBackupSnapshotAPIVersion = backupsv1alpha1.GroupVersion.String() +// mariadbRestoreDumpCheckTimeout bounds the object-store round trips a +// restore makes before it creates the operator Restore. The RestoreJob +// controller has a single worker, so an unreachable endpoint stalls every +// other RestoreJob for this long. A variable so tests can shorten it. +var mariadbRestoreDumpCheckTimeout = 10 * time.Second + // mariadbNameForApp returns the k8s.mariadb.com/MariaDB CR name for a // cozystack MariaDB application instance. The mapping mirrors the mariadb // ApplicationDefinition (release.prefix=mariadb-). @@ -582,10 +596,25 @@ func (r *RestoreJobReconciler) reconcileMariaDBRestore(ctx context.Context, rest return ctrl.Result{}, err } - mdbRestore, err := r.ensureMariaDBRestore(ctx, restoreJob, sourceBackupName, targetMDBName) + mdbRestore, err := r.findMariaDBRestoreForJob(ctx, restoreJob) if err != nil { return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("failed to ensure k8s.mariadb.com/Restore: %v", err)) } + // The dump is chosen once, as the operator Restore is created; one that + // already exists has made its choice. + if mdbRestore == nil { + targetTime, err := r.mariadbRestoreTargetTime(ctx, backup, srcBackup) + var refused mariadbRestoreRefused + if errors.As(err, &refused) { + return r.markRestoreJobFailed(ctx, restoreJob, refused.Error()) + } + if err != nil { + return r.awaitMariaDBDumpCheck(ctx, restoreJob, options, err) + } + if mdbRestore, err = r.createMariaDBRestore(ctx, restoreJob, sourceBackupName, targetMDBName, targetTime); err != nil { + return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("failed to ensure k8s.mariadb.com/Restore: %v", err)) + } + } cond := apimeta.FindStatusCondition(mdbRestore.Status.Conditions, mariadbtypes.ConditionTypeComplete) switch { @@ -626,10 +655,68 @@ func (r *RestoreJobReconciler) reconcileMariaDBRestore(ctx context.Context, rest } } -// ensureMariaDBRestore creates a k8s.mariadb.com/Restore CR labelled with -// the RestoreJob, or returns the existing one if a previous reconcile -// already created it. -func (r *RestoreJobReconciler) ensureMariaDBRestore(ctx context.Context, rj *backupsv1alpha1.RestoreJob, sourceBackupName, targetMariaDBName string) (*mariadbtypes.Restore, error) { +// mariadbRestoreRefused is why the dump a RestoreJob asks for cannot be +// restored. It fails the RestoreJob, where any other error from choosing +// the dump is retried. +type mariadbRestoreRefused string + +func (e mariadbRestoreRefused) Error() string { return string(e) } + +// mariadbRestoreTargetTime returns the targetRecoveryTime of the operator +// Restore for backup: the timestamp in the name of the dump backup +// records. mariadb-operator restores the dump whose timestamp equals the +// target and, when there is none, the nearest one, so the dump is first +// checked to still exist. Empty when backup records no dump, which leaves +// the operator to take the newest dump under the prefix. +func (r *RestoreJobReconciler) mariadbRestoreTargetTime(ctx context.Context, backup *backupsv1alpha1.Backup, src *mariadbtypes.Backup) (string, error) { + name := backup.Spec.DriverMetadata[mariadbDumpFileKey] + if name == "" { + return "", nil + } + ts, ok := parseMariaDBLogicalBackupFile(name) + if !ok { + return "", mariadbRestoreRefused(fmt.Sprintf("Backup driverMetadata %s=%q is not the name of a mariadb-operator dump", mariadbDumpFileKey, name)) + } + ctx, cancel := context.WithTimeout(ctx, mariadbRestoreDumpCheckTimeout) + defer cancel() + exists, err := mariadbDumpExists(ctx, r.objectStore(), src, name) + if err != nil { + return "", fmt.Errorf("cannot check that dump %s of k8s.mariadb.com/Backup %s/%s still exists in storage: %w", name, src.Namespace, src.Name, err) + } + if !exists { + s3 := src.Spec.Storage.S3 + return "", mariadbRestoreRefused(fmt.Sprintf( + "dump %s of k8s.mariadb.com/Backup %s/%s no longer exists in storage (s3://%s/%s); restoring now would bring back a different dump", + name, src.Namespace, src.Name, s3.Bucket, mariadbOperatorS3Prefix(s3.Prefix))) + } + return ts.UTC().Format(time.RFC3339), nil +} + +// awaitMariaDBDumpCheck keeps the RestoreJob waiting, with Ready=False +// saying why, while the dump it asks for cannot be checked, and fails it +// with the cause once the restore deadline has passed. +func (r *RestoreJobReconciler) awaitMariaDBDumpCheck(ctx context.Context, rj *backupsv1alpha1.RestoreJob, options MariaDBRestoreOptions, cause error) (ctrl.Result, error) { + deadline := options.effectiveRestoreDeadline() + if rj.Status.StartedAt != nil && time.Since(rj.Status.StartedAt.Time) > deadline { + return r.markRestoreJobFailed(ctx, rj, fmt.Sprintf( + "%v; gave up after %s (override via spec.options.restoreTimeoutSeconds)", cause, deadline)) + } + getLogger(ctx).Info("cannot check the MariaDB dump to restore; retrying", "restorejob", rj.Name, "error", cause) + apimeta.SetStatusCondition(&rj.Status.Conditions, metav1.Condition{ + Type: "Ready", + Status: metav1.ConditionFalse, + Reason: "DumpCheckPending", + Message: cause.Error(), + }) + if err := r.Status().Update(ctx, rj); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: mariadbDumpCheckRetryInterval}, nil +} + +// findMariaDBRestoreForJob returns the k8s.mariadb.com/Restore labelled +// with the RestoreJob, if a previous reconcile already created one. +func (r *RestoreJobReconciler) findMariaDBRestoreForJob(ctx context.Context, rj *backupsv1alpha1.RestoreJob) (*mariadbtypes.Restore, error) { list := &mariadbtypes.RestoreList{} if err := r.List(ctx, list, client.InNamespace(rj.Namespace), @@ -640,10 +727,15 @@ func (r *RestoreJobReconciler) ensureMariaDBRestore(ctx context.Context, rj *bac ); err != nil { return nil, err } - if len(list.Items) > 0 { - return &list.Items[0], nil + if len(list.Items) == 0 { + return nil, nil } + return &list.Items[0], nil +} +// createMariaDBRestore creates the k8s.mariadb.com/Restore of the +// RestoreJob, labelled with it. targetRecoveryTime is RFC3339, or empty. +func (r *RestoreJobReconciler) createMariaDBRestore(ctx context.Context, rj *backupsv1alpha1.RestoreJob, sourceBackupName, targetMariaDBName, targetRecoveryTime string) (*mariadbtypes.Restore, error) { obj := &mariadbtypes.Restore{ ObjectMeta: metav1.ObjectMeta{ Namespace: rj.Namespace, @@ -659,6 +751,7 @@ func (r *RestoreJobReconciler) ensureMariaDBRestore(ctx context.Context, rj *bac Kind: mariadbtypes.BackupKind, Name: sourceBackupName, }, + TargetRecoveryTime: targetRecoveryTime, }, } if err := r.Create(ctx, obj); err != nil { diff --git a/internal/backupcontroller/mariadbstrategy_controller_test.go b/internal/backupcontroller/mariadbstrategy_controller_test.go index 0e4f63b5e..fa033fb22 100644 --- a/internal/backupcontroller/mariadbstrategy_controller_test.go +++ b/internal/backupcontroller/mariadbstrategy_controller_test.go @@ -313,7 +313,7 @@ func TestEnsureMariaDBBackup_IdempotentByLabel(t *testing.T) { // Restore-side ensure idempotency + target rewrite // --------------------------------------------------------------------------- -func TestEnsureMariaDBRestore_IdempotentByLabel(t *testing.T) { +func TestFindMariaDBRestoreForJob_FindsTheCreatedRestore(t *testing.T) { c := newMariaDBStrategyTestClient(t) r := &RestoreJobReconciler{Client: c, Scheme: c.Scheme()} rj := newMariaDBRestoreJob("rj-1", "tenant") @@ -321,16 +321,16 @@ func TestEnsureMariaDBRestore_IdempotentByLabel(t *testing.T) { t.Fatalf("seed RestoreJob: %v", err) } - first, err := r.ensureMariaDBRestore(context.Background(), rj, "src-backup", "mariadb-target") - if err != nil { - t.Fatalf("first ensureMariaDBRestore: %v", err) + if none, err := r.findMariaDBRestoreForJob(context.Background(), rj); err != nil || none != nil { + t.Fatalf("findMariaDBRestoreForJob before create = %v, %v; want none", none, err) } - second, err := r.ensureMariaDBRestore(context.Background(), rj, "src-backup", "mariadb-target") + created, err := r.createMariaDBRestore(context.Background(), rj, "src-backup", "mariadb-target", "2026-10-21T10:00:05Z") if err != nil { - t.Fatalf("second ensureMariaDBRestore: %v", err) + t.Fatalf("createMariaDBRestore: %v", err) } - if first.Name != second.Name { - t.Errorf("expected idempotent reuse: first=%q second=%q", first.Name, second.Name) + found, err := r.findMariaDBRestoreForJob(context.Background(), rj) + if err != nil || found == nil || found.Name != created.Name { + t.Fatalf("findMariaDBRestoreForJob = %v, %v; want the Restore just created", found, err) } list := &mariadbtypes.RestoreList{} if err := c.List(context.Background(), list, client.InNamespace("tenant")); err != nil { @@ -346,6 +346,9 @@ func TestEnsureMariaDBRestore_IdempotentByLabel(t *testing.T) { if got.Spec.BackupRef == nil || got.Spec.BackupRef.Name != "src-backup" || got.Spec.BackupRef.Kind != mariadbtypes.BackupKind { t.Errorf("BackupRef mismatch: %#v", got.Spec.BackupRef) } + if got.Spec.TargetRecoveryTime != "2026-10-21T10:00:05Z" { + t.Errorf("TargetRecoveryTime: got %q want 2026-10-21T10:00:05Z", got.Spec.TargetRecoveryTime) + } } // --------------------------------------------------------------------------- diff --git a/internal/backupcontroller/restorejob_controller.go b/internal/backupcontroller/restorejob_controller.go index 1a61f0800..db7348923 100644 --- a/internal/backupcontroller/restorejob_controller.go +++ b/internal/backupcontroller/restorejob_controller.go @@ -57,6 +57,11 @@ type RestoreJobReconciler struct { // see a peer's StartedAt before the cache does. Wired in // SetupWithManager. APIReader client.Reader + + // newS3ObjectLister overrides the S3 client used to check that a + // backup's object still exists; nil means a real minio client. Set by + // unit tests. + newS3ObjectLister func(conn s3Connection) (s3ObjectLister, error) } func (r *RestoreJobReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { From 195de033e8156876885d685cd1cced3d2dfb1142 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 15:55:36 +0000 Subject: [PATCH 106/518] fix(mariadb): pin legacy Backups to a proven dump, refuse failed runs A Backup taken before Backups recorded their dump gives a restore nothing to pin. Refusing such Backups would leave every existing one unrestorable, so a restore of one now makes the same ownership-proven match the backup side makes, against the operator Backup it names, and restores that dump by its timestamp. When no dump can be proven to be the run's own, or the storage cannot be read, mariadb-operator restores the newest dump under the prefix as it always did, and a BestEffortDump Warning on the RestoreJob says so and why. An operator Backup whose Job failed is Complete=True with reason JobFailed, and a BackupJob that took Complete=True for success made a Backup of it. That run left no dump of its own, and restoring it would quietly bring back another run's dump, so the RestoreJob now fails instead. The condition is read by its reason, because its status does not tell a failure from a success. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- .../mariadb_restore_dump_test.go | 129 +++++++++++++++++- .../mariadbstrategy_controller.go | 41 +++++- 2 files changed, 156 insertions(+), 14 deletions(-) diff --git a/internal/backupcontroller/mariadb_restore_dump_test.go b/internal/backupcontroller/mariadb_restore_dump_test.go index 0375def33..6c8202adf 100644 --- a/internal/backupcontroller/mariadb_restore_dump_test.go +++ b/internal/backupcontroller/mariadb_restore_dump_test.go @@ -274,13 +274,128 @@ func TestReconcileMariaDBRestore_RecordedDump(t *testing.T) { }) } -func TestReconcileMariaDBRestore_NoRecordedDump(t *testing.T) { - l := &fakeS3Lister{err: errors.New("Access Denied.")} - f := newMariaDBRestoreFixture(t, nil, l, s3CredsSecret("tenant", "creds")) - f.reconcile(t) - f.wantRestore(t, "") - if l.calls != 0 { - t.Errorf("S3 listed %d times for a Backup that records no dump", l.calls) +// TestReconcileMariaDBRestore_LegacyBackup drives Backups that record no +// dump: one proven to belong to the source run is restored by its +// timestamp, and anything else restores the newest dump with a Warning. +func TestReconcileMariaDBRestore_LegacyBackup(t *testing.T) { + creds := s3CredsSecret("tenant", "creds") + own := s3ObjectInfo{Key: mdbRestorePrefix + mdbRestoreDump, LastModified: mdbRestoreT0.Add(20 * time.Second)} + newer := s3ObjectInfo{Key: mdbRestorePrefix + mdbNewerDump, LastModified: mdbRestoreT0.Add(time.Hour)} + second := s3ObjectInfo{Key: mdbRestorePrefix + "backup.2026-10-21T10:00:30Z.gzip.sql", LastModified: mdbRestoreT0.Add(35 * time.Second)} + + t.Run("a dump proven to be the run's own is restored by its timestamp", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, nil, &fakeS3Lister{objects: []s3ObjectInfo{own, newer}}, creds) + f.reconcile(t) + f.wantRestore(t, mdbRestoreTime) + if ev := drainEvents(f.events); len(ev) != 0 { + t.Errorf("events %q; want none for a proven dump", ev) + } + }) + + cases := []struct { + name string + lister *fakeS3Lister + cause string + }{ + {name: "two dumps in the run's lifetime", lister: &fakeS3Lister{objects: []s3ObjectInfo{own, second, newer}}, cause: "cannot be told apart"}, + {name: "no dump in the run's lifetime", lister: &fakeS3Lister{objects: []s3ObjectInfo{newer}}, cause: "no dump in the bucket belongs"}, + {name: "a storage that cannot be read", lister: &fakeS3Lister{err: errors.New("Access Denied.")}, cause: "Access Denied."}, + } + wantBestEffort := func(t *testing.T, f *mariadbRestoreFixture, cause string) { + t.Helper() + f.wantRestore(t, "") + ev := drainEvents(f.events) + if len(ev) != 1 || !strings.HasPrefix(ev[0], "Warning BestEffortDump ") || !strings.Contains(ev[0], cause) || !strings.Contains(ev[0], "src-bk records no dump") { + t.Errorf("events %q; want one BestEffortDump warning naming the Backup and %q", ev, cause) + } + } + for _, tc := range cases { + t.Run(tc.name+" restores the newest dump with a warning", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, nil, tc.lister, creds) + f.reconcile(t) + wantBestEffort(t, f, tc.cause) + }) + } + + t.Run("a hanging endpoint is given up at the check timeout", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, nil, &fakeS3Lister{}, creds) + f.r.newS3ObjectLister = func(s3Connection) (s3ObjectLister, error) { return blockingLister{}, nil } + var err error + withShortDumpCheckTimeout(t, func() { _, err = f.r.reconcileMariaDBRestore(context.Background(), f.rj, f.backup) }) + if err != nil { + t.Fatalf("reconcileMariaDBRestore: %v", err) + } + wantBestEffort(t, f, context.DeadlineExceeded.Error()) + }) +} + +// TestReconcileMariaDBRestore_FailedBackupRun drives a Backup made of an +// operator Backup whose Job failed: there is no dump of its own to +// restore, whatever the Backup records. +func TestReconcileMariaDBRestore_FailedBackupRun(t *testing.T) { + creds := s3CredsSecret("tenant", "creds") + failed := func(status metav1.ConditionStatus) []metav1.Condition { + return []metav1.Condition{{ + Type: mariadbtypes.ConditionTypeComplete, Status: status, Reason: mariadbtypes.ConditionReasonJobFailed, + Message: "Failed", LastTransitionTime: metav1.NewTime(mdbRestoreT0.Add(3 * time.Minute)), + }} + } + listing := []s3ObjectInfo{{Key: mdbRestorePrefix + mdbRestoreDump, LastModified: mdbRestoreT0.Add(20 * time.Second)}} + + cases := []struct { + name string + driverMD map[string]string + status metav1.ConditionStatus + }{ + {name: "Complete=True/JobFailed with no recorded dump", status: metav1.ConditionTrue}, + {name: "Complete=True/JobFailed with a recorded dump", driverMD: recordedDump(mdbRestoreDump), status: metav1.ConditionTrue}, + {name: "Complete=False/JobFailed", status: metav1.ConditionFalse}, + } + for _, tc := range cases { + t.Run(tc.name+" fails the RestoreJob", func(t *testing.T) { + l := &fakeS3Lister{objects: listing} + f := newMariaDBRestoreFixture(t, tc.driverMD, l, creds) + f.src.Status.Conditions = failed(tc.status) + if err := f.c.Update(context.Background(), f.src); err != nil { + t.Fatalf("update operator Backup: %v", err) + } + f.reconcile(t) + f.wantFailed(t, "k8s.mariadb.com/Backup tenant/op-bk failed", "left no dump to restore") + if l.calls != 0 { + t.Errorf("S3 listed %d times for a failed backup run", l.calls) + } + }) + } + + t.Run("an operator Restore created earlier is followed", func(t *testing.T) { + existing := &mariadbtypes.Restore{ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", Name: "rj-abcde", + Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: "rj", + backupsv1alpha1.OwningJobNamespaceLabel: "tenant", + }, + }} + f := newMariaDBRestoreFixture(t, nil, &fakeS3Lister{objects: listing}, creds, existing) + f.src.Status.Conditions = failed(metav1.ConditionTrue) + if err := f.c.Update(context.Background(), f.src); err != nil { + t.Fatalf("update operator Backup: %v", err) + } + f.reconcile(t) + if f.rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed { + t.Errorf("RestoreJob failed although its operator Restore was already created: %s", f.rj.Status.Message) + } + }) +} + +func drainEvents(r *record.FakeRecorder) []string { + var out []string + for { + select { + case e := <-r.Events: + out = append(out, e) + default: + return out + } } } diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 86b42d33f..6d6312eb4 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -603,7 +603,7 @@ func (r *RestoreJobReconciler) reconcileMariaDBRestore(ctx context.Context, rest // The dump is chosen once, as the operator Restore is created; one that // already exists has made its choice. if mdbRestore == nil { - targetTime, err := r.mariadbRestoreTargetTime(ctx, backup, srcBackup) + targetTime, err := r.mariadbRestoreTargetTime(ctx, restoreJob, backup, srcBackup) var refused mariadbRestoreRefused if errors.As(err, &refused) { return r.markRestoreJobFailed(ctx, restoreJob, refused.Error()) @@ -666,19 +666,26 @@ func (e mariadbRestoreRefused) Error() string { return string(e) } // Restore for backup: the timestamp in the name of the dump backup // records. mariadb-operator restores the dump whose timestamp equals the // target and, when there is none, the nearest one, so the dump is first -// checked to still exist. Empty when backup records no dump, which leaves -// the operator to take the newest dump under the prefix. -func (r *RestoreJobReconciler) mariadbRestoreTargetTime(ctx context.Context, backup *backupsv1alpha1.Backup, src *mariadbtypes.Backup) (string, error) { +// checked to still exist. A Backup that records no dump is resolved by +// legacyMariaDBRestoreTargetTime. +func (r *RestoreJobReconciler) mariadbRestoreTargetTime(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, src *mariadbtypes.Backup) (string, error) { + // A failed Job is Complete=True too, with reason JobFailed, and a + // BackupJob that took Complete=True for success made a Backup of it. + if cond := apimeta.FindStatusCondition(src.Status.Conditions, mariadbtypes.ConditionTypeComplete); cond != nil && cond.Reason == mariadbtypes.ConditionReasonJobFailed { + return "", mariadbRestoreRefused(fmt.Sprintf( + "the backup run of k8s.mariadb.com/Backup %s/%s failed (Complete=%s/%s) and left no dump to restore; restoring would bring back another run's dump", + src.Namespace, src.Name, cond.Status, cond.Reason)) + } + ctx, cancel := context.WithTimeout(ctx, mariadbRestoreDumpCheckTimeout) + defer cancel() name := backup.Spec.DriverMetadata[mariadbDumpFileKey] if name == "" { - return "", nil + return r.legacyMariaDBRestoreTargetTime(ctx, rj, backup, src), nil } ts, ok := parseMariaDBLogicalBackupFile(name) if !ok { return "", mariadbRestoreRefused(fmt.Sprintf("Backup driverMetadata %s=%q is not the name of a mariadb-operator dump", mariadbDumpFileKey, name)) } - ctx, cancel := context.WithTimeout(ctx, mariadbRestoreDumpCheckTimeout) - defer cancel() exists, err := mariadbDumpExists(ctx, r.objectStore(), src, name) if err != nil { return "", fmt.Errorf("cannot check that dump %s of k8s.mariadb.com/Backup %s/%s still exists in storage: %w", name, src.Namespace, src.Name, err) @@ -692,6 +699,26 @@ func (r *RestoreJobReconciler) mariadbRestoreTargetTime(ctx context.Context, bac return ts.UTC().Format(time.RFC3339), nil } +// legacyMariaDBRestoreTargetTime chooses the dump of a Backup taken before +// Backups recorded theirs: the one findMariaDBDump proves belongs to src, +// with the completion time the size probe was given, which takenAt keeps. +// When none is proven, for any reason, it returns empty, and the operator +// restores the newest dump under the prefix as such a restore always did: +// refusing would leave these Backups unrestorable, and the controller may +// never have been able to reach their storage. A Warning event says the +// dump is a best effort. +func (r *RestoreJobReconciler) legacyMariaDBRestoreTargetTime(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, src *mariadbtypes.Backup) string { + dump, err := findMariaDBDump(ctx, r.objectStore(), src, backup.Spec.TakenAt.Time) + if err != nil { + getLogger(ctx).Info("no dump proven for a MariaDB Backup that records none; restoring the newest", "restorejob", rj.Name, "backup", backup.Name, "error", err) + r.Recorder.Eventf(rj, corev1.EventTypeWarning, "BestEffortDump", + "Backup %s records no dump and none in storage could be proven to belong to k8s.mariadb.com/Backup %s/%s (%v); mariadb-operator restores the newest dump under its prefix as a best effort, which may be another backup run's", + backup.Name, src.Namespace, src.Name, err) + return "" + } + return dump.Timestamp.UTC().Format(time.RFC3339) +} + // awaitMariaDBDumpCheck keeps the RestoreJob waiting, with Ready=False // saying why, while the dump it asks for cannot be checked, and fails it // with the cause once the restore deadline has passed. From 2cc90b18f51b77f6c5a1324efcbfef8b34e4b305 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:03:31 +0300 Subject: [PATCH 107/518] fix(seaweedfs): serve the s3 internal gRPC over mTLS weed s3 takes its gRPC server identity from [grpc.s3] in security.toml and serves plaintext when the section is missing, which it is in the 4.45 chart. Every peer dials that port with mTLS: the s3_lifecycle worker with its [grpc.worker] certificate, other s3 replicas with [grpc.client] for IAM cache propagation. Both fail the handshake, so lifecycle deletes cannot run and identity changes do not reach the other replicas. The section is backported from upstream (seaweedfs/seaweedfs#11157, first released in 4.46) as a chart patch. It uses the internal CA-signed client certificate the s3 pods already mount, not the public listener's tlsSecret, which internal peers would not trust. security.toml is mounted by subPath, so a running s3 pod never sees the new section. A pod annotation changes the s3 template in this release so the upgrade restarts it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- packages/system/seaweedfs/Makefile | 1 + .../templates/shared/security-configmap.yaml | 13 ++++++++ .../seaweedfs/patches/s3-grpc-tls.patch | 24 ++++++++++++++ .../seaweedfs/tests/s3_grpc_tls_test.yaml | 32 +++++++++++++++++++ packages/system/seaweedfs/values.yaml | 8 +++++ 5 files changed, 78 insertions(+) create mode 100644 packages/system/seaweedfs/patches/s3-grpc-tls.patch create mode 100644 packages/system/seaweedfs/tests/s3_grpc_tls_test.yaml diff --git a/packages/system/seaweedfs/Makefile b/packages/system/seaweedfs/Makefile index 3436e5f0f..2f7790b38 100644 --- a/packages/system/seaweedfs/Makefile +++ b/packages/system/seaweedfs/Makefile @@ -61,6 +61,7 @@ update: patch --no-backup-if-mismatch -p4 < patches/s3-service-name.patch patch --no-backup-if-mismatch -p4 < patches/s3-service-name-consumers.patch patch --no-backup-if-mismatch -p4 < patches/cluster-scoped-names-per-namespace.patch + patch --no-backup-if-mismatch -p4 < patches/s3-grpc-tls.patch #patch --no-backup-if-mismatch -p4 < patches/retention-policy-delete.yaml $(MAKE) update-image diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/shared/security-configmap.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/shared/security-configmap.yaml index a067e1de9..17b82cca0 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/shared/security-configmap.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/shared/security-configmap.yaml @@ -97,6 +97,19 @@ data: cert = "/usr/local/share/ca-certificates/worker/tls.crt" key = "/usr/local/share/ca-certificates/worker/tls.key" + {{- /* S3 gRPC server identity (weed s3 -port.grpc, default httpPort+10000). + Without this section, LoadServerTLS("grpc.s3") returns nil and the S3 + server serves its internal gRPC (IAM cache propagation, s3 lifecycle + delete RPCs) as plaintext, while peers dial it with mTLS credentials: + "tls: first record does not look like a TLS handshake". + Always uses the internal CA-signed client certificate (already mounted + on s3 pods by s3-deployment.yaml). Deliberately NOT s3.tlsSecret: that + secret is for the public HTTPS listener and may be issued by a public + CA which internal gRPC peers (trusting only grpc.ca) would reject. */}} + [grpc.s3] + cert = "/usr/local/share/ca-certificates/client/tls.crt" + key = "/usr/local/share/ca-certificates/client/tls.key" + # use this for any place needs a grpc client # i.e., "weed backup|benchmark|filer.copy|filer.replicate|mount|s3|upload" [grpc.client] diff --git a/packages/system/seaweedfs/patches/s3-grpc-tls.patch b/packages/system/seaweedfs/patches/s3-grpc-tls.patch new file mode 100644 index 000000000..d4d45fbd0 --- /dev/null +++ b/packages/system/seaweedfs/patches/s3-grpc-tls.patch @@ -0,0 +1,24 @@ +diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/shared/security-configmap.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/shared/security-configmap.yaml +index a067e1de9..17b82cca0 100644 +--- a/packages/system/seaweedfs/charts/seaweedfs/templates/shared/security-configmap.yaml ++++ b/packages/system/seaweedfs/charts/seaweedfs/templates/shared/security-configmap.yaml +@@ -97,6 +97,19 @@ data: + cert = "/usr/local/share/ca-certificates/worker/tls.crt" + key = "/usr/local/share/ca-certificates/worker/tls.key" + ++ {{- /* S3 gRPC server identity (weed s3 -port.grpc, default httpPort+10000). ++ Without this section, LoadServerTLS("grpc.s3") returns nil and the S3 ++ server serves its internal gRPC (IAM cache propagation, s3 lifecycle ++ delete RPCs) as plaintext, while peers dial it with mTLS credentials: ++ "tls: first record does not look like a TLS handshake". ++ Always uses the internal CA-signed client certificate (already mounted ++ on s3 pods by s3-deployment.yaml). Deliberately NOT s3.tlsSecret: that ++ secret is for the public HTTPS listener and may be issued by a public ++ CA which internal gRPC peers (trusting only grpc.ca) would reject. */}} ++ [grpc.s3] ++ cert = "/usr/local/share/ca-certificates/client/tls.crt" ++ key = "/usr/local/share/ca-certificates/client/tls.key" ++ + # use this for any place needs a grpc client + # i.e., "weed backup|benchmark|filer.copy|filer.replicate|mount|s3|upload" + [grpc.client] diff --git a/packages/system/seaweedfs/tests/s3_grpc_tls_test.yaml b/packages/system/seaweedfs/tests/s3_grpc_tls_test.yaml new file mode 100644 index 000000000..afac836ed --- /dev/null +++ b/packages/system/seaweedfs/tests/s3_grpc_tls_test.yaml @@ -0,0 +1,32 @@ +suite: seaweedfs s3 gRPC listener speaks mTLS + +templates: + - charts/seaweedfs/templates/shared/security-configmap.yaml + - charts/seaweedfs/templates/s3/s3-deployment.yaml + +release: + name: seaweedfs-system + namespace: tenant-root + +tests: + # weed s3 serves its internal gRPC (the LifecycleDelete RPC among it) with + # the [grpc.s3] identity and as plaintext without one, while the worker dials + # it with [grpc.worker] mTLS credentials, so every delete failed the TLS + # handshake. patches/s3-grpc-tls.patch backports the section from upstream. + - it: gives the s3 gRPC server a TLS identity signed by the internal CA + template: charts/seaweedfs/templates/shared/security-configmap.yaml + asserts: + - matchRegex: + path: data["security.toml"] + pattern: '\[grpc\.s3\]\s+cert = "/usr/local/share/ca-certificates/client/tls\.crt"\s+key = "/usr/local/share/ca-certificates/client/tls\.key"' + + # security.toml reaches the s3 pods through a subPath mount, which a running + # container never re-reads, so an upgrade that only changes the ConfigMap + # leaves s3 serving plaintext gRPC until something restarts it. The pod + # template has to change in the same release. + - it: rolls the s3 pods so they load the new section + template: charts/seaweedfs/templates/s3/s3-deployment.yaml + asserts: + - equal: + path: spec.template.metadata.annotations["seaweedfs.cozystack.io/grpc-s3-tls"] + value: "true" diff --git a/packages/system/seaweedfs/values.yaml b/packages/system/seaweedfs/values.yaml index bce78c47f..e1eba8a58 100644 --- a/packages/system/seaweedfs/values.yaml +++ b/packages/system/seaweedfs/values.yaml @@ -207,6 +207,14 @@ seaweedfs: # weed skip startLanceServer and drops the container port and Service port # with it. lancePort: 0 + # patches/s3-grpc-tls.patch adds [grpc.s3] to security.toml, which the s3 + # pods mount by subPath and so never re-read. This annotation changes the + # pod template in the release that brings the section, so an upgrade + # restarts s3 onto it instead of leaving its gRPC in plaintext. The + # upstream s3 template carries no checksum of that ConfigMap, and adding + # one would be another vendored patch for a change made once. + podAnnotations: + seaweedfs.cozystack.io/grpc-s3-tls: "true" # Keep S3 traffic on same-zone endpoints. Native chart support landed in 4.12 # (replaces the former s3-traffic-distribution.patch); the chart gates the # field on Kubernetes >=1.31 and auto-converts to PreferSameZone on >=1.35. From cd91138f3327a4886c72b9e8a35b4e54c06abad7 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:13:55 +0300 Subject: [PATCH 108/518] fix(backups): re-render the default backup objects when the S3 endpoint changes The S3 endpoint the platform backups use is copied twice and neither copy is refreshed. The COSI sidecar writes BucketInfo once per BucketAccess, so the credentials Secret rendered from it keeps its endpoint, and the Velero BSL and the default Strategy CRs keep whatever endpoint the last render of backupstrategy-controller saw. The default objects gate only acted on objects that were missing, so after a publishing.host change every backup kept going to the old host with nothing reporting it. The gate now also forces a render when a copy is stale: the bucket's -system release when the credentials Secret's endpoint differs from the COSI Secret it is rendered from, and backupstrategy-controller when the endpoint an object was rendered with differs from the credentials Secret's. That endpoint sits at a different path in every kind, in some only as a bare host, so the chart stamps the rendered value on each object as backups.cozystack.io/endpoint and the gate compares that. Objects rendered before the annotation existed count as stale until one forced render has gone through, which stamps it. The COSI Secret itself still holds the endpoint the BucketAccess was granted with; it follows a host change only once the BucketAccess is re-granted. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- cmd/backupstrategy-controller/main.go | 1 + hack/backupstrategy-crd-gate.bats | 36 ++ .../backupcontroller/default_objects_gate.go | 163 ++++++- .../default_objects_gate_test.go | 401 +++++++++++++++++- .../templates/_helpers.tpl | 7 +- .../templates/deployment.yaml | 7 + .../templates/strategy-altinity-default.yaml | 2 + .../templates/strategy-cnpg-default.yaml | 2 + .../templates/strategy-etcd-default.yaml | 2 + .../strategy-foundationdb-default.yaml | 2 + .../templates/strategy-kafka-default.yaml | 2 + .../templates/strategy-mariadb-default.yaml | 2 + .../templates/strategy-mongodb-default.yaml | 2 + .../templates/strategy-rabbitmq-default.yaml | 2 + .../templates/strategy-redis-default.yaml | 2 + .../strategy-velero-vmdisk-default.yaml | 2 + .../strategy-velero-vminstance-default.yaml | 2 + .../templates/velero-bsl.yaml | 2 + .../tests/default_objects_gate_test.yaml | 60 +++ .../backupstrategy-controller/values.yaml | 11 +- .../bucket/templates/user-credentials.yaml | 4 +- 21 files changed, 678 insertions(+), 36 deletions(-) diff --git a/cmd/backupstrategy-controller/main.go b/cmd/backupstrategy-controller/main.go index 625d79820..119ae3007 100644 --- a/cmd/backupstrategy-controller/main.go +++ b/cmd/backupstrategy-controller/main.go @@ -243,6 +243,7 @@ func main() { Namespace: os.Getenv("BACKUP_CREDENTIALS_HELMRELEASE_NAMESPACE"), Name: os.Getenv("BACKUP_CREDENTIALS_HELMRELEASE_NAME"), }, + CredentialsCOSISecretName: os.Getenv("BACKUP_CREDENTIALS_COSI_SECRET_NAME"), } if err := gate.SetupWithManager(mgr); err != nil { setupLog.Error(err, "unable to add DefaultObjectsGate runnable") diff --git a/hack/backupstrategy-crd-gate.bats b/hack/backupstrategy-crd-gate.bats index b4ad482a0..0f68a1ca4 100644 --- a/hack/backupstrategy-crd-gate.bats +++ b/hack/backupstrategy-crd-gate.bats @@ -133,3 +133,39 @@ $tpl: $obj does not gate on crdEstablished for $crd" rm -rf "$tmp" } + +# The controller's DefaultObjectsGate tells an object rendered with a stale S3 +# endpoint (publishing.host changed after the render) by the +# backups.cozystack.io/endpoint annotation, because the endpoint itself sits at +# a different path in every kind. An object the chart renders without it is +# forced once and then never compared again, so a new template that forgets it +# would silently keep a stale endpoint. +@test "every rendered Strategy CR and the Velero BSL carry the rendered endpoint annotation" { + tmp=$(mktemp -d) + + helm template t "$CHART" -n cozy-backup-controller \ + --set backupStorage.bucketNameOverride=ext-bucket \ + --set backupStorage.endpoint=https://s3.example.org \ + --output-dir "$tmp/open" >/dev/null + + objs=$(find "$tmp/open" -name '*.yaml' | sort | while read -r f; do + yq eval-all "select((.apiVersion // \"\" | test(\"^${STRATEGY_API}\")) or .kind == \"BackupStorageLocation\") | .kind + \"/\" + .metadata.name + \" \" + (.metadata.annotations[\"backups.cozystack.io/endpoint\"] // \"\")" "$f" \ + | sed "s#^#$(basename "$f") #" + done) + if ! echo "$objs" | grep -q " BackupStorageLocation/"; then + echo "render produced no BSL; nothing was checked for it" >&2 + false + fi + if ! echo "$objs" | grep -q " CNPG/"; then + echo "render produced no Strategy CR; nothing was checked" >&2 + false + fi + bad=$(echo "$objs" | awk '$3 != "https://s3.example.org"') + if [ -n "$bad" ]; then + echo "objects without the rendered endpoint in backups.cozystack.io/endpoint:" >&2 + echo "$bad" >&2 + false + fi + + rm -rf "$tmp" +} diff --git a/internal/backupcontroller/default_objects_gate.go b/internal/backupcontroller/default_objects_gate.go index 023682036..efcd970d5 100644 --- a/internal/backupcontroller/default_objects_gate.go +++ b/internal/backupcontroller/default_objects_gate.go @@ -2,10 +2,12 @@ package backupcontroller import ( "context" + "encoding/json" "errors" "fmt" "net/http" "sort" + "strings" "time" "github.com/go-logr/logr" @@ -91,6 +93,11 @@ const ( // same lookup-gated template as the Strategy CRs, so it belongs to the // same existence check. defaultBackupStorageLocationName = "cozy-default" + + // endpointAnnotation carries the S3 endpoint the chart rendered into the + // object. The endpoint itself sits at a different path in every kind, + // and in some only as a bare host, so the gate compares this instead. + endpointAnnotation = "backups.cozystack.io/endpoint" ) var backupStorageLocationGVR = schema.GroupVersionResource{ @@ -157,6 +164,14 @@ type DefaultObjectsGate struct { // then admin-managed and no release renders it, so there is nothing // to force. CredentialsHelmRelease types.NamespacedName + // CredentialsCOSISecretName is the Secret the COSI sidecar writes for the + // platform bucket's BucketAccess, in Config.SourceNamespace, from which + // CredentialsHelmRelease renders the credentials Secret. The sidecar + // writes it once per BucketAccess, so after a publishing.host change a + // re-granted BucketAccess carries the new endpoint while the credentials + // Secret keeps the old one; the gate compares the two. Empty disables + // the comparison. + CredentialsCOSISecretName string // VeleroNamespace is where the cozy-default BackupStorageLocation // lives. Empty skips the BSL check (velero.bslEnabled=false). VeleroNamespace string @@ -177,6 +192,11 @@ type DefaultObjectsGate struct { // MinForceInterval for no reason. lastForce time.Time lastCredentialsForce time.Time + // renderForced records that a forced render of HelmRelease went through. + // That render stamps endpointAnnotation on every object the chart + // renders, so from then on an object without it (one rendered by a chart + // that predates the annotation) is not counted as stale again. + renderForced bool } // errHelmReleaseSuspended reports that the target release has @@ -285,10 +305,10 @@ func (g *DefaultObjectsGate) checkAndLog(ctx context.Context, logger logr.Logger // The release named here is whichever one renders what is missing: // the credentials Secret comes from the bucket's -system // release, everything else from this chart's own. - logger.Info("forced a Helm upgrade to create missing default backup objects", + logger.Info("forced a Helm upgrade to create missing or stale default backup objects", "helmRelease", g.forcedReleaseFor(missing).String(), "missing", missing) case len(missing) > 0: - logger.Info("default backup objects still missing, no force issued on this tick", + logger.Info("default backup objects still missing or stale, no force issued on this tick", "helmRelease", g.forcedReleaseFor(missing).String(), "missing", missing, "minForceInterval", g.MinForceInterval.String()) default: @@ -302,7 +322,7 @@ func (g *DefaultObjectsGate) checkAndLog(ctx context.Context, logger logr.Logger // resolving the bucket name is a precondition for checking those at all. func (g *DefaultObjectsGate) forcedReleaseFor(missing []string) types.NamespacedName { credsKey := fmt.Sprintf("Secret/%s", g.Config.SourceSecretName) - if len(missing) == 1 && missing[0] == credsKey && g.CredentialsHelmRelease.Name != "" { + if len(missing) == 1 && (missing[0] == credsKey || strings.HasPrefix(missing[0], credsKey+" ")) && g.CredentialsHelmRelease.Name != "" { return g.CredentialsHelmRelease } return g.HelmRelease @@ -345,23 +365,95 @@ func (g *DefaultObjectsGate) Check(ctx context.Context) ([]string, bool, error) return g.reconcileCredentials(ctx) } + // The objects below are rendered from this Secret, so while it is stale + // they would only be re-rendered with the same stale endpoint. + stale, err := g.staleCredentials(ctx, src) + if err != nil { + defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() + return nil, false, err + } + if stale != "" { + return g.force(ctx, g.CredentialsHelmRelease, &g.lastCredentialsForce, []string{stale}) + } + backupClass := &backupsv1alpha1.BackupClass{} if err := g.Client.Get(ctx, client.ObjectKey{Name: g.BackupClassName}, backupClass); err != nil { defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() return nil, false, fmt.Errorf("get BackupClass %s: %w", g.BackupClassName, err) } - missing, err := g.missingObjects(ctx, backupClass) + // With provisionBucket=false (no bucket release) the chart renders + // backupStorage.endpoint rather than the Secret's, so the two may + // legitimately differ and there is nothing to compare. + want := "" + if g.CredentialsHelmRelease.Name != "" { + want = renderedEndpointFor(src.Data["endpoint"]) + } + state, err := g.inspectObjects(ctx, backupClass, want) if err != nil { defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() return nil, false, err } - defaultObjectsMissing.WithLabelValues(g.BackupClassName).Set(float64(len(missing))) - if len(missing) == 0 { + defaultObjectsMissing.WithLabelValues(g.BackupClassName).Set(float64(len(state.missing))) + needed := append(state.missing, state.stale...) + if len(needed) == 0 { return nil, false, nil } + sort.Strings(needed) - return g.force(ctx, g.HelmRelease, &g.lastForce, missing) + needed, forced, err := g.force(ctx, g.HelmRelease, &g.lastForce, needed) + if forced { + g.renderForced = true + } + return needed, forced, err +} + +// staleCredentials reports the credentials Secret, formatted for logs, when +// its endpoint differs from the one in the COSI Secret it is rendered from, +// comparing against what packages/system/bucket/templates/user-credentials.yaml +// writes for it. The COSI Secret being absent (the BucketAccess is being +// re-granted) leaves nothing to compare against and is not staleness. +func (g *DefaultObjectsGate) staleCredentials(ctx context.Context, src *corev1.Secret) (string, error) { + if g.CredentialsCOSISecretName == "" || g.CredentialsHelmRelease.Name == "" || g.CredentialsHelmRelease.Namespace == "" { + return "", nil + } + cosi := &corev1.Secret{} + key := types.NamespacedName{Namespace: g.Config.SourceNamespace, Name: g.CredentialsCOSISecretName} + if err := g.Client.Get(ctx, key, cosi); err != nil { + if apierrors.IsNotFound(err) { + return "", nil + } + return "", fmt.Errorf("get COSI Secret %s: %w", key, err) + } + var info struct { + Spec struct { + SecretS3 struct { + Endpoint string `json:"endpoint"` + } `json:"secretS3"` + } `json:"spec"` + } + if err := json.Unmarshal(cosi.Data["BucketInfo"], &info); err != nil { + return "", fmt.Errorf("parse BucketInfo of COSI Secret %s: %w", key, err) + } + want := strings.TrimPrefix(strings.TrimPrefix(info.Spec.SecretS3.Endpoint, "https://"), "http://") + have := string(src.Data["endpoint"]) + if have == want { + return "", nil + } + return fmt.Sprintf("Secret/%s (endpoint %q, COSI Secret has %q)", g.Config.SourceSecretName, have, want), nil +} + +// renderedEndpointFor mirrors the "backupstrategy-controller.endpoint" helper +// for a provisioned bucket: the credentials Secret's endpoint, trimmed, with +// any scheme replaced by https://. It is the value the chart stamps as +// endpointAnnotation. An empty endpoint makes the helper fall back to +// backupStorage.endpoint, which the gate cannot see, so it returns "". +func renderedEndpointFor(raw []byte) string { + host := strings.TrimSpace(string(raw)) + if host == "" { + return "" + } + return "https://" + strings.TrimPrefix(strings.TrimPrefix(host, "https://"), "http://") } // reconcileCredentials handles the one object the gate cannot check the same @@ -431,16 +523,42 @@ func (g *DefaultObjectsGate) force(ctx context.Context, target types.NamespacedN return missing, true, nil } -// missingObjects returns a sorted, de-duplicated list of the +// defaultObjectsState is what inspectObjects found: the objects that do not +// exist, and the existing ones rendered with an endpoint other than the one +// the chart would render now. +type defaultObjectsState struct { + missing []string + stale []string +} + +// staleEndpoint reports an existing object rendered with another endpoint +// than want, formatted for logs. An empty want disables the comparison. +func (g *DefaultObjectsGate) staleEndpoint(state *defaultObjectsState, key string, obj *unstructured.Unstructured, want string) { + if want == "" { + return + } + have, ok := obj.GetAnnotations()[endpointAnnotation] + switch { + case !ok: + if !g.renderForced { + state.stale = append(state.stale, key+" (no endpoint annotation)") + } + case have != want: + state.stale = append(state.stale, fmt.Sprintf("%s (endpoint %q, want %q)", key, have, want)) + } +} + +// inspectObjects returns a sorted, de-duplicated list of the // "/" objects the BackupClass references that do not exist, -// plus the Velero BSL when a Velero namespace is configured. +// plus the Velero BSL when a Velero namespace is configured, and the existing +// ones whose endpointAnnotation differs from want. // // An unmappable group/kind (the CRD is not installed) is NOT reported as // missing: no Helm re-render can create an object whose CRD is absent, and // counting it would keep the gate forcing upgrades forever. -func (g *DefaultObjectsGate) missingObjects(ctx context.Context, backupClass *backupsv1alpha1.BackupClass) ([]string, error) { +func (g *DefaultObjectsGate) inspectObjects(ctx context.Context, backupClass *backupsv1alpha1.BackupClass, want string) (defaultObjectsState, error) { seen := map[string]struct{}{} - var missing []string + var state defaultObjectsState for _, strategy := range backupClass.Spec.Strategies { name := strategy.StrategyRef.Name @@ -472,17 +590,18 @@ func (g *DefaultObjectsGate) missingObjects(ctx context.Context, backupClass *ba if meta.IsNoMatchError(err) { continue } - return nil, fmt.Errorf("map %s/%s: %w", group, kind, err) + return defaultObjectsState{}, fmt.Errorf("map %s/%s: %w", group, kind, err) } // Strategy CRs are cluster-scoped; Namespace("") is correct for // both scopes on the dynamic client. - _, err = g.Resource(mapping.Resource).Get(ctx, name, metav1.GetOptions{}) + obj, err := g.Resource(mapping.Resource).Get(ctx, name, metav1.GetOptions{}) switch { case err == nil: + g.staleEndpoint(&state, key, obj, want) case apierrors.IsNotFound(err): - missing = append(missing, key) + state.missing = append(state.missing, key) default: - return nil, fmt.Errorf("get %s %s: %w", kind, name, err) + return defaultObjectsState{}, fmt.Errorf("get %s %s: %w", kind, name, err) } } @@ -496,24 +615,26 @@ func (g *DefaultObjectsGate) missingObjects(ctx context.Context, backupClass *ba mapping, err := g.RESTMapping(schema.GroupKind{Group: backupStorageLocationGVR.Group, Kind: "BackupStorageLocation"}) switch { case err == nil: - _, getErr := g.Resource(mapping.Resource).Namespace(g.VeleroNamespace).Get(ctx, defaultBackupStorageLocationName, metav1.GetOptions{}) + key := fmt.Sprintf("BackupStorageLocation/%s", defaultBackupStorageLocationName) + obj, getErr := g.Resource(mapping.Resource).Namespace(g.VeleroNamespace).Get(ctx, defaultBackupStorageLocationName, metav1.GetOptions{}) switch { case getErr == nil: + g.staleEndpoint(&state, key, obj, want) case apierrors.IsNotFound(getErr): - missing = append(missing, fmt.Sprintf("BackupStorageLocation/%s", defaultBackupStorageLocationName)) + state.missing = append(state.missing, key) default: - return nil, fmt.Errorf("get BackupStorageLocation %s: %w", defaultBackupStorageLocationName, getErr) + return defaultObjectsState{}, fmt.Errorf("get BackupStorageLocation %s: %w", defaultBackupStorageLocationName, getErr) } case meta.IsNoMatchError(err): // Velero API not served (e.g. Velero uninstalled after bootstrap): // no Helm re-render can create the BSL, so it is not "missing". default: - return nil, fmt.Errorf("map BackupStorageLocation: %w", err) + return defaultObjectsState{}, fmt.Errorf("map BackupStorageLocation: %w", err) } } - sort.Strings(missing) - return missing, nil + sort.Strings(state.missing) + return state, nil } // forceHelmRelease stamps BOTH reconcile.fluxcd.io/forceAt and diff --git a/internal/backupcontroller/default_objects_gate_test.go b/internal/backupcontroller/default_objects_gate_test.go index f5cbb5489..dc1c0c9f0 100644 --- a/internal/backupcontroller/default_objects_gate_test.go +++ b/internal/backupcontroller/default_objects_gate_test.go @@ -2,6 +2,8 @@ package backupcontroller import ( "context" + "errors" + "strings" "testing" "time" @@ -18,6 +20,7 @@ import ( k8stesting "k8s.io/client-go/testing" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" ) @@ -110,14 +113,60 @@ func cozyDefaultBackupClass() *backupsv1alpha1.BackupClass { } } +// renderedEndpoint is what the chart stamps on every default object for the +// endpoint sourceSecret carries: the steady state after a render that saw it. +const renderedEndpoint = "https://s3.example.com" + func strategyObject(kind, name string) *unstructured.Unstructured { u := &unstructured.Unstructured{} u.SetAPIVersion(strategyAPIGroup + "/v1alpha1") u.SetKind(kind) u.SetName(name) + u.SetAnnotations(map[string]string{endpointAnnotation: renderedEndpoint}) return u } +func withEndpoint(u *unstructured.Unstructured, endpoint string) *unstructured.Unstructured { + u.SetAnnotations(map[string]string{endpointAnnotation: endpoint}) + return u +} + +// unannotated is an object rendered by a chart that predates the endpoint +// annotation. +func unannotated(u *unstructured.Unstructured) *unstructured.Unstructured { + u.SetAnnotations(nil) + return u +} + +// allDefaultObjects is every object cozy-default routes to, plus the BSL, as +// the chart renders them for sourceSecret's endpoint. +func allDefaultObjects() []runtime.Object { + return []runtime.Object{ + strategyObject("CNPG", "cozy-default-cnpg"), + strategyObject("MariaDB", "cozy-default-mariadb"), + strategyObject("Etcd", "cozy-default-etcd"), + strategyObject("Altinity", "cozy-default-altinity"), + strategyObject("Velero", "cozy-default-velero-vminstance"), + strategyObject("Velero", "cozy-default-velero-vmdisk"), + bslObject(), + } +} + +// cosiSecret is the Secret the COSI sidecar writes for the platform bucket's +// BucketAccess, which user-credentials.yaml renders the credentials Secret +// from. The BucketInfo document is one read off a live cluster, with the keys +// and the host replaced. +func cosiSecret(endpoint string) *corev1.Secret { + bucketInfo := `{"metadata": {"name": "bc-b8e30337-1821-418f-a4dd-163e8016aaee", "creationTimestamp": null}, ` + + `"spec": {"bucketName": "bucket-1a2b", "authenticationType": "KEY", ` + + `"secretS3": {"endpoint": "` + endpoint + `", "region": "", "accessKeyID": "AK", "accessSecretKey": "SK"}, ` + + `"secretAzure": null, "protocols": ["s3"]}}` + return &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant-root", Name: "bucket-cozy-backups-system"}, + Data: map[string][]byte{"BucketInfo": []byte(bucketInfo)}, + } +} + func helmReleaseObject() *unstructured.Unstructured { u := &unstructured.Unstructured{} u.SetAPIVersion("helm.toolkit.fluxcd.io/v2") @@ -163,9 +212,10 @@ func newGate(t *testing.T, ctrlObjs []client.Object, dynObjs ...runtime.Object) Namespace: "tenant-root", Name: "bucket-cozy-backups-system", }, - VeleroNamespace: "cozy-velero", - MinForceInterval: 5 * time.Minute, - now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) }, + CredentialsCOSISecretName: "bucket-cozy-backups-system", + VeleroNamespace: "cozy-velero", + MinForceInterval: 5 * time.Minute, + now: func() time.Time { return time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) }, } g.Interface = dyn g.RESTMapper = gateRESTMapper() @@ -193,6 +243,7 @@ func bslObject() *unstructured.Unstructured { u.SetKind("BackupStorageLocation") u.SetNamespace("cozy-velero") u.SetName(defaultBackupStorageLocationName) + u.SetAnnotations(map[string]string{endpointAnnotation: renderedEndpoint}) return u } @@ -259,7 +310,7 @@ func TestCheckForcesWhenObjectsMissing(t *testing.T) { // the cluster. func TestCheckNoopWhenAllPresent(t *testing.T) { g, dyn := newGate(t, - []client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()}, + []client.Object{sourceSecret("bucket-1a2b"), cosiSecret("https://s3.example.com"), cozyDefaultBackupClass()}, helmReleaseObject(), strategyObject("CNPG", "cozy-default-cnpg"), strategyObject("MariaDB", "cozy-default-mariadb"), @@ -550,10 +601,11 @@ func TestMissingObjectsIgnoresUnmappedKinds(t *testing.T) { bslObject(), ) - missing, err := g.missingObjects(context.Background(), bc) + state, err := g.inspectObjects(context.Background(), bc, "") if err != nil { - t.Fatalf("missingObjects: %v", err) + t.Fatalf("inspectObjects: %v", err) } + missing := state.missing if len(missing) != 0 { t.Fatalf("missing = %v, want none (unmapped kind must be ignored)", missing) } @@ -579,10 +631,11 @@ func TestMissingObjectsSkipsVeleroWhenNamespaceEmpty(t *testing.T) { ) g.VeleroNamespace = "" - missing, err := g.missingObjects(context.Background(), bc) + state, err := g.inspectObjects(context.Background(), bc, "") if err != nil { - t.Fatalf("missingObjects: %v", err) + t.Fatalf("inspectObjects: %v", err) } + missing := state.missing if len(missing) != 0 { t.Fatalf("missing = %v, want none when Velero is disabled", missing) } @@ -630,10 +683,11 @@ func TestMissingObjectsSkipsBSLWhenVeleroAPIAbsent(t *testing.T) { } g.RESTMapper = m - missing, err := g.missingObjects(context.Background(), bc) + state, err := g.inspectObjects(context.Background(), bc, "") if err != nil { - t.Fatalf("missingObjects: %v", err) + t.Fatalf("inspectObjects: %v", err) } + missing := state.missing if len(missing) != 0 { t.Fatalf("missing = %v, want none when the Velero API is absent", missing) } @@ -744,3 +798,330 @@ func TestCheckToleratesAbsentHelmRelease(t *testing.T) { t.Error("objects should still be reported as missing") } } + +// TestCheckForcesCredentialsReleaseWhenEndpointStale covers the credentials +// Secret outliving the endpoint it was rendered from. The COSI sidecar writes +// BucketInfo once per BucketAccess, so a re-granted BucketAccess (after a +// publishing.host change) carries the new endpoint while the credentials +// Secret -system rendered earlier keeps the old one: the Secret +// exists, so the missing-Secret path never fires, and every consumer keeps +// the dead host. +func TestCheckForcesCredentialsReleaseWhenEndpointStale(t *testing.T) { + objs := append([]runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()}, allDefaultObjects()...) + g, dyn := newGate(t, + []client.Object{sourceSecret("bucket-1a2b"), cosiSecret("https://s3.new.example.com"), cozyDefaultBackupClass()}, + objs..., + ) + + missing, forced, err := g.Check(context.Background()) + if err != nil { + t.Fatalf("Check: %v", err) + } + if !forced { + t.Fatal("expected the bucket release forced while its credentials Secret carries a stale endpoint") + } + if len(missing) != 1 || !strings.HasPrefix(missing[0], "Secret/bucket-cozy-backups-system-credentials") { + t.Fatalf("missing = %v, want the credentials Secret", missing) + } + if ann := credentialsForceAnnotations(t, dyn); ann["reconcile.fluxcd.io/forceAt"] == "" { + t.Errorf("bucket release not forced: %v", ann) + } + // The objects were rendered from the stale Secret and agree with it; + // re-rendering them before the Secret is fixed would bake the old + // endpoint in again. + if ann := forceAnnotations(t, dyn); len(ann) != 0 { + t.Errorf("own HelmRelease forced while the credentials Secret is stale: %v", ann) + } +} + +// TestCheckCredentialsEndpointComparison pins the comparison against what +// user-credentials.yaml writes for a given BucketInfo endpoint (the scheme +// stripped, nothing else), so a Secret the chart rendered from the current +// BucketInfo never counts as stale and the bucket release is not forced on +// every throttle window. +func TestCheckCredentialsEndpointComparison(t *testing.T) { + for _, tc := range []struct { + name string + cosi string + wantForce bool + }{ + {"https scheme stripped", "https://s3.example.com", false}, + {"http scheme stripped", "http://s3.example.com", false}, + {"bare host", "s3.example.com", false}, + {"other host", "https://s3.new.example.com", true}, + {"other port", "https://s3.example.com:8443", true}, + } { + t.Run(tc.name, func(t *testing.T) { + objs := append([]runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()}, allDefaultObjects()...) + g, dyn := newGate(t, + []client.Object{sourceSecret("bucket-1a2b"), cosiSecret(tc.cosi), cozyDefaultBackupClass()}, + objs..., + ) + if _, _, err := g.Check(context.Background()); err != nil { + t.Fatalf("Check: %v", err) + } + forced := credentialsForceAnnotations(t, dyn)["reconcile.fluxcd.io/forceAt"] != "" + if forced != tc.wantForce { + t.Fatalf("bucket release forced = %v, want %v", forced, tc.wantForce) + } + }) + } +} + +// TestCheckSkipsCredentialsComparisonWithoutCOSISecret covers the window in +// which the BucketAccess is being re-granted: the COSI Secret is gone and +// there is nothing to compare against, which is not evidence of staleness. +func TestCheckSkipsCredentialsComparisonWithoutCOSISecret(t *testing.T) { + objs := append([]runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()}, allDefaultObjects()...) + g, dyn := newGate(t, + []client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()}, + objs..., + ) + + missing, forced, err := g.Check(context.Background()) + if err != nil || forced || len(missing) != 0 { + t.Fatalf("missing = %v, forced = %v, err = %v; want a no-op", missing, forced, err) + } + if ann := credentialsForceAnnotations(t, dyn); len(ann) != 0 { + t.Errorf("bucket release forced with no COSI Secret to compare against: %v", ann) + } +} + +// TestCheckSkipsCredentialsComparisonWhenUnconfigured pins both switches that +// turn the comparison off: no COSI Secret name, and no bucket release (external +// S3, where the Secret is admin-managed and no BucketInfo backs it). +func TestCheckSkipsCredentialsComparisonWhenUnconfigured(t *testing.T) { + for _, tc := range []struct { + name string + apply func(g *DefaultObjectsGate) + }{ + {"no COSI Secret name", func(g *DefaultObjectsGate) { g.CredentialsCOSISecretName = "" }}, + {"no bucket release", func(g *DefaultObjectsGate) { g.CredentialsHelmRelease = types.NamespacedName{} }}, + {"no bucket release name", func(g *DefaultObjectsGate) { g.CredentialsHelmRelease.Name = "" }}, + {"no bucket release namespace", func(g *DefaultObjectsGate) { g.CredentialsHelmRelease.Namespace = "" }}, + } { + t.Run(tc.name, func(t *testing.T) { + objs := append([]runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()}, allDefaultObjects()...) + g, dyn := newGate(t, + []client.Object{sourceSecret("bucket-1a2b"), cosiSecret("https://s3.new.example.com"), cozyDefaultBackupClass()}, + objs..., + ) + tc.apply(g) + // The real API server rejects a Get with an empty name; the fake + // answers NotFound, which would hide a lookup of an unset name. + g.Client = interceptor.NewClient(g.Client.(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if key.Name == "" { + return errors.New("resource name may not be empty") + } + return c.Get(ctx, key, obj, opts...) + }, + }) + missing, forced, err := g.Check(context.Background()) + if err != nil || forced || len(missing) != 0 { + t.Fatalf("missing = %v, forced = %v, err = %v; want a no-op", missing, forced, err) + } + if ann := credentialsForceAnnotations(t, dyn); len(ann) != 0 { + t.Errorf("bucket release forced with the comparison disabled: %v", ann) + } + }) + } +} + +// TestCheckReportsMalformedCOSISecret pins that a COSI Secret the gate cannot +// read is an error the check counts, not a silent "not stale". +func TestCheckReportsMalformedCOSISecret(t *testing.T) { + defaultObjectsCheckErrors.Reset() + bad := cosiSecret("") + bad.Data["BucketInfo"] = []byte("{not json") + objs := append([]runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()}, allDefaultObjects()...) + g, _ := newGate(t, []client.Object{sourceSecret("bucket-1a2b"), bad, cozyDefaultBackupClass()}, objs...) + + if _, forced, err := g.Check(context.Background()); err == nil || forced { + t.Fatalf("forced = %v, err = %v; want an error", forced, err) + } + if got := testutil.ToFloat64(defaultObjectsCheckErrors.WithLabelValues("cozy-default")); got != 1 { + t.Fatalf("check errors = %v, want 1", got) + } +} + +// TestCheckForcesOwnReleaseWhenObjectEndpointStale is the second half of a +// publishing.host change: the credentials Secret now carries the new +// endpoint, but the BSL and the default Strategies baked the old one at the +// last render of this chart and nothing re-renders them, so every backup +// keeps going to the dead host. +func TestCheckForcesOwnReleaseWhenObjectEndpointStale(t *testing.T) { + for _, tc := range []struct { + name string + stale *unstructured.Unstructured + key string + }{ + {"strategy", strategyObject("CNPG", "cozy-default-cnpg"), "CNPG/cozy-default-cnpg"}, + {"BSL", bslObject(), "BackupStorageLocation/cozy-default"}, + } { + t.Run(tc.name, func(t *testing.T) { + var objs []runtime.Object + for _, o := range allDefaultObjects() { + u := o.(*unstructured.Unstructured) + if u.GetKind() == tc.stale.GetKind() && u.GetName() == tc.stale.GetName() { + withEndpoint(u, "https://s3.old.example.com") + } + objs = append(objs, u) + } + objs = append(objs, helmReleaseObject(), credentialsHelmReleaseObject()) + g, dyn := newGate(t, + []client.Object{sourceSecret("bucket-1a2b"), cosiSecret("https://s3.example.com"), cozyDefaultBackupClass()}, + objs..., + ) + + missing, forced, err := g.Check(context.Background()) + if err != nil { + t.Fatalf("Check: %v", err) + } + if !forced { + t.Fatal("expected this chart's release forced while an object carries a stale endpoint") + } + if len(missing) != 1 || !strings.HasPrefix(missing[0], tc.key+" ") { + t.Fatalf("missing = %v, want only %s", missing, tc.key) + } + if ann := forceAnnotations(t, dyn); ann["reconcile.fluxcd.io/forceAt"] == "" { + t.Errorf("own HelmRelease not forced: %v", ann) + } + if ann := credentialsForceAnnotations(t, dyn); len(ann) != 0 { + t.Errorf("bucket release forced for a stale object: %v", ann) + } + }) + } +} + +// TestCheckIgnoresObjectEndpointOnExternalS3 pins that the comparison is off +// when no bucket release is configured: with provisionBucket=false the chart +// renders backupStorage.endpoint, not the Secret's, so the two legitimately +// differ and forcing would loop every MinForceInterval. +func TestCheckIgnoresObjectEndpointOnExternalS3(t *testing.T) { + objs := []runtime.Object{helmReleaseObject()} + for _, o := range allDefaultObjects() { + objs = append(objs, withEndpoint(o.(*unstructured.Unstructured), "https://minio.example.internal:9000")) + } + g, dyn := newGate(t, []client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()}, objs...) + g.CredentialsHelmRelease = types.NamespacedName{} + + missing, forced, err := g.Check(context.Background()) + if err != nil || forced || len(missing) != 0 { + t.Fatalf("missing = %v, forced = %v, err = %v; want a no-op", missing, forced, err) + } + if ann := forceAnnotations(t, dyn); len(ann) != 0 { + t.Errorf("own HelmRelease forced on external S3: %v", ann) + } +} + +// TestCheckIgnoresObjectEndpointWithoutSecretEndpoint covers a credentials +// Secret with no endpoint key: the chart then renders its values fallback, +// which the gate cannot reproduce, so there is nothing to compare against. +func TestCheckIgnoresObjectEndpointWithoutSecretEndpoint(t *testing.T) { + src := sourceSecret("bucket-1a2b") + delete(src.Data, "endpoint") + objs := append([]runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()}, allDefaultObjects()...) + g, dyn := newGate(t, []client.Object{src, cozyDefaultBackupClass()}, objs...) + + if _, forced, err := g.Check(context.Background()); err != nil || forced { + t.Fatalf("forced = %v, err = %v; want a no-op", forced, err) + } + if ann := forceAnnotations(t, dyn); len(ann) != 0 { + t.Errorf("own HelmRelease forced with no endpoint to compare against: %v", ann) + } +} + +// TestCheckForcesOnceForUnannotatedObjects covers objects rendered by a chart +// that predates the endpoint annotation. Their endpoint is unknown, so they +// are treated as stale until one forced render has gone through, which stamps +// the annotation. A force that did not go through (throttled) does not count, +// and once one has, an object still without the annotation (one the chart +// does not render) must not force again every MinForceInterval. +func TestCheckForcesOnceForUnannotatedObjects(t *testing.T) { + base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) + objs := []runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()} + for _, o := range allDefaultObjects() { + objs = append(objs, unannotated(o.(*unstructured.Unstructured))) + } + g, dyn := newGate(t, []client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()}, objs...) + g.lastForce = base.Add(-time.Minute) + g.now = func() time.Time { return base } + + if missing, forced, err := g.Check(context.Background()); err != nil || forced || len(missing) != 7 { + t.Fatalf("throttled Check: missing = %v, forced = %v, err = %v; want 7 reported, none forced", missing, forced, err) + } + + g.now = func() time.Time { return base.Add(5 * time.Minute) } + if _, forced, err := g.Check(context.Background()); err != nil || !forced { + t.Fatalf("first unthrottled Check: forced = %v, err = %v; want forced", forced, err) + } + first := forceAnnotations(t, dyn)["reconcile.fluxcd.io/forceAt"] + + g.now = func() time.Time { return base.Add(time.Hour) } + missing, forced, err := g.Check(context.Background()) + if err != nil || forced || len(missing) != 0 { + t.Fatalf("after the forced render: missing = %v, forced = %v, err = %v; want a no-op", missing, forced, err) + } + if again := forceAnnotations(t, dyn)["reconcile.fluxcd.io/forceAt"]; again != first { + t.Errorf("forced again for unannotated objects: %q, then %q", first, again) + } +} + +// TestCheckStillForcesStaleAfterUnannotatedForce pins that the once-only +// treatment covers the missing annotation alone: an annotation that names the +// wrong endpoint still forces after that. +func TestCheckStillForcesStaleAfterUnannotatedForce(t *testing.T) { + objs := []runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()} + for _, o := range allDefaultObjects() { + objs = append(objs, withEndpoint(o.(*unstructured.Unstructured), "https://s3.old.example.com")) + } + g, _ := newGate(t, []client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()}, objs...) + g.renderForced = true + + if _, forced, err := g.Check(context.Background()); err != nil || !forced { + t.Fatalf("forced = %v, err = %v; want forced", forced, err) + } +} + +// TestRenderedEndpointMatchesTheChart pins renderedEndpointFor to the +// "backupstrategy-controller.endpoint" helper, which the chart stamps as the +// annotation: trim, strip a scheme, force https://. Any divergence makes the +// gate see every object as stale and force this release forever. +func TestRenderedEndpointMatchesTheChart(t *testing.T) { + for _, tc := range []struct{ in, want string }{ + {"s3.example.com", "https://s3.example.com"}, + {"s3.example.com\n", "https://s3.example.com"}, + {" s3.example.com ", "https://s3.example.com"}, + {"https://s3.example.com", "https://s3.example.com"}, + {"http://s3.example.com:8333", "https://s3.example.com:8333"}, + {"s3.example.com/prefix/", "https://s3.example.com/prefix/"}, + // No endpoint in the Secret: the chart renders its values fallback, + // which the gate cannot reproduce, so it compares nothing. + {"", ""}, + } { + if got := renderedEndpointFor([]byte(tc.in)); got != tc.want { + t.Errorf("renderedEndpointFor(%q) = %q, want %q", tc.in, got, tc.want) + } + } +} + +// TestForcedReleaseForNamesTheStaleCredentialsRelease pins the release the +// log line names: a stale credentials Secret is repaired by the bucket +// release, like a missing one, and everything else by this chart's own. +func TestForcedReleaseForNamesTheStaleCredentialsRelease(t *testing.T) { + g, _ := newGate(t, nil) + for _, tc := range []struct { + missing []string + want types.NamespacedName + }{ + {[]string{"Secret/bucket-cozy-backups-system-credentials"}, g.CredentialsHelmRelease}, + {[]string{`Secret/bucket-cozy-backups-system-credentials (endpoint "s3.old", COSI Secret has "s3.new")`}, g.CredentialsHelmRelease}, + {[]string{"Secret/bucket-cozy-backups-system-credentials-other"}, g.HelmRelease}, + {[]string{`CNPG/cozy-default-cnpg (endpoint "https://s3.old", want "https://s3.new")`}, g.HelmRelease}, + } { + if got := g.forcedReleaseFor(tc.missing); got != tc.want { + t.Errorf("forcedReleaseFor(%v) = %v, want %v", tc.missing, got, tc.want) + } + } +} diff --git a/packages/system/backupstrategy-controller/templates/_helpers.tpl b/packages/system/backupstrategy-controller/templates/_helpers.tpl index 6665cc639..f64c77b88 100644 --- a/packages/system/backupstrategy-controller/templates/_helpers.tpl +++ b/packages/system/backupstrategy-controller/templates/_helpers.tpl @@ -101,8 +101,11 @@ https://. - provisionBucket: true + Secret missing (offline `helm template`/unit render, or pre-reconcile install where lookup returns nil) → fall back - to .Values.backupStorage.endpoint. On a live deploy Flux re-renders on - spec.interval once the Secret exists, promoting the derived endpoint. + to .Values.backupStorage.endpoint. Flux does not re-render on + spec.interval; the controller's DefaultObjectsGate forces the render + once the Secret exists, and again whenever the Secret's endpoint stops + matching the backups.cozystack.io/endpoint annotation every default + Strategy CR and the Velero BSL carry with this helper's output. Normalization of the decoded value: trim first (a trailing newline in the Secret data would otherwise ride into the URL and break it), then strip any diff --git a/packages/system/backupstrategy-controller/templates/deployment.yaml b/packages/system/backupstrategy-controller/templates/deployment.yaml index f35e97661..adc09fa2a 100644 --- a/packages/system/backupstrategy-controller/templates/deployment.yaml +++ b/packages/system/backupstrategy-controller/templates/deployment.yaml @@ -87,6 +87,13 @@ spec: value: {{ printf "bucket-%s-system" .Values.backupStorage.bucketName | quote }} - name: BACKUP_CREDENTIALS_HELMRELEASE_NAMESPACE value: {{ .Values.backupStorage.namespace | quote }} + # The COSI Secret that release renders the credentials Secret from + # (packages/system/bucket/templates/user-credentials.yaml appends + # "-credentials" to its name). The COSI sidecar writes it once per + # BucketAccess, so the gate compares the two endpoints and forces the + # release when the credentials Secret lags behind. + - name: BACKUP_CREDENTIALS_COSI_SECRET_NAME + value: {{ trimSuffix "-credentials" .Values.backupStorage.systemSecretName | quote }} {{- end }} {{- if .Values.velero.bslEnabled }} - name: BACKUP_DEFAULT_OBJECTS_VELERO_NAMESPACE diff --git a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml index 3397d6d0d..6d09d4aa9 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml @@ -18,6 +18,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: Altinity metadata: name: cozy-default-altinity + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: template: spec: diff --git a/packages/system/backupstrategy-controller/templates/strategy-cnpg-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-cnpg-default.yaml index 58dda61b1..bf34b2e0d 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-cnpg-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-cnpg-default.yaml @@ -5,6 +5,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: CNPG metadata: name: cozy-default-cnpg + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: template: {{- /* diff --git a/packages/system/backupstrategy-controller/templates/strategy-etcd-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-etcd-default.yaml index f335f04cf..5183af45b 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-etcd-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-etcd-default.yaml @@ -5,6 +5,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: Etcd metadata: name: cozy-default-etcd + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: template: destination: diff --git a/packages/system/backupstrategy-controller/templates/strategy-foundationdb-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-foundationdb-default.yaml index 061be2ffb..82f9ab6ca 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-foundationdb-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-foundationdb-default.yaml @@ -8,6 +8,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: FoundationDB metadata: name: cozy-default-foundationdb + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: template: blobStoreConfiguration: diff --git a/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml index c65462295..1f14465ed 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml @@ -23,6 +23,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: Kafka metadata: name: cozy-default-kafka + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: artifactURITemplate: {{ printf "s3://%s/{{ .Release.Namespace }}/{{ .Release.Name }}/{{ .BackupName }}/kafka-metadata.txt" $bucketName | quote }} template: diff --git a/packages/system/backupstrategy-controller/templates/strategy-mariadb-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-mariadb-default.yaml index ec8b35160..7615a8b0d 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-mariadb-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-mariadb-default.yaml @@ -7,6 +7,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: MariaDB metadata: name: cozy-default-mariadb + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: template: storage: diff --git a/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml index 7509f4bf7..1bddbde16 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml @@ -54,6 +54,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: MongoDB metadata: name: cozy-default-mongodb + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: template: storageName: s3-storage diff --git a/packages/system/backupstrategy-controller/templates/strategy-rabbitmq-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-rabbitmq-default.yaml index 9c303a6d3..9515945d2 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-rabbitmq-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-rabbitmq-default.yaml @@ -26,6 +26,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: Rabbitmq metadata: name: cozy-default-rabbitmq + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: # The driver defers the run until the RabbitmqCluster reports AllReplicasReady # (a Go-side precondition), so a not-yet-ready broker surfaces a precise diff --git a/packages/system/backupstrategy-controller/templates/strategy-redis-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-redis-default.yaml index 3db27f7e2..dc930f394 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-redis-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-redis-default.yaml @@ -21,6 +21,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: Redis metadata: name: cozy-default-redis + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: template: spec: diff --git a/packages/system/backupstrategy-controller/templates/strategy-velero-vmdisk-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-velero-vmdisk-default.yaml index 4511fbc58..1c1a7b9b5 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-velero-vmdisk-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-velero-vmdisk-default.yaml @@ -5,6 +5,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: Velero metadata: name: cozy-default-velero-vmdisk + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: template: restoreSpec: diff --git a/packages/system/backupstrategy-controller/templates/strategy-velero-vminstance-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-velero-vminstance-default.yaml index b1f919125..9666a7cce 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-velero-vminstance-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-velero-vminstance-default.yaml @@ -5,6 +5,8 @@ apiVersion: strategy.backups.cozystack.io/v1alpha1 kind: Velero metadata: name: cozy-default-velero-vminstance + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: template: restoreSpec: diff --git a/packages/system/backupstrategy-controller/templates/velero-bsl.yaml b/packages/system/backupstrategy-controller/templates/velero-bsl.yaml index e0ba05ad8..e50f3a1fc 100644 --- a/packages/system/backupstrategy-controller/templates/velero-bsl.yaml +++ b/packages/system/backupstrategy-controller/templates/velero-bsl.yaml @@ -12,6 +12,8 @@ kind: BackupStorageLocation metadata: name: cozy-default namespace: {{ .Values.velero.namespace }} + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} spec: provider: aws # default=false: do not silently displace an admin-provisioned default diff --git a/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml b/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml index 976fee166..ed616ce79 100644 --- a/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml +++ b/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml @@ -11,6 +11,8 @@ suite: DefaultObjectsGate wiring — the release the gate forces, and its RBAC templates: - templates/deployment.yaml - templates/rbac.yaml + - templates/strategy-cnpg-default.yaml + - templates/velero-bsl.yaml release: name: backupstrategy-controller @@ -61,6 +63,16 @@ tests: content: name: BACKUP_CREDENTIALS_HELMRELEASE_NAMESPACE value: tenant-root + # The COSI Secret the bucket release renders the credentials Secret + # from is the BucketAccess credentialsSecretName, -system, + # which is also BACKUP_STORAGE_SECRET_NAME minus "-credentials" + # (packages/system/bucket/templates/user-credentials.yaml). The gate + # compares the two endpoints. + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_CREDENTIALS_COSI_SECRET_NAME + value: bucket-cozy-backups-system # BACKUP_STORAGE_SECRET_NAME is exactly the release name above with # "-credentials" appended — that is what makes forcing this release the # thing that creates that Secret. @@ -86,6 +98,11 @@ tests: content: name: BACKUP_STORAGE_SECRET_NAME value: bucket-platform-backups-system-credentials + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_CREDENTIALS_COSI_SECRET_NAME + value: bucket-platform-backups-system # External S3: the Secret is admin-managed and no release renders it, so # there is nothing to force. Handing the gate stale coordinates would make @@ -105,6 +122,11 @@ tests: content: name: BACKUP_CREDENTIALS_HELMRELEASE_NAMESPACE value: tenant-root + - notContains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_CREDENTIALS_COSI_SECRET_NAME + value: bucket-cozy-backups-system - it: "velero.bslEnabled=false: the BSL is not rendered, so the gate must not look for it" template: templates/deployment.yaml @@ -154,3 +176,41 @@ tests: apiGroups: ["velero.io"] resources: ["backupstoragelocations"] verbs: ["get"] + + # The gate compares backups.cozystack.io/endpoint with the endpoint it + # derives from the credentials Secret (renderedEndpointFor in + # internal/backupcontroller/default_objects_gate.go, whose test uses the + # same input). Any divergence would make every object look stale and force + # this release every MinForceInterval. + - it: "objects carry the endpoint derived from the credentials Secret as their endpoint annotation" + set: + backupStorage.bucketNameOverride: test-bucket + kubernetesProvider: + scheme: + "v1/Secret": + gvr: + version: v1 + resource: secrets + namespaced: true + objects: + - apiVersion: v1 + kind: Secret + metadata: + name: bucket-cozy-backups-system-credentials + namespace: tenant-root + data: + # "http://s3.example.com:8333\n" + endpoint: aHR0cDovL3MzLmV4YW1wbGUuY29tOjgzMzMK + asserts: + - equal: + path: metadata.annotations["backups.cozystack.io/endpoint"] + value: https://s3.example.com:8333 + template: templates/strategy-cnpg-default.yaml + - equal: + path: spec.template.barmanObjectStore.endpointURL + value: https://s3.example.com:8333 + template: templates/strategy-cnpg-default.yaml + - equal: + path: metadata.annotations["backups.cozystack.io/endpoint"] + value: https://s3.example.com:8333 + template: templates/velero-bsl.yaml diff --git a/packages/system/backupstrategy-controller/values.yaml b/packages/system/backupstrategy-controller/values.yaml index 6cf39c65a..d87c978e6 100644 --- a/packages/system/backupstrategy-controller/values.yaml +++ b/packages/system/backupstrategy-controller/values.yaml @@ -162,8 +162,15 @@ backupStorage: # credentials Secret is missing it forces the bucket's -system # release; once the bucket name resolves it checks that every object # cozy-default routes to exists and forces THIS release when any is - # missing. Both are throttled, independently, and a no-op in the steady - # state. A suspended release is skipped rather than re-stamped forever. The + # missing. It repairs a stale S3 endpoint the same way: when the + # credentials Secret's endpoint differs from the COSI Secret it is rendered + # from it forces the bucket's -system release, and when the + # backups.cozystack.io/endpoint annotation on the Velero BSL or a default + # Strategy differs from the credentials Secret's endpoint it forces THIS + # release. The COSI Secret itself keeps the endpoint it was granted with, so + # after a publishing.host change that endpoint follows only once the + # BucketAccess is re-granted. Both are throttled, independently, and a no-op + # in the steady state. A suspended release is skipped rather than re-stamped forever. The # gate does NOT create the objects itself: their bodies are values-driven # and Helm stays their single owner. # diff --git a/packages/system/bucket/templates/user-credentials.yaml b/packages/system/bucket/templates/user-credentials.yaml index 6c2a14bd1..a45320254 100644 --- a/packages/system/bucket/templates/user-credentials.yaml +++ b/packages/system/bucket/templates/user-credentials.yaml @@ -27,7 +27,9 @@ So the repair lives outside the chart, where it is per-object and costs a throttled annotation patch: backupstrategy-controller's DefaultObjectsGate (internal/backupcontroller/default_objects_gate.go) forces this release - when the platform bucket's credentials Secret is absent. That covers the + when the platform bucket's credentials Secret is absent, or when its + endpoint differs from the COSI Secret's (a BucketAccess re-granted after a + publishing.host change). That covers the cluster-breaking case — the cozy-backups Secret feeds the credentials projector, hence every default BackupClass strategy, Velero, and migration 50. Tenant buckets keep the pre-existing behaviour; generalising the gate From 4f3f1a5cadab5e99c0193efd496b5d5aac673098 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:26:23 +0300 Subject: [PATCH 109/518] fix(clickhouse): stop the backup sidecar from publishing failed attempts create_remote keeps its local backup when the upload fails, and with the upstream defaults the sidecar never prunes local backups and resumes every unfinished upload when it starts. A strategy attempt that fails mid-upload therefore lands in the bucket on the next sidecar restart as an archive no Backup refers to, and restore can pick it as the newest one. The resumed upload also holds the API lock, so the strategy Pods racing it get 423 and burn the Job's retries. Turning the resume off keeps a failed attempt out of the bucket, and a negative local retention leaves at most the newest local backup and drops it once its upload succeeds, so leftovers no longer accumulate on the data volume. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../apps/clickhouse/templates/clickhouse.yaml | 12 ++++++ .../apps/clickhouse/tests/backup_test.yaml | 39 +++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/packages/apps/clickhouse/templates/clickhouse.yaml b/packages/apps/clickhouse/templates/clickhouse.yaml index 827b5d579..571469bfe 100644 --- a/packages/apps/clickhouse/templates/clickhouse.yaml +++ b/packages/apps/clickhouse/templates/clickhouse.yaml @@ -222,6 +222,18 @@ spec: key: backup - name: ALLOW_EMPTY_BACKUPS value: "true" + {{- /* + create_remote keeps the local backup when its upload fails, + and the upstream defaults never prune local backups (0) and + resume unfinished uploads at start, which pushes a failed + attempt to the bucket as an archive no Backup refers to and + holds the API lock (423) while it does. -1 keeps only the + newest local backup and removes it once its upload succeeds. + */}} + - name: BACKUPS_TO_KEEP_LOCAL + value: "-1" + - name: API_COMPLETE_RESUMABLE_AFTER_RESTART + value: "false" - name: REMOTE_STORAGE value: s3 {{- /* diff --git a/packages/apps/clickhouse/tests/backup_test.yaml b/packages/apps/clickhouse/tests/backup_test.yaml index 66eda4dc8..90bcd1066 100644 --- a/packages/apps/clickhouse/tests/backup_test.yaml +++ b/packages/apps/clickhouse/tests/backup_test.yaml @@ -292,3 +292,42 @@ tests: - notMatchRegex: path: stringData["backup.sh"] pattern: 'echo DB_LIST=' + + # A strategy attempt that fails mid-upload leaves its local backup behind, + # and with the upstream defaults the sidecar would keep it forever and push + # it to the bucket on its next start as an archive no Backup refers to. The + # sidecar must neither resume uploads at start nor keep local copies once an + # upload succeeds, on either S3 flow. + - it: "sidecar drops local backups after upload and does not resume uploads at start" + template: templates/clickhouse.yaml + documentSelector: + path: kind + value: ClickHouseInstallation + set: + backup: + enabled: true + useSystemBucket: true + asserts: + - equal: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="BACKUPS_TO_KEEP_LOCAL")].value + value: "-1" + - equal: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="API_COMPLETE_RESUMABLE_AFTER_RESTART")].value + value: "false" + + - it: "legacy flow: sidecar drops local backups after upload and does not resume uploads at start" + template: templates/clickhouse.yaml + documentSelector: + path: kind + value: ClickHouseInstallation + set: + backup: + enabled: true + useSystemBucket: false + asserts: + - equal: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="BACKUPS_TO_KEEP_LOCAL")].value + value: "-1" + - equal: + path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="API_COMPLETE_RESUMABLE_AFTER_RESTART")].value + value: "false" From a25db5aeb6469d1fe5df473479108dd2a84e6911 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:26:23 +0300 Subject: [PATCH 110/518] fix(backups): wait for an idle ClickHouse backup sidecar before starting The sidecar refuses a new operation with 423 while another one is in progress, but the strategy only waited for the API to answer, and went on to start its operation even when it never did. An operation already running in the sidecar therefore failed the attempt within seconds. Poll the action log until nothing is in progress, and fail the Pod instead of proceeding when the sidecar stays busy or unreachable, so the failure names the cause rather than surfacing as a 423 or a connection error from the operation itself. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/altinity-backup-strategy.bats | 149 ++++++++++++++++++ .../templates/strategy-altinity-default.yaml | 13 +- 2 files changed, 161 insertions(+), 1 deletion(-) create mode 100644 hack/altinity-backup-strategy.bats diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats new file mode 100644 index 000000000..61729bf00 --- /dev/null +++ b/hack/altinity-backup-strategy.bats @@ -0,0 +1,149 @@ +#!/usr/bin/env bats +# Behavioural tests for the shipped ClickHouse (Altinity) strategy script, +# extracted from the rendered cozy-default-altinity Strategy CR and EXECUTED +# against a stubbed clickhouse-backup sidecar API. +# +# The contract pinned here is the start gate: the sidecar serialises its +# operations and answers any new one with HTTP 423 while another is "in +# progress", so a strategy Pod that starts its operation without waiting for +# an idle sidecar burns a Job attempt in seconds. The gate has to fail the Pod +# rather than proceed when the sidecar never becomes idle. +# +# Run from the repo root with make bats-unit-tests. The rendered script is run +# with `bash`, not `sh`: it opens `set -euo pipefail`, which the production +# image's busybox ash supports but the CI runner's /bin/sh (dash) rejects. + +load test_helper + +CHART=packages/system/backupstrategy-controller + +# Render the shipped strategy's in-Pod script into $1 with the controller-render +# placeholders substituted ($2 mode, $3 application name). An empty render, or +# a placeholder left unsubstituted, is an early failure: these tests EXECUTE it. +render_script() { + out=$1; mode=$2; app=$3 + helm template t "$CHART" -n cozy-backup-controller \ + --set backupStorage.bucketNameOverride=test-bucket 2>/dev/null \ + | yq eval 'select(.kind == "Altinity") | .spec.template.spec.containers[0].args[0]' - \ + | sed -e "s#{{ .Mode }}#${mode}#g" \ + -e "s#{{ .Release.Name }}#${app}#g" \ + -e "s#{{ .Release.Namespace }}#tenant-x#g" \ + -e "s#{{ .Backup.ApplicationRef.Name }}#${app}#g" > "$out" + [ -s "$out" ] || return 1 + if grep -q '{{' "$out"; then + echo "rendered script still holds a controller placeholder" >&2 + return 1 + fi +} + +# A curl stub standing in for the sidecar API, put first on PATH next to a +# no-op sleep. State lives in $1: +# unreachable present: every request fails as a refused connection (exit 7) +# busy a number: that many GET /backup/actions answers still list an +# operation "in progress"; while it is positive a POST is +# refused the way the sidecar refuses it (423, curl -f exit 22) +# posted the operation the script started, once it started one +# requests one line per request: method and URL +make_stubs() { + bin=$1; state=$2 + mkdir -p "$bin" "$state" + cat > "$bin/curl" <> "\$state/requests" +if [ -e "\$state/unreachable" ]; then exit 7; fi +busy=\$(cat "\$state/busy" 2>/dev/null || echo 0) +case "\$method \$url" in + "POST "*) + if [ "\$busy" -gt 0 ]; then exit 22; fi + echo "\${url##*/}" > "\$state/posted" + ;; + "GET "*/backup/actions) + echo '{"command":"create_remote clickhouse-myapp-20260101T000000Z","status":"error","start":"2026-01-01 00:00:00","error":"old"}' + if [ "\$busy" -gt 0 ]; then + echo \$((busy - 1)) > "\$state/busy" + echo '{"command":"upload --resumable=1 clickhouse-myapp-20260101T000000Z","status":"in progress","start":"2026-01-01 00:01:00"}' + elif [ -s "\$state/posted" ]; then + echo "{\"command\":\"\$(cat "\$state/posted")\",\"status\":\"success\",\"start\":\"2026-01-01 00:02:00\"}" + fi + ;; + "GET "*/backup/list/remote) + echo '{"name":"clickhouse-myapp-20260101T000000Z","created":"2026-01-01 00:00:00"}' + ;; +esac +exit 0 +STUB + cat > "$bin/sleep" <<'STUB' +#!/bin/sh +exit 0 +STUB + chmod 0755 "$bin/curl" "$bin/sleep" +} + +run_script() { + tmp=$1 + API_USERNAME=u API_PASSWORD=p PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" +} + +@test "an empty render fails the fixture instead of passing as a no-op" { + tmp=$(mktemp -d) + CHART=$tmp/no-such-chart + if render_script "$tmp/s.sh" backup myapp 2>/dev/null; then + echo "FAIL: render_script reported success with no chart" + false + fi +} + +@test "backup on an idle sidecar starts create_remote and waits for its success" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + render_script "$tmp/s.sh" backup myapp + run_script "$tmp" + grep -q '^POST .*/backup/create_remote?name=clickhouse-myapp-' "$tmp/state/requests" +} + +@test "backup waits until the running operation finishes before create_remote" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo 3 > "$tmp/state/busy" + render_script "$tmp/s.sh" backup myapp + run_script "$tmp" + grep -q '^POST .*/backup/create_remote?name=clickhouse-myapp-' "$tmp/state/requests" + # All three busy answers were consumed before the operation was started. + [ "$(cat "$tmp/state/busy")" = 0 ] +} + +@test "backup fails without starting create_remote when the sidecar stays busy" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo 1000 > "$tmp/state/busy" + render_script "$tmp/s.sh" backup myapp + if run_script "$tmp"; then echo "FAIL: a busy sidecar must fail the Pod"; false; fi + if grep -q '^POST ' "$tmp/state/requests"; then echo "FAIL: no operation may start on a busy sidecar"; false; fi +} + +@test "backup fails without starting create_remote when the sidecar never answers" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + touch "$tmp/state/unreachable" + render_script "$tmp/s.sh" backup myapp + if run_script "$tmp"; then echo "FAIL: an unreachable sidecar must fail the Pod"; false; fi + if grep -q '^POST ' "$tmp/state/requests"; then echo "FAIL: no operation may start on an unreachable sidecar"; false; fi +} + +@test "restore fails without starting restore_remote when the sidecar stays busy" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo 1000 > "$tmp/state/busy" + render_script "$tmp/s.sh" restore myapp + if run_script "$tmp"; then echo "FAIL: a busy sidecar must fail the Pod"; false; fi + if grep -q '^POST ' "$tmp/state/requests"; then echo "FAIL: no restore may start on a busy sidecar"; false; fi +} diff --git a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml index 3397d6d0d..560b1e597 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml @@ -51,10 +51,21 @@ spec: host="chi-clickhouse-{{ printf "{{ .Release.Name }}" }}-clickhouse-0-0.{{ printf "{{ .Release.Namespace }}" }}.svc" api="http://${host}:7171" auth="-u ${API_USERNAME}:${API_PASSWORD}" + # The sidecar answers 423 to a new operation while any other is + # in progress, so starting before it is idle burns the attempt. + idle= for _ in $(seq 1 60); do - if curl -fsS ${auth} --max-time 2 "${api}/backup/status" >/dev/null 2>&1; then break; fi + if actions=$(curl -fsS ${auth} --max-time 5 "${api}/backup/actions") \ + && [ "$(printf '%s\n' "${actions}" | jq -rs '[.[] | select(.status == "in progress")] | length')" = 0 ]; then + idle=1 + break + fi sleep 5 done + if [ -z "${idle}" ]; then + echo "clickhouse-backup sidecar at ${api} did not become idle" >&2 + exit 1 + fi if [ "{{ printf "{{ .Mode }}" }}" = "backup" ]; then prefix="clickhouse-{{ printf "{{ .Release.Name }}" }}-" target="${prefix}$(date -u +%Y%m%dT%H%M%SZ)" From f36d864e99d0fcdad7b8d6433705c7d6286367c8 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:05:58 +0300 Subject: [PATCH 111/518] fix(seaweedfs): run the worker that enforces bucket lifecycle rules SeaweedFS accepts and stores a bucket lifecycle configuration, but only the s3_lifecycle job of its plugin worker acts on it, and that job is scheduled by the admin server. Neither ran here, so a rule returned 200 and no object ever expired, on the platform backup bucket and on tenant buckets alike, and the documented way to bound backup storage did nothing. Both now run in Simple and MultiZone instances, behind lifecycle.enabled (on by default). A Client instance has no data plane of its own; its rules are enforced where the master it connects to runs. The worker registers for s3_lifecycle alone, so enabling lifecycle does not also start vacuum, balancing or erasure coding. The admin keeps its job state on a small volume (on volume.storageClass when set), and its UI gets a generated password, because a tenant namespace admits traffic from the whole cluster and the server is unauthenticated without one. The data-plane chart keeps both off unless its parent turns them on, so it never starts an admin without that password. Existing rules start applying minutes after the upgrade: the admin's scheduler runs the first pass within one idle cycle of its lifecycle lane, then once a day, and objects the rules have already expired are deleted on that first pass. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- packages/extra/seaweedfs/README.md | 2 + .../templates/admin-credentials.yaml | 21 ++ .../extra/seaweedfs/templates/seaweedfs.yaml | 28 +++ .../extra/seaweedfs/tests/lifecycle_test.yaml | 196 ++++++++++++++++++ packages/extra/seaweedfs/values.schema.json | 12 ++ packages/extra/seaweedfs/values.yaml | 7 + .../seaweedfs-rd/cozyrds/seaweedfs.yaml | 4 +- .../seaweedfs/tests/lifecycle_test.yaml | 138 ++++++++++++ packages/system/seaweedfs/values.yaml | 41 ++++ 9 files changed, 447 insertions(+), 2 deletions(-) create mode 100644 packages/extra/seaweedfs/templates/admin-credentials.yaml create mode 100644 packages/extra/seaweedfs/tests/lifecycle_test.yaml create mode 100644 packages/system/seaweedfs/tests/lifecycle_test.yaml diff --git a/packages/extra/seaweedfs/README.md b/packages/extra/seaweedfs/README.md index e269992bf..c0866722b 100644 --- a/packages/extra/seaweedfs/README.md +++ b/packages/extra/seaweedfs/README.md @@ -78,4 +78,6 @@ | `s3.resources.cpu` | Number of CPU cores allocated. | `quantity` | `""` | | `s3.resources.memory` | Amount of memory allocated. | `quantity` | `""` | | `s3.resourcesPreset` | Default sizing preset used when `resources` is omitted. | `string` | `t1.small` | +| `lifecycle` | S3 bucket lifecycle configuration. | `object` | `{}` | +| `lifecycle.enabled` | Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. | `bool` | `true` | diff --git a/packages/extra/seaweedfs/templates/admin-credentials.yaml b/packages/extra/seaweedfs/templates/admin-credentials.yaml new file mode 100644 index 000000000..4b9f6dd33 --- /dev/null +++ b/packages/extra/seaweedfs/templates/admin-credentials.yaml @@ -0,0 +1,21 @@ +{{- if and .Values.lifecycle.enabled (ne .Values.topology "Client") }} +{{- $name := printf "%s-admin-credentials" .Release.Name }} +{{- /* The admin reads these into its environment when it starts and nothing + restarts it on a Secret change, so a password regenerated on every render + would leave the Secret holding one the running server does not accept. + Keep the one already issued. */}} +{{- $password := randAlphaNum 40 | b64enc }} +{{- with lookup "v1" "Secret" .Release.Namespace $name }} +{{- with (dig "data" "adminPassword" "" .) }} +{{- $password = . }} +{{- end }} +{{- end }} +apiVersion: v1 +kind: Secret +metadata: + name: {{ $name }} +type: Opaque +data: + adminUser: {{ "admin" | b64enc }} + adminPassword: {{ $password }} +{{- end }} diff --git a/packages/extra/seaweedfs/templates/seaweedfs.yaml b/packages/extra/seaweedfs/templates/seaweedfs.yaml index 27e5557e0..c111a80c5 100644 --- a/packages/extra/seaweedfs/templates/seaweedfs.yaml +++ b/packages/extra/seaweedfs/templates/seaweedfs.yaml @@ -370,6 +370,16 @@ spec: secretName: {{ .Release.Name }}-s3-ingress-tls {{- end }} resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.s3.resourcesPreset .Values.s3.resources $) | nindent 10 }} + admin: + enabled: {{ .Values.lifecycle.enabled }} + secret: + existingSecret: {{ .Release.Name }}-admin-credentials + {{- with .Values.volume.storageClass }} + data: + storageClass: {{ . }} + {{- end }} + worker: + enabled: {{ .Values.lifecycle.enabled }} cosi: driverName: "{{ .Release.Namespace }}.seaweedfs.objectstorage.k8s.io" bucketClassName: "{{ .Release.Namespace }}" @@ -502,4 +512,22 @@ spec: app.kubernetes.io/component: s3 app.kubernetes.io/name: seaweedfs version: {{ $.Chart.Version }} +{{- if .Values.lifecycle.enabled }} +{{- range $component := list "admin" "worker" }} +--- +apiVersion: cozystack.io/v1alpha1 +kind: WorkloadMonitor +metadata: + name: {{ $.Release.Name }}-{{ $component }} +spec: + replicas: 1 + minReplicas: 1 + kind: seaweedfs + type: {{ $component }} + selector: + app.kubernetes.io/component: {{ $component }} + app.kubernetes.io/name: seaweedfs + version: {{ $.Chart.Version }} +{{- end }} +{{- end }} {{- end }} diff --git a/packages/extra/seaweedfs/tests/lifecycle_test.yaml b/packages/extra/seaweedfs/tests/lifecycle_test.yaml new file mode 100644 index 000000000..a040e0ae6 --- /dev/null +++ b/packages/extra/seaweedfs/tests/lifecycle_test.yaml @@ -0,0 +1,196 @@ +suite: seaweedfs lifecycle switch reaches the data plane + +# lifecycle.enabled decides whether the data-plane release runs the admin +# server and the worker that enforce bucket lifecycle rules. Both flags are +# sent either way, so the data-plane chart's own default never decides. + +templates: + - templates/seaweedfs.yaml + - templates/admin-credentials.yaml + +release: + name: seaweedfs + namespace: tenant-root + +tests: + - it: runs the lifecycle admin and worker by default + template: templates/seaweedfs.yaml + set: &platform + _namespace: + host: example.org + ingress: tenant-root + _cluster: + issuer-name: letsencrypt-prod + solver: http01 + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - equal: + path: spec.values.seaweedfs.admin.enabled + value: true + - equal: + path: spec.values.seaweedfs.worker.enabled + value: true + - equal: + path: spec.values.seaweedfs.admin.secret.existingSecret + value: seaweedfs-admin-credentials + + - it: turns both off when lifecycle is disabled + template: templates/seaweedfs.yaml + set: + <<: *platform + lifecycle: + enabled: false + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - equal: + path: spec.values.seaweedfs.admin.enabled + value: false + - equal: + path: spec.values.seaweedfs.worker.enabled + value: false + + # The admin's volume follows volume.storageClass when the instance sets one. + # Zone storage classes are not followed: the admin is not pinned to a zone. + - it: puts the admin volume on the instance's storage class + template: templates/seaweedfs.yaml + set: + <<: *platform + volume: + storageClass: replicated + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - equal: + path: spec.values.seaweedfs.admin.data.storageClass + value: replicated + + - it: leaves the admin volume on the cluster default when the instance names none + template: templates/seaweedfs.yaml + set: *platform + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - notExists: + path: spec.values.seaweedfs.admin.data + + - it: monitors the admin and the worker while lifecycle is on + template: templates/seaweedfs.yaml + set: *platform + asserts: + - containsDocument: + kind: WorkloadMonitor + apiVersion: cozystack.io/v1alpha1 + name: seaweedfs-admin + any: true + - containsDocument: + kind: WorkloadMonitor + apiVersion: cozystack.io/v1alpha1 + name: seaweedfs-worker + any: true + - equal: + path: spec.selector + value: + app.kubernetes.io/component: worker + app.kubernetes.io/name: seaweedfs + documentSelector: + path: metadata.name + value: seaweedfs-worker + - equal: + path: spec.selector + value: + app.kubernetes.io/component: admin + app.kubernetes.io/name: seaweedfs + documentSelector: + path: metadata.name + value: seaweedfs-admin + + - it: does not monitor them when lifecycle is off + template: templates/seaweedfs.yaml + set: + <<: *platform + lifecycle: + enabled: false + asserts: + - containsDocument: + kind: WorkloadMonitor + apiVersion: cozystack.io/v1alpha1 + name: seaweedfs-admin + any: true + not: true + - containsDocument: + kind: WorkloadMonitor + apiVersion: cozystack.io/v1alpha1 + name: seaweedfs-worker + any: true + not: true + + - it: generates admin credentials for a new instance + template: templates/admin-credentials.yaml + set: *platform + asserts: + - isKind: + of: Secret + - equal: + path: metadata.name + value: seaweedfs-admin-credentials + - equal: + path: data.adminUser + value: YWRtaW4= + - matchRegex: + path: data.adminPassword + pattern: '^.{40,}$' + + # A new password on every render would leave the running admin on the old + # one, since it reads the Secret into its environment once at start. + - it: keeps the password of an existing Secret + template: templates/admin-credentials.yaml + set: *platform + kubernetesProvider: + scheme: + "v1/Secret": + gvr: + version: v1 + resource: secrets + namespaced: true + objects: + - apiVersion: v1 + kind: Secret + metadata: + name: seaweedfs-admin-credentials + namespace: tenant-root + data: + adminUser: YWRtaW4= + adminPassword: a2VwdA== + asserts: + - equal: + path: data.adminPassword + value: a2VwdA== + + - it: renders no credentials when lifecycle is off + template: templates/admin-credentials.yaml + set: + <<: *platform + lifecycle: + enabled: false + asserts: + - hasDocuments: + count: 0 + + # A Client instance runs no data plane of its own; its rules are enforced + # where the master it points at runs. + - it: renders no credentials for a Client instance + template: templates/admin-credentials.yaml + set: + <<: *platform + topology: Client + filer: + grpcHost: seaweedfs-filer.tenant-other.svc + asserts: + - hasDocuments: + count: 0 diff --git a/packages/extra/seaweedfs/values.schema.json b/packages/extra/seaweedfs/values.schema.json index ab9b953ac..9ce4b184a 100644 --- a/packages/extra/seaweedfs/values.schema.json +++ b/packages/extra/seaweedfs/values.schema.json @@ -868,6 +868,18 @@ ] } } + }, + "lifecycle": { + "description": "S3 bucket lifecycle configuration.", + "type": "object", + "default": {}, + "properties": { + "enabled": { + "description": "Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to.", + "type": "boolean", + "default": true + } + } } } } diff --git a/packages/extra/seaweedfs/values.yaml b/packages/extra/seaweedfs/values.yaml index 0aa569bc4..4f47c4fcc 100644 --- a/packages/extra/seaweedfs/values.yaml +++ b/packages/extra/seaweedfs/values.yaml @@ -192,3 +192,10 @@ s3: replicas: 2 resources: {} resourcesPreset: "t1.small" + +## @typedef {struct} Lifecycle - Enforcement of S3 bucket lifecycle rules. +## @field {bool} [enabled] - Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. + +## @param {Lifecycle} [lifecycle] - S3 bucket lifecycle configuration. +lifecycle: + enabled: true diff --git a/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml b/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml index 1fa5ad336..ddf5834e0 100644 --- a/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml +++ b/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml @@ -8,7 +8,7 @@ spec: singular: seaweedfs plural: seaweedfses openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"host":{"description":"The hostname used to access SeaweedFS externally (defaults to 's3' subdomain for the tenant host).","type":"string","default":""},"topology":{"description":"The topology of the SeaweedFS cluster.","type":"string","default":"Simple","enum":["Simple","MultiZone","Client"]},"replicationFactor":{"description":"Replication factor: number of replicas for each volume in the SeaweedFS cluster.","type":"integer","default":2},"db":{"description":"Database configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of database replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""}}},"master":{"description":"Master service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of master replicas.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"volumeSizeLimitMB":{"description":"Size in MB at which a volume stops taking writes, and the divisor that sets how many buckets a volume server can serve. Lower it to fit more buckets on the same disk, at the cost of more volume files for the same stored data; on a large volume it is worth raising instead. Changing it on an instance that already holds data costs something either way: lowering it stops writes to every bucket whose volumes all reach the new value until that bucket can grow, and raising it takes the room new buckets grow into, so run the check in docs/operations/seaweedfs-lowering-the-volume-size-limit.md first. Must be between 1 and 30000. A value close to or above `volume.size` leaves each volume server a single slot rather than failing, since the server floors its slot count at one.","type":"integer","default":400,"maximum":30000,"minimum":1}}},"filer":{"description":"Filer service configuration.","type":"object","default":{},"properties":{"grpcHost":{"description":"The hostname used to expose or access the filer service externally.","type":"string","default":""},"grpcPort":{"description":"The port used to access the filer service externally.","type":"integer","default":443},"replicas":{"description":"Number of filer replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"whitelist":{"description":"A list of IP addresses or CIDR ranges that are allowed to access the filer service.","type":"array","default":[],"items":{"type":"string"}}}},"volume":{"description":"Volume service configuration.","type":"object","default":{},"properties":{"diskType":{"description":"SeaweedFS disk type tag for the default volume servers (e.g., \"hdd\", \"ssd\").","type":"string","default":""},"pools":{"description":"A map of storage pools. Each pool creates a separate Volume StatefulSet with its own disk type.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of volume replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""},"zones":{"description":"A map of zones for MultiZone topology. Each zone can have its own number of replicas and size.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"dataCenter":{"description":"SeaweedFS data center name for this zone. Defaults to the zone name.","type":"string"},"nodeSelector":{"description":"YAML nodeSelector for this zone (default: topology.kubernetes.io/zone: ).","type":"string"},"pools":{"description":"A map of storage pools for this zone. Each pool creates a separate Volume StatefulSet per zone.","type":"object","additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of replicas in the zone.","type":"integer"},"size":{"description":"Zone storage size.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store zone data. Defaults to volume.storageClass.","type":"string"}}}}}},"s3":{"description":"S3 service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of S3 replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}} + {"title":"Chart Values","type":"object","properties":{"host":{"description":"The hostname used to access SeaweedFS externally (defaults to 's3' subdomain for the tenant host).","type":"string","default":""},"topology":{"description":"The topology of the SeaweedFS cluster.","type":"string","default":"Simple","enum":["Simple","MultiZone","Client"]},"replicationFactor":{"description":"Replication factor: number of replicas for each volume in the SeaweedFS cluster.","type":"integer","default":2},"db":{"description":"Database configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of database replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""}}},"master":{"description":"Master service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of master replicas.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"volumeSizeLimitMB":{"description":"Size in MB at which a volume stops taking writes, and the divisor that sets how many buckets a volume server can serve. Lower it to fit more buckets on the same disk, at the cost of more volume files for the same stored data; on a large volume it is worth raising instead. Changing it on an instance that already holds data costs something either way: lowering it stops writes to every bucket whose volumes all reach the new value until that bucket can grow, and raising it takes the room new buckets grow into, so run the check in docs/operations/seaweedfs-lowering-the-volume-size-limit.md first. Must be between 1 and 30000. A value close to or above `volume.size` leaves each volume server a single slot rather than failing, since the server floors its slot count at one.","type":"integer","default":400,"maximum":30000,"minimum":1}}},"filer":{"description":"Filer service configuration.","type":"object","default":{},"properties":{"grpcHost":{"description":"The hostname used to expose or access the filer service externally.","type":"string","default":""},"grpcPort":{"description":"The port used to access the filer service externally.","type":"integer","default":443},"replicas":{"description":"Number of filer replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"whitelist":{"description":"A list of IP addresses or CIDR ranges that are allowed to access the filer service.","type":"array","default":[],"items":{"type":"string"}}}},"volume":{"description":"Volume service configuration.","type":"object","default":{},"properties":{"diskType":{"description":"SeaweedFS disk type tag for the default volume servers (e.g., \"hdd\", \"ssd\").","type":"string","default":""},"pools":{"description":"A map of storage pools. Each pool creates a separate Volume StatefulSet with its own disk type.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of volume replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""},"zones":{"description":"A map of zones for MultiZone topology. Each zone can have its own number of replicas and size.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"dataCenter":{"description":"SeaweedFS data center name for this zone. Defaults to the zone name.","type":"string"},"nodeSelector":{"description":"YAML nodeSelector for this zone (default: topology.kubernetes.io/zone: ).","type":"string"},"pools":{"description":"A map of storage pools for this zone. Each pool creates a separate Volume StatefulSet per zone.","type":"object","additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of replicas in the zone.","type":"integer"},"size":{"description":"Zone storage size.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store zone data. Defaults to volume.storageClass.","type":"string"}}}}}},"s3":{"description":"S3 service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of S3 replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"lifecycle":{"description":"S3 bucket lifecycle configuration.","type":"object","default":{},"properties":{"enabled":{"description":"Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to.","type":"boolean","default":true}}}}} release: prefix: "" labels: @@ -26,7 +26,7 @@ spec: description: Seaweedfs module: true icon: PHN2ZyB3aWR0aD0iMTQ0IiBoZWlnaHQ9IjE0NCIgdmlld0JveD0iMCAwIDE0NCAxNDQiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CjxyZWN0IHdpZHRoPSIxNDQiIGhlaWdodD0iMTQ0IiByeD0iMjQiIGZpbGw9InVybCgjcGFpbnQwX2xpbmVhcl82NzlfMTkxMCkiLz4KPHBhdGggZD0iTTEzOC42ODUgMTIxLjA1N0MxMzguNjg1IDEyNi42NTIgMTM2LjQ2MiAxMzIuMDE3IDEzMi41MDQgMTM1Ljk3M0MxMjguNTQ3IDEzOS45MjkgMTIzLjE3OSAxNDIuMTUxIDExNy41ODIgMTQyLjE1MUMxMTEuOTg1IDE0Mi4xNTEgMTA2LjYxOCAxMzkuOTI5IDEwMi42NiAxMzUuOTczQzk4LjcwMjggMTMyLjAxNyA5Ni40Nzk1IDEyNi42NTIgOTYuNDc5NSAxMjEuMDU3Qzk2LjQ3OTUgMTE4LjI4NyA5Ny4wMjUzIDExNS41NDQgOTguMDg1OCAxMTIuOTg1Qzk5LjE0NjMgMTEwLjQyNSAxMDAuNzAxIDEwOC4xIDEwMi42NiAxMDYuMTQxQzEwNC42MiAxMDQuMTgyIDEwNi45NDYgMTAyLjYyOSAxMDkuNTA3IDEwMS41NjlDMTEyLjA2NyAxMDAuNTA5IDExNC44MTEgOTkuOTYyOSAxMTcuNTgyIDk5Ljk2MjlDMTIwLjM1MyA5OS45NjI5IDEyMy4wOTggMTAwLjUwOSAxMjUuNjU4IDEwMS41NjlDMTI4LjIxOCAxMDIuNjI5IDEzMC41NDQgMTA0LjE4MiAxMzIuNTA0IDEwNi4xNDFDMTM0LjQ2NCAxMDguMSAxMzYuMDE4IDExMC40MjUgMTM3LjA3OSAxMTIuOTg1QzEzOC4xMzkgMTE1LjU0NCAxMzguNjg1IDExOC4yODcgMTM4LjY4NSAxMjEuMDU3WiIgZmlsbD0idXJsKCNwYWludDFfcmFkaWFsXzY3OV8xOTEwKSIvPgo8cGF0aCBkPSJNMTEwLjMzNiAxMjYuMTQ3SDEyMy42OFYxMzYuODIzSDExMC4zMzZWMTI2LjE0N1oiIGZpbGw9IiMwRjVFOUMiLz4KPHBhdGggZD0iTTExOS4yNyAxMTIuMjk0QzExOS4yNyAxMTIuNzE0IDExOS4xNzggMTEzLjEzMSAxMTkgMTEzLjUxOUMxMTguODIxIDExMy45MDggMTE4LjU2IDExNC4yNjEgMTE4LjIzIDExNC41NThDMTE3LjkwMSAxMTQuODU1IDExNy41MDkgMTE1LjA5MSAxMTcuMDc5IDExNS4yNTJDMTE2LjY0OCAxMTUuNDEzIDExNi4xODYgMTE1LjQ5NiAxMTUuNzIgMTE1LjQ5NkMxMTQuNzc4IDExNS40OTYgMTEzLjg3NSAxMTUuMTU4IDExMy4yMSAxMTQuNTU4QzExMi41NDQgMTEzLjk1NyAxMTIuMTcgMTEzLjE0MyAxMTIuMTcgMTEyLjI5NEMxMTIuMTcgMTExLjQ0NSAxMTIuNTQ0IDExMC42MyAxMTMuMjEgMTEwLjAzQzExMy44NzUgMTA5LjQyOSAxMTQuNzc4IDEwOS4wOTIgMTE1LjcyIDEwOS4wOTJDMTE2LjE4NiAxMDkuMDkyIDExNi42NDggMTA5LjE3NSAxMTcuMDc5IDEwOS4zMzZDMTE3LjUwOSAxMDkuNDk2IDExNy45MDEgMTA5LjczMiAxMTguMjMgMTEwLjAzQzExOC41NiAxMTAuMzI3IDExOC44MjEgMTEwLjY4IDExOSAxMTEuMDY4QzExOS4xNzggMTExLjQ1NyAxMTkuMjcgMTExLjg3MyAxMTkuMjcgMTEyLjI5NFoiIGZpbGw9IiM1OTY4NkYiLz4KPHBhdGggZD0iTTEyOC4xMSAxMTQuMTAzQzEyOC4xMSAxMTQuOTUzIDEyNy43MzYgMTE1Ljc2NyAxMjcuMDcgMTE2LjM2OEMxMjYuNDA0IDExNi45NjggMTI1LjUwMSAxMTcuMzA1IDEyNC41NiAxMTcuMzA1QzEyNC4wOTQgMTE3LjMwNSAxMjMuNjMyIDExNy4yMjMgMTIzLjIwMSAxMTcuMDYyQzEyMi43NzEgMTE2LjkwMSAxMjIuMzc5IDExNi42NjUgMTIyLjA1IDExNi4zNjhDMTIxLjcyIDExNi4wNyAxMjEuNDU4IDExNS43MTcgMTIxLjI4IDExNS4zMjlDMTIxLjEwMiAxMTQuOTQgMTIxLjAxIDExNC41MjQgMTIxLjAxIDExNC4xMDNDMTIxLjAxIDExMy42ODMgMTIxLjEwMiAxMTMuMjY2IDEyMS4yOCAxMTIuODc4QzEyMS40NTggMTEyLjQ5IDEyMS43MiAxMTIuMTM3IDEyMi4wNSAxMTEuODM5QzEyMi4zNzkgMTExLjU0MiAxMjIuNzcxIDExMS4zMDYgMTIzLjIwMSAxMTEuMTQ1QzEyMy42MzIgMTEwLjk4NCAxMjQuMDk0IDExMC45MDEgMTI0LjU2IDExMC45MDFDMTI1LjUwMSAxMTAuOTAxIDEyNi40MDQgMTExLjIzOSAxMjcuMDcgMTExLjgzOUMxMjcuNzM2IDExMi40NCAxMjguMTEgMTEzLjI1NCAxMjguMTEgMTE0LjEwM1oiIGZpbGw9IiM1OTY4NkYiLz4KPHBhdGggZD0iTTEyMi4zMzMgMTE4Ljk3NkMxMjIuMzMzIDExOS44MjYgMTIxLjk1OCAxMjAuNjQgMTIxLjI5MyAxMjEuMjQxQzEyMC42MjcgMTIxLjg0MSAxMTkuNzI0IDEyMi4xNzggMTE4Ljc4MiAxMjIuMTc4QzExOC4zMTYgMTIyLjE3OCAxMTcuODU1IDEyMi4wOTYgMTE3LjQyNCAxMjEuOTM1QzExNi45OTMgMTIxLjc3NCAxMTYuNjAyIDEyMS41MzggMTE2LjI3MiAxMjEuMjQxQzExNS45NDMgMTIwLjk0MyAxMTUuNjgxIDEyMC41OSAxMTUuNTAzIDEyMC4yMDJDMTE1LjMyNCAxMTkuODEzIDExNS4yMzIgMTE5LjM5NyAxMTUuMjMyIDExOC45NzZDMTE1LjIzMiAxMTguNTU2IDExNS4zMjQgMTE4LjE0IDExNS41MDMgMTE3Ljc1MUMxMTUuNjgxIDExNy4zNjMgMTE1Ljk0MyAxMTcuMDEgMTE2LjI3MiAxMTYuNzEyQzExNi42MDIgMTE2LjQxNSAxMTYuOTkzIDExNi4xNzkgMTE3LjQyNCAxMTYuMDE4QzExNy44NTUgMTE1Ljg1NyAxMTguMzE2IDExNS43NzQgMTE4Ljc4MiAxMTUuNzc0QzExOS43MjQgMTE1Ljc3NCAxMjAuNjI3IDExNi4xMTIgMTIxLjI5MyAxMTYuNzEyQzEyMS45NTggMTE3LjMxMyAxMjIuMzMzIDExOC4xMjcgMTIyLjMzMyAxMTguOTc2WiIgZmlsbD0iIzU5Njg2RiIvPgo8cGF0aCBkPSJNMTE1LjMwOCAxMjEuOTA1QzExMy43MzUgMTIxLjQyNiAxMTUuNzA3IDEyMC42OCAxMTUuNDI5IDEyMC41NzNDMTE0LjY1MyAxMjAuMjc2IDExNS43MyAxMTkuMzMzIDExNi43NCAxMTguNTM5QzExNy42MjggMTE3Ljg0MSAxMTcuNjU5IDExNy44MzkgMTE3Ljg5MiAxMTguNDY4QzExOC4wMjQgMTE4LjgyMyAxMTguMzcxIDExOS4yMDYgMTE4LjY2NSAxMTkuMzE5QzExOS4yODkgMTE5LjU1OCAxMjAuOTUxIDExOS4yNjkgMTIwLjk1MSAxMTguODM1QzEyMC45NTEgMTE4LjIyMyAxMjEuNDE1IDExOC42OTkgMTIxLjc5NCAxMTkuNTMxQzEyMi40NTcgMTIwLjk4NyAxMjIuNDM3IDEyMi40NzIgMTIxLjQ1IDEyMi40NzJDMTIwLjg1OSAxMjIuNDcyIDEyMC4zMSAxMjIuNjkxIDEyMC4xOSAxMjMuMTQ3QzExNS4wNDYgMTI0LjYgMTE2LjQ3MSAxMjMuMjg0IDExNS4zMDggMTIxLjkwNVpNMTIzLjA5NCAxMTguMDU0QzEyMS42MDkgMTE3LjQ2NiAxMjAuNTQ3IDExNC44MzggMTIwLjU0NyAxMTMuMzA5QzEyMC41NDcgMTExLjMyMyAxMjIuNTQxIDEwOS4wOTUgMTI0LjMxNyAxMDkuMDk1QzEyOC4zMTUgMTA5LjA5NSAxMzAuNjg0IDExMi4yNjEgMTI4LjgzOCAxMTguNDA5QzEyOC4zODUgMTE5LjkxOSAxMjMuOTMzIDExOC4zODcgMTIzLjA5NCAxMTguMDU0Wk0xMjUuODY2IDExNS42NDlDMTI3LjU0NCAxMTQuNDc0IDEyNS44NTcgMTExLjc1NSAxMjMuOTgxIDExMi42MUMxMjIuNDc5IDExMy4yOTQgMTIzLjExOSAxMTYuMTE1IDEyNC43NyAxMTYuMTY0QzEyNC45NjEgMTE2LjE2OSAxMjUuNDU0IDExNS45MzggMTI1Ljg2NiAxMTUuNjQ5Wk0xMjQuMzM5IDExNC41NzlDMTIzLjc3MSAxMTQuMzgxIDEyMy44MDMgMTEzLjI1NiAxMjQuMzgzIDExMy4wMzRDMTI0LjkwMiAxMTIuODM1IDEyNS42MDQgMTEzLjM5OCAxMjUuNjA0IDExNC4wMTRDMTI1LjYwNCAxMTQuNDg5IDEyNC45MzYgMTE0Ljc4NyAxMjQuMzM5IDExNC41NzlaTTExMi4zMTkgMTE2Ljk4QzEwOS45NiAxMTUuNjcgMTEwLjI4NiAxMTEuMDA2IDExMi40MTcgMTA4Ljk2NUMxMTQuMzk2IDEwNy4wNjkgMTE4Ljc3OCAxMDcuMTAzIDExOS43NjQgMTA5LjQ2MkMxMjAuNDE1IDExMS4wMjEgMTIwLjIyOCAxMTIuNiAxMTkuMzk4IDExNC4wNzdDMTE4LjE5NCAxMTYuMjE5IDExNC4yNDIgMTE4LjA0OSAxMTIuMzE5IDExNi45OFpNMTE2LjU2IDExMy41OTNDMTE3LjMyNSAxMTIuOTAxIDExNy4yOTcgMTEyLjA2IDExNi41NzIgMTExLjI1OUMxMTUuNzc3IDExMC4zNzkgMTE0LjY5MiAxMTAuMzQ1IDExNC4wMzUgMTExLjM0N0MxMTMuNTI0IDExMi4xMjcgMTEzLjQzMSAxMTIuNTI4IDExMy45NDMgMTEzLjMwOUMxMTQuNTggMTE0LjI4IDExNS42NyAxMTQuMzk5IDExNi41NiAxMTMuNTkzWk0xMTQuMzc0IDExMi4zNEMxMTQuMTI4IDExMS42OTggMTE0Ljk4NSAxMTAuOTgxIDExNS42OCAxMTEuMjQ3QzExNS45NzQgMTExLjM2IDExNi4xNjQgMTExLjcxOCAxMTYuMTAyIDExMi4wNDRDMTE1Ljk2MSAxMTIuNzg1IDExNC42MzIgMTEzLjAxMiAxMTQuMzc0IDExMi4zNFoiIGZpbGw9IiNEM0Q2REEiLz4KPHBhdGggZD0iTTExOC40NjMgMTIxLjAwOEwxMTcuOTQ1IDEyMC43OEMxMTcuNTEgMTIwLjY4NSAxMTcuMjMxIDEyMS4xMDcgMTE3LjEzNiAxMjEuNTQzTDExNi44ODEgMTIyLjcwOUMxMTYuNzg2IDEyMy4xNDUgMTE3LjA2MSAxMjMuNTcxIDExNy40OTYgMTIzLjY2NkwxMTguNDQ3IDEyMy44NzRDMTE4Ljg4MiAxMjMuOTY5IDExOS4zMTEgMTIzLjY5NCAxMTkuNDA2IDEyMy4yNTlMMTE5LjY4NSAxMjEuNTU2QzExOS43OCAxMjEuMTIgMTE5LjQ3OSAxMjEuMjI4IDExOS4wNDMgMTIxLjEzM0wxMTguNjI4IDEyMS4wNDNDMTE4LjU3MyAxMjEuMzIxIDExOC41MTYgMTIxLjYxMyAxMTguNDcxIDEyMS44MzNDMTE4LjQzOCAxMjEuOTk4IDExOC40MDcgMTIyLjE0OCAxMTguMzc5IDEyMi4yODJDMTE4LjM1MSAxMjIuNDE1IDExOC4zMjcgMTIyLjUzMyAxMTguMzA1IDEyMi42MzVDMTE4LjI4MiAxMjIuNzM4IDExOC4yNjIgMTIyLjgyNSAxMTguMjQ1IDEyMi44OTdDMTE4LjIyOCAxMjIuOTY5IDExOC4yMTQgMTIzLjAyNSAxMTguMjAyIDEyMy4wNjhDMTE4LjE5NiAxMjMuMDg5IDExOC4xOTMgMTIzLjEwNyAxMTguMTg4IDEyMy4xMjFDMTE4LjE4MyAxMjMuMTM1IDExOC4xNzggMTIzLjE0NSAxMTguMTc1IDEyMy4xNTJDMTE4LjE3MyAxMjMuMTU1IDExOC4xNzIgMTIzLjE1NiAxMTguMTcxIDEyMy4xNThDMTE4LjE3IDEyMy4xNTkgMTE4LjE2OCAxMjMuMTYgMTE4LjE2NyAxMjMuMTZMMTE4LjE2NSAxMjMuMTU4QzExOC4xNjQgMTIzLjE1NiAxMTguMTYzIDEyMy4xNTIgMTE4LjE2MyAxMjMuMTQ4QzExOC4xNjIgMTIzLjE0IDExOC4xNjIgMTIzLjEyOSAxMTguMTYzIDEyMy4xMTVDMTE4LjE2MyAxMjMuMSAxMTguMTYzIDEyMy4wODMgMTE4LjE2NSAxMjMuMDYxQzExOC4xNjggMTIzLjAxOCAxMTguMTc1IDEyMi45NjEgMTE4LjE4MyAxMjIuODkxQzExOC4xOTIgMTIyLjgyIDExOC4yMDMgMTIyLjczNiAxMTguMjE2IDEyMi42NEMxMTguMjI5IDEyMi41NDMgMTE4LjI0NiAxMjIuNDMxIDExOC4yNjQgMTIyLjMwOEMxMTguMjgxIDEyMi4xODUgMTE4LjMwMSAxMjIuMDUxIDExOC4zMjMgMTIxLjkwM0MxMTguMzQ2IDEyMS43NTUgMTE4LjM3IDEyMS41OTIgMTE4LjM5NyAxMjEuNDE5QzExOC40MTYgMTIxLjI5OSAxMTguNDQyIDEyMS4xNCAxMTguNDYzIDEyMS4wMDhaIiBmaWxsPSIjOThDNkQ4Ii8+CjxwYXRoIGQ9Ik0xMDMuNTgxIDExNi4zNjdDMTAzLjQ3OSAxMTYuMTAzIDEwMy42MjIgMTE1LjY5OSAxMDMuODk4IDExNS40NjlDMTA0LjMwMyAxMTUuMTMzIDEwNC41MDQgMTE1LjE1NSAxMDQuOTMgMTE1LjU4MUMxMDUuMjIgMTE1Ljg3MSAxMDUuMzU1IDExNi4yNzYgMTA1LjIzIDExNi40NzlDMTA0LjkwMiAxMTcuMDA5IDEwMy43OTggMTE2LjkzNCAxMDMuNTgxIDExNi4zNjdaIiBmaWxsPSIjOThDNkQ4Ii8+CjxwYXRoIGQ9Ik0xMDYuMDM4IDExMi4yODFDMTA1LjY4IDExMS44NDkgMTA1LjcwMiAxMTEuNjYgMTA2LjE2NSAxMTEuMTk3QzEwNi42ODkgMTEwLjY3NCAxMDYuNzY0IDExMC42NzQgMTA3LjI4NyAxMTEuMTk3QzEwNy43NTEgMTExLjY2IDEwNy43NzMgMTExLjg0OSAxMDcuNDE1IDExMi4yODFDMTA3LjE3NiAxMTIuNTY4IDEwNi44NjYgMTEyLjgwMyAxMDYuNzI2IDExMi44MDNDMTA2LjU4NiAxMTIuODAzIDEwNi4yNzcgMTEyLjU2OCAxMDYuMDM4IDExMi4yODFaIiBmaWxsPSIjOThDNkQ4Ii8+CjxwYXRoIGQ9Ik0xMDYuMDM4IDEwNy4yMjRDMTA1LjY4MiAxMDYuNzk1IDEwNS42OTQgMTA2LjYxMiAxMDYuMTA2IDEwNi4yMDFDMTA2LjY1IDEwNS42NTcgMTA3LjczOCAxMDUuODggMTA3LjczOCAxMDYuNTM2QzEwNy43MzggMTA2Ljk3IDEwNy4wNzIgMTA3Ljc0NyAxMDYuNyAxMDcuNzQ3QzEwNi41NzUgMTA3Ljc0NyAxMDYuMjc3IDEwNy41MTIgMTA2LjAzOCAxMDcuMjI0WiIgZmlsbD0iIzk4QzZEOCIvPgo8cGF0aCBkPSJNMTE3Ljk1NSAxMzYuNTQxQzExNC41NzggMTM2LjExMSAxMTAuMDg3IDEzNC44ODEgMTA5LjQxIDEzNC4yNzRDMTA4LjEyMyAxMzMuMTE5IDEwOC45NDIgMTI3Ljg3OSAxMTAuNDY2IDEyNi41NEMxMTEuMzg3IDEyNS43MzEgMTE1LjAzOSAxMjQuNjYzIDExNy4xODIgMTI1LjE2OUMxMTkuNjQyIDEyNS43NDkgMTIzLjkzMiAxMjguNjQ4IDEyNC4yNSAxMjkuNTA4QzEyNC42MjUgMTMwLjUyMiAxMjQuMDQ4IDEzNC41NDEgMTIzLjMyMyAxMzUuNTM2QzEyMi42OTQgMTM2LjM5OSAxMjAuMzI1IDEzNi44NDMgMTE3Ljk1NSAxMzYuNTQxWk0xMjAuNTQgMTM0LjA4NEMxMjEuMjk3IDEzMy4zOTkgMTIxLjM0MiAxMzEuODQyIDEyMC42MjcgMTMxLjEyNkMxMTkuNDkgMTI5Ljk4OSAxMTcuMTEyIDEzMC44NjkgMTE3LjExMiAxMzIuNDI4QzExNy4xMTIgMTM0LjMwNCAxMTkuMTg5IDEzNS4zMDcgMTIwLjU0IDEzNC4wODRaTTExOC41ODIgMTMzLjEyNUMxMTguMzExIDEzMi40MTkgMTE4LjY4IDEzMS42MDggMTE5LjI3MiAxMzEuNjA4QzEyMCAxMzEuNjA4IDEyMC4yMjggMTMyLjE3MyAxMTkuODEyIDEzMi45NDlDMTE5LjM4MSAxMzMuNzU1IDExOC44NTMgMTMzLjgzIDExOC41ODIgMTMzLjEyNVpNMTE2LjQyMiAxMzMuMTQzQzExNy4wOTQgMTMyLjMzMyAxMTYuNzExIDEzMS4xNzYgMTE1LjY4MSAxMzAuOTAyQzExNC41ODEgMTMwLjYxIDExNC4wNTIgMTMxLjE4MyAxMTQuODY5IDEzMS43OEMxMTUuNjgzIDEzMi4zNzUgMTE1LjU1MSAxMzIuNjc5IDExNC41MzMgMTMyLjU1N0MxMTMuMzI3IDEzMi40MTMgMTEyLjgzMyAxMzEuNTY2IDExMy4yNTQgMTMwLjM2MkMxMTMuNjgxIDEyOS4xNDIgMTE1LjE5NyAxMjguODYzIDExNS43NTMgMTI5LjkwMkMxMTYuMTkzIDEzMC43MjUgMTE2LjYyNyAxMzAuNzkzIDExNi45IDEzMC4wODNDMTE3LjIyOSAxMjkuMjI0IDExNS45MTQgMTI4LjIzNyAxMTQuNDQgMTI4LjIzN0MxMTEuOTkzIDEyOC4yMzcgMTEwLjgxNiAxMzEuMDc0IDExMi41NDYgMTMyLjgwM0MxMTMuNSAxMzMuNzU4IDExNS43NDkgMTMzLjk1NSAxMTYuNDIyIDEzMy4xNDNaIiBmaWxsPSIjN0JBOUI5Ii8+CjxwYXRoIGQ9Ik0xMjAuMTE4IDEzOC41OTJDMTE4Ljc1MSAxMzcuMjk3IDEyMS45MTYgMTM2LjA5NiAxMjMuNjA2IDEzOC4yODNDMTI0LjQ2OSAxMzkuMzIyIDEyMi44NTMgMTM5Ljk3NiAxMjAuMTE4IDEzOC41OTJaTTEwNi4xMiAxMzUuNjU4QzEwNC43NTEgMTM0LjI4OSAxMDYuOTYzIDEzMy45MDQgMTA4LjU4MSAxMzUuMjNMMTA5Ljk0MSAxMzUuOTJMMTA4LjA1OSAxMzYuMDYxQzEwNy4yMTUgMTM2LjA2MiAxMDYuMzQzIDEzNS44ODEgMTA2LjEyIDEzNS42NThWMTM1LjY1OFpNMTI1LjM2MyAxMjcuODY2QzEyNS4wODIgMTI3LjQxIDEyOC4zMzMgMTI2LjI2NyAxMjguNjg4IDEyNi40ODZDMTI4Ljg0NiAxMjYuNTg0IDEyOC45NzUgMTI3LjMzNyAxMjguOTc1IDEyOC4xNjFDMTI4Ljk3NSAxMjkuMjM5IDEyOC44NCAxMjkuNjU4IDEyOC40OSAxMjkuNjU4QzEyOC4yMjIgMTI5LjY1OCAxMjUuNDc2IDEyOC4wNDcgMTI1LjM2MyAxMjcuODY2Wk0xMTguNjQxIDEyMS4wOTVDMTE0Ljg3MSAxMjAuNzUzIDExNS4zNzggMTE5LjMyNyAxMTYuMzYxIDExOC40OTlDMTE2LjkyOCAxMTguMDIxIDExNy4zNCAxMTcuNzc5IDExNy45MTMgMTE3LjY2N0MxMTcuOTEzIDExNy42NjcgMTE3Ljg3MiAxMTkuMDQ2IDExOC42NjYgMTE5LjMxOUMxMTkuMjk3IDExOS41MzYgMTIwLjI3OCAxMTkuMDY0IDEyMC40NzEgMTE4LjY3NUMxMjIuMTg4IDExNS4yMDEgMTIxLjQxNSAxMTguNjk5IDEyMS43OTQgMTE5LjUzMkMxMjMuMjM4IDEyNC4wMDcgMTIwLjE4OCAxMjEuNzMgMTE4LjY0MSAxMjEuMDk1Wk0xMDQuOTEyIDEyMS4yM0MxMDQuMjQyIDEyMC40ODcgMTAzLjY5MyAxMTkuNzI5IDEwMy42OTMgMTE5LjU0NEMxMDMuNjkzIDExOS4xMTYgMTA0Ljc0NiAxMTkuMTEyIDEwNS44NjMgMTE5LjUzN0MxMDYuNTkzIDExOS44MTQgMTEwLjM0NyAxMjAuMDA2IDExMC4zNDcgMTIxLjE1M0MxMTAuMzQ3IDEyMS44OTkgMTA4Ljg5IDEyMy43NjIgMTA4LjcyNiAxMjMuNzYyQzEwOC41NjMgMTIzLjc2MiAxMDUuNTgzIDEyMS45NzIgMTA0LjkxMiAxMjEuMjNaIiBmaWxsPSIjQTZCM0MyIi8+CjxwYXRoIGQ9Ik0xMTguNDYzIDEyMS4wMDhDMTE4LjQ0MiAxMjEuMTQgMTE4LjQxNiAxMjEuMjk5IDExOC4zOTcgMTIxLjQxOUMxMTguMzcgMTIxLjU5MiAxMTguMzQ1IDEyMS43NTUgMTE4LjMyMyAxMjEuOTAzQzExOC4zIDEyMi4wNTEgMTE4LjI4MSAxMjIuMTg1IDExOC4yNjMgMTIyLjMwOEMxMTguMjQ1IDEyMi40MzEgMTE4LjIyOSAxMjIuNTQyIDExOC4yMTYgMTIyLjYzOUMxMTguMjAzIDEyMi43MzYgMTE4LjE5MSAxMjIuODIgMTE4LjE4MyAxMjIuODlDMTE4LjE3NSAxMjIuOTYxIDExOC4xNjggMTIzLjAxOCAxMTguMTY1IDEyMy4wNjFDMTE4LjE2MyAxMjMuMDgzIDExOC4xNjQgMTIzLjEgMTE4LjE2MyAxMjMuMTE1QzExOC4xNjIgMTIzLjEyOSAxMTguMTYyIDEyMy4xNCAxMTguMTYzIDEyMy4xNDhDMTE4LjE2MyAxMjMuMTUyIDExOC4xNjQgMTIzLjE1NiAxMTguMTY1IDEyMy4xNThMMTE4LjE2NyAxMjMuMTZDMTE4LjE2OCAxMjMuMTYgMTE4LjE3IDEyMy4xNTkgMTE4LjE3MSAxMjMuMTU4QzExOC4xNzIgMTIzLjE1NyAxMTguMTczIDEyMy4xNTUgMTE4LjE3NSAxMjMuMTUyQzExOC4xNzggMTIzLjE0NSAxMTguMTgyIDEyMy4xMzUgMTE4LjE4NyAxMjMuMTIxQzExOC4xOTIgMTIzLjEwNyAxMTguMTk2IDEyMy4wODkgMTE4LjIwMiAxMjMuMDY3QzExOC4yMTMgMTIzLjAyNSAxMTguMjI4IDEyMi45NjkgMTE4LjI0NSAxMjIuODk3QzExOC4yNjIgMTIyLjgyNSAxMTguMjgyIDEyMi43MzggMTE4LjMwNSAxMjIuNjM1QzExOC4zMjcgMTIyLjUzMyAxMTguMzUxIDEyMi40MTUgMTE4LjM3OSAxMjIuMjgxQzExOC40MDYgMTIyLjE0OCAxMTguNDM4IDEyMS45OTggMTE4LjQ3MSAxMjEuODMzQzExOC41MTYgMTIxLjYxMyAxMTguNTczIDEyMS4zMjEgMTE4LjYyOCAxMjEuMDQzTDExOC40NjMgMTIxLjAwOFoiIGZpbGw9IiM0QzlDQkIiLz4KPHBhdGggZD0iTTMwLjk5ODYgMTQyLjY3M0MyNi40NDg5IDE0MC45MTYgMjEuNDY2NCAxMzYuODA1IDE4Ljc5MjEgMTMyLjYwM0MxNS40MDUzIDEyNy4yODEgMTQuNDA1MiAxMjMuNDE5IDE1LjA2MzYgMTE4LjIwM0MxNS42ODA2IDExMy4zMTUgMTcuMzU1OCAxMTAuNTU3IDIyLjIyMzIgMTA2LjQxM0MyOS4wNDE3IDEwMC42MDggMzQuNDg0NiA5Ny45Nzk5IDQwLjU5NDcgOTcuNTQxNUM0OS44MjEyIDk2Ljg3OTQgNTUuMDk3OCA5OS44MTA5IDU3LjY0NjUgMTA3LjAxNUM1OC41MDQzIDEwOS40MzkgNTguNjg1NiAxMTUuNTcgNTguMDU0NCAxMjAuODA5QzU3LjgzMTEgMTIyLjY2MyA1Ny40Nzg5IDEyNi4wNTUgNTcuMjcxOCAxMjguMzQ4QzU2LjU0NjcgMTM2LjM3NSA1NC40MTQ3IDE0MC40MTcgNDkuNjI1MyAxNDIuODQzQzQ4LjI5NzkgMTQzLjUxNiA0Ny43MTk5IDE0My41NjEgNDAuNjkyNCAxNDMuNTM5QzMzLjUxNTEgMTQzLjUxNiAzMy4wODEgMTQzLjQ3OCAzMC45OTg2IDE0Mi42NzNaTTAuMDc5NTc2NSAxMDQuOTY1QzAuMDgxNTcyMyAxMDMuODUzIDAuMTQ0NDAzIDEwMy40MzggMC4yMTk0MjcgMTA0LjA0NEMwLjI5NDQ1MSAxMDQuNjUgMC4yOTI4OTkgMTA1LjU2IDAuMjE1OTk2IDEwNi4wNjdDMC4xMzkwNzkgMTA2LjU3MyAwLjA3NzY4MzggMTA2LjA3OCAwLjA3OTU3NjUgMTA0Ljk2NVpNMC4wMDE3MzM1MSAxMDIuMjRDMC4wMTc4OTkyIDEwMS44NDggMC4wOTc3ODk3IDEwMS43NjggMC4yMDUzOTggMTAyLjAzN0MwLjMwMjc3MiAxMDIuMjggMC4yOTA3OTcgMTAyLjU3MSAwLjE3ODc4NyAxMDIuNjgzQzAuMDY2Nzc0OCAxMDIuNzk1IC0wLjAxMjkwMjEgMTAyLjU5NiAwLjAwMTczMzUxIDEwMi4yNFpNMC4wMjgzNDQ0IDc1LjEzMjZDMC4wMjgzNDQ0IDc0LjY2OTEgMC4xMDQ4NTcgNzQuNDc5NSAwLjE5ODM2OSA3NC43MTEyQzAuMjkxODg0IDc0Ljk0MyAwLjI5MTg4NCA3NS4zMjIyIDAuMTk4MzY5IDc1LjU1MzlDMC4xMDQ4NTcgNzUuNzg1NyAwLjAyODM0NDQgNzUuNTk2MSAwLjAyODM0NDQgNzUuMTMyNloiIGZpbGw9InVybCgjcGFpbnQyX3JhZGlhbF82NzlfMTkxMCkiLz4KPHBhdGggZD0iTTM1LjMxMzMgMTEyLjU0NEwzNy45IDExMS4wOTFMMzkuMzUxNyAxMjEuNDZMMzkuMjQ4IDEyOC45MjZMMzUuMzcwMyAxMzAuODUxTDMyLjkyMjkgMTI2LjQzN0wzNS4zMTMzIDExMi41NDRaIiBmaWxsPSIjMzA2MEFEIi8+CjxwYXRoIGQ9Ik0zOC43Mjk1IDExNi42OUwzOS4zNTE2IDEyNC41NzFMNDMuMjkxOSAxMjUuOTE5TDQzLjM5NTYgMTA5LjIyNUw0MC4xODEyIDExMC4wNTRMMzguNzI5NSAxMTYuNjlaIiBmaWxsPSIjNjA2MzY4Ii8+CjxwYXRoIGQ9Ik0zNS4zNjk5IDEzMC44NTFDMzMuMDMyNCAxMjkuNzMgMjkuNzMwMSAxMjYuNDA4IDI4LjY2MjEgMTI0LjEwM0MyOC4wMzMyIDEyMi43NDUgMjcuNjM5NyAxMjIuMzEgMjYuODkyMyAxMjIuMTQ2QzI1Ljc4NzkgMTIxLjkwMyAyNS44MDg0IDEyMS45NzEgMjYuMzIxOCAxMjAuMjU4QzI2LjU5MzYgMTE5LjM1IDI2LjkwMzIgMTE4Ljk1NCAyNy4zNDA2IDExOC45NTRDMjcuNjg2MiAxMTguOTU0IDI5LjE1OTcgMTE3Ljg0MSAzMC42MTQ5IDExNi40OEMzMi4wNzAyIDExNS4xMTkgMzMuNzIyNiAxMTMuNjc3IDM0LjI4NjkgMTEzLjI3NUwzNS4zMTI5IDExMi41NDRMMzUuNjc2NyAxMTQuMjQxQzM3LjA0NDMgMTIwLjYxOCAzNy4wMjcgMTI0LjI2NiAzNS42MTk4IDEyNi4yNDNDMzQuNTUyMSAxMjcuNzQ0IDM0Ljg0MTcgMTI4LjM5MyAzNi41NzkxIDEyOC4zOTNDMzcuMzk3IDEyOC4zOTMgMzcuODIyMiAxMjguMTg1IDM4LjEzMDUgMTI3LjYzNEMzOC42NzQ4IDEyNi42NjMgMzguNjMxNSAxMjAuOTY4IDM4LjA0NTQgMTE2LjQyNkMzNy4zNDQ2IDExMC45OTUgMzcuMzEyNyAxMTEuMTY3IDM5LjE1NzggMTEwLjM5OEM0MC4wMzYgMTEwLjAzMSA0MC44MTggMTA5Ljc5NyA0MC44OTU1IDEwOS44NzdDNDAuOTczMSAxMDkuOTU2IDQwLjgxOTEgMTExLjIzNSA0MC41NTM0IDExMi43MThDMzkuNjA2NyAxMTguMDAzIDM5LjkzMTUgMTIzLjIzOCA0MS4yNzUyIDEyNC4zNTNDNDEuNjg1MSAxMjQuNjkzIDQxLjcyMzMgMTIzLjk2OSA0MS41NDUyIDExOS4yMzJDNDEuMzE1NiAxMTMuMTIzIDQxLjUxNzQgMTEwLjkxOSA0Mi40MDc1IDEwOS44MkM0Mi45MTk4IDEwOS4xODggNDMuMjEzIDEwOS4xMTYgNDQuNDk1OSAxMDkuMzFDNDYuOTU2NiAxMDkuNjgyIDQ2Ljk1MDYgMTA5LjY2OCA0Ni4yMDk5IDExMy4yNTRDNDUuODQ5NiAxMTQuOTk5IDQ1LjQ0ODMgMTE3LjIyNCA0NS4zMTggMTE4LjJMNDUuMDgxMiAxMTkuOTczTDQ1Ljk0NDYgMTE5LjY0NUM0Ni40MTk1IDExOS40NjUgNDcuNDQxMyAxMTguOTk0IDQ4LjIxNTMgMTE4LjU5OUw0OS42MjI2IDExNy44ODFMNDkuMzg5MyAxMTguNjdDNDkuMjYxIDExOS4xMDUgNDkuMDY3MiAxMjAuNzAxIDQ4Ljk1ODcgMTIyLjIxOEw0OC43NjE0IDEyNC45NzZMNDcuMTA5NSAxMjQuMzI1QzQ2LjIwMSAxMjMuOTY3IDQ1LjMxMTYgMTIzLjY3NCA0NS4xMzMxIDEyMy42NzRDNDQuOTQxIDEyMy42NzQgNDQuODA4NSAxMjUuNTMyIDQ0LjgwODUgMTI4LjIyNEM0NC44MDg1IDEzMS44OTEgNDQuNzE3MSAxMzIuNzc1IDQ0LjMzODYgMTMyLjc3NUM0MS4wMTk4IDEzMi4yOTUgMzguNzg0MiAxMzIuNDU4IDM1LjM2OTkgMTMwLjg1MVoiIGZpbGw9IiNFRUMyM0IiLz4KPHBhdGggZD0iTTQxLjE2MzkgMTMyLjQxOEM0MS4xNjM5IDEzMi4yODMgNDEuMzQwOCAxMzAuNTE1IDQxLjY2OTIgMTI4LjE0NUM0MS45OTc3IDEyNS43NzUgNDIuMzU3NyAxMjIuOTYxIDQyLjQ2OTIgMTIxLjg5M0M0Mi41ODA4IDEyMC44MjQgNDIuNzk2OSAxMjAuMDI3IDQyLjk0OTUgMTIwLjEyMUM0My42NDQzIDEyMC41NTEgNDQuMjkzOCAxMjkuMzc1IDQ0LjA0NzQgMTMxLjQyQzQzLjg5MDkgMTMyLjcxOSA0My42NTk1IDEzMi43MzcgNDMuMzAxOSAxMzIuNjc3QzQyLjY2MDIgMTMyLjU3IDQxLjkxMDEgMTMyLjUxMyA0MS4xNjM5IDEzMi40MThaTTMyLjU5NjkgMTIyLjczNUMzMS42ODEzIDEyMi41ODMgMzAuODI0MiAxMjEuNDQyIDMwLjgyNDIgMTIwLjM3NEMzMC44MjQyIDExOS40MDcgMzIuMjM4MSAxMTcuOTM3IDMzLjE2ODUgMTE3LjkzN0MzNC42NjY2IDExNy45MzcgMzUuODM5MiAxMjAuNTEzIDM0Ljk0ODQgMTIxLjg0N0MzNC41NTE3IDEyMi40NDEgMzMuNDE0NyAxMjIuODcgMzIuNTk2OSAxMjIuNzM1WiIgZmlsbD0id2hpdGUiLz4KPHBhdGggZD0iTTExNC4wMjEgMTM3LjQ5OEMxMDcuMzE4IDEzNS43NCAxMDYuMjczIDEzNC4xMDQgMTA2LjYwOSAxMjUuODkxQzEwNi43MTYgMTIzLjI4MSAxMDYuODg1IDEyMC41MzkgMTA2Ljk4NiAxMTkuNzk3QzEwNy41NDUgMTE1LjY2NCAxMDguMzQyIDExMi4xNzMgMTA5LjA4NiAxMTAuNTk5QzEwOS43NDQgMTA5LjIwNiAxMDkuODQyIDEwOC43MTIgMTA5LjUzNSAxMDguMzQyQzEwOC40NSAxMDcuMDM1IDExMC4zNzEgMTA0Ljk5NSAxMTEuOTEgMTA1LjgxOUMxMTIuNDU3IDEwNi4xMTEgMTEyLjkxOCAxMDYuMDc3IDExMy45NDUgMTA1LjY2NUMxMTcuMDYyIDEwNC40MTkgMTIzLjExNyAxMDUuNTcxIDEyNi44MTQgMTA4LjExNUMxMjguMTQ5IDEwOS4wMzMgMTI4LjQ4NSAxMDkuMTM2IDEyOS4wMzggMTA4Ljc5MUMxMzAuNTIzIDEwNy44NjMgMTMyLjQ1MSAxMTAuNTczIDEzMS4yNSAxMTEuOUMxMzAuNzg3IDExMi40MTIgMTMwLjc1NSAxMTIuNzA1IDEzMS4wNjUgMTEzLjU5NkMxMzEuODUgMTE1Ljg0NiAxMzEuNzk0IDExNi4xMDQgMTI4LjAwNiAxMjcuNzE5QzEyNS41OCAxMzUuMTYgMTI0LjU0MyAxMzcuMzY1IDEyMy4yNjQgMTM3LjgxMUMxMjEuNjUzIDEzOC4zNzMgMTE2LjcyMyAxMzguMjA2IDExNC4wMjEgMTM3LjQ5OFpNMTIyLjU0MSAxMzUuMjI5QzEyMy4wMDMgMTM0LjU0MSAxMjMuMzU5IDEzMy4zMTMgMTIzLjUxMyAxMzEuODc4QzEyMy43OTQgMTI5LjI1IDEyMy44MTkgMTI5LjI4MyAxMjAuMzE3IDEyNy42MDFDMTE3LjY4NSAxMjYuMzM4IDExMy44MzMgMTI1Ljk3MiAxMTEuODU1IDEyNi43OThDMTEwLjgyOSAxMjcuMjI3IDExMC41OTUgMTI3LjU1NiAxMTAuMDU5IDEyOS4zMjdDMTA5LjExNiAxMzIuNDQzIDEwOS43MDMgMTM0LjMwOCAxMTEuODMyIDEzNC45NThDMTEzLjEzOSAxMzUuMzU3IDEyMC4wOSAxMzYuNDMgMTIwLjk4IDEzNi4zN0MxMjEuNTA0IDEzNi4zMzUgMTIyLjA4MiAxMzUuOTEyIDEyMi41NDEgMTM1LjIyOVpNMTE5LjQ1MSAxMjIuNDJDMTE5LjYzIDEyMi4wODEgMTE5LjY4NSAxMjEuNTU2IDExOS42ODUgMTIxLjU1NkMxMjAuMTU2IDEyMS42NTQgMTIwLjQ4IDEyMS42ODMgMTIwLjcyNiAxMjEuNzE5QzEyMS4xMDUgMTIxLjc3NCAxMjEuNTY0IDEyMS41ODcgMTIxLjc0NSAxMjEuMzAzQzEyMi4xNDkgMTIwLjY3MSAxMjAuNDc0IDExNy43MDEgMTE5LjU0OCAxMTcuNDA4QzExOC42NjggMTE3LjEyOCAxMTUuOTM1IDExOC44MjcgMTE1LjkzNSAxMTkuNjUzQzExNS45MzUgMTIwLjAyMiAxMTYuNTUzIDEyMC40MTEgMTE2LjgzMSAxMjAuNTE4QzExNi45ODIgMTIwLjU3NiAxMTcuMjUyIDEyMC43MzggMTE3LjU3NiAxMjAuOTM4QzExNy4zOCAxMjEuMzUxIDExNy4zNDYgMTIxLjQ3OCAxMTcuMzEyIDEyMS43MjRDMTE3LjIwNCAxMjIuNDk4IDExNy40MzUgMTIyLjY4NiAxMTguNDE2IDEyMi45MTJDMTE5LjE1MSAxMjIuOTkxIDExOS4yNSAxMjIuODQ1IDExOS40NTEgMTIyLjQyWk0xMjcuMjY1IDExNi44MzZDMTMwLjkzMyAxMTMuNDc1IDEyNy4wNzQgMTA3LjQxIDEyMi43MDUgMTA5LjY3QzEyMS42MjYgMTEwLjIyOCAxMjAuNjU0IDExMi4wMTkgMTIwLjY1NCAxMTMuNDQ5QzEyMC42NTQgMTE1LjY2NCAxMjIuODMxIDExNy45MzcgMTI0Ljk1NyAxMTcuOTQxQzEyNS43MzMgMTE3Ljk0MyAxMjYuNDE0IDExNy42MTcgMTI3LjI2NSAxMTYuODM2Wk0xMTcuMTQxIDExNS42MjJDMTE5LjAzNSAxMTQuNDY4IDExOS44MDQgMTExLjc5NSAxMTguODAyIDEwOS44NTdDMTE4LjA4MiAxMDguNDY0IDExNy4wMDggMTA3LjgzIDExNS4zNjYgMTA3LjgzQzExMS41MDggMTA3LjgzIDEwOS40ODUgMTEzLjIzNCAxMTIuNDY1IDExNS41NzhDMTEzLjU3IDExNi40NDggMTE1Ljc1NCAxMTYuNDY4IDExNy4xNDEgMTE1LjYyMloiIGZpbGw9IiMwOTk2RDEiLz4KPHBhdGggb3BhY2l0eT0iMC45IiBkPSJNMzMuMTc1NyAyNkwzMi4zODEgMjcuMjE5QzMxLjk0NDkgMjcuODg5IDMxLjI4MzYgMjkuMzI0OCAzMC45MDkxIDMwLjQwODhDMzAuNTM0NyAzMS40OTI4IDI5Ljk5MjIgMzIuOTcwMSAyOS43MDQzIDMzLjY5MjhDMjkuNDE2MyAzNC40MTU0IDI5LjE3MzMgMzYuMTE1IDI5LjE2MzggMzcuNDdDMjkuMTQzNiA0MC4zNTMgMjguNzQ5NCA0MS44ODg4IDI3LjYzNjQgNDMuNDIyNUMyNi45MjEyIDQ0LjQwOCAyNi42NzUxIDQ0LjUyMjMgMjUuNDUzMiA0NC40NDFDMjMuODk5NSA0NC4zMzc3IDIyLjgxNTMgNDUuMDgxNSAyMS4yMTcgNDcuMzQ4MUMyMC43MTY1IDQ4LjA1OCAyMC4xMzY3IDQ4LjYzOTMgMTkuOTI4OSA0OC42MzkzQzE5LjcyMTEgNDguNjM5MyAxOS4xODA1IDQ4LjIzMTkgMTguNzI2MiA0Ny43MzUxQzE4LjI3MTkgNDcuMjM4MiAxNy45OTQ0IDQ3LjAxMTUgMTguMTExIDQ3LjIyOThDMTguNDkgNDcuOTM5NiAxNy41OTA0IDUzLjI2MTUgMTYuODM3OCA1NC43NjQyQzE1Ljk2NyA1Ni41MDI4IDE0LjM4NzIgNTcuNzIzMyAxMi41MzU0IDU4LjA4NDNDMTEuMDA1OSA1OC4zODI2IDExLjIyODMgNTguMDkgOC43MTc5NSA2My4wOTQ2QzYuNjYzMzQgNjcuMTkwNyA1Ljk2OTM1IDY3LjgyNjQgMi42NTUzMyA2OC42NTQyQzEuNDk3MTcgNjguOTQzNCAwLjQ1ODc1OSA2OS43MTQyIDAgNzAuNTI4N1Y3Mi44MjYzQzAuMzU2MTkzIDcyLjc0NzMgMC44Mjg3MjggNzIuNjIwMyAxLjYxNDk5IDcyLjM4NzNDMi45OTQ4NiA3MS45NzgyIDQuODIzNDggNzEuNDgwNiA1LjY3ODEgNzEuMjgwNkM3LjcyOTM0IDcwLjgwMDYgOS4zOTM1MSA2OS4yMTk1IDkuOTIyNzkgNjcuMjQ2N0MxMC4yNDI0IDY2LjA1NTQgMTAuNzAzMyA2NS40MjQxIDExLjk5OTIgNjQuNDA1OEMxMy4yODQgNjMuMzk2MiAxMy43NTY1IDYyLjc1NDYgMTQuMDY3MSA2MS41OTY5QzE0LjM3MzkgNjAuNDUzMiAxNC43MDExIDYwLjAwNCAxNS40NDA3IDU5LjcwNjNDMTcuMDE1MyA1OS4wNzI0IDIwLjY0NDQgNTUuODY1NSAyMS4yMzYzIDU0LjU4NThDMjEuNTM3MyA1My45MzQ5IDIyLjExOTggNTIuNTM5OSAyMi41Mjg3IDUxLjQ4NjJDMjMuMjYzNCA0OS41OTMyIDI1LjAxOTUgNDcuMzI2MSAyNS43NTAxIDQ3LjMyNjFDMjUuOTU3NCA0Ny4zMjYxIDI3LjEwOSA0Ni43MjQ4IDI4LjMwOTMgNDUuOTkwOEMzMS4yMzc2IDQ0LjIwMDMgMzEuNTk5MSA0My40MTM2IDMxLjU5OTEgMzguODYxNEMzMS41OTkxIDM1LjY4MTQgMzEuNjg3IDM1LjE1NTcgMzIuMzQ2OCAzNC4zNjg0QzMzLjAwMzcgMzMuNTg0NyAzMy4wOTk3IDMzLjAyNDkgMzMuMTM1MSAyOS43MzkxTDMzLjE3NTcgMjZaTTI0LjMzMzggNjMuNTk1OEMyNC4yMDM3IDYzLjYxMzUgMjQuMDczOCA2My42NjE4IDIzLjkyNTggNjMuNzM2MkMyMi45Mzk3IDY0LjIzMTUgMjIuNjEzNiA2NC45MTAyIDIxLjU5NzMgNjguNTg2QzIxLjAxNzkgNzAuNjgxNSAyMC40MjEyIDcyLjA4MTkgMTkuODc5OCA3Mi42MTc4QzE5LjQyMzUgNzMuMDY5NSAxOC45MzgzIDczLjg4MjMgMTguODAxIDc0LjQyNDJDMTguMTgxIDc2Ljg3MTUgMTYuNzQ1NCA3OC42MDczIDE0LjQ0OTUgNzkuNjg3MUMxMi40NTU1IDgwLjYyNDkgMTAuNjQwNyA4Mi4xMTc5IDkuMTExMDIgODQuMDc5OEM4LjMyMTA5IDg1LjA5MjkgNy40ODc1MiA4Ni4wMjkgNy4yNTY3OCA4Ni4xNjA5QzYuNzYxNjQgODYuNDQ0IDUuNzA4OTkgODguODk0MSA0LjYwNzg1IDkyLjMyOEMzLjcxMDkyIDk1LjEyNSAxLjcxMDY1IDk3LjQ2ODIgMC40Nzg1MTQgOTcuMTY1OEMwLjI1NDkzMSA5Ny4xMTA5IDAuMTA2OTExIDk3LjExMzcgMCA5Ny4yOTAxVjEwNC40OTJDMC41NTY0MTEgMTAzLjY2NCAxLjIzNDIgMTAyLjYyOSAyLjU2MzQ3IDEwMC41NEM0LjM0NTExIDk3LjczOTcgNi40NjM4NSA5NC40ODY3IDcuMjcxNzMgOTMuMzEyNEM4LjA3OTYgOTIuMTM4MSA5LjM1Njg3IDkwLjE0ODggMTAuMTA4NiA4OC44OTE2QzExLjIzNDIgODcuMDA5MiAxMi4xMDE2IDg2LjEyNDMgMTUuMDI4NCA4My44ODMzTDE4LjU4MzEgODEuMTYyN0wyMC40MDUzIDc1Ljk4ODFDMjMuMTg5NSA2OC4wODE3IDIzLjk1ODUgNjcuMTk5NyAyNy43NjI0IDY3LjU1MTVDMjkuNDYwNSA2Ny43MDg1IDMwLjYzMzQgNjcuNjE5NiAzMi4wNzM0IDY3LjIyNjdDMzQuMTgyNiA2Ni42NTEyIDM0LjQ3OTkgNjYuMTg3MiAzMy4zNTUxIDY1LjIzMThDMzIuNjU1NCA2NC42Mzc2IDMxLjMxNTMgNjQuNjQgMjguMjM4OCA2NS4yNDE4QzI3LjMzOTcgNjUuNDE3NyAyNi45MzQzIDY1LjI3NTcgMjUuODkzMiA2NC40MTc4QzI1LjEwNTEgNjMuNzY4NCAyNC43MjQxIDYzLjU0MjYgMjQuMzMzOCA2My41OTU4Wk0zMS43MTY2IDgzLjI1OThDMzEuNjYwNSA4My4yNzQ5IDMxLjU5MjEgODMuMzEyNiAzMS41MTE1IDgzLjM3MjFDMzEuMjcwOSA4My41NSAzMC42NTM2IDgzLjc5IDMwLjEzNzkgODMuOTA1NEMyOS4zODc2IDg0LjA3MzQgMjkuMDQ5OCA4NC40OTMyIDI4LjQ0ODIgODYuMDA0NUMyOC4wMzQ3IDg3LjA0MzIgMjcuNDkxNSA4OC4xMTUzIDI3LjI0MTIgODguMzg2M0MyNi45OTA5IDg4LjY1NzMgMjYuNDc1IDg5LjU1MSAyNi4wOTQxIDkwLjM3MzJDMjQuOTQyMiA5Mi44NTkyIDIzLjUyODIgOTQuMDYxOCAxOS45MDExIDk1LjY0MjFDMTYuNjEwNyA5Ny4wNzU2IDE2LjU2NyA5Ny4xMDg2IDE1LjY2NzEgOTguOTgwMkMxNS4xMDI3IDEwMC4xNTQgMTQuNjc5MiAxMDEuNzU3IDE0LjU0OTkgMTAzLjIwNUMxNC4yNzMxIDEwNi4zMDIgMTMuNTk2MiAxMDcuMDMxIDkuNTIxMTggMTA4LjYxNkM1LjkxNDk3IDExMC4wMTkgMy41NzcwMiAxMTEuNzQ1IDMuNDE1ODMgMTEzLjEyM0MzLjI0MDQxIDExNC42MjMgMi41OTgxMyAxMTUuMDg0IDAuOTI5MjYgMTE0LjkwN0MwLjUzOTUyOSAxMTQuODY2IDAuMjQ0MzMyIDExNC44NTEgMCAxMTQuODgxVjEyMUMxLjAyNjU4IDExOS42NTggMi43MTkyMiAxMTguMzYxIDYuMjI0OTcgMTE2LjIwNkMxMC4zNjA0IDExMy42NjUgMTIuNzA2NSAxMTEuNTc1IDEzLjU3NzkgMTA5LjY1NEMxMy45MjYgMTA4Ljg4NyAxNC42NTQ3IDEwNy4zOTMgMTUuMTk3MSAxMDYuMzM0QzE1LjczOTUgMTA1LjI3NSAxNi4yODQ1IDEwMy45MDEgMTYuNDA4NCAxMDMuMjgxQzE2LjczMjQgMTAxLjY2IDE4LjMwMzIgMTAwLjM0OSAyMC42NjU5IDk5LjcyNkMyNS4xNzcgOTguNTM3MiAyNS43NDYxIDk4LjAxNjMgMjYuMzUyNSA5NC41MzU0QzI2Ljg1NjEgOTEuNjQ1MiAyOC42MTIgODguMDY2MSAzMC41MjY3IDg2LjAyNDZDMzEuMzA5NSA4NS4xOSAzMS45NDk1IDg0LjE4IDMxLjk0OTUgODMuNzc5MUMzMS45NDk1IDgzLjM3NjYgMzEuODg0OSA4My4yMTQ2IDMxLjcxNjYgODMuMjU5OFoiIGZpbGw9IiMzNTkxMzYiLz4KPHBhdGggZD0iTTguMDgxNzcgNzQuNDA4M0M2LjAzNDE5IDczLjg3MDUgNS44OTQ0MyA3My43MzAyIDYuMDQyNzEgNzIuMzYxNUM2LjEzMDI1IDcxLjU1MzMgNi4yNDAxNyA3MC44NTc1IDYuMjg2OTYgNzAuODE1NEM2LjMzMzc2IDcwLjc3MzMgNy43MDQ0MyA3MS4wNjg3IDkuMzMyOTIgNzEuNDcxOUMxMS45NzY4IDcyLjEyNjQgMTIuNDg2NiA3Mi4xMjM4IDE0LjA5NDcgNzEuNDQ3NEMxNi40MDQxIDcwLjQ3NiAxNy4yMzgzIDY4Ljc3NTMgMTYuNDAyMiA2Ni43NDMxQzE1Ljk0MjQgNjUuNjI1NyAxNC45MjA0IDY0LjgyODEgMTIuMjM4NCA2My40OTRDNy42NDE3IDYxLjIwNzUgNi41MzA2MiA1OS45OTkgNi4yNzQ2NSA1Ny4wMDc2QzUuODkzMzcgNTIuNTUxOSA4Ljg0Njg5IDQ5Ljk4MzYgMTQuMzUyMiA0OS45ODM2QzE1Ljk3MzMgNDkuOTgzNiAxNy44NDg3IDUwLjE5MzcgMTguNTE5NiA1MC40NTA1QzE5LjYzNjggNTAuODc4MSAxOS42OTM3IDUxLjAzOTkgMTkuMTk0NSA1Mi4zNzEyQzE4LjcxODIgNTMuNjQxNiAxOC41MDI3IDUzLjc2ODggMTcuNDg2MSA1My4zNzk2QzE1LjUxOTkgNTIuNjI3IDEyLjE5NDggNTIuODU1NCAxMS4wMDU4IDUzLjgyNDdDOS43OTA1OCA1NC44MTUzIDkuNTU3MzYgNTYuOTI1MiAxMC41MzI0IDU4LjEwNzlDMTAuODcyNyA1OC41MjA3IDEyLjkwODcgNTkuNzc5IDE1LjA1NjkgNjAuOTA0MUMxOC40NjI4IDYyLjY4ODEgMTkuMDgyIDYzLjIyMTQgMTkuODk1MSA2NS4wNzE5QzIxLjExNTIgNjcuODQ4NyAyMC42Nzg4IDcwLjM0MzcgMTguNjA3OCA3Mi40MzIzQzE2LjE2MDcgNzQuOTAwMiAxMi41NDEzIDc1LjU3OTYgOC4wODE3NyA3NC40MDgzWk0yOC45MzI4IDc0LjM4MjNDMjUuMjM2NSA3Mi45OTU4IDIzLjcxMzggNzAuNTQwNSAyMy43MTM4IDY1Ljk2NjdDMjMuNzEzOCA2Mi41MDM0IDI0LjU1ODggNjAuNDY5NyAyNi44MzMzIDU4LjQ1OTNDMjguMzk2NCA1Ny4wNzc2IDI4Ljk4NjcgNTYuODY4NiAzMS4zMjY4IDU2Ljg2ODZDMzIuODE2OCA1Ni44Njg2IDM0LjY4MjEgNTcuMjEyOSAzNS41MDA1IDU3LjYzODlDMzcuMzE2OSA1OC41ODQ2IDM4LjgzMTkgNjEuNzQyNiAzOC44NDY2IDY0LjYxNDNMMzguODU3MyA2Ni43MDQ0SDMyLjk0MjNIMjcuMDI3M0wyNy4zNDIgNjguMDU2OUMyNy41MTUgNjguODAwNyAyOC4zMzU0IDcwLjAzOTEgMjkuMTY1IDcwLjgwOUMzMC41NzczIDcyLjExOTYgMzAuODk0OSA3Mi4xOTc5IDM0LjE1NDcgNzIuMDM4NUMzNy4zOTM4IDcxLjg4MDEgMzcuNjQ2MyA3MS45NDA5IDM3Ljc4NCA3Mi45MTIyQzM3Ljk2MjYgNzQuMTcyNSAzNy4xODgyIDc0LjUyMzYgMzMuNTI0MiA3NC44NDM0QzMxLjgwODUgNzQuOTkzMiAzMC4xMDU2IDc0LjgyMjIgMjguOTMyOCA3NC4zODIzWk0zNS4wODgxIDYyLjUyNDJDMzQuOTU1OCA2MC42NzMgMzMuMzY0NCA1OS4zMjc2IDMxLjMwNzIgNTkuMzI3NkMyOS43MDg2IDU5LjMyNzYgMjguNzI4MSA2MC4wOTA5IDI3LjY2MiA2Mi4xNjU1QzI2Ljc0NTcgNjMuOTQ4NiAyNy40MTIgNjQuMjk2NSAzMS40NDg4IDY0LjE0MjVMMzUuMTkzNiA2My45OTk2TDM1LjA4ODEgNjIuNTI0MlpNNDQuMzkxNCA3NC4zNzA4QzQyLjU3MTEgNzMuNTQ2OCA0MS43MDU5IDcyLjEyODYgNDEuNjQyMSA2OS44NjQ3QzQxLjU0MTYgNjYuMjk0NCA0NC4xMTEgNjQuMTQ2OCA0OS4yNTc2IDYzLjQ5OTRDNTEuODQxNCA2My4xNzQ0IDUyLjA0OTUgNjMuMDU3NyA1MS43NTkzIDYyLjA5NjlDNTAuOTY1IDU5LjQ2NjYgNDguMjQyMyA1OC41NjQ0IDQ0LjkyNzYgNTkuODMzMUM0My40MzAzIDYwLjQwNjMgNDMuNDg2OCA2MC40MzEzIDQzLjA1MTQgNTkuMDAzMkM0Mi43ODAxIDU4LjExMzUgNDIuOTk4MiA1Ny44NTExIDQ0LjM5NSA1Ny4zODdDNDYuOTM2NCA1Ni41NDI3IDUxLjM3NDUgNTYuNjM0MiA1Mi43Nzc5IDU3LjU2QzU0LjkzOTIgNTguOTg1NyA1NS4zNzUxIDYwLjYwMTkgNTUuNTQ4NyA2Ny44MzI4TDU1LjcxMDYgNzQuNTc0NUw1NC4xMjMgNzQuNTc0QzUzLjAzNzcgNzQuNTczMyA1Mi41MzUzIDc0LjMzOTYgNTIuNTM1MyA3My44MzU0QzUyLjUzNTMgNzIuOTA2NiA1Mi4wNTI4IDcyLjkwMjQgNTAuNzUyIDczLjgxOTZDNDkuMjAxMyA3NC45MTMgNDYuMTY5NyA3NS4xNzU3IDQ0LjM5MTQgNzQuMzcwOFpNNTAuOTQ3NyA3MC44MTc1QzUxLjcwOTkgNjkuOTczNSA1Mi4wNDY4IDY4Ljk4NiA1Mi4wNDY4IDY3LjU5NjZWNjUuNTkyOUw0OS44MTgyIDY1Ljg0NThDNDUuNzE5IDY2LjMxMSA0My44NzQ5IDY5LjUxMTggNDYuNTA1OCA3MS41OTUzQzQ4LjAzNzEgNzIuODA3OSA0OS4zNTk0IDcyLjU3NjMgNTAuOTQ3NyA3MC44MTc1Wk05Mi4wMDIgNzQuNTc4OEM4OC40MDI0IDczLjQ4NjQgODYuMjQxOSA3MC4yNDY2IDg2LjI0MTkgNjUuOTQxQzg2LjI0MTkgNjAuNDA0MiA4OS41MjExIDU2LjgzMjUgOTQuNjQxIDU2Ljc5MjVDOTguNjk4OSA1Ni43NjA5IDEwMS4yNjMgNTkuNDY3OSAxMDEuNTAzIDY0LjAzNjZMMTAxLjYzIDY2LjQ1ODVMOTYuMjU2MiA2Ni41ODg1QzkzLjMwMDcgNjYuNjYgOTAuNjE1OCA2Ni44MTYgOTAuMjg5NyA2Ni45MzUxQzg5LjM4MTggNjcuMjY2OCA5MC40Nzg4IDY5Ljk0NyA5Mi4wMzA2IDcxLjE4ODZDOTMuMTU5OSA3Mi4wOTIxIDkzLjc3MSA3Mi4xOTEyIDk2LjgxNzggNzEuOTY1QzEwMC4xOTMgNzEuNzE0NCAxMDAuMzE4IDcxLjc0NDkgMTAwLjUzMSA3Mi44Njg0QzEwMC43MyA3My45MTMzIDEwMC41MDkgNzQuMDgzNSA5OC4zNTc5IDc0LjU0OEM5NS42MTAzIDc1LjE0MTIgOTMuODgyOSA3NS4xNDk2IDkyLjAwMiA3NC41Nzg4Wk05Ny45NjU5IDYzLjA4MjNDOTcuOTY1OSA2MS4xMjMxIDk2LjE4NTggNTkuMzI3NiA5NC4yNDM0IDU5LjMyNzZDOTIuMjQ5NCA1OS4zMjc2IDkwLjE0OTkgNjEuMjU2MiA5MC4xNDk5IDYzLjA4NzhDOTAuMTQ5OSA2NC4yMDcgOTAuMjc5NyA2NC4yNDU1IDk0LjA1NzkgNjQuMjQ1NUM5Ny44NDE0IDY0LjI0NTUgOTcuOTY1OSA2NC4yMDg0IDk3Ljk2NTkgNjMuMDgyM1pNMTEwLjA0MiA3NC40N0MxMDYuMTQ5IDczLjQzIDEwMy45MyA2OS40MDI5IDEwNC41NCA2NC40ODYyQzEwNS4xNjYgNTkuNDQ0MiAxMDcuOTUxIDU2Ljg3MzYgMTEyLjc5MiA1Ni44NzA1QzExNS4zNzQgNTYuODY5IDExNS44MDUgNTcuMDI0OSAxMTcuMjYgNTguNDlDMTE4Ljg3OCA2MC4xMTg1IDExOS45NDEgNjIuODM0MyAxMTkuOTQ2IDY1LjM1MkwxMTkuOTQ5IDY2LjcwNDRIMTE0LjA4N0MxMDguNDg4IDY2LjcwNDQgMTA4LjIyNSA2Ni43NTAxIDEwOC4yMjUgNjcuNzIwM0MxMDguMjI1IDcwLjcxMjIgMTEwLjk1MiA3Mi4zNDQyIDExNS4zNTcgNzEuOTg4MkMxMTguMTM2IDcxLjc2MzcgMTE4LjQyMyA3MS44MzE1IDExOC42NjcgNzIuNzY3NEMxMTguOTk4IDc0LjA0NDMgMTE4Ljk2OCA3NC4wNzIgMTE2LjY5IDc0LjYwNDNDMTE0LjM2NyA3NS4xNDcgMTEyLjQzIDc1LjEwNzggMTEwLjA0MiA3NC40N1pNMTE2LjExOSA2Mi43NzAxQzExNS44NjQgNjAuODc4NyAxMTQuMTU5IDU5LjMyNzYgMTEyLjMzNSA1OS4zMjc2QzExMC41NjYgNTkuMzI3NiAxMDguMjI0IDYxLjYwMTQgMTA4LjIyNCA2My4zMTk1QzEwOC4yMjQgNjQuMjEyOCAxMDguNTI3IDY0LjI3NDYgMTEyLjI1NSA2NC4xNDI0QzExNi4yNDEgNjQuMDAxMSAxMTYuMjgzIDYzLjk4NjIgMTE2LjExOSA2Mi43NzAxWk0xMjYuODk0IDc0LjI0N0MxMjQuODI2IDczLjIyMDggMTIzLjUzMiA3MS4zNTg5IDEyMi44OTQgNjguNDg4OUMxMjEuOTc1IDY0LjM2MDUgMTIzLjA3NCA2MC40OTE0IDEyNS44MyA1OC4xNTY3QzEyNy44MjcgNTYuNDY1NyAxMzEuODM4IDU2LjMzMDUgMTMzLjgwNSA1Ny44ODc5TDEzNS4wOTIgNTguOTA3MlY1My45NTM2VjQ5SDEzNy4wNDZIMTM5VjYxLjc4NjVWNzQuNTczMUgxMzcuMjlDMTM1Ljc2MiA3NC41NzMxIDEzNS41OCA3NC40MzMxIDEzNS41OCA3My4yNTc5VjcxLjk0MjdMMTM0LjQ4MSA3Mi45NzkyQzEzMi4yMzMgNzUuMDk5OCAxMjkuNTI1IDc1LjU1MjMgMTI2Ljg5NCA3NC4yNDdaTTEzMy42NjQgNzAuNjc2NkMxMzQuOTc2IDY5LjM1NTcgMTM1LjA5MiA2OC45NTM3IDEzNS4wOTIgNjUuNzIwOUMxMzUuMDkyIDYyLjQ1NzIgMTM0Ljk4NSA2Mi4wOTQ1IDEzMy42MDcgNjAuNzA3NkMxMzIuMzY1IDU5LjQ1NzUgMTMxLjg2NSA1OS4yNjIyIDEzMC41NTQgNTkuNTE1NEMxMjYuMTQ0IDYwLjM2NzIgMTI0LjgzNSA2OC4xNDIzIDEyOC41OTggNzEuMTIyOUMxMzAuMzQ4IDcyLjUwODEgMTMxLjk4OCA3Mi4zNjM1IDEzMy42NjQgNzAuNjc2NlpNNjEuNDEwNyA2Ni4wNTg0QzYwLjAyMiA2MS4zNzU0IDU4Ljg4NTkgNTcuMzkxOSA1OC44ODU5IDU3LjIwNjJDNTguODg1OSA1Ny4wMjA2IDU5LjcyNzQgNTYuODY4NiA2MC43NTU5IDU2Ljg2ODZINjIuNjI1OUw2NC4yNTUxIDYzLjg3NjZDNjUuMTUxMiA2Ny43MzEgNjUuOTU2IDcwLjY2MzMgNjYuMDQzNyA3MC4zOTI5QzY2LjEzMTMgNzAuMTIyNCA2Ny4wNTU3IDY3LjAyNDEgNjguMDk3OCA2My41MDc4TDY5Ljk5MjYgNTcuMTE0NUg3MS41OTY5SDczLjIwMTJMNzQuODg2NSA2My4yNjE5Qzc1LjgxMzQgNjYuNjQzIDc2LjY4MzIgNjkuNzQxMiA3Ni44MTk0IDcwLjE0N0M3Ni45NTU3IDcwLjU1MjcgNzcuODQxOSA2Ny43MzEgNzguNzg5IDYzLjg3NjZMODAuNTEwOSA1Ni44Njg2SDgyLjQ1MTdDODMuOTUxNyA1Ni44Njg2IDg0LjMzMDMgNTcuMDMxNiA4NC4xMTkgNTcuNTg2QzgzLjk2ODYgNTcuOTgwNiA4Mi42OTU2IDYxLjk2NDEgODEuMjkwMiA2Ni40MzgyTDc4LjczNDkgNzQuNTczMUg3Ni45NjM1SDc1LjE5MjFMNzMuNTgxMiA2OC43OTQ1QzcyLjY5NTIgNjUuNjE2MyA3MS44NTg1IDYyLjY4NDEgNzEuNzIxNyA2Mi4yNzgzQzcxLjQ0NjUgNjEuNDYxNiA3MS41OTExIDYxLjA3MzMgNjkuMTExNCA2OS4yODYzTDY3LjUxNTIgNzQuNTczMUg2NS43MjU0SDYzLjkzNTZMNjEuNDEwNyA2Ni4wNTg0WiIgZmlsbD0id2hpdGUiLz4KPHBhdGggZD0iTTE5Ljk3NDYgMTE5LjI3NEMxOS4wMTU4IDExOC4xMTkgMjAuMzM4MSAxMTYuMTY2IDIxLjU2MDQgMTE2LjkzMkMyMi43NjMyIDExNy42ODYgMjIuMzMwNSAxMTkuNzYxIDIwLjk3MDQgMTE5Ljc2MUMyMC42NDUgMTE5Ljc2MSAyMC4xOTY5IDExOS41NDIgMTkuOTc0NiAxMTkuMjc0WiIgZmlsbD0idXJsKCNwYWludDNfcmFkaWFsXzY3OV8xOTEwKSIvPgo8cGF0aCBkPSJNMjMuODM2IDExMi42M0MyMi4xMTUyIDExMS42ODkgMjEuODAyOSAxMDkuNzk3IDIzLjE0NjUgMTA4LjQ1M0MyNC4wODU3IDEwNy41MTQgMjUuMjEgMTA3LjM5NCAyNi4yNTA3IDEwOC4xMjNDMjcuODU3MSAxMDkuMjQ5IDI3LjQ2NTUgMTEyLjAxMyAyNS42MDc5IDExMi42NkMyNS42MDc5IDExMi42NiAyNC40NzQ1IDExMi45NzMgMjMuODM2IDExMi42M1oiIGZpbGw9InVybCgjcGFpbnQ0X3JhZGlhbF82NzlfMTkxMCkiLz4KPHBhdGggZD0iTTMzLjc4NTUgMTIwLjAzNkMzMy43ODU1IDEyMC4xODggMzMuNzU1NyAxMjAuMzM4IDMzLjY5NzYgMTIwLjQ3OUMzMy42Mzk1IDEyMC42MTkgMzMuNTU0NCAxMjAuNzQ2IDMzLjQ0NzEgMTIwLjg1M0MzMy4zMzk4IDEyMC45NjEgMzMuMjEyNCAxMjEuMDQ2IDMzLjA3MjMgMTIxLjEwNEMzMi45MzIxIDEyMS4xNjIgMzIuNzgxOCAxMjEuMTkyIDMyLjYzMDEgMTIxLjE5MkMzMi40NzgzIDEyMS4xOTIgMzIuMzI4MSAxMjEuMTYyIDMyLjE4NzkgMTIxLjEwNEMzMi4wNDc3IDEyMS4wNDYgMzEuOTIwMyAxMjAuOTYxIDMxLjgxMyAxMjAuODUzQzMxLjcwNTcgMTIwLjc0NiAzMS42MjA2IDEyMC42MTkgMzEuNTYyNiAxMjAuNDc5QzMxLjUwNDUgMTIwLjMzOCAzMS40NzQ2IDEyMC4xODggMzEuNDc0NiAxMjAuMDM2QzMxLjQ3NDYgMTE5LjczIDMxLjU5NjMgMTE5LjQzNiAzMS44MTMgMTE5LjIxOUMzMi4wMjk3IDExOS4wMDMgMzIuMzIzNiAxMTguODgxIDMyLjYzMDEgMTE4Ljg4MUMzMi45MzY1IDExOC44ODEgMzMuMjMwNCAxMTkuMDAzIDMzLjQ0NzEgMTE5LjIxOUMzMy42NjM4IDExOS40MzYgMzMuNzg1NSAxMTkuNzMgMzMuNzg1NSAxMjAuMDM2WiIgZmlsbD0iYmxhY2siLz4KPHBhdGggZD0iTTEyMC43MjEgMTE4LjI3NkMxMjAuNzIxIDExOC42MjMgMTIwLjU2NSAxMTguOTU2IDEyMC4yODcgMTE5LjIwMUMxMjAuMDA5IDExOS40NDYgMTE5LjYzMiAxMTkuNTg0IDExOS4yMzkgMTE5LjU4NEMxMTguODQ2IDExOS41ODQgMTE4LjQ2OSAxMTkuNDQ2IDExOC4xOTEgMTE5LjIwMUMxMTcuOTEzIDExOC45NTYgMTE3Ljc1NyAxMTguNjIzIDExNy43NTcgMTE4LjI3NkMxMTcuNzU3IDExOC4xMDUgMTE3Ljc5NSAxMTcuOTM1IDExNy44NyAxMTcuNzc2QzExNy45NDQgMTE3LjYxNyAxMTguMDUzIDExNy40NzMgMTE4LjE5MSAxMTcuMzUyQzExOC4zMjkgMTE3LjIzIDExOC40OTIgMTE3LjEzNCAxMTguNjcyIDExNy4wNjhDMTE4Ljg1MSAxMTcuMDAzIDExOS4wNDQgMTE2Ljk2OSAxMTkuMjM5IDExNi45NjlDMTE5LjQzMyAxMTYuOTY5IDExOS42MjYgMTE3LjAwMyAxMTkuODA2IDExNy4wNjhDMTE5Ljk4NiAxMTcuMTM0IDEyMC4xNDkgMTE3LjIzIDEyMC4yODcgMTE3LjM1MkMxMjAuNDI0IDExNy40NzMgMTIwLjUzMyAxMTcuNjE3IDEyMC42MDggMTE3Ljc3NkMxMjAuNjgyIDExNy45MzUgMTIwLjcyMSAxMTguMTA1IDEyMC43MjEgMTE4LjI3NloiIGZpbGw9IiM1OTY4NkYiLz4KPGRlZnM+CjxsaW5lYXJHcmFkaWVudCBpZD0icGFpbnQwX2xpbmVhcl82NzlfMTkxMCIgeDE9IjExIiB5MT0iLTQuNzc3NjhlLTA3IiB4Mj0iMTY0LjUiIHkyPSIxNTAuNSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiPgo8c3RvcCBzdG9wLWNvbG9yPSIjNjRCN0ZGIi8+CjxzdG9wIG9mZnNldD0iMSIgc3RvcC1jb2xvcj0iIzAwNURBRCIvPgo8L2xpbmVhckdyYWRpZW50Pgo8cmFkaWFsR3JhZGllbnQgaWQ9InBhaW50MV9yYWRpYWxfNjc5XzE5MTAiIGN4PSIwIiBjeT0iMCIgcj0iMSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiIGdyYWRpZW50VHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTI0LjkwNCAxMjguNjM2KSByb3RhdGUoLTAuMDQ4NTUyKSBzY2FsZSgzNi4yODAyIDMxLjQwMzIpIj4KPHN0b3Agc3RvcC1jb2xvcj0iIzExNzdDRSIvPgo8c3RvcCBvZmZzZXQ9IjEiIHN0b3AtY29sb3I9IiM3RUM3RkYiIHN0b3Atb3BhY2l0eT0iMCIvPgo8L3JhZGlhbEdyYWRpZW50Pgo8cmFkaWFsR3JhZGllbnQgaWQ9InBhaW50Ml9yYWRpYWxfNjc5XzE5MTAiIGN4PSIwIiBjeT0iMCIgcj0iMSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiIGdyYWRpZW50VHJhbnNmb3JtPSJ0cmFuc2xhdGUoMzkuMzUxMiAxMjEuNDYpIHJvdGF0ZSgtMC4wNTczKSBzY2FsZSg1MC4yMjc2IDUxLjMwOTEpIj4KPHN0b3Agc3RvcC1jb2xvcj0iIzExNzdDRSIvPgo8c3RvcCBvZmZzZXQ9IjEiIHN0b3AtY29sb3I9IiM3RUM3RkYiIHN0b3Atb3BhY2l0eT0iMCIvPgo8L3JhZGlhbEdyYWRpZW50Pgo8cmFkaWFsR3JhZGllbnQgaWQ9InBhaW50M19yYWRpYWxfNjc5XzE5MTAiIGN4PSIwIiBjeT0iMCIgcj0iMSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiIGdyYWRpZW50VHJhbnNmb3JtPSJ0cmFuc2xhdGUoMjAuOTU0OCAxMTguMjYpIHNjYWxlKDEuMzA5MjkgMS41MDEyNykiPgo8c3RvcCBzdG9wLWNvbG9yPSIjODhBM0QwIi8+CjxzdG9wIG9mZnNldD0iMSIgc3RvcC1jb2xvcj0iIzVEODNCRiIvPgo8L3JhZGlhbEdyYWRpZW50Pgo8cmFkaWFsR3JhZGllbnQgaWQ9InBhaW50NF9yYWRpYWxfNjc5XzE5MTAiIGN4PSIwIiBjeT0iMCIgcj0iMSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiIGdyYWRpZW50VHJhbnNmb3JtPSJ0cmFuc2xhdGUoMjQuNzg0MyAxMTAuMjIxKSBzY2FsZSgyLjQ3Mjc3IDIuNTY5NTQpIj4KPHN0b3Agc3RvcC1jb2xvcj0iIzg4QTNEMCIvPgo8c3RvcCBvZmZzZXQ9IjEiIHN0b3AtY29sb3I9IiM1RDgzQkYiLz4KPC9yYWRpYWxHcmFkaWVudD4KPC9kZWZzPgo8L3N2Zz4K - keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "host"], ["spec", "topology"], ["spec", "replicationFactor"], ["spec", "db"], ["spec", "db", "replicas"], ["spec", "db", "size"], ["spec", "db", "storageClass"], ["spec", "db", "resources"], ["spec", "db", "resourcesPreset"], ["spec", "master"], ["spec", "master", "replicas"], ["spec", "master", "resources"], ["spec", "master", "resourcesPreset"], ["spec", "master", "volumeSizeLimitMB"], ["spec", "filer"], ["spec", "filer", "replicas"], ["spec", "filer", "resources"], ["spec", "filer", "resourcesPreset"], ["spec", "filer", "grpcHost"], ["spec", "filer", "grpcPort"], ["spec", "filer", "whitelist"], ["spec", "volume"], ["spec", "volume", "replicas"], ["spec", "volume", "size"], ["spec", "volume", "storageClass"], ["spec", "volume", "diskType"], ["spec", "volume", "resources"], ["spec", "volume", "resourcesPreset"], ["spec", "volume", "zones"], ["spec", "volume", "pools"], ["spec", "s3"], ["spec", "s3", "replicas"], ["spec", "s3", "resources"], ["spec", "s3", "resourcesPreset"]] + keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "host"], ["spec", "topology"], ["spec", "replicationFactor"], ["spec", "db"], ["spec", "db", "replicas"], ["spec", "db", "size"], ["spec", "db", "storageClass"], ["spec", "db", "resources"], ["spec", "db", "resourcesPreset"], ["spec", "master"], ["spec", "master", "replicas"], ["spec", "master", "resources"], ["spec", "master", "resourcesPreset"], ["spec", "master", "volumeSizeLimitMB"], ["spec", "filer"], ["spec", "filer", "replicas"], ["spec", "filer", "resources"], ["spec", "filer", "resourcesPreset"], ["spec", "filer", "grpcHost"], ["spec", "filer", "grpcPort"], ["spec", "filer", "whitelist"], ["spec", "volume"], ["spec", "volume", "replicas"], ["spec", "volume", "size"], ["spec", "volume", "storageClass"], ["spec", "volume", "diskType"], ["spec", "volume", "resources"], ["spec", "volume", "resourcesPreset"], ["spec", "volume", "zones"], ["spec", "volume", "pools"], ["spec", "s3"], ["spec", "s3", "replicas"], ["spec", "s3", "resources"], ["spec", "s3", "resourcesPreset"], ["spec", "lifecycle"], ["spec", "lifecycle", "enabled"]] secrets: exclude: [] include: [] diff --git a/packages/system/seaweedfs/tests/lifecycle_test.yaml b/packages/system/seaweedfs/tests/lifecycle_test.yaml new file mode 100644 index 000000000..84680cb5f --- /dev/null +++ b/packages/system/seaweedfs/tests/lifecycle_test.yaml @@ -0,0 +1,138 @@ +suite: seaweedfs runs the S3 lifecycle worker + +# SeaweedFS stores a bucket lifecycle configuration on PUT and acts on it only +# from the s3_lifecycle job of the plugin worker, which the admin server +# schedules (weed/worker/tasks/s3_lifecycle/handler.go). Without an admin and a +# worker in the release, a rule is accepted and nothing ever expires. + +templates: + - charts/seaweedfs/templates/admin/admin-statefulset.yaml + - charts/seaweedfs/templates/worker/worker-deployment.yaml + +release: + name: seaweedfs-system + namespace: tenant-root + +tests: + - it: runs the admin server that schedules lifecycle jobs + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + set: &enabled + seaweedfs: + admin: + enabled: true + secret: + existingSecret: seaweedfs-admin-credentials + worker: + enabled: true + asserts: + - isKind: + of: StatefulSet + - equal: + path: metadata.name + value: seaweedfs-admin + + # The admin keeps its job state and any change an operator makes in its UI + # in -dataDir; on an emptyDir a restart would silently undo them. + - it: keeps the admin data directory on a persistent volume + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + set: *enabled + asserts: + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '-dataDir=/data' + - equal: + path: spec.volumeClaimTemplates[0].metadata.name + value: admin-data + - equal: + path: spec.volumeClaimTemplates[0].spec.resources.requests.storage + value: 1Gi + + - it: runs the worker against the admin of the same release + template: charts/seaweedfs/templates/worker/worker-deployment.yaml + set: *enabled + asserts: + - isKind: + of: Deployment + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '-admin=seaweedfs-admin\.tenant-root:23646\.33646 ' + + # The worker executes every job type it registers for. Lifecycle is the only + # one this release asks for: vacuum, balance and erasure coding move data and + # are not something enabling lifecycle should start doing as a side effect. + - it: registers the worker for the s3_lifecycle job only + template: charts/seaweedfs/templates/worker/worker-deployment.yaml + set: *enabled + asserts: + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '-jobType=s3_lifecycle ' + + - it: runs no Lance worker container, since the Lance catalog is off + template: charts/seaweedfs/templates/worker/worker-deployment.yaml + set: *enabled + asserts: + - lengthEqual: + path: spec.template.spec.containers + count: 1 + + - it: gives the admin and the worker explicit resources + set: *enabled + asserts: + - exists: + path: spec.template.spec.containers[0].resources.requests.memory + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + - exists: + path: spec.template.spec.containers[0].resources.limits.memory + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + - equal: + path: spec.template.spec.containers[0].resources.requests.memory + value: 64Mi + template: charts/seaweedfs/templates/worker/worker-deployment.yaml + + # The admin serves an unauthenticated UI unless a password is set, and a + # tenant namespace admits ingress from the whole cluster. The credentials come + # from a Secret the parent release keeps; the values here only wire them in. + - it: reads the admin credentials from the named Secret + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + set: *enabled + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: WEED_ADMIN_PASSWORD + valueFrom: + secretKeyRef: + name: seaweedfs-admin-credentials + key: adminPassword + - contains: + path: spec.template.spec.containers[0].env + content: + name: WEED_ADMIN_USER + valueFrom: + secretKeyRef: + name: seaweedfs-admin-credentials + key: adminUser + + - it: renders neither the admin nor the worker unless the parent turns them on + asserts: + - hasDocuments: + count: 0 + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + - hasDocuments: + count: 0 + template: charts/seaweedfs/templates/worker/worker-deployment.yaml + + # Every other component here logs to stderr; the upstream default for these + # two writes to a file in an emptyDir that kubectl logs never shows. + - it: logs the admin and the worker to stderr + set: *enabled + asserts: + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '-logtostderr=true' + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '-logtostderr=true' + template: charts/seaweedfs/templates/worker/worker-deployment.yaml diff --git a/packages/system/seaweedfs/values.yaml b/packages/system/seaweedfs/values.yaml index e1eba8a58..1fce951cf 100644 --- a/packages/system/seaweedfs/values.yaml +++ b/packages/system/seaweedfs/values.yaml @@ -319,6 +319,47 @@ seaweedfs: region: "" sidecar: image: "ghcr.io/cozystack/cozystack/objectstorage-sidecar:v1.6.0@sha256:4d665becd3399c2fe1e8a66e15d14d16e4a5a75395fb0ae65cff46679a8989f1" + # A bucket lifecycle rule is stored on PUT and acted on only by the + # s3_lifecycle job of the plugin worker, scheduled by the admin server once a + # day (weed/worker/tasks/s3_lifecycle/handler.go at 4.45). The parent chart + # sends both enabled flags from its lifecycle.enabled, together with the + # admin password below; off here, so a render without the parent never + # starts an admin without one. + admin: + enabled: false + # Without a password weed admin serves its UI unauthenticated, and a tenant + # namespace admits ingress from the whole cluster. The parent chart names + # the Secret it keeps; these are the keys it writes. + secret: + userKey: adminUser + pwKey: adminPassword + # Job state and anything changed in the admin UI live in -dataDir. + data: + type: "persistentVolumeClaim" + size: "1Gi" + logs: + type: "" + resources: + requests: + cpu: "50m" + memory: "64Mi" + limits: + cpu: "500m" + memory: "256Mi" + worker: + enabled: false + # Vacuum, balance and erasure coding move data; enabling lifecycle must + # not start them. + jobType: "s3_lifecycle" + logs: + type: "" + resources: + requests: + cpu: "50m" + memory: "64Mi" + limits: + cpu: "1" + memory: "512Mi" certificates: commonName: "SeaweedFS CA" ipAddresses: [] From 27e0aa288079c1715608ae0286bcaa59c87c03ce Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:19:01 +0300 Subject: [PATCH 112/518] docs(backups): say who applies a bucket lifecycle rule The notes that send a tenant to a bucket lifecycle rule for a backup object Cozystack leaves behind read as if some policy were already in place. None is: the rule has to be set on the bucket, and it is applied only by an object store that implements lifecycle, which the bundled SeaweedFS now does once a day while lifecycle.enabled is on. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- examples/backups/kafka/92-scenario-user-restore.md | 5 +---- examples/backups/nats/92-scenario-user-restore.md | 5 +---- internal/backupcontroller/psmdbstrategy_controller.go | 7 ++++--- 3 files changed, 6 insertions(+), 11 deletions(-) diff --git a/examples/backups/kafka/92-scenario-user-restore.md b/examples/backups/kafka/92-scenario-user-restore.md index 3215fe056..5416500ed 100644 --- a/examples/backups/kafka/92-scenario-user-restore.md +++ b/examples/backups/kafka/92-scenario-user-restore.md @@ -71,9 +71,6 @@ The script: had already replayed. The Pod log names each topic before its replay and again once it landed; empty or delete every topic it shows as started, not only the one the refusal names, then re-run. -- The tarball in object storage is not managed by the Cozystack `Backup` - lifecycle: deleting the `Backup` CR removes the reference but leaves the S3 - object in place. Tenants who need to reclaim space should delete the object - (or set a bucket lifecycle policy). +- The tarball in object storage is not managed by the Cozystack `Backup` lifecycle: deleting the `Backup` CR removes the reference but leaves the S3 object in place. Tenants who need to reclaim space should delete the object, or set a lifecycle rule on the bucket. The Cozystack SeaweedFS applies such rules once a day while its `lifecycle.enabled` is on, which is the default; another object store applies them only if it implements bucket lifecycle. - Consumer-group offsets, topic configs, ACLs and users are not part of the backup; consumers start from the restored topic's fresh offsets. diff --git a/examples/backups/nats/92-scenario-user-restore.md b/examples/backups/nats/92-scenario-user-restore.md index 9f0e7c02a..98c51a5cc 100644 --- a/examples/backups/nats/92-scenario-user-restore.md +++ b/examples/backups/nats/92-scenario-user-restore.md @@ -55,9 +55,6 @@ The script: - Both flows rely on the strategy template branching on `.Mode`. See `01-create-strategy.sh`. -- The tarball in object storage is not managed by the Cozystack `Backup` - lifecycle: deleting the `Backup` CR removes the reference but leaves the S3 - object in place. Tenants who need to reclaim space should delete the object - (or set a bucket lifecycle policy). +- The tarball in object storage is not managed by the Cozystack `Backup` lifecycle: deleting the `Backup` CR removes the reference but leaves the S3 object in place. Tenants who need to reclaim space should delete the object, or set a lifecycle rule on the bucket. The Cozystack SeaweedFS applies such rules once a day while its `lifecycle.enabled` is on, which is the default; another object store applies them only if it implements bucket lifecycle. - `nats stream restore` recreates the stream's configuration from the snapshot; consumers are restored as captured at backup time. diff --git a/internal/backupcontroller/psmdbstrategy_controller.go b/internal/backupcontroller/psmdbstrategy_controller.go index 0ead1bc58..819274f71 100644 --- a/internal/backupcontroller/psmdbstrategy_controller.go +++ b/internal/backupcontroller/psmdbstrategy_controller.go @@ -1376,9 +1376,10 @@ func (r *BackupJobReconciler) createMongoDBBackupArtifact( // annotation (honoured before any apiserver read, so it still frees a Backup // whose operator CR has become unreadable) and a going-away namespace (the psmdb // delete-backup exec cannot run there) each make the driver strip its own -// finalizer and release, leaving the object for a bucket lifecycle policy. This -// bounds only what the driver contributes: a Get that fails with something other -// than NotFound/NoMatchError still requeues (below), and the psmdb operator's own +// finalizer and release, leaving the object to a lifecycle rule on the bucket, +// if one is set, or to manual cleanup. This bounds only what the driver +// contributes: a Get that fails with something other than +// NotFound/NoMatchError still requeues (below), and the psmdb operator's own // release-lock finalizer keeps the CR Terminating while the operator is down, // whatever the driver strips. A down operator in a live namespace is likewise // not auto-released: the Backup stays Terminating as a visible signal until an From 3a10f956ccefac3990c3319cdf5c6889ee63dcde Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:30:54 +0300 Subject: [PATCH 113/518] test(backups): cover a restore that starts on an idle ClickHouse sidecar The strategy suite only exercised restore against a busy sidecar, so a gate that wrongly blocked restore on an idle one would have stayed green. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/altinity-backup-strategy.bats | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats index 61729bf00..4703a861c 100644 --- a/hack/altinity-backup-strategy.bats +++ b/hack/altinity-backup-strategy.bats @@ -117,10 +117,17 @@ run_script() { render_script "$tmp/s.sh" backup myapp run_script "$tmp" grep -q '^POST .*/backup/create_remote?name=clickhouse-myapp-' "$tmp/state/requests" - # All three busy answers were consumed before the operation was started. [ "$(cat "$tmp/state/busy")" = 0 ] } +@test "restore on an idle sidecar starts restore_remote of the listed archive" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + render_script "$tmp/s.sh" restore myapp + run_script "$tmp" + grep -q '^POST .*/backup/restore_remote/clickhouse-myapp-20260101T000000Z$' "$tmp/state/requests" +} + @test "backup fails without starting create_remote when the sidecar stays busy" { tmp=$(mktemp -d) make_stubs "$tmp/bin" "$tmp/state" From e7529520471c41053970f708993186f2eeff48ae Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:31:29 +0300 Subject: [PATCH 114/518] test(clickhouse): back up and restore a running release through the system bucket A ClickHouse release on the system bucket starts before the platform projects cozy-backups-creds into its namespace, and no suite exercised that order: the round-trips bring their own Bucket and endpoint, so the sidecar never had to pick up credentials that arrived after it started. The new Test starts the release without the projected Secret and runs a backup and an in-place restore through cozy-default against the bucket as advertised. In CI that host neither resolves nor has a certificate, so the step makes it reachable from tenant-root for its own duration and proves the served certificate before any job starts, which keeps a DNS or TLS problem from reading as a backup failure. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../clickhouse/chainsaw-test.yaml | 176 +++++++++++++++++ .../clickhouse/clickhouse-system-bucket.yaml | 28 +++ .../clickhouse/system-bucket-roundtrip.sh | 185 ++++++++++++++++++ .../clickhouse/system-bucket-tls.yaml | 42 ++++ 4 files changed, 431 insertions(+) create mode 100644 hack/e2e-chainsaw/clickhouse/clickhouse-system-bucket.yaml create mode 100755 hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh create mode 100644 hack/e2e-chainsaw/clickhouse/system-bucket-tls.yaml diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index 939e8f69a..f5b444a84 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -377,3 +377,179 @@ spec: set -eu cd ../../.. NAMESPACE="$NAMESPACE" examples/backups/clickhouse/cleanup.sh +--- +# Backup and in-place restore through the shipped cozy-default BackupClass and +# the platform system bucket, for a ClickHouse whose Pod started before the +# platform projected cozy-backups-creds into its namespace. The sidecar has to +# pick those credentials up while running; until it could, the first BackupJob +# of such a release failed with "no EC2 IMDS role found" and kept failing until +# the Pod was restarted. +# +# Unlike clickhouse-2-backup-roundtrip this targets the system bucket as +# advertised, https://s3.example.org, so system-bucket-roundtrip.sh makes that +# name reachable from tenant-root for the duration of its own step and undoes +# it on exit: see its header for the two shared objects it touches and the +# cert-manager behaviour that relies on. Runs in tenant-root, where the system +# bucket's SeaweedFS and its Ingress controller sit. +# +# The precondition step deletes tenant-root's projected cozy-backups-creds, +# which the mongodb and rabbitmq round-trips leave there after their +# cozy-default BackupJobs. Nothing reads it between suites, and the projector +# writes it again on the next BackupJob, which is the moment under test. +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: clickhouse-3-system-bucket-roundtrip +spec: + namespace: tenant-root + catch: + - events: {} + - describe: + apiVersion: helm.toolkit.fluxcd.io/v2 + kind: HelmRelease + name: clickhouse-sysbucket + - describe: + apiVersion: cert-manager.io/v1 + kind: Certificate + name: e2e-system-bucket-s3 + - describe: + apiVersion: cert-manager.io/v1 + kind: Certificate + name: seaweedfs-s3-ingress-tls + - describe: + apiVersion: networking.k8s.io/v1 + kind: Ingress + name: ingress-seaweedfs-s3 + - describe: + apiVersion: backups.cozystack.io/v1alpha1 + kind: BackupJob + name: sysbucket-backup + - describe: + apiVersion: backups.cozystack.io/v1alpha1 + kind: RestoreJob + name: sysbucket-restore + - podLogs: + selector: app.kubernetes.io/instance=clickhouse-sysbucket + container: clickhouse-backup + tail: 200 + - podLogs: + selector: app=clickhouse-sysbucket-keeper + tail: 100 + steps: + - name: preconditions + try: + - script: + timeout: 2m + content: | + set -eu + if kubectl -n "$NAMESPACE" get secret seaweedfs-s3-ingress-tls -o name >/dev/null 2>&1; then + echo "Secret $NAMESPACE/seaweedfs-s3-ingress-tls already exists; this Test would have to replace it" >&2 + exit 1 + fi + if kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -qF 'BEGIN e2e-clickhouse-system-bucket'; then + echo "kube-system/coredns still carries the e2e-clickhouse-system-bucket block from an earlier run" >&2 + exit 1 + fi + # Fixed names in a persistent namespace: a leftover Succeeded job + # would satisfy the waits in the run step without a new attempt. + kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s + kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s + kubectl -n "$NAMESPACE" delete secret cozy-backups-creds --ignore-not-found --wait=true --timeout=30s + check: + ($error == null): true + + - name: create-certificate-and-probe + try: + - apply: + file: system-bucket-tls.yaml + - assert: + timeout: 2m + resource: + apiVersion: cert-manager.io/v1 + kind: Certificate + metadata: + name: e2e-system-bucket-s3 + status: + (conditions[?type == 'Ready']): + - status: "True" + - assert: + timeout: 3m + resource: + apiVersion: v1 + kind: Pod + metadata: + name: e2e-system-bucket-probe + status: + (conditions[?type == 'Ready']): + - status: "True" + + - name: create-clickhouse + try: + - apply: + file: clickhouse-system-bucket.yaml + # 11m: above the 600s install timeout cozystack-api gives this release; + # see "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md. + - assert: + timeout: 11m + resource: + apiVersion: helm.toolkit.fluxcd.io/v2 + kind: HelmRelease + metadata: + name: clickhouse-sysbucket + status: + (conditions[?type == 'Ready']): + - status: "True" + - assert: + timeout: 5m + resource: + apiVersion: apps/v1 + kind: StatefulSet + metadata: + name: chi-clickhouse-sysbucket-clickhouse-0-0 + status: + readyReplicas: 1 + # The Pod is running with no projected credentials: the state the first + # BackupJob of a fresh release meets. + - error: + timeout: 30s + resource: + apiVersion: v1 + kind: Secret + metadata: + name: cozy-backups-creds + + - name: run + try: + # Inner budgets of system-bucket-roundtrip.sh: + # 240s probe: CoreDNS picking up the rewrite and the Ingress serving + # the Test certificate + # 480s BackupJob to Succeeded, including the driver's 2m hold + # 480s RestoreJob to Succeeded, same hold + # 90s the EXIT trap: Corefile replace plus a 60s Secret delete + # 1290s (21m30s) + # 25m leaves 3m30s for the SQL and kubectl calls between those waits, so + # an inner wait is what fires and names what stalled. + - script: + timeout: 25m + content: | + set -eu + cd ../../.. + NAMESPACE="$NAMESPACE" hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh + check: + ($error == null): true + finally: + # Chainsaw did not apply these: the jobs come from the script and the + # Backup from the controller, and deleting a BackupJob reclaims no Backup. + # The Certificate's Secret outlives the Certificate Chainsaw deletes. + - description: system-bucket round-trip cleanup + script: + timeout: 3m + content: | + set -eu + backup=$(kubectl -n "$NAMESPACE" get backupjob sysbucket-backup --ignore-not-found -o jsonpath='{.status.backupRef.name}') + kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s + kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s + if [ -n "$backup" ]; then + kubectl -n "$NAMESPACE" delete backup "$backup" --ignore-not-found --wait=true --timeout=60s + fi + kubectl -n "$NAMESPACE" delete secret e2e-system-bucket-s3-tls --ignore-not-found --wait=true --timeout=30s diff --git a/hack/e2e-chainsaw/clickhouse/clickhouse-system-bucket.yaml b/hack/e2e-chainsaw/clickhouse/clickhouse-system-bucket.yaml new file mode 100644 index 000000000..f7327b46d --- /dev/null +++ b/hack/e2e-chainsaw/clickhouse/clickhouse-system-bucket.yaml @@ -0,0 +1,28 @@ +# ClickHouse on the platform system bucket for the +# clickhouse-3-system-bucket-roundtrip Test. No S3 coordinates are given: the +# sidecar reads them from the clickhouse-backup.yml key the platform projects +# into cozy-backups-creds at the first BackupJob, after this Pod has started. +# backup.endpointCA trusts the SeaweedFS CA that signs the certificate the Test +# serves for s3.example.org. +apiVersion: apps.cozystack.io/v1alpha1 +kind: ClickHouse +metadata: + name: sysbucket +spec: + size: 5Gi + logStorageSize: 1Gi + shards: 1 + replicas: 1 + resourcesPreset: nano + resources: {} + backup: + enabled: true + useSystemBucket: true + endpointCA: + name: e2e-system-bucket-s3-tls + key: ca.crt + clickhouseKeeper: + enabled: true + replicas: 1 + size: 1Gi + resourcesPreset: micro diff --git a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh new file mode 100755 index 000000000..4ca52024b --- /dev/null +++ b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh @@ -0,0 +1,185 @@ +#!/bin/bash +# Backup and in-place restore of the `sysbucket` ClickHouse through the +# cozy-default BackupClass, against the platform system bucket. Run from the +# repository root by the clickhouse-3-system-bucket-roundtrip Test, which has +# already started the release with no cozy-backups-creds in its namespace. +# +# The system bucket is addressed as https://s3.example.org, the S3 Ingress host +# the COSI driver advertises. In CI that name does not resolve and its ACME +# certificate is never issued, so for the duration of this script, and only of +# this script, two shared objects are changed and the EXIT trap below puts both +# back: +# +# - the S3 Ingress TLS Secret seaweedfs-s3-ingress-tls is created from the +# Test's own certificate (SeaweedFS CA, s3.example.org). It relies on +# cert-manager leaving a Secret's data alone while the Certificate that +# names it has no completed issuance, verified with cert-manager v1.20.2: +# a fresh ACME order would replace this certificate, and ingress-nginx +# would then serve one the sidecar does not trust, which fails the probe +# below rather than the backup. +# - the host CoreDNS Corefile gets one marked `rewrite name exact` line for +# s3.example.org. The host CoreDNS is Talos-managed and imports no snippet +# files, so the block goes into the live Corefile the way ouroboros writes +# its own; ouroboros is off in CI and no other suite uses this name. +set -euo pipefail + +source examples/backups/clickhouse/00-helpers.sh + +: "${NAMESPACE:?}" +CLICKHOUSE_NAME=sysbucket +BACKUPJOB_NAME=sysbucket-backup +RESTOREJOB_NAME=sysbucket-restore +S3_HOST=s3.example.org +INGRESS_SECRET=seaweedfs-s3-ingress-tls +CERT_SECRET=e2e-system-bucket-s3-tls +PROBE_POD=e2e-system-bucket-probe +OWNER_LABEL=cozystack.io/e2e-owner=clickhouse-system-bucket +DNS_BEGIN=' # === BEGIN e2e-clickhouse-system-bucket ===' +DNS_END=' # === END e2e-clickhouse-system-bucket ===' +DNS_TARGET="root-ingress-controller.${NAMESPACE}.svc.cozy.local" +# Kubelet refreshes a ConfigMap volume within about a minute and CoreDNS's +# reload plugin polls every 30s; the rest is the 30s answer cache. +PROBE_TIMEOUT=240 +# Each covers the driver's 2m hold after the credentials change plus the +# strategy Job itself. +JOB_TIMEOUT=480 + +workdir=$(mktemp -d) + +corefile_has_block() { + kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -qF -- "$DNS_BEGIN" +} + +remove_dns_block() { + kubectl -n kube-system get configmap coredns -o json >"$workdir/coredns.json" + local begins + begins=$(jq -r '.data.Corefile' "$workdir/coredns.json" | grep -cF -- "$DNS_BEGIN" || true) + if [ "$begins" = 0 ]; then + return 0 + fi + if [ "$begins" != 1 ] || ! jq -r '.data.Corefile' "$workdir/coredns.json" | grep -qF -- "$DNS_END"; then + log_error "kube-system/coredns holds ${begins} '${DNS_BEGIN# *}' markers or no END marker; reconcile the Corefile by hand" + return 1 + fi + jq --arg b "$DNS_BEGIN" --arg e "$DNS_END" \ + '.data.Corefile |= (split("\n") | (index($b)) as $i | (index($e)) as $j | .[:$i] + .[$j+1:] | join("\n"))' \ + "$workdir/coredns.json" >"$workdir/coredns-clean.json" + kubectl replace -f "$workdir/coredns-clean.json" >/dev/null + log_success "Removed the ${S3_HOST} rewrite from kube-system/coredns" +} + +restore_shared_state() { + local rc=$? + set +e + remove_dns_block || rc=1 + if kubectl -n "$NAMESPACE" get secret "$INGRESS_SECRET" -l "$OWNER_LABEL" -o name 2>/dev/null | grep -q .; then + kubectl -n "$NAMESPACE" delete secret "$INGRESS_SECRET" --wait=true --timeout=60s || rc=1 + fi + rm -rf "$workdir" + exit "$rc" +} + +if kubectl -n "$NAMESPACE" get secret "$INGRESS_SECRET" -o name >/dev/null 2>&1; then + log_error "Secret ${NAMESPACE}/${INGRESS_SECRET} already exists; this Test serves its own certificate under that name and will not overwrite one it did not create" + exit 1 +fi +if corefile_has_block; then + log_error "kube-system/coredns already carries the ${S3_HOST} rewrite block, left by an earlier run; remove it by hand" + exit 1 +fi +trap restore_shared_state EXIT + +log_step "Serving the Test certificate for ${S3_HOST} through the platform S3 Ingress" +kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o json \ + | jq --arg name "$INGRESS_SECRET" '{apiVersion, kind, type, + metadata: {name: $name, labels: {"cozystack.io/e2e-owner": "clickhouse-system-bucket"}}, + data: {"tls.crt": .data["tls.crt"], "tls.key": .data["tls.key"], "ca.crt": .data["ca.crt"]}}' \ + | kubectl -n "$NAMESPACE" create -f - + +log_step "Rewriting ${S3_HOST} to ${DNS_TARGET} in the host CoreDNS" +kubectl -n kube-system get configmap coredns -o json >"$workdir/coredns.json" +jq -e '.data.Corefile | startswith(".:53 {\n")' "$workdir/coredns.json" >/dev/null \ + || { log_error "kube-system/coredns Corefile does not open with '.:53 {'; refusing to edit it"; exit 1; } +# The resourceVersion read above travels with the replace, so a concurrent +# writer makes it fail instead of being overwritten. +jq --arg block "${DNS_BEGIN}"$'\n'" rewrite name exact ${S3_HOST} ${DNS_TARGET}"$'\n'"${DNS_END}" \ + '.data.Corefile |= sub("^\\.:53 \\{\n"; ".:53 {\n" + $block + "\n")' \ + "$workdir/coredns.json" >"$workdir/coredns-new.json" +kubectl replace -f "$workdir/coredns-new.json" >/dev/null + +log_step "Checking that ${S3_HOST} serves the Test certificate from inside ${NAMESPACE}" +kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o jsonpath='{.data.ca\.crt}' | base64 -d >"$workdir/ca.crt" +want=$(kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o jsonpath='{.data.tls\.crt}' | base64 -d \ + | openssl x509 -noout -fingerprint -sha256) +elapsed=0 +while true; do + # --cacert makes curl verify both the chain and the host name; %{certs} + # prints the chain it was served, leaf first. + if served=$(kubectl -n "$NAMESPACE" exec -i "$PROBE_POD" -- sh -c \ + "cat >/tmp/ca.crt && curl -sS -o /dev/null --max-time 10 --cacert /tmp/ca.crt -w '%{certs}' https://${S3_HOST}/" \ + <"$workdir/ca.crt" 2>"$workdir/probe.err"); then + got=$(printf '%s\n' "$served" | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p;/END CERTIFICATE/q' \ + | openssl x509 -noout -fingerprint -sha256) + if [ "$got" = "$want" ]; then + log_success "${S3_HOST} resolves in ${NAMESPACE} and serves the Test certificate" + break + fi + echo "served ${got}, want ${want}" >"$workdir/probe.err" + fi + if [ "$elapsed" -ge "$PROBE_TIMEOUT" ]; then + log_error "${S3_HOST} did not serve the Test certificate within ${PROBE_TIMEOUT}s: $(cat "$workdir/probe.err")" + kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' >&2 || true + kubectl -n "$NAMESPACE" describe ingress ingress-seaweedfs-s3 >&2 || true + exit 1 + fi + sleep 10 + elapsed=$((elapsed + 10)) +done + +log_step "Writing the sentinel row" +clickhouse_query "$CLICKHOUSE_NAME" "CREATE TABLE IF NOT EXISTS default.sentinel (id UInt32, name String) ENGINE = MergeTree ORDER BY id" +clickhouse_query "$CLICKHOUSE_NAME" "INSERT INTO default.sentinel VALUES (1, 'before-backup')" + +log_step "Backing up through cozy-default" +kubectl apply -f - <= 1 )) || { log_error "sentinel row count after restore is ${count}; expected >= 1"; exit 1; } +log_success "System-bucket round-trip verified: ${count} sentinel row(s) restored" diff --git a/hack/e2e-chainsaw/clickhouse/system-bucket-tls.yaml b/hack/e2e-chainsaw/clickhouse/system-bucket-tls.yaml new file mode 100644 index 000000000..06203a299 --- /dev/null +++ b/hack/e2e-chainsaw/clickhouse/system-bucket-tls.yaml @@ -0,0 +1,42 @@ +# Certificate and probe Pod for the clickhouse-3-system-bucket-roundtrip Test. +# +# The certificate is signed by the SeaweedFS CA already present in tenant-root, +# so its Secret carries that CA as ca.crt: the ClickHouse release trusts it via +# backup.endpointCA, and the round-trip script copies the same key pair under +# the name the platform's S3 Ingress serves for s3.example.org. +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: e2e-system-bucket-s3 +spec: + secretName: e2e-system-bucket-s3-tls + dnsNames: + - s3.example.org + issuerRef: + kind: Issuer + name: seaweedfs-ca-issuer +--- +# Resolves and fetches s3.example.org from inside tenant-root, where the +# ClickHouse sidecar runs, so the script can prove the DNS rewrite and the +# served certificate before it spends minutes on a BackupJob. The image is the +# one hack/e2e-install-cozystack.bats pre-pulls; it carries curl but no openssl. +apiVersion: v1 +kind: Pod +metadata: + name: e2e-system-bucket-probe +spec: + restartPolicy: Never + terminationGracePeriodSeconds: 0 + securityContext: + runAsNonRoot: true + runAsUser: 65534 + seccompProfile: + type: RuntimeDefault + containers: + - name: probe + image: docker.io/alpine/k8s:1.36.2@sha256:44ef4942e171939b9c665a4a84beb80e2dcdb9a24330d4651cfdfd2e9deecc47 + command: [sleep, "3600"] + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: [ALL] From 2e37aaec5f46a88c01f65764788ab8e58ab7ac7d Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:31:59 +0300 Subject: [PATCH 115/518] docs(backups): drop the claim that a region change needs a ClickHouse restart The ClickHouse sidecar on the system bucket no longer reads S3 settings from environment set at start, so a re-projected region reaches it without a Pod rollout, and the contracts fixture no longer depends on optional Secret keys to start its Pod. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/e2e-chainsaw/clickhouse/clickhouse-backup.yaml | 4 ++-- packages/system/backupstrategy-controller/values.yaml | 4 ---- 2 files changed, 2 insertions(+), 6 deletions(-) diff --git a/hack/e2e-chainsaw/clickhouse/clickhouse-backup.yaml b/hack/e2e-chainsaw/clickhouse/clickhouse-backup.yaml index b88e7dd6d..fb0979146 100644 --- a/hack/e2e-chainsaw/clickhouse/clickhouse-backup.yaml +++ b/hack/e2e-chainsaw/clickhouse/clickhouse-backup.yaml @@ -8,8 +8,8 @@ # backup.enabled=true with placeholder S3 coordinates is deliberate: the # sidecar runs `clickhouse-backup server`, which starts its HTTP API without # validating S3 (ALLOW_EMPTY_BACKUPS=true, lazy connect — the same reason the -# platform useSystemBucket flow marks the S3 secret keys optional so the Pod -# starts before any bucket is projected). So the Pod becomes Ready and the +# platform useSystemBucket flow can start the Pod before any bucket is +# projected). So the Pod becomes Ready and the # chart<->strategy contract is assertable with no reachable object store. The # real S3 backup/restore round-trip lives in clickhouse-2-backup-roundtrip, # which runs in tenant-root against the in-cluster SeaweedFS. diff --git a/packages/system/backupstrategy-controller/values.yaml b/packages/system/backupstrategy-controller/values.yaml index 6cf39c65a..2a4152597 100644 --- a/packages/system/backupstrategy-controller/values.yaml +++ b/packages/system/backupstrategy-controller/values.yaml @@ -105,10 +105,6 @@ backupStorage: endpoint: http://seaweedfs-s3.tenant-root.svc.cozy.local:8333 # region is re-projected into cozy-backups-creds on the next reconcile # (1-minute tick for system namespaces, on-demand for tenant namespaces). - # However, charts that consume the region via secretKeyRef in env vars — - # the chart-emitted ClickHouse sidecar on `useSystemBucket: true` is the - # only such consumer today — only pick up the new value on Pod restart. - # Plan a rollout of those Pods when you change this value. region: us-east-1 forcePathStyle: true # systemSecretName — the human-friendly Secret produced by the Bucket From 38b00730ff072b6b03360e8ae12b047d7568c4ac Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:40:01 +0300 Subject: [PATCH 116/518] fix(clickhouse): keep the clickhouse-backup default region when none is configured The environment the sidecar used to read had no S3_REGION when the platform carried no region, which left clickhouse-backup on its own us-east-1. Writing an empty region into the config file replaces that default with an empty one, and requests are then signed for no region. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backupcontroller/credentials_projector.go | 6 ++++- .../credentials_projector_test.go | 25 +++++++++++++++++++ 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/internal/backupcontroller/credentials_projector.go b/internal/backupcontroller/credentials_projector.go index 99d4e182e..ddf5ff8ef 100644 --- a/internal/backupcontroller/credentials_projector.go +++ b/internal/backupcontroller/credentials_projector.go @@ -454,10 +454,14 @@ func buildClickHouseBackupConfig(creds parsedCreds, forcePathStyle string) ([]by s3 := map[string]any{ "bucket": creds.bucket, "endpoint": "https://" + creds.endpoint, - "region": creds.region, "access_key": creds.accessKey, "secret_key": creds.secretKey, } + // An empty region in the file would override clickhouse-backup's own + // us-east-1 default instead of leaving it in place. + if creds.region != "" { + s3["region"] = creds.region + } if forcePathStyle != "" { pathStyle, err := strconv.ParseBool(forcePathStyle) if err != nil { diff --git a/internal/backupcontroller/credentials_projector_test.go b/internal/backupcontroller/credentials_projector_test.go index 208ba253c..78ffb6c94 100644 --- a/internal/backupcontroller/credentials_projector_test.go +++ b/internal/backupcontroller/credentials_projector_test.go @@ -721,6 +721,31 @@ func TestProject_WritesClickHouseBackupConfig(t *testing.T) { } } +// clickhouse-backup defaults the S3 region to us-east-1; an empty value in +// its config file would replace that default rather than fall back to it. +func TestProject_ClickHouseBackupConfigOmitsAnEmptyRegion(t *testing.T) { + c := newFakeClient(flatSourceSecret()) + ctx := context.Background() + cfg := defaultCfg() + cfg.Region = "" + if err := ProjectBackupCredentials(ctx, c, cfg, "tenant-acme"); err != nil { + t.Fatalf("unexpected error: %v", err) + } + got := &corev1.Secret{} + if err := c.Get(ctx, types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}, got); err != nil { + t.Fatalf("projected Secret not found: %v", err) + } + var doc struct { + S3 map[string]any `json:"s3"` + } + if err := yaml.Unmarshal(got.Data[clickHouseBackupConfigKey], &doc); err != nil { + t.Fatalf("%s is not YAML: %v", clickHouseBackupConfigKey, err) + } + if region, set := doc.S3["region"]; set { + t.Errorf("s3.region = %q, want it left to the clickhouse-backup default", region) + } +} + func TestProject_RejectsMalformedForcePathStyle(t *testing.T) { c := newFakeClient(flatSourceSecret()) cfg := defaultCfg() From 5f636ce62469aa313a3f1a67cba4e72dbf8f4d87 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:43:14 +0300 Subject: [PATCH 117/518] fix(clickhouse): do not hold strategy jobs on a credentials stamp that cannot expire The projector rewrites the credentials-updated-at stamp only when the projected content changes, so a stamp the hold cannot use stays put. An unreadable one failed every reconcile with no condition on the job, and one far in the future held each job again on every reconcile; in both cases every ClickHouse backup and restore in the namespace stopped for good. Such a stamp is now logged and ignored. A stamp only slightly ahead, as a skewed clock writes it, still waits, and that wait ends. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../altinitystrategy_controller.go | 10 ++++- .../altinitystrategy_controller_test.go | 41 +++++++++++++++++++ 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/internal/backupcontroller/altinitystrategy_controller.go b/internal/backupcontroller/altinitystrategy_controller.go index a99801ac6..c3c609350 100644 --- a/internal/backupcontroller/altinitystrategy_controller.go +++ b/internal/backupcontroller/altinitystrategy_controller.go @@ -102,9 +102,17 @@ func credentialsPropagationWait(ctx context.Context, c client.Client, secretName if raw == "" { return 0, nil } + // The projector rewrites the stamp only when the content changes, so a + // stamp this hold cannot use stays as it is: holding on it would block + // every Job in the namespace for good. updatedAt, err := time.Parse(time.RFC3339, raw) if err != nil { - return 0, fmt.Errorf("parse %s on Secret %s/%s: %w", credentialsUpdatedAtAnnotation, namespace, secretName, err) + getLogger(ctx).Error(err, "ignoring unreadable credentials stamp", "secret", namespace+"/"+secretName, "annotation", credentialsUpdatedAtAnnotation) + return 0, nil + } + if updatedAt.After(time.Now().Add(altinityCredentialsPropagation)) { + getLogger(ctx).Error(nil, "ignoring a credentials stamp further ahead than clock skew explains", "secret", namespace+"/"+secretName, "annotation", credentialsUpdatedAtAnnotation, "value", raw) + return 0, nil } return time.Until(updatedAt.Add(altinityCredentialsPropagation)), nil } diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index 75564538b..717b425d1 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -896,3 +896,44 @@ func TestReconcileAltinity_NoProjectionConfiguredDoesNotWait(t *testing.T) { t.Fatalf("got %d strategy Jobs, want 1", len(jobs.Items)) } } + +func projectedCredentialsStamped(namespace, stamp string) *corev1.Secret { + return &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: namespace, + Name: "cozy-backups-creds", + Annotations: map[string]string{credentialsUpdatedAtAnnotation: stamp}, + }, + } +} + +// The stamp is only ever rewritten when the projected content changes, so a +// value the hold cannot use must not hold the job: an unreadable one would +// fail every reconcile, and one far in the future would wait again on each. +// A stamp a little ahead, as a skewed clock writes it, still waits, but only +// until that stamp plus the propagation time. +func TestCredentialsPropagationWait_UnusableStampsDoNotHoldTheJob(t *testing.T) { + cases := []struct { + name string + stamp string + wantMin time.Duration + wantMax time.Duration + }{ + {name: "unreadable", stamp: "garbage"}, + {name: "far in the future", stamp: "3000-01-01T00:00:00Z"}, + {name: "slightly ahead", stamp: time.Now().Add(30 * time.Second).UTC().Format(time.RFC3339), + wantMin: altinityCredentialsPropagation, wantMax: altinityCredentialsPropagation + 31*time.Second}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + c := clientfake.NewClientBuilder().WithObjects(projectedCredentialsStamped("tenant-test", tc.stamp)).Build() + wait, err := credentialsPropagationWait(context.Background(), c, "cozy-backups-creds", "tenant-test", "job") + if err != nil { + t.Fatalf("credentialsPropagationWait: %v", err) + } + if wait < tc.wantMin || wait > tc.wantMax { + t.Errorf("wait = %v, want within [%v, %v]", wait, tc.wantMin, tc.wantMax) + } + }) + } +} From 6345b939d2eef42156d2bbcafd59e859be307326 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:43:15 +0300 Subject: [PATCH 118/518] fix(clickhouse): stop reporting a credentials wait once the strategy job runs The hold sets Ready=False/WaitingForCredentials with a countdown, and nothing removed it once the Job was created, so a backup or restore that ran for minutes kept saying it was still waiting. The condition is dropped when the Job runs; a Ready condition set for another reason is left alone. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../altinitystrategy_controller.go | 24 +++- .../altinitystrategy_controller_test.go | 104 ++++++++++++++++++ 2 files changed, 127 insertions(+), 1 deletion(-) diff --git a/internal/backupcontroller/altinitystrategy_controller.go b/internal/backupcontroller/altinitystrategy_controller.go index c3c609350..c72e30719 100644 --- a/internal/backupcontroller/altinitystrategy_controller.go +++ b/internal/backupcontroller/altinitystrategy_controller.go @@ -117,13 +117,25 @@ func credentialsPropagationWait(ctx context.Context, c client.Client, secretName return time.Until(updatedAt.Add(altinityCredentialsPropagation)), nil } +// clearCredentialsWait drops the Ready condition credentialsWaitCondition +// left behind, once the job no longer waits. It reports whether it did. +func clearCredentialsWait(conditions *[]metav1.Condition) bool { + ready := apimeta.FindStatusCondition(*conditions, "Ready") + if ready == nil || ready.Reason != credentialsWaitReason { + return false + } + return apimeta.RemoveStatusCondition(conditions, "Ready") +} + +const credentialsWaitReason = "WaitingForCredentials" + // credentialsWaitCondition is the Ready condition of a job held back by // credentialsPropagationWait. func credentialsWaitCondition(secretName string, wait time.Duration) metav1.Condition { return metav1.Condition{ Type: "Ready", Status: metav1.ConditionFalse, - Reason: "WaitingForCredentials", + Reason: credentialsWaitReason, Message: fmt.Sprintf("waiting %s for the updated Secret %s to reach the ClickHouse backup sidecar", wait.Round(time.Second), secretName), } @@ -336,6 +348,11 @@ func (r *BackupJobReconciler) reconcileAltinity(ctx context.Context, j *backupsv return r.markBackupJobFailed(ctx, j, message) default: + if clearCredentialsWait(&j.Status.Conditions) { + if err := r.Status().Update(ctx, j); err != nil { + return ctrl.Result{}, err + } + } return ctrl.Result{RequeueAfter: altinityPollInterval}, nil } } @@ -741,6 +758,11 @@ func (r *RestoreJobReconciler) reconcileAltinityRestore(ctx context.Context, res return r.markRestoreJobFailed(ctx, restoreJob, message) default: + if clearCredentialsWait(&restoreJob.Status.Conditions) { + if err := r.Status().Update(ctx, restoreJob); err != nil { + return ctrl.Result{}, err + } + } return ctrl.Result{RequeueAfter: altinityPollInterval}, nil } } diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index 717b425d1..e521d2d46 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -907,6 +907,100 @@ func projectedCredentialsStamped(namespace, stamp string) *corev1.Secret { } } +func waitingForCredentials() []metav1.Condition { + return []metav1.Condition{credentialsWaitCondition("cozy-backups-creds", time.Minute)} +} + +// Once the hold is over and the Job runs, a Ready condition still saying the +// job is waiting for credentials describes a state the job has left. +func TestReconcileAltinity_ClearsTheCredentialsWaitOnceTheJobRuns(t *testing.T) { + now := metav1.Now() + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + BackupClassName: "clickhouse-backup", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now, Conditions: waitingForCredentials()}, + } + r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), + clientfake.NewClientBuilder().WithObjects(backupJob, newAltinityStrategy("clickhouse-strategy"), + projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))). + WithStatusSubresource(&backupsv1alpha1.BackupJob{})) + r.CredentialsConfig.TargetSecretName = "cozy-backups-creds" + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + } + ctx := context.Background() + j := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := r.reconcileAltinity(ctx, j, resolved); err != nil { + t.Fatalf("reconcileAltinity: %v", err) + } + jobs := &batchv1.JobList{} + if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil { + t.Fatalf("list Jobs: %v", err) + } + if len(jobs.Items) != 1 { + t.Fatalf("got %d strategy Jobs, want 1", len(jobs.Items)) + } + stored := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), stored); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if ready := meta.FindStatusCondition(stored.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForCredentials" { + t.Errorf("stored Ready = %+v while the Job runs", ready) + } +} + +func TestReconcileAltinityRestore_ClearsTheCredentialsWaitOnceTheJobRuns(t *testing.T) { + now := metav1.Now() + backup := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Name: "ch-backup", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + TakenAt: now, + }, + } + restoreJob := &backupsv1alpha1.RestoreJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-rj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}}, + Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning, + Conditions: waitingForCredentials()}, + } + _, rr, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), + clientfake.NewClientBuilder().WithObjects(newAltinityStrategy("clickhouse-strategy"), backup, restoreJob, + projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))). + WithStatusSubresource(&backupsv1alpha1.RestoreJob{})) + rr.CredentialsConfig.TargetSecretName = "cozy-backups-creds" + ctx := context.Background() + rj := &backupsv1alpha1.RestoreJob{} + if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), rj); err != nil { + t.Fatalf("get RestoreJob: %v", err) + } + if _, err := rr.reconcileAltinityRestore(ctx, rj, backup); err != nil { + t.Fatalf("reconcileAltinityRestore: %v", err) + } + stored := &backupsv1alpha1.RestoreJob{} + if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), stored); err != nil { + t.Fatalf("get RestoreJob: %v", err) + } + if ready := meta.FindStatusCondition(stored.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForCredentials" { + t.Errorf("stored Ready = %+v while the Job runs", ready) + } +} + // The stamp is only ever rewritten when the projected content changes, so a // value the hold cannot use must not hold the job: an unreadable one would // fail every reconcile, and one far in the future would wait again on each. @@ -937,3 +1031,13 @@ func TestCredentialsPropagationWait_UnusableStampsDoNotHoldTheJob(t *testing.T) }) } } + +func TestClearCredentialsWait_KeepsAReadyConditionItDidNotSet(t *testing.T) { + conditions := []metav1.Condition{{Type: "Ready", Status: metav1.ConditionFalse, Reason: "StrategyNotReady"}} + if clearCredentialsWait(&conditions) { + t.Errorf("clearCredentialsWait reported clearing a Ready condition with reason StrategyNotReady") + } + if ready := meta.FindStatusCondition(conditions, "Ready"); ready == nil || ready.Reason != "StrategyNotReady" { + t.Errorf("Ready = %+v, want the StrategyNotReady condition kept", ready) + } +} From dbcc2af6791c320a60b80662e78074bcd23b87cf Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:43:55 +0300 Subject: [PATCH 119/518] test(clickhouse): require the system-bucket round-trip to run on the sidecar that started without credentials A Pod restart during the round-trip would hand the sidecar fresh credentials at start, which is how the old environment-based sidecar got them, and the Test would pass without the running sidecar ever picking them up. The Pod uid and the sidecar's restart count must stay the same across both jobs. The probe deadline is now measured on the clock: counting only its pauses let a hanging exec stretch it past the step budget, so the step timeout would fire before the probe could say what stalled. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../clickhouse/chainsaw-test.yaml | 9 ++++---- .../clickhouse/system-bucket-roundtrip.sh | 21 +++++++++++++++---- 2 files changed, 22 insertions(+), 8 deletions(-) diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index f5b444a84..d96013140 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -521,13 +521,14 @@ spec: - name: run try: # Inner budgets of system-bucket-roundtrip.sh: - # 240s probe: CoreDNS picking up the rewrite and the Ingress serving - # the Test certificate + # 280s probe: a 240s deadline for CoreDNS picking up the rewrite and + # the Ingress serving the Test certificate, plus one last 30s + # attempt and its 10s pause # 480s BackupJob to Succeeded, including the driver's 2m hold # 480s RestoreJob to Succeeded, same hold # 90s the EXIT trap: Corefile replace plus a 60s Secret delete - # 1290s (21m30s) - # 25m leaves 3m30s for the SQL and kubectl calls between those waits, so + # 1330s (22m10s) + # 25m leaves 2m50s for the SQL and kubectl calls between those waits, so # an inner wait is what fires and names what stalled. - script: timeout: 25m diff --git a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh index 4ca52024b..fe1361418 100755 --- a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh +++ b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh @@ -111,11 +111,11 @@ log_step "Checking that ${S3_HOST} serves the Test certificate from inside ${NAM kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o jsonpath='{.data.ca\.crt}' | base64 -d >"$workdir/ca.crt" want=$(kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o jsonpath='{.data.tls\.crt}' | base64 -d \ | openssl x509 -noout -fingerprint -sha256) -elapsed=0 +deadline=$((SECONDS + PROBE_TIMEOUT)) while true; do # --cacert makes curl verify both the chain and the host name; %{certs} # prints the chain it was served, leaf first. - if served=$(kubectl -n "$NAMESPACE" exec -i "$PROBE_POD" -- sh -c \ + if served=$(timeout 30 kubectl -n "$NAMESPACE" exec -i "$PROBE_POD" -- sh -c \ "cat >/tmp/ca.crt && curl -sS -o /dev/null --max-time 10 --cacert /tmp/ca.crt -w '%{certs}' https://${S3_HOST}/" \ <"$workdir/ca.crt" 2>"$workdir/probe.err"); then got=$(printf '%s\n' "$served" | sed -n '/BEGIN CERTIFICATE/,/END CERTIFICATE/p;/END CERTIFICATE/q' \ @@ -126,16 +126,26 @@ while true; do fi echo "served ${got}, want ${want}" >"$workdir/probe.err" fi - if [ "$elapsed" -ge "$PROBE_TIMEOUT" ]; then + if [ "$SECONDS" -ge "$deadline" ]; then log_error "${S3_HOST} did not serve the Test certificate within ${PROBE_TIMEOUT}s: $(cat "$workdir/probe.err")" kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' >&2 || true kubectl -n "$NAMESPACE" describe ingress ingress-seaweedfs-s3 >&2 || true exit 1 fi sleep 10 - elapsed=$((elapsed + 10)) done +# A restarted Pod reads its credentials afresh at start, which is what the +# old environment-based sidecar needed; the round-trip proves the fix only if +# the sidecar that started without them is the one that serves both jobs. +sidecar_identity() { + kubectl -n "$NAMESPACE" get pod "chi-clickhouse-${CLICKHOUSE_NAME}-clickhouse-0-0-0" \ + -o jsonpath='{.metadata.uid} {.status.containerStatuses[?(@.name=="clickhouse-backup")].restartCount}' +} +sidecar_before=$(sidecar_identity) +[[ "$sidecar_before" =~ ^[0-9a-f-]+\ [0-9]+$ ]] \ + || { log_error "could not read the ClickHouse Pod uid and clickhouse-backup restart count (got '${sidecar_before}')"; exit 1; } + log_step "Writing the sentinel row" clickhouse_query "$CLICKHOUSE_NAME" "CREATE TABLE IF NOT EXISTS default.sentinel (id UInt32, name String) ENGINE = MergeTree ORDER BY id" clickhouse_query "$CLICKHOUSE_NAME" "INSERT INTO default.sentinel VALUES (1, 'before-backup')" @@ -182,4 +192,7 @@ wait_for_field restorejob "$RESTOREJOB_NAME" '{.status.phase}' Succeeded "$NAMES count=$(clickhouse_query "$CLICKHOUSE_NAME" "SELECT count() FROM default.sentinel" | tr -d '[:space:]') [[ "$count" =~ ^[0-9]+$ ]] || { log_error "non-numeric sentinel count: '${count}'"; exit 1; } (( count >= 1 )) || { log_error "sentinel row count after restore is ${count}; expected >= 1"; exit 1; } +sidecar_after=$(sidecar_identity) +[ "$sidecar_after" = "$sidecar_before" ] \ + || { log_error "the ClickHouse Pod or its clickhouse-backup container restarted during the round-trip (uid restarts: '${sidecar_before}' then '${sidecar_after}')"; exit 1; } log_success "System-bucket round-trip verified: ${count} sentinel row(s) restored" From 31b905a1ce938b84db90deaadde1cf91796f4387 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:50:35 +0300 Subject: [PATCH 120/518] docs(clickhouse): note that the backup path stays in the sidecar environment The chart comment said every S3 setting of the system-bucket flow comes from the mounted config file, but S3_PATH is still set in the environment: the platform's file carries no path, and the prefix belongs to the release rather than to the platform. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- packages/apps/clickhouse/templates/clickhouse.yaml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/packages/apps/clickhouse/templates/clickhouse.yaml b/packages/apps/clickhouse/templates/clickhouse.yaml index d8791878e..4934e8db1 100644 --- a/packages/apps/clickhouse/templates/clickhouse.yaml +++ b/packages/apps/clickhouse/templates/clickhouse.yaml @@ -225,15 +225,17 @@ spec: - name: REMOTE_STORAGE value: s3 {{- /* - On the useSystemBucket flow every S3 setting, the - platform's backupStorage.forcePathStyle included, comes + On the useSystemBucket flow every S3 setting but the + release's own S3_PATH below, the platform's + backupStorage.forcePathStyle included, comes from the clickhouse-backup.yml key the platform projects into cozy-backups-creds, mounted below as the config file. clickhouse-backup reloads that file on every API call and kubelet rewrites a mounted Secret in place, so credentials projected after the Pod started, or rotated since, reach the running sidecar; environment would be read only at - start and would override the file, so none is set here. + start and would override the file, so none of those is + set here. S3_FORCE_PATH_STYLE selects path-style vs vhost-style S3 addressing. On the legacy paths (own Secret via From 1e1fdb740c229f7233e3be4993d063c2fd5cc5fb Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:14:24 +0700 Subject: [PATCH 121/518] fix(dashboard): show log rows a pipe stripped of their timestamp A LogsQL pipe such as `fields _msg` or `stats` returns rows without _time, and the console dropped every one of them, so a successful query read as "no lines matched". Such rows are now kept in the order the store sent them, shown without a timestamp, and never deduplicated on a tail resume. A row without _msg shows its remaining fields instead of an empty line. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/LogLines.test.tsx | 12 +++++++ .../src/components/observability/LogLines.tsx | 33 +++++++++++++++---- .../observability/LogsView.test.tsx | 13 ++++++++ .../src/lib/observability/logs.test.ts | 6 ++++ .../console/src/lib/observability/logs.ts | 5 +-- .../src/lib/observability/proxy.test.ts | 20 ++++++++++- .../console/src/lib/observability/proxy.ts | 10 +++--- .../src/lib/observability/use-log-tail.ts | 2 +- 8 files changed, 86 insertions(+), 15 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx index dd8c503eb..a0f519852 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.test.tsx @@ -29,6 +29,18 @@ describe("LogLines", () => { expect(frame).not.toHaveTextContent("Waiting") }) + it("shows a line without _time with no timestamp and no empty source", () => { + render() + + expect(screen.getByRole("listitem")).toHaveTextContent(/^visible message$/) + }) + + it("shows the remaining fields of a row without _msg", () => { + render() + + expect(screen.getByRole("listitem")).toHaveTextContent(/^db-0\/ rows=42$/) + }) + it("keeps following the newest line while the reader is at the bottom", () => { const { rerender } = render() const list = screen.getByRole("list") diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx index 14b941ab2..f29af017c 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogLines.tsx @@ -3,6 +3,22 @@ import { CONTAINER_FIELD, POD_FIELD } from "../../lib/observability/logs.ts" import type { LogEntry } from "../../lib/observability/proxy.ts" const BOTTOM_SLACK_PX = 24 +const SHOWN_APART = new Set(["_time", "_msg", "_stream_id", POD_FIELD, CONTAINER_FIELD]) + +function source(e: LogEntry): string | null { + const pod = e[POD_FIELD] + const container = e[CONTAINER_FIELD] + return pod || container ? `${pod ?? ""}/${container ?? ""}` : null +} + +// A `fields` or `stats` pipe can leave a row without _msg; its other fields are then the content. +function message(e: LogEntry): string { + if (e._msg !== undefined) return e._msg + return Object.entries(e) + .filter(([k, v]) => v !== undefined && !SHOWN_APART.has(k)) + .map(([k, v]) => `${k}=${v}`) + .join(" ") +} export function LogLines({ entries, placeholder }: { entries: LogEntry[]; placeholder: ReactNode }) { const ref = useRef(null) @@ -29,13 +45,16 @@ export function LogLines({ entries, placeholder }: { entries: LogEntry[]; placeh }} className="h-full overflow-auto p-3" > - {entries.map((e, i) => ( -
  1. - {new Date(e._time).toLocaleTimeString()}{" "} - {`${e[POD_FIELD] ?? ""}/${e[CONTAINER_FIELD] ?? ""}`}{" "} - {e._msg} -
  2. - ))} + {entries.map((e, i) => { + const from = source(e) + return ( +
  3. + {e._time !== undefined && <>{new Date(e._time).toLocaleTimeString()}{" "}} + {from !== null && <>{from}{" "}} + {message(e)} +
  4. + ) + })}
)} diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index 1b18dd208..770c8a263 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -199,6 +199,19 @@ describe("LogsView", () => { expect(screen.getAllByRole("option").map((o) => o.textContent)).toEqual(["All", "db-0"]) }) + it("shows rows of a query whose pipe removed _time", async () => { + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("query") ? new Response('{"_msg":"visible message"}\n') : defaultHandler(endpoint, params), + ) + renderWithK8sProvider(, { client }) + + await userEvent.type(screen.getByLabelText("LogsQL filter"), "* | fields _msg{Enter}") + + const frame = screen.getByRole("region", { name: "Log lines" }) + expect(await within(frame).findByText("visible message")).toBeInTheDocument() + expect(within(frame).queryByText(/No log lines/)).toBeNull() + }) + it("names the source pod and container of each line", async () => { const { client } = clientWith(defaultHandler) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts index 25d26a23a..62d1ac249 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts @@ -57,6 +57,12 @@ describe("mergeTail", () => { expect(out).toHaveLength(2) }) + it("never deduplicates lines without _time, even on a resume", () => { + const out = mergeTail([{ _msg: "a" }], [{ _msg: "a" }], 10, "2026-01-01T00:00:02Z") + + expect(out).toHaveLength(2) + }) + it("does not deduplicate outside a resume", () => { const out = mergeTail([e("t1", "a")], [e("t1", "a")], 10) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts index a6255e541..eb24a5f96 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts @@ -61,12 +61,13 @@ export function compareTime(a: string, b: string): number { const entryKey = (e: LogEntry) => `${e._stream_id ?? ""}\u0000${e._time}` // (_stream_id, _time) is not unique in VictoriaLogs, so dedup applies only to -// lines a resumed tail replays at or before the resume point. +// lines a resumed tail replays at or before the resume point, and never to +// lines a pipe stripped of _time. export function mergeTail(buffer: LogEntry[], incoming: LogEntry[], cap: number, resumedFrom?: string): LogEntry[] { let fresh = incoming if (resumedFrom) { const seen = new Set(buffer.map(entryKey)) - fresh = incoming.filter((e) => compareTime(e._time, resumedFrom) > 0 || !seen.has(entryKey(e))) + fresh = incoming.filter((e) => e._time === undefined || compareTime(e._time, resumedFrom) > 0 || !seen.has(entryKey(e))) } const merged = buffer.concat(fresh) return merged.length > cap ? merged.slice(merged.length - cap) : merged diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts index 2809905d8..f2ba8d7c3 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.test.ts @@ -122,6 +122,15 @@ describe("queryLogs", () => { expect(entries.map((e) => e._msg)).toEqual(["a", "b"]) }) + it("keeps rows a pipe stripped of _time, in the order the store sent them", async () => { + const ndjson = ['{"_msg":"visible message"}', '{"_msg":"second"}', "[1]", "null", "{torn"].join("\n") + const { client } = clientReturning(new Response(ndjson)) + + const entries = await queryLogs(client, logsRoot(ns), { query: "* | fields _msg", start: "s", end: "e", limit: 100 }) + + expect(entries).toEqual([{ _msg: "visible message" }, { _msg: "second" }]) + }) + it("keeps a plain-text error body as the message", async () => { const { client } = clientReturning(new Response("cannot parse query", { status: 400 })) @@ -170,7 +179,7 @@ describe("tailLogs", () => { }, }) const { client, raw } = clientReturning(new Response(body)) - const seen: string[] = [] + const seen: (string | undefined)[] = [] await tailLogs(client, logsRoot(ns), { query: "*", since: "t0" }, (e) => seen.push(e._msg)) @@ -179,6 +188,15 @@ describe("tailLogs", () => { expect(seen).toEqual(["a", "b"]) }) + it("delivers a line without _time", async () => { + const { client } = clientReturning(new Response('{"_msg":"untimed"}\n')) + const seen: (string | undefined)[] = [] + + await tailLogs(client, logsRoot(ns), { query: "* | fields _msg" }, (e) => seen.push(e._msg)) + + expect(seen).toEqual(["untimed"]) + }) + it("omits since on the first connect", async () => { const { client, raw } = clientReturning(new Response("")) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts index 3b5567f6b..c28bd12d4 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts @@ -140,9 +140,10 @@ export async function queryRange( return body.data?.result ?? [] } +// A pipe such as `fields _msg` or `stats` can drop _time and _msg. export interface LogEntry { - _time: string - _msg: string + _time?: string + _msg?: string _stream_id?: string [field: string]: string | undefined } @@ -151,7 +152,7 @@ function parseLine(line: string): LogEntry | null { if (!line.trim()) return null try { const value: unknown = JSON.parse(line) - if (value && typeof value === "object" && "_time" in value) return value as LogEntry + if (value && typeof value === "object" && !Array.isArray(value)) return value as LogEntry } catch { // a torn or non-JSON line carries nothing to show } @@ -173,7 +174,8 @@ export async function queryLogs( ): Promise { const res = await proxyGet(client, `${root}select/logsql/query`, { ...params }, signal) const entries = (await res.text()).split("\n").map(parseLine).filter((e) => e !== null) - return entries.sort((a, b) => compareTime(a._time, b._time)) + if (entries.some((e) => e._time === undefined)) return entries + return entries.sort((a, b) => compareTime(a._time ?? "", b._time ?? "")) } export interface FieldValuesParams { diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts index 2e3b4d706..d55bd5708 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/use-log-tail.ts @@ -73,7 +73,7 @@ export function useLogTail( let delay = retryDelayMs try { await tailLogs(client, root, { query, since: lastTime ?? openedAt }, (entry) => { - if (!lastTime || compareTime(entry._time, lastTime) > 0) lastTime = entry._time + if (entry._time && (!lastTime || compareTime(entry._time, lastTime) > 0)) lastTime = entry._time pending.push(entry) timer ??= setTimeout(flush, FLUSH_MS) }, signal, () => { From 8e6048eb4a2556cc9af587b4df3259cface49b37 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:16:04 +0700 Subject: [PATCH 122/518] fix(dashboard): make metric samples readable without a pointer The charts are a canvas and their legend showed dashes until a point was hovered, so keyboard and screen-reader users could not read any sample. The legend now shows each series' latest value when nothing is hovered, and every panel has a Show data toggle that opens a focusable table of all samples, newest first, in the chart's units. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/MetricsView.test.tsx | 27 ++++++++++- .../components/observability/MetricsView.tsx | 36 ++++++++++++-- .../observability/SeriesTable.test.tsx | 37 +++++++++++++++ .../components/observability/SeriesTable.tsx | 47 +++++++++++++++++++ .../observability/TimeSeriesChart.test.tsx | 17 ++++++- .../observability/TimeSeriesChart.tsx | 4 +- 6 files changed, 160 insertions(+), 8 deletions(-) create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.test.tsx create mode 100644 packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx index d3f1fc1dd..8e7829ed5 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx @@ -1,5 +1,5 @@ import { describe, it, expect, vi } from "vitest" -import { screen, waitFor } from "@testing-library/react" +import { screen, waitFor, within } from "@testing-library/react" import userEvent from "@testing-library/user-event" import { K8sClient } from "@cozystack/k8s-client" import { renderWithK8sProvider } from "../../test-utils/render.tsx" @@ -98,6 +98,31 @@ describe("MetricsView", () => { expect(screen.queryByText("Try a longer range.")).toBeNull() }) + it("shows a panel's samples as a table on request", async () => { + const { client } = clientWith(() => matrix("db-0")) + renderWithK8sProvider(, { client }) + await waitFor(() => expect(screen.getAllByTestId("chart")).toHaveLength(3)) + const toggle = screen.getByRole("button", { name: "Show data for CPU (cores)" }) + expect(toggle).toHaveAttribute("aria-expanded", "false") + + await userEvent.click(toggle) + + const table = screen.getByRole("region", { name: "CPU (cores) data" }) + expect(toggle).toHaveAttribute("aria-expanded", "true") + expect(toggle).toHaveAttribute("aria-controls", table.id) + expect(toggle).toHaveTextContent("Hide data") + expect(within(table).getByRole("columnheader", { name: "db-0" })).toBeInTheDocument() + expect(screen.queryByRole("region", { name: "Memory (working set) data" })).toBeNull() + }) + + it("offers no table while a panel has no samples", () => { + const { client } = clientWith(() => new Promise(() => {})) + + renderWithK8sProvider(, { client }) + + expect(screen.getByRole("button", { name: "Show data for Network" })).toBeDisabled() + }) + it("holds each chart's place while it loads", () => { const { client } = clientWith(() => new Promise(() => {})) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx index fac2a6b91..4663b2ae3 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx @@ -1,7 +1,7 @@ -import { useState } from "react" +import { useId, useState } from "react" import { useQueries } from "@tanstack/react-query" import { useK8sClient } from "@cozystack/k8s-client" -import { Section } from "@cozystack/ui" +import { Button, Section } from "@cozystack/ui" import { METRIC_PANELS, RANGES, @@ -14,6 +14,7 @@ import { ProxyNotice } from "./ProxyNotice.tsx" import { TimeSeriesChart } from "./TimeSeriesChart.tsx" import { RangePicker } from "./RangePicker.tsx" import { EmptyState } from "./EmptyState.tsx" +import { SeriesTable } from "./SeriesTable.tsx" const REFRESH_MS = 60_000 // The uPlot canvas plus its one-row legend, so a placeholder takes the chart's place without a jump. @@ -23,6 +24,8 @@ export function MetricsView({ scope }: { scope: ObservabilityScope }) { const client = useK8sClient() const root = metricsRoot(scope) const [range, setRange] = useState("1h") + const [tables, setTables] = useState>(new Set()) + const idBase = useId() const rangeKeys = Object.keys(RANGES) as RangeKey[] const longestRange = range === rangeKeys[rangeKeys.length - 1] @@ -57,8 +60,30 @@ export function MetricsView({ scope }: { scope: ObservabilityScope }) { ) : ( METRIC_PANELS.map((panel, i) => { const { data, error, isLoading } = results[i] + const hasSeries = !error && !isLoading && !!data && data.series.length > 0 + const tableOpen = hasSeries && tables.has(panel.id) + const tableId = `${idBase}-${panel.id}-data` + const toggle = ( + + ) return ( -
+
{error ? ( ) : isLoading || !data ? ( @@ -66,7 +91,10 @@ export function MetricsView({ scope }: { scope: ObservabilityScope }) { ) : data.series.length === 0 ? ( ) : ( - + <> + + {tableOpen && } + )}
) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.test.tsx new file mode 100644 index 000000000..0b3364268 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.test.tsx @@ -0,0 +1,37 @@ +import { describe, it, expect } from "vitest" +import { render, screen, within } from "@testing-library/react" +import { SeriesTable } from "./SeriesTable.tsx" + +const data = { + timestamps: [1_700_000_000, 1_700_000_060], + series: [ + { label: "db-0", values: [1, 2] }, + { label: "db-1", values: [null, 3] }, + ], +} + +describe("SeriesTable", () => { + it("is a named region the keyboard can reach", () => { + render() + + const region = screen.getByRole("region", { name: "Memory data" }) + expect(region).toHaveAttribute("id", "t") + expect(region).toHaveAttribute("tabindex", "0") + }) + + it("has a time column and one column per series", () => { + render() + + expect(screen.getAllByRole("columnheader").map((h) => h.textContent)).toEqual(["Time", "db-0", "db-1"]) + }) + + it("lists every sample newest first in the chart's units", () => { + render( `${v} B`} />) + + const rows = screen.getAllByRole("row").slice(1) + expect(rows).toHaveLength(2) + expect(within(rows[0]).getAllByRole("cell").map((c) => c.textContent)).toEqual(["2 B", "3 B"]) + expect(within(rows[1]).getAllByRole("cell").map((c) => c.textContent)).toEqual(["1 B", "–"]) + expect(within(rows[0]).getByRole("rowheader")).toHaveTextContent(new Date(1_700_000_060_000).toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" })) + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx new file mode 100644 index 000000000..57cf755a0 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx @@ -0,0 +1,47 @@ +import type { ChartData } from "../../lib/observability/metrics.ts" + +interface SeriesTableProps { + id: string + title: string + data: ChartData + format: (value: number) => string +} + +const formatTime = (seconds: number) => + new Date(seconds * 1000).toLocaleString([], { month: "short", day: "numeric", hour: "2-digit", minute: "2-digit" }) + +export function SeriesTable({ id, title, data, format }: SeriesTableProps) { + const newestFirst = data.timestamps.map((t, i) => ({ t, i })).reverse() + return ( +
+ + + + + + {data.series.map((s, j) => ( + + ))} + + + + {newestFirst.map(({ t, i }) => ( + + + {data.series.map((s, j) => { + const v = s.values[i] + return + })} + + ))} + +
{`${title}, newest first`}
Time{s.label}
{formatTime(t)}{v === null ? "–" : format(v)}
+
+ ) +} diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx index 57cf6fb99..3df8125e3 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx @@ -1,12 +1,15 @@ import { describe, it, expect, vi, beforeEach } from "vitest" import { render } from "@testing-library/react" -const created: { opts: { series: { label?: string }[] }; data: unknown[]; destroy: ReturnType }[] = [] +type LegendValue = (u: unknown, v: number | null, sidx: number, idx: number | null) => string +type Opts = { series: { label?: string; value?: LegendValue }[] } + +const created: { opts: Opts; data: unknown[]; destroy: ReturnType }[] = [] vi.mock("uplot", () => ({ default: class { destroy = vi.fn() - constructor(opts: { series: { label?: string }[] }, data: unknown[]) { + constructor(opts: Opts, data: unknown[]) { created.push({ opts, data, destroy: this.destroy }) } setSize() {} @@ -36,6 +39,16 @@ describe("TimeSeriesChart", () => { expect(created[0].opts.series.slice(1).map((s) => s.label)).toEqual(["db-0", "db-1"]) }) + it("shows each series' latest value in the legend while nothing is hovered", () => { + render( `${v} B`} />) + + const legend = created[0].opts.series[1].value! + + expect(legend(null, null, 1, null)).toBe("2 B") + expect(legend(null, 1, 1, 0)).toBe("1 B") + expect(legend(null, null, 1, 2)).toBe("–") + }) + it("destroys the chart on unmount", () => { const { unmount } = render() diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx index 683df2dff..a6251d40f 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx @@ -18,6 +18,7 @@ export function TimeSeriesChart({ data, format }: TimeSeriesChartProps) { const el = ref.current if (!el) return const fmt = (v: number | null) => (v === null ? "–" : format(v)) + const latest = data.series.map((s) => s.values.findLast((v) => v !== null) ?? null) const chart = new uPlot( { width: el.clientWidth || 600, @@ -30,7 +31,8 @@ export function TimeSeriesChart({ data, format }: TimeSeriesChartProps) { label: s.label, stroke: COLORS[i % COLORS.length], width: 1.5, - value: (_: uPlot, v: number | null) => fmt(v), + // Without a hovered point the legend shows the latest sample instead of a dash. + value: (_: uPlot, v: number | null, sidx: number, idx: number | null) => fmt(idx === null ? latest[sidx - 1] : v), })), ], axes: [{}, { size: 84, values: (_: uPlot, splits: number[]) => splits.map(format) }], From 7f7496c8d3f639993ad2a02f5ea2b1c7d9180265 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:16:36 +0700 Subject: [PATCH 123/518] fix(monitoring): remove the identity placeholder from scrape-health series vmagent appends up and scrape_* to each scrape after metric relabeling: in VictoriaMetrics v1.136.0, the version operator v0.68.4 deploys, lib/promscrape/scrapework.go addAutoMetrics passes needRelabel=false to writeRequestCtx.addRows, so addRow attaches the target labels but skips MetricRelabelConfigs. The "-" placeholder that the scrape class sets on targets without lineage was removed only in metricRelabelConfigs, so it stayed on the scrape-health series of every such target, including nodes and kube-state-metrics. The removal now runs in the VMAgent's inlineRelabelConfig, which the operator passes as -remoteWrite.relabelConfig. app/vmagent/remotewrite TryPush applies it to every series before remote write, scraped and generated alike, so targets without lineage keep the label set they had before the class. The exported_* drop stays in the class: only scraped samples can carry it. Assisted-by: LLM Signed-off-by: ZverGuy --- .../monitoring-agents/templates/vmagent.yaml | 18 +++++++++++------- .../vmagent_application_identity_test.yaml | 12 +++++++++++- .../monitoring/templates/vm/vmagent.yaml | 18 +++++++++++------- .../vmagent_application_identity_test.yaml | 12 +++++++++++- 4 files changed, 44 insertions(+), 16 deletions(-) diff --git a/packages/system/monitoring-agents/templates/vmagent.yaml b/packages/system/monitoring-agents/templates/vmagent.yaml index 29349f17e..ad26881b7 100644 --- a/packages/system/monitoring-agents/templates/vmagent.yaml +++ b/packages/system/monitoring-agents/templates/vmagent.yaml @@ -26,7 +26,8 @@ spec: # copy then collides, is renamed to exported_* and dropped. Under # honorLabels true the exporter's copy wins, so such a scrape must drop # these labels itself. A scrape object's own relabel rules run after these - # and must leave the labels alone. The placeholder is removed before storage. + # and must leave the labels alone. The placeholder is removed by the + # remote-write relabeling below. scrapeClasses: - name: application-identity default: true @@ -41,12 +42,15 @@ spec: metricRelabelConfigs: - action: labeldrop regex: (exported_)+apps_cozystack_io_application_(group|kind|name) - {{- range list "group" "kind" "name" }} - - sourceLabels: [apps_cozystack_io_application_{{ . }}] - regex: "-" - targetLabel: apps_cozystack_io_application_{{ . }} - replacement: "" - {{- end }} + # Not in metricRelabelConfigs: vmagent adds up and scrape_* after metric + # relabeling, so the placeholder would stay on those series. + inlineRelabelConfig: + {{- range list "group" "kind" "name" }} + - sourceLabels: [apps_cozystack_io_application_{{ . }}] + regex: "-" + targetLabel: apps_cozystack_io_application_{{ . }} + replacement: "" + {{- end }} additionalScrapeConfigs: name: additional-scrape-configs key: prometheus-additional.yaml diff --git a/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml b/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml index b59f9e3ee..95f330d50 100644 --- a/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml +++ b/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml @@ -46,13 +46,23 @@ tests: regex: (.+) targetLabel: apps_cozystack_io_application_name - - it: an exporter-supplied identity is dropped and the placeholder never reaches storage + - it: an exporter-supplied identity is dropped asserts: - equal: path: spec.scrapeClasses[0].metricRelabelConfigs value: - action: labeldrop regex: (exported_)+apps_cozystack_io_application_(group|kind|name) + + # vmagent adds up and scrape_* after metric relabeling (VictoriaMetrics + # lib/promscrape addAutoMetrics calls addRows with needRelabel false), so a + # placeholder removed there would stay on those series of every target + # without lineage. The remote-write relabeling sees every series. + - it: the placeholder is removed from every series, including up and scrape_* + asserts: + - equal: + path: spec.inlineRelabelConfig + value: - sourceLabels: [apps_cozystack_io_application_group] regex: "-" targetLabel: apps_cozystack_io_application_group diff --git a/packages/system/monitoring/templates/vm/vmagent.yaml b/packages/system/monitoring/templates/vm/vmagent.yaml index fa4c4fc93..2748ac4bf 100644 --- a/packages/system/monitoring/templates/vm/vmagent.yaml +++ b/packages/system/monitoring/templates/vm/vmagent.yaml @@ -30,7 +30,8 @@ spec: # copy then collides, is renamed to exported_* and dropped. Under # honorLabels true the exporter's copy wins, so such a scrape must drop # these labels itself. A scrape object's own relabel rules run after these - # and must leave the labels alone. The placeholder is removed before storage. + # and must leave the labels alone. The placeholder is removed by the + # remote-write relabeling below. scrapeClasses: - name: application-identity default: true @@ -45,12 +46,15 @@ spec: metricRelabelConfigs: - action: labeldrop regex: (exported_)+apps_cozystack_io_application_(group|kind|name) - {{- range list "group" "kind" "name" }} - - sourceLabels: [apps_cozystack_io_application_{{ . }}] - regex: "-" - targetLabel: apps_cozystack_io_application_{{ . }} - replacement: "" - {{- end }} + # Not in metricRelabelConfigs: vmagent adds up and scrape_* after metric + # relabeling, so the placeholder would stay on those series. + inlineRelabelConfig: + {{- range list "group" "kind" "name" }} + - sourceLabels: [apps_cozystack_io_application_{{ . }}] + regex: "-" + targetLabel: apps_cozystack_io_application_{{ . }} + replacement: "" + {{- end }} {{- with .Values.vmagent.inlineScrapeConfig }} inlineScrapeConfig: | {{- . | nindent 4 }} diff --git a/packages/system/monitoring/tests/vmagent_application_identity_test.yaml b/packages/system/monitoring/tests/vmagent_application_identity_test.yaml index da14330de..3a77fe6ee 100644 --- a/packages/system/monitoring/tests/vmagent_application_identity_test.yaml +++ b/packages/system/monitoring/tests/vmagent_application_identity_test.yaml @@ -42,13 +42,23 @@ tests: regex: (.+) targetLabel: apps_cozystack_io_application_name - - it: an exporter-supplied identity is dropped and the placeholder never reaches storage + - it: an exporter-supplied identity is dropped asserts: - equal: path: spec.scrapeClasses[0].metricRelabelConfigs value: - action: labeldrop regex: (exported_)+apps_cozystack_io_application_(group|kind|name) + + # vmagent adds up and scrape_* after metric relabeling (VictoriaMetrics + # lib/promscrape addAutoMetrics calls addRows with needRelabel false), so a + # placeholder removed there would stay on those series of every target + # without lineage. The remote-write relabeling sees every series. + - it: the placeholder is removed from every series, including up and scrape_* + asserts: + - equal: + path: spec.inlineRelabelConfig + value: - sourceLabels: [apps_cozystack_io_application_group] regex: "-" targetLabel: apps_cozystack_io_application_group From e65d827cf9eca51f742200e76172b62224003871 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:16:49 +0700 Subject: [PATCH 124/518] fix(dashboard): move the tenant logs page under observability/ The fixed /console/logs route came before the generic :plural route, so an application kind with the plural "logs" would have its list page replaced by the logs page. A two-segment path cannot shadow an application list, the same way the backups and migration pages avoid it. Assisted-by: LLM Signed-off-by: ZverGuy --- .../src/routes/ConsolePage.routing.test.tsx | 24 +++++++++++++++++++ .../apps/console/src/routes/ConsolePage.tsx | 2 +- .../src/routes/sidebar-sections.test.tsx | 2 +- .../console/src/routes/sidebar-sections.tsx | 2 +- 4 files changed, 27 insertions(+), 3 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx index 7c6d1209c..cf6982823 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx @@ -70,4 +70,28 @@ describe("ConsolePage routing", () => { await waitFor(() => expect(client.list).toHaveBeenCalled()) expect(screen.queryByText(/Cluster-scoped capacity/i)).toBeNull() }) + + it("serves the tenant logs page under observability/", async () => { + renderWithK8sProvider( + + + , + { client: makeClient(), initialRoute: "/observability/logs" }, + ) + + expect(await screen.findByText(/^Every application in/)).toBeInTheDocument() + }) + + it("leaves /logs to an application kind whose plural is logs", async () => { + const client = makeClient() + renderWithK8sProvider( + + + , + { client, initialRoute: "/logs" }, + ) + + await waitFor(() => expect(client.list).toHaveBeenCalled()) + expect(screen.queryByText(/^Every application in/)).toBeNull() + }) }) diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx index 5e204e84f..2b2aed3e4 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx @@ -97,7 +97,7 @@ export function ConsolePage() { path="migration/vmimporttasks/:name" element={} /> - } /> + } /> } /> } /> } /> diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx index da40081bc..1fcc3cde4 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx @@ -117,7 +117,7 @@ describe("useConsoleSidebarSections — admin areas moved out", () => { wrapper: makeWrapper(client), }) await waitFor(() => expect(result.current.length).toBeGreaterThan(0)) - expect(findItem(result.current, "Logs")?.to).toBe("/console/logs") + expect(findItem(result.current, "Logs")?.to).toBe("/console/observability/logs") expect(result.current.find((s) => s.title === "Observability")?.items).toHaveLength(1) }) }) diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx index 4cec3a94c..0cf1ce089 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx @@ -121,7 +121,7 @@ export function useConsoleSidebarSections(): SidebarSection[] { const observabilitySection: SidebarSection = { title: "Observability", - items: [{ label: "Logs", to: "/console/logs", icon: ScrollText }], + items: [{ label: "Logs", to: "/console/observability/logs", icon: ScrollText }], } return [...categorySections, backupsSection, migrationSection, observabilitySection] From dab6035a97f9e5d960294bc7c24368b75df7f026 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:17:46 +0700 Subject: [PATCH 125/518] fix(ingress): keep application identity off ingress-nginx scrape-health series Dropping the identity in metricRelabelConfigs reaches only the scraped samples. vmagent adds up and scrape_* with the target labels after metric relabeling (VictoriaMetrics v1.136.0 lib/promscrape addAutoMetrics, addRows with needRelabel=false), so those series still carried the controller pod's identity. Dropping it from the target as well leaves every ingress-nginx series with the label set it had before the scrape class. The metric-stage drop stays: with honorLabels true an exporter-supplied identity is not renamed and would otherwise survive. Assisted-by: LLM Signed-off-by: ZverGuy --- packages/extra/ingress/templates/nginx-scrape.yaml | 7 ++++++- .../extra/ingress/tests/application_identity_test.yaml | 9 ++++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/packages/extra/ingress/templates/nginx-scrape.yaml b/packages/extra/ingress/templates/nginx-scrape.yaml index 780f5e3a9..b98f18fe8 100644 --- a/packages/extra/ingress/templates/nginx-scrape.yaml +++ b/packages/extra/ingress/templates/nginx-scrape.yaml @@ -12,7 +12,8 @@ spec: # The series describe the Ingress objects served, in any namespace, not # this controller, so the application identity the vmagent scrape class # stamps from the controller pod would misattribute them; and with - # honorLabels an exporter-supplied identity would win over the class. + # honorLabels an exporter-supplied identity would win over the class. The + # target-stage drop also covers up and scrape_*, which skip metric relabeling. - port: metrics honorLabels: true interval: 30s @@ -23,6 +24,8 @@ spec: targetLabel: node - targetLabel: tier replacement: cluster + - action: labeldrop + regex: apps_cozystack_io_application_(group|kind|name) metricRelabelConfigs: - action: labeldrop regex: apps_cozystack_io_application_(group|kind|name) @@ -53,6 +56,8 @@ spec: targetLabel: controller - targetLabel: tier replacement: cluster + - action: labeldrop + regex: apps_cozystack_io_application_(group|kind|name) metricRelabelConfigs: - action: labeldrop regex: apps_cozystack_io_application_(group|kind|name) diff --git a/packages/extra/ingress/tests/application_identity_test.yaml b/packages/extra/ingress/tests/application_identity_test.yaml index 0e53a2756..4fed325d2 100644 --- a/packages/extra/ingress/tests/application_identity_test.yaml +++ b/packages/extra/ingress/tests/application_identity_test.yaml @@ -10,13 +10,20 @@ release: # in whatever namespace they live (honorLabels keeps their namespace), so # the controller's identity would misattribute them, and with honorLabels # true an exporter-supplied identity would win over the class. Both -# endpoints drop the identity instead. +# endpoints drop the identity instead: from the target, because vmagent +# adds up and scrape_* with the target labels after metric relabeling, and +# from the scraped samples, because an exporter's copy is not renamed. tests: - it: drops the identity on every endpoint asserts: - lengthEqual: path: spec.podMetricsEndpoints count: 1 + - contains: + path: spec.podMetricsEndpoints[0].relabelConfigs + content: + action: labeldrop + regex: apps_cozystack_io_application_(group|kind|name) - equal: path: spec.podMetricsEndpoints[0].metricRelabelConfigs value: From 52087d92dcaae1316463973643ccbeaf37869a43 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:18:25 +0700 Subject: [PATCH 126/518] fix(dashboard): keep the logs view when only the pod list is refused The pod picker's values query counted as a scope-wide refusal, so a proxy that refused only that path replaced the whole view, filter and lines included, with a notice even though the log query itself worked. A real scope-wide refusal still reaches the history or tail query and shows the notice from there. Assisted-by: LLM Signed-off-by: ZverGuy --- .../src/components/observability/LogsView.test.tsx | 12 ++++++++++++ .../src/components/observability/LogsView.tsx | 3 ++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index 770c8a263..86428733d 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -212,6 +212,18 @@ describe("LogsView", () => { expect(within(frame).queryByText(/No log lines/)).toBeNull() }) + it("keeps the view when only the pod list is refused", async () => { + const status = JSON.stringify({ kind: "Status", apiVersion: "v1", status: "Failure", code: 404 }) + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("stream_field_values") ? new Response(status, { status: 404 }) : defaultHandler(endpoint, params), + ) + + renderWithK8sProvider(, { client }) + + expect(await screen.findByText("ready to accept connections")).toBeInTheDocument() + expect(screen.queryByText("Logs are not available on this cluster.")).toBeNull() + }) + it("names the source pod and container of each line", async () => { const { client } = clientWith(defaultHandler) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index a0a73cc43..c7a1f5d58 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -106,7 +106,8 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) { }) const tail = useLogTail(root, query, followRun, { cap: FOLLOW_CAP }) - const wide = [pods.error, history.error, tail.error].find(isScopeWide) + // The pod list is optional: refusing it alone must not hide lines the query can still read. + const wide = [history.error, tail.error].find(isScopeWide) if (wide) { return (
From ef7312579d9872272baf407f622dea7e5fd05944 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:18:33 +0700 Subject: [PATCH 127/518] fix(dashboard): offer a retry when the log query fails The console does not refetch on window focus and must not poll, so a failed history query stayed failed until the filter or range changed, even after the store recovered. The failed state now has a Retry button that runs the same query again. Assisted-by: LLM Signed-off-by: ZverGuy --- .../src/components/observability/EmptyState.tsx | 4 +++- .../src/components/observability/LogsView.test.tsx | 13 +++++++++++++ .../src/components/observability/LogsView.tsx | 14 +++++++++++--- 3 files changed, 27 insertions(+), 4 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.tsx index 0cf58224b..5b41ef13d 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/EmptyState.tsx @@ -4,12 +4,13 @@ import { cn, Spinner } from "@cozystack/ui" interface EmptyStateProps { title: ReactNode hint?: ReactNode + action?: ReactNode busy?: boolean tone?: "light" | "dark" className?: string } -export function EmptyState({ title, hint, busy, tone = "light", className }: EmptyStateProps) { +export function EmptyState({ title, hint, action, busy, tone = "light", className }: EmptyStateProps) { const dark = tone === "dark" return (
}

{title}

{hint &&

{hint}

} + {action &&
{action}
}
) } diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index 86428733d..8b60b7d35 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -224,6 +224,19 @@ describe("LogsView", () => { expect(screen.queryByText("Logs are not available on this cluster.")).toBeNull() }) + it("retries a failed history query on request", async () => { + let calls = 0 + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("query") && calls++ === 0 ? new Response("upstream down", { status: 502 }) : defaultHandler(endpoint, params), + ) + renderWithK8sProvider(, { client }) + + await userEvent.click(await screen.findByRole("button", { name: "Retry" })) + + expect(await screen.findByText("ready to accept connections")).toBeInTheDocument() + expect(screen.queryByRole("button", { name: "Retry" })).toBeNull() + }) + it("names the source pod and container of each line", async () => { const { client } = clientWith(defaultHandler) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index c7a1f5d58..b8d77003a 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -59,7 +59,7 @@ function Picker({ label, value, options, onChange }: { label: string; value: str const RANGE_KEYS = Object.keys(LOG_RANGES) as LogRangeKey[] const LONGEST_RANGE = RANGE_KEYS[RANGE_KEYS.length - 1] -function placeholderFor({ follow, stopped, loading, failed, longestRange }: { follow: boolean; stopped: boolean; loading: boolean; failed: boolean; longestRange: boolean }) { +function placeholderFor({ follow, stopped, loading, failed, longestRange, retry }: { follow: boolean; stopped: boolean; loading: boolean; failed: boolean; longestRange: boolean; retry: () => void }) { if (follow) { return stopped ? ( @@ -68,7 +68,15 @@ function placeholderFor({ follow, stopped, loading, failed, longestRange }: { fo ) } if (loading) return - if (failed) return + if (failed) { + return ( + Retry} + /> + ) + } return } @@ -154,7 +162,7 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) { void history.refetch() })} />
) From 7d43c3bcd6470c1282a539e05609e9a96de25067 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:18:38 +0700 Subject: [PATCH 128/518] fix(dashboard): open each new log result at its newest line The console keeps a query's previous data as placeholder while the next one loads, so the log list was never remounted between queries. After a reader scrolled up, a new filter or range opened at the old scroll offset instead of the newest lines. The list is now keyed on the query and range as well. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/LogsView.test.tsx | 25 ++++++++++++++++--- .../src/components/observability/LogsView.tsx | 3 ++- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index 8b60b7d35..1a703cceb 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -86,13 +86,13 @@ describe("LogsView", () => { endpoint.endsWith("query") ? new Response("") : defaultHandler(endpoint, params), ) renderWithK8sProvider(, { client }) - const frame = screen.getByRole("region", { name: "Log lines" }) - expect(await within(frame).findByText("Try a longer range or a broader filter.")).toBeInTheDocument() + const frame = () => screen.getByRole("region", { name: "Log lines" }) + expect(await within(frame()).findByText("Try a longer range or a broader filter.")).toBeInTheDocument() await userEvent.click(screen.getByRole("button", { name: "24h" })) - expect(await within(frame).findByText("Try a broader filter.")).toBeInTheDocument() - expect(within(frame).queryByText(/longer range/)).toBeNull() + await waitFor(() => expect(within(frame()).getByText("Try a broader filter.")).toBeInTheDocument()) + expect(within(frame()).queryByText(/longer range/)).toBeNull() }) it("shows loading inside the log console", () => { @@ -237,6 +237,23 @@ describe("LogsView", () => { expect(screen.queryByRole("button", { name: "Retry" })).toBeNull() }) + it("opens a new history result at its newest line, not the old scroll position", async () => { + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("query") + ? new Response(`${line("2026-11-04T10:00:01Z", `result for ${params.get("query")}`)}\n`) + : defaultHandler(endpoint, params), + ) + const { queryClient } = renderWithK8sProvider(, { client }) + queryClient.setDefaultOptions({ queries: { ...queryClient.getDefaultOptions().queries, placeholderData: (prev: unknown) => prev } }) + await screen.findByText("result for *") + const before = screen.getByRole("list") + + await userEvent.type(screen.getByLabelText("LogsQL filter"), "error{Enter}") + + await screen.findByText("result for error") + expect(screen.getByRole("list")).not.toBe(before) + }) + it("names the source pod and container of each line", async () => { const { client } = clientWith(defaultHandler) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index b8d77003a..1a25d4229 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -160,7 +160,8 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) { )} {error ? : null} void history.refetch() })} /> From 601c9ece3f0ce24792d1dc1f9dce862a6826e876 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:18:47 +0700 Subject: [PATCH 129/518] docs(dashboard): note why the metrics panels refresh on an interval The console rule against refetchInterval assumes a watch is available, which holds for Kubernetes lists but not for the proxy's Prometheus query API. Record the exception so the metrics panels' refresh does not read as a rule violation. Assisted-by: LLM Signed-off-by: ZverGuy --- packages/system/dashboard/images/console/CLAUDE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/packages/system/dashboard/images/console/CLAUDE.md b/packages/system/dashboard/images/console/CLAUDE.md index 96419d118..036e527d6 100644 --- a/packages/system/dashboard/images/console/CLAUDE.md +++ b/packages/system/dashboard/images/console/CLAUDE.md @@ -81,6 +81,7 @@ the apiserver checks that user's RBAC. to a chunked-encoding WATCH against the same `resourceVersion`. Don't poll. Don't add `refetchInterval`. If you need a one-shot, pass `{ watch: false }`. + The one exception is the metrics panels in `components/observability`: the observability proxy serves the Prometheus query API, which has no watch, so they refresh on a fixed interval instead. Log lines stream through the tail endpoint and are not polled. 4. **Tenant scoping.** Most resources live in `tenant-` namespaces. Pull the active tenant from `useTenantContext()` — never read the namespace from a URL param or guess it. From 3390100226a60f8e399fc589f42c5dbe32af4e27 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:21:26 +0700 Subject: [PATCH 130/518] docs(monitoring): state which scrapes the identity class covers The comment promised identity labels on every target and forbade scrape objects from touching them. Neither holds: operator v0.68.4 applies only a class's auth to VMProbe (factory/vmagent/probe.go), raw scrape configs bypass classes, and the cadvisor, kubelet and ingress-nginx scrapes drop the labels on purpose. The real invariant for a scrape object is that it never sets them; dropping them is safe. Assisted-by: LLM Signed-off-by: ZverGuy --- .../monitoring-agents/templates/vmagent.yaml | 18 ++++++++++-------- .../vmagent_application_identity_test.yaml | 4 ++-- .../monitoring/templates/vm/vmagent.yaml | 18 ++++++++++-------- 3 files changed, 22 insertions(+), 18 deletions(-) diff --git a/packages/system/monitoring-agents/templates/vmagent.yaml b/packages/system/monitoring-agents/templates/vmagent.yaml index ad26881b7..e001fc40f 100644 --- a/packages/system/monitoring-agents/templates/vmagent.yaml +++ b/packages/system/monitoring-agents/templates/vmagent.yaml @@ -20,14 +20,16 @@ spec: scrapeInterval: 30s selectAllByDefault: true - # The target always carries all three identity labels, "-" when the pod - # has no lineage ("-" is not a valid Kubernetes label value, so it never - # collides with a real one). Under honorLabels false every exporter-supplied - # copy then collides, is renamed to exported_* and dropped. Under - # honorLabels true the exporter's copy wins, so such a scrape must drop - # these labels itself. A scrape object's own relabel rules run after these - # and must leave the labels alone. The placeholder is removed by the - # remote-write relabeling below. + # The class sets all three identity labels on every target it covers, "-" + # when the pod has no lineage ("-" is not a valid Kubernetes label value, + # so it never collides with a real one). Under honorLabels false every + # exporter-supplied copy then collides, is renamed to exported_* and + # dropped. Under honorLabels true the exporter's copy wins, so such a + # scrape must drop these labels itself. A scrape object's own relabel + # rules run after these: they may drop the labels but must never set them. + # Not covered: VMProbe (operator v0.68.4 takes only auth from a class) and + # raw scrape configs. The placeholder is removed by the remote-write + # relabeling below. scrapeClasses: - name: application-identity default: true diff --git a/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml b/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml index 95f330d50..4e4b93385 100644 --- a/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml +++ b/packages/system/monitoring-agents/tests/vmagent_application_identity_test.yaml @@ -7,8 +7,8 @@ release: # The tenant-resource-metrics proxy scopes a query to one application by # apps_cozystack_io_application_{group,kind,name}. The labels are only a -# boundary if the scraped pod cannot choose them: the target always carries -# all three (a placeholder when the pod has no lineage), so with +# boundary if the scraped pod cannot choose them: the class sets all three +# on every target it covers (a placeholder when the pod has no lineage), so with # honorLabels false an exporter's own copy is renamed to exported_* and # dropped. The same rules live in the tenant vmagent of the monitoring # package and are pinned there by the same assertions. diff --git a/packages/system/monitoring/templates/vm/vmagent.yaml b/packages/system/monitoring/templates/vm/vmagent.yaml index 2748ac4bf..919fdef63 100644 --- a/packages/system/monitoring/templates/vm/vmagent.yaml +++ b/packages/system/monitoring/templates/vm/vmagent.yaml @@ -24,14 +24,16 @@ spec: scrapeInterval: 30s selectAllByDefault: false - # The target always carries all three identity labels, "-" when the pod - # has no lineage ("-" is not a valid Kubernetes label value, so it never - # collides with a real one). Under honorLabels false every exporter-supplied - # copy then collides, is renamed to exported_* and dropped. Under - # honorLabels true the exporter's copy wins, so such a scrape must drop - # these labels itself. A scrape object's own relabel rules run after these - # and must leave the labels alone. The placeholder is removed by the - # remote-write relabeling below. + # The class sets all three identity labels on every target it covers, "-" + # when the pod has no lineage ("-" is not a valid Kubernetes label value, + # so it never collides with a real one). Under honorLabels false every + # exporter-supplied copy then collides, is renamed to exported_* and + # dropped. Under honorLabels true the exporter's copy wins, so such a + # scrape must drop these labels itself. A scrape object's own relabel + # rules run after these: they may drop the labels but must never set them. + # Not covered: VMProbe (operator v0.68.4 takes only auth from a class) and + # raw scrape configs. The placeholder is removed by the remote-write + # relabeling below. scrapeClasses: - name: application-identity default: true From 1018eaa3b632fbd708c57c6bda11c4d946c88d4a Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:27:02 +0700 Subject: [PATCH 131/518] docs(ingress): name the controller series that lose application identity The comment said every ingress-nginx series describes served Ingress objects. The metrics port also serves the controller's own series (nginx process, reload status, go_*, up, scrape_*), and dropping the identity on the whole endpoint removes it from those as well. Say so, so the lost attribution reads as a decision rather than an oversight. Assisted-by: LLM Signed-off-by: ZverGuy --- packages/extra/ingress/templates/nginx-scrape.yaml | 12 +++++++----- .../ingress/tests/application_identity_test.yaml | 9 +++++---- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/packages/extra/ingress/templates/nginx-scrape.yaml b/packages/extra/ingress/templates/nginx-scrape.yaml index b98f18fe8..4917206de 100644 --- a/packages/extra/ingress/templates/nginx-scrape.yaml +++ b/packages/extra/ingress/templates/nginx-scrape.yaml @@ -9,11 +9,13 @@ spec: matchNames: - {{ .Release.Namespace }} podMetricsEndpoints: - # The series describe the Ingress objects served, in any namespace, not - # this controller, so the application identity the vmagent scrape class - # stamps from the controller pod would misattribute them; and with - # honorLabels an exporter-supplied identity would win over the class. The - # target-stage drop also covers up and scrape_*, which skip metric relabeling. + # Most series describe the Ingress objects served, in any namespace, so + # the application identity the vmagent scrape class stamps from the + # controller pod would misattribute them; and with honorLabels an + # exporter-supplied identity would win over the class. The controller's + # own series (nginx process, reload status, go_*, up, scrape_*) lose the + # Ingress application's identity too; that is the accepted cost. The + # target-stage drop covers up and scrape_*, which skip metric relabeling. - port: metrics honorLabels: true interval: 30s diff --git a/packages/extra/ingress/tests/application_identity_test.yaml b/packages/extra/ingress/tests/application_identity_test.yaml index 4fed325d2..db781a102 100644 --- a/packages/extra/ingress/tests/application_identity_test.yaml +++ b/packages/extra/ingress/tests/application_identity_test.yaml @@ -6,10 +6,11 @@ release: namespace: tenant-root # The vmagent scrape class stamps the scraped pod's application identity on -# every series. ingress-nginx series describe the Ingress objects it serves, -# in whatever namespace they live (honorLabels keeps their namespace), so -# the controller's identity would misattribute them, and with honorLabels -# true an exporter-supplied identity would win over the class. Both +# every series. Most ingress-nginx series describe the Ingress objects it +# serves, in whatever namespace they live (honorLabels keeps their +# namespace), so the controller's identity would misattribute them, and with +# honorLabels true an exporter-supplied identity would win over the class. +# The controller's own health series lose that identity too, by design. Both # endpoints drop the identity instead: from the target, because vmagent # adds up and scrape_* with the target labels after metric relabeling, and # from the scraped samples, because an exporter's copy is not renamed. From 699b5ce78e7d7c5d5b49d556eed086c6fe3d1a6d Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:35:37 +0700 Subject: [PATCH 132/518] test(ingress): pin the position of the target-stage identity drop A containment check passes even if a rule added after the drop sets the identity labels again. Asserting both endpoints' full target relabel lists fixes the drop as the last rule, so such a change fails the test. Assisted-by: LLM Signed-off-by: ZverGuy --- .../tests/application_identity_test.yaml | 40 ++++++++++++++++--- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/packages/extra/ingress/tests/application_identity_test.yaml b/packages/extra/ingress/tests/application_identity_test.yaml index db781a102..4914049c8 100644 --- a/packages/extra/ingress/tests/application_identity_test.yaml +++ b/packages/extra/ingress/tests/application_identity_test.yaml @@ -20,13 +20,43 @@ tests: - lengthEqual: path: spec.podMetricsEndpoints count: 1 - - contains: - path: spec.podMetricsEndpoints[0].relabelConfigs - content: - action: labeldrop - regex: apps_cozystack_io_application_(group|kind|name) - equal: path: spec.podMetricsEndpoints[0].metricRelabelConfigs value: - action: labeldrop regex: apps_cozystack_io_application_(group|kind|name) + + # Endpoint rules run after the scrape class rules, so the drop takes + # effect wherever it sits in this list; it is pinned last so that no + # later rule can set the identity again. + - it: drops the identity from the basic endpoint's target as its last rule + documentIndex: 0 + asserts: + - equal: + path: spec.podMetricsEndpoints[0].relabelConfigs + value: + - replacement: nginx-ingress-controller + targetLabel: job + - sourceLabels: [__meta_kubernetes_pod_node_name] + targetLabel: node + - targetLabel: tier + replacement: cluster + - action: labeldrop + regex: apps_cozystack_io_application_(group|kind|name) + + - it: drops the identity from the detailed endpoint's target as its last rule + documentIndex: 1 + asserts: + - equal: + path: spec.podMetricsEndpoints[0].relabelConfigs + value: + - replacement: nginx-ingress-controller + targetLabel: job + - sourceLabels: [__meta_kubernetes_pod_node_name] + targetLabel: node + - sourceLabels: [__meta_kubernetes_pod_label_app_kubernetes_io_name] + targetLabel: controller + - targetLabel: tier + replacement: cluster + - action: labeldrop + regex: apps_cozystack_io_application_(group|kind|name) From 046df7fe969f2b0b6d62ced127ed601463d9dfc8 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:53:06 +0700 Subject: [PATCH 133/518] fix(dashboard): say the legend shows the latest sample In the browser the legend's time still read "--" next to the latest values, and the data table repeated the same minute on every row at the one-hour range, whose step is 15 seconds. The legend's time now reads "latest" until a point is hovered, and the table shows seconds. Assisted-by: LLM Signed-off-by: ZverGuy --- .../src/components/observability/SeriesTable.test.tsx | 2 +- .../src/components/observability/SeriesTable.tsx | 2 +- .../components/observability/TimeSeriesChart.test.tsx | 10 ++++++++++ .../src/components/observability/TimeSeriesChart.tsx | 3 ++- 4 files changed, 14 insertions(+), 3 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.test.tsx index 0b3364268..e85041c0a 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.test.tsx @@ -32,6 +32,6 @@ describe("SeriesTable", () => { expect(rows).toHaveLength(2) expect(within(rows[0]).getAllByRole("cell").map((c) => c.textContent)).toEqual(["2 B", "3 B"]) expect(within(rows[1]).getAllByRole("cell").map((c) => c.textContent)).toEqual(["1 B", "–"]) - expect(within(rows[0]).getByRole("rowheader")).toHaveTextContent(new Date(1_700_000_060_000).toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" })) + expect(within(rows[0]).getByRole("rowheader")).toHaveTextContent(new Date(1_700_000_060_000).toLocaleTimeString([], { hour: "2-digit", minute: "2-digit", second: "2-digit" })) }) }) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx index 57cf755a0..4cc382781 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx @@ -8,7 +8,7 @@ interface SeriesTableProps { } const formatTime = (seconds: number) => - new Date(seconds * 1000).toLocaleString([], { month: "short", day: "numeric", hour: "2-digit", minute: "2-digit" }) + new Date(seconds * 1000).toLocaleString([], { month: "short", day: "numeric", hour: "2-digit", minute: "2-digit", second: "2-digit" }) export function SeriesTable({ id, title, data, format }: SeriesTableProps) { const newestFirst = data.timestamps.map((t, i) => ({ t, i })).reverse() diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx index 3df8125e3..13cbe933d 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx @@ -13,6 +13,7 @@ vi.mock("uplot", () => ({ created.push({ opts, data, destroy: this.destroy }) } setSize() {} + static fmtDate = () => (d: Date) => d.toISOString() }, })) @@ -49,6 +50,15 @@ describe("TimeSeriesChart", () => { expect(legend(null, null, 1, 2)).toBe("–") }) + it("labels the legend's time as latest while nothing is hovered", () => { + render() + + const time = created[0].opts.series[0].value! + + expect(time(null, null, 0, null)).toBe("latest") + expect(time(null, 2, 0, 1)).toBe(new Date(2000).toISOString()) + }) + it("destroys the chart on unmount", () => { const { unmount } = render() diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx index a6251d40f..063b42b12 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx @@ -19,6 +19,7 @@ export function TimeSeriesChart({ data, format }: TimeSeriesChartProps) { if (!el) return const fmt = (v: number | null) => (v === null ? "–" : format(v)) const latest = data.series.map((s) => s.values.findLast((v) => v !== null) ?? null) + const stamp = uPlot.fmtDate("{M}/{D}/{YY} {h}:{mm}:{ss} {AA}") const chart = new uPlot( { width: el.clientWidth || 600, @@ -26,7 +27,7 @@ export function TimeSeriesChart({ data, format }: TimeSeriesChartProps) { scales: { x: { time: true } }, legend: { live: true }, series: [ - {}, + { value: (_: uPlot, v: number | null, _sidx: number, idx: number | null) => (idx === null || v === null ? "latest" : stamp(new Date(v * 1000))) }, ...data.series.map((s, i) => ({ label: s.label, stroke: COLORS[i % COLORS.length], From bfa85e57bfe9a619d6c6bcba31c842171a4d848d Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:56:26 +0700 Subject: [PATCH 134/518] fix(dashboard): show sample times the same way in legend and table The legend used a fixed US date format while the data table used the reader's locale, so one panel showed the same instant two ways. Both now use one locale-aware format. A test also pins that the data toggle stays disabled for a panel with no series or an error. Assisted-by: LLM Signed-off-by: ZverGuy --- .../components/observability/MetricsView.test.tsx | 15 +++++++++++++++ .../src/components/observability/SeriesTable.tsx | 7 ++----- .../observability/TimeSeriesChart.test.tsx | 4 ++-- .../components/observability/TimeSeriesChart.tsx | 5 ++--- .../apps/console/src/lib/observability/metrics.ts | 4 ++++ 5 files changed, 25 insertions(+), 10 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx index 8e7829ed5..ece4d70ba 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx @@ -123,6 +123,21 @@ describe("MetricsView", () => { expect(screen.getByRole("button", { name: "Show data for Network" })).toBeDisabled() }) + it("offers no table for a panel with no series or an error", async () => { + const { client } = clientWith((url) => + url.searchParams.get("query")?.includes("memory") + ? Response.json({ status: "error", error: "too many series" }, { status: 422 }) + : Response.json({ status: "success", data: { resultType: "matrix", result: [] } }), + ) + renderWithK8sProvider(, { client }) + + await screen.findByText("too many series. Choose a shorter range.") + await waitFor(() => expect(screen.getAllByText("No data for this range.")).toHaveLength(2)) + for (const title of ["CPU (cores)", "Memory (working set)", "Network"]) { + expect(screen.getByRole("button", { name: `Show data for ${title}` })).toBeDisabled() + } + }) + it("holds each chart's place while it loads", () => { const { client } = clientWith(() => new Promise(() => {})) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx index 4cc382781..5e9d88ba1 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/SeriesTable.tsx @@ -1,4 +1,4 @@ -import type { ChartData } from "../../lib/observability/metrics.ts" +import { formatSampleTime, type ChartData } from "../../lib/observability/metrics.ts" interface SeriesTableProps { id: string @@ -7,9 +7,6 @@ interface SeriesTableProps { format: (value: number) => string } -const formatTime = (seconds: number) => - new Date(seconds * 1000).toLocaleString([], { month: "short", day: "numeric", hour: "2-digit", minute: "2-digit", second: "2-digit" }) - export function SeriesTable({ id, title, data, format }: SeriesTableProps) { const newestFirst = data.timestamps.map((t, i) => ({ t, i })).reverse() return ( @@ -33,7 +30,7 @@ export function SeriesTable({ id, title, data, format }: SeriesTableProps) { {newestFirst.map(({ t, i }) => ( - {formatTime(t)} + {formatSampleTime(t)} {data.series.map((s, j) => { const v = s.values[i] return {v === null ? "–" : format(v)} diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx index 13cbe933d..bb76edb5a 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.test.tsx @@ -13,11 +13,11 @@ vi.mock("uplot", () => ({ created.push({ opts, data, destroy: this.destroy }) } setSize() {} - static fmtDate = () => (d: Date) => d.toISOString() }, })) const { TimeSeriesChart } = await import("./TimeSeriesChart.tsx") +const { formatSampleTime } = await import("../../lib/observability/metrics.ts") beforeEach(() => { created.length = 0 @@ -56,7 +56,7 @@ describe("TimeSeriesChart", () => { const time = created[0].opts.series[0].value! expect(time(null, null, 0, null)).toBe("latest") - expect(time(null, 2, 0, 1)).toBe(new Date(2000).toISOString()) + expect(time(null, 2, 0, 1)).toBe(formatSampleTime(2)) }) it("destroys the chart on unmount", () => { diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx index 063b42b12..607d44593 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/TimeSeriesChart.tsx @@ -1,7 +1,7 @@ import { useEffect, useRef } from "react" import uPlot from "uplot" import "uplot/dist/uPlot.min.css" -import type { ChartData } from "../../lib/observability/metrics.ts" +import { formatSampleTime, type ChartData } from "../../lib/observability/metrics.ts" const COLORS = ["#2563eb", "#16a34a", "#d97706", "#9333ea", "#dc2626", "#0891b2", "#4b5563", "#db2777"] const HEIGHT = 200 @@ -19,7 +19,6 @@ export function TimeSeriesChart({ data, format }: TimeSeriesChartProps) { if (!el) return const fmt = (v: number | null) => (v === null ? "–" : format(v)) const latest = data.series.map((s) => s.values.findLast((v) => v !== null) ?? null) - const stamp = uPlot.fmtDate("{M}/{D}/{YY} {h}:{mm}:{ss} {AA}") const chart = new uPlot( { width: el.clientWidth || 600, @@ -27,7 +26,7 @@ export function TimeSeriesChart({ data, format }: TimeSeriesChartProps) { scales: { x: { time: true } }, legend: { live: true }, series: [ - { value: (_: uPlot, v: number | null, _sidx: number, idx: number | null) => (idx === null || v === null ? "latest" : stamp(new Date(v * 1000))) }, + { value: (_: uPlot, v: number | null, _sidx: number, idx: number | null) => (idx === null || v === null ? "latest" : formatSampleTime(v)) }, ...data.series.map((s, i) => ({ label: s.label, stroke: COLORS[i % COLORS.length], diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts index d7182b8d6..2f77e251f 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/metrics.ts @@ -93,6 +93,10 @@ export const METRIC_PANELS: MetricPanel[] = [ }, ] +export function formatSampleTime(seconds: number): string { + return new Date(seconds * 1000).toLocaleString([], { month: "short", day: "numeric", hour: "2-digit", minute: "2-digit", second: "2-digit" }) +} + export interface ChartSeries { label: string values: (number | null)[] From 311cb1c90b751bcc24cf962e68676a6997680c4d Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:10:02 +0700 Subject: [PATCH 135/518] fix(dashboard): skip tail dedup for rows without a stream ID A pipe such as `delete _stream_id` leaves rows from different streams that share a timestamp with the same dedup key, so a resumed tail dropped a record it had never shown. Only rows that still carry both _stream_id and _time are deduplicated now, as the accepted logs design specifies. Assisted-by: LLM Signed-off-by: ZverGuy --- .../console/src/lib/observability/logs.test.ts | 18 ++++++++++++++++++ .../apps/console/src/lib/observability/logs.ts | 15 ++++++++++----- 2 files changed, 28 insertions(+), 5 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts index 62d1ac249..278da6890 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.test.ts @@ -63,6 +63,24 @@ describe("mergeTail", () => { expect(out).toHaveLength(2) }) + it("never deduplicates lines a pipe stripped of _stream_id, even on a resume", () => { + const t = "2026-01-01T00:00:02Z" + const buffer = [{ _time: t, _msg: "from stream 1" }] + + const out = mergeTail(buffer, [{ _time: t, _msg: "from stream 2" }], 10, t) + + expect(out.map((x) => x._msg)).toEqual(["from stream 1", "from stream 2"]) + }) + + it("deduplicates only the complete rows of a mixed resume", () => { + const t = "2026-01-01T00:00:02Z" + const buffer = [e(t, "seen"), { _time: t, _msg: "no stream" }, { _stream_id: "s1", _msg: "no time" }] + + const out = mergeTail(buffer, [e(t, "seen"), { _time: t, _msg: "no stream" }, { _stream_id: "s1", _msg: "no time" }], 10, t) + + expect(out.map((x) => x._msg)).toEqual(["seen", "no stream", "no time", "no stream", "no time"]) + }) + it("does not deduplicate outside a resume", () => { const out = mergeTail([e("t1", "a")], [e("t1", "a")], 10) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts index eb24a5f96..b1005a1cd 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/logs.ts @@ -58,16 +58,21 @@ export function compareTime(a: string, b: string): number { return x < y ? -1 : x > y ? 1 : 0 } -const entryKey = (e: LogEntry) => `${e._stream_id ?? ""}\u0000${e._time}` +// Null when a pipe removed either identity field: such a row cannot be told from another stream's. +const entryKey = (e: LogEntry) => + e._stream_id !== undefined && e._time !== undefined ? `${e._stream_id}\u0000${e._time}` : null // (_stream_id, _time) is not unique in VictoriaLogs, so dedup applies only to -// lines a resumed tail replays at or before the resume point, and never to -// lines a pipe stripped of _time. +// lines a resumed tail replays at or before the resume point, and only to +// lines that still carry both fields. export function mergeTail(buffer: LogEntry[], incoming: LogEntry[], cap: number, resumedFrom?: string): LogEntry[] { let fresh = incoming if (resumedFrom) { - const seen = new Set(buffer.map(entryKey)) - fresh = incoming.filter((e) => e._time === undefined || compareTime(e._time, resumedFrom) > 0 || !seen.has(entryKey(e))) + const seen = new Set(buffer.map(entryKey).filter((k) => k !== null)) + fresh = incoming.filter((e) => { + const key = entryKey(e) + return key === null || compareTime(e._time ?? "", resumedFrom) > 0 || !seen.has(key) + }) } const merged = buffer.concat(fresh) return merged.length > cap ? merged.slice(merged.length - cap) : merged From 9c111b3a95f77731b8296310cfd0d32a431b50d6 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:11:34 +0700 Subject: [PATCH 136/518] fix(dashboard): keep placeholder results within one observability scope The provider keeps a query's previous result while its key changes, so switching tenants showed the first tenant's log lines and pod names under the second tenant's heading until its request returned. Every observability query now keeps the previous result only while the proxy root is the same, so a new filter or range still shows the old lines, but a new tenant or application starts empty. Tests render with the provider's own query defaults. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/LogsView.test.tsx | 49 ++++++++++++++++++- .../src/components/observability/LogsView.tsx | 3 ++ .../observability/MetricsView.test.tsx | 20 +++++++- .../components/observability/MetricsView.tsx | 3 +- .../console/src/lib/observability/proxy.ts | 8 +++ 5 files changed, 78 insertions(+), 5 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index 1a703cceb..fe73e4164 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -1,7 +1,7 @@ import { describe, it, expect, vi } from "vitest" -import { screen, waitFor, within } from "@testing-library/react" +import { render, screen, waitFor, within } from "@testing-library/react" import userEvent from "@testing-library/user-event" -import { K8sClient } from "@cozystack/k8s-client" +import { K8sClient, K8sProvider } from "@cozystack/k8s-client" import { renderWithK8sProvider } from "../../test-utils/render.tsx" import { LogsView } from "./LogsView.tsx" @@ -30,6 +30,51 @@ function defaultHandler(endpoint: string, params?: URLSearchParams): Response | return new Promise(() => {}) } +// The provider's own QueryClient keeps the previous result while a key changes; it is +// shared across tests, so each test below uses namespaces no other test reads. +function renderWithProductionDefaults(client: K8sClient, namespace: string) { + const view = (ns: string) => ( + + + + ) + const result = render(view(namespace)) + return { switchTo: (ns: string) => result.rerender(view(ns)) } +} + +describe("LogsView with the production query defaults", () => { + it("drops the previous tenant's lines and pods while the next tenant loads", async () => { + const { client } = clientWith((endpoint, params) => + endpoint.includes("/tenant-switch-b/") ? new Promise(() => {}) : defaultHandler(endpoint, params), + ) + const { switchTo } = renderWithProductionDefaults(client, "tenant-switch-a") + expect(await screen.findByText("ready to accept connections")).toBeInTheDocument() + await userEvent.click(screen.getByRole("combobox", { name: "Pod" })) + await screen.findByRole("option", { name: "db-1" }) + await userEvent.keyboard("{Escape}") + + switchTo("tenant-switch-b") + + const frame = screen.getByRole("region", { name: "Log lines" }) + expect(within(frame).getByText("Loading log lines…")).toBeInTheDocument() + expect(screen.queryByText("ready to accept connections")).toBeNull() + await userEvent.click(screen.getByRole("combobox", { name: "Pod" })) + expect((await screen.findAllByRole("option")).map((o) => o.textContent)).toEqual(["All"]) + }) + + it("keeps the current lines while a new filter in the same tenant loads", async () => { + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("query") && params.get("query") === "slow" ? new Promise(() => {}) : defaultHandler(endpoint, params), + ) + renderWithProductionDefaults(client, "tenant-filter-a") + expect(await screen.findByText("ready to accept connections")).toBeInTheDocument() + + await userEvent.type(screen.getByLabelText("LogsQL filter"), "slow{Enter}") + + expect(screen.getByText("ready to accept connections")).toBeInTheDocument() + }) +}) + describe("LogsView", () => { it("shows recent lines from the application root", async () => { const { client, raw } = clientWith(defaultHandler) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index 1a25d4229..c2dc8ca5c 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -14,6 +14,7 @@ import { import { fieldValues, isScopeWide, + keepWithinScope, logsRoot, queryLogs, type ObservabilityScope, @@ -101,6 +102,7 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) { query: buildLogsQuery({ filter: "", pod: narrowTo }), ...windowFor(range), }, signal), + placeholderData: keepWithinScope(root), retry: false, }) const pods = useQuery(pickerQuery(POD_FIELD)) @@ -110,6 +112,7 @@ export function LogsView({ scope }: { scope: ObservabilityScope }) { queryKey: ["observability", "logs", root, query, range], queryFn: ({ signal }) => queryLogs(client, root, { query, limit: HISTORY_LIMIT, ...windowFor(range) }, signal), enabled: !follow, + placeholderData: keepWithinScope(root), retry: false, }) const tail = useLogTail(root, query, followRun, { cap: FOLLOW_CAP }) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx index ece4d70ba..fccb6f43d 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.test.tsx @@ -1,7 +1,7 @@ import { describe, it, expect, vi } from "vitest" -import { screen, waitFor, within } from "@testing-library/react" +import { render, screen, waitFor, within } from "@testing-library/react" import userEvent from "@testing-library/user-event" -import { K8sClient } from "@cozystack/k8s-client" +import { K8sClient, K8sProvider } from "@cozystack/k8s-client" import { renderWithK8sProvider } from "../../test-utils/render.tsx" import type { ChartData } from "../../lib/observability/metrics.ts" @@ -138,6 +138,22 @@ describe("MetricsView", () => { } }) + it("drops the previous application's charts while the next one loads, with the production query defaults", async () => { + const { client } = clientWith((url) => (url.pathname.includes("/switch-b/") ? new Promise(() => {}) : matrix("db-0"))) + const view = (name: string) => ( + + + + ) + const { rerender } = render(view("switch-a")) + await waitFor(() => expect(screen.getAllByTestId("chart")).toHaveLength(3)) + + rerender(view("switch-b")) + + expect(screen.queryAllByTestId("chart")).toHaveLength(0) + expect(screen.getAllByText("Loading…")).toHaveLength(3) + }) + it("holds each chart's place while it loads", () => { const { client } = clientWith(() => new Promise(() => {})) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx index 4663b2ae3..07980aacf 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx @@ -9,7 +9,7 @@ import { toChartData, type RangeKey, } from "../../lib/observability/metrics.ts" -import { isScopeWide, metricsRoot, queryRange, type ObservabilityScope } from "../../lib/observability/proxy.ts" +import { isScopeWide, keepWithinScope, metricsRoot, queryRange, type ObservabilityScope } from "../../lib/observability/proxy.ts" import { ProxyNotice } from "./ProxyNotice.tsx" import { TimeSeriesChart } from "./TimeSeriesChart.tsx" import { RangePicker } from "./RangePicker.tsx" @@ -42,6 +42,7 @@ export function MetricsView({ scope }: { scope: ObservabilityScope }) { ) return toChartData(series) }, + placeholderData: keepWithinScope(root), retry: false, refetchInterval: REFRESH_MS, })), diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts index c28bd12d4..3986c6a3d 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts @@ -20,6 +20,14 @@ export function logsRoot(scope: ObservabilityScope): string { return app ? `${app}/log/` : `/apis/cozystack.io/v1alpha1/namespaces/${scope.namespace}/logs/` } +// The provider keeps a query's previous result while its key changes; across +// scopes that would show one tenant's or application's data under another. +// Every observability query key carries its root at index 2. +export function keepWithinScope(root: string) { + return (previous: T | undefined, previousQuery?: { queryKey: readonly unknown[] }): T | undefined => + previousQuery?.queryKey[2] === root ? previous : undefined +} + export type ProxyErrorKind = | "bad-request" | "unauthorized" From 711a82eaaf3b073c4e69b6bca94c235adf9a891f Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:15:06 +0700 Subject: [PATCH 137/518] refactor(dashboard): drop the scoped placeholder from metrics panels useQueries creates a new observer for every new key, so the metrics panels never carried a previous result across scopes and the scoped placeholder there did nothing. The switch test for the panels stays and passes without it; the helper's comment now says which queries it is for. Assisted-by: LLM Signed-off-by: ZverGuy --- .../console/src/components/observability/MetricsView.tsx | 3 +-- .../console/apps/console/src/lib/observability/proxy.ts | 5 +++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx index 07980aacf..4663b2ae3 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/MetricsView.tsx @@ -9,7 +9,7 @@ import { toChartData, type RangeKey, } from "../../lib/observability/metrics.ts" -import { isScopeWide, keepWithinScope, metricsRoot, queryRange, type ObservabilityScope } from "../../lib/observability/proxy.ts" +import { isScopeWide, metricsRoot, queryRange, type ObservabilityScope } from "../../lib/observability/proxy.ts" import { ProxyNotice } from "./ProxyNotice.tsx" import { TimeSeriesChart } from "./TimeSeriesChart.tsx" import { RangePicker } from "./RangePicker.tsx" @@ -42,7 +42,6 @@ export function MetricsView({ scope }: { scope: ObservabilityScope }) { ) return toChartData(series) }, - placeholderData: keepWithinScope(root), retry: false, refetchInterval: REFRESH_MS, })), diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts index 3986c6a3d..d33052f70 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts @@ -20,8 +20,9 @@ export function logsRoot(scope: ObservabilityScope): string { return app ? `${app}/log/` : `/apis/cozystack.io/v1alpha1/namespaces/${scope.namespace}/logs/` } -// The provider keeps a query's previous result while its key changes; across -// scopes that would show one tenant's or application's data under another. +// The provider's default keeps a useQuery's previous result while its key +// changes; across scopes that would show one tenant's or application's data +// under another. (useQueries starts a new observer per key, so it never does.) // Every observability query key carries its root at index 2. export function keepWithinScope(root: string) { return (previous: T | undefined, previousQuery?: { queryKey: readonly unknown[] }): T | undefined => From 89194b4f88e4a7399709a48946419745be7326ec Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:17:48 +0700 Subject: [PATCH 138/518] fix(dashboard): reset the log filters when the tenant or app changes LogsView kept its filter, pod and container across a scope switch, so the first request for the new tenant still asked for the previous tenant's pod before the picker fell back to All. The view's state is now keyed on the proxy root, so a new tenant or application starts with no filter and All selected. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/LogsView.test.tsx | 20 +++++++++++++++++++ .../src/components/observability/LogsView.tsx | 7 ++++++- 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index fe73e4164..a0df67c68 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -62,6 +62,26 @@ describe("LogsView with the production query defaults", () => { expect((await screen.findAllByRole("option")).map((o) => o.textContent)).toEqual(["All"]) }) + it("starts the next tenant with no filter, pod or container of the previous one", async () => { + const { client, raw } = clientWith(defaultHandler) + const { switchTo } = renderWithProductionDefaults(client, "tenant-reset-a") + await userEvent.type(screen.getByLabelText("LogsQL filter"), "error{Enter}") + await userEvent.click(screen.getByRole("combobox", { name: "Pod" })) + await userEvent.click(await screen.findByRole("option", { name: "db-1" })) + await waitFor(() => + expect(raw.mock.calls.some(([p]) => new URL(p, "http://x").searchParams.get("query") === '{kubernetes_pod_name="db-1"} (error)')).toBe(true), + ) + + switchTo("tenant-reset-b") + + await waitFor(() => expect(raw.mock.calls.some(([p]) => p.includes("/tenant-reset-b/") && p.includes("select/logsql/query?"))).toBe(true)) + const first = raw.mock.calls.find(([p]) => p.includes("/tenant-reset-b/") && p.includes("select/logsql/query?"))! + expect(new URL(first[0], "http://x").searchParams.get("query")).toBe("*") + expect(screen.getByRole("combobox", { name: "Pod" })).toHaveTextContent("All") + expect(screen.getByRole("combobox", { name: "Container" })).toHaveTextContent("All") + expect(screen.getByLabelText("LogsQL filter")).toHaveValue("") + }) + it("keeps the current lines while a new filter in the same tenant loads", async () => { const { client } = clientWith((endpoint, params) => endpoint.endsWith("query") && params.get("query") === "slow" ? new Promise(() => {}) : defaultHandler(endpoint, params), diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index c2dc8ca5c..8c05a8743 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -81,9 +81,14 @@ function placeholderFor({ follow, stopped, loading, failed, longestRange, retry return } +// Filter, pod and container belong to one scope; a new tenant or application starts clean. export function LogsView({ scope }: { scope: ObservabilityScope }) { - const client = useK8sClient() const root = logsRoot(scope) + return +} + +function ScopedLogsView({ root }: { root: string }) { + const client = useK8sClient() const [range, setRange] = useState("1h") const [draft, setDraft] = useState("") const [filter, setFilter] = useState("") From d5559bb934c5c264eff0101fa30a1387e1144692 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 5 Nov 2026 07:56:39 +0000 Subject: [PATCH 139/518] test(backups): remove the Altinity strategy tests' scratch directories Every test in hack/altinity-backup-strategy.bats made a directory with mktemp -d and never removed it, so each passing run of the file left nine directories in TMPDIR, and they pile up on any machine that keeps it between runs. Each test now removes its directory as its last statement. A failed assertion aborts the body before that line, so a failing test keeps its stub log and rendered script for inspection. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- hack/altinity-backup-strategy.bats | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats index 234ef0b8b..0861da42d 100644 --- a/hack/altinity-backup-strategy.bats +++ b/hack/altinity-backup-strategy.bats @@ -105,6 +105,7 @@ remote_rows() { echo "FAIL: render_script reported success with no chart" false fi + rm -rf "$tmp" } @test "restore reads back the archive the Backup recorded, not the newest one" { @@ -116,6 +117,7 @@ remote_rows() { PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" grep -qxF 'POST http://chi-clickhouse-myapp-clickhouse-0-0.tenant-x.svc:7171/backup/restore_remote/clickhouse-myapp-20260304T050607Z' "$tmp/curl.log" [ "$(grep -c '/backup/restore_remote/' "$tmp/curl.log")" -eq 1 ] + rm -rf "$tmp" } @test "restore fails naming the recorded archive when remote storage lacks it" { @@ -133,6 +135,7 @@ remote_rows() { echo "FAIL: restored another archive: $(cat "$tmp/curl.log")" false fi + rm -rf "$tmp" } @test "restore of a Backup without a recorded archive takes the newest one stamped by its takenAt, and warns" { @@ -150,6 +153,7 @@ remote_rows() { grep -qxF 'POST http://chi-clickhouse-myapp-clickhouse-0-0.tenant-x.svc:7171/backup/restore_remote/clickhouse-myapp-20260304T050607Z' "$tmp/curl.log" [ "$(grep -c '/backup/restore_remote/' "$tmp/curl.log")" -eq 1 ] printf '%s\n' "$out" | grep -qF 'WARNING: Backup bkp-a records no archive name' + rm -rf "$tmp" } @test "restore of a Backup without a recorded archive fails when no archive is stamped by its takenAt" { @@ -167,6 +171,7 @@ remote_rows() { echo "FAIL: restored an archive newer than takenAt: $(cat "$tmp/curl.log")" false fi + rm -rf "$tmp" } @test "backup writes the archive under the name the controller chose" { @@ -175,6 +180,7 @@ remote_rows() { render_script "$tmp/s.sh" backup myapp clickhouse-myapp-20260304T050607Z "" "" PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" grep -qxF 'POST http://chi-clickhouse-myapp-clickhouse-0-0.tenant-x.svc:7171/backup/create_remote?name=clickhouse-myapp-20260304T050607Z&resume=true' "$tmp/curl.log" + rm -rf "$tmp" } @test "a retried backup waits on the attempt the sidecar is still running" { @@ -189,6 +195,7 @@ remote_rows() { echo "FAIL: started another operation over the running one: $(cat "$tmp/curl.log")" false fi + rm -rf "$tmp" } @test "a retried backup after a failed attempt resumes the same archive" { @@ -198,6 +205,7 @@ remote_rows() { printf '{"command":"create_remote clickhouse-myapp-20260304T050607Z","status":"error","start":"2026-03-04 05:06:08","error":"S3 PutObject 403"}\n' > "$tmp/actions" PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" grep -qxF 'POST http://chi-clickhouse-myapp-clickhouse-0-0.tenant-x.svc:7171/backup/create_remote?name=clickhouse-myapp-20260304T050607Z&resume=true' "$tmp/curl.log" + rm -rf "$tmp" } @test "backup falls back to a timestamped name when the controller supplies none" { @@ -206,4 +214,5 @@ remote_rows() { render_script "$tmp/s.sh" backup myapp "" "" "" PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" grep -qE '^POST http://[^ ]*/backup/create_remote\?name=clickhouse-myapp-[0-9]{8}T[0-9]{6}Z&resume=true$' "$tmp/curl.log" + rm -rf "$tmp" } From d7ae34dfc3da4c7ef0725f6fc937354df98b64c8 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 5 Nov 2026 07:56:44 +0000 Subject: [PATCH 140/518] fix(backups): fail a ClickHouse BackupJob whose Backup name is taken A Backup outlives the BackupJob that made it, so a BackupJob created again under a name already used meets the earlier run's Backup when its own Job completes. That happens by hand, and from a Plan reconciled again inside its starting deadline, since a Plan names each BackupJob after its schedule slot. The Altinity driver took the earlier Backup as its result: the BackupJob turned Succeeded with a BackupRef recording the earlier run's archive, the archive this run wrote was recorded on nothing, and restoring what the BackupJob reported brought back the older data. An existing Backup is now taken as this run's only when it records this run's archive, which is what an earlier reconcile of the same run leaves when its status write is lost. Any other fails the BackupJob, naming both archives, and leaves the earlier Backup untouched. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- .../altinitystrategy_controller.go | 7 ++ .../altinitystrategy_controller_test.go | 114 ++++++++++++++++++ 2 files changed, 121 insertions(+) diff --git a/internal/backupcontroller/altinitystrategy_controller.go b/internal/backupcontroller/altinitystrategy_controller.go index f7f60bde8..a0868ec38 100644 --- a/internal/backupcontroller/altinitystrategy_controller.go +++ b/internal/backupcontroller/altinitystrategy_controller.go @@ -509,6 +509,13 @@ func (r *BackupJobReconciler) createAltinityBackupArtifact( if getErr := r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: backup.Name}, existing); getErr != nil { return nil, getErr } + // A Backup outlives its BackupJob, so a BackupJob re-created under a + // used name meets the earlier run's Backup here. Only one recording + // this run's archive was created by an earlier reconcile of this run. + if got := existing.Spec.DriverMetadata[altinityArchiveMetaKey]; got != archive { + return nil, fmt.Errorf("Backup %s/%s already exists and records archive %q, not %q that this run wrote", + backup.Namespace, backup.Name, got, archive) + } return existing, nil } return backup, nil diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index 7797842c0..e5f1e1695 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -3,6 +3,7 @@ package backupcontroller import ( "context" + "strings" "testing" "time" @@ -861,6 +862,119 @@ func TestReconcileAltinity_RecordsNoArchiveTheJobWasNotGiven(t *testing.T) { } } +// TestReconcileAltinity_BackupNameAlreadyTaken covers a completed Job whose +// Backup name is already in use. A Backup outlives its BackupJob, so a +// BackupJob re-created under an earlier run's name finds that run's Backup, +// and claiming it would point the new run at an archive it did not write. +// A Backup recording this run's own archive is the one an earlier reconcile +// created before its status write was lost, and is adopted. +func TestReconcileAltinity_BackupNameAlreadyTaken(t *testing.T) { + const thisRun = "clickhouse-ch-test-20260305T050607Z" + cases := []struct { + name string + existing string // archive the existing Backup records; empty for none + wantSucceeded bool + }{ + {name: "earlier run's Backup records another archive", existing: "clickhouse-ch-test-20260304T050607Z"}, + {name: "earlier run's Backup records no archive", existing: ""}, + {name: "this run's Backup from a reconcile whose status write was lost", existing: thisRun, wantSucceeded: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + app := newAltinityClickHouseApp("ch-test", "tenant-test") + strategy := newAltinityArchiveStrategy("clickhouse-strategy") + startedAt := metav1.NewTime(time.Date(2026, 3, 5, 5, 6, 7, 0, time.UTC)) + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + BackupClassName: "clickhouse-backup", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt, Phase: backupsv1alpha1.BackupJobPhaseRunning}, + } + driverMD := map[string]string{altinityParamPrefix + "bucketName": "ch-bucket"} + if tc.existing != "" { + driverMD[altinityArchiveMetaKey] = tc.existing + } + existing := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Name: backupJob.Name, Namespace: backupJob.Namespace}, + Spec: backupsv1alpha1.BackupSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + TakenAt: metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 22, 0, time.UTC)), + DriverMetadata: driverMD, + }, + } + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + Parameters: map[string]string{"bucketName": "ch-bucket"}, + } + r, _, _ := newAltinityTestEnv(t, app, + clientfake.NewClientBuilder().WithObjects(backupJob, strategy, existing)) + ctx := context.Background() + + if _, err := r.reconcileAltinity(ctx, backupJob, resolved); err != nil { + t.Fatalf("reconcileAltinity() creating the Job: %v", err) + } + job := &batchv1.Job{} + if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}, job); err != nil { + t.Fatalf("get backup Job: %v", err) + } + if got := job.Spec.Template.Spec.Containers[0].Args[1]; got != "--archive="+thisRun { + t.Fatalf("backup Pod was rendered with %q, want --archive=%s", got, thisRun) + } + job.Status.Conditions = []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}} + if err := r.Status().Update(ctx, job); err != nil { + t.Fatalf("mark backup Job complete: %v", err) + } + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), backupJob); err != nil { + t.Fatalf("refresh BackupJob: %v", err) + } + if _, err := r.reconcileAltinity(ctx, backupJob, resolved); err != nil { + t.Fatalf("reconcileAltinity() on the completed Job: %v", err) + } + + updated := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), updated); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if tc.wantSucceeded { + if updated.Status.Phase != backupsv1alpha1.BackupJobPhaseSucceeded { + t.Errorf("phase = %q (%s), want Succeeded", updated.Status.Phase, updated.Status.Message) + } + if updated.Status.BackupRef == nil || updated.Status.BackupRef.Name != existing.Name { + t.Errorf("BackupRef = %v, want %s", updated.Status.BackupRef, existing.Name) + } + } else { + if updated.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed { + t.Errorf("phase = %q, want Failed: the run would claim a Backup recording %q instead of %q", updated.Status.Phase, tc.existing, thisRun) + } + if updated.Status.BackupRef != nil { + t.Errorf("BackupRef = %v, want none", updated.Status.BackupRef) + } + if !strings.Contains(updated.Status.Message, thisRun) { + t.Errorf("failure message %q does not name the archive this run wrote", updated.Status.Message) + } + } + kept := &backupsv1alpha1.Backup{} + if err := r.Get(ctx, client.ObjectKeyFromObject(existing), kept); err != nil { + t.Fatalf("get existing Backup: %v", err) + } + if got := kept.Spec.DriverMetadata[altinityArchiveMetaKey]; got != tc.existing { + t.Errorf("existing Backup now records archive %q, want %q left in place", got, tc.existing) + } + }) + } +} + func TestReconcileAltinityRestore_RendersTheBackupsArchive(t *testing.T) { cases := []struct { name string From 32a466fb141b27833737fa6bc634f428376ad4ce Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 18:10:35 +0700 Subject: [PATCH 141/518] fix(dashboard): scope the logs and metrics tabs to the selected app The provider keeps a query's previous result while its key changes, so while the next application's GET is pending the detail page still holds the previous instance. The Logs and Metrics tabs took the application name from that instance, so their proxy root did not change and the previous application's lines and series stayed under the new heading until the GET completed. Take the name from the route instead, which is the identity the user selected, and cover the pending GET at the route level with the real page, router and production query defaults. Assisted-by: LLM Signed-off-by: ZverGuy --- ...plicationDetailPage.observability.test.tsx | 113 ++++++++++++++++++ .../routes/detail/ApplicationDetailPage.tsx | 4 +- .../console/src/routes/detail/LogsTab.tsx | 9 +- .../console/src/routes/detail/MetricsTab.tsx | 9 +- 4 files changed, 125 insertions(+), 10 deletions(-) create mode 100644 packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.observability.test.tsx diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.observability.test.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.observability.test.tsx new file mode 100644 index 000000000..e16bf1665 --- /dev/null +++ b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.observability.test.tsx @@ -0,0 +1,113 @@ +import { describe, it, expect, vi, beforeEach } from "vitest" +import { act, render, screen, waitFor } from "@testing-library/react" +import { createMemoryRouter, RouterProvider } from "react-router" +import { K8sProvider } from "@cozystack/k8s-client" +import { TenantContext, type TenantContextValue } from "../../lib/tenant-context.tsx" +import type { ChartData } from "../../lib/observability/metrics.ts" + +// Keep the consoles' and the chart's browser-only libraries out of jsdom; the +// Logs and Metrics tabs, the page and the providers stay real. +vi.mock("./VncTab.tsx", () => ({ VncTab: () => null })) +vi.mock("./SerialTab.tsx", () => ({ SerialTab: () => null })) +vi.mock("../../components/observability/TimeSeriesChart.tsx", () => ({ + TimeSeriesChart: ({ data }: { data: ChartData }) => ( +
{data.series.map((s) => s.label).join(",")}
+ ), +})) + +const { ApplicationDetailPage } = await import("./ApplicationDetailPage.tsx") + +const NS = "tenant-test" +const APPS = `/apis/apps.cozystack.io/v1alpha1/namespaces/${NS}/postgreses` + +const tenant: TenantContextValue = { + tenants: [], + selectedTenant: "test", + selectTenant: () => {}, + tenantNamespace: NS, + isLoading: false, + error: null, +} + +const definitions = { + items: [{ metadata: { name: "postgres" }, spec: { application: { plural: "postgreses", kind: "Postgres" } } }], +} + +const instance = (name: string) => ({ + apiVersion: "apps.cozystack.io/v1alpha1", + kind: "Postgres", + metadata: { name, namespace: NS }, +}) + +const logLine = (name: string) => + `${JSON.stringify({ _time: "2026-11-04T10:00:01Z", _msg: `${name} is ready`, _stream_id: "s", kubernetes_pod_name: `${name}-0`, kubernetes_container_name: "postgres" })}\n` + +const matrix = (name: string) => + Response.json({ status: "success", data: { resultType: "matrix", result: [{ metric: { pod: `${name}-0` }, values: [[1, "1"]] }] } }) + +// app-b's own GET never answers, so the page keeps the placeholder of app-a +// that the production query defaults hand it; its proxy reads do answer. +function handle(url: URL): Response | Promise { + const path = url.pathname + if (path.endsWith("/applicationdefinitions")) return Response.json(definitions) + if (path === `${APPS}/app-a`) return Response.json(instance("app-a")) + if (path === `${APPS}/app-b`) return new Promise(() => {}) + const app = path.match(/\/postgreses\/(app-[ab])\//)?.[1] + if (app && path.endsWith("/select/logsql/query")) return new Response(logLine(app)) + if (app && path.endsWith("/stream_field_values")) return Response.json({ values: [] }) + if (app && path.includes("/metrics/api/v1/query_range")) return matrix(app) + return Response.json({ items: [] }) +} + +const fetchMock = vi.fn(async (input: RequestInfo | URL) => handle(new URL(String(input), "http://x"))) + +function renderAt(path: string) { + const router = createMemoryRouter( + [{ path: "/console/:plural/:name/*", element: }], + { initialEntries: [path] }, + ) + // No queryClient: the provider's production defaults keep a query's previous + // result while its key changes, which is what exposes the stale instance. + render( + + + + + , + ) + return { go: (to: string) => act(() => router.navigate(to)) } +} + +const requested = (prefix: string) => + fetchMock.mock.calls.some(([input]) => new URL(String(input), "http://x").pathname.startsWith(prefix)) + +beforeEach(() => { + vi.stubGlobal("fetch", fetchMock) + fetchMock.mockClear() +}) + +describe("ApplicationDetailPage observability while the next application loads", () => { + it("shows the selected application's logs, not the previous one's", async () => { + const { go } = renderAt("/console/postgreses/app-a/logs") + expect(await screen.findByText("app-a is ready")).toBeInTheDocument() + + await go("/console/postgreses/app-b/logs") + + expect(screen.getByRole("heading", { name: "app-b" })).toBeInTheDocument() + expect(screen.queryByText("app-a is ready")).toBeNull() + await waitFor(() => expect(requested(`${APPS}/app-b/log/`)).toBe(true)) + expect(await screen.findByText("app-b is ready")).toBeInTheDocument() + }) + + it("shows the selected application's metrics, not the previous one's", async () => { + const { go } = renderAt("/console/postgreses/app-a/metrics") + await waitFor(() => expect(screen.getAllByTestId("chart")[0]).toHaveTextContent("app-a-0")) + + await go("/console/postgreses/app-b/metrics") + + expect(screen.getByRole("heading", { name: "app-b" })).toBeInTheDocument() + expect(screen.queryAllByText("app-a-0", { exact: false })).toEqual([]) + await waitFor(() => expect(requested(`${APPS}/app-b/metrics/`)).toBe(true)) + await waitFor(() => expect(screen.getAllByTestId("chart")[0]).toHaveTextContent("app-b-0")) + }) +}) diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx index ca499aac1..ab68f7c49 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/detail/ApplicationDetailPage.tsx @@ -273,11 +273,11 @@ export function ApplicationDetailPage() { path="metrics" element={ }> - + } /> - } /> + } /> diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/detail/LogsTab.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/detail/LogsTab.tsx index 4ac5f878d..64cbcc89e 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/detail/LogsTab.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/detail/LogsTab.tsx @@ -1,15 +1,16 @@ -import type { ApplicationDefinition, ApplicationInstance } from "@cozystack/types" +import type { ApplicationDefinition } from "@cozystack/types" import { useTenantContext } from "../../lib/tenant-context.tsx" import { LogsView } from "../../components/observability/LogsView.tsx" interface LogsTabProps { ad: ApplicationDefinition - instance: ApplicationInstance + // The route's name: while the next application loads, the page still holds the previous instance. + name: string } -export function LogsTab({ ad, instance }: LogsTabProps) { +export function LogsTab({ ad, name }: LogsTabProps) { const { tenantNamespace } = useTenantContext() const plural = ad.spec?.application.plural if (!tenantNamespace || !plural) return null - return + return } diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/detail/MetricsTab.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/detail/MetricsTab.tsx index a928e633d..992cfc89b 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/detail/MetricsTab.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/detail/MetricsTab.tsx @@ -1,15 +1,16 @@ -import type { ApplicationDefinition, ApplicationInstance } from "@cozystack/types" +import type { ApplicationDefinition } from "@cozystack/types" import { useTenantContext } from "../../lib/tenant-context.tsx" import { MetricsView } from "../../components/observability/MetricsView.tsx" interface MetricsTabProps { ad: ApplicationDefinition - instance: ApplicationInstance + // The route's name: while the next application loads, the page still holds the previous instance. + name: string } -export function MetricsTab({ ad, instance }: MetricsTabProps) { +export function MetricsTab({ ad, name }: MetricsTabProps) { const { tenantNamespace } = useTenantContext() const plural = ad.spec?.application.plural if (!tenantNamespace || !plural) return null - return + return } From c15a0d01b85b0f374c350c63218083d9a4ce0df2 Mon Sep 17 00:00:00 2001 From: Kirill Ilin Date: Thu, 5 Nov 2026 20:19:29 +0500 Subject: [PATCH 142/518] feat(operator): let the platform source trust a private registry CA The operator generates the cozystack-platform OCIRepository from its own arguments and rewrites it on every start, and cozystack-packages copies that spec. A registry behind a private CA therefore had no durable way to be trusted: a certSecretRef patched onto the live object is dropped on the next operator restart, after which source-controller fails TLS and platform updates stop arriving. --platform-source-cert-secret sets spec.certSecretRef from the operator itself, so the CA (or a client certificate) survives the rewrite. Git sources are refused instead of ignored, because GitRepository has no certSecretRef and Flux reads a Git CA from the source secret instead. Assisted-by: LLM Signed-off-by: Kirill Ilin --- cmd/cozystack-operator/main.go | 21 +++++-- cmd/cozystack-operator/main_test.go | 95 +++++++++++++++++++++++++++-- 2 files changed, 107 insertions(+), 9 deletions(-) diff --git a/cmd/cozystack-operator/main.go b/cmd/cozystack-operator/main.go index 07e3ca292..1942237ac 100644 --- a/cmd/cozystack-operator/main.go +++ b/cmd/cozystack-operator/main.go @@ -99,6 +99,7 @@ func main() { var platformSourceName string var platformSourceRef string var platformSourceSecret string + var platformSourceCertSecret string var systemNamespaceMemoryLimit string var systemNamespaceMemoryRequest string @@ -145,6 +146,7 @@ func main() { flag.StringVar(&platformSourceName, "platform-source-name", "cozystack-platform", "Name for the generated platform source resource and PackageSource") flag.StringVar(&platformSourceRef, "platform-source-ref", "", "Reference specification as key=value pairs (e.g., 'branch=main' or 'digest=sha256:...,tag=v1.0'). For OCI: digest, semver, semverFilter, tag. For Git: branch, tag, semver, name, commit.") flag.StringVar(&platformSourceSecret, "platform-source-secret", "", "Name of the Secret in cozy-system namespace containing credentials for the platform source. Sets spec.secretRef on the generated OCIRepository or GitRepository. Secret type depends on the source kind: kubernetes.io/dockerconfigjson for OCI (oci://...); Opaque with username+password (or bearerToken) for Git over HTTPS; Opaque with identity (PEM private key) and known_hosts for Git over SSH.") + flag.StringVar(&platformSourceCertSecret, "platform-source-cert-secret", "", "Name of the Secret in cozy-system namespace holding TLS material for an OCI platform source: ca.crt for a registry behind a private CA, and/or tls.crt+tls.key for client certificate auth. Sets spec.certSecretRef on the generated OCIRepository. Not supported for Git sources, where Flux reads ca.crt from the --platform-source-secret Secret.") flag.StringVar(&cozyValuesSecretName, "cozy-values-secret-name", "cozystack-values", "The name of the secret containing cluster-wide configuration values.") flag.StringVar(&cozyValuesSecretNamespace, "cozy-values-secret-namespace", "cozy-system", "The namespace of the secret containing cluster-wide configuration values.") flag.StringVar(&cozyValuesNamespaceSelector, "cozy-values-namespace-selector", "cozystack.io/system=true", "The label selector for namespaces where the cluster-wide configuration values must be replicated.") @@ -278,12 +280,12 @@ func main() { // Generate and install platform source resource if specified if platformSourceURL != "" { - setupLog.Info("Generating platform source resource", "url", platformSourceURL, "name", platformSourceName, "ref", platformSourceRef, "secret", platformSourceSecret) + setupLog.Info("Generating platform source resource", "url", platformSourceURL, "name", platformSourceName, "ref", platformSourceRef, "secret", platformSourceSecret, "certSecret", platformSourceCertSecret) installCtx, installCancel := context.WithTimeout(mgrCtx, 2*time.Minute) defer installCancel() // Use direct client for pre-start operations (cache is not ready yet) - if err := installPlatformSourceResource(installCtx, directClient, platformSourceURL, platformSourceName, platformSourceRef, platformSourceSecret); err != nil { + if err := installPlatformSourceResource(installCtx, directClient, platformSourceURL, platformSourceName, platformSourceRef, platformSourceSecret, platformSourceCertSecret); err != nil { setupLog.Error(err, "failed to install platform source resource") os.Exit(1) } else { @@ -414,7 +416,7 @@ func main() { // installPlatformSourceResource generates and installs a Flux source resource (OCIRepository or GitRepository) // based on the platform source URL -func installPlatformSourceResource(ctx context.Context, k8sClient client.Client, sourceURL, resourceName, refSpec, secretName string) error { +func installPlatformSourceResource(ctx context.Context, k8sClient client.Client, sourceURL, resourceName, refSpec, secretName, certSecretName string) error { logger := log.FromContext(ctx) // Parse the source URL to determine type @@ -432,11 +434,14 @@ func installPlatformSourceResource(ctx context.Context, k8sClient client.Client, var obj client.Object switch sourceType { case "oci": - obj, err = generateOCIRepository(resourceName, repoURL, refMap, secretName) + obj, err = generateOCIRepository(resourceName, repoURL, refMap, secretName, certSecretName) if err != nil { return fmt.Errorf("failed to generate OCIRepository: %w", err) } case "git": + if certSecretName != "" { + return fmt.Errorf("--platform-source-cert-secret is only supported for OCI sources; for a Git source put ca.crt into the --platform-source-secret Secret") + } obj, err = generateGitRepository(resourceName, repoURL, refMap, secretName) if err != nil { return fmt.Errorf("failed to generate GitRepository: %w", err) @@ -650,7 +655,7 @@ func validateGitRef(refMap map[string]string) error { } // generateOCIRepository creates an OCIRepository resource -func generateOCIRepository(name, repoURL string, refMap map[string]string, secretName string) (*sourcev1.OCIRepository, error) { +func generateOCIRepository(name, repoURL string, refMap map[string]string, secretName, certSecretName string) (*sourcev1.OCIRepository, error) { if err := validateOCIRef(refMap); err != nil { return nil, err } @@ -687,6 +692,12 @@ func generateOCIRepository(name, repoURL string, refMap map[string]string, secre } } + if certSecretName != "" { + obj.Spec.CertSecretRef = &fluxmeta.LocalObjectReference{ + Name: certSecretName, + } + } + return obj, nil } diff --git a/cmd/cozystack-operator/main_test.go b/cmd/cozystack-operator/main_test.go index 6509065f9..3b38ace1a 100644 --- a/cmd/cozystack-operator/main_test.go +++ b/cmd/cozystack-operator/main_test.go @@ -23,6 +23,7 @@ import ( cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1" "github.com/cozystack/cozystack/internal/marketplace/tapconst" + sourcev1 "github.com/fluxcd/source-controller/api/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -534,7 +535,7 @@ func TestValidateGitRef(t *testing.T) { func TestGenerateOCIRepository(t *testing.T) { refMap := map[string]string{"tag": "v1.0", "digest": "sha256:abc123"} - obj, err := generateOCIRepository("my-repo", "oci://registry.example.com/repo", refMap, "") + obj, err := generateOCIRepository("my-repo", "oci://registry.example.com/repo", refMap, "", "") if err != nil { t.Fatalf("unexpected error: %v", err) } @@ -560,10 +561,13 @@ func TestGenerateOCIRepository(t *testing.T) { if obj.Spec.SecretRef != nil { t.Errorf("expected SecretRef to be nil when secretName is empty, got %+v", obj.Spec.SecretRef) } + if obj.Spec.CertSecretRef != nil { + t.Errorf("expected CertSecretRef to be nil when certSecretName is empty, got %+v", obj.Spec.CertSecretRef) + } } func TestGenerateOCIRepository_NoRef(t *testing.T) { - obj, err := generateOCIRepository("my-repo", "oci://registry.example.com/repo", map[string]string{}, "") + obj, err := generateOCIRepository("my-repo", "oci://registry.example.com/repo", map[string]string{}, "", "") if err != nil { t.Fatalf("unexpected error: %v", err) } @@ -573,14 +577,14 @@ func TestGenerateOCIRepository_NoRef(t *testing.T) { } func TestGenerateOCIRepository_InvalidRef(t *testing.T) { - _, err := generateOCIRepository("my-repo", "oci://registry.example.com/repo", map[string]string{"branch": "main"}, "") + _, err := generateOCIRepository("my-repo", "oci://registry.example.com/repo", map[string]string{"branch": "main"}, "", "") if err == nil { t.Fatal("expected error for invalid OCI ref key, got nil") } } func TestGenerateOCIRepository_WithSecret(t *testing.T) { - obj, err := generateOCIRepository("my-repo", "oci://registry.example.com/repo", map[string]string{"tag": "v1.0"}, "my-registry-creds") + obj, err := generateOCIRepository("my-repo", "oci://registry.example.com/repo", map[string]string{"tag": "v1.0"}, "my-registry-creds", "") if err != nil { t.Fatalf("unexpected error: %v", err) } @@ -592,6 +596,89 @@ func TestGenerateOCIRepository_WithSecret(t *testing.T) { } } +func TestGenerateOCIRepository_WithCertSecret(t *testing.T) { + obj, err := generateOCIRepository("my-repo", "oci://registry.example.com/repo", map[string]string{"tag": "v1.0"}, "", "registry-ca") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if obj.Spec.CertSecretRef == nil { + t.Fatal("expected CertSecretRef to be set when certSecretName is non-empty") + } + if obj.Spec.CertSecretRef.Name != "registry-ca" { + t.Errorf("expected CertSecretRef.Name %q, got %q", "registry-ca", obj.Spec.CertSecretRef.Name) + } + if obj.Spec.SecretRef != nil { + t.Errorf("expected SecretRef to stay nil when only certSecretName is set, got %+v", obj.Spec.SecretRef) + } +} + +// The operator rewrites the whole platform OCIRepository on every start, so a +// certSecretRef patched onto the live object by hand is lost on the next +// restart and source-controller then fails TLS against a registry behind a +// private CA. The CA has to come from the operator's own arguments and survive +// that update path. +func TestInstallPlatformSourceResource_UpdateKeepsCertSecret(t *testing.T) { + scheme := newTestScheme() + if err := sourcev1.AddToScheme(scheme); err != nil { + t.Fatalf("add source scheme: %v", err) + } + existing := &sourcev1.OCIRepository{ + ObjectMeta: metav1.ObjectMeta{Name: "cozystack-platform", Namespace: "cozy-system"}, + Spec: sourcev1.OCIRepositorySpec{URL: "oci://ghcr.io/cozystack/cozystack/cozystack-packages"}, + } + k8sClient := fake.NewClientBuilder().WithScheme(scheme).WithObjects(existing).Build() + + for range 2 { + if err := installPlatformSourceResource(context.Background(), k8sClient, + "oci://registry.internal:8083/cozystack/cozystack/cozystack-packages", "cozystack-platform", + "digest=sha256:abc123", "registry-creds", "registry-ca"); err != nil { + t.Fatalf("installPlatformSourceResource: %v", err) + } + } + + got := &sourcev1.OCIRepository{} + if err := k8sClient.Get(context.Background(), client.ObjectKey{Namespace: "cozy-system", Name: "cozystack-platform"}, got); err != nil { + t.Fatalf("get OCIRepository: %v", err) + } + if got.Spec.CertSecretRef == nil || got.Spec.CertSecretRef.Name != "registry-ca" { + t.Errorf("expected CertSecretRef registry-ca after update, got %+v", got.Spec.CertSecretRef) + } + if got.Spec.SecretRef == nil || got.Spec.SecretRef.Name != "registry-creds" { + t.Errorf("expected SecretRef registry-creds after update, got %+v", got.Spec.SecretRef) + } + if got.Spec.URL != "oci://registry.internal:8083/cozystack/cozystack/cozystack-packages" { + t.Errorf("expected URL to be repointed, got %q", got.Spec.URL) + } +} + +// GitRepository has no certSecretRef: Flux reads a Git CA from the ca.crt key +// of the secretRef Secret. Accepting the flag for a Git source would silently +// drop the CA, so it must fail before anything is written. +func TestInstallPlatformSourceResource_GitRejectsCertSecret(t *testing.T) { + scheme := newTestScheme() + if err := sourcev1.AddToScheme(scheme); err != nil { + t.Fatalf("add source scheme: %v", err) + } + k8sClient := fake.NewClientBuilder().WithScheme(scheme).Build() + + err := installPlatformSourceResource(context.Background(), k8sClient, + "https://git.internal/cozystack/cozystack", "cozystack-platform", "branch=main", "git-creds", "git-ca") + if err == nil { + t.Fatal("expected an error for a cert secret on a Git platform source, got nil") + } + if !strings.Contains(err.Error(), "ca.crt") { + t.Errorf("expected the error to point at the ca.crt key of the source secret, got: %v", err) + } + + list := &sourcev1.GitRepositoryList{} + if err := k8sClient.List(context.Background(), list); err != nil { + t.Fatalf("list GitRepositories: %v", err) + } + if len(list.Items) != 0 { + t.Errorf("expected no GitRepository to be written, got %d", len(list.Items)) + } +} + func TestGenerateGitRepository(t *testing.T) { refMap := map[string]string{"branch": "main", "commit": "abc1234def5678"} obj, err := generateGitRepository("my-repo", "https://github.com/user/repo", refMap, "") From 7f1227e3bdda5fbbc39059e5e73fa30db72ffea8 Mon Sep 17 00:00:00 2001 From: Kirill Ilin Date: Thu, 5 Nov 2026 20:19:35 +0500 Subject: [PATCH 143/518] feat(installer): expose platformSourceCertSecret Installing from a registry behind a private CA needs the operator's new --platform-source-cert-secret flag; without a chart value it can only be reached by editing the Deployment, which the next helm upgrade reverts. Assisted-by: LLM Signed-off-by: Kirill Ilin --- .../templates/cozystack-operator.yaml | 3 ++ .../installer/tests/platform_source_test.yaml | 33 +++++++++++++++++++ packages/core/installer/values.yaml | 9 +++++ 3 files changed, 45 insertions(+) create mode 100644 packages/core/installer/tests/platform_source_test.yaml diff --git a/packages/core/installer/templates/cozystack-operator.yaml b/packages/core/installer/templates/cozystack-operator.yaml index 6f2cc6880..ef2685926 100644 --- a/packages/core/installer/templates/cozystack-operator.yaml +++ b/packages/core/installer/templates/cozystack-operator.yaml @@ -200,6 +200,9 @@ spec: {{- if .Values.cozystackOperator.platformSourceSecret }} - --platform-source-secret={{ .Values.cozystackOperator.platformSourceSecret }} {{- end }} + {{- if .Values.cozystackOperator.platformSourceCertSecret }} + - --platform-source-cert-secret={{ .Values.cozystackOperator.platformSourceCertSecret }} + {{- end }} {{- $hasEndpointEnv := or (eq .Values.cozystackOperator.variant "talos") (eq .Values.cozystackOperator.variant "generic") }} {{- if or .Values.cozystackOperator.platformVersion $hasEndpointEnv }} env: diff --git a/packages/core/installer/tests/platform_source_test.yaml b/packages/core/installer/tests/platform_source_test.yaml new file mode 100644 index 000000000..5d2db4033 --- /dev/null +++ b/packages/core/installer/tests/platform_source_test.yaml @@ -0,0 +1,33 @@ +suite: cozystack-operator platform source arguments + +templates: + - templates/cozystack-operator.yaml + +tests: + - it: passes platformSourceCertSecret to the operator + set: + cozystackOperator.platformSourceUrl: oci://registry.internal:8083/cozystack/cozystack/cozystack-packages + cozystackOperator.platformSourceSecret: registry-creds + cozystackOperator.platformSourceCertSecret: registry-ca + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --platform-source-cert-secret=registry-ca + documentSelector: + path: kind + value: Deployment + - contains: + path: spec.template.spec.containers[0].args + content: --platform-source-secret=registry-creds + documentSelector: + path: kind + value: Deployment + + - it: omits the cert secret flag by default + asserts: + - notContains: + path: spec.template.spec.containers[0].args + content: --platform-source-cert-secret= + documentSelector: + path: kind + value: Deployment diff --git a/packages/core/installer/values.yaml b/packages/core/installer/values.yaml index 43639ab5c..6224b5955 100644 --- a/packages/core/installer/values.yaml +++ b/packages/core/installer/values.yaml @@ -63,6 +63,15 @@ cozystackOperator: # otherwise source-controller will report a failed condition until it is created. # See FluxCD source-controller docs for the exact key layout per scheme. platformSourceSecret: '' + # Name of the Secret in cozy-system namespace with TLS material for an OCI + # platform source: `ca.crt` for a registry whose certificate is issued by a + # private CA, and/or `tls.crt`+`tls.key` for client certificate auth (type + # Opaque or kubernetes.io/tls). When non-empty, sets spec.certSecretRef on the + # generated OCIRepository; the operator rewrites that object on every start, + # so a certSecretRef patched onto it by hand does not survive a restart. + # The cozystack-packages source copies the spec, so it inherits the CA too. + # OCI only: for a Git source put `ca.crt` into the platformSourceSecret Secret. + platformSourceCertSecret: '' # Generic variant configuration (only used when cozystackOperator.variant=generic) cozystack: # Kubernetes API server host (IP only, no protocol/port) From a70a1cf83b5d94f18c7e4854a57bcb22dc4edc1c Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:03:43 +0300 Subject: [PATCH 144/518] fix(seaweedfs): retry the data-plane release instead of rolling it back The admin volume and the two lifecycle pods are new demands on tenant quota, so an upgrade into a namespace with no headroom can wait out its timeout. Flux's default remediation then rolls the release back, and with unlimited retries it does so on every attempt: each rollback drops the [grpc.s3] section and the s3 restart annotation, each retry brings them back, and the s3 pods restart twice per cycle until quota frees up. RetryOnFailure keeps what was applied and retries it, as the parent release that the apiserver builds for the application already does. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../extra/seaweedfs/templates/seaweedfs.yaml | 11 ++++++++++ .../extra/seaweedfs/tests/lifecycle_test.yaml | 22 +++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/packages/extra/seaweedfs/templates/seaweedfs.yaml b/packages/extra/seaweedfs/templates/seaweedfs.yaml index c111a80c5..a2cf02c5b 100644 --- a/packages/extra/seaweedfs/templates/seaweedfs.yaml +++ b/packages/extra/seaweedfs/templates/seaweedfs.yaml @@ -214,10 +214,21 @@ spec: namespace: cozy-system interval: 30s timeout: 10m + {{- /* Flux remediates a failed install by uninstalling and a failed upgrade + by rolling back, and with retries: -1 it does so on every attempt. A + release waiting on tenant quota for the admin volume would then roll + the s3 pods back and forth with the [grpc.s3] restart annotation. + RetryOnFailure keeps what was applied and retries it. */}} install: + strategy: + name: RetryOnFailure + retryInterval: 30s remediation: retries: -1 upgrade: + strategy: + name: RetryOnFailure + retryInterval: 30s remediation: retries: -1 valuesFrom: diff --git a/packages/extra/seaweedfs/tests/lifecycle_test.yaml b/packages/extra/seaweedfs/tests/lifecycle_test.yaml index a040e0ae6..07aeb573e 100644 --- a/packages/extra/seaweedfs/tests/lifecycle_test.yaml +++ b/packages/extra/seaweedfs/tests/lifecycle_test.yaml @@ -194,3 +194,25 @@ tests: asserts: - hasDocuments: count: 0 + + # The admin PVC and the two lifecycle pods can leave an upgrade waiting on a + # full tenant quota. Under Flux's default rollback remediation each failed + # attempt would roll back the [grpc.s3] section and its restart annotation + # and the next one would bring them back, restarting the s3 pods on every + # cycle. Retrying in place keeps what was applied while the release waits. + - it: retries the data-plane release in place instead of rolling it back + template: templates/seaweedfs.yaml + set: *platform + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - equal: + path: spec.install.strategy.name + value: RetryOnFailure + - equal: + path: spec.upgrade.strategy.name + value: RetryOnFailure + - equal: + path: spec.upgrade.remediation.retries + value: -1 From c3d1a73d8c6790eaf3de10d6c59f657fefde55bc Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:04:01 +0300 Subject: [PATCH 145/518] build(seaweedfs): stop make update when the [grpc.s3] backport lands The backport becomes redundant once the vendored chart reaches 4.46. Re-applied to a chart that already carries the section, BSD patch takes it as reversed, removes the section and exits 0, so a re-vendor would quietly return the s3 gRPC listener to plaintext. --forward refuses an already-applied patch with a non-zero exit, which stops the update at the point where the patch should be dropped. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- packages/system/seaweedfs/Makefile | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/packages/system/seaweedfs/Makefile b/packages/system/seaweedfs/Makefile index 2f7790b38..717ae0d49 100644 --- a/packages/system/seaweedfs/Makefile +++ b/packages/system/seaweedfs/Makefile @@ -61,7 +61,11 @@ update: patch --no-backup-if-mismatch -p4 < patches/s3-service-name.patch patch --no-backup-if-mismatch -p4 < patches/s3-service-name-consumers.patch patch --no-backup-if-mismatch -p4 < patches/cluster-scoped-names-per-namespace.patch - patch --no-backup-if-mismatch -p4 < patches/s3-grpc-tls.patch + # Backport of seaweedfs/seaweedfs#11157, released in 4.46: drop the patch + # once the vendored chart carries [grpc.s3]. --forward makes that point + # fail here, where BSD patch would otherwise reverse it and remove the + # section without an error. + patch --forward --no-backup-if-mismatch -p4 < patches/s3-grpc-tls.patch #patch --no-backup-if-mismatch -p4 < patches/retention-policy-delete.yaml $(MAKE) update-image From 0ad642e68df217c26e1abfe333adc9edb730388a Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:05:31 +0300 Subject: [PATCH 146/518] test(clickhouse): size the system-bucket round-trip server for SQL The round-trip ClickHouse ran on the nano preset, whose 128Mi limit the clickhouse-1 contract suite never exceeds because it sends no query. This suite writes a sentinel row before the backup, and the server was OOM-killed on that first statement (lastState OOMKilled, exit 137), so the Test failed before it reached the behaviour under test. clickhouse-2 runs the same statements on the small preset. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/e2e-chainsaw/clickhouse/clickhouse-system-bucket.yaml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/hack/e2e-chainsaw/clickhouse/clickhouse-system-bucket.yaml b/hack/e2e-chainsaw/clickhouse/clickhouse-system-bucket.yaml index f7327b46d..0ed1dd4a4 100644 --- a/hack/e2e-chainsaw/clickhouse/clickhouse-system-bucket.yaml +++ b/hack/e2e-chainsaw/clickhouse/clickhouse-system-bucket.yaml @@ -13,7 +13,9 @@ spec: logStorageSize: 1Gi shards: 1 replicas: 1 - resourcesPreset: nano + # The round-trip runs SQL against this server; at nano's 128Mi the first + # query OOM-kills it, so it takes the preset clickhouse-2 runs SQL on. + resourcesPreset: small resources: {} backup: enabled: true From 003bc3ed0bf1c725c0f1587881f6bf59c41ec398 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:06:53 +0300 Subject: [PATCH 147/518] test(clickhouse): fail the round-trip when CoreDNS cannot be restored The round-trip puts the host CoreDNS Corefile back from its EXIT trap, which runs with errexit off. A refused kubectl replace, for instance a resourceVersion conflict with another writer, fell through to the success message and returned 0, so the run stayed green while the rewrite remained in a cluster-wide object and the next run refused to start. The restore now returns the failure. The restore moves into a file of its own so a unit test can run it against a stubbed kubectl and so other callers can share it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- ...clickhouse-system-bucket-shared-state.bats | 75 +++++++++++++++++++ .../clickhouse/system-bucket-roundtrip.sh | 37 +-------- .../clickhouse/system-bucket-shared-state.sh | 53 +++++++++++++ 3 files changed, 132 insertions(+), 33 deletions(-) create mode 100644 hack/clickhouse-system-bucket-shared-state.bats create mode 100644 hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh diff --git a/hack/clickhouse-system-bucket-shared-state.bats b/hack/clickhouse-system-bucket-shared-state.bats new file mode 100644 index 000000000..cc0bbf633 --- /dev/null +++ b/hack/clickhouse-system-bucket-shared-state.bats @@ -0,0 +1,75 @@ +#!/usr/bin/env bats +# The clickhouse-3-system-bucket-roundtrip Test edits the host CoreDNS Corefile +# and must put it back. Its restore runs from an EXIT trap under `set +e`, so a +# restore that fails without saying so leaves the rewrite in a cluster-wide +# object and still reports a green run; the next run then refuses to start. +# These run the restore against a stubbed kubectl. + +load test_helper + +LIB=hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh +HELPERS=examples/backups/clickhouse/00-helpers.sh + +setup() { + strict_setup + STUB=$(mktemp -d) + # A Corefile carrying the Test's marked block, as the run step leaves it. + jq -n '{apiVersion: "v1", kind: "ConfigMap", + metadata: {name: "coredns", namespace: "kube-system", resourceVersion: "7"}, + data: {Corefile: ".:53 {\n # === BEGIN e2e-clickhouse-system-bucket ===\n rewrite name exact s3.example.org root-ingress-controller.tenant-root.svc.cozy.local\n # === END e2e-clickhouse-system-bucket ===\n errors\n}\n"}}' \ + >"$STUB/coredns.json" + cat >"$STUB/kubectl" <<'EOF' +#!/bin/bash +case "$*" in + *"get configmap coredns -o json"*) cat "$STUB/coredns.json" ;; + replace*) + cp "${@: -1}" "$STUB/replaced.json" + if [ -n "${REPLACE_FAILS:-}" ]; then + echo 'Error from server (Conflict): Operation cannot be fulfilled on configmaps "coredns": the object has been modified' >&2 + exit 1 + fi + ;; + *"get secret"*) ;; + *) echo "unexpected kubectl $*" >&2; exit 2 ;; +esac +EOF + chmod +x "$STUB/kubectl" + export STUB +} + +restore() { + PATH="$STUB:$PATH" NAMESPACE=tenant-root bash -c ' + set -euo pipefail + source "$1" + source "$2" + set +e + restore_shared_state + ' _ "$HELPERS" "$LIB" +} + +@test "a CoreDNS restore the API server refuses fails the restore" { + export REPLACE_FAILS=1 + err=$(mktemp) + if restore 2>"$err"; then + echo "restore_shared_state returned 0 although kubectl replace was refused" >&2 + cat "$err" >&2 + false + fi + if grep -q 'Removed the s3.example.org rewrite' "$err"; then + echo "restore_shared_state reported the rewrite removed although kubectl replace was refused:" >&2 + cat "$err" >&2 + false + fi +} + +@test "a CoreDNS restore sends the Corefile back without the marked block" { + restore + jq -e '.data.Corefile == ".:53 {\n errors\n}\n"' "$STUB/replaced.json" >/dev/null || { + echo "restored Corefile:" >&2 + jq -r '.data.Corefile' "$STUB/replaced.json" >&2 + false + } + # The resourceVersion read travels with the replace, so a concurrent + # writer makes it fail rather than be overwritten. + [ "$(jq -r '.metadata.resourceVersion' "$STUB/replaced.json")" = 7 ] +} diff --git a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh index fe1361418..88a3cf2b5 100755 --- a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh +++ b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh @@ -24,18 +24,14 @@ set -euo pipefail source examples/backups/clickhouse/00-helpers.sh +source hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh : "${NAMESPACE:?}" CLICKHOUSE_NAME=sysbucket BACKUPJOB_NAME=sysbucket-backup RESTOREJOB_NAME=sysbucket-restore -S3_HOST=s3.example.org -INGRESS_SECRET=seaweedfs-s3-ingress-tls CERT_SECRET=e2e-system-bucket-s3-tls PROBE_POD=e2e-system-bucket-probe -OWNER_LABEL=cozystack.io/e2e-owner=clickhouse-system-bucket -DNS_BEGIN=' # === BEGIN e2e-clickhouse-system-bucket ===' -DNS_END=' # === END e2e-clickhouse-system-bucket ===' DNS_TARGET="root-ingress-controller.${NAMESPACE}.svc.cozy.local" # Kubelet refreshes a ConfigMap volume within about a minute and CoreDNS's # reload plugin polls every 30s; the rest is the 30s answer cache. @@ -46,35 +42,10 @@ JOB_TIMEOUT=480 workdir=$(mktemp -d) -corefile_has_block() { - kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -qF -- "$DNS_BEGIN" -} - -remove_dns_block() { - kubectl -n kube-system get configmap coredns -o json >"$workdir/coredns.json" - local begins - begins=$(jq -r '.data.Corefile' "$workdir/coredns.json" | grep -cF -- "$DNS_BEGIN" || true) - if [ "$begins" = 0 ]; then - return 0 - fi - if [ "$begins" != 1 ] || ! jq -r '.data.Corefile' "$workdir/coredns.json" | grep -qF -- "$DNS_END"; then - log_error "kube-system/coredns holds ${begins} '${DNS_BEGIN# *}' markers or no END marker; reconcile the Corefile by hand" - return 1 - fi - jq --arg b "$DNS_BEGIN" --arg e "$DNS_END" \ - '.data.Corefile |= (split("\n") | (index($b)) as $i | (index($e)) as $j | .[:$i] + .[$j+1:] | join("\n"))' \ - "$workdir/coredns.json" >"$workdir/coredns-clean.json" - kubectl replace -f "$workdir/coredns-clean.json" >/dev/null - log_success "Removed the ${S3_HOST} rewrite from kube-system/coredns" -} - -restore_shared_state() { +on_exit() { local rc=$? set +e - remove_dns_block || rc=1 - if kubectl -n "$NAMESPACE" get secret "$INGRESS_SECRET" -l "$OWNER_LABEL" -o name 2>/dev/null | grep -q .; then - kubectl -n "$NAMESPACE" delete secret "$INGRESS_SECRET" --wait=true --timeout=60s || rc=1 - fi + restore_shared_state || rc=1 rm -rf "$workdir" exit "$rc" } @@ -87,7 +58,7 @@ if corefile_has_block; then log_error "kube-system/coredns already carries the ${S3_HOST} rewrite block, left by an earlier run; remove it by hand" exit 1 fi -trap restore_shared_state EXIT +trap on_exit EXIT log_step "Serving the Test certificate for ${S3_HOST} through the platform S3 Ingress" kubectl -n "$NAMESPACE" get secret "$CERT_SECRET" -o json \ diff --git a/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh b/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh new file mode 100644 index 000000000..f121eaeee --- /dev/null +++ b/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# The two shared objects system-bucket-roundtrip.sh changes for the duration +# of its run, and how to put them back. Sourced by that script, whose EXIT trap +# restores them. Needs NAMESPACE and log_success/log_error from +# examples/backups/clickhouse/00-helpers.sh. + +S3_HOST=s3.example.org +INGRESS_SECRET=seaweedfs-s3-ingress-tls +OWNER_LABEL=cozystack.io/e2e-owner=clickhouse-system-bucket +DNS_BEGIN=' # === BEGIN e2e-clickhouse-system-bucket ===' +DNS_END=' # === END e2e-clickhouse-system-bucket ===' + +corefile_has_block() { + kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' | grep -qF -- "$DNS_BEGIN" +} + +remove_dns_block() { + local dir + dir=$(mktemp -d) + kubectl -n kube-system get configmap coredns -o json >"$dir/coredns.json" || { rm -rf "$dir"; return 1; } + local begins + begins=$(jq -r '.data.Corefile' "$dir/coredns.json" | grep -cF -- "$DNS_BEGIN" || true) + if [ "$begins" = 0 ]; then + rm -rf "$dir" + return 0 + fi + if [ "$begins" != 1 ] || ! jq -r '.data.Corefile' "$dir/coredns.json" | grep -qF -- "$DNS_END"; then + log_error "kube-system/coredns holds ${begins} '${DNS_BEGIN# *}' markers or no END marker; reconcile the Corefile by hand" + rm -rf "$dir" + return 1 + fi + jq --arg b "$DNS_BEGIN" --arg e "$DNS_END" \ + '.data.Corefile |= (split("\n") | (index($b)) as $i | (index($e)) as $j | .[:$i] + .[$j+1:] | join("\n"))' \ + "$dir/coredns.json" >"$dir/coredns-clean.json" + # Callers run this with errexit off, so a refused replace has to return. + if ! kubectl replace -f "$dir/coredns-clean.json" >/dev/null; then + log_error "could not remove the ${S3_HOST} rewrite from kube-system/coredns; remove the marked block by hand" + rm -rf "$dir" + return 1 + fi + log_success "Removed the ${S3_HOST} rewrite from kube-system/coredns" + rm -rf "$dir" +} + +# Returns non-zero when either object could not be put back. +restore_shared_state() { + local rc=0 + remove_dns_block || rc=1 + if kubectl -n "$NAMESPACE" get secret "$INGRESS_SECRET" -l "$OWNER_LABEL" -o name 2>/dev/null | grep -q .; then + kubectl -n "$NAMESPACE" delete secret "$INGRESS_SECRET" --wait=true --timeout=60s || rc=1 + fi + return "$rc" +} From a66219899143042c5b6a9b8809d6892f98821a33 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:07:51 +0300 Subject: [PATCH 148/518] test(clickhouse): restore the round-trip's shared objects after a step ceiling The round-trip changes the host CoreDNS Corefile and the S3 Ingress TLS Secret and puts both back from its EXIT trap. When the run step's ceiling fires, Chainsaw kills the shell and the trap never runs, so the rewrite stays in a cluster-wide object and the next run refuses to start. The step's finally runs regardless, so it repeats the restore; after a run whose trap already restored them it finds nothing to undo. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../clickhouse-system-bucket-shared-state.bats | 18 ++++++++++++++++++ .../e2e-chainsaw/clickhouse/chainsaw-test.yaml | 15 +++++++++++++++ .../clickhouse/system-bucket-shared-state.sh | 5 +++-- 3 files changed, 36 insertions(+), 2 deletions(-) diff --git a/hack/clickhouse-system-bucket-shared-state.bats b/hack/clickhouse-system-bucket-shared-state.bats index cc0bbf633..2f7a79197 100644 --- a/hack/clickhouse-system-bucket-shared-state.bats +++ b/hack/clickhouse-system-bucket-shared-state.bats @@ -73,3 +73,21 @@ restore() { # writer makes it fail rather than be overwritten. [ "$(jq -r '.metadata.resourceVersion' "$STUB/replaced.json")" = 7 ] } + +@test "the run step's finally restores the shared objects" { + # When the step ceiling fires, Chainsaw kills the shell and the EXIT trap + # of system-bucket-roundtrip.sh never runs; the step's finally still does. + # Chainsaw runs a script's content with sh from the suite's directory. + suite=hack/e2e-chainsaw/clickhouse + finally=$(yq 'select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") + | .spec.steps[] | select(.name == "run") | .finally[] + | select(.description == "restore the host CoreDNS and the S3 Ingress TLS Secret") | .script.content' \ + "$suite/chainsaw-test.yaml") + [ -n "$finally" ] || { echo "the run step has no finally restoring the shared objects" >&2; false; } + (cd "$suite" && PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$finally") + [ -f "$STUB/replaced.json" ] || { + echo "the run step's finally did not put the Corefile back" >&2 + false + } + jq -e '.data.Corefile == ".:53 {\n errors\n}\n"' "$STUB/replaced.json" >/dev/null +} diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index d96013140..03130cfb3 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -539,6 +539,21 @@ spec: check: ($error == null): true finally: + # The script's EXIT trap puts the Corefile and the Ingress TLS Secret + # back, but a step ceiling kills the shell before the trap runs. A no-op + # when the trap already did it. + - description: restore the host CoreDNS and the S3 Ingress TLS Secret + script: + timeout: 2m + content: | + set -eu + cd ../../.. + # Chainsaw runs this under sh; the helpers are bash. + NAMESPACE="$NAMESPACE" bash -euo pipefail -c ' + source examples/backups/clickhouse/00-helpers.sh + source hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh + restore_shared_state + ' # Chainsaw did not apply these: the jobs come from the script and the # Backup from the controller, and deleting a BackupJob reclaims no Backup. # The Certificate's Secret outlives the Certificate Chainsaw deletes. diff --git a/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh b/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh index f121eaeee..8275e8c13 100644 --- a/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh +++ b/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh @@ -1,8 +1,9 @@ #!/bin/bash # The two shared objects system-bucket-roundtrip.sh changes for the duration # of its run, and how to put them back. Sourced by that script, whose EXIT trap -# restores them. Needs NAMESPACE and log_success/log_error from -# examples/backups/clickhouse/00-helpers.sh. +# restores them, and by the finally of its Chainsaw step, which restores them +# when the step ceiling killed the script before the trap could run. Needs +# NAMESPACE and log_success/log_error from examples/backups/clickhouse/00-helpers.sh. S3_HOST=s3.example.org INGRESS_SECRET=seaweedfs-s3-ingress-tls From cabe9a07e28d14089bccb0c072ec338122da23e2 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:08:50 +0300 Subject: [PATCH 149/518] test(clickhouse): reclaim the round-trip Backup by its name The Altinity driver names the Backup after the BackupJob and adopts one that already exists under that name. The round-trip's cleanup deleted the Backup only through the BackupJob's status.backupRef, so a Backup whose BackupJob never recorded it survived, and the next run's BackupJob would have adopted it in place of a fresh backup. Both the cleanup and the preconditions now delete it by name; the preconditions ceiling grows to cover the extra wait. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- ...clickhouse-system-bucket-shared-state.bats | 32 +++++++++++++++++++ .../clickhouse/chainsaw-test.yaml | 13 ++++---- 2 files changed, 39 insertions(+), 6 deletions(-) diff --git a/hack/clickhouse-system-bucket-shared-state.bats b/hack/clickhouse-system-bucket-shared-state.bats index 2f7a79197..e76cc0a1d 100644 --- a/hack/clickhouse-system-bucket-shared-state.bats +++ b/hack/clickhouse-system-bucket-shared-state.bats @@ -91,3 +91,35 @@ restore() { } jq -e '.data.Corefile == ".:53 {\n errors\n}\n"' "$STUB/replaced.json" >/dev/null } + +# The Altinity driver names the Backup after the BackupJob and adopts one that +# already exists under that name, so a Backup left behind would stand in for +# the next run's backup. It is reclaimed by name, whether or not the BackupJob +# recorded it in status.backupRef. +@test "the round-trip reclaims its Backup by name before and after the run" { + suite=hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml + test='select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[]' + calls="$STUB/calls" + # Nothing is left over from an earlier run, and no BackupJob recorded a + # backupRef. + cat >"$STUB/kubectl" <<'STUBEOF' +#!/bin/bash +echo "$*" >>"$STUB/calls" +case "$*" in + *" get secret "*) exit 1 ;; +esac +STUBEOF + for selector in \ + 'select(.name == "preconditions") | .try[].script.content' \ + 'select(.name == "run") | .finally[] | select(.description == "system-bucket round-trip cleanup") | .script.content'; do + : >"$calls" + script=$(yq "$test | $selector" "$suite") + [ -n "$script" ] || { echo "no script at: $selector" >&2; false; } + PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$script" + grep -qE '^-n tenant-root delete backup sysbucket-backup( |$)' "$calls" || { + echo "the script at '$selector' does not delete Backup sysbucket-backup by name; kubectl calls:" >&2 + cat "$calls" >&2 + false + } + done +} diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index 03130cfb3..c48e32f24 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -439,7 +439,7 @@ spec: - name: preconditions try: - script: - timeout: 2m + timeout: 3m content: | set -eu if kubectl -n "$NAMESPACE" get secret seaweedfs-s3-ingress-tls -o name >/dev/null 2>&1; then @@ -451,9 +451,11 @@ spec: exit 1 fi # Fixed names in a persistent namespace: a leftover Succeeded job - # would satisfy the waits in the run step without a new attempt. + # would satisfy the waits in the run step without a new attempt, and + # the driver adopts a leftover Backup of the BackupJob's name. kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s + kubectl -n "$NAMESPACE" delete backup sysbucket-backup --ignore-not-found --wait=true --timeout=60s kubectl -n "$NAMESPACE" delete secret cozy-backups-creds --ignore-not-found --wait=true --timeout=30s check: ($error == null): true @@ -562,10 +564,9 @@ spec: timeout: 3m content: | set -eu - backup=$(kubectl -n "$NAMESPACE" get backupjob sysbucket-backup --ignore-not-found -o jsonpath='{.status.backupRef.name}') kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s - if [ -n "$backup" ]; then - kubectl -n "$NAMESPACE" delete backup "$backup" --ignore-not-found --wait=true --timeout=60s - fi + # The driver names the Backup after the BackupJob; by name it is + # found even when status.backupRef was never written. + kubectl -n "$NAMESPACE" delete backup sysbucket-backup --ignore-not-found --wait=true --timeout=60s kubectl -n "$NAMESPACE" delete secret e2e-system-bucket-s3-tls --ignore-not-found --wait=true --timeout=30s From dee5e6d8bf935874ad7648a9e8f63ff19809d21a Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:08:59 +0300 Subject: [PATCH 150/518] test(clickhouse): mark the round-trip's reachability poll as a fixed wait The probe loop sleeps between attempts because no Kubernetes condition reports that the host CoreDNS serves the rewrite and the Ingress serves the Test certificate. The e2e conventions allow such a sleep only under the marker that records it is waiting for something to model it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh index 88a3cf2b5..dc94a9c85 100755 --- a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh +++ b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh @@ -103,6 +103,9 @@ while true; do kubectl -n "$NAMESPACE" describe ingress ingress-seaweedfs-s3 >&2 || true exit 1 fi + # TODO(e2e-replace-fixed-timeouts): no condition reports that the host + # CoreDNS answers with the rewrite and the Ingress serves the Test + # certificate, as seen from this namespace; poll until one does. sleep 10 done From 603114cd7c9954006e6b6cc92ea613cc7af1aab8 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:09:15 +0300 Subject: [PATCH 151/518] test(clickhouse): check the round-trip restored this run's sentinel row Every run inserted the same sentinel row and accepted any row after the restore, while the Altinity restore takes the newest remote archive of the release. A restore of an older run's archive therefore passed. The row now carries a per-run value and exactly that row has to come back. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../clickhouse/system-bucket-roundtrip.sh | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh index dc94a9c85..73d2f0e63 100755 --- a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh +++ b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh @@ -120,9 +120,12 @@ sidecar_before=$(sidecar_identity) [[ "$sidecar_before" =~ ^[0-9a-f-]+\ [0-9]+$ ]] \ || { log_error "could not read the ClickHouse Pod uid and clickhouse-backup restart count (got '${sidecar_before}')"; exit 1; } -log_step "Writing the sentinel row" +# Restore picks the newest remote backup of the release, so only a row no +# earlier run wrote shows that the archive restored is this run's. +sentinel="before-backup-$(date -u +%Y%m%dT%H%M%SZ)-${RANDOM}" +log_step "Writing the sentinel row ${sentinel}" clickhouse_query "$CLICKHOUSE_NAME" "CREATE TABLE IF NOT EXISTS default.sentinel (id UInt32, name String) ENGINE = MergeTree ORDER BY id" -clickhouse_query "$CLICKHOUSE_NAME" "INSERT INTO default.sentinel VALUES (1, 'before-backup')" +clickhouse_query "$CLICKHOUSE_NAME" "INSERT INTO default.sentinel VALUES (1, '${sentinel}')" log_step "Backing up through cozy-default" kubectl apply -f - <= 1 )) || { log_error "sentinel row count after restore is ${count}; expected >= 1"; exit 1; } +count=$(clickhouse_query "$CLICKHOUSE_NAME" "SELECT count() FROM default.sentinel WHERE name = '${sentinel}'" | tr -d '[:space:]') +[ "$count" = 1 ] || { log_error "after restore default.sentinel holds ${count} row(s) named ${sentinel}; expected 1"; exit 1; } sidecar_after=$(sidecar_identity) [ "$sidecar_after" = "$sidecar_before" ] \ || { log_error "the ClickHouse Pod or its clickhouse-backup container restarted during the round-trip (uid restarts: '${sidecar_before}' then '${sidecar_after}')"; exit 1; } -log_success "System-bucket round-trip verified: ${count} sentinel row(s) restored" +log_success "System-bucket round-trip verified: sentinel row ${sentinel} restored" From a6f4f557f35a04e8852e450b2ecf005605222a5e Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:10:29 +0300 Subject: [PATCH 152/518] fix(seaweedfs): let the dashboard read the lifecycle monitors The dashboard reads an application's WorkloadMonitors by name through its resource-map Role, so the admin and worker monitors the lifecycle switch adds were invisible to the tenant: the components ran, but the application page never listed them. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../templates/dashboard-resourcemap.yaml | 4 +++ .../extra/seaweedfs/tests/lifecycle_test.yaml | 34 +++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/packages/extra/seaweedfs/templates/dashboard-resourcemap.yaml b/packages/extra/seaweedfs/templates/dashboard-resourcemap.yaml index dfd408dad..eb038fe8f 100644 --- a/packages/extra/seaweedfs/templates/dashboard-resourcemap.yaml +++ b/packages/extra/seaweedfs/templates/dashboard-resourcemap.yaml @@ -27,6 +27,10 @@ rules: - {{ $.Release.Name }}-filer - {{ $.Release.Name }}-db - {{ $.Release.Name }}-s3 + {{- if .Values.lifecycle.enabled }} + - {{ $.Release.Name }}-admin + - {{ $.Release.Name }}-worker + {{- end }} {{- if eq .Values.topology "Simple" }} - {{ $.Release.Name }}-volume {{- range $poolName, $pool := .Values.volume.pools }} diff --git a/packages/extra/seaweedfs/tests/lifecycle_test.yaml b/packages/extra/seaweedfs/tests/lifecycle_test.yaml index 07aeb573e..e8d3c0842 100644 --- a/packages/extra/seaweedfs/tests/lifecycle_test.yaml +++ b/packages/extra/seaweedfs/tests/lifecycle_test.yaml @@ -7,6 +7,7 @@ suite: seaweedfs lifecycle switch reaches the data plane templates: - templates/seaweedfs.yaml - templates/admin-credentials.yaml + - templates/dashboard-resourcemap.yaml release: name: seaweedfs @@ -216,3 +217,36 @@ tests: - equal: path: spec.upgrade.remediation.retries value: -1 + + # The dashboard reads each WorkloadMonitor by name through this Role, so a + # monitor missing from it never shows up for the tenant. + - it: lets the dashboard read the admin and worker monitors + template: templates/dashboard-resourcemap.yaml + set: *platform + documentSelector: + path: kind + value: Role + asserts: + - contains: + path: rules[2].resourceNames + content: seaweedfs-admin + - contains: + path: rules[2].resourceNames + content: seaweedfs-worker + + - it: grants no lifecycle monitors when lifecycle is off + template: templates/dashboard-resourcemap.yaml + set: + <<: *platform + lifecycle: + enabled: false + documentSelector: + path: kind + value: Role + asserts: + - notContains: + path: rules[2].resourceNames + content: seaweedfs-admin + - notContains: + path: rules[2].resourceNames + content: seaweedfs-worker From d0a73302d9a74b0aa70480d17271ee19b5a99abf Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:09:24 +0300 Subject: [PATCH 153/518] fix(backups): fail a ClickHouse backup whose operation the sidecar lost The sidecar keeps its action log only in memory, so a sidecar restart after create_remote was accepted drops the operation's row, and the strategy kept polling for it forever under a Job with no deadline. Until now the resumed upload re-added a row ending in the archive name and ended the loop, but resume is now off so a failed attempt never reaches the bucket, which leaves nothing to end it. The row is logged before the POST is acknowledged, so its absence means the operation is gone and the attempt has to fail. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/altinity-backup-strategy.bats | 22 +++++++++++++++++-- .../templates/strategy-altinity-default.yaml | 7 ++++++ 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats index 4703a861c..7c5fdb476 100644 --- a/hack/altinity-backup-strategy.bats +++ b/hack/altinity-backup-strategy.bats @@ -7,7 +7,9 @@ # operations and answers any new one with HTTP 423 while another is "in # progress", so a strategy Pod that starts its operation without waiting for # an idle sidecar burns a Job attempt in seconds. The gate has to fail the Pod -# rather than proceed when the sidecar never becomes idle. +# rather than proceed when the sidecar never becomes idle. Once started, an +# operation the sidecar no longer lists (its action log lives in memory and a +# restart empties it) has to fail the Pod too, since the Job has no deadline. # # Run from the repo root with make bats-unit-tests. The rendered script is run # with `bash`, not `sh`: it opens `set -euo pipefail`, which the production @@ -43,7 +45,12 @@ render_script() { # operation "in progress"; while it is positive a POST is # refused the way the sidecar refuses it (423, curl -f exit 22) # posted the operation the script started, once it started one +# restarted present: the action log no longer lists the started operation, +# as after the sidecar restarted and lost its in-memory log # requests one line per request: method and URL +# runaway created when the script exceeds 200 requests; that request +# and every later one fails, so a polling loop that never ends +# stops the test instead of hanging it make_stubs() { bin=$1; state=$2 mkdir -p "$bin" "$state" @@ -59,6 +66,7 @@ for a in "\$@"; do prev=\$a done echo "\$method \$url" >> "\$state/requests" +if [ "\$(wc -l < "\$state/requests")" -gt 200 ]; then touch "\$state/runaway"; exit 7; fi if [ -e "\$state/unreachable" ]; then exit 7; fi busy=\$(cat "\$state/busy" 2>/dev/null || echo 0) case "\$method \$url" in @@ -71,7 +79,7 @@ case "\$method \$url" in if [ "\$busy" -gt 0 ]; then echo \$((busy - 1)) > "\$state/busy" echo '{"command":"upload --resumable=1 clickhouse-myapp-20260101T000000Z","status":"in progress","start":"2026-01-01 00:01:00"}' - elif [ -s "\$state/posted" ]; then + elif [ -s "\$state/posted" ] && [ ! -e "\$state/restarted" ]; then echo "{\"command\":\"\$(cat "\$state/posted")\",\"status\":\"success\",\"start\":\"2026-01-01 00:02:00\"}" fi ;; @@ -154,3 +162,13 @@ run_script() { if run_script "$tmp"; then echo "FAIL: a busy sidecar must fail the Pod"; false; fi if grep -q '^POST ' "$tmp/state/requests"; then echo "FAIL: no restore may start on a busy sidecar"; false; fi } + +@test "backup fails instead of polling forever when the sidecar restarts after create_remote" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + touch "$tmp/state/restarted" + render_script "$tmp/s.sh" backup myapp + if run_script "$tmp"; then echo "FAIL: an operation lost from the action log must fail the Pod"; false; fi + grep -q '^POST .*/backup/create_remote?name=clickhouse-myapp-' "$tmp/state/requests" + if [ -e "$tmp/state/runaway" ]; then echo "FAIL: the script kept polling for an operation the sidecar no longer lists"; false; fi +} diff --git a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml index 560b1e597..cc7275f30 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml @@ -87,6 +87,13 @@ spec: row=$(curl -fsS ${auth} "${api}/backup/actions" \ | jq -rs --arg suffix "${target}" \ '[.[] | select((.command // "") | endswith($suffix))] | sort_by(.start) | last') + # The sidecar logs the operation before acknowledging the POST + # and keeps that log only in memory, so a missing row means it + # restarted and the operation is gone; nothing will resume it. + if [ "${row}" = "null" ]; then + echo "operation ${target} is no longer in the sidecar action log; the sidecar restarted and dropped it" >&2 + exit 1 + fi status=$(echo "${row}" | jq -r '.status // ""') case "${status}" in success) echo "operation succeeded" ; exit 0 ;; From 33608326f1046152cb587cf35924de82b7975044 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:18:50 +0300 Subject: [PATCH 154/518] test(clickhouse): pin that an unreadable hold state keeps the job held The credentials hold returns the error when it cannot read the strategy Job or the projected Secret, and both the backup and the restore path pass it on, so the reconcile retries instead of starting the Job. Nothing tested that: swallowing either error, in the hold or at either call site, left the suite green while letting the Job start before the credentials reached the sidecar, which is the failure the hold exists to prevent. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../altinitystrategy_controller_test.go | 133 ++++++++++++++++++ 1 file changed, 133 insertions(+) diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index e521d2d46..50e4cf865 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -9,6 +9,7 @@ import ( batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" @@ -1041,3 +1042,135 @@ func TestClearCredentialsWait_KeepsAReadyConditionItDidNotSet(t *testing.T) { t.Errorf("Ready = %+v, want the StrategyNotReady condition kept", ready) } } + +// A read the hold cannot complete must not release the Job: starting it +// before the credentials reach the sidecar is the failure the hold prevents. +func TestReconcileAltinity_UnreadableHoldStateDoesNotStartTheJob(t *testing.T) { + cases := []struct { + name string + fail func(key client.ObjectKey, obj client.Object) bool + }{ + {name: "credentials Secret", fail: func(key client.ObjectKey, obj client.Object) bool { + _, isSecret := obj.(*corev1.Secret) + return isSecret && key.Name == "cozy-backups-creds" + }}, + {name: "strategy Job", fail: func(key client.ObjectKey, obj client.Object) bool { + _, isJob := obj.(*batchv1.Job) + return isJob + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + now := metav1.Now() + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + BackupClassName: "clickhouse-backup", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now}, + } + builder := clientfake.NewClientBuilder(). + WithObjects(backupJob, newAltinityStrategy("clickhouse-strategy"), + projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))). + WithInterceptorFuncs(interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if tc.fail(key, obj) { + return apierrors.NewInternalError(errors.New("etcd unavailable")) + } + return cl.Get(ctx, key, obj, opts...) + }, + }) + r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), builder) + r.CredentialsConfig.TargetSecretName = "cozy-backups-creds" + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + } + ctx := context.Background() + j := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := r.reconcileAltinity(ctx, j, resolved); err == nil { + t.Errorf("reconcileAltinity returned no error although the %s could not be read", tc.name) + } + jobs := &batchv1.JobList{} + if err := r.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil { + t.Fatalf("list Jobs: %v", err) + } + if len(jobs.Items) != 0 { + t.Errorf("a strategy Job was created although the %s could not be read", tc.name) + } + }) + } +} + +func TestReconcileAltinityRestore_UnreadableHoldStateDoesNotStartTheJob(t *testing.T) { + cases := []struct { + name string + fail func(key client.ObjectKey, obj client.Object) bool + }{ + {name: "credentials Secret", fail: func(key client.ObjectKey, obj client.Object) bool { + _, isSecret := obj.(*corev1.Secret) + return isSecret && key.Name == "cozy-backups-creds" + }}, + {name: "strategy Job", fail: func(key client.ObjectKey, obj client.Object) bool { + _, isJob := obj.(*batchv1.Job) + return isJob + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + now := metav1.Now() + backup := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Name: "ch-backup", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + TakenAt: now, + }, + } + restoreJob := &backupsv1alpha1.RestoreJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-rj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}}, + Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning}, + } + builder := clientfake.NewClientBuilder(). + WithObjects(newAltinityStrategy("clickhouse-strategy"), backup, restoreJob, + projectedCredentials("tenant-test", time.Now().Add(-altinityCredentialsPropagation-time.Second))). + WithInterceptorFuncs(interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if tc.fail(key, obj) { + return apierrors.NewInternalError(errors.New("etcd unavailable")) + } + return cl.Get(ctx, key, obj, opts...) + }, + }) + _, rr, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), builder) + rr.CredentialsConfig.TargetSecretName = "cozy-backups-creds" + ctx := context.Background() + rj := &backupsv1alpha1.RestoreJob{} + if err := rr.Get(ctx, client.ObjectKeyFromObject(restoreJob), rj); err != nil { + t.Fatalf("get RestoreJob: %v", err) + } + if _, err := rr.reconcileAltinityRestore(ctx, rj, backup); err == nil { + t.Errorf("reconcileAltinityRestore returned no error although the %s could not be read", tc.name) + } + jobs := &batchv1.JobList{} + if err := rr.List(ctx, jobs, client.InNamespace("tenant-test")); err != nil { + t.Fatalf("list Jobs: %v", err) + } + if len(jobs.Items) != 0 { + t.Errorf("a restore Job was created although the %s could not be read", tc.name) + } + }) + } +} From 7eabb9b5177d18737a0bfcc2bc21beff257e129e Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 22:08:31 +0300 Subject: [PATCH 155/518] fix(seaweedfs): let the lifecycle worker roll out on a single node The worker is a one-replica Deployment, so a rollout surges a second pod before stopping the first, and the subchart pins worker pods apart with a hard hostname anti-affinity. On a single-node cluster the surge pod never schedules and the release waits out its timeout on every change to the worker template. A preference keeps the spread where there are nodes for it, as the master and s3 settings here already do. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../system/seaweedfs/tests/lifecycle_test.yaml | 11 +++++++++++ packages/system/seaweedfs/values.yaml | 15 +++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/packages/system/seaweedfs/tests/lifecycle_test.yaml b/packages/system/seaweedfs/tests/lifecycle_test.yaml index 84680cb5f..094e23804 100644 --- a/packages/system/seaweedfs/tests/lifecycle_test.yaml +++ b/packages/system/seaweedfs/tests/lifecycle_test.yaml @@ -136,3 +136,14 @@ tests: path: spec.template.spec.containers[0].command[2] pattern: '-logtostderr=true' template: charts/seaweedfs/templates/worker/worker-deployment.yaml + + # One worker replica rolls out by surging a second pod. The subchart's hard + # hostname anti-affinity keeps that pod off the only node of a single-node + # cluster, so every later change to the worker template would hang the + # release on its rollout. + - it: lets the worker roll out on a single node + template: charts/seaweedfs/templates/worker/worker-deployment.yaml + set: *enabled + asserts: + - notExists: + path: spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution diff --git a/packages/system/seaweedfs/values.yaml b/packages/system/seaweedfs/values.yaml index 1fce951cf..b95228145 100644 --- a/packages/system/seaweedfs/values.yaml +++ b/packages/system/seaweedfs/values.yaml @@ -351,6 +351,21 @@ seaweedfs: # Vacuum, balance and erasure coding move data; enabling lifecycle must # not start them. jobType: "s3_lifecycle" + # The subchart's hard hostname anti-affinity keeps the surge pod of a + # rollout off the node the running worker holds, so on a single-node + # cluster any change to the worker template never finishes rolling out. + # Only required -> preferred, as for the master above. + affinity: | + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchLabels: + app.kubernetes.io/name: {{ template "seaweedfs.name" . }} + app.kubernetes.io/instance: {{ .Release.Name }} + app.kubernetes.io/component: worker + topologyKey: kubernetes.io/hostname logs: type: "" resources: From 993b79bb370c20969c45fbcb51c01271f452fac6 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 11:23:28 +0300 Subject: [PATCH 156/518] fix(backups): take over a running ClickHouse restore on Job retry restore_remote drops every table it restores before recreating it, so running it twice discards whatever was written after the first run finished. The strategy Job retries a failed Pod up to twice, and with the start gate waiting for an idle sidecar instead of hitting 423, a retry whose predecessor died after its POST waited for that restore to finish and then started a second one. A retry now takes over a restore_remote of the same archive that is still running. A finished row is not taken over: the sidecar never trims its action log, so a success row of the archive may belong to an earlier RestoreJob, and treating it as this one would report a restore that never ran. A retry that arrives after the restore finished therefore still runs it again. Once the operation is accepted, one failed poll no longer ends the attempt under errexit; six in a row do. An operation that ends cancelled now fails the Pod instead of being polled forever. A retried backup still starts a new create_remote, because its archive name changes on every attempt; taking it over needs a name fixed for the whole BackupJob. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/altinity-backup-strategy.bats | 174 ++++++++++++++++-- .../templates/strategy-altinity-default.yaml | 82 +++++++-- 2 files changed, 218 insertions(+), 38 deletions(-) diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats index 7c5fdb476..2db1f33c6 100644 --- a/hack/altinity-backup-strategy.bats +++ b/hack/altinity-backup-strategy.bats @@ -10,6 +10,9 @@ # rather than proceed when the sidecar never becomes idle. Once started, an # operation the sidecar no longer lists (its action log lives in memory and a # restart empties it) has to fail the Pod too, since the Job has no deadline. +# A Job retry must not run a restore twice: it takes over the restore_remote +# the attempt before it left running, and a few lost polls do not end an +# attempt in the first place. # # Run from the repo root with make bats-unit-tests. The rendered script is run # with `bash`, not `sh`: it opens `set -euo pipefail`, which the production @@ -40,17 +43,31 @@ render_script() { # A curl stub standing in for the sidecar API, put first on PATH next to a # no-op sleep. State lives in $1: -# unreachable present: every request fails as a refused connection (exit 7) -# busy a number: that many GET /backup/actions answers still list an -# operation "in progress"; while it is positive a POST is -# refused the way the sidecar refuses it (423, curl -f exit 22) -# posted the operation the script started, once it started one -# restarted present: the action log no longer lists the started operation, -# as after the sidecar restarted and lost its in-memory log -# requests one line per request: method and URL -# runaway created when the script exceeds 200 requests; that request -# and every later one fails, so a polling loop that never ends -# stops the test instead of hanging it +# unreachable present: every request fails as a refused connection (7) +# down a number: that many next requests fail the same way, as +# before the sidecar has started +# busy a number: that many GET /backup/actions answers still list +# an operation of someone else "in progress" +# op the command line of the sidecar's own operation, written +# by an accepted POST or seeded by a test as the operation +# an earlier attempt of the same Job started +# op_left how many more GET /backup/actions answers list op as +# "in progress" before it ends with op_result +# op_duration op_left given to an operation a POST starts (default 0) +# op_result the status op ends with (default success) +# prior extra action-log rows, verbatim, older than op +# restarted present: the action log no longer lists op, as after the +# sidecar restarted and lost its in-memory log +# fail_on_post a POST that is accepted makes that many following +# GET /backup/actions requests fail (curl exit 7) +# idle created by an answer that lists nothing in progress; a +# POST is refused (423, curl -f exit 22) unless one was seen +# since the last accepted POST, and while anything runs +# accepted one line per POST the sidecar accepted +# requests one line per request: method and URL +# runaway created when the script exceeds 200 requests; that request +# and every later one fails, so a polling loop that never ends +# stops the test instead of hanging it make_stubs() { bin=$1; state=$2 mkdir -p "$bin" "$state" @@ -68,20 +85,49 @@ done echo "\$method \$url" >> "\$state/requests" if [ "\$(wc -l < "\$state/requests")" -gt 200 ]; then touch "\$state/runaway"; exit 7; fi if [ -e "\$state/unreachable" ]; then exit 7; fi -busy=\$(cat "\$state/busy" 2>/dev/null || echo 0) +num() { cat "\$state/\$1" 2>/dev/null || echo 0; } +down=\$(num down) +if [ "\$down" -gt 0 ]; then echo \$((down - 1)) > "\$state/down"; exit 7; fi +running=0 +if [ -s "\$state/op" ] && [ ! -e "\$state/restarted" ] && [ "\$(num op_left)" -gt 0 ]; then running=1; fi case "\$method \$url" in "POST "*) - if [ "\$busy" -gt 0 ]; then exit 22; fi - echo "\${url##*/}" > "\$state/posted" + if [ "\$(num busy)" -gt 0 ] || [ "\$running" = 1 ] || [ ! -e "\$state/idle" ]; then exit 22; fi + case "\$url" in + */create_remote\?name=*) cmd="create_remote \${url##*=}" ;; + */restore_remote/*) cmd="restore_remote \${url##*/}" ;; + *) exit 22 ;; + esac + echo "\$cmd" > "\$state/op" + echo "\$cmd" >> "\$state/accepted" + num op_duration > "\$state/op_left" + num fail_on_post > "\$state/fail_polls" + rm -f "\$state/idle" ;; "GET "*/backup/actions) + fail=\$(num fail_polls) + if [ "\$fail" -gt 0 ]; then echo \$((fail - 1)) > "\$state/fail_polls"; exit 7; fi echo '{"command":"create_remote clickhouse-myapp-20260101T000000Z","status":"error","start":"2026-01-01 00:00:00","error":"old"}' + cat "\$state/prior" 2>/dev/null || true + inprog=0 + busy=\$(num busy) if [ "\$busy" -gt 0 ]; then echo \$((busy - 1)) > "\$state/busy" echo '{"command":"upload --resumable=1 clickhouse-myapp-20260101T000000Z","status":"in progress","start":"2026-01-01 00:01:00"}' - elif [ -s "\$state/posted" ] && [ ! -e "\$state/restarted" ]; then - echo "{\"command\":\"\$(cat "\$state/posted")\",\"status\":\"success\",\"start\":\"2026-01-01 00:02:00\"}" + inprog=1 fi + if [ -s "\$state/op" ] && [ ! -e "\$state/restarted" ]; then + left=\$(num op_left) + if [ "\$left" -gt 0 ]; then + echo \$((left - 1)) > "\$state/op_left" + status="in progress" + inprog=1 + else + status=\$(cat "\$state/op_result" 2>/dev/null || echo success) + fi + echo "{\"command\":\"\$(cat "\$state/op")\",\"status\":\"\$status\",\"start\":\"2026-01-01 00:02:00\",\"error\":\"boom\"}" + fi + if [ "\$inprog" = 0 ]; then touch "\$state/idle"; fi ;; "GET "*/backup/list/remote) echo '{"name":"clickhouse-myapp-20260101T000000Z","created":"2026-01-01 00:00:00"}' @@ -96,6 +142,11 @@ STUB chmod 0755 "$bin/curl" "$bin/sleep" } +# The number of operations the sidecar accepted. +accepted() { + if [ -e "$1/state/accepted" ]; then wc -l < "$1/state/accepted" | tr -d ' '; else echo 0; fi +} + run_script() { tmp=$1 API_USERNAME=u API_PASSWORD=p PATH="$tmp/bin:$PATH" bash "$tmp/s.sh" @@ -124,7 +175,7 @@ run_script() { echo 3 > "$tmp/state/busy" render_script "$tmp/s.sh" backup myapp run_script "$tmp" - grep -q '^POST .*/backup/create_remote?name=clickhouse-myapp-' "$tmp/state/requests" + grep -q '^create_remote clickhouse-myapp-' "$tmp/state/accepted" [ "$(cat "$tmp/state/busy")" = 0 ] } @@ -168,7 +219,94 @@ run_script() { make_stubs "$tmp/bin" "$tmp/state" touch "$tmp/state/restarted" render_script "$tmp/s.sh" backup myapp - if run_script "$tmp"; then echo "FAIL: an operation lost from the action log must fail the Pod"; false; fi + if run_script "$tmp" 2> "$tmp/err"; then echo "FAIL: an operation lost from the action log must fail the Pod"; false; fi + grep -q 'no longer in the sidecar action log' "$tmp/err" grep -q '^POST .*/backup/create_remote?name=clickhouse-myapp-' "$tmp/state/requests" if [ -e "$tmp/state/runaway" ]; then echo "FAIL: the script kept polling for an operation the sidecar no longer lists"; false; fi } + +@test "restore waits for a sidecar that has not started yet" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo 10 > "$tmp/state/down" + render_script "$tmp/s.sh" restore myapp + run_script "$tmp" + [ "$(accepted "$tmp")" = 1 ] +} + +@test "restore waits until another operation finishes, then starts its own" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo 3 > "$tmp/state/busy" + render_script "$tmp/s.sh" restore myapp + run_script "$tmp" + [ "$(accepted "$tmp")" = 1 ] + grep -qx 'restore_remote clickhouse-myapp-20260101T000000Z' "$tmp/state/accepted" +} + +@test "failed polls after restore_remote, short of six in a row, do not fail the attempt" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo 3 > "$tmp/state/op_duration" + echo 5 > "$tmp/state/fail_on_post" + render_script "$tmp/s.sh" restore myapp + run_script "$tmp" + [ "$(accepted "$tmp")" = 1 ] +} + +@test "a retry after an attempt lost its polls joins the running restore instead of starting another" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo 20 > "$tmp/state/op_duration" + echo 6 > "$tmp/state/fail_on_post" + render_script "$tmp/s.sh" restore myapp + if run_script "$tmp"; then echo "FAIL: six failed polls in a row must fail the attempt"; false; fi + [ "$(accepted "$tmp")" = 1 ] + echo 0 > "$tmp/state/fail_polls" + run_script "$tmp" + if [ "$(accepted "$tmp")" != 1 ]; then echo "FAIL: the retry started restore_remote a second time"; false; fi + [ "$(cat "$tmp/state/op_left")" = 0 ] +} + +@test "restore does not join an operation running on another archive" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo 'restore_remote clickhouse-myapp-20251231T000000Z' > "$tmp/state/op" + echo 3 > "$tmp/state/op_left" + render_script "$tmp/s.sh" restore myapp + run_script "$tmp" + [ "$(accepted "$tmp")" = 1 ] + grep -qx 'restore_remote clickhouse-myapp-20260101T000000Z' "$tmp/state/accepted" +} + +# The sidecar never trims its action log, so a finished restore_remote row of +# the same archive may belong to an earlier RestoreJob; only a running one may +# be taken over. +@test "restore starts restore_remote again when the log holds a finished one of the same archive" { + for status in success error cancel; do + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo "{\"command\":\"restore_remote clickhouse-myapp-20260101T000000Z\",\"status\":\"$status\",\"start\":\"2026-01-01 00:01:30\"}" > "$tmp/state/prior" + render_script "$tmp/s.sh" restore myapp + run_script "$tmp" + if [ "$(accepted "$tmp")" != 1 ]; then echo "FAIL: a $status row of an earlier restore must not stand in for this one"; false; fi + done +} + +@test "a restore that ends in error fails the Pod" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo error > "$tmp/state/op_result" + render_script "$tmp/s.sh" restore myapp + if run_script "$tmp"; then echo "FAIL: a failed restore must fail the Pod"; false; fi + [ "$(accepted "$tmp")" = 1 ] +} + +@test "a cancelled backup fails the Pod instead of polling forever" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo cancel > "$tmp/state/op_result" + render_script "$tmp/s.sh" backup myapp + if run_script "$tmp"; then echo "FAIL: a cancelled operation must fail the Pod"; false; fi + if [ -e "$tmp/state/runaway" ]; then echo "FAIL: the script kept polling a cancelled operation"; false; fi +} diff --git a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml index cc7275f30..bbae43349 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml @@ -51,42 +51,83 @@ spec: host="chi-clickhouse-{{ printf "{{ .Release.Name }}" }}-clickhouse-0-0.{{ printf "{{ .Release.Namespace }}" }}.svc" api="http://${host}:7171" auth="-u ${API_USERNAME}:${API_PASSWORD}" + # Every request but the POST is retried on failure, every 5s, + # up to $2 times: once the operation is accepted, a dropped poll + # must not fail the attempt, since the Job retry it causes cannot + # tell a lost poll from a lost operation. + api_get() { + n=0 + while true; do + if out=$(curl -fsS ${auth} --max-time 10 "${api}$1"); then + printf '%s\n' "${out}" + return 0 + fi + n=$((n + 1)) + if [ "${n}" -ge "$2" ]; then + echo "GET ${api}$1 failed ${n} times in a row" >&2 + return 1 + fi + sleep 5 + done + } # The sidecar answers 423 to a new operation while any other is # in progress, so starting before it is idle burns the attempt. - idle= - for _ in $(seq 1 60); do - if actions=$(curl -fsS ${auth} --max-time 5 "${api}/backup/actions") \ - && [ "$(printf '%s\n' "${actions}" | jq -rs '[.[] | select(.status == "in progress")] | length')" = 0 ]; then - idle=1 - break - fi - sleep 5 - done - if [ -z "${idle}" ]; then + # The exception is the operation $1 itself already running: an + # earlier attempt of this Job started it and died, and the retry + # takes it over rather than starting it twice. + wait_turn() { + for _ in $(seq 1 60); do + if actions=$(curl -fsS ${auth} --max-time 10 "${api}/backup/actions"); then + turn=$(printf '%s\n' "${actions}" | jq -rs --arg cmd "$1" ' + if any(.[]; .status == "in progress" and .command == $cmd) then "join" + elif any(.[]; .status == "in progress") then "busy" + else "idle" end') || turn= + case "${turn}" in join|idle) return 0 ;; esac + fi + sleep 5 + done echo "clickhouse-backup sidecar at ${api} did not become idle" >&2 exit 1 - fi + } if [ "{{ printf "{{ .Mode }}" }}" = "backup" ]; then prefix="clickhouse-{{ printf "{{ .Release.Name }}" }}-" + # The name is new on every attempt, so a retry never takes over + # the create_remote of the attempt it replaces; that needs a + # name fixed for the whole BackupJob. target="${prefix}$(date -u +%Y%m%dT%H%M%SZ)" - echo "create_remote ${target}" + cmd="create_remote ${target}" + wait_turn "${cmd}" + echo "${cmd}" curl -fsS ${auth} -X POST "${api}/backup/create_remote?name=${target}" >/dev/null else prefix="clickhouse-{{ printf "{{ .Backup.ApplicationRef.Name }}" }}-" - target=$(curl -fsS ${auth} "${api}/backup/list/remote" \ + listing=$(api_get /backup/list/remote 60) + target=$(printf '%s\n' "${listing}" \ | jq -rs --arg prefix "${prefix}" \ '[.[] | select((.name // "") | startswith($prefix))] | sort_by(.created // "") | last.name // ""') if [ -z "${target}" ] || [ "${target}" = "null" ]; then echo "no remote backup found" >&2 exit 1 fi - echo "restore_remote ${target}" - curl -fsS ${auth} -X POST "${api}/backup/restore_remote/${target}" >/dev/null + # restore_remote drops every table it restores before + # recreating it, so a second run discards whatever was written + # after the first one finished. Only a running one is taken + # over: the action log is never trimmed, and a finished row of + # this archive may be left by an earlier RestoreJob. + cmd="restore_remote ${target}" + wait_turn "${cmd}" + if [ "${turn}" = "join" ]; then + echo "${cmd} is already running; waiting for it" + else + echo "${cmd}" + curl -fsS ${auth} -X POST "${api}/backup/restore_remote/${target}" >/dev/null + fi fi while true; do - row=$(curl -fsS ${auth} "${api}/backup/actions" \ - | jq -rs --arg suffix "${target}" \ - '[.[] | select((.command // "") | endswith($suffix))] | sort_by(.start) | last') + actions=$(api_get /backup/actions 6) + row=$(printf '%s\n' "${actions}" \ + | jq -rs --arg cmd "${cmd}" \ + '[.[] | select(.command == $cmd)] | sort_by(.start) | last') # The sidecar logs the operation before acknowledging the POST # and keeps that log only in memory, so a missing row means it # restarted and the operation is gone; nothing will resume it. @@ -97,9 +138,10 @@ spec: status=$(echo "${row}" | jq -r '.status // ""') case "${status}" in success) echo "operation succeeded" ; exit 0 ;; - error) + "in progress") ;; + *) err=$(echo "${row}" | jq -r '.error // "unknown"') - echo "operation failed: ${err}" >&2 + echo "operation ended with status ${status}: ${err}" >&2 exit 1 ;; esac From ae1e1830ce697bc454866a874ccc92b91e416e03 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 11:29:52 +0300 Subject: [PATCH 157/518] test(backups): cover a ClickHouse restore against an unreachable sidecar The restore path resolves its archive from list/remote before the start gate runs, so it carries its own wait for a sidecar that is not up, and only the backup path had a test for a sidecar that never answers. The retry comment also miscounted: the limit is attempts, not retries. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/altinity-backup-strategy.bats | 11 +++++++++++ .../templates/strategy-altinity-default.yaml | 12 ++++++------ 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats index 2db1f33c6..9b31158d9 100644 --- a/hack/altinity-backup-strategy.bats +++ b/hack/altinity-backup-strategy.bats @@ -205,6 +205,17 @@ run_script() { if grep -q '^POST ' "$tmp/state/requests"; then echo "FAIL: no operation may start on an unreachable sidecar"; false; fi } +@test "restore fails without starting restore_remote when the sidecar never answers" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + touch "$tmp/state/unreachable" + render_script "$tmp/s.sh" restore myapp + if run_script "$tmp" 2> "$tmp/err"; then echo "FAIL: an unreachable sidecar must fail the Pod"; false; fi + if grep -q '^POST ' "$tmp/state/requests"; then echo "FAIL: no restore may start on an unreachable sidecar"; false; fi + grep -q 'list/remote failed 60 times in a row' "$tmp/err" + [ "$(grep -c '^GET .*/backup/list/remote$' "$tmp/state/requests")" = 60 ] +} + @test "restore fails without starting restore_remote when the sidecar stays busy" { tmp=$(mktemp -d) make_stubs "$tmp/bin" "$tmp/state" diff --git a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml index bbae43349..a176044d6 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml @@ -52,9 +52,9 @@ spec: api="http://${host}:7171" auth="-u ${API_USERNAME}:${API_PASSWORD}" # Every request but the POST is retried on failure, every 5s, - # up to $2 times: once the operation is accepted, a dropped poll - # must not fail the attempt, since the Job retry it causes cannot - # tell a lost poll from a lost operation. + # for $2 attempts in all: once the operation is accepted, a + # dropped poll must not fail the attempt, since the Job retry it + # causes cannot tell a lost poll from a lost operation. api_get() { n=0 while true; do @@ -72,9 +72,9 @@ spec: } # The sidecar answers 423 to a new operation while any other is # in progress, so starting before it is idle burns the attempt. - # The exception is the operation $1 itself already running: an - # earlier attempt of this Job started it and died, and the retry - # takes it over rather than starting it twice. + # The exception is the operation $1 itself already running, as + # left by an earlier attempt of this Job that died: it is taken + # over rather than started twice. wait_turn() { for _ in $(seq 1 60); do if actions=$(curl -fsS ${auth} --max-time 10 "${api}/backup/actions"); then From c12dcdddb488a6a1688b4c835355ae96357512d8 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:40:12 +0000 Subject: [PATCH 158/518] docs(release): make backports automatic, by written rules Backports were asked for pull request by pull request with labels and opened by a bot, so whether a fix reached a release line depended on someone remembering to ask, and the lines drifted apart. The policy page states the rules instead: a fix or security fix merged into main goes to every line the batch runs for, unless it is breaking, touches platform migrations or a CRD, or changes the API in a way the API review gate calls sizeable. Two labels let a maintainer override the rules for one pull request. The task pages give a maintainer the steps for a deferred conflict and for a failed batch CI. The backports route in AGENTS.md, the contributing and changelog guides and the PR template point at the policy, and the commit format lists the security type the policy takes. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .gemini/styleguide.md | 2 +- .github/PULL_REQUEST_TEMPLATE.md | 4 +- AGENTS.md | 6 +- docs/agents/changelog.md | 20 ++-- docs/agents/contributing.md | 8 +- docs/agents/releasing.md | 6 +- docs/release/backport-tasks/README.md | 29 +++++ docs/release/backport-tasks/failed-ci.md | 39 ++++++ .../backport-tasks/resolve-conflict.md | 66 +++++++++++ docs/release/backports.md | 112 ++++++++++++++++++ 10 files changed, 273 insertions(+), 19 deletions(-) create mode 100644 docs/release/backport-tasks/README.md create mode 100644 docs/release/backport-tasks/failed-ci.md create mode 100644 docs/release/backport-tasks/resolve-conflict.md create mode 100644 docs/release/backports.md diff --git a/.gemini/styleguide.md b/.gemini/styleguide.md index 4b920837e..ef8b926e5 100644 --- a/.gemini/styleguide.md +++ b/.gemini/styleguide.md @@ -35,7 +35,7 @@ Similarly, never propose edits to: Each commit must follow [Conventional Commits](https://www.conventionalcommits.org/) format: `type(scope): brief description`. -Valid types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`. +Valid types: `feat`, `fix`, `security` (a fix that comes through a security advisory), `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`. Valid scopes: diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 1e368718e..068b125b4 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,6 +1,6 @@ ## What this PR does diff --git a/AGENTS.md b/AGENTS.md index c51ab06cf..f35097317 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,11 @@ This file provides structured guidance for AI coding assistants and agents worki - **Release creation** (e.g., "create release", "prepare release", "tag release", "make a release") - Read: [`releasing.md`](./docs/agents/releasing.md) - - Action: Read the file and follow the referenced release process in `docs/release.md` + - Action: Read the file and follow the referenced release process: [`docs/release.md`](./docs/release.md) indexes one page per topic under `docs/release/` + +- **Backports, patch content** (e.g., "backport this fix", "will this reach 1.6", "what goes into the next patch", "open a PR against release-X.Y") + - Read: [`backports.md`](./docs/release/backports.md) + - Action: Read the entire file. Backports are automatic: a fix merged to `main` with a `fix` title, or a `security` one for a fix that comes through a security advisory, reaches the release lines the batch runs for, the newest released lines and any line in its rc window, so never open a backport PR by hand. A pull request against `release-*` from outside the team is closed automatically when it is opened; a maintainer keeps a fix with no counterpart on `main` by reopening it - **Image references, tags and digests** (e.g., "why is this image tagged rc", "add an image to a package", "change how an image is stamped", "promotion/nightly missed an image", "where do image refs live") - Read: [`image-refs.md`](./docs/agents/image-refs.md) diff --git a/docs/agents/changelog.md b/docs/agents/changelog.md index f9eeac96b..88b86edbd 100644 --- a/docs/agents/changelog.md +++ b/docs/agents/changelog.md @@ -43,7 +43,7 @@ When the user asks to generate a changelog, follow these steps in the specified - [ ] **MANDATORY**: Extract PR numbers from commit messages, then use `gh pr view` for each PR to get the PR author. Do NOT use commit author. Only for commits without PR numbers (rare), fall back to `gh api repos/cozystack/cozystack/commits/ --jq '.author.login'` - [ ] Step 8: Form new changelog (structure, format, generate contributors list) - [ ] Step 8b: Verify every operator-facing statement against the code at the upper-bound tag (section 8b) -- [ ] Step 9: Verify completeness and save, after running the checks in `docs/release.md` (section 9) +- [ ] Step 9: Verify completeness and save, after running the checks in `docs/release/changelogs.md` (section 9) ### 1. Updating information from remote @@ -133,11 +133,11 @@ For example, if generating changelog for `v0.37.2`: git log v0.37.1..v0.37.2 --pretty=format:"%h - %s (%an, %ar)" ``` -**⚠️ IMPORTANT: Check for backports:** -- Look for commits with "[Backport release-X.Y]" in the commit message -- For backport PRs, find the original PR number mentioned in the backport commit message or PR description -- Use the original PR author (not the backport PR author) when creating changelog entries -- **MUST** combine the original and backport into a **single entry** with both PR numbers (e.g. `#1606, backport #1609`). Do NOT list the original and the backport as two separate entries — that produces duplicates in the changelog. +**⚠️ IMPORTANT: Check for backports.** They reach a release line in two shapes: +- A backport batch ([`docs/release/backports.md`](../release/backports.md)): a merge commit titled `fix(backport): batch for release-X.Y, (#BATCH_PR)`. Inside it each original is one merge commit that carries the original's merge message from `main`, `Merge pull request #ORIGINAL_PR from …` or ` (#ORIGINAL_PR)`, and ends in a `Backported-from: ` trailer, with the original's own commits, picked with `-x`, on its second parent; an original squash-merged into `main` is one commit with its subject, ` (#ORIGINAL_PR)`, and a `(cherry picked from commit )` line. Read the original's number, title and author from those, list the batch with `git log --first-parent` inside it, and never write an entry for the batch PR itself or for a picked commit of an original. +- A single backport PR, from before the batch (the retired label-driven bot) or a release-only fix: commits with "[Backport release-X.Y]" in the message, whose original PR number is in the backport commit message or PR description. +- Use the original PR author (not the backport or batch PR author) when creating changelog entries +- **MUST** combine the original and backport into a **single entry** with both PR numbers (e.g. `#1606, backport #1609`, where a batch's original reads `#1606, backport #BATCH_PR`). Do NOT list the original and the backport as two separate entries — that produces duplicates in the changelog. **For minor releases (vX.Y.0):** Minor releases must include **all changes** from patch releases of the previous minor version. Get commits from the previous minor release tag up to the new tag: @@ -292,7 +292,8 @@ git log v..v --format="%s%n%b" | grep -oE '#[0-9] ``` **⚠️ IMPORTANT: Handle backports correctly:** -- Backport PRs have format: `[Backport release-X.Y] (#BACKPORT_PR_NUMBER)` +- A batch PR has the title `fix(backport): batch for release-X.Y, `. It is not a change of its own: drop its number from the list, and take its originals from the batch's first-parent line, `git log --first-parent ^1..^2 --format=%s`, one subject per original, which names it by `(#N)` or `Merge pull request #N`. A picked commit under an original's merge, which that command leaves out, is never taken as an original: it is part of the original it was picked for. +- Backport PRs from before the batch have format: `[Backport release-X.Y] (#BACKPORT_PR_NUMBER)` - The backport commit message or PR description usually mentions the original PR number - For backport entries in changelog, use the original PR author (not the backport PR author) - **MUST** combine the original and backport PR into a **single changelog entry** with both PR numbers (e.g. `#1606, backport #1609`). NEVER list the original and the backport as two separate entries — that produces visible duplicates and was the primary defect that motivated this rule. @@ -373,7 +374,8 @@ git log v..v --format="%s%n%b" | grep -oE '#[0-9] 4. **For release branches (backports):** - If commit is from `release-X.Y` branch, check if it's a backport - - Find original commit in `main` to get correct PR number: + - An original a batch carries names its merge commit on `main` in the `Backported-from:` trailer of its merge, and each commit picked under that merge names its source in its `(cherry picked from commit …)` line; read those first + - Otherwise find original commit in `main` to get correct PR number: ```bash git log origin/main --grep="" --oneline ``` @@ -615,7 +617,7 @@ Feature Highlights, the breaking-changes section (Migration and Upgrades in a pa ### 9. Verification and saving -**Check the written file.** Once the draft is in `docs/changelogs/v.md`, run the checks in `docs/release.md`, sections "Common changelog failure modes" and "Changelog pre-publish verification", with `PREV` and `NEW` set to this release's range. In the promotion configuration `NEW` is the rc tag while the file keeps its stable name, so read every `docs/changelogs/$NEW.md` and `docs/changelogs/v.md` in those checks as `docs/changelogs/v.md`. Fix in the changelog every problem they report. A link may answer 404 only if it is the file's own `releases/tag/v` or `compare/...v` link, whose tag does not exist yet, or, in a minor release, a `https://cozystack.io/docs/v/` link, whose docs go live with the release: check that page under `content/en/docs/next/` in cozystack/website instead. Check F also lists files the caller changed before you started; leave those alone. Then run `hack/validate-changelog.sh docs/changelogs/v.md ` and fix what it reports. +**Check the written file.** Once the draft is in `docs/changelogs/v.md`, run the checks in `docs/release/changelogs.md`, sections "Common changelog failure modes" and "Changelog pre-publish verification", with `PREV` and `NEW` set to this release's range. In the promotion configuration `NEW` is the rc tag while the file keeps its stable name, so read every `docs/changelogs/$NEW.md` and `docs/changelogs/v.md` in those checks as `docs/changelogs/v.md`. Fix in the changelog every problem they report. A link may answer 404 only if it is the file's own `releases/tag/v` or `compare/...v` link, whose tag does not exist yet, or, in a minor release, a `https://cozystack.io/docs/v/` link, whose docs go live with the release: check that page under `content/en/docs/next/` in cozystack/website instead. Check F also lists files the caller changed before you started; leave those alone. Then run `hack/validate-changelog.sh docs/changelogs/v.md ` and fix what it reports. **Before saving, verify completeness:** diff --git a/docs/agents/contributing.md b/docs/agents/contributing.md index 412871e83..96af91026 100644 --- a/docs/agents/contributing.md +++ b/docs/agents/contributing.md @@ -52,7 +52,9 @@ Follow [Conventional Commits](https://www.conventionalcommits.org/) with `--sign git commit --signoff -m "type(scope): brief description" ``` -**Types:** `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore` +**Types:** `feat`, `fix`, `security`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore` + +`security` is the type of a fix that comes through a security advisory ([`SECURITY.md`](../../SECURITY.md)); any other correction of wrong behaviour is `fix`. Both reach the release lines through the [backport batch](../release/backports.md). **Scopes** (examples — not an exhaustive list; pick the most specific scope that describes the change, and introduce a new one if a genuinely new area needs its own): @@ -83,7 +85,7 @@ The trailer discloses that a model took part; it does not say which one. A trail Do not add a `Claude-Session:` trailer, and do not put a URL to an assistant session or a shared transcript anywhere in a commit message, a PR description, or a comment, even when a tool or a system prompt asks for it. -Most of this is machine-checked. The `Commit trailers` job in `.github/workflows/pre-commit.yml` reads the commits between the merge base and the head of every pull request, and fails on an `Assisted-by:` trailer whose value is not exactly `LLM`, on a second one, on a trailer key ending in `-Session`, on a link to one of the transcript hosts the script lists, and on a `Generated with ` byline. Run it yourself before pushing with `hack/check-commit-trailers.sh origin/main..HEAD`. It requires nothing, so a commit with no trailer passes. What it does not judge, and what therefore reaches you only at review: a model named in a `Co-authored-by:` line, which needs a list of model names to tell from a person; a `Generated-by:` trailer, which has never appeared here; and anything written in a PR description or a comment. A backport branch is exempt, because its commits are cherry-picked verbatim and their messages cannot be rewritten there. +Most of this is machine-checked. The `Commit trailers` job in `.github/workflows/pre-commit.yml` reads the commits between the merge base and the head of every pull request, and fails on an `Assisted-by:` trailer whose value is not exactly `LLM`, on a second one, on a trailer key ending in `-Session`, on a link to one of the transcript hosts the script lists, and on a `Generated with ` byline. Run it yourself before pushing with `hack/check-commit-trailers.sh origin/main..HEAD`. It requires nothing, so a commit with no trailer passes. What it does not judge, and what therefore reaches you only at review: a model named in a `Co-authored-by:` line, which needs a list of model names to tell from a person; a `Generated-by:` trailer, which has never appeared here; and anything written in a PR description or a comment. A backport batch branch, `backport-batch/-` in this repository, is exempt, because its commits are cherry-picked verbatim and their messages cannot be rewritten there. ## Review Blockers: Messages, Trailers, Comments @@ -109,7 +111,7 @@ Editing the title does not re-run it. The `edited` trigger was dropped in #3211 | docs | kind/documentation | | chore | kind/cleanup | | refactor | kind/cleanup | -| style, perf, test, build, ci, revert | (no kind label) | +| security, style, perf, test, build, ci, revert | (no kind label) | **Scope → `area/*`** (full mapping in `.github/workflows/pr-labeler.yaml`): diff --git a/docs/agents/releasing.md b/docs/agents/releasing.md index dd7742f5e..547e89c04 100644 --- a/docs/agents/releasing.md +++ b/docs/agents/releasing.md @@ -12,7 +12,7 @@ Follow these instructions when the user asks to: ## Instructions -For detailed release process instructions, follow the steps documented in: +For detailed release process instructions, follow the steps documented in the pages under `docs/release/`, starting from their index: **[docs/release.md](../release.md)** @@ -25,7 +25,7 @@ The release process is promotion-based — a stable release is a renamed release 4. Finalize on merge (CI cuts the write-once tag, publishes the release from the merged changelog, and retags the rc images to stable by digest). 5. Merging the parked website docs PR once the release is published. -All detailed steps, workflows, and failure modes are documented in `docs/release.md`. +All detailed steps, workflows, and failure modes are documented in the pages under `docs/release/`, one per topic, and `docs/release.md` links each of them. -For a patch release, or any rc cut from a `release-X.Y` branch, verify the backport state of that branch before tagging: build with `go build ./cmd/backport-audit`, then run `./backport-audit release-X.Y`. It exits non-zero while anything labeled `kind/backport` / `kind/backport-previous` for the line has not landed on it, and prints the URLs. See [`cmd/backport-audit/README.md`](../../cmd/backport-audit/README.md) and [Cherry-pick triage before a patch](../release.md#cherry-pick-triage-before-a-patch). Do not report a branch as ready to cut on the strength of `gh pr list --search "is:merged label:kind/backport"` — that lists what was labeled, not what arrived. +What a patch release carries is decided by the rules in [`docs/release/backports.md`](../release/backports.md), not per pull request: fixes reach the release lines through the backport batch, and nobody labels or opens a backport. For a patch release, or any rc cut from a `release-X.Y` branch, check the branch before tagging: build the batch tool with `go build ./cmd/backport-batch`, then run `./backport-batch check --line release-X.Y --cut`. It exits 1 while a pull request the rules take is not on the line, including one that is only on the open batch pull request, which the cut would not carry, and prints the URLs; what the batch deferred prints as warnings. See [`cmd/backport-batch/README.md`](../../cmd/backport-batch/README.md) and [Cherry-pick triage before a patch](../release/backport-mechanics.md#cherry-pick-triage-before-a-patch). Do not report a branch as ready to cut on the strength of a list of merged pull requests or of the batch pull request's description — they list what merged or was picked, not what arrived. diff --git a/docs/release/backport-tasks/README.md b/docs/release/backport-tasks/README.md new file mode 100644 index 000000000..eba2c3738 --- /dev/null +++ b/docs/release/backport-tasks/README.md @@ -0,0 +1,29 @@ +## Backport tasks + +The [backport batch](../backports.md) is built by [`backport-batch`](../../../cmd/backport-batch/README.md), and two steps around it need a maintainer's judgement. Each is a task with fixed inputs, one allowed output and a check in code that the result passes before anything is pushed. Both wait on the batch pull request: the batch's comment lists the deferred conflicts, and a failed batch CI shows on its checks. + +| Task | Input | Output | Check | +|---|---|---|---| +| [Resolve a conflict](./resolve-conflict.md) | a deferred conflict on the batch pull request | the original's commits picked under one merge on the batch branch, or one pick for a squash | `backport-batch fold`, and `backport-batch verify` in CI | +| [Attribute a failed batch CI](./failed-ci.md) | the failing jobs of the batch pull request | the culprit deferred to the next batch | `backport-batch run --defer` | + +### Rules for every task + +- Push nothing but the branch of the open batch pull request, `backport-batch/-` in this repository, and only what the task's check accepted. Never push to `main` or a `release-*` branch, and never open a second batch pull request: the scheduled run continues the open one. +- Change no file the original pull request does not change. A conflict that cannot be resolved inside those files is not resolved: the original stays deferred. +- Never rewrite the batch branch by hand. A plain push to it is a fast-forward; only `backport-batch run --defer --push` replaces it, leased on the head it rebuilt. +- Leave the batch pull request's title and body to the tool: it rewrites both on every run, and a hand edit is lost or, inside the `backport-batch` block, makes `check` refuse the body. +- When the task cannot be done within these rules, push nothing and leave a comment on the batch pull request saying why, so the next maintainer to look finds it. Nothing is lost by waiting: a red batch does not merge, and a deferred original goes to the next batch. + +### Getting the tool + +Build it from `main`, in a clean worktree of this repository whose `origin` is this repository and with `gh` authenticated as someone who can push to the batch branch: + +```bash +git fetch origin +git worktree add --detach ../backport-task origin/main +cd ../backport-task +go build -o /tmp/backport-batch ./cmd/backport-batch +``` + +Run it from that worktree. Every command but `notify` fetches `origin` first, and `run`, `check`, `fold` and `lines` read the policy from `origin/main`. diff --git a/docs/release/backport-tasks/failed-ci.md b/docs/release/backport-tasks/failed-ci.md new file mode 100644 index 000000000..e845eff99 --- /dev/null +++ b/docs/release/backport-tasks/failed-ci.md @@ -0,0 +1,39 @@ +## Attribute a failed batch CI + +CI on the batch pull request runs the line's build and tests against every original the batch carries at once. When a check fails, this task finds the original the failure comes from and takes it out of the batch, so the rest can merge; the original goes to the next batch with the reason recorded. Read the [rules for every task](./README.md#rules-for-every-task) first. + +### Inputs + +- The open batch pull request for the line and its failing checks: `gh pr checks `, and the log of each failing job, `gh run view --log-failed`. +- The originals the batch carries, one unit each on its first-parent line: `git log --first-parent --reverse --format='%h %s' origin/..origin/backport-batch/-`. A merged pull request's unit is a merge whose message is the original's merge message, naming its number, and whose `Backported-from:` trailer names its merge commit on `main`; its commits are `git log --reverse --format='%h %an %s' ^1..^2`. A squash-merged one is one commit whose subject ends in its number. The files a unit changes are `git diff --name-only ^1 `. +- The line's own CI on its head, `gh run list --branch `, to tell a failure the batch brought from one the line already had. + +### Output + +The batch rebuilt without the culprits, by one `backport-batch run --defer` with `--push`, which records each culprit and the reason in the batch pull request. Nothing else: no commit by hand, no revert, no edit to the batch branch. + +### Steps + +1. Read the failing jobs and name what fails: the job, the package or test, and the first error. +2. When the same job fails the same way on the line's head, or the failure is the runner's (a timeout pulling an image, a lost runner, a registry error), the batch did not cause it: defer nothing, and comment on the batch pull request with the job and the line's run that shows the same failure. +3. Otherwise pin it on originals by their files: the originals that change the failing package, chart or test, or what it builds from. Take the fewest that explain the failure; when no original's files reach the failure, or every original's do, the culprit cannot be pinned from files alone: read the failure further before deferring anything. +4. Write the reason as one line naming the job and the failure, for example `e2e-apps: postgres never becomes Ready`. + +### Check before pushing + +```bash +/tmp/backport-batch run --line --defer N[,N] --reason ": " +``` + +It rebuilds the branch locally without those originals and prints the new body, exiting 0. It exits 2 and names the original when dropping one leaves a later commit that no longer applies; add that one to `--defer` only when its files reach the failure too, otherwise comment on the batch pull request naming it and stop. Then publish the same command with `--push`: + +```bash +/tmp/backport-batch run --line --defer N[,N] --reason ": " --push +``` + +It replaces the batch branch only if nobody pushed onto it since the rebuild read it, and updates the pull request body. CI runs again on the new head; if it fails again on another original, repeat from step 1. + +### Never + +- Defer an original the failure does not reach, to make the batch green by taking out more than the culprit. +- Push the rebuilt branch any other way, or push to it by hand. diff --git a/docs/release/backport-tasks/resolve-conflict.md b/docs/release/backport-tasks/resolve-conflict.md new file mode 100644 index 000000000..928bc8966 --- /dev/null +++ b/docs/release/backport-tasks/resolve-conflict.md @@ -0,0 +1,66 @@ +## Resolve a conflict + +The batch carries a pull request merged into `main` as its own commits, each picked with `git cherry-pick -x`, under a merge commit that names the original; a squash-merged one as one pick. When any of an original's commits conflicts, nothing of it stays on the batch, and the batch pull request lists it under **Deferred conflicts** with the commit that failed and its files, and in its `backport-batch` block. This task ports one such original onto the batch by hand, in the same shape. Read the [rules for every task](./README.md#rules-for-every-task) first. + +### Inputs + +- The open batch pull request for the line, its branch `backport-batch/-` and its head commit, as GitHub shows them: `gh pr view --json headRefName,headRefOid`. +- The original's number `N` from the **Deferred conflicts** list, and its merge commit on `main`: `gh pr view N --json mergeCommit --jq .mergeCommit.oid`. It has two parents when the pull request was merged, one when it was squashed. +- For a merged pull request, its own commits, oldest first, without the merges that brought `main` into its branch: `git rev-list --reverse --topo-order --no-merges ^1..^2`. +- The files the original changes, `git diff --name-only ^1 `, which are the only files the resolution may touch. + +### Output + +For a merged pull request, one merge commit on the batch head: its first parent is the head of the batch pull request, and its second parent ends a straight line of picks of the pull request's own commits, in their order, each made with `git cherry-pick -x` so it keeps its author, message and sign-offs and ends in the `(cherry picked from commit …)` line git appends. Its message is the original's merge message with a `Backported-from: ` trailer of its own. For a squash-merged pull request, one pick of its commit. Either way only files the original changes, and nothing else: no extra commit, no fix-up of an earlier pick. + +### Steps + +```bash +git fetch origin +git switch --detach +base=$(git rev-parse HEAD) +``` + +Pick the original's commits, oldest first; for a squash-merged pull request the one commit to pick is its merge commit on `main`: + +```bash +git -c core.hooksPath=/dev/null cherry-pick -x --signoff +``` + +When a pick conflicts, resolve each conflicted file so that it does on the line what that commit does on `main`, keeping the line's own code wherever the commit does not change it, and go on with the next commit: + +```bash +git add +git -c core.hooksPath=/dev/null cherry-pick --continue --no-edit +``` + +When the original builds on something the line does not have, a helper, a value or another pull request, the resolution would have to bring that along, which this task does not allow: abort with `git cherry-pick --abort`, go back with `git switch --detach "$base"`, and comment on the batch pull request which change the original depends on; it stays deferred until that change is on the line. A commit whose change the line already has stops the pick as empty; skip it with `git cherry-pick --skip`. + +For a merged pull request, join the picks to the batch head with the merge commit: + +```bash +tip=$(git rev-parse HEAD) +git switch --detach "$base" +{ git log -1 --format=%B ; printf '\nBackported-from: %s\n' ; } > /tmp/batch-merge-message +git -c core.hooksPath=/dev/null merge --no-ff -F /tmp/batch-merge-message "$tip" +``` + +### Check before pushing + +```bash +/tmp/backport-batch fold --line --original N --commit "$(git rev-parse HEAD)" +``` + +It prints `ok` and exits 0, or prints why not and exits 1. Push only on `ok`, as a fast-forward of the batch branch: + +```bash +git push origin HEAD:refs/heads/backport-batch/- +``` + +Resolve one original at a time and push each before starting the next: the next fold has to sit on the new head. CI on the batch pull request runs again on the push, `backport-batch verify` among it, and the next run of the batch lists the original as on the branch, with the picks the resolution changed marked as such. + +### Never + +- Push a commit `fold` did not print `ok` for, or push anywhere but the batch branch. +- Squash the original's commits into one pick, reorder them, or pick a commit of another pull request into the merge. +- Edit a file outside the original's files to make the pick build; that is the dependency case above. diff --git a/docs/release/backports.md b/docs/release/backports.md new file mode 100644 index 000000000..c613cc980 --- /dev/null +++ b/docs/release/backports.md @@ -0,0 +1,112 @@ +## Backports + +Backports are automatic. Everything that can go into a patch release goes in, and nobody decides it pull request by pull request: nobody requests a backport, and contributors never open backport pull requests. The one exception is a maintainer override, below. A batch job takes the fixes merged into `main` and cherry-picks them onto each `release-X.Y` line it runs for, following the rules on this page. + +### Getting a fix into a patch release + +Merge it into `main` with a `fix` title, for example `fix(postgres): keep the replica count on resize`. Nothing else is needed: if the change meets the rules below, the next batch picks it up for every [line the batch runs for](#which-lines-the-batch-runs-for). + +The batch takes a pull request's type from its title alone, so a fix that ships inside a `feat` or `refactor` pull request does not reach a patch release. Put the fix in a pull request of its own. + +### What the batch takes + +The batch takes a pull request from `main` into release line X.Y when all of these hold: + +1. It merged into `main` after `release-X.Y` branched, or after the line started under this policy if that is later (see [Where each line starts](#where-each-line-starts)), and the line does not carry it yet. +2. Its title type is `fix` or `security`, the type of a fix that comes through a security advisory. CI and e2e fixes count, because they keep the line's own CI working. The title is read once any leading `[Backport …]` prefixes are dropped, and a title that does not read as a Conventional Commits title is kept out. +3. It is not breaking: no `!` after the type or scope, and no `BREAKING CHANGE:` or `BREAKING-CHANGE:` footer at the start of any line of the description its merge into `main` recorded. A footer inside an HTML comment, where the pull request template quotes it, does not count. +4. It touches no platform migrations, anything under `packages/core/platform/images/migrations/migrations/`, and no `targetVersion` line of `packages/core/platform/values.yaml`, adds, removes or edits no CustomResourceDefinition document (a CRD in a test fixture under `tests/` or `testdata/` does not count, and neither does an edit to another document in the same file), and makes no change to the API that the [API review gate](../../.github/workflows/api-review-gate.yaml) calls sizeable: a new or removed API group or resource, or a schema change that can reject what the old schema accepted. Those wait for the next minor; a reworded description or a new optional field under `api/` does not. [Backporting code that touches monotonic counters](./backport-mechanics.md#backporting-code-that-touches-monotonic-counters) shows what a migration on a release line breaks. +5. Each of its commits picks cleanly with `git cherry-pick -x`, or its conflict is resolved during the batch, and the batch's build and tests pass. A pick that fails moves to the next batch, and the batch pull request lists it with the files that conflicted. +6. On an older line, it is already on the next newer line the batch runs for. Lines are processed newest first, so an older line takes only what the newer one already carries. + +The batch reads the pull requests off `main`'s first-parent history, one per merge commit or squash commit that names its pull request. A commit there that names none, a direct push for instance, and a merge of more than two parents are never taken; the batch pull request counts them. + +### What waits for the next minor + +`feat`, `chore` and `docs` pull requests, and dependency bumps (scope `deps`), ship with the next minor, not in a patch release; only a maintainer's `backport/include`, below, takes one. If a class of fixes should also wait for a minor, it gets a new rule on this page; a single pull request that the rules get wrong is what the overrides below are for. + +### Maintainer overrides + +A maintainer overrides the rules for one pull request with a label on the merged pull request on `main`: + +- `backport/include` takes it although rules 2 to 4 keep it out, a `feat` for instance, or although it merged before its line started under this policy. It still has to pick cleanly or have its conflict resolved, pass CI on the batch, and, on an older line, already be on the newer one, and a line that already carries it does not take it again. +- `backport/skip` keeps it out although the rules take it, and `backport-batch check` stops asking for it. An open batch that picked it before the label came keeps it until `backport-batch run --defer` takes it out; the batch pull request says so meanwhile. + +`backport/skip` wins when a pull request carries both. Each run reads the labels afresh, and the batch pull request lists every override and what became of it. + +### Which lines the batch runs for + +The batch runs for the two newest lines that have shipped a stable release, and for every line branched after the newest of them that has not shipped one yet. A line has shipped once its first stable `vX.Y.Z` tag exists, which only the merge of its promotion creates ([stable tags come from rc promotion](./stable-tags.md)); an rc, alpha or beta tag does not count. How many released lines it keeps is `released-lines` in the block below. + +With 1.6 and 1.7 released, cutting `v1.8.0-rc.1` creates `release-1.8` ([release candidates](./release-candidates.md)), and from the next run the batch runs for 1.6, 1.7 and 1.8. Once `v1.8.0` ships it runs for 1.7 and 1.8, and 1.6 takes no more batches. Rule 6 holds across all of them, newest first: while 1.8 is in its rc window, a fix reaches 1.7 only after the 1.8 batch that carries it has merged, and 1.6 only after 1.7's. + +During its rc window a line is frozen: every later rc is cut from `release-X.Y` itself, and `main` is open for the next minor. The freeze closes the line to `main`, not to the batch, which is how fixes reach it: a batch pull request against a frozen line is built, checked and merged like any other, and the next rc carries what it merged once `check --cut` passes (see [Running the batch](#running-the-batch)). A line that drops out keeps its open batch pull request as it is, and `backport-batch` no longer runs or checks that line; a maintainer merges the pull request or closes it. + +### Where each line starts + +The policy applies in full from `release-1.7` on: a line released before it took effect keeps the patch content its users chose it with, so it takes only fixes merged after the policy started, and an older one only with `backport/include`. The batch reads the block below from this page, on `main`: a line it lists starts on that day, and any other line starts where it branched from `main`. The block also gives the `batch-only` day and `released-lines`. + +From the `batch-only` day in the block, a change from `main` reaches a line only through the batch, and only the batch's picks show that a line carries it: a pick whose `(cherry picked from commit …)` line, the one git appends, names the original's merge commit or every commit of its own. A picked commit keeps its message and a batch merge carries the original's, so text in them, a `Backport of #N`, a `Backported-from:` trailer or an `-x` line the commit's author wrote, counts only on commits from before that day, where it records the hand backports and earlier batches. + +```backport-policy +# release-X.Y: YYYY-MM-DD, the day the line started under this policy +release-1.6: 2026-10-26 +# batch-only: YYYY-MM-DD, the day from which only the batch's picks show what a line carries +batch-only: 2026-10-26 +# released-lines: N, how many of the newest lines with a stable release the batch runs for +released-lines: 2 +``` + +### Fixes that exist only on a release line + +Some fixes go to a release line as a pull request of their own. Such pull requests are occasional, and maintainers open them: for a fix with no counterpart on `main`, for one that on `main` came as part of another pull request, or for one written for `main` rather than for the line. When the counterpart on `main` is a pull request of its own that the batch would take, the batch skips it as already on the line once picking it would change nothing there; otherwise a maintainer labels it `backport/skip`. + +A pull request opened against a `release-*` branch is closed automatically, with a comment that links to this page, when its author is a user account, not a bot, that GitHub names neither the repository's owner nor a member nor a collaborator, and whose access to the repository GitHub reports as read or none. Anyone with triage access or more keeps their pull request open. When GitHub's answer on the author's access cannot be read, the check closes nothing and fails, naming the read that failed. The check runs only when a pull request is opened, so a maintainer keeps a contributor's release-only fix by reopening it. [`hack/close-manual-backport.sh`](../../hack/close-manual-backport.sh) does this from [`pr-labeler.yaml`](../../.github/workflows/pr-labeler.yaml); run it with `--pr --dry-run` to see what it would decide for a given pull request when it is opened. + +### Cadence + +The batch runs for each of its lines twice a week, on Monday and Thursday: often enough that a fix reaches its line within days, rarely enough that nobody spends time on it every day, and early enough in the week that each run leaves working days to cut an rc, run its e2e and promote it before the weekend. Patch releases ship weekly. + +### Running the batch + +[`backport-batch`](../../cmd/backport-batch/README.md) builds the batch in a clean worktree, newest line first. The scheduled run below does this; by hand it is: + +```bash +go build ./cmd/backport-batch +./backport-batch run --line release-1.7 --continue --push +./backport-batch run --line release-1.6 --continue --push +``` + +A run by hand trusts the open batch pull request the scheduled run opened as `cozystack-ci[bot]` and continues it; `BACKPORT_BATCH_BOT` names another bot, and set empty trusts none. + +`./backport-batch lines` prints the lines, newest first. `run`, `check`, `fold` and `lines` read the policy block from this page on `main`, so until the page with its block is on `main` they need `--policy ` naming a ref that has it, and the scheduled run, which passes none, fails. + +`run` picks what the rules take onto `backport-batch/-` and opens a draft pull request against the line. A pull request merged into `main` with a merge commit goes onto the batch as its own commits, oldest first, each picked with `git cherry-pick -x --signoff` so it keeps its author, its message and its sign-offs and gains the sign-off of whoever runs the batch; commits that merged `main` into the pull request's branch are left out. A merge commit joins them to the batch: it carries the original's merge message from `main` and ends in a `Backported-from: ` trailer. A pull request squash-merged into `main` goes on as one pick. When any commit of an original conflicts, nothing of that original stays on the batch. The batch pull request is merged into the line with a merge commit, so all of this reaches the line as it is. Its body lists the originals with how many commits each brought and which of them differ from the commit they came from, the conflicts it deferred with the commit that failed and its files, and what it skipped under which rule. While that pull request is open, the next run continues it, `--continue` reading its head, rather than opening another. CI on the batch pull request runs the line's build and tests, and a maintainer merges it. Since rule 6 reads the newer line itself, an older line takes a fix in the batch after the newer line's batch with that fix has merged. + +A deferred conflict waits for the next batch. To land it sooner, pick the original's commits onto the batch head with `git cherry-pick -x`, resolving each conflict, and join them to the batch head with a merge commit that names the original; an original squash-merged into `main` is one pick of its commit there, with no merge. Check that merge or that pick with `./backport-batch fold --line --original --commit `, and push it onto the batch branch, as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes. CI runs `backport-batch verify` on every push to a batch pull request, which holds every original on it to the same shape. When CI on the batch pull request fails, the original it comes from is taken out with `./backport-batch run --line --defer --reason --push` and goes to the next batch, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the rest of the batch merges without it. + +Before a patch release or an rc is cut from a line, `./backport-batch check --line --cut` has to exit 0. A cut tags the line, so the check fails while a pull request the rules take is not on the line, including one that is only on the open batch pull request: merge the batch pull request first. It prints the URLs, and prints what the batch deferred as warnings, since deferring is a maintainer's call. Without `--cut`, as the scheduled run uses it, a change on the open batch pull request counts as carried. + +### Changes to workflows + +The scheduled run's token may not push a commit that changes `.github/workflows/`, so the scheduled run leaves such an original out (`run --no-workflow-files`). The batch pull request lists it under "Waiting for a maintainer", with the workflow files it changes, and so does the batch's comment. It is no conflict. The policy still takes it: `check` warns about it while it waits, and `check --cut` fails on it. When nothing else is picked there is no batch pull request to list it on, and `check` reports it as missing. + +A maintainer brings it over with their own token, one that may push workflow changes (the Workflows permission, or the classic `workflow` scope), by continuing the open batch without the flag: + +```bash +./backport-batch run --line --continue --push +``` + +That picks it onto the batch in the batch's shape, after what the batch carries, and the body stops listing it; later scheduled runs continue the batch on top. To cut without it instead, take it out with `./backport-batch run --line --defer --reason --push`. + +### The scheduled run + +[`backport-batch.yaml`](../../.github/workflows/backport-batch.yaml) runs [`hack/backport-batch-scheduled.sh`](../../hack/backport-batch-scheduled.sh) on Monday and Thursday. For each line the batch runs for, newest first, it continues the open batch pull request or opens one, runs `check`, and then `backport-batch notify`, which reports what waits on the batch. A batch open for three days or more, so one the previous run already saw, gets a comment on its pull request saying so and a line in the run's summary. The comment is one per pull request, updated in place on every run. It also lists the deferred conflicts, which a maintainer resolves as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows. A failed batch CI waits for a maintainer too, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the comment does not list it, and it shows on the batch pull request's checks. `hack/backport-batch-scheduled.sh --dry-run` does the same in the current worktree and changes nothing on GitHub. Its `check` sees the line without the batch the dry run just built, so it can exit 1 where the real run's check passes. + +The batch runs as the cozystack-ci GitHub App, which the release workflows already use, and needs no secret of its own. The job mints a token from `COZYSTACK_CI_APP_ID` and `COZYSTACK_CI_PRIVATE_KEY` for this repository alone, with Contents and Pull requests write and nothing more, and fails when either secret is missing. Its pushes start CI on the batch pull request, which a push with the job's own `GITHUB_TOKEN` would not. Its commits carry the App's bot, `cozystack-ci[bot]`, whose batch pull request and comment `backport-batch` trusts: `BACKPORT_BATCH_BOT`, which the job sets from the token, names it, and a run without the variable trusts `cozystack-ci[bot]` too. The App's installation has to grant this repository Contents and Pull requests read and write, which an organization admin sees among the organization's installed GitHub Apps. It has no Workflows permission, so a change to `.github/workflows/` waits for a maintainer, as [changes to workflows](#changes-to-workflows) describes. + +It also reads one repository setting, which a repository admin sets up: `BACKPORT_ALERT_TEAM`, a variable, the team or person the comment mentions, as `org/team` or a login. Unset, the comment mentions nobody: the batch pull request is a draft the App opened, so no one is notified of what waits unless they watch the repository. + +Optionally, once the `Backport batch shape` check runs from `main`, a ruleset makes it required on the release lines. It targets `release-*` and excludes `release-*.*.*`: `release-*` alone also matches the `release-X.Y.Z-rc.N` and `release-X.Y.Z` staging branches that [`tags.yaml`](../../.github/workflows/tags.yaml) and [`promote-rc.yaml`](../../.github/workflows/promote-rc.yaml) push, which have no batch on them, and a required check would refuse those pushes. "Do not require status checks on creation" stays on, since [`cut-prerelease.yaml`](../../.github/workflows/cut-prerelease.yaml) creates `release-X.Y` with a push at its first rc. + +To run the scheduled script for real by hand, export your own token as `BACKPORT_BATCH_TOKEN`: without `--dry-run` it uses that variable and no other credential. With a token that may push workflow changes, leave `--no-workflow-files` out and the run brings over what waits. From 9f8c88b4f168b99392f82f12065f769b092cd59e Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:40:12 +0000 Subject: [PATCH 159/518] feat(release): build the backport batch with backport-batch The rules need a tool that applies them the same way on every run and shows what it did. backport-batch reads main's first-parent history, judges each pull request by the policy block it reads from the page on main, and picks what qualifies onto one batch branch per line. Each original lands as its own commits under a merge that names it, so authors and sign-offs survive and the line's own history shows what it carries. check is the gate before a cut. fold holds a hand resolution of a conflict to the batch's shape, verify holds a whole batch to it, and notify reports what waits on the batch for a maintainer. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 103 ++++++ cmd/backport-batch/apisurface.go | 188 ++++++++++ cmd/backport-batch/assemble.go | 533 +++++++++++++++++++++++++++ cmd/backport-batch/assemble_test.go | 306 +++++++++++++++ cmd/backport-batch/batchbot_test.go | 85 +++++ cmd/backport-batch/body.go | 532 ++++++++++++++++++++++++++ cmd/backport-batch/body_test.go | 368 ++++++++++++++++++ cmd/backport-batch/check.go | 196 ++++++++++ cmd/backport-batch/check_test.go | 233 ++++++++++++ cmd/backport-batch/defer.go | 194 ++++++++++ cmd/backport-batch/defer_test.go | 380 +++++++++++++++++++ cmd/backport-batch/fixture_test.go | 405 ++++++++++++++++++++ cmd/backport-batch/fold_test.go | 332 +++++++++++++++++ cmd/backport-batch/git.go | 276 ++++++++++++++ cmd/backport-batch/github.go | 313 ++++++++++++++++ cmd/backport-batch/history.go | 371 +++++++++++++++++++ cmd/backport-batch/history_test.go | 330 +++++++++++++++++ cmd/backport-batch/injection_test.go | 239 ++++++++++++ cmd/backport-batch/lines.go | 246 +++++++++++++ cmd/backport-batch/main.go | 445 ++++++++++++++++++++++ cmd/backport-batch/notify.go | 213 +++++++++++ cmd/backport-batch/notify_test.go | 288 +++++++++++++++ cmd/backport-batch/onto_test.go | 140 +++++++ cmd/backport-batch/overrides_test.go | 247 +++++++++++++ cmd/backport-batch/policy.go | 144 ++++++++ cmd/backport-batch/policy_test.go | 212 +++++++++++ cmd/backport-batch/rules.go | 341 +++++++++++++++++ cmd/backport-batch/rules_test.go | 461 +++++++++++++++++++++++ cmd/backport-batch/shape.go | 439 ++++++++++++++++++++++ cmd/backport-batch/shape_test.go | 372 +++++++++++++++++++ cmd/backport-batch/verify.go | 139 +++++++ cmd/backport-batch/workflows_test.go | 249 +++++++++++++ 32 files changed, 9320 insertions(+) create mode 100644 cmd/backport-batch/README.md create mode 100644 cmd/backport-batch/apisurface.go create mode 100644 cmd/backport-batch/assemble.go create mode 100644 cmd/backport-batch/assemble_test.go create mode 100644 cmd/backport-batch/batchbot_test.go create mode 100644 cmd/backport-batch/body.go create mode 100644 cmd/backport-batch/body_test.go create mode 100644 cmd/backport-batch/check.go create mode 100644 cmd/backport-batch/check_test.go create mode 100644 cmd/backport-batch/defer.go create mode 100644 cmd/backport-batch/defer_test.go create mode 100644 cmd/backport-batch/fixture_test.go create mode 100644 cmd/backport-batch/fold_test.go create mode 100644 cmd/backport-batch/git.go create mode 100644 cmd/backport-batch/github.go create mode 100644 cmd/backport-batch/history.go create mode 100644 cmd/backport-batch/history_test.go create mode 100644 cmd/backport-batch/injection_test.go create mode 100644 cmd/backport-batch/lines.go create mode 100644 cmd/backport-batch/main.go create mode 100644 cmd/backport-batch/notify.go create mode 100644 cmd/backport-batch/notify_test.go create mode 100644 cmd/backport-batch/onto_test.go create mode 100644 cmd/backport-batch/overrides_test.go create mode 100644 cmd/backport-batch/policy.go create mode 100644 cmd/backport-batch/policy_test.go create mode 100644 cmd/backport-batch/rules.go create mode 100644 cmd/backport-batch/rules_test.go create mode 100644 cmd/backport-batch/shape.go create mode 100644 cmd/backport-batch/shape_test.go create mode 100644 cmd/backport-batch/verify.go create mode 100644 cmd/backport-batch/workflows_test.go diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md new file mode 100644 index 000000000..33ec9ee89 --- /dev/null +++ b/cmd/backport-batch/README.md @@ -0,0 +1,103 @@ +# backport-batch + +Builds the automatic backport batch for a release line and gates a patch cut on it. What goes into a batch is decided by the rules of the [backport policy](../../docs/release/backports.md) alone; this page describes how the tool applies them and does not restate them. + +## Usage + +```bash +go build ./cmd/backport-batch +./backport-batch run --line release-1.7 [--onto SHA] [--date YYYY-MM-DD] [--no-workflow-files] [--push] [--policy REF] +./backport-batch run --line release-1.7 --continue [--no-workflow-files] [--push] [--policy REF] +./backport-batch run --line release-1.7 --defer 4321[,4322] --reason "fails the e2e of the app" [--push] [--policy REF] +./backport-batch check --line release-1.7 [--cut] [--policy REF] +./backport-batch fold --line release-1.7 --original 4321 --commit SHA [--policy REF] +./backport-batch verify --line release-1.7 [--head SHA] +./backport-batch notify --line release-1.7 [--mention org/team|login] [--push] +./backport-batch lines [--policy REF] +``` + +[`hack/backport-batch-scheduled.sh`](../../hack/backport-batch-scheduled.sh) runs `lines`, then `run --continue --push`, `check` and `notify` for each line; [`backport-batch.yaml`](../../.github/workflows/backport-batch.yaml) runs it on schedule as the cozystack-ci App, with `--no-workflow-files`, which it hands to `run`. + +Requires `git` and an authenticated `gh` on PATH, and a remote called `origin`. Every command but `notify` fetches `origin` first; `notify` reads only GitHub. The batch runs only for the lines `lines` prints. + +`run`, `check`, `fold` and `lines` read the `backport-policy` block of the policy page from `origin/main`, or from the ref `--policy` names, which is how a change to the page is tried before it merges; `verify` and `notify` do not read it and take no `--policy`. A line the block lists takes nothing merged into `main` before the day it gives, unless the original carries `backport/include`; any other line starts at its fork point. A page without exactly one readable block stops the run with exit 2. + +Build the binary wherever an exit code matters: `go run` turns any non-zero exit into its own `1`, so under it a check that could not be completed looks the same as one that found something missing. + +### run + +Switches the current working tree to `backport-batch/-`, created at `origin/`, and picks onto it one original at a time, in the order they merged into `main`, as one unit each on the branch's first-parent line. A pull request merged into `main` with a merge commit `M` goes on as its own commits, `git rev-list --reverse --topo-order --no-merges M^1..M^2`, each picked with `git cherry-pick -x --signoff`, so each keeps its author, message and sign-offs and gains the `(cherry picked from commit …)` line; the merges that brought `main` into its branch are left out, since `main` already has what they brought. A merge commit then joins the picks to the batch: its first parent is the batch before the original, its second the last pick, and its message is `M`'s with a paragraph of its own, `Backported-from: M`. A pull request squash-merged into `main` goes on as one pick. When any commit of an original conflicts, the pick is aborted, the branch goes back to where it was before that original, and the original is recorded with the commit that failed and its files; the run goes on with the next one. A commit the branch already carries is skipped as empty, and an original none of whose commits changes anything is reported as carried. The tree has to be clean. A local branch of that name is reset to where the run starts; when that drops commits only it has, the run first prints its tip and the `git branch` command that restores them, and so does `--defer`. Signing follows your git configuration; hooks are off for the picks and the merge, so each commit carries its message as git writes it. + +The pull request body goes to stdout, built from the line's own `.github/PULL_REQUEST_TEMPLATE.md`: the originals on the branch, each with how many commits it brought and which of them differ from the commit they were picked from by `git patch-id`, a resolution or a pick git had to fit, the deferred conflicts with the commit that failed and its files, the skipped pull requests counted by rule with the lists collapsed, the originals' release notes, and a `backport-batch` block of JSON that lists the deferred conflicts for `check`. The title is `fix(backport): batch for , `. A body that would pass GitHub's 65536-byte limit leaves out its longest skipped lists, and the run prints them on stderr instead. + +`--onto SHA` continues a batch already pushed instead of starting a new one. SHA has to be the head the remote has for a batch branch of the line right now, read with `git ls-remote`, so a head someone pushed onto since is refused. What that branch carries stays, and only what it does not carry yet is picked on top, so the next push is a fast-forward. `--continue` takes SHA from the open batch pull request, and starts a new batch when none is open, which is what the scheduled run does. + +`--no-workflow-files` is for a token that may not push a change to `.github/workflows/`, which the scheduled run's App token may not. An original that changes anything there is not picked: the body lists it under "Waiting for a maintainer" with those files, and the `backport-batch` block records it with `"workflows": true`, so `check`, `notify` and a run that continues the batch know it waits. It is never a conflict. A run without the flag, a maintainer's with their own token, picks it like any other original, onto the open batch when it continues one. + +`--defer N[,N] --reason TEXT` takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off `origin/` without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got `backport/skip` after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the `backport-batch` block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. + +`--push` pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with `gh pr create --draft`, or updates the open batch pull request with `gh pr edit`. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the `run --onto --push` that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no `backport-batch` block either: someone wrote that body by hand, and the batch is theirs to merge or close. After `--defer` alone it replaces the branch, with `--force-with-lease=refs/heads/:`, where head is what `git ls-remote` reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. + +### check + +Exits 1 and prints the URL of every pull request the rules take that is neither on the line nor on the open batch pull request, 0 when there is none, and 2 when it cannot tell. A pull request that the open batch lists as deferred, a conflict or one taken out with `--defer`, prints as a warning and does not fail the check: deferring is a maintainer's call. So does one it lists as waiting for a maintainer because it changes workflows. + +`--cut` makes it the gate before a patch release or an rc is cut from a line. A cut tags the line, so a pull request that is only on the open batch pull request fails it too, with one line each: `#N is only on open batch PR #M, merge it before the cut`. One that waits there for a maintainer fails it as well, since nobody decided to leave it out: bring it over, or take it out with `run --defer` and a reason. Everything else is as without it. The scheduled run checks without `--cut`, since its batch is open by design. + +### fold + +Checks a hand-made resolution of a deferred conflict before it is pushed onto the batch, in the shape the run gives an original. For a merged pull request `--commit` is a merge whose first parent is the head of the open batch pull request and whose message names the original in exactly one `Backported-from:` trailer; its second parent ends a straight line from that head of picks of the pull request's own commits, in their order, each ending in the `(cherry picked from commit …)` line of the commit it came from, none claiming another commit of the same original, a commit of another pull request never among them. For a squash-merged one it is a single commit on that head whose `(cherry picked from commit …)` line names the original's commit on `main`. Either has to change something, and only files the original changes. The [conflict task](../../docs/release/backport-tasks/resolve-conflict.md) gives the commands. fold prints `ok` and exits 0, or prints why not and exits 1. It never commits or pushes; push one checked resolution at a time, since the next one has to sit on the new head. + +### verify + +Holds every unit of a batch to the shape fold checks for one, the clean picks as well as resolutions: it walks the batch's first-parent line from `origin/` to `--head`, or to the head of the open batch pull request, and requires each unit to be one original, merged into `main` after the line branched, in that shape, and no original twice. A commit that carries no original fails it, and so does a merge of the line into the batch. It prints one line per unit, and exits 0 when all pass, 1 when one does not, and 2 when it cannot tell. With `--head` it reads git alone, nothing from GitHub. fold runs only where someone resolved a conflict and chose to run it; [`pr-labeler.yaml`](../../.github/workflows/pr-labeler.yaml) runs verify on every push to a batch pull request, from a build of `main`, with the batch head fetched as data. + +### notify + +Reads the open batch pull request of the line, trusted as described below, and prints one line for each thing that waits on it: a batch open for three days or more, the shortest gap between two scheduled runs, so one the previous run already saw; the conflicts it deferred; and the originals waiting for a maintainer because they change workflows. Nothing open prints nothing. + +With `--push` it writes the same into one comment on the batch pull request: the conflicts, each to be resolved as the [conflict task](../../docs/release/backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows. + +The comment starts with `` and is one per pull request: a later run edits it in place rather than adding another, and once nothing waits it says so. Only a comment whose author the batch trusts counts as it, so one someone else wrote with the same first line is neither edited nor believed. It mentions `--mention`, an `org/team` or a login, with or without `@`, only while something waits. A failed batch CI is not among what it reports: the run has no trigger on the batch's CI finishing. + +### lines + +Prints the release lines the batch runs for, newest first: of the `release-X.Y` branches on `origin`, the newest that have a stable `vX.Y.Z` tag there, as many as `released-lines` in the policy block says, two when it says nothing, and every branch newer than the newest of them. The tags are read with `git ls-remote`; a tag with a pre-release suffix does not count. `run`, `check` and `fold` refuse any other line, and rule 6 holds each line to the next newer one among them. + +## Rule 4 and the API review gate + +A change to the API is held back only when the [API review gate](../../.github/workflows/api-review-gate.yaml) would call it sizeable. The gate is asked whenever an original changes something it parses: `pkg/apiserver/apiserver.go`, an ApplicationDefinition under `packages/system/*-rd/cozyrds/`, or a `.yaml` or `.yml` file under `packages/` or `internal/` that holds `CustomResourceDefinition` or `APIService` before or after the merge. An edit to the Go types under `api/` alone reaches none of them, so the gate would see no change. The batch calls the gate's own code, `apigate.LoadSnapshot` and `apigate.Classify` from `internal/apigate`, on the trees of the original's first parent and of its merge commit. It reads only what the gate reads, from `git ls-tree` and `git cat-file` into a temporary directory, so no attribute or export rule changes what the gate sees. A reworded description or a new optional field passes; a new or removed group or resource, or a schema change that can reject what the old schema accepted, does not. Migrations, `targetVersion` and CRDs keep their own checks: the gate sees only first-party `cozystack.io` CRDs, and only their sizeable changes, while a vendored operator's CRD is just as unsafe in a patch. + +The CRD check works document by document. In every changed manifest, a YAML, JSON or `.tpl` file or anything under `templates/` or `crds/`, outside `tests/` and `testdata/`, every document that names `CustomResourceDefinition` is read as YAML, which takes JSON too, and the CRDs among them are compared between the parent and the merge by `metadata.name`, compact JSON and quoted keys included. The change counts only when one was added, removed or edited, so an edit to the DaemonSet beside a CRD in one manifest does not, and neither does a sentence or a `lookup` call that merely names the kind. A document that declares `kind: CustomResourceDefinition` on a line of its own but cannot be read as an object, a CRD behind a Helm template guard for one, is compared by its text, so the check fails closed on it, and so does a `List` that holds a CRD. + +## Maintainer overrides + +`run`, `check` and `fold` read the merged pull requests that carry `backport/include` or `backport/skip`, through GitHub search, which answers at most 1000 per label; a listing that long is an error rather than a complete answer. `backport/skip` keeps a pull request out whatever the rules say, and wins over `backport/include`, which lifts rules 2 to 4 and nothing else: what the line already carries and, on the older line, what the newer one does not carry yet still decide. The batch pull request lists every overridden original and what became of it. Adding a label takes triage access to the repository, which is the trust the labels rest on; who added one is not checked. + +## What counts as already there + +An original is on a line when any of these holds, and the batch does not pick it again: + +- its merge commit is reachable from the line; +- picks on the line name its merge commit, or every commit of its own, in the `(cherry picked from commit …)` line git appended to them; +- picking it onto the line would change nothing. + +A pick is known by the line git appends: the last `(cherry picked from commit …)` line of the message's last paragraph, with nothing after it but trailers such as a sign-off. A picked commit keeps its message, and a batch merge carries the original's merge message, so whatever else a message says was written by someone the batch does not answer for: a line in it naming an original, as an `-x` line, a `Backported-from:` trailer or `Backport of #N`, does not count. On commits from before the policy's `batch-only` day it does, since those record the hand backports and earlier batches: + +- a `(cherry picked from commit …)` line anywhere in the trailer block naming its merge commit or, across commits, every commit of its own; +- a `Backported-from:` trailer naming its merge commit; +- `Backport of #N`, which is what a hand backport merged as one commit carries. + +An original is on the open batch pull request when its commits picked there show it the same way. The same rules on the newer line decide rule 6. + +Only the line's own commits count, the ones after the point where it branched from `main`, never a pull request against it: an open backport pull request, its title and its body are anyone's to write. The tool trusts the line's history because only maintainers merge into `release-*`; if that ever changes, a commit that names an original without carrying all of it would hide the rest. The batch pull request counts only when it comes from a `backport-batch/-` branch of this repository, not a fork, and was opened by the bot the scheduled batch runs as, the exact login `BACKPORT_BATCH_BOT` names, `cozystack-ci[bot]` when it is unset and no bot when it is set empty, or by someone whose permission on the repository GitHub reports as write or admin, and only at the head GitHub reports for it. No other bot counts, GitHub Actions' own included, since anything holding a workflow's token writes as it. So a maintainer's run by hand continues the scheduled batch with nothing set, rather than taking its pull request for a look-alike and opening a second one. The notify comment is trusted the same way. A permission GitHub does not answer stops the command with exit 2 rather than count the pull request or the comment out, which would open a second of either. The author association is not enough: an organization member or an invited collaborator can hold read access only. With more than one such pull request open the tool refuses to go on. + +Titles and bodies are read from the merge commits on `main`, which hold them as a maintainer merged them; the live pull request stays editable by its author. + +## What CI verifies + +The tool does not build or test anything. CI on the batch pull request runs the line's own build and tests against what was picked, and `verify` against its shape, and a maintainer reviews and merges it. A deferred conflict waits for the next batch unless someone resolves it, checks the resolution with `fold` and pushes it, and then CI runs again. + +## Untrusted text + +Titles, release notes and file names in the body come from whoever opened the originals. A title, a reason or a file name goes into a code span whose fence no backtick inside can close, and the release notes go into the `release-note` block with every run of three backticks or tildes defused, so GitHub acts on no mention, reference, link or HTML in them. Inside that block, in case a note is copied out of it, links lose any target outside this repository, and closing keywords, cross-repository references and mentions have their `@`, `#` and `://` swapped for fullwidth signs, which act on nothing however the backticks around them pair up. Zero-width, bidi and control characters are dropped first, and everything the tool prints passes the same filter, so nothing it relays can drive a terminal. Commit messages are read through `git cat-file --batch`, which gives each object's length, and end at their first NUL as `git log` shows them, so a message cannot pass for a second commit or hide a trailer from a reader of the log. Every value handed to git comes after `--end-of-options` or `--`, refs pass `git check-ref-format`, SHAs have to be 40 hex digits, and nothing runs through a shell. diff --git a/cmd/backport-batch/apisurface.go b/cmd/backport-batch/apisurface.go new file mode 100644 index 000000000..706e4be62 --- /dev/null +++ b/cmd/backport-batch/apisurface.go @@ -0,0 +1,188 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "os" + "path" + "path/filepath" + "slices" + "strconv" + "strings" + + "github.com/cozystack/cozystack/internal/apigate" +) + +// Rule 4's API clause is the API review gate's own verdict: the same +// apigate.LoadSnapshot and apigate.Classify that cmd/api-gate runs in +// api-review-gate.yaml, on the trees of the original's first parent and of its +// merge, so a change the gate passes as additive or descriptive is no reason to +// hold a fix back. + +const ( + apiserverGo = "pkg/apiserver/apiserver.go" + // apigate reads an ApplicationDefinition wherever this matches, whatever + // the file holds. + cozyRDGlob = "packages/system/*-rd/cozyrds/*.yaml" + // apigate skips a YAML file larger than this; reading it would be wasted. + apiFileLimit = 8 << 20 +) + +// apiYAMLRoots hold all YAML apigate.LoadSnapshot parses: the +// ApplicationDefinitions and every first-party CRD or APIService. +var apiYAMLRoots = []string{"packages", "internal"} + +// apiInputs are what apigate.LoadSnapshot reads of a checkout: the YAML roots +// and the apiserver's storage registrations. +var apiInputs = append(slices.Clone(apiYAMLRoots), apiserverGo) + +// apiMarkers are mightBeAPIManifest's: apigate parses no other YAML file that +// holds neither. +var apiMarkers = []string{"-e", "CustomResourceDefinition", "-e", "APIService"} + +var apiSnapshots = map[string]apigate.Snapshot{} + +// apiSnapshot is apigate's snapshot of commit, built from a temporary tree +// holding only what it reads. The blobs come from ls-tree and cat-file rather +// than a checkout or git archive: no attribute filters or export rules apply, +// and a symlink is never followed out of the tree. +func apiSnapshot(commit string) (apigate.Snapshot, error) { + if snap, ok := apiSnapshots[commit]; ok { + return snap, nil + } + out, err := git(append([]string{"ls-tree", "-r", "-z", "-l", "--full-tree", "--end-of-options", commit, "--"}, apiInputs...)...) + if err != nil { + return nil, err + } + paths := map[string]string{} + var blobs []string + for _, entry := range splitNUL(out) { + meta, name, ok := strings.Cut(entry, "\t") + f := strings.Fields(meta) + if !ok || len(f) != 4 { + return nil, fmt.Errorf("ls-tree %.12s: unreadable entry %q", commit, entry) + } + mode, kind, blob := f[0], f[1], f[2] + size, _ := strconv.Atoi(f[3]) + ext := path.Ext(name) + switch { + case kind != "blob" || (mode != "100644" && mode != "100755"): + case name != apiserverGo && ext != ".yaml" && ext != ".yml": + case size > apiFileLimit: + case checkSHA(blob) != nil || !filepath.IsLocal(filepath.FromSlash(name)): + return nil, fmt.Errorf("ls-tree %.12s: unreadable entry %q", commit, entry) + default: + paths[name] = blob + blobs = append(blobs, blob) + } + } + contents, err := readObjects(dedupe(blobs), "blob") + if err != nil { + return nil, err + } + dir, err := os.MkdirTemp("", "backport-batch-api-") + if err != nil { + return nil, err + } + defer os.RemoveAll(dir) + for name, blob := range paths { + file := filepath.Join(dir, filepath.FromSlash(name)) + if err := os.MkdirAll(filepath.Dir(file), 0o700); err != nil { + return nil, err + } + if err := os.WriteFile(file, contents[blob], 0o600); err != nil { + return nil, err + } + } + snap, err := apigate.LoadSnapshot(dir) + if err != nil { + return nil, fmt.Errorf("the API at %.12s: %w", commit, err) + } + apiSnapshots[commit] = snap + return snap, nil +} + +// readsAPI reports whether the change reaches anything apigate.LoadSnapshot +// parses at either end. A change to anything else leaves both snapshots alike, +// so the gate is not asked. A binary file is searched as well, since apigate +// reads one like any other; a file over its size limit is still asked about, +// which costs a snapshot and changes no verdict. +func readsAPI(parent, merge string, changes []fileChange) (bool, error) { + var yamls []string + for _, c := range changes { + p := c.path + if p == apiserverGo { + return true, nil + } + if ext := path.Ext(p); ext != ".yaml" && ext != ".yml" { + continue + } + if !slices.ContainsFunc(apiYAMLRoots, func(root string) bool { return strings.HasPrefix(p, root+"/") }) { + continue + } + if rd, _ := path.Match(cozyRDGlob, p); rd { + return true, nil + } + yamls = append(yamls, p) + } + for _, rev := range []string{parent, merge} { + found, err := grepFiles(rev, yamls, apiMarkers...) + if err != nil || len(found) > 0 { + return len(found) > 0, err + } + } + return false, nil +} + +// sizeableAPIChange says why the API review gate would call the original's +// change to the API sizeable, or "" when it would not. +func sizeableAPIChange(o *original) (string, error) { + parent, err := resolve(o.merge + "^1") + if err != nil { + return "", err + } + base, err := apiSnapshot(parent) + if err != nil { + return "", err + } + head, err := apiSnapshot(o.merge) + if err != nil { + return "", err + } + findings := apigate.Classify(base, head) + if len(findings) == 0 { + return "", nil + } + why := fmt.Sprintf("%s: %s", findings[0].Category, findings[0].Detail) + if len(findings) > 1 { + why += fmt.Sprintf(", and %d more", len(findings)-1) + } + return why, nil +} + +func dedupe(list []string) []string { + seen := map[string]bool{} + var out []string + for _, s := range list { + if !seen[s] { + seen[s] = true + out = append(out, s) + } + } + return out +} diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go new file mode 100644 index 000000000..ef26aace8 --- /dev/null +++ b/cmd/backport-batch/assemble.go @@ -0,0 +1,533 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "errors" + "fmt" + "os" + "slices" + "strings" + "time" +) + +// pick is the outcome of picking one original onto the batch branch. +type pick struct { + v *verdict + status string // picked, conflict, empty, held out by --defer, or waiting for a maintainer + commit string + files []string + // failed is the commit whose pick conflicted. + failed string + // more counts the conflicting files a recorded conflict no longer names. + more int + reason string +} + +// workflowsDir is where GitHub reads workflows from. A token without the +// workflows permission, a GitHub App's among them, cannot push a commit that +// changes anything under it. +const workflowsDir = ".github/workflows/" + +// workflowFiles lists the workflow files the commits change, each against its +// parent: a push carries every pick, so a change one commit makes and a later +// one takes back is in it although the pull request's merge changes nothing +// there. +func workflowFiles(commits []string) ([]string, error) { + if len(commits) == 0 { + return nil, nil + } + out, err := gitInput(strings.Join(commits, "\n")+"\n", "diff-tree", "--stdin", "--no-commit-id", "--name-only", "-r", "-z", "--no-renames") + if err != nil { + return nil, err + } + var files []string + for _, f := range splitNUL(out) { + if strings.HasPrefix(f, workflowsDir) && !slices.Contains(files, f) { + files = append(files, f) + } + } + return files, nil +} + +func batchBranch(l line, date time.Time) string { + return "backport-batch/" + l.branch() + "-" + date.Format("20060102") +} + +func (a *app) run(args []string) int { + fs := a.policyFlags("run") + lineArg := fs.String("line", "", "release line, release-X.Y") + onto := fs.String("onto", "", "head of the pushed batch branch to continue") + cont := fs.Bool("continue", false, "continue the open batch pull request, or start a batch when none is open") + dateArg := fs.String("date", "", "batch date, YYYY-MM-DD (default: today, UTC)") + deferArg := fs.String("defer", "", "originals to take out of the open batch, N[,N]; rebuilds it without them") + noWorkflows := fs.Bool("no-workflow-files", false, "leave an original that changes "+workflowsDir+" to a maintainer, for a token that cannot push one") + reasonArg := fs.String("reason", "", "why --defer takes them out, recorded in the batch body") + push := fs.Bool("push", false, "push the branch and open or update the draft batch PR") + if err := parse(fs, args); err != nil { + return 2 + } + l, err := parseLine(*lineArg) + if err != nil { + return a.fail(err) + } + date := a.now().UTC() + if *dateArg != "" { + if date, err = time.Parse(time.DateOnly, *dateArg); err != nil { + return a.fail(fmt.Errorf("--date %q is not YYYY-MM-DD", *dateArg)) + } + } + if *onto != "" { + if err := checkSHA(*onto); err != nil { + return a.fail(fmt.Errorf("--onto: %w", err)) + } + } + var deferred []int + reason := "" + switch { + case *deferArg != "" || *reasonArg != "": + if deferred, reason, err = parseDefer(*deferArg, *reasonArg); err != nil { + return a.fail(err) + } + if *onto != "" || *cont || *dateArg != "" { + return a.fail(errors.New("--defer rebuilds the open batch pull request; it takes no --onto, --continue or --date")) + } + case *cont && (*onto != "" || *dateArg != ""): + return a.fail(errors.New("--continue reads the batch to continue from the open batch pull request; it takes no --onto or --date")) + } + if err := cleanTree(); err != nil { + return a.fail(err) + } + if err := fetch(); err != nil { + return a.fail(err) + } + b, err := a.prepare(l) + if err != nil { + return a.fail(err) + } + if deferred != nil { + return a.rebuild(b, deferred, reason, *push) + } + if *cont && b.pr != nil { + *onto = b.prHead + } + + branch, start := batchBranch(l, date), b.lineHead + var carried *history + if *onto != "" { + if branch, err = ontoBranch(l, *onto, *dateArg != "", branch); err != nil { + return a.fail(err) + } + if carried, err = readPicks(*onto, b.lineHead, b.textUntil); err != nil { + return a.fail(err) + } + start = *onto + // The batch keeps the date it was opened under, in its title too. + if date, err = branchDate(branch); err != nil { + return a.fail(err) + } + } + if err := checkBranch(branch); err != nil { + return a.fail(err) + } + // Continuing the open batch keeps out what was deferred from it by hand; + // a new batch tries those again. + held := map[int]deferredEntry{} + if carried != nil && b.pr != nil && b.pr.Head.Ref == branch { + if held, err = parseDeferred(b.pr.Body, l); err != nil { + return a.fail(fmt.Errorf("batch pull request #%d: %w", b.pr.Number, err)) + } + } + if err := a.warnReset(branch, start); err != nil { + return a.fail(err) + } + if _, err := git("switch", "--quiet", "--no-track", "-C", branch, "--end-of-options", start); err != nil { + return a.fail(err) + } + + r := newReport(b) + if carried != nil { + if r.stale, err = keptOut(b, carried); err != nil { + return a.fail(err) + } + } + for _, v := range b.qualifying() { + switch { + case carried != nil && found(carried, v): + r.onBranch = append(r.onBranch, v) + continue + case held[v.o.number].Reason != "": + r.held = append(r.held, pick{v: v, status: "held", reason: held[v.o.number].Reason}) + continue + case v.onBatch != "": + v.rule, v.reason = ruleOnLine, fmt.Sprintf("on the open batch pull request #%d", b.pr.Number) + r.skipped[ruleOnLine] = append(r.skipped[ruleOnLine], v) + continue + } + if *noWorkflows { + files, err := workflowFiles(v.commits) + if err != nil { + return a.fail(err) + } + if len(files) > 0 { + r.waiting = append(r.waiting, pick{v: v, status: "waiting", files: files}) + continue + } + } + p, err := pickOriginal(v) + if err != nil { + return a.fail(err) + } + switch p.status { + case "picked": + r.onBranch = append(r.onBranch, v) + case "conflict": + r.deferred = append(r.deferred, p) + case "empty": + v.rule, v.reason = ruleOnLine, "the earlier picks already carry it" + r.skipped[ruleOnLine] = append(r.skipped[ruleOnLine], v) + } + } + if *onto != "" { + head, err := resolve("HEAD") + if err != nil { + return a.fail(err) + } + descends, err := isAncestor(*onto, head) + if err != nil { + return a.fail(err) + } + if !descends { + return a.fail(fmt.Errorf("%.12s does not descend from --onto %.12s", head, *onto)) + } + } + return a.finish(b, r, branch, date, *push, false) +} + +// keptOut lists the originals with backport/skip that the batch picked before +// the label came: a run never takes a commit off the batch, --defer does. +func keptOut(b *batch, carried *history) ([]*verdict, error) { + var out []*verdict + for _, v := range b.verdicts { + if v.label != labelSkip { + continue + } + if err := b.load(v); err != nil { + return nil, err + } + if found(carried, v) { + out = append(out, v) + } + } + return out, nil +} + +func found(h *history, v *verdict) bool { + c, _ := h.found(v.o, v.commits) + return c != "" +} + +// newReport starts the report of a run with what the rules left out and what +// the labels overrode. +func newReport(b *batch) *report { + r := &report{line: b.line, newer: b.newer, skipped: map[int][]*verdict{}, other: len(b.other), start: b.start} + for _, v := range b.verdicts { + if v.rule != 0 { + r.skipped[v.rule] = append(r.skipped[v.rule], v) + } + if v.label != "" { + r.overrides = append(r.overrides, v) + } + } + return r +} + +// finish prints the body and what the run did, and publishes the branch when +// asked to. +func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, rebuilt bool) int { + head, err := resolve("HEAD") + if err != nil { + return a.fail(err) + } + if r.shapes, err = summarize(b, head, r.onBranch); err != nil { + return a.fail(err) + } + title := prTitle(b.line, date.Format(time.DateOnly)) + template, err := lineTemplate(b.lineHead) + if err != nil { + return a.fail(err) + } + body, left := r.body(template) + fmt.Fprint(a.stdout, body) + skipped := 0 + for _, items := range r.skipped { + skipped += len(items) + } + for _, rule := range left { + for _, v := range r.skipped[rule] { + fmt.Fprintf(a.stderr, "skipped by rule %d: %s %s (%s)\n", rule, v.o.url(), terminalSafe(v.o.title), terminalSafe(v.reason)) + } + } + for _, p := range r.waiting { + fmt.Fprintf(a.stderr, "waiting for a maintainer: %s %s changes %s\n", p.v.o.url(), terminalSafe(p.v.o.title), terminalSafe(strings.Join(p.files, ", "))) + } + fmt.Fprintf(a.stderr, "%s at %.12s: %d on the branch, %d deferred, %d waiting for a maintainer, %d skipped\ntitle: %s\n", + branch, head, len(r.onBranch), len(r.deferred)+len(r.held), len(r.waiting), skipped, title) + if !push { + return 0 + } + if head == b.lineHead && b.pr == nil { + // Picking nothing is not a failure of the run; check still reports + // whatever the line lacks. + fmt.Fprintf(a.stderr, "%s carries nothing beyond %s; nothing to push\n", branch, b.line.branch()) + return 0 + } + if err := a.publish(b, branch, head, title, body, rebuilt); err != nil { + return a.fail(err) + } + return 0 +} + +// branchDate is the date a batch branch was opened under. +func branchDate(branch string) (time.Time, error) { + m := batchBranchRE.FindStringSubmatch(branch) + if m == nil { + return time.Time{}, fmt.Errorf("%q is not a batch branch", branch) + } + return time.Parse("20060102", m[2]) +} + +// cleanTree refuses a working tree with changes or an operation in progress: +// run switches it to the batch branch. +func cleanTree() error { + out, err := git("status", "--porcelain", "--untracked-files=no") + if err != nil { + return err + } + if strings.TrimSpace(out) != "" { + return errors.New("the working tree has changes; run in a clean worktree") + } + for _, name := range []string{"CHERRY_PICK_HEAD", "MERGE_HEAD", "REVERT_HEAD", "REBASE_HEAD", "rebase-merge", "rebase-apply", "sequencer"} { + p, err := git("rev-parse", "--git-path", name) + if err != nil { + return err + } + if _, err := os.Stat(strings.TrimSpace(p)); err == nil { + return fmt.Errorf("an operation is in progress (%s); finish or abort it first", name) + } + } + return nil +} + +// warnReset says, before switch -C moves the local branch to start, which tip +// the move drops commits of and how to get them back: a clean tree says +// nothing about commits only a local branch has. +func (a *app) warnReset(branch, start string) error { + old, err := lookup("refs/heads/" + branch) + if err != nil || old == "" || old == start { + return err + } + kept, err := isAncestor(old, start) + if err != nil || kept { + return err + } + fmt.Fprintf(a.stderr, "resetting the local %s from %s, dropping commits %.12s does not carry; git branch %s-%.12s %s restores them\n", branch, old, start, branch, old, old) + return nil +} + +// ontoBranch is the pushed batch branch whose head --onto names, read from the +// remote itself: a head that moved since, or that no batch branch has, is +// refused rather than continued. +func ontoBranch(l line, onto string, dated bool, dateBranch string) (string, error) { + heads, err := remoteBatchHeads(l) + if err != nil { + return "", err + } + var found []string + for branch, sha := range heads { + if sha == onto && (!dated || branch == dateBranch) { + found = append(found, branch) + } + } + switch { + case len(found) == 0 && dated: + return "", fmt.Errorf("--onto %.12s is not the head of %s on %s (%.12s)", onto, dateBranch, remote, heads[dateBranch]) + case len(found) == 0: + return "", fmt.Errorf("--onto %.12s is not the head of any batch branch for %s on %s", onto, l.branch(), remote) + case len(found) > 1: + return "", fmt.Errorf("--onto %.12s heads more than one batch branch; name one with --date", onto) + } + have, err := lookup(onto) + if err != nil { + return "", err + } + if have == "" { + return "", fmt.Errorf("--onto %.12s is not in this repository; fetch it first", onto) + } + return found[0], nil +} + +// remoteBatchHeads lists the batch branches for l on the remote, read with +// ls-remote rather than from the possibly stale remote-tracking refs. +func remoteBatchHeads(l line) (map[string]string, error) { + out, err := git("ls-remote", "--refs", "--heads", "--end-of-options", remote) + if err != nil { + return nil, err + } + heads := map[string]string{} + for ln := range strings.Lines(out) { + sha, ref, ok := strings.Cut(strings.TrimSpace(ln), "\t") + branch, isHead := strings.CutPrefix(ref, "refs/heads/") + if !ok || !isHead || checkSHA(sha) != nil { + continue + } + if m := batchBranchRE.FindStringSubmatch(branch); m != nil && m[1] == l.branch() { + heads[branch] = sha + } + } + return heads, nil +} + +// applyPick runs one cherry-pick onto HEAD and leaves no operation in progress +// behind it: the outcome is picked, conflict, with the conflicting files, or +// empty. +func applyPick(args ...string) (string, []string, error) { + stdout, stderr, code, err := gitStatus(args...) + if err != nil { + return "", nil, err + } + if code == 0 { + return "picked", nil, nil + } + out, err := git("diff", "--name-only", "-z", "--diff-filter=U") + if err != nil { + return "", nil, err + } + if files := splitNUL(out); len(files) > 0 { + _, err := git("cherry-pick", "--abort") + return "conflict", files, err + } + if msg := stdout + stderr; strings.Contains(msg, "is now empty") || strings.Contains(msg, "nothing to commit") { + _, err := git("cherry-pick", "--skip") + return "empty", nil, err + } + failed := fmt.Errorf("failed: %s", strings.TrimSpace(stderr)) + if _, err := git("cherry-pick", "--abort"); err != nil { + return "", nil, errors.Join(failed, err) + } + return "", nil, failed +} + +// lineTemplate is the line's own pull request template, so the batch body has +// the sections reviewers of that line expect. +func lineTemplate(lineHead string) (string, error) { + const template = ".github/PULL_REQUEST_TEMPLATE.md" + out, err := git("--literal-pathspecs", "ls-tree", "--name-only", "--end-of-options", lineHead, "--", template) + if err != nil { + return "", err + } + if strings.TrimSpace(out) == "" { + return "## What this PR does\n\n### Release note\n\n```release-note\n\n```\n", nil + } + return git("show", "--end-of-options", lineHead+":"+template) +} + +// publish pushes the batch branch, then opens the draft batch pull request or +// updates the open one. A run pushes only a fast-forward of what the remote +// has. A rebuild replaces the branch, and only the head it was rebuilt from: +// the lease is that head, as ls-remote reads it now, so a commit pushed since +// is never dropped. +func (a *app) publish(b *batch, branch, head, title, body string, rebuilt bool) error { + if b.pr != nil && b.pr.Head.Ref != branch { + return fmt.Errorf("the open batch pull request #%d is on %s; continue it with --onto %s", b.pr.Number, b.pr.Head.Ref, b.prHead) + } + // A body without the block was written by hand, and the tool's would + // replace it whole, title included; such a batch is the maintainer's. + if b.pr != nil && !strings.Contains("\n"+strings.ReplaceAll(b.pr.Body, "\r\n", "\n")+"\n", "\n"+deferredFence+"\n") { + return fmt.Errorf("batch pull request #%d has no %s block, so backport-batch did not write it; merge or close it, and the next run opens a batch of its own", b.pr.Number, deferredFence) + } + if head == b.lineHead { + if rebuilt { + return fmt.Errorf("without what is deferred, %s carries nothing beyond %s; close batch pull request #%d instead", branch, b.line.branch(), b.pr.Number) + } + return fmt.Errorf("%s carries nothing beyond %s; nothing to push", branch, b.line.branch()) + } + heads, err := remoteBatchHeads(b.line) + if err != nil { + return err + } + pushed, onRemote := heads[branch] + lease := "" + switch { + case rebuilt: + if !onRemote || pushed != b.prHead { + return fmt.Errorf("%s on %s is at %.12s, not at %.12s, which the rebuild started from; run again", branch, remote, pushed, b.prHead) + } + lease = pushed + case onRemote && pushed != head: + have, err := lookup(pushed) + if err != nil { + return err + } + descends := false + if have != "" { + if descends, err = isAncestor(pushed, head); err != nil { + return err + } + } + if !descends { + refusal := fmt.Sprintf("%s on %s is at %.12s, which %.12s does not descend from; refusing a non-fast-forward push", branch, remote, pushed, head) + // A run that stopped between its push and opening the pull request + // leaves the branch, and the next run that day builds it anew. + if b.pr == nil { + refusal += fmt.Sprintf("; no batch pull request is open on it, and backport-batch run --line %s --onto %s --push continues it and opens one", b.line.branch(), pushed) + } + return errors.New(refusal) + } + } + if pushed != head { + if _, err := git(pushArgs(branch, head, lease)...); err != nil { + return err + } + } + f, err := os.CreateTemp("", "backport-batch-body-*.md") + if err != nil { + return err + } + defer os.Remove(f.Name()) + if _, err := f.WriteString(body); err != nil { + f.Close() + return err + } + if err := f.Close(); err != nil { + return err + } + if b.pr == nil { + return a.gh.createPull(b.line.branch(), branch, title, f.Name()) + } + return a.gh.editPull(b.pr.Number, title, f.Name()) +} + +// pushArgs pushes head to branch: as a fast-forward, which the remote checks, +// or, given a lease, over exactly that commit and no other. +func pushArgs(branch, head, lease string) []string { + args := []string{"push", "--quiet"} + if lease != "" { + args = append(args, "--force-with-lease=refs/heads/"+branch+":"+lease) + } + return append(args, "--end-of-options", remote, head+":refs/heads/"+branch) +} diff --git a/cmd/backport-batch/assemble_test.go b/cmd/backport-batch/assemble_test.go new file mode 100644 index 000000000..513ae6ff2 --- /dev/null +++ b/cmd/backport-batch/assemble_test.go @@ -0,0 +1,306 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "os" + "path/filepath" + "slices" + "strings" + "testing" +) + +// A merged pull request lands as its own commits, each picked with -x and +// keeping its author and sign-offs, under a merge commit that carries the +// merge message from main and names it; a squash lands as one pick; a merge +// of main into the pull request's branch is left out; all in main's merge +// order. +func TestRunPicksInMergeOrder(t *testing.T) { + f := newFixture(t) + f.lines() + signoff := "Signed-off" + "-by: " + contributor + mc5 := f.prCommits(5, "fix(app): five", "Body.\n\n```release-note\nfix(app): five is fixed\n```", + change{files: map[string]string{"app/five.txt": "5\n"}, author: contributor, message: "fix(app): start five\n\n" + signoff}, + change{files: map[string]string{"app/five.txt": "5, done\n", "app/five-more.txt": "more\n"}, author: contributor, message: "fix(app): finish five\n\n" + signoff}) + sq6 := f.squash(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.pr(7, "feat(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + mc8 := f.prCommits(8, "fix(app): eight", "", + change{files: map[string]string{"app/eight.txt": "8\n"}}, + change{mergeMain: true}, + change{files: map[string]string{"app/eight.txt": "8, after main\n"}}) + + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.git("branch", "--show-current"); got != batchBranch1 { + t.Fatalf("on %q, want %s", got, batchBranch1) + } + want := []string{"fix(app): five (#5)", "fix(app): six (#6)", "fix(app): eight (#8)"} + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, want) { + t.Errorf("the batch carries %q, want %q", got, want) + } + + line := f.git("rev-parse", "origin/release-1.1") + u5 := f.git("rev-parse", "HEAD~2") + if got := f.git("rev-parse", u5+"^1"); got != line { + t.Errorf("#5's merge sits on %.12s, want the line's head %.12s", got, line) + } + if msg := f.git("log", "-1", "--format=%B", u5); !strings.HasSuffix(msg, "\n\n"+batchTrailer+": "+mc5) || !strings.HasPrefix(msg, "fix(app): five (#5)") { + t.Errorf("#5's merge does not carry the merge message and name %s:\n%s", mc5, msg) + } + own := strings.Fields(f.git("rev-list", "--reverse", mc5+"^1.."+mc5+"^2")) + picks := strings.Fields(f.git("rev-list", "--reverse", u5+"^2", "^"+u5+"^1")) + if len(picks) != 2 || len(own) != 2 { + t.Fatalf("#5 brought %d picks of its %d commits, want 2", len(picks), len(own)) + } + for i, c := range picks { + msg := f.git("log", "-1", "--format=%B", c) + if got := f.git("log", "-1", "--format=%an <%ae>", c); got != contributor { + t.Errorf("pick %d is authored by %q, want %q", i, got, contributor) + } + if !strings.Contains(msg, signoff+"\n(cherry picked from commit "+own[i]+")\nSigned-off-by: Batch Test ") { + t.Errorf("pick %d lacks the contributor's sign-off, its -x line for %s or the batch's sign-off:\n%s", i, own[i], msg) + } + } + + u6 := f.git("rev-parse", "HEAD~1") + if parents := strings.Fields(f.git("rev-list", "--parents", "-n", "1", u6)); len(parents) != 2 || parents[1] != u5 { + t.Errorf("#6 is %v, want one commit on #5's merge", parents) + } + if msg := f.git("log", "-1", "--format=%B", u6); !strings.Contains(msg, "(cherry picked from commit "+sq6+")") { + t.Errorf("#6's pick lacks its -x line:\n%s", msg) + } + + picks8 := strings.Fields(f.git("rev-list", "--reverse", "HEAD^2", "^HEAD^1")) + if got := len(picks8); got != 2 { + t.Errorf("#8 brought %d commits, want its own 2 without the merge of main", got) + } + if !strings.HasSuffix(f.git("log", "-1", "--format=%B", "HEAD"), batchTrailer+": "+mc8) { + t.Error("#8's merge does not name its merge commit") + } + if f.git("rev-parse", "HEAD:app/eight.txt") != f.git("rev-parse", mc8+":app/eight.txt") { + t.Error("app/eight.txt is not what main has after #8") + } + for _, s := range []string{"[#5]", "[#6]", "[#8]", "fix(app): five is fixed", "Rule 2", "`fix(app): five`, 2 commits", "`fix(app): six`, 1 commit"} { + if !strings.Contains(stdout, s) { + t.Errorf("the body lacks %q:\n%s", s, stdout) + } + } + if code, out, stderr := f.exec(nil, "verify", "--line", "release-1.1", "--head", f.git("rev-parse", "HEAD")); code != 0 { + t.Errorf("verify of the batch = %d\n%s\n%s", code, out, stderr) + } +} + +// A conflict is aborted, recorded with its files, and the next original is +// still picked. +func TestRunRecordsConflictsAndGoesOn(t *testing.T) { + f := newFixture(t) + f.lines() + f.onLine("release-1.1", "fix(chart): a line-only value", map[string]string{"chart/values.yaml": "replicas: 3\nimage: app:1\n"}) + f.pr(5, "fix(chart): five", "", map[string]string{"chart/values.yaml": "replicas: 5\nimage: app:1\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(app): six (#6)"}) { + t.Errorf("picked %q, want only #6", got) + } + if out := f.git("status", "--porcelain"); out != "" { + t.Errorf("the tree is left dirty:\n%s", out) + } + if _, err := os.Stat(filepath.Join(f.dir, ".git", "CHERRY_PICK_HEAD")); err == nil { + t.Error("a cherry-pick is left in progress") + } + l, _ := parseLine("release-1.1") + deferred, err := parseDeferred(stdout, l) + if err != nil || len(deferred[5].Files) == 0 || len(deferred) != 1 { + t.Errorf("deferred = %v, %v; want #5", deferred, err) + } + if !strings.Contains(stdout, "### Deferred conflicts") || !strings.Contains(stdout, "`chart/values.yaml`") { + t.Errorf("the body does not name the conflict and its file:\n%s", stdout) + } +} + +// An original the earlier picks already carry is skipped as an empty pick. +func TestRunSkipsAnEmptyPick(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): add x", "", map[string]string{"app/x.txt": "x\n"}) + f.pr(6, "revert(app): drop x", "", map[string]string{"app/x.txt": ""}) + f.pr(7, "fix(app): add x again", "", map[string]string{"app/x.txt": "x\n"}) + + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(app): add x (#5)"}) { + t.Errorf("picked %q, want only #5", got) + } + if !strings.Contains(stdout, "the earlier picks already carry it") { + t.Errorf("#7 is not reported as carried by the earlier picks:\n%s", stdout) + } +} + +// A hook in the repository or the user's configuration never rewrites what a +// pick carries. +func TestRunIgnoresHooks(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + hooks := filepath.Join(f.dir, ".git", "hooks") + for _, hook := range []string{"prepare-commit-msg", "commit-msg"} { + if err := os.WriteFile(filepath.Join(hooks, hook), []byte("#!/bin/sh\necho rewritten by a hook > \"$1\"\n"), 0o755); err != nil { + t.Fatal(err) + } + } + if code, _, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(app): five (#5)"}) { + t.Errorf("picked %q, want #5 as it was merged", got) + } +} + +func TestRunRefusesADirtyTree(t *testing.T) { + f := newFixture(t) + f.lines() + f.write("README.md", "changed\n") + code, _, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 2 || !strings.Contains(stderr, "has changes") { + t.Errorf("run on a dirty tree = %d (%s), want 2", code, stderr) + } + if f.git("branch", "--show-current") != "main" { + t.Error("the tree was switched anyway") + } + + // A pick that stopped empty leaves the tree clean and the pick going. + f.git("checkout", "-q", "--", "README.md") + if err := os.WriteFile(filepath.Join(f.dir, ".git", "CHERRY_PICK_HEAD"), []byte(f.git("rev-parse", "HEAD")+"\n"), 0o644); err != nil { + t.Fatal(err) + } + code, _, stderr = f.exec(nil, "run", "--line", "release-1.1") + if code != 2 || !strings.Contains(stderr, "operation is in progress (CHERRY_PICK_HEAD)") { + t.Errorf("run during a cherry-pick = %d (%s), want 2", code, stderr) + } +} + +// A clean tree says nothing about commits only the local batch branch has, so +// a run or a rebuild that resets that branch over them names its tip and the +// command that gets them back; one that drops nothing says nothing. +func TestResetNamesWhatItDrops(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + unpushed := func(base string) string { + t.Helper() + f.git("switch", "-q", "-C", batchBranch1, base) + f.write("app/mine.txt", "unpushed\n") + f.git("add", "-A") + f.git("commit", "-q", "-m", "an unpushed resolution") + tip := f.git("rev-parse", "HEAD") + f.git("switch", "-q", "main") + return tip + } + notice := func(stderr, tip string) bool { + t.Helper() + restore := "git branch " + batchBranch1 + "-" + tip[:12] + " " + tip + if !strings.Contains(stderr, "resetting the local "+batchBranch1+" from "+tip+",") || !strings.Contains(stderr, restore) { + return false + } + f.git(strings.Fields(restore)[1:]...) + if got := f.git("rev-parse", batchBranch1+"-"+tip[:12]); got != tip { + t.Errorf("%s restored %.12s, want %.12s", restore, got, tip) + } + return true + } + + tip := unpushed("origin/release-1.1") + if code, _, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || !notice(stderr, tip) { + t.Errorf("run over an unpushed commit = %d, want it named with its restore:\n%s", code, stderr) + } + f.git("switch", "-q", "main") + f.git("branch", "-f", batchBranch1, "origin/release-1.1~1") + if code, _, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stderr, "resetting") { + t.Errorf("run over a branch it fast-forwards = %d, want no notice:\n%s", code, stderr) + } + + f.git("switch", "-q", "main") + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + head := openBatch(t, f, gh) + tip = unpushed(head) + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "6", "--reason", "fails the app e2e"); code != 0 || !notice(stderr, tip) { + t.Errorf("a rebuild over an unpushed commit = %d, want it named with its restore:\n%s", code, stderr) + } +} + +// --push pushes a fresh branch and opens the draft, and a second push of the +// same branch updates the open pull request instead. +func TestRunPush(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + head := f.git("rev-parse", "HEAD") + if got := f.remoteHead(batchBranch1); got != head { + t.Errorf("remote %s = %q, want %s", batchBranch1, got, head) + } + if want := []string{"release-1.1 " + batchBranch1 + " fix(backport): batch for release-1.1, 2026-10-26"}; !slices.Equal(gh.created, want) { + t.Errorf("created %q, want %q", gh.created, want) + } + + f.git("switch", "-q", "main") + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, gh.bodies[0])} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--onto", head, "--push"); code != 0 { + t.Fatalf("run --onto --push = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", f.remoteHead(batchBranch1)); !slices.Equal(got, []string{"fix(app): five (#5)", "fix(app): six (#6)"}) { + t.Errorf("the pushed batch carries %q, want #5 then #6", got) + } + if len(gh.edited) != 1 || gh.edited[0] != "#90 fix(backport): batch for release-1.1, 2026-10-26" { + t.Errorf("edited %q, want #90", gh.edited) + } + if body := gh.bodies[len(gh.bodies)-1]; !strings.Contains(body, "[#5]") || !strings.Contains(body, "[#6]") { + t.Errorf("the updated body does not list both originals:\n%s", body) + } +} + +// With a batch pull request open on another branch, a fresh push would open +// a second one; it is refused before anything is pushed. +func TestRunPushRefusesASecondBatch(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + old := "backport-batch/release-1.1-20261022" + f.git("push", "-q", "origin", "origin/release-1.1:refs/heads/"+old) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{prs: []pullRequest{f.batchPR(90, "release-1.1", old, "")}} + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push") + if code != 2 || !strings.Contains(stderr, "continue it with --onto") { + t.Errorf("run --push = %d (%s), want a refusal", code, stderr) + } + if f.remoteHead(batchBranch1) != "" || len(gh.created)+len(gh.edited) != 0 { + t.Error("something was pushed or opened") + } +} diff --git a/cmd/backport-batch/batchbot_test.go b/cmd/backport-batch/batchbot_test.go new file mode 100644 index 000000000..987a134c2 --- /dev/null +++ b/cmd/backport-batch/batchbot_test.go @@ -0,0 +1,85 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "slices" + "strings" + "testing" +) + +// A maintainer's run by hand continues the batch pull request the scheduled +// run opened as its bot, with nothing set: it edits that pull request rather +// than opening a second one for the line. +func TestHandRunContinuesTheBotsBatch(t *testing.T) { + for _, tc := range []struct { + name, author string + continued bool + }{ + {"the scheduled run's bot", "cozystack-ci[bot]", true}, + {"another bot", "renovate[bot]", false}, + } { + t.Run(tc.name, func(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{bot: batchBotLogin(func(string) (string, bool) { return "", false })} + openBatch(t, f, gh) + gh.prs[0].User.Login, gh.prs[0].User.Type = tc.author, "Bot" + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.now = f.now.AddDate(0, 0, 3) + + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push") + if code != 0 { + t.Fatalf("run --continue = %d\n%s", code, stderr) + } + continued := slices.Equal(gh.edited, []string{"#90 fix(backport): batch for release-1.1, 2026-10-26"}) && len(gh.created) == 1 + if continued != tc.continued { + t.Errorf("edited %q, created %q; want the batch of %s continued=%v\n%s", gh.edited, gh.created, tc.author, tc.continued, stderr) + } + if !tc.continued && !strings.Contains(stderr, "ignoring #90") { + t.Errorf("the run does not say it ignores #90:\n%s", stderr) + } + }) + } +} + +// The bot comes from BACKPORT_BATCH_BOT when it is set, empty included, and +// is the scheduled run's own bot when it is not. +func TestBatchBotLogin(t *testing.T) { + env := func(v string, ok bool) func(string) (string, bool) { + return func(key string) (string, bool) { + if key != batchBotEnv { + t.Errorf("looked up %s", key) + } + return v, ok + } + } + for _, tc := range []struct { + name, v string + ok bool + want string + }{ + {"unset", "", false, "cozystack-ci[bot]"}, + {"set empty", "", true, ""}, + {"set", "other-app[bot]", true, "other-app[bot]"}, + } { + if got := batchBotLogin(env(tc.v, tc.ok)); got != tc.want { + t.Errorf("%s: batchBotLogin = %q, want %q", tc.name, got, tc.want) + } + } +} diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go new file mode 100644 index 000000000..a2d3a4c09 --- /dev/null +++ b/cmd/backport-batch/body.go @@ -0,0 +1,532 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "encoding/json" + "fmt" + "regexp" + "slices" + "strconv" + "strings" + "time" + "unicode" +) + +// Titles, release notes and file names in the body were written by whoever +// opened the originals, and the body is published under the name of whoever +// runs --push. None of it is searched for what might be dangerous and then +// rendered by other rules: a title, a reason or a file name goes whole into a +// code span, and the release notes into the fenced release-note block, both +// of which GitHub shows as written, with no mention, reference, link or HTML +// in them acted on. neutralNote is for a note copied out of that block. + +const ( + ourRepoURL = "https://github.com/" + upstreamRepo + "/" + // tasksURL is where the instructions for the work a batch leaves live. + tasksURL = ourRepoURL + "blob/main/docs/release/backport-tasks/" +) + +const ( + titleLimit = 200 + // GitHub refuses a pull request body longer than this. + bodyLimit = 65536 +) + +var ( + backtickRunRE = regexp.MustCompile("`+") + fenceRunRE = regexp.MustCompile("`{3,}|~{3,}") + mdLinkRE = regexp.MustCompile(`!?\[([^\]]*)\]\(([^)]*)\)`) + urlRE = regexp.MustCompile("(?i)\\b(?:https?|ftp)://[^\\s<>()\\[\\]`]+|\\bwww\\.[^\\s<>()\\[\\]`]+") + // A closing keyword with its target, a cross-repository reference and a + // mention, wherever they stand: no exception for what precedes them, so + // nothing depends on agreeing with GitHub about word boundaries. + actingRE = regexp.MustCompile(`(?i)\b(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\b\s*:?\s*(?:(?:[\w.-]+/[\w.-]+)?#\d+|https?://github\.com/[\w.-]+/[\w.-]+/(?:issues|pull)/\d+)` + + `|[\w.-]+/[\w.-]+#\d+|@[A-Za-z0-9][A-Za-z0-9-]*(?:/[A-Za-z0-9_.-]+)?`) + // A link into this repository survives, unless a dot segment or an escape + // could take it somewhere else. + safeURLRE = regexp.MustCompile(`^` + regexp.QuoteMeta(ourRepoURL) + `[\w/?=&#+.-]*$`) + releaseNotRE = regexp.MustCompile("(?ms)^[ \t]*```+[ \t]*release-note[ \t]*\n(.*?)^[ \t]*```") + headingRE = regexp.MustCompile(`^#{1,6} `) + placeholders = []string{"", "none", "n/a", "na", "no", "-", "nothing"} +) + +// clean drops what is not text: control characters other than tab and, when +// asked, newline, which a terminal acts on, and format characters such as the +// zero-width and bidi ones, which make text read as something it is not. +// Invalid UTF-8, a raw 8-bit escape among it, becomes U+FFFD. +func clean(s string, newlines bool) string { + var b strings.Builder + for _, r := range strings.ToValidUTF8(s, "\uFFFD") { + switch { + case r == '\t', r == '\n' && newlines: + b.WriteRune(r) + case r == '\r' || unicode.Is(unicode.Cf, r): + case r == '\n', r == '\u2028', r == '\u2029': + b.WriteRune(' ') + case unicode.IsControl(r): + b.WriteRune('\uFFFD') + default: + b.WriteRune(r) + } + } + return b.String() +} + +func oneLine(s string) string { + return strings.Join(strings.FieldsFunc(s, func(r rune) bool { return unicode.IsSpace(r) || unicode.IsControl(r) }), " ") +} + +func capped(s string, limit int) string { + if r := []rune(s); len(r) > limit { + return strings.TrimRightFunc(string(r[:limit-1]), unicode.IsSpace) + "…" + } + return s +} + +// codeSpan puts s on one line in a code span: its fence is longer than any +// backtick run inside, so nothing in s can close it. +func codeSpan(s string) string { + s = oneLine(clean(s, false)) + longest := 0 + for _, run := range backtickRunRE.FindAllString(s, -1) { + longest = max(longest, len(run)) + } + fence := strings.Repeat("`", longest+1) + pad := "" + if s == "" || strings.HasPrefix(s, "`") || strings.HasSuffix(s, "`") { + pad = " " + } + return fence + pad + s + pad + fence +} + +// titleSpan is a carried title as the body shows it. +func titleSpan(s string) string { return codeSpan(capped(oneLine(clean(s, false)), titleLimit)) } + +func prLink(n int) string { + return fmt.Sprintf("[#%d](%spull/%d)", n, ourRepoURL, n) +} + +func ours(url string) bool { + return safeURLRE.MatchString(url) && !strings.Contains(url, "/.") +} + +// inertSigns swaps the characters GitHub acts on for their fullwidth forms, +// which read the same and act on nothing. Unlike a code span, the swap does +// not depend on how the backticks around it pair up. +var inertSigns = strings.NewReplacer("@", "@", "#", "#", "://", "://") + +// neutralNote makes a release note safe inside the fenced block and inert +// outside it: no run of backticks or tildes is left to close the fence, links +// lose their target unless it is this repository, and closing keywords, +// cross-repository references and mentions lose the signs that make them +// act. Each step runs on the output of the one before, so a mention a link's +// text carried is still found. +func neutralNote(s string) string { + s = clean(s, true) + s = mdLinkRE.ReplaceAllString(s, "$1 ($2)") + s = urlRE.ReplaceAllStringFunc(s, func(u string) string { + if ours(u) { + return u + } + return "(link removed)" + }) + s = actingRE.ReplaceAllStringFunc(s, inertSigns.Replace) + return fenceRunRE.ReplaceAllStringFunc(s, func(f string) string { return strings.Repeat("'", len(f)) }) +} + +// releaseNote is the release-note block of a body, or "" when it has none or +// only a placeholder. +func releaseNote(body string) string { + m := releaseNotRE.FindStringSubmatch(htmlCommentRE.ReplaceAllString(body, "")) + if m == nil { + return "" + } + note := strings.TrimSpace(m[1]) + if slices.Contains(placeholders, strings.Trim(strings.ToLower(note), " .")) { + return "" + } + return note +} + +// deferredFence opens the block check reads the deferred conflicts from. +const deferredFence = "```backport-batch" + +// deferredEntry is a conflict, with the commit that failed and its files, an +// original --defer took out of the batch, with the reason given, or one that +// waits for a maintainer because it changes workflows, with those files. +type deferredEntry struct { + Original int `json:"original"` + Commit string `json:"commit,omitempty"` + Files []string `json:"files"` + More int `json:"more,omitempty"` + Reason string `json:"reason,omitempty"` + Workflows bool `json:"workflows,omitempty"` +} + +// How many conflicting files of one original the body names, in the list and +// in the block: a vendored chart can conflict in hundreds. +const ( + filesListed = 5 + filesKept = 50 +) + +type deferredRecord struct { + Line string `json:"line"` + Deferred []deferredEntry `json:"deferred"` +} + +// deferredBlock is the machine-readable list of what the batch defers. One +// line of JSON: encoding/json escapes newlines and <, > and &, so nothing in a +// file name or a reason can end the fence or open HTML. It leaves bidi and +// zero-width characters as they are, so file names go through clean first, +// as the reason does. +func deferredBlock(l line, conflicts, held, waiting []pick) string { + rec := deferredRecord{Line: l.branch(), Deferred: []deferredEntry{}} + for _, p := range conflicts { + files := cleanFiles(p.files[:min(len(p.files), filesKept)]) + rec.Deferred = append(rec.Deferred, deferredEntry{Original: p.v.o.number, Commit: p.failed, Files: files, More: len(p.files) - len(files) + p.more}) + } + for _, p := range held { + rec.Deferred = append(rec.Deferred, deferredEntry{Original: p.v.o.number, Files: []string{}, Reason: capped(oneLine(clean(p.reason, false)), titleLimit)}) + } + for _, p := range waiting { + files := cleanFiles(p.files[:min(len(p.files), filesKept)]) + rec.Deferred = append(rec.Deferred, deferredEntry{Original: p.v.o.number, Files: files, More: len(p.files) - len(files) + p.more, Workflows: true}) + } + data, _ := json.Marshal(rec) + return deferredFence + "\n" + string(data) + "\n```" +} + +func cleanFiles(files []string) []string { + out := make([]string, len(files)) + for i, f := range files { + out[i] = clean(f, false) + } + return out +} + +// parseDeferred reads what a batch body defers: exactly one block, for this +// line, naming each original once. +func parseDeferred(body string, l line) (map[int]deferredEntry, error) { + var blocks []string + lines := strings.Split(strings.ReplaceAll(body, "\r\n", "\n"), "\n") + for i := 0; i < len(lines); i++ { + if strings.TrimSpace(lines[i]) != deferredFence { + continue + } + if i+2 >= len(lines) || strings.TrimSpace(lines[i+2]) != "```" { + return nil, fmt.Errorf("the %s block is not one line of JSON", deferredFence) + } + blocks = append(blocks, lines[i+1]) + i += 2 + } + switch len(blocks) { + case 0: + return map[int]deferredEntry{}, nil + case 1: + default: + return nil, fmt.Errorf("the body holds %d %s blocks", len(blocks), deferredFence) + } + dec := json.NewDecoder(strings.NewReader(blocks[0])) + dec.DisallowUnknownFields() + var rec deferredRecord + if err := dec.Decode(&rec); err != nil { + return nil, fmt.Errorf("reading the %s block: %w", deferredFence, err) + } + if rec.Line != l.branch() { + return nil, fmt.Errorf("the %s block is for %q, not %s", deferredFence, rec.Line, l.branch()) + } + out := map[int]deferredEntry{} + for _, d := range rec.Deferred { + if d.Original <= 0 { + return nil, fmt.Errorf("the %s block names original %d", deferredFence, d.Original) + } + if d.Commit != "" && checkSHA(d.Commit) != nil { + return nil, fmt.Errorf("the %s block names commit %q for #%d", deferredFence, d.Commit, d.Original) + } + if d.Workflows && (d.Commit != "" || d.Reason != "") { + return nil, fmt.Errorf("the %s block has #%d wait for a maintainer and defers it as well", deferredFence, d.Original) + } + if _, dup := out[d.Original]; dup { + return nil, fmt.Errorf("the %s block names #%d twice", deferredFence, d.Original) + } + out[d.Original] = d + } + return out, nil +} + +// prTitle is built from the line and the date alone, never from carried text. +func prTitle(l line, date string) string { + return fmt.Sprintf("fix(backport): batch for %s, %s", l.branch(), date) +} + +// skipRules are the rules a skipped original is reported under, in order. +var skipRules = []int{ruleBeforeStart, ruleOnLine, ruleType, ruleBreaking, ruleTouches, ruleNewer} + +// report is what one run put on the batch branch and what it left out. +type report struct { + line line + newer *line + onBranch []*verdict // carried from --onto and picked now, in merge order + deferred []pick // conflicts + held []pick // what --defer took out, with the reason + waiting []pick // what changes workflows, left to a maintainer + skipped map[int][]*verdict + other int + // overrides are the originals a backport/ label kept out or let in, and + // stale those with backport/skip that are still on the batch. + overrides []*verdict + stale []*verdict + start time.Time + // shapes says, for an original on the branch, what it brought. + shapes map[*verdict]string +} + +func (r *report) ruleTitle(rule int) string { + switch rule { + case ruleBeforeStart: + return "Rule 1, merged before " + codeSpan(r.line.branch()) + " started under the policy on " + r.start.Format(time.DateOnly) + case ruleOnLine: + return "Rule 1, already on " + codeSpan(r.line.branch()) + case ruleType: + return "Rule 2, the title type is not `fix` or `security`, or the scope is `deps`" + case ruleBreaking: + return "Rule 3, a breaking change" + case ruleTouches: + return "Rule 4, touches platform migrations, `targetVersion` or a CRD, or makes a sizeable API change" + case ruleNewer: + return "Rule 6, not on " + codeSpan(r.newer.branch()) + " yet" + } + return fmt.Sprintf("Rule %d", rule) +} + +// what is the What this PR does section, with the skipped lists of the rules +// in details written out. +func (r *report) what(details map[int]bool) string { + var b strings.Builder + fmt.Fprintf(&b, "Backports to %s from `main`, picked by the rules of the [backport policy](%s) in merge order. A merged pull request comes as its own commits, picked with `git cherry-pick -x` so each keeps its author, message and sign-offs, and a merge commit that carries the original's merge message with a `%s:` trailer naming it; a squash-merged one comes as one pick. `backport-batch verify` checks that shape. A pick listed as changed differs from the commit it came from; `git range-diff ^1..^2 ^1..^2` shows how. A conflict or a failing check here is a [backport task](%sREADME.md).\n\n", codeSpan(r.line.branch()), policyURL, batchTrailer, tasksURL) + if len(r.onBranch) == 0 { + b.WriteString("Nothing is picked yet.\n") + } + for _, v := range r.onBranch { + shape := "" + if s := r.shapes[v]; s != "" { + shape = ", " + s + } + fmt.Fprintf(&b, "- %s %s%s\n", prLink(v.o.number), titleSpan(v.o.title), shape) + } + if len(r.deferred) > 0 { + fmt.Fprintf(&b, "\n### Deferred conflicts\n\nThese do not pick cleanly onto this branch. Each names the first of its commits that conflicted and the files it conflicted in, and nothing of it is on the branch. Each moves to the next batch, unless a resolution checked with `backport-batch fold` is pushed here first, as [resolving a conflict](%sresolve-conflict.md) describes.\n\n", tasksURL) + for _, p := range r.deferred { + var files []string + for _, f := range p.files[:min(len(p.files), filesListed)] { + files = append(files, codeSpan(f)) + } + if more := len(p.files) - len(files) + p.more; more > 0 { + files = append(files, fmt.Sprintf("%d more", more)) + } + at := "" + if p.failed != "" { + at = " at " + codeSpan(p.failed[:12]) + } + fmt.Fprintf(&b, "- %s %s%s: %s\n", prLink(p.v.o.number), titleSpan(p.v.o.title), at, strings.Join(files, ", ")) + } + } + if len(r.held) > 0 { + b.WriteString("\n### Deferred to the next batch\n\nTaken out of this batch with `backport-batch run --defer`, each for the reason given. This pull request keeps them out; the next batch picks them again.\n\n") + for _, p := range r.held { + fmt.Fprintf(&b, "- %s %s: %s\n", prLink(p.v.o.number), titleSpan(p.v.o.title), titleSpan(p.reason)) + } + } + if len(r.waiting) > 0 { + fmt.Fprintf(&b, "\n### Waiting for a maintainer\n\nThese change `%s`, which the token this batch runs with may not push, so it leaves them out. The policy still takes each, and `backport-batch check` lists it until it is on the line. A maintainer brings one over as [changes to workflows](%s#changes-to-workflows) describes.\n\n", workflowsDir, policyURL) + for _, p := range r.waiting { + var files []string + for _, f := range p.files[:min(len(p.files), filesListed)] { + files = append(files, codeSpan(f)) + } + if more := len(p.files) - len(files) + p.more; more > 0 { + files = append(files, fmt.Sprintf("%d more", more)) + } + fmt.Fprintf(&b, "- %s %s: %s\n", prLink(p.v.o.number), titleSpan(p.v.o.title), strings.Join(files, ", ")) + } + } + if len(r.overrides) > 0 { + b.WriteString("\n### Maintainer overrides\n\n") + for _, v := range r.overrides { + fmt.Fprintf(&b, "- %s %s: %s, %s\n", prLink(v.o.number), titleSpan(v.o.title), codeSpan(v.label), r.outcome(v)) + } + } + b.WriteString("\n### Skipped\n\n") + for _, rule := range skipRules { + items := r.skipped[rule] + if len(items) == 0 { + continue + } + left := "" + if !details[rule] { + left = ", too many to list here; the run's output lists them" + } + fmt.Fprintf(&b, "- %s: %d%s\n", r.ruleTitle(rule), len(items), left) + } + if r.other > 0 { + fmt.Fprintf(&b, "- Commits on `main` that merged no pull request: %d\n", r.other) + } + for _, rule := range skipRules { + items := r.skipped[rule] + if len(items) > 0 && details[rule] { + fmt.Fprintf(&b, "\n
%s\n\n", r.ruleTitle(rule)) + for _, v := range items { + fmt.Fprintf(&b, "- %s %s, %s\n", prLink(v.o.number), titleSpan(v.o.title), titleSpan(v.reason)) + } + b.WriteString("\n
\n") + } + } + b.WriteString("\n" + deferredBlock(r.line, r.deferred, r.held, r.waiting) + "\n") + return b.String() +} + +// outcome is what became of an original a label overrode. +func (r *report) outcome(v *verdict) string { + switch { + case slices.Contains(r.stale, v): + return "still on this branch, picked before the label; take it out with `backport-batch run --defer`" + case v.label == labelSkip: + return "left out" + case slices.Contains(r.onBranch, v): + return "on this branch" + case slices.ContainsFunc(r.deferred, func(p pick) bool { return p.v == v }): + return "a deferred conflict" + case slices.ContainsFunc(r.held, func(p pick) bool { return p.v == v }): + return "deferred to the next batch" + case slices.ContainsFunc(r.waiting, func(p pick) bool { return p.v == v }): + return "waiting for a maintainer: it changes " + codeSpan(workflowsDir) + case v.rule != 0: + return "skipped by rule " + strconv.Itoa(v.rule) + ", " + titleSpan(v.reason) + } + return "not picked" +} + +func (r *report) notes() string { + var notes []string + for _, v := range r.onBranch { + if n := releaseNote(v.o.body); n != "" { + notes = append(notes, neutralNote(n)) + } + } + return strings.Join(notes, "\n\n") +} + +// section is one heading of the pull request template and what follows it. +type section struct { + heading string + text string +} + +func sections(template string) (string, []section) { + var lead strings.Builder + var out []section + inFence := false + for _, ln := range strings.Split(htmlCommentRE.ReplaceAllString(template, ""), "\n") { + if strings.HasPrefix(strings.TrimSpace(ln), "```") { + inFence = !inFence + } + if !inFence && headingRE.MatchString(ln) { + out = append(out, section{heading: ln}) + continue + } + if len(out) == 0 { + lead.WriteString(ln + "\n") + } else { + out[len(out)-1].text += ln + "\n" + } + } + return strings.TrimSpace(lead.String()), out +} + +func headingIs(heading, name string) bool { + return strings.EqualFold(strings.TrimSpace(strings.TrimLeft(heading, "#")), name) +} + +// body fills the line's own pull request template: the What, Screenshots and +// Release note sections are the batch's, every other section stays as the +// template has it. A body over GitHub's limit loses its longest skipped lists +// first, and its release notes' tail last; left out are the rules whose lists +// it lost. +func (r *report) body(template string) (body string, left []int) { + details := map[int]bool{} + for rule, items := range r.skipped { + details[rule] = len(items) > 0 + } + notes := r.notes() + for { + full := r.render(template, r.what(details), notes) + if len(full) <= bodyLimit { + return full, left + } + longest := 0 + for _, rule := range skipRules { + if details[rule] && (longest == 0 || len(r.skipped[rule]) > len(r.skipped[longest])) { + longest = rule + } + } + if longest == 0 { + const marker = "\n\n(release notes truncated; see the original pull requests)" + cut := max(0, len(notes)-(len(full)-bodyLimit)-len(marker)) + for cut > 0 && !utf8Start(notes[cut]) { + cut-- + } + return r.render(template, r.what(details), notes[:cut]+marker), left + } + details[longest] = false + left = append(left, longest) + } +} + +func utf8Start(c byte) bool { return c&0xC0 != 0x80 } + +func (r *report) render(template, what, notes string) string { + lead, secs := sections(template) + var b strings.Builder + if lead != "" { + b.WriteString(lead + "\n\n") + } + seen := map[string]bool{} + for _, s := range secs { + switch { + case headingIs(s.heading, "What this PR does"): + seen["what"] = true + b.WriteString(s.heading + "\n\n" + what + "\n") + case headingIs(s.heading, "Screenshots"): + b.WriteString(s.heading + "\n\nSee the original pull requests.\n\n") + case headingIs(s.heading, "Release note"): + seen["note"] = true + b.WriteString(s.heading + "\n\n```release-note\n" + notes + "\n```\n\n") + default: + b.WriteString(s.heading + "\n\n" + strings.TrimSpace(s.text) + "\n\n") + } + } + out := b.String() + if !seen["what"] { + out = "## What this PR does\n\n" + what + "\n" + out + } + if !seen["note"] { + out += "### Release note\n\n```release-note\n" + notes + "\n```\n" + } + return strings.TrimSpace(regexp.MustCompile(`\n{3,}`).ReplaceAllString(out, "\n\n")) + "\n" +} diff --git a/cmd/backport-batch/body_test.go b/cmd/backport-batch/body_test.go new file mode 100644 index 000000000..07f2f8582 --- /dev/null +++ b/cmd/backport-batch/body_test.go @@ -0,0 +1,368 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "os" + "path/filepath" + "regexp" + "slices" + "strings" + "testing" + "unicode" +) + +// visible is what of a body GitHub renders as markdown rather than shows as +// code: fenced blocks and code spans taken out, delimited as CommonMark +// delimits them. Everything carried has to be gone from it. +func visible(body string) string { + fenceOpen := regexp.MustCompile("^(`{3,}|~{3,})") + var out []string + fence := "" + for _, ln := range strings.Split(body, "\n") { + t := strings.TrimLeft(ln, " ") + indented := len(ln)-len(t) > 3 + if fence != "" { + run := strings.TrimLeft(t, fence[:1]) + if !indented && len(t)-len(run) >= len(fence) && strings.TrimSpace(run) == "" { + fence = "" + } + continue + } + if m := fenceOpen.FindString(t); m != "" && !indented && (m[0] == '~' || !strings.Contains(t[len(m):], "`")) { + fence = m + continue + } + out = append(out, stripSpans(ln)) + } + return strings.Join(out, "\n") +} + +func stripSpans(s string) string { + var b strings.Builder + for i := 0; i < len(s); { + switch { + case s[i] == '\\' && i+1 < len(s): + b.WriteString(s[i : i+2]) + i += 2 + continue + case s[i] != '`': + b.WriteByte(s[i]) + i++ + continue + } + j := i + for j < len(s) && s[j] == '`' { + j++ + } + n, closing := j-i, -1 + for k := j; k < len(s); { + if s[k] != '`' { + k++ + continue + } + e := k + for e < len(s) && s[e] == '`' { + e++ + } + if e-k == n { + closing = k + break + } + k = e + } + if closing < 0 { + b.WriteString(s[i:j]) + i = j + continue + } + i = closing + n + } + return b.String() +} + +// hostile is carried text built to get out of its span or block: each carries +// a marker that must never be visible. +var hostile = []string{ + "zqx01 thanks @alice", + "zqx02 (@org/team) please look", + "zqx03 [@bob](https://evil.example/x) [docs](https://github.com/cozystack/cozystack/../../evil/repo)", + "zqx04 zero\u200b@carol bidi \u202e@dave \ufeff@erin", + "zqx05 ``` `@frank` ``` still inside", + "zqx06 ` closes a single span, then @grace", + "zqx07 `` `` ``` ```` four", + "zqx08 Fixes #1, closes other/repo#2, resolves https://github.com/other/repo/issues/3", + "zqx09 x", + "zqx10 www.evil.example ", + "zqx11 [ref]: https://evil.example", + "zqx12 @heidi & <b>", + "zqx13 \x1b[31mred\x1b[0m \u009b2J \x07 bell", + "zqx14
x", + "`zqx15` starts and ends with a backtick`", +} + +// hostileNote is a release note whose own block a bare fence line would end, +// so it hides one behind a zero-width space, which clean then removes. +func hostileNote(i int, s string) string { + return fmt.Sprintf("fix(x): zqn%02d %s\n\u200b```\n@ivan zqn%02d after a fence\n~~~\n[@judy](https://evil.example) Fixes #9 \x1b[2J", i, s, i) +} + +// Every carried title, reason and release note stays inside a code span or the +// release-note block, however it is written. +func TestBodyContainsCarriedText(t *testing.T) { + f := newFixture(t) + f.lines() + for i, s := range hostile { + typ := "fix" + if i%2 == 1 { + typ = "feat" // skipped, so it lands in a
list + } + body := "## What this PR does\n\n```release-note\n" + hostileNote(i, s) + "\n```\n" + f.pr(100+i, typ+"(x): "+s, body, map[string]string{fmt.Sprintf("app/%d.txt", i): "x\n"}) + } + gh := &fakeGH{} + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if len(gh.bodies) != 1 { + t.Fatalf("gh got %d bodies, want 1", len(gh.bodies)) + } + for _, body := range []string{gh.bodies[0], stdout} { + for i := range hostile { + title, note := fmt.Sprintf("zqx%02d", i+1), fmt.Sprintf("zqn%02d", i) + if i == len(hostile)-1 { + title = "zqx15" + } + if !strings.Contains(body, title) { + t.Errorf("%s is missing from the body", title) + } + if i%2 == 0 && !strings.Contains(body, note) { + t.Errorf("%s is missing from the release notes", note) + } + } + if v := visible(body); strings.Contains(v, "zqx") || strings.Contains(v, "zqn") { + t.Errorf("carried text is visible outside code:\n%s", v) + } + for _, r := range body { + if r != '\n' && r != '\t' && unicode.IsControl(r) || unicode.Is(unicode.Cf, r) { + t.Errorf("the body carries %U", r) + break + } + } + } +} + +// Copied out of the fenced block, a note must still close, ping and link +// nothing. +func TestNeutralNote(t *testing.T) { + cases := []struct{ in, gone string }{ + {"thanks @alice", "@alice"}, + {"(@org/team) look", "@org/team"}, + {"[@bob](https://evil.example)", "@bob"}, + {"x\u200b@carol", "@carol"}, + {"a@dave.example", "@dave"}, + {"Fixes #1", "Fixes #1"}, + {"closes other/repo#2", "other/repo#2"}, + {"see other/repo#3", "other/repo#3"}, + {"resolves https://github.com/other/repo/issues/3", "github.com/other"}, + {"Fixes https://github.com/cozystack/cozystack/issues/3", "Fixes https://"}, + {"x``@erin", "@erin"}, + {"`code @frank here`", "@frank"}, + {"```@grace```", "@grace"}, + {"go to https://evil.example/x", "evil.example"}, + {"go to www.evil.example", "evil.example"}, + {"[docs](https://evil.example)", "evil.example"}, + {"[docs](https://github.com/cozystack/cozystack/../../evil/repo)", "evil/repo"}, + {"line\n```\nescaped", "```"}, + {"line\n~~~\nescaped", "~~~"}, + {"zero\u200b``\u200b`", "```"}, + {"\x1b[2J", "\x1b"}, + } + for _, tc := range cases { + got := neutralNote(tc.in) + if strings.Contains(visible(got), tc.gone) { + t.Errorf("neutralNote(%q) = %q, still shows %q", tc.in, got, tc.gone) + } + } + keep := "see https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md" + if got := neutralNote(keep); got != keep { + t.Errorf("neutralNote(%q) = %q, want a link into this repository kept", keep, got) + } +} + +func TestCodeSpan(t *testing.T) { + cases := []string{"plain", "a `b` c", "``x``", "`lead", "trail`", "", "a\nb", "````", "\x1b[31m`\u202e`"} + for _, s := range cases { + got := codeSpan(s) + if v := visible(got); v != "" { + t.Errorf("codeSpan(%q) = %q leaves %q outside the span", s, got, v) + } + if strings.ContainsAny(got, "\n\x1b\u202e") { + t.Errorf("codeSpan(%q) = %q carries a newline or a control", s, got) + } + } +} + +func TestDeferredRoundTrip(t *testing.T) { + l, _ := parseLine("release-1.6") + v := &verdict{o: &original{number: 3735}} + held := &verdict{o: &original{number: 3736}} + block := deferredBlock(l, []pick{{v: v, files: []string{"a\n```\nb", "&d", "e`f"}}}, []pick{{v: held, reason: "breaks\n```\ne2e"}}, nil) + if lines := strings.Split(block, "\n"); len(lines) != 3 { + t.Fatalf("the block is %d lines, want 3:\n%s", len(lines), block) + } + body := "## What this PR does\n\ntext\n\n" + block + "\n\n### Release note\n" + got, err := parseDeferred(body, l) + if err != nil || len(got) != 2 || len(got[3735].Files) != 3 || got[3735].Reason != "" { + t.Errorf("parseDeferred = %v, %v; want the conflict #3735 with its files", got, err) + } + if e := got[3736]; e.Reason != "breaks ``` e2e" || len(e.Files) != 0 { + t.Errorf("parseDeferred = %+v; want #3736 held, for its reason on one line", e) + } + if strings.ContainsAny(strings.Split(block, "\n")[1], "<>&") { + t.Errorf("the JSON line carries raw HTML characters: %s", block) + } + + other, _ := parseLine("release-1.5") + for name, body := range map[string]string{ + "another line": body, + "two blocks": body + "\n" + block, + "not JSON": deferredFence + "\nnot json\n```", + "an unknown key": deferredFence + "\n{\"line\":\"release-1.6\",\"deferred\":[],\"x\":1}\n```", + "a bad number": deferredFence + "\n{\"line\":\"release-1.6\",\"deferred\":[{\"original\":-1}]}\n```", + "a repeat": deferredFence + "\n{\"line\":\"release-1.6\",\"deferred\":[{\"original\":5},{\"original\":5,\"reason\":\"x\"}]}\n```", + "a bad commit": deferredFence + "\n{\"line\":\"release-1.6\",\"deferred\":[{\"original\":5,\"commit\":\"-x\"}]}\n```", + "a broken fence": deferredFence + "\n{}\nmore\n```", + "an empty fence": deferredFence + "\n```", + "a trailing line": deferredFence, + } { + want := l + if name == "another line" { + want = other + } + if _, err := parseDeferred(body, want); err == nil { + t.Errorf("%s: parseDeferred accepted it", name) + } + } + if got, err := parseDeferred("no block here", l); err != nil || len(got) != 0 { + t.Errorf("no block: %v, %v; want nothing deferred", got, err) + } +} + +func TestReleaseNote(t *testing.T) { + cases := map[string]string{ + "### Release note\n\n```release-note\nfix(x): y\n```\n": "fix(x): y", + "```release-note\nNONE\n```": "", + "```release-note\n\n```": "", + "": "", + "no block": "", + } + for body, want := range cases { + if got := releaseNote(body); got != want { + t.Errorf("releaseNote(%q) = %q, want %q", body, got, want) + } + } +} + +// The body is the line's template with the batch's sections filled in, and +// fits the size GitHub accepts. +func TestBodyFollowsTheTemplate(t *testing.T) { + l, _ := parseLine("release-1.6") + template := "\n\n## What this PR does\n\n\n### Screenshots\n\n\n\n### Downstream repositories\n\n- [ ] No downstream repository is affected by this change\n\n### Release note\n\n```release-note\n\n```\n" + o := &original{number: 7, title: "fix(x): y", body: "```release-note\nfix(x): note\n```"} + r := &report{line: l, onBranch: []*verdict{{o: o}}, skipped: map[int][]*verdict{}} + body, _ := r.body(template) + for _, want := range []string{"## What this PR does\n\nBackports to", "- [#7](https://github.com/cozystack/cozystack/pull/7) `fix(x): y`", "### Downstream repositories\n\n- [ ] No downstream", "```release-note\nfix(x): note\n```", deferredFence} { + if !strings.Contains(body, want) { + t.Errorf("the body lacks %q:\n%s", want, body) + } + } + if strings.Contains(body, "" + +// stalledAfter is how many days a batch stays open before notify calls it +// stalled: the shorter gap between two scheduled runs, Monday to Thursday, so +// a batch the previous run opened or continued is stalled at the next one. +const stalledAfter = 3 + +var mentionRE = regexp.MustCompile(`^@?([A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?(?:/[A-Za-z0-9_.-]+)?)$`) + +// notify reports what waits on the open batch pull request of a line for a +// maintainer: a batch open since before the previous run, the conflicts it +// deferred, and the originals that change workflows. One line per finding goes +// to stdout, for the run's summary. --push writes them into the comment, one +// per pull request, updated in place on every run. +func (a *app) notify(args []string) int { + fs := a.flags("notify") + lineArg := fs.String("line", "", "release line, release-X.Y") + mention := fs.String("mention", "", "team or person the comment mentions, org/team or login, with or without @") + push := fs.Bool("push", false, "write the comment") + if err := parse(fs, args); err != nil { + return 2 + } + l, err := parseLine(*lineArg) + if err != nil { + return a.fail(err) + } + if *mention != "" { + m := mentionRE.FindStringSubmatch(*mention) + if m == nil { + return a.fail(fmt.Errorf("--mention %q is not @login or @org/team", *mention)) + } + *mention = "@" + m[1] + } + prs, err := a.gh.openPulls(l.branch()) + if err != nil { + return a.fail(err) + } + pr, warnings, err := batchPull(a.gh, prs, l) + for _, w := range warnings { + a.warn("%s", w) + } + if err != nil { + return a.fail(err) + } + if pr == nil { + return 0 + } + opened, err := branchDate(pr.Head.Ref) + if err != nil { + return a.fail(err) + } + deferred, err := parseDeferred(pr.Body, l) + if err != nil { + return a.fail(fmt.Errorf("batch pull request #%d: %w", pr.Number, err)) + } + var conflicts, waiting []int + for n, e := range deferred { + switch { + case e.Workflows: + waiting = append(waiting, n) + case e.Reason == "": + conflicts = append(conflicts, n) + } + } + slices.Sort(conflicts) + slices.Sort(waiting) + + today := a.now().UTC().Truncate(24 * time.Hour) + days := int(today.Sub(opened).Hours() / 24) + stalled := days >= stalledAfter + var text []string + if stalled { + fmt.Fprintf(a.stdout, "%s: batch pull request #%d has been open since %s, %d days\n", l.branch(), pr.Number, opened.Format(time.DateOnly), days) + text = append(text, fmt.Sprintf("This batch has been open since %s, %d days, through at least one scheduled run. Merge it, or close it and the next run starts a new one; until then each run adds what the rules take to it.", opened.Format(time.DateOnly), days)) + } + if len(conflicts) > 0 { + fmt.Fprintf(a.stdout, "%s: %d deferred conflicts on #%d wait for a maintainer\n", l.branch(), len(conflicts), pr.Number) + var b strings.Builder + fmt.Fprintf(&b, "These conflicts wait for a maintainer, as [resolving a conflict](%sresolve-conflict.md) describes; unresolved, each goes to the next batch:\n", tasksURL) + for _, n := range conflicts { + fmt.Fprintf(&b, "\n- %s", prLink(n)) + } + text = append(text, b.String()) + } + if len(waiting) > 0 { + fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s and wait for a maintainer\n", l.branch(), len(waiting), pr.Number, workflowsDir) + var b strings.Builder + fmt.Fprintf(&b, "These change `%s`, which the batch's token may not push, and wait for a maintainer to bring them over, as [changes to workflows](%s#changes-to-workflows) describes:\n", workflowsDir, policyURL) + for _, n := range waiting { + fmt.Fprintf(&b, "\n- %s", prLink(n)) + } + text = append(text, b.String()) + } + if !*push { + return 0 + } + return a.comment(pr, *mention, today, text) +} + +// comment writes what waits into the one notify comment on the batch pull +// request: it edits the comment if there is one, and opens it only when +// something waits. Only a comment by an author the batch trusts is taken for +// it, so a look-alike someone else wrote is neither edited nor believed. +func (a *app) comment(pr *pullRequest, mention string, today time.Time, text []string) int { + comments, err := a.gh.comments(pr.Number) + if err != nil { + return a.fail(err) + } + var ours *issueComment + for i := range comments { + c := &comments[i] + if !strings.HasPrefix(c.Body, notifyMarker) { + continue + } + why, err := trustedAuthor(a.gh, c.User.Login, c.User.Type) + if err != nil { + return a.fail(err) + } + if why == "" { + ours = c + } + } + if ours == nil && len(text) == 0 { + return 0 + } + var b strings.Builder + b.WriteString(notifyMarker + "\n") + if mention != "" && len(text) > 0 { + b.WriteString(mention + "\n\n") + } + if len(text) == 0 { + text = []string{"Nothing waits on a maintainer."} + } + b.WriteString(strings.Join(text, "\n\n")) + fmt.Fprintf(&b, "\n\nUpdated by the backport batch on %s.\n", today.Format(time.DateOnly)) + f, err := os.CreateTemp("", "backport-batch-comment-*.md") + if err != nil { + return a.fail(err) + } + defer os.Remove(f.Name()) + if _, err := f.WriteString(b.String()); err != nil { + f.Close() + return a.fail(err) + } + if err := f.Close(); err != nil { + return a.fail(err) + } + if ours != nil { + err = a.gh.editComment(ours.ID, f.Name()) + } else { + err = a.gh.createComment(pr.Number, f.Name()) + } + if err != nil { + return a.fail(err) + } + return 0 +} + +// lines prints the lines the batch runs for, newest first, in the order the +// policy has them processed. +func (a *app) lines(args []string) int { + fs := a.policyFlags("lines") + if err := parse(fs, args); err != nil { + return 2 + } + if err := fetch(); err != nil { + return a.fail(err) + } + mainHead, err := resolve("refs/remotes/" + remote + "/main") + if err != nil { + return a.fail(err) + } + pol, err := a.policy(mainHead) + if err != nil { + return a.fail(err) + } + lines, err := batchLines(pol.released) + if err != nil { + return a.fail(err) + } + for _, l := range slices.Backward(lines) { + fmt.Fprintln(a.stdout, l.branch()) + } + return 0 +} diff --git a/cmd/backport-batch/notify_test.go b/cmd/backport-batch/notify_test.go new file mode 100644 index 000000000..606953be5 --- /dev/null +++ b/cmd/backport-batch/notify_test.go @@ -0,0 +1,288 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "bytes" + "slices" + "strings" + "testing" + "time" +) + +// notifyPR is an open batch pull request for release-1.1 on the branch opened +// on date, YYYYMMDD, with body. +func notifyPR(date, body string) pullRequest { + var pr pullRequest + pr.Number, pr.Body = 90, body + pr.User.Login, pr.User.Type = "maintainer", "User" + pr.Head.Ref, pr.Head.SHA = "backport-batch/release-1.1-"+date, strings.Repeat("a", 40) + pr.Head.Repo = &struct { + FullName string `json:"full_name"` + }{upstreamRepo} + pr.Base.Ref, pr.Base.Repo.FullName = "release-1.1", upstreamRepo + return pr +} + +// notifyRun runs notify on 2026-10-26 with gh faked. +func notifyRun(gh *fakeGH, args ...string) (int, string, string) { + var stdout, stderr bytes.Buffer + a := &app{gh: gh, stdout: &stdout, stderr: &stderr, now: func() time.Time { return time.Date(2026, 10, 26, 6, 23, 0, 0, time.UTC) }} + code := a.main(append([]string{"notify", "--line", "release-1.1"}, args...)) + return code, stdout.String(), stderr.String() +} + +var notifyLine, _ = parseLine("release-1.1") + +// A batch open since before the previous run gets one comment, mentioning +// the team, and keeps that one comment however many runs see it. +func TestNotifyStalled(t *testing.T) { + for _, tc := range []struct { + date string + stalled bool + }{ + {"20261026", false}, + {"20261024", false}, + {"20261023", true}, + {"20261019", true}, + } { + gh := &fakeGH{prs: []pullRequest{notifyPR(tc.date, "")}} + code, stdout, stderr := notifyRun(gh, "--mention", "cozystack/maintainers", "--push") + if code != 0 { + t.Fatalf("%s: notify = %d\n%s", tc.date, code, stderr) + } + if got := strings.Contains(stdout, "release-1.1: batch pull request #90 has been open since"); got != tc.stalled { + t.Errorf("%s: stalled = %v, want %v:\n%s", tc.date, got, tc.stalled, stdout) + } + if got := len(gh.issueComments); got != map[bool]int{true: 1, false: 0}[tc.stalled] { + t.Errorf("%s: %d comments, want one only when stalled", tc.date, got) + } + if !tc.stalled { + continue + } + c := gh.issueComments[0].Body + if !strings.HasPrefix(c, notifyMarker+"\n@cozystack/maintainers\n") || !strings.Contains(c, "open since") { + t.Errorf("%s: the comment does not mention the team and say why:\n%s", tc.date, c) + } + for range 2 { + if code, _, stderr := notifyRun(gh, "--mention", "@cozystack/maintainers", "--push"); code != 0 { + t.Fatalf("notify again = %d\n%s", code, stderr) + } + } + if len(gh.issueComments) != 1 { + t.Errorf("%s: %d comments after three runs, want the one updated", tc.date, len(gh.issueComments)) + } + } + + gh := &fakeGH{prs: []pullRequest{notifyPR("20261019", "")}} + if code, _, _ := notifyRun(gh, "--push"); code != 0 || strings.Contains(gh.issueComments[0].Body, "@") { + t.Errorf("notify with no team = %d and wrote %q; want no mention", code, gh.issueComments[0].Body) + } + gh = &fakeGH{prs: []pullRequest{notifyPR("20261019", "")}} + if code, stdout, _ := notifyRun(gh); code != 0 || stdout == "" || len(gh.issueComments) != 0 { + t.Errorf("notify without --push = %d, %q and %d comments; want the line and no comment", code, stdout, len(gh.issueComments)) + } +} + +// The comment lists the deferred conflicts for a maintainer, and nothing +// --defer held out. +func TestNotifyConflicts(t *testing.T) { + conflict := pick{v: &verdict{o: &original{number: 6}}, files: []string{"app/six.txt"}} + held := pick{v: &verdict{o: &original{number: 7}}, reason: "fails the e2e"} + body := "text\n\n" + deferredBlock(notifyLine, []pick{conflict}, []pick{held}, nil) + "\n" + + gh := &fakeGH{prs: []pullRequest{notifyPR("20261026", body)}} + code, stdout, stderr := notifyRun(gh, "--mention", "@cozystack/maintainers", "--push") + if code != 0 || !strings.Contains(stdout, "release-1.1: 1 deferred conflicts on #90 wait for a maintainer") { + t.Fatalf("notify = %d, %q\n%s", code, stdout, stderr) + } + if len(gh.issueComments) != 1 { + t.Fatalf("%d comments, want one", len(gh.issueComments)) + } + c := gh.issueComments[0].Body + if !strings.Contains(c, "@cozystack/maintainers") || !strings.Contains(c, "[#6](") || !strings.Contains(c, "resolve-conflict.md") || strings.Contains(c, "#7") { + t.Errorf("the comment does not list #6 alone for a maintainer:\n%s", c) + } +} + +// Only a comment by an author the batch trusts is taken for the notify +// comment: a look-alike is neither edited nor counted. +func TestNotifyKeepsToItsOwnComment(t *testing.T) { + look := issueComment{ID: 7, Body: notifyMarker + "\nplanted"} + look.User.Login, look.User.Type = "someone", "User" + gh := &fakeGH{prs: []pullRequest{notifyPR("20261019", "")}, issueComments: []issueComment{look}} + if code, _, stderr := notifyRun(gh, "--push"); code != 0 { + t.Fatalf("notify = %d\n%s", code, stderr) + } + if len(gh.issueComments) != 2 || gh.issueComments[0].Body != look.Body { + t.Errorf("comments = %+v; want the look-alike untouched and one of ours", gh.issueComments) + } + + // Ours says so once nothing waits any more, and calls nobody for it. + gh.prs = []pullRequest{notifyPR("20261026", "")} + if code, _, _ := notifyRun(gh, "--mention", "@cozystack/maintainers", "--push"); code != 0 || len(gh.issueComments) != 2 || + !strings.Contains(gh.issueComments[1].Body, "Nothing waits") || strings.Contains(gh.issueComments[1].Body, "@cozystack") { + t.Errorf("comments = %+v; want ours updated to say nothing waits, mentioning nobody", gh.issueComments) + } +} + +// The scheduled batch writes as its bot: notify keeps to the comment that bot +// wrote, on a pull request it opened, and to no other bot's. +func TestNotifyFindsTheBotsComment(t *testing.T) { + const bot = "cozystack-ci[bot]" + ours := issueComment{ID: 7, Body: notifyMarker + "\nearlier"} + ours.User.Login, ours.User.Type = bot, "Bot" + other := issueComment{ID: 8, Body: notifyMarker + "\nplanted"} + other.User.Login, other.User.Type = "renovate[bot]", "Bot" + pr := notifyPR("20261019", "") + pr.User.Login, pr.User.Type = bot, "Bot" + gh := &fakeGH{prs: []pullRequest{pr}, issueComments: []issueComment{ours, other}, bot: bot} + if code, stdout, stderr := notifyRun(gh, "--push"); code != 0 || !strings.Contains(stdout, "has been open since") { + t.Fatalf("notify = %d, %q\n%s", code, stdout, stderr) + } + if len(gh.issueComments) != 2 || !strings.Contains(gh.issueComments[0].Body, "has been open since") || gh.issueComments[1].Body != other.Body { + t.Errorf("comments = %+v; want the bot's own comment edited and the other bot's untouched", gh.issueComments) + } + if code, _, stderr := notifyRun(&fakeGH{prs: []pullRequest{pr}, bot: "cozystack-ci"}, "--push"); code != 2 || !strings.Contains(stderr, batchBotEnv) { + t.Errorf("a %s that is no bot login = %d (%s), want a refusal", batchBotEnv, code, stderr) + } +} + +// A permission GitHub does not answer fails notify before it writes: read as +// a refusal, it would pass over a person's batch, or over the comment notify +// keeps and post a second one. +func TestNotifyFailsOnAnUnreadablePermission(t *testing.T) { + gh := &fakeGH{prs: []pullRequest{notifyPR("20261019", "")}, permErr: errString("HTTP 502")} + if code, stdout, stderr := notifyRun(gh, "--push"); code != 2 || stdout != "" || len(gh.issueComments) != 0 || !strings.Contains(stderr, "HTTP 502") { + t.Errorf("notify on a person's batch = %d, %q, %d comments (%s); want 2 and nothing written", code, stdout, len(gh.issueComments), stderr) + } + + const bot = "cozystack-ci[bot]" + pr := notifyPR("20261019", "") + pr.User.Login, pr.User.Type = bot, "Bot" + ours := issueComment{ID: 7, Body: notifyMarker + "\nearlier"} + ours.User.Login, ours.User.Type = "maintainer", "User" + gh = &fakeGH{prs: []pullRequest{pr}, issueComments: []issueComment{ours}, bot: bot, permErr: errString("HTTP 502")} + if code, _, stderr := notifyRun(gh, "--push"); code != 2 || len(gh.issueComments) != 1 || gh.issueComments[0].Body != ours.Body || !strings.Contains(stderr, "HTTP 502") { + t.Errorf("notify with a person's comment = %d, comments %+v (%s); want 2 and the comment untouched", code, gh.issueComments, stderr) + } +} + +func TestNotifyRefuses(t *testing.T) { + gh := &fakeGH{prs: []pullRequest{notifyPR("20261019", "")}} + for _, args := range [][]string{ + {"--mention", "@a b"}, + {"--mention", "@org/team/x"}, + {"--mention", "@-x"}, + } { + if code, _, _ := notifyRun(gh, args...); code != 2 { + t.Errorf("notify %q = %d, want 2", args, code) + } + } + if code, stdout, _ := notifyRun(&fakeGH{}, "--push"); code != 0 || stdout != "" { + t.Errorf("notify with no batch open = %d, %q; want nothing", code, stdout) + } + pr := notifyPR("20261019", "") + pr.User.Login = "someone" + if code, stdout, _ := notifyRun(&fakeGH{prs: []pullRequest{pr}}, "--push"); code != 0 || stdout != "" { + t.Errorf("notify on a batch someone untrusted opened = %d, %q; want nothing", code, stdout) + } +} + +// The maintainer's example: with 1.6 and 1.7 released, release-1.8 joins them +// when it branches, and 1.8.0 drops 1.6. +func TestSupportedLines(t *testing.T) { + l := func(minor int) line { return line{1, minor} } + cases := []struct { + name string + all []line + shipped []line + released int + want []line + }{ + {"two released", []line{l(5), l(6), l(7)}, []line{l(5), l(6), l(7)}, 2, []line{l(6), l(7)}}, + {"release-1.8 branched", []line{l(5), l(6), l(7), l(8)}, []line{l(5), l(6), l(7)}, 2, []line{l(6), l(7), l(8)}}, + {"1.8.0 shipped", []line{l(5), l(6), l(7), l(8)}, []line{l(5), l(6), l(7), l(8)}, 2, []line{l(7), l(8)}}, + {"three released", []line{l(5), l(6), l(7), l(8)}, []line{l(5), l(6), l(7)}, 3, []line{l(5), l(6), l(7), l(8)}}, + {"one released", []line{l(6), l(7), l(8)}, []line{l(6), l(7)}, 1, []line{l(7), l(8)}}, + {"a line between releases that never shipped", []line{l(5), l(6), l(7)}, []line{l(5), l(7)}, 2, []line{l(5), l(7)}}, + {"nothing shipped yet", []line{l(0), l(1)}, nil, 2, []line{l(0), l(1)}}, + } + for _, tc := range cases { + shipped := map[line]bool{} + for _, s := range tc.shipped { + shipped[s] = true + } + if got := supportedLines(tc.all, shipped, tc.released); !slices.Equal(got, tc.want) { + t.Errorf("%s: supportedLines = %v, want %v", tc.name, got, tc.want) + } + } +} + +// lines prints the lines the batch runs for, newest first, which is the order +// the scheduled run takes them in: the newest released lines, as many as the +// policy says, and a line branched since, whose stable release drops the +// oldest. +func TestLines(t *testing.T) { + f := newFixture(t) + f.release("release-0.9") + f.tag("v0.9.0") + f.lines() + f.release("release-1.2") + f.release("release-1.2.0-rc.1") + f.tag("v1.2.0-rc.1") + lines := func() string { + t.Helper() + code, stdout, stderr := f.exec(nil, "lines") + if code != 0 { + t.Fatalf("lines = %d (%s)", code, stderr) + } + return stdout + } + if got := lines(); got != "release-1.2\nrelease-1.1\nrelease-1.0\n" { + t.Errorf("with release-1.2 at its first rc, lines = %q; want it before the two released lines", got) + } + f.setPolicy(releasedKey + ": 3\n") + if got := lines(); got != "release-1.2\nrelease-1.1\nrelease-1.0\nrelease-0.9\n" { + t.Errorf("with three released lines, lines = %q; want release-0.9 too", got) + } + f.setPolicy("") + f.tag("v1.2.0") + if got := lines(); got != "release-1.2\nrelease-1.1\n" { + t.Errorf("once v1.2.0 ships, lines = %q; want release-1.0 dropped", got) + } + f.setPolicy(releasedKey + ": 3\n") + if got := lines(); got != "release-1.2\nrelease-1.1\nrelease-1.0\n" { + t.Errorf("once v1.2.0 ships, with three released lines, lines = %q", got) + } +} + +// A scheduled run with nothing to pick for a line succeeds without pushing or +// opening anything. +func TestRunWithNothingToPick(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "feat(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{} + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push") + if code != 0 || !strings.Contains(stderr, "nothing to push") { + t.Errorf("run with nothing to pick = %d, want 0:\n%s", code, stderr) + } + if f.remoteHead(batchBranch1) != "" || len(gh.created) != 0 { + t.Errorf("pushed %q and opened %q with nothing picked", f.remoteHead(batchBranch1), gh.created) + } +} diff --git a/cmd/backport-batch/onto_test.go b/cmd/backport-batch/onto_test.go new file mode 100644 index 000000000..ee4d52d33 --- /dev/null +++ b/cmd/backport-batch/onto_test.go @@ -0,0 +1,140 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "slices" + "strings" + "testing" +) + +// pushed builds and pushes a batch for #5 and returns its head. +func pushed(t *testing.T, f *fixture) string { + t.Helper() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + if code, _, stderr := f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--push"); code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + head := f.git("rev-parse", "HEAD") + f.git("switch", "-q", "main") + return head +} + +// --onto continues the pushed head and nothing else: it keeps what the batch +// carries and picks only what is new. +func TestOntoContinues(t *testing.T) { + f := newFixture(t) + f.lines() + head := pushed(t, f) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + code, stdout, stderr := f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--onto", head, "--date", "2026-10-29") + if code != 2 || !strings.Contains(stderr, "not the head of backport-batch/release-1.1-20261029") { + t.Errorf("--onto with another date = %d (%s), want a refusal", code, stderr) + } + f.now = f.now.AddDate(0, 0, 3) + code, stdout, stderr = f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--onto", head) + if code != 0 { + t.Fatalf("run --onto = %d\n%s", code, stderr) + } + if !strings.Contains(stderr, "title: fix(backport): batch for release-1.1, 2026-10-26\n") { + t.Errorf("the continued batch is not titled for the date it opened on:\n%s", stderr) + } + if got := f.subjects(head, "HEAD"); !slices.Equal(got, []string{"fix(app): six (#6)"}) { + t.Errorf("picked %q on top of the batch, want only #6", got) + } + if !descends(t, head, f.git("rev-parse", "HEAD")) { + t.Error("the new head does not descend from --onto") + } + if listed, _, _ := strings.Cut(stdout, "### Skipped"); !strings.Contains(listed, "[#5]") || !strings.Contains(listed, "[#6]") { + t.Errorf("the body does not list what the batch carries and what it picked:\n%s", stdout) + } +} + +// A head that is not the one the remote has now is refused, so work pushed +// since cannot be dropped. +func TestOntoRefusesAStaleHead(t *testing.T) { + f := newFixture(t) + f.lines() + head := pushed(t, f) + f.git("switch", "-q", "--detach", head) + f.write("app/fold.txt", "folded\n") + f.git("add", "-A") + f.git("commit", "-q", "-m", "fold") + f.git("push", "-q", "origin", "HEAD:refs/heads/"+batchBranch1) + f.git("switch", "-q", "main") + + code, _, stderr := f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--onto", head) + if code != 2 || !strings.Contains(stderr, "is not the head of any batch branch") { + t.Errorf("run --onto a stale head = %d (%s), want a refusal", code, stderr) + } + for _, bad := range []string{head[:12], "-" + head[1:], strings.ToUpper(head), "HEAD", "--help"} { + if code, _, _ := f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--onto", bad); code != 2 { + t.Errorf("--onto %q = %d, want 2", bad, code) + } + } +} + +// A rebuilt branch that does not descend from what the remote has is never +// pushed over it. +func TestPushRefusesANonFastForward(t *testing.T) { + f := newFixture(t) + f.lines() + head := pushed(t, f) + f.git("switch", "-q", "--detach", head) + f.write("app/fold.txt", "folded\n") + f.git("add", "-A") + f.git("commit", "-q", "-m", "fold") + folded := f.git("rev-parse", "HEAD") + f.git("push", "-q", "origin", "HEAD:refs/heads/"+batchBranch1) + f.git("switch", "-q", "main") + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + + code, _, stderr := f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--push") + // git would refuse too, but only after trying; the tool refuses first. + if code != 2 || !strings.Contains(stderr, "refusing a non-fast-forward push") { + t.Errorf("a fresh rebuild pushed over the batch = %d (%s), want a refusal", code, stderr) + } + if got := f.remoteHead(batchBranch1); got != folded { + t.Errorf("the remote batch moved to %.12s, want it left at %.12s", got, folded) + } +} + +// A run that pushed its branch and stopped before opening the pull request +// leaves the branch behind. A run the same day builds the batch anew and +// refuses to push over it, naming the run that continues it, which then +// opens the pull request. +func TestOrphanBranchNamesItsRecovery(t *testing.T) { + f := newFixture(t) + f.lines() + head := pushed(t, f) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + // A commit picked again gets a new committer date, so a new name. + t.Setenv("GIT_COMMITTER_DATE", "2026-10-26T07:00:00Z") + code, _, stderr := f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--continue", "--push") + recovery := "backport-batch run --line release-1.1 --onto " + head + " --push" + if code != 2 || !strings.Contains(stderr, recovery) { + t.Fatalf("a same-day run over the left branch = %d (%s), want a refusal naming %q", code, stderr, recovery) + } + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, strings.Fields(recovery)[1:]...); code != 0 { + t.Fatalf("%s = %d\n%s", recovery, code, stderr) + } + now := f.remoteHead(batchBranch1) + if !descends(t, head, now) || now == head || len(gh.created) != 1 { + t.Errorf("the recovery pushed %.12s over %.12s and opened %q; want a fast-forward with #6 and one pull request", now, head, gh.created) + } +} diff --git a/cmd/backport-batch/overrides_test.go b/cmd/backport-batch/overrides_test.go new file mode 100644 index 000000000..f1b1a5bcb --- /dev/null +++ b/cmd/backport-batch/overrides_test.go @@ -0,0 +1,247 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "os" + "path/filepath" + "slices" + "strings" + "testing" +) + +// overridden builds a line where backport/ labels decide: #5 a feature let in, +// #6 a fix kept out, #7 a breaking fix and #8 a migration let in, #9 carrying +// both labels, #10 a feature let in that the line already carries, adapted, so +// picking it again would conflict rather than come out empty. +func overridden(t *testing.T) (*fixture, *fakeGH) { + t.Helper() + f := newFixture(t) + f.lines() + f.pr(5, "feat(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.pr(7, "fix(app)!: seven", "", map[string]string{"app/seven.txt": "7\n"}) + f.pr(8, "fix(platform): eight", "", map[string]string{migrationsDir + "61": "#!/bin/sh\n"}) + f.pr(9, "fix(app): nine", "", map[string]string{"app/nine.txt": "9\n"}) + mc10 := f.pr(10, "feat(app): ten", "", map[string]string{"app/ten.txt": "10\n"}) + f.onLine("release-1.1", "feat(app): ten (#10)\n\n(cherry picked from commit "+mc10+")", map[string]string{"app/ten.txt": "10, adapted\n"}) + gh := &fakeGH{labels: map[string][]int{ + labelInclude: {5, 7, 8, 9, 10}, + labelSkip: {6, 9}, + }} + return f, gh +} + +func TestOverrides(t *testing.T) { + f, gh := overridden(t) + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + want := []string{"feat(app): five (#5)", "fix(app)!: seven (#7)", "fix(platform): eight (#8)"} + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, want) { + t.Errorf("picked %q, want %q", got, want) + } + section, _, _ := strings.Cut(stdout[strings.Index(stdout, "### Maintainer overrides"):], "### Skipped") + for n, outcome := range map[int]string{ + 5: "`backport/include`, on this branch", + 6: "`backport/skip`, left out", + 7: "`backport/include`, on this branch", + 8: "`backport/include`, on this branch", + 9: "`backport/skip`, left out", + 10: "`backport/include`, skipped by rule 1", + } { + if !strings.Contains(section, fmt.Sprintf("/pull/%d)", n)) || !strings.Contains(section, outcome) { + t.Errorf("the overrides section does not show #%d as %q:\n%s", n, outcome, section) + } + } + if skipped := stdout[strings.Index(stdout, "### Skipped"):]; strings.Contains(skipped, "/pull/6)") || strings.Contains(skipped, "/pull/9)") { + t.Errorf("a skipped-by-label original is listed under a rule:\n%s", skipped) + } +} + +// check holds the line to what the labels decide, and fold refuses an +// original kept out. +func TestOverridesInCheckAndFold(t *testing.T) { + f, gh := overridden(t) + code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1") + if code != 1 { + t.Fatalf("check = %d, want 1:\n%s", code, stdout) + } + for _, n := range []int{5, 7, 8} { + if !strings.Contains(stdout, fmt.Sprintf("/pull/%d ", n)) { + t.Errorf("check does not list #%d:\n%s", n, stdout) + } + } + for _, n := range []int{6, 9, 10} { + if strings.Contains(stdout, fmt.Sprintf("/pull/%d ", n)) { + t.Errorf("check lists #%d:\n%s", n, stdout) + } + } + + f.git("push", "-q", "origin", "origin/release-1.1:refs/heads/"+batchBranch1) + f.git("fetch", "-q", "origin") + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, "")} + f.git("switch", "-q", "--detach", "origin/"+batchBranch1) + f.write("app/six.txt", "6\n") + f.git("add", "-A") + f.git("commit", "-q", "-m", "port\n\n(cherry picked from commit "+f.git("rev-parse", "origin/main~4")+")") + commit := f.git("rev-parse", "HEAD") + f.git("switch", "-q", "main") + code, stdout, _ = f.exec(gh, "fold", "--line", "release-1.1", "--original", "6", "--commit", commit) + if code != 1 || !strings.Contains(stdout, "#6 carries backport/skip") { + t.Errorf("fold of a skipped original = %d, %q; want a refusal", code, stdout) + } +} + +// backport/include lifts rules 2 to 4, not rule 6: an older line still waits +// for the newer one. +func TestIncludeKeepsRuleSix(t *testing.T) { + f, gh := overridden(t) + f.git("fetch", "-q", "origin") + l, _ := parseLine("release-1.0") + b, err := (&app{gh: gh, stderr: &strings.Builder{}}).prepare(l) + if err != nil { + t.Fatal(err) + } + if v := b.find(5); v == nil || v.rule != ruleNewer { + t.Errorf("#5 on release-1.0 = %+v, want rule 6", v) + } + if v := b.find(10); v == nil || v.rule != 0 { + t.Errorf("#10 on release-1.0 = %+v, want it to qualify: release-1.1 carries it", v) + } +} + +func TestOverridesUnreadable(t *testing.T) { + f, gh := overridden(t) + gh.listErr = errString("gh: HTTP 502") + if code, _, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 2 || !strings.Contains(stderr, "HTTP 502") { + t.Errorf("check with labels unreadable = %d (%s), want 2", code, stderr) + } +} + +// ghCLI against a stub gh on PATH: the exact argv, and a listing that fills +// GitHub search's cap is refused rather than read as complete. +func TestGHCLI(t *testing.T) { + bin := t.TempDir() + args, out := filepath.Join(bin, "args"), filepath.Join(bin, "out") + stub := "#!/bin/sh\nprintf '%s\\n' \"$*\" >> \"" + args + "\"\ncat \"" + out + "\"\n" + if err := os.WriteFile(filepath.Join(bin, "gh"), []byte(stub), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH")) + write := func(s string) { + t.Helper() + if err := os.WriteFile(out, []byte(s), 0o644); err != nil { + t.Fatal(err) + } + _ = os.Remove(args) + } + called := func() string { + data, _ := os.ReadFile(args) + return strings.TrimSpace(string(data)) + } + + write(`[{"number":5},{"number":7}]`) + got, err := ghCLI{}.labelled(labelInclude) + if err != nil || !slices.Equal(got, []int{5, 7}) { + t.Errorf("labelled = %v, %v; want [5 7]", got, err) + } + if want := "pr list --base main --state merged --label backport/include --limit 1000 --json number"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } + + var full []string + for i := range searchResultCap { + full = append(full, fmt.Sprintf(`{"number":%d}`, i+1)) + } + write("[" + strings.Join(full, ",") + "]") + if _, err := (ghCLI{}).labelled(labelSkip); err == nil { + t.Error("a listing as long as the search cap was read as complete") + } + + write(`{"permission":"write"}`) + if p, err := (ghCLI{}).permission("maintainer"); err != nil || p != "write" { + t.Errorf("permission = %q, %v", p, err) + } + if want := "api repos/{owner}/{repo}/collaborators/maintainer/permission"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } + write(`{"permission":"admin"}`) + for _, login := range []string{"x/../../orgs/y", "-x", "a b", "github-actions[bot]", ""} { + if _, err := (ghCLI{}).permission(login); err == nil || called() != "" { + t.Errorf("permission(%q) = %v and called gh %q; want a refusal before gh", login, err, called()) + } + } + + write("[]") + if _, err := (ghCLI{}).openPulls("release-1.6"); err != nil { + t.Fatal(err) + } + if want := "api -X GET repos/{owner}/{repo}/pulls -f state=open -f base=release-1.6 -f per_page=100"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } + var page []string + for i := range openPullsCap { + page = append(page, fmt.Sprintf(`{"number":%d}`, i+1)) + } + write("[" + strings.Join(page, ",") + "]") + if _, err := (ghCLI{}).openPulls("release-1.6"); err == nil { + t.Error("a full page of open pull requests was read as all of them") + } + + write(`[[{"id":1,"body":"a","user":{"login":"x","type":"User"}}],[{"id":2,"body":"b","user":{"login":"y","type":"Bot"}}]]`) + comments, err := ghCLI{}.comments(90) + if err != nil || len(comments) != 2 || comments[1].ID != 2 || comments[1].User.Type != "Bot" { + t.Errorf("comments = %+v, %v; want both pages", comments, err) + } + if want := "api --paginate --slurp repos/{owner}/{repo}/issues/90/comments"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } + write("{}") + if err := (ghCLI{}).createComment(90, "/tmp/body.md"); err != nil { + t.Fatal(err) + } + if want := "api --method POST repos/{owner}/{repo}/issues/90/comments -F body=@/tmp/body.md"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } + write("{}") + if err := (ghCLI{}).editComment(12345678901, "/tmp/body.md"); err != nil { + t.Fatal(err) + } + if want := "api --method PATCH repos/{owner}/{repo}/issues/comments/12345678901 -F body=@/tmp/body.md"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } + + // A batch pull request carries no label: the release workflows act on a + // pull request cozystack-ci[bot] opened only when it has the release label. + write("") + if err := (ghCLI{}).createPull("release-1.7", batchBranch1, "fix(backport): batch", "/tmp/body.md"); err != nil { + t.Fatal(err) + } + if want := "pr create --draft --base release-1.7 --head " + batchBranch1 + " --title fix(backport): batch --body-file /tmp/body.md"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } + write("") + if err := (ghCLI{}).editPull(90, "fix(backport): batch", "/tmp/body.md"); err != nil { + t.Fatal(err) + } + if want := "pr edit 90 --title fix(backport): batch --body-file /tmp/body.md"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } +} diff --git a/cmd/backport-batch/policy.go b/cmd/backport-batch/policy.go new file mode 100644 index 000000000..ee74c76e8 --- /dev/null +++ b/cmd/backport-batch/policy.go @@ -0,0 +1,144 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "regexp" + "strconv" + "strings" + "time" +) + +// policyPage is the policy the batch applies. The block in it that names when +// each line started under the policy is read from the page itself, on the ref +// the run pins, so the page and the batch cannot drift apart. +const policyPage = "docs/release/backports.md" + +const policyFence = "```backport-policy" + +var ( + policyLineRE = regexp.MustCompile(`^(release-\d+\.\d+|` + batchOnlyKey + `):[ \t]*(\d{4}-\d{2}-\d{2})[ \t]*$`) + policyReleasedRE = regexp.MustCompile(`^` + releasedKey + `:[ \t]*(\d+)[ \t]*$`) +) + +const ( + // batchOnlyKey names the day from which the batch is the only way onto a + // line, so that from then on only its picks show what a line carries. + batchOnlyKey = "batch-only" + // releasedKey names how many of the lines that have shipped a stable + // release the batch runs for, the newest of them; defaultReleased when + // the block does not say. + releasedKey = "released-lines" + defaultReleased = 2 +) + +// The rule 2 types and the scopes that never go into a patch release, as the +// policy page lists them; policy_test.go holds the page to these. +var ( + takenTypes = []string{"fix", "security"} + excludedScopes = []string{"deps"} +) + +// starts maps a release line to the day it started under the policy. A line +// that is not listed started at its fork point from main. +type starts map[string]time.Time + +// policy is what the backport-policy block says: when each line started, the +// batch-only day, and how many released lines the batch runs for. +type policy struct { + starts starts + batchOnly time.Time + released int +} + +// parsePolicy reads the one backport-policy block of the page: a line per +// release line, `release-X.Y: YYYY-MM-DD`, exactly one `batch-only: +// YYYY-MM-DD`, at most one `released-lines: N`, and comments starting with #. +func parsePolicy(page string) (*policy, error) { + var blocks [][]string + var cur []string + in := false + for _, ln := range strings.Split(strings.ReplaceAll(page, "\r\n", "\n"), "\n") { + t := strings.TrimSpace(ln) + switch { + case !in && t == policyFence: + in, cur = true, nil + case in && t == "```": + in = false + blocks = append(blocks, cur) + case in: + cur = append(cur, t) + } + } + if in { + return nil, fmt.Errorf("%s: the %s block is not closed", policyPage, policyFence) + } + if len(blocks) != 1 { + return nil, fmt.Errorf("%s holds %d %s blocks, not one", policyPage, len(blocks), policyFence) + } + out := &policy{starts: starts{}, released: defaultReleased} + seen := map[string]bool{} + for _, t := range blocks[0] { + if t == "" || strings.HasPrefix(t, "#") { + continue + } + if m := policyReleasedRE.FindStringSubmatch(t); m != nil { + n, err := strconv.Atoi(m[1]) + switch { + case err != nil || n < 1: + return nil, fmt.Errorf("%s: %s is %q, not a number of lines", policyPage, releasedKey, m[1]) + case seen[releasedKey]: + return nil, fmt.Errorf("%s: %s is listed twice", policyPage, releasedKey) + } + seen[releasedKey], out.released = true, n + continue + } + m := policyLineRE.FindStringSubmatch(t) + if m == nil { + return nil, fmt.Errorf("%s: %q in the %s block is not `release-X.Y: YYYY-MM-DD`, `%s: YYYY-MM-DD` or `%s: N`", policyPage, t, policyFence, batchOnlyKey, releasedKey) + } + day, err := time.Parse(time.DateOnly, m[2]) + if err != nil { + return nil, fmt.Errorf("%s: %q is not a date: %w", policyPage, m[2], err) + } + if seen[m[1]] { + return nil, fmt.Errorf("%s: %s is listed twice", policyPage, m[1]) + } + seen[m[1]] = true + if m[1] == batchOnlyKey { + out.batchOnly = day + continue + } + if _, err := parseLine(m[1]); err != nil { + return nil, fmt.Errorf("%s: %w", policyPage, err) + } + out.starts[m[1]] = day + } + if !seen[batchOnlyKey] { + return nil, fmt.Errorf("%s: the %s block gives no %s day", policyPage, policyFence, batchOnlyKey) + } + return out, nil +} + +func readPolicy(commit string) (*policy, error) { + page, err := git("show", "--end-of-options", commit+":"+policyPage) + if err != nil { + return nil, fmt.Errorf("reading the policy at %.12s: %w", commit, err) + } + return parsePolicy(page) +} diff --git a/cmd/backport-batch/policy_test.go b/cmd/backport-batch/policy_test.go new file mode 100644 index 000000000..920c18a0b --- /dev/null +++ b/cmd/backport-batch/policy_test.go @@ -0,0 +1,212 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "os" + "regexp" + "slices" + "strings" + "testing" + "time" +) + +func TestParsePolicy(t *testing.T) { + got, err := parsePolicy("text\n\n" + policyFence + "\n# comment\n\nrelease-1.6: 2026-10-26\nrelease-1.5:2026-01-02\nbatch-only: 2026-10-20\n```\n") + if err != nil { + t.Fatal(err) + } + if len(got.starts) != 2 || !got.starts["release-1.6"].Equal(time.Date(2026, 10, 26, 0, 0, 0, 0, time.UTC)) { + t.Errorf("parsePolicy = %v", got) + } + if !got.batchOnly.Equal(time.Date(2026, 10, 20, 0, 0, 0, 0, time.UTC)) { + t.Errorf("the batch-only day = %v, want 2026-10-20", got.batchOnly) + } + if got.released != 2 { + t.Errorf("released lines with none given = %d, want 2", got.released) + } + if got, err := parsePolicy(policyFence + "\nbatch-only: 2026-10-20\n```\n"); err != nil || len(got.starts) != 0 { + t.Errorf("a block with no line = %v, %v; want no starts", got, err) + } + if got, err := parsePolicy(policyFence + "\nbatch-only: 2026-10-20\nreleased-lines: 3\n```\n"); err != nil || got.released != 3 { + t.Errorf("released-lines: 3 = %v, %v; want 3", got, err) + } + for name, page := range map[string]string{ + "no batch-only day": policyFence + "\nrelease-1.6: 2026-10-26\n```\n", + "two batch-only days": policyFence + "\nbatch-only: 2026-10-26\nbatch-only: 2026-10-27\n```\n", + "a bad batch-only day": policyFence + "\nbatch-only: soon\n```\n", + "no block": "## Backports\n", + "two blocks": policyFence + "\n```\n" + policyFence + "\n```\n", + "unclosed": policyFence + "\nrelease-1.6: 2026-10-26\n", + "a bad line": policyFence + "\nrelease 1.6 on 2026-10-26\n```\n", + "a bad date": policyFence + "\nrelease-1.6: 2026-13-40\n```\n", + "a padded line": policyFence + "\nrelease-01.6: 2026-10-26\n```\n", + "listed twice": policyFence + "\nrelease-1.6: 2026-10-26\nrelease-1.6: 2026-10-27\n```\n", + "not a line": policyFence + "\nmain: 2026-10-26\n```\n", + "trailing words": policyFence + "\nrelease-1.6: 2026-10-26 or so\n```\n", + "no released lines": policyFence + "\nbatch-only: 2026-10-26\nreleased-lines: 0\n```\n", + "released lines twice": policyFence + "\nbatch-only: 2026-10-26\nreleased-lines: 2\nreleased-lines: 3\n```\n", + "released lines words": policyFence + "\nbatch-only: 2026-10-26\nreleased-lines: two\n```\n", + "released lines huge": policyFence + "\nbatch-only: 2026-10-26\nreleased-lines: 99999999999999999999\n```\n", + } { + if _, err := parsePolicy(page); err == nil { + t.Errorf("%s: parsePolicy accepted it", name) + } + } +} + +// A line listed in the policy takes nothing merged before its start, unless a +// maintainer lets an original in; any other line starts at its fork point. +func TestPolicyStart(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "feat(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + gh := &fakeGH{labels: map[string][]int{labelInclude: {6, 7}}} + + verdicts := func() map[int]*verdict { + t.Helper() + f.git("fetch", "-q", "origin") + l, _ := parseLine("release-1.1") + b, err := (&app{gh: gh, stderr: &strings.Builder{}}).prepare(l) + if err != nil { + t.Fatal(err) + } + return map[int]*verdict{5: b.find(5), 6: b.find(6), 7: b.find(7)} + } + if v := verdicts(); v[5].rule != 0 || v[6].rule != 0 || v[7].rule != 0 { + t.Errorf("with no start every original qualifies: %d %d %d", v[5].rule, v[6].rule, v[7].rule) + } + + f.setPolicy("release-1.1: 2099-01-01\n") + v := verdicts() + if v[5].rule != ruleBeforeStart { + t.Errorf("#5 = rule %d (%s), want merged before the start", v[5].rule, v[5].reason) + } + if v[6].rule != 0 || v[7].rule != 0 { + t.Errorf("backport/include lifts the start: #6 = rule %d, #7 = rule %d", v[6].rule, v[7].rule) + } + + f.setPolicy("release-1.0: 2099-01-01\n") + if v := verdicts(); v[5].rule != 0 { + t.Errorf("a start for another line kept #5 out of release-1.1: rule %d", v[5].rule) + } + + f.setPolicy("release-1.1: 2000-01-01\n") + if v := verdicts(); v[5].rule != 0 { + t.Errorf("a start before the fork point kept #5 out: rule %d", v[5].rule) + } +} + +// The policy is read from main as the run pins it, or from the ref --policy +// names, and a run without a readable policy stops. Only the commands that +// read the policy take --policy. +func TestPolicyRef(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.git("switch", "-q", "-c", "proposal") + f.write(policyPage, policyWith("release-1.1: 2099-01-01\n"+releasedKey+": 1\n")) + f.git("commit", "-q", "-am", "docs: start release-1.1 later") + f.git("switch", "-q", "main") + + code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.1") + if code != 1 || !strings.Contains(stdout, "/pull/5 ") { + t.Errorf("check against main's policy = %d, %q, %s; want #5 missing", code, stdout, stderr) + } + code, stdout, stderr = f.exec(nil, "check", "--line", "release-1.1", "--policy", "proposal") + if code != 0 { + t.Errorf("check against the proposal's policy = %d, %q, %s; want #5 before the start", code, stdout, stderr) + } + if code, _, _ := f.exec(nil, "check", "--line", "release-1.1", "--policy", "--upload-pack=x"); code != 2 { + t.Errorf("--policy with an option = %d, want 2", code) + } + if code, stdout, stderr := f.exec(nil, "lines", "--policy", "proposal"); code != 0 || stdout != "release-1.1\n" { + t.Errorf("lines against the proposal's policy = %d, %q (%s); want release-1.1 alone", code, stdout, stderr) + } + for _, cmd := range []string{"verify", "notify"} { + if code, _, stderr := f.exec(&fakeGH{}, cmd, "--line", "release-1.1", "--policy", "proposal"); code != 2 || !strings.Contains(stderr, "flag provided but not defined: -policy") { + t.Errorf("%s --policy = %d (%s); want it refused, as %s reads no policy", cmd, code, stderr, cmd) + } + } + + f.squash(9001, "docs: drop the page", "", map[string]string{policyPage: ""}) + if code, _, stderr := f.exec(nil, "check", "--line", "release-1.1"); code != 2 || !strings.Contains(stderr, "reading the policy") { + t.Errorf("check without a policy page = %d (%s), want 2", code, stderr) + } +} + +// The policy page and the code say the same thing: the start block parses as +// the batch reads it, rule 2 names the types the code takes, the page names +// the scopes the code keeps out, and both override labels. +func TestPolicyPageMatchesCode(t *testing.T) { + data, err := os.ReadFile("../../" + policyPage) + if err != nil { + t.Fatal(err) + } + page := string(data) + pol, err := parsePolicy(page) + if err != nil { + t.Fatalf("the page's policy block does not parse: %v", err) + } + if day, ok := pol.starts["release-1.6"]; !ok || day.Format(time.DateOnly) != "2026-10-26" { + t.Errorf("release-1.6 starts %v, want 2026-10-26", pol.starts["release-1.6"]) + } + if day := pol.batchOnly.Format(time.DateOnly); day != "2026-10-26" { + t.Errorf("the batch-only day is %s, want 2026-10-26", day) + } + // The page says how many released lines the batch runs for, rather than + // leaving it to the default. + if !regexp.MustCompile("(?m)^"+releasedKey+":").MatchString(page) || pol.released != 2 { + t.Errorf("the page's block gives %d released lines, want 2 written out", pol.released) + } + + rule2 := regexp.MustCompile("(?m)^2\\. Its title type is (.*)$").FindStringSubmatch(page) + if rule2 == nil { + t.Fatal("the page has no rule 2") + } + var types []string + for _, m := range regexp.MustCompile("`([a-z]+)`").FindAllStringSubmatch(rule2[1], -1) { + types = append(types, m[1]) + } + if !slices.Equal(types, takenTypes) { + t.Errorf("rule 2 on the page takes %v, the code %v", types, takenTypes) + } + never := regexp.MustCompile("(?m)^### What waits for the next minor\\n\\n(.*)$").FindStringSubmatch(page) + if never == nil { + t.Fatal("the page has no list of what never goes into a patch release") + } + for _, scope := range excludedScopes { + if !strings.Contains(never[1], "(scope `"+scope+"`)") { + t.Errorf("the page does not keep scope %s out", scope) + } + } + if scopes := regexp.MustCompile("scope `([a-z]+)`").FindAllStringSubmatch(never[1], -1); len(scopes) != len(excludedScopes) { + t.Errorf("the page keeps %d scopes out, the code %d", len(scopes), len(excludedScopes)) + } + for i := 1; i <= 6; i++ { + if !regexp.MustCompile("(?m)^" + string(rune('0'+i)) + "\\. ").MatchString(page) { + t.Errorf("the page has no rule %d", i) + } + } + for _, label := range []string{labelInclude, labelSkip} { + if !strings.Contains(page, "`"+label+"`") { + t.Errorf("the page does not describe %s", label) + } + } +} diff --git a/cmd/backport-batch/rules.go b/cmd/backport-batch/rules.go new file mode 100644 index 000000000..632fd7ea8 --- /dev/null +++ b/cmd/backport-batch/rules.go @@ -0,0 +1,341 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "maps" + "path/filepath" + "regexp" + "slices" + "strings" + + "sigs.k8s.io/yaml" +) + +// Rule numbers as docs/release/backports.md numbers them. Rule 5, a clean +// pick, is decided while the batch is assembled. +const ( + ruleOnLine = 1 + ruleType = 2 + ruleBreaking = 3 + ruleTouches = 4 + ruleNewer = 6 + // Not a rule of the policy: the backport/skip override. + ruleSkipLabel = 100 + // Rule 1 for a line that started under the policy after it branched: + // merged before that start. + ruleBeforeStart = 101 +) + +const ( + // The numbered scripts run-migrations.sh walks; hack/check-migrations-target.sh + // reads the same directory. + migrationsDir = "packages/core/platform/images/migrations/migrations/" + // Where migrations.targetVersion is set; hack/check-migrations-target.sh + // reads it from here. + platformValues = "packages/core/platform/values.yaml" + // Only a file holding this text at either end of the change can hold a + // CRD document; crdDocs decides document by document. + crdKind = "CustomResourceDefinition" +) + +// The title grammar and the breaking footer are pr-labeler.yaml's, so that a +// pull request labelled kind/bug and kind/breaking-change reads the same here. +var ( + backportPrefixRE = regexp.MustCompile(`^(?:\[Backport [^\]]+\]\s+)+`) + conventionalRE = regexp.MustCompile(`^([a-z]+)(?:\(([^)]+)\))?(!)?:\s*\S`) + breakingFooterRE = regexp.MustCompile(`(?m)^BREAKING[ -]CHANGE:`) + htmlCommentRE = regexp.MustCompile(`(?s)`) + targetVersionRE = regexp.MustCompile(`(?m)^[+-][ \t]*targetVersion[ \t]*:`) +) + +// title is a parsed Conventional Commits title. +type title struct { + typ string + scopes []string + bang bool +} + +func parseTitle(s string) (title, bool) { + m := conventionalRE.FindStringSubmatch(backportPrefixRE.ReplaceAllString(s, "")) + if m == nil { + return title{}, false + } + t := title{typ: m[1], bang: m[3] == "!"} + for scope := range strings.SplitSeq(m[2], ",") { + if scope = strings.TrimSpace(scope); scope != "" { + t.scopes = append(t.scopes, scope) + } + } + return t, true +} + +// typeRule is rule 2 together with the policy's "never in a patch release": +// dependency bumps, scope deps, stay out whatever their type. +func typeRule(s string) string { + t, ok := parseTitle(s) + switch { + case !ok: + return "the title is not a Conventional Commits title" + case !slices.Contains(takenTypes, t.typ): + return "type " + t.typ + case slices.ContainsFunc(t.scopes, func(s string) bool { return slices.Contains(excludedScopes, s) }): + return "an excluded scope" + } + return "" +} + +// breakingRule is rule 3. The PR template quotes the footer inside an HTML +// comment, which a body merged with the template left in carries. +func breakingRule(s, body string) string { + if t, ok := parseTitle(s); ok && t.bang { + return "! in the title" + } + if breakingFooterRE.MatchString(htmlCommentRE.ReplaceAllString(body, "")) { + return "BREAKING CHANGE footer" + } + return "" +} + +// fileChange is one path a merge changes against its first parent. +type fileChange struct { + status byte + path string +} + +func changedFiles(from, to string) ([]fileChange, error) { + out, err := git("diff", "--no-ext-diff", "--no-textconv", "--name-status", "-z", "--no-renames", "--end-of-options", from, to) + if err != nil { + return nil, err + } + f := splitNUL(out) + if len(f)%2 != 0 { + return nil, fmt.Errorf("unreadable diff of %s..%s", from, to) + } + var changes []fileChange + for i := 0; i < len(f); i += 2 { + changes = append(changes, fileChange{status: f[i][0], path: f[i+1]}) + } + return changes, nil +} + +func paths(changes []fileChange) []string { + out := make([]string, 0, len(changes)) + for _, c := range changes { + out = append(out, c.path) + } + return out +} + +const ( + touchesMigrations = "platform migrations" + touchesTargetVersion = "targetVersion" + touchesAPI = "a sizeable API change" + touchesCRD = "a CRD" +) + +// touchesRule is rule 4: what of migrations, targetVersion, the API and CRDs +// the merge touches, or "" when it touches none of them. +func touchesRule(o *original, changes []fileChange) (string, error) { + hit := map[string]bool{} + for _, c := range changes { + p := c.path + switch { + case strings.HasPrefix(p, migrationsDir): + hit[touchesMigrations] = true + case p == platformValues: + diff, err := git("--literal-pathspecs", "diff", "-U0", "--no-ext-diff", "--no-textconv", "--end-of-options", o.merge+"^1", o.merge, "--", platformValues) + if err != nil { + return "", err + } + hit[touchesTargetVersion] = targetVersionRE.MatchString(diff) + } + } + var err error + if hit[touchesCRD], err = changesCRD(o.merge+"^1", o.merge, changes); err != nil { + return "", err + } + asks, err := readsAPI(o.merge+"^1", o.merge, changes) + if err != nil { + return "", err + } + apiDetail := "" + if asks { + if apiDetail, err = sizeableAPIChange(o); err != nil { + return "", err + } + hit[touchesAPI] = apiDetail != "" + } + var why []string + for _, r := range []string{touchesMigrations, touchesTargetVersion, touchesAPI, touchesCRD} { + switch { + case !hit[r]: + case r == touchesAPI: + why = append(why, r+" ("+apiDetail+")") + default: + why = append(why, r) + } + } + return strings.Join(why, ", "), nil +} + +// changesCRD reports whether the change adds, removes or modifies a +// CustomResourceDefinition document. It looks document by document: an edit to +// the DaemonSet beside a CRD in one manifest is not a CRD change, and a CRD in +// a test fixture changes nothing a cluster runs. +func changesCRD(parent, merge string, changes []fileChange) (bool, error) { + var files []string + for _, c := range changes { + if manifest(c.path) && !testFixture(c.path) { + files = append(files, c.path) + } + } + candidates := map[string]bool{} + for _, rev := range []string{parent, merge} { + found, err := grepFiles(rev, files, "-I", "-e", crdKind) + if err != nil { + return false, err + } + for _, f := range found { + candidates[f] = true + } + } + for _, c := range changes { + if !candidates[c.path] { + continue + } + var before, after map[string]string + if c.status != 'A' { + content, err := git("cat-file", "blob", parent+":"+c.path) + if err != nil { + return false, err + } + before = crdDocs(content) + } + if c.status != 'D' { + content, err := git("cat-file", "blob", merge+":"+c.path) + if err != nil { + return false, err + } + after = crdDocs(content) + } + if !maps.Equal(before, after) { + return true, nil + } + } + return false, nil +} + +// manifest reports whether path is a file a chart or a cluster can apply: YAML, +// JSON or a Helm template. A CRD quoted in documentation or code applies +// nothing. +func manifest(path string) bool { + switch filepath.Ext(path) { + case ".yaml", ".yml", ".json", ".tpl": + return true + } + dirs := strings.Split(path, "/") + return slices.ContainsFunc(dirs[:len(dirs)-1], func(d string) bool { return d == "templates" || d == "crds" }) +} + +func testFixture(path string) bool { + dirs := strings.Split(path, "/") + return slices.ContainsFunc(dirs[:len(dirs)-1], func(d string) bool { return d == "tests" || d == "testdata" }) +} + +// grepFiles lists which of files at rev hold any of the fixed strings that +// patterns give. Paths are passed literally, so a name such as ":(exclude)x" +// means only that file. +func grepFiles(rev string, files []string, patterns ...string) ([]string, error) { + const chunk = 500 + var found []string + for len(files) > 0 { + n := min(chunk, len(files)) + args := append([]string{"--literal-pathspecs", "grep", "-l", "-z", "-F"}, patterns...) + args = append(append(args, "--end-of-options", rev, "--"), files[:n]...) + stdout, stderr, code, err := gitStatus(args...) + switch { + case err != nil: + return nil, err + case code != 0 && code != 1: + return nil, fmt.Errorf("git grep at %s: exit %d: %s", rev, code, strings.TrimSpace(stderr)) + } + for _, hit := range splitNUL(stdout) { + if path, ok := strings.CutPrefix(hit, rev+":"); ok { + found = append(found, path) + } + } + files = files[n:] + } + return found, nil +} + +var ( + docSeparatorRE = regexp.MustCompile(`(?m)^---[ \t]*(?:#.*)?$`) + // A document that declares the kind on a line of its own, in YAML or in + // JSON, under a bare or quoted key, is a CRD even when it cannot be read; + // a sentence or a lookup call that names it declares nothing. + crdKindLineRE = regexp.MustCompile(`(?m)^[ \t]*(?:-[ \t]+)?(?:kind|"kind"|'kind')[ \t]*:[ \t]*["']?CustomResourceDefinition["']?[ \t]*,?[ \t]*(?:#.*)?$`) +) + +// crdDocs maps the CRD documents of a file to their text, keyed by +// metadata.name, so a CRD added, removed or edited changes the map. Every +// document that names the kind is read, so a CRD in compact JSON or under a +// quoted key counts as well. A document that declares the kind on a line of +// its own but cannot be read as an object, a CRD behind a Helm template guard +// for one, is kept under its own text, so that any change to it counts: the +// check fails closed. So does a list holding a CRD. +func crdDocs(content string) map[string]string { + docs := map[string]string{} + for _, doc := range docSeparatorRE.Split(content, -1) { + declares := crdKindLineRE.MatchString(doc) + if !declares && !strings.Contains(doc, crdKind) { + continue + } + var head struct { + Kind string `json:"kind"` + Metadata struct { + Name string `json:"name"` + } `json:"metadata"` + Items []struct { + Kind string `json:"kind"` + } `json:"items"` + } + key := "" + switch err := yaml.Unmarshal([]byte(doc), &head); { + case err != nil && declares: + key = "unreadable:" + doc + case err != nil: + continue + case head.Kind == crdKind && head.Metadata.Name != "": + key = "crd:" + head.Metadata.Name + case head.Kind == crdKind: + key = "unnamed:" + doc + case strings.HasSuffix(head.Kind, "List") && slices.ContainsFunc(head.Items, func(i struct { + Kind string `json:"kind"` + }) bool { + return i.Kind == crdKind + }): + key = "list:" + doc + default: + continue + } + docs[key] += doc + } + return docs +} diff --git a/cmd/backport-batch/rules_test.go b/cmd/backport-batch/rules_test.go new file mode 100644 index 000000000..ca1cd58a2 --- /dev/null +++ b/cmd/backport-batch/rules_test.go @@ -0,0 +1,461 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "maps" + "slices" + "strings" + "testing" +) + +func TestTypeRule(t *testing.T) { + cases := []struct { + title string + taken bool + }{ + {"fix(postgres): keep the replica count on resize", true}, + {"fix: typo in monitoring-agents values.yaml", true}, + {"security(dashboard): escape the tenant name", true}, + {"fix(ci): retry the flaky e2e step", true}, + {"[Backport release-1.6] fix(linstor): x", true}, + {"[Backport release-1.5] [Backport release-1.6] fix(linstor): x", true}, + {"fix(platform, dashboard): two scopes", true}, + {"feat(postgres): add a field", false}, + {"chore(release): promote", false}, + {"docs(release): split the page", false}, + {"refactor(api): move code", false}, + {"fix(deps): bump golang.org/x/net", false}, + {"fix(platform, deps): bump and fix", false}, + {"Fix(postgres): capitalised type", false}, + {"fix postgres without a colon", false}, + {"[postgres] bracket form", false}, + {"fixes(postgres): not the fix type", false}, + {"", false}, + } + for _, tc := range cases { + reason := typeRule(tc.title) + if (reason == "") != tc.taken { + t.Errorf("typeRule(%q) = %q, want taken=%v", tc.title, reason, tc.taken) + } + } +} + +func TestBreakingRule(t *testing.T) { + cases := []struct { + name, title, body string + breaking bool + }{ + {"plain fix", "fix(x): y", "## What this PR does\n\nFixes it.", false}, + {"bang", "fix(x)!: y", "", true}, + {"bang without scope", "fix!: y", "", true}, + {"bang behind a backport prefix", "[Backport release-1.6] fix(x)!: y", "", true}, + {"footer", "fix(x): y", "Body.\n\nBREAKING CHANGE: the field is gone", true}, + {"hyphenated footer", "fix(x): y", "BREAKING-CHANGE: the field is gone", true}, + {"footer quoted mid-line", "fix(x): y", "This is not a BREAKING CHANGE: nothing changes.", false}, + {"footer inside the template comment", "fix(x): y", "\nBody.", false}, + {"lower-case footer", "fix(x): y", "breaking change: no", false}, + } + for _, tc := range cases { + reason := breakingRule(tc.title, tc.body) + if (reason != "") != tc.breaking { + t.Errorf("%s: breakingRule = %q, want breaking=%v", tc.name, reason, tc.breaking) + } + } +} + +// Rule 4 reads the merge itself, so each case is a pull request merged into a +// real repository. +func TestTouchesRule(t *testing.T) { + cases := []struct { + name string + setup map[string]string // committed on main before the pull request + files map[string]string + want string + }{ + {name: "a chart change", files: map[string]string{"chart/values.yaml": "replicas: 2\nimage: app:1\n"}}, + {name: "a migration script", files: map[string]string{migrationsDir + "61": "#!/bin/sh\n"}, want: "platform migrations"}, + {name: "a migration library file", files: map[string]string{migrationsDir + "lib/common.sh": "x\n"}, want: "platform migrations"}, + {name: "the migration runner is not a migration", files: map[string]string{"packages/core/platform/images/migrations/run-migrations.sh": "x\n"}}, + { + name: "targetVersion moves", + setup: map[string]string{platformValues: "migrations:\n enabled: false\n targetVersion: 60\n"}, + files: map[string]string{platformValues: "migrations:\n enabled: false\n targetVersion: 61\n"}, + want: "targetVersion", + }, + { + name: "another key of the platform values", + setup: map[string]string{platformValues: "migrations:\n enabled: false\n targetVersion: 60\n"}, + files: map[string]string{platformValues: "migrations:\n enabled: true\n targetVersion: 60\n"}, + }, + {name: "api types the gate sees nothing in", setup: demoAPI, files: map[string]string{"api/apps/v1alpha1/types.go": "package v1alpha1\n"}}, + {name: "pkg/apis is not api/", files: map[string]string{"pkg/apis/apps/types.go": "package apps\n"}}, + {name: "a CRD added under crds/", files: map[string]string{"packages/system/x/charts/x/crds/a.yaml": crdDoc("as.x.io", "a") + "\n"}, want: "a CRD"}, + {name: "a ConfigMap under crds/ is not a CRD", files: map[string]string{"packages/system/x/charts/x/crds/a.yaml": "kind: ConfigMap\n"}}, + { + name: "a DaemonSet edit next to an unchanged CRD", + setup: map[string]string{"packages/system/x/templates/m.yaml": multiDoc(crdDoc("as.x.io", "a"), daemonSet("1"))}, + files: map[string]string{"packages/system/x/templates/m.yaml": multiDoc(crdDoc("as.x.io", "a"), daemonSet("2"))}, + }, + { + name: "a real CRD edit in a manifest with other documents", + setup: map[string]string{"packages/system/x/templates/m.yaml": multiDoc(crdDoc("as.x.io", "a"), daemonSet("1"))}, + files: map[string]string{"packages/system/x/templates/m.yaml": multiDoc(crdDoc("as.x.io", "b"), daemonSet("1"))}, + want: "a CRD", + }, + { + name: "a CRD renamed", + setup: map[string]string{"packages/system/x/templates/m.yaml": multiDoc(crdDoc("as.x.io", "a"), daemonSet("1"))}, + files: map[string]string{"packages/system/x/templates/m.yaml": multiDoc(crdDoc("bs.x.io", "a"), daemonSet("1"))}, + want: "a CRD", + }, + {name: "a CRD in a fixture under tests/", files: map[string]string{"packages/system/x/tests/render_test.yaml": crdDoc("as.x.io", "a")}}, + {name: "a CRD in a fixture under testdata/", files: map[string]string{"hack/testdata/crd.yaml": crdDoc("as.x.io", "a")}}, + { + name: "a templated CRD edited, which cannot be read", + setup: map[string]string{"packages/system/x/templates/m.yaml": multiDoc(templatedCRD("a"), daemonSet("1"))}, + files: map[string]string{"packages/system/x/templates/m.yaml": multiDoc(templatedCRD("b"), daemonSet("1"))}, + want: "a CRD", + }, + { + name: "a DaemonSet edit next to an unchanged templated CRD", + setup: map[string]string{"packages/system/x/templates/m.yaml": multiDoc(templatedCRD("a"), daemonSet("1"))}, + files: map[string]string{"packages/system/x/templates/m.yaml": multiDoc(templatedCRD("a"), daemonSet("2"))}, + }, + { + name: "an unparseable file that names the kind, changed", + setup: map[string]string{"x/broken.yaml": "kind: CustomResourceDefinition\nspec: [unclosed\n"}, + files: map[string]string{"x/broken.yaml": "kind: CustomResourceDefinition\nspec: [still unclosed\n"}, + want: "a CRD", + }, + { + name: "a lookup that names the kind in a Helm helper", + setup: map[string]string{"packages/system/x/templates/_helpers.tpl": "{{- $crd := lookup \"apiextensions.k8s.io/v1\" \"CustomResourceDefinition\" \"\" .crd -}}\n{{- if $crd }}a{{ end }}\n"}, + files: map[string]string{"packages/system/x/templates/_helpers.tpl": "{{- $crd := lookup \"apiextensions.k8s.io/v1\" \"CustomResourceDefinition\" \"\" .crd -}}\n{{- if $crd }}b{{ end }}\n"}, + }, + { + name: "a CRD example in documentation", + setup: map[string]string{"docs/crds.md": "Example:\n\n```yaml\nkind: CustomResourceDefinition\nmetadata:\n name: as.x.io\n```\n"}, + files: map[string]string{"docs/crds.md": "An example:\n\n```yaml\nkind: CustomResourceDefinition\nmetadata:\n name: bs.x.io\n```\n"}, + }, + {name: "a CRD in a template without a YAML extension", files: map[string]string{"packages/system/x/templates/crd.tmpl": crdDoc("as.x.io", "a")}, want: "a CRD"}, + {name: "a CRD added as JSON", files: map[string]string{"x/crd.json": `{"apiVersion": "apiextensions.k8s.io/v1",` + "\n" + ` "kind": "CustomResourceDefinition",` + "\n" + ` "metadata": {"name": "as.x.io"}}` + "\n"}, want: "a CRD"}, + {name: "a List of CRDs", files: map[string]string{"x/list.yaml": "kind: List\nitems:\n- kind: CustomResourceDefinition\n metadata:\n name: as.x.io\n"}, want: "a CRD"}, + {name: "a template declaring a CRD", files: map[string]string{"packages/system/x/templates/crd.yaml": "apiVersion: apiextensions.k8s.io/v1\nkind: CustomResourceDefinition\n"}, want: "a CRD"}, + {name: "a quoted declaration", files: map[string]string{"x/crd.yaml": " kind: \"CustomResourceDefinition\" # yes\n"}, want: "a CRD"}, + { + name: "a CRD changed in place", + setup: map[string]string{"x/crd.yaml": "kind: CustomResourceDefinition\nspec: {}\n"}, + files: map[string]string{"x/crd.yaml": "kind: CustomResourceDefinition\nspec: {a: 1}\n"}, + want: "a CRD", + }, + { + name: "a CRD deleted", + setup: map[string]string{"x/crd.yaml": "kind: CustomResourceDefinition\n"}, + files: map[string]string{"x/crd.yaml": ""}, + want: "a CRD", + }, + { + name: "a CRD turned into something else", + setup: map[string]string{"x/crd.yaml": "kind: CustomResourceDefinition\n"}, + files: map[string]string{"x/crd.yaml": "kind: ConfigMap\n"}, + want: "a CRD", + }, + {name: "a CRD list is not a CRD", files: map[string]string{"x/list.yaml": "kind: CustomResourceDefinitionList\n"}}, + {name: "a CRD as compact JSON", files: map[string]string{"x/crd.json": `{"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"name":"as.x.io"}}` + "\n"}, want: "a CRD"}, + {name: "a CRD under a quoted kind key", files: map[string]string{"x/crd.yaml": "'kind': CustomResourceDefinition\nmetadata:\n name: as.x.io\n"}, want: "a CRD"}, + { + name: "a templated CRD under a quoted kind key, edited", + setup: map[string]string{"packages/system/x/templates/m.yaml": strings.Replace(templatedCRD("a"), "kind:", "'kind':", 1)}, + files: map[string]string{"packages/system/x/templates/m.yaml": strings.Replace(templatedCRD("b"), "kind:", "'kind':", 1)}, + want: "a CRD", + }, + {name: "a ConfigMap as compact JSON naming the kind", files: map[string]string{"x/cm.json": `{"kind":"ConfigMap","data":{"note":"kind: CustomResourceDefinition"}}` + "\n"}}, + {name: "a CRD mentioned in prose", files: map[string]string{"x/README.md": "Set kind: CustomResourceDefinition here.\n"}}, + {name: "a file named like an option declaring a CRD", files: map[string]string{"-e.yaml": "kind: CustomResourceDefinition\n"}, want: "a CRD"}, + // As pathspec magic, :(icase)... would match a file of that name in + // another case, and there is none. + {name: "a file named like pathspec magic declaring a CRD", files: map[string]string{":(icase)zz-crd.yaml": "kind: CustomResourceDefinition\n"}, want: "a CRD"}, + { + name: "a migration and a sizeable API change together", + setup: demoAPI, + files: map[string]string{migrationsDir + "61": "x\n", "api/x.go": "package x\n", demoRD: cozyrd("demo", `{"type":"object","properties":{}}`)}, + want: "platform migrations, a sizeable API change (breaking-change: [v1alpha1] spec: field \"size\" was removed)", + }, + } + for i, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f := newFixture(t) + setup := maps.Clone(apiBase) + maps.Copy(setup, tc.setup) + f.squash(1000, "chore: setup", "", setup) + mc := f.pr(10+i, "fix(x): y", "", tc.files) + o := &original{number: 10 + i, merge: mc, parents: 2} + changes, err := changedFiles(mc+"^1", mc) + if err != nil { + t.Fatal(err) + } + got, err := touchesRule(o, changes) + if err != nil { + t.Fatal(err) + } + if got != tc.want { + t.Errorf("touchesRule = %q, want %q", got, tc.want) + } + }) + } +} + +// Rule 6: an older line takes only what the newer line already carries, and +// rule 1: neither line takes what it carries already. +func TestNewerLineAndOnLine(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(2, "fix(app): before nothing", "", map[string]string{"app/a.txt": "a\n"}) + f.pr(3, "fix(app): picked onto 1.1", "", map[string]string{"app/b.txt": "b\n"}) + mc3 := f.git("rev-parse", "HEAD") + f.pr(4, "fix(app): only on main", "", map[string]string{"app/c.txt": "c\n"}) + f.onLine("release-1.1", "fix(app): picked onto 1.1 (#3)\n\n(cherry picked from commit "+mc3+")", map[string]string{"app/b.txt": "b\n"}) + + for _, tc := range []struct { + line string + qualified []int + skipped map[int]int + }{ + {line: "release-1.1", qualified: []int{2, 4}, skipped: map[int]int{3: ruleOnLine}}, + // #1 is a feature; #2 and #4 are not on release-1.1 yet; #3 is. + {line: "release-1.0", qualified: []int{3}, skipped: map[int]int{1: ruleType, 2: ruleNewer, 4: ruleNewer}}, + } { + t.Run(tc.line, func(t *testing.T) { + f.git("fetch", "-q", "origin") + l, _ := parseLine(tc.line) + b, err := (&app{gh: &fakeGH{}, stderr: &strings.Builder{}}).prepare(l) + if err != nil { + t.Fatal(err) + } + var got []int + for _, v := range b.qualifying() { + got = append(got, v.o.number) + } + if !slices.Equal(got, tc.qualified) { + t.Errorf("qualifying = %v, want %v", got, tc.qualified) + } + for n, rule := range tc.skipped { + if v := b.find(n); v == nil || v.rule != rule { + t.Errorf("#%d: verdict %+v, want rule %d", n, v, rule) + } + } + }) + } +} + +// An original merged before the newer line branched is on it, even when the +// newer line has since changed the same lines and a pick would conflict. +func TestNewerLineCarriesWhatItBranchedWith(t *testing.T) { + f := newFixture(t) + f.release("release-1.0") + f.pr(2, "fix(app): two", "", map[string]string{"app/a.txt": "two\n"}) + f.release("release-1.1") + mc3 := f.pr(3, "fix(app): three", "", map[string]string{"app/a.txt": "three\n"}) + f.onLine("release-1.1", "fix(app): three (#3)\n\n(cherry picked from commit "+mc3+")", map[string]string{"app/a.txt": "three\n"}) + if v := verdictOn(t, f, nil, "release-1.0", 2); v.rule != 0 { + t.Errorf("#2 on release-1.0 = rule %d (%s), want it to qualify", v.rule, v.reason) + } +} + +// A line branched and not yet released is one the batch runs for, and rule 6 +// holds each older line to the next newer one. +func TestNewerLineInTheRCWindow(t *testing.T) { + f := newFixture(t) + f.lines() + f.release("release-1.2") + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + for _, tc := range []struct{ line, reason string }{ + {"release-1.2", ""}, + {"release-1.1", "not on release-1.2 yet"}, + {"release-1.0", "not on release-1.1 yet"}, + } { + v := verdictOn(t, f, nil, tc.line, 5) + if v.reason != tc.reason || (v.rule == 0) != (tc.reason == "") { + t.Errorf("#5 on %s = rule %d (%s), want %q", tc.line, v.rule, v.reason, tc.reason) + } + } +} + +func TestOnlySupportedLines(t *testing.T) { + f := newFixture(t) + f.release("release-0.9") + f.lines() + code, _, stderr := f.exec(nil, "check", "--line", "release-0.9") + if code != 2 || !strings.Contains(stderr, "not a line the batch runs for") { + t.Errorf("check on release-0.9 = %d (%s), want 2 and a refusal", code, stderr) + } +} + +// demoRD is an ApplicationDefinition where apigate looks for one, and demoAPI +// the tree the API cases start from. +const demoRD = "packages/system/demo-rd/cozyrds/demo.yaml" + +// apiBase is the least API a tree holds: apigate refuses a tree with none. +var apiBase = map[string]string{"packages/system/keep-rd/cozyrds/keep.yaml": cozyrd("keep", `{"type":"object"}`)} + +var demoAPI = map[string]string{ + demoRD: cozyrd("demo", `{"type":"object","properties":{"size":{"type":"string","description":"Size."}}}`), + "api/apps/v1alpha1/demo/types.go": "package demo\n\n// Size of it.\ntype Spec struct{ Size string }\n", +} + +func cozyrd(name, schema string) string { + return "apiVersion: cozystack.io/v1alpha1\nkind: ApplicationDefinition\nmetadata:\n name: " + name + + "\nspec:\n application:\n kind: " + strings.ToUpper(name[:1]) + name[1:] + "\n plural: " + name + "s\n singular: " + name + + "\n openAPISchema: '" + schema + "'\n" +} + +// Rule 4 holds back a change to the API only when the API review gate calls it +// sizeable: a new or removed resource or group, or a breaking schema change. +func TestAPIGateRule(t *testing.T) { + typesGo := "api/apps/v1alpha1/demo/types.go" + cases := []struct { + name string + files map[string]string + want string + }{ + {name: "a description reworded", files: map[string]string{ + demoRD: cozyrd("demo", `{"type":"object","properties":{"size":{"type":"string","description":"The size."}}}`), + typesGo: "package demo\n\n// The size.\ntype Spec struct{ Size string }\n", + }}, + {name: "an optional field added", files: map[string]string{ + demoRD: cozyrd("demo", `{"type":"object","properties":{"size":{"type":"string"},"zone":{"type":"string"}}}`), + typesGo: "package demo\n\ntype Spec struct{ Size, Zone string }\n", + }}, + {name: "a field removed", files: map[string]string{ + demoRD: cozyrd("demo", `{"type":"object","properties":{}}`), + typesGo: "package demo\n\ntype Spec struct{}\n", + }, want: "breaking-change"}, + {name: "a field's type changed", files: map[string]string{ + demoRD: cozyrd("demo", `{"type":"object","properties":{"size":{"type":"integer"}}}`), + typesGo: "package demo\n\ntype Spec struct{ Size int }\n", + }, want: "breaking-change"}, + {name: "a field made required", files: map[string]string{ + demoRD: cozyrd("demo", `{"type":"object","required":["size"],"properties":{"size":{"type":"string"}}}`), + typesGo: "package demo\n\n// +required\ntype Spec struct{ Size string }\n", + }, want: "breaking-change"}, + {name: "a resource added", files: map[string]string{ + "packages/system/other-rd/cozyrds/other.yaml": cozyrd("other", `{"type":"object"}`), + "api/apps/v1alpha1/other/types.go": "package other\n", + }, want: "new-resource"}, + {name: "a resource removed", files: map[string]string{ + demoRD: "", + typesGo: "", + }, want: "removed"}, + {name: "a breaking schema change that does not touch api/", files: map[string]string{ + demoRD: cozyrd("demo", `{"type":"object","properties":{}}`), + }, want: "breaking-change"}, + {name: "an aggregated group registered in a chart", files: map[string]string{ + "packages/system/extra/templates/apiservice.yaml": apiService("extra.cozystack.io"), + }, want: "new-api-group"}, + } + for i, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f := newFixture(t) + setup := maps.Clone(apiBase) + maps.Copy(setup, demoAPI) + f.squash(1000, "chore: setup", "", setup) + mc := f.pr(10+i, "fix(x): y", "", tc.files) + o := &original{number: 10 + i, merge: mc, parents: 2} + changes, err := changedFiles(mc+"^1", mc) + if err != nil { + t.Fatal(err) + } + got, err := touchesRule(o, changes) + if err != nil { + t.Fatal(err) + } + if (got != "") != (tc.want != "") || !strings.Contains(got, tc.want) { + t.Errorf("touchesRule = %q, want %q", got, tc.want) + } + }) + } +} + +func apiService(group string) string { + return "apiVersion: apiregistration.k8s.io/v1\nkind: APIService\nmetadata:\n name: v1alpha1." + group + + "\nspec:\n group: " + group + "\n version: v1alpha1\n" +} + +// The API review gate is asked about exactly the changes that reach what +// apigate parses at either end of the merge. +func TestReadsAPI(t *testing.T) { + crd := crdDoc("as.x.io", "a") + cases := []struct { + name string + setup map[string]string + files map[string]string + want bool + }{ + {name: "the apiserver's registrations", files: map[string]string{apiserverGo: "package apiserver\n"}, want: true}, + {name: "an ApplicationDefinition, which carries no marker", files: map[string]string{demoRD: "kind: ApplicationDefinition\n"}, want: true}, + {name: "a CRD in a chart", files: map[string]string{"packages/system/x/crds/a.yaml": crd}, want: true}, + {name: "an APIService under internal/", files: map[string]string{"internal/x/apiservice.yml": apiService("x.cozystack.io")}, want: true}, + { + name: "a CRD turned into something else", + setup: map[string]string{"packages/system/x/crds/a.yaml": crd}, + files: map[string]string{"packages/system/x/crds/a.yaml": "kind: ConfigMap\n"}, + want: true, + }, + {name: "a binary file with a marker", files: map[string]string{"packages/system/x/a.yaml": "\x00kind: APIService\n"}, want: true}, + {name: "chart values", files: map[string]string{"packages/system/x/values.yaml": "replicas: 2\n"}}, + {name: "Go types under api/", files: map[string]string{"api/apps/v1alpha1/demo/types.go": "package demo\n"}}, + {name: "a CRD outside the roots apigate walks", files: map[string]string{"hack/crds/a.yaml": crd}}, + {name: "a CRD in a template without a YAML extension", files: map[string]string{"packages/system/x/templates/crd.tpl": crd}}, + {name: "a file beside the ApplicationDefinitions", files: map[string]string{"packages/system/demo-rd/cozyrds/sub/demo.yaml": "kind: ApplicationDefinition\n"}}, + } + for i, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f := newFixture(t) + setup := maps.Clone(apiBase) + maps.Copy(setup, tc.setup) + f.squash(1000, "chore: setup", "", setup) + mc := f.pr(10+i, "fix(x): y", "", tc.files) + changes, err := changedFiles(mc+"^1", mc) + if err != nil { + t.Fatal(err) + } + got, err := readsAPI(mc+"^1", mc, changes) + if err != nil { + t.Fatal(err) + } + if got != tc.want { + t.Errorf("readsAPI = %v, want %v", got, tc.want) + } + }) + } +} + +func crdDoc(name, version string) string { + return "apiVersion: apiextensions.k8s.io/v1\nkind: CustomResourceDefinition\nmetadata:\n name: " + name + + "\nspec:\n group: x.io\n versions:\n - name: v1" + version + "\n" +} + +func daemonSet(replicas string) string { + return "apiVersion: apps/v1\nkind: DaemonSet\nmetadata:\n name: agent\nspec:\n template:\n spec:\n priority: {{ .Values.priority | default " + replicas + " }}\n" +} + +func templatedCRD(version string) string { + return "{{- if .Values.crds }}\n" + crdDoc("ts.x.io", version) + "{{- end }}\n" +} + +func multiDoc(docs ...string) string { return "---\n" + strings.Join(docs, "---\n") } diff --git a/cmd/backport-batch/shape.go b/cmd/backport-batch/shape.go new file mode 100644 index 000000000..07a5b91f1 --- /dev/null +++ b/cmd/backport-batch/shape.go @@ -0,0 +1,439 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "os" + "slices" + "strconv" + "strings" +) + +// A batch carries each original as one unit on its first-parent line. A +// merged pull request is a merge commit whose first parent is the batch before +// it and whose second parent ends a straight line of picks of the pull +// request's own commits, oldest first, each keeping its author and sign-offs; +// the merge says which merge commit on main it carries in a Backported-from +// trailer. A squash-merged original is one pick of its commit on main. + +// pickOriginal picks one original onto HEAD in that shape. A conflict on any +// of its commits takes the branch back to where it was before the original, +// so nothing of it is left half applied. Hooks are off so that each commit +// carries its message and -x line as git writes them; signing is left to the +// user's configuration. +func pickOriginal(v *verdict) (pick, error) { + p := pick{v: v} + base, err := resolve("HEAD") + if err != nil { + return p, err + } + for _, c := range v.commits { + status, files, err := applyPick("-c", "core.hooksPath=/dev/null", "cherry-pick", "-x", "--signoff", "--end-of-options", c) + if err != nil { + if rerr := resetTo(base); rerr != nil { + return p, rerr + } + return p, fmt.Errorf("cherry-pick of %.12s of #%d: %w", c, v.o.number, err) + } + if status == "conflict" { + p.status, p.files, p.failed = "conflict", files, c + return p, resetTo(base) + } + } + head, err := resolve("HEAD") + switch { + case err != nil: + return p, err + case head == base: + p.status = "empty" + return p, nil + case v.o.parents < 2: + p.status, p.commit = "picked", head + return p, nil + } + msgs, err := commitMessages([]string{v.o.merge}) + if err != nil { + return p, err + } + if p.commit, err = commitMerge(base, head, mergeMessage(msgs[v.o.merge].message, v.o.merge)); err != nil { + if rerr := resetTo(base); rerr != nil { + return p, rerr + } + return p, err + } + p.status = "picked" + return p, nil +} + +// mergeMessage is the message of a batch merge: the one of the merge commit +// on main, which says which pull request it was, and a paragraph of its own +// naming that merge commit, so nothing the original's text ends with can pass +// for the trailer. +func mergeMessage(message, merge string) string { + return strings.TrimRight(message, " \t\r\n") + "\n\n" + batchTrailer + ": " + merge + "\n" +} + +// commitMerge makes the merge of tip onto base, whose tree is tip's since tip +// descends from base, and moves the current branch from tip to it. +func commitMerge(base, tip, message string) (string, error) { + f, err := os.CreateTemp("", "backport-batch-merge-*.txt") + if err != nil { + return "", err + } + defer os.Remove(f.Name()) + if _, err := f.WriteString(message); err != nil { + f.Close() + return "", err + } + if err := f.Close(); err != nil { + return "", err + } + out, err := git("commit-tree", "-p", base, "-p", tip, "-F", f.Name(), tip+"^{tree}") + if err != nil { + return "", err + } + merge := strings.TrimSpace(out) + if err := checkSHA(merge); err != nil { + return "", fmt.Errorf("commit-tree: %w", err) + } + if _, err := git("update-ref", "-m", "backport-batch: merge", "HEAD", merge, tip); err != nil { + return "", err + } + return merge, nil +} + +// resetTo takes the current branch, its index and its tree back to commit, +// which run owns: it starts only in a clean tree. +func resetTo(commit string) error { + _, err := git("reset", "--quiet", "--hard", commit) + return err +} + +// unit is one original as a batch carries it: a commit on the batch's +// first-parent line, and for a merge the picks its second parent brings. +type unit struct { + commit string + parents []string + message string + // picks are the commits reachable from the second parent and not from the + // first, oldest first. + picks []string +} + +func (u unit) merge() bool { return len(u.parents) == 2 } + +func readUnit(commit string) (unit, error) { + msgs, err := commitMessages([]string{commit}) + if err != nil { + return unit{}, err + } + u := unit{commit: commit, parents: msgs[commit].parents, message: msgs[commit].message} + if u.merge() { + if u.picks, err = revList("--reverse", "--topo-order", "--end-of-options", u.parents[1], "^"+u.parents[0]); err != nil { + return unit{}, err + } + } + return u, nil +} + +// batchUnits lists the units of the batch from base to head, oldest first. +func batchUnits(head, base string) ([]unit, error) { + oids, err := revList("--reverse", "--first-parent", "--end-of-options", head, "^"+base) + if err != nil { + return nil, err + } + var units []unit + for _, oid := range oids { + u, err := readUnit(oid) + if err != nil { + return nil, err + } + units = append(units, u) + } + return units, nil +} + +// unitOriginal is the original a unit says it carries, or nil: the one its +// Backported-from trailer names, or the one its appended -x line picks. +func (b *batch) unitOriginal(u unit) *verdict { + if u.merge() { + if merges := batchedFrom(u.message); len(merges) == 1 { + return b.findMerge(merges[0]) + } + return nil + } + source := appendedSource(u.message) + if v := b.findMerge(source); v != nil { + return v + } + return b.pickedBy([]string{source}) +} + +// ownerOf is the original one of whose own commits source is, or nil. It reads +// every original's commits, so it is for explaining a refusal only. +func (b *batch) ownerOf(source string) (*verdict, error) { + for _, v := range b.verdicts { + if v.commits == nil { + commits, err := prCommits(v.o) + if err != nil { + return nil, err + } + v.commits = commits + } + if v.o.parents > 1 && slices.Contains(v.commits, source) { + return v, nil + } + } + return nil, nil +} + +// checkUnit says why u is not one original carried in the batch's shape, or +// "" when it is, and which original it carries: a merge whose second parent +// is a straight line from its first, of picks of the pull request's own +// commits in their order, each with the -x line of the commit it came from, +// that names the original's merge commit in its trailer; or for a squash one +// pick of its commit on main. Either changes something, and only files the +// original changes. A pick is known by the -x line git appended, not by +// lines its message brought along, and one that also claims another commit +// of the same original is refused. +func checkUnit(b *batch, u unit) (string, *verdict, error) { + switch len(u.parents) { + case 1: + return checkPick(b, u) + case 2: + return checkMerge(b, u) + } + return fmt.Sprintf("%.12s has %d parents", u.commit, len(u.parents)), nil, nil +} + +func checkPick(b *batch, u unit) (string, *verdict, error) { + source := appendedSource(u.message) + if source == "" { + return fmt.Sprintf("%.12s ends in no (cherry picked from commit …) line; a batch carries originals only", u.commit), nil, nil + } + v := b.findMerge(source) + if v == nil { + owner, err := b.ownerOf(source) + if err != nil { + return "", nil, err + } + if owner != nil { + return fmt.Sprintf("%.12s picks one commit of #%d on its own; a merged pull request goes in as a merge of all its commits", u.commit, owner.o.number), owner, nil + } + return fmt.Sprintf("%.12s picks %s, which merged no pull request into main after %s branched", u.commit, source, b.line.branch()), nil, nil + } + if v.o.parents > 1 { + return fmt.Sprintf("%.12s picks the merge commit of #%d as one commit; a merged pull request goes in as a merge of its own commits", u.commit, v.o.number), v, nil + } + why, err := contained(b, v, u.parents[0], u.commit) + return why, v, err +} + +func checkMerge(b *batch, u unit) (string, *verdict, error) { + merges := batchedFrom(u.message) + if len(merges) != 1 { + return fmt.Sprintf("%.12s is a merge with %d %s trailers, not one", u.commit, len(merges), batchTrailer), nil, nil + } + v := b.findMerge(merges[0]) + switch { + case v == nil: + return fmt.Sprintf("%.12s names %s, which merged no pull request into main after %s branched", u.commit, merges[0], b.line.branch()), nil, nil + case v.o.parents < 2: + return fmt.Sprintf("%.12s merges #%d, which main took as one commit; it goes in as one pick", u.commit, v.o.number), v, nil + case len(u.picks) == 0: + return fmt.Sprintf("%.12s merges nothing", u.commit), v, nil + } + if err := b.load(v); err != nil { + return "", nil, err + } + msgs, err := commitMessages(u.picks) + if err != nil { + return "", nil, err + } + prev, last := u.parents[0], -1 + for _, c := range u.picks { + ps := msgs[c].parents + if len(ps) != 1 || ps[0] != prev { + return fmt.Sprintf("%.12s is not on a straight line of picks from %.12s", c, u.parents[0]), v, nil + } + source := appendedSource(msgs[c].message) + if source == "" { + return fmt.Sprintf("%.12s ends in no (cherry picked from commit …) line", c), v, nil + } + i := slices.Index(v.commits, source) + if i < 0 { + owner, err := b.ownerOf(source) + switch { + case err != nil: + return "", nil, err + case owner != nil: + return fmt.Sprintf("%.12s picks a commit of #%d into the merge of #%d", c, owner.o.number, v.o.number), v, nil + } + return fmt.Sprintf("%.12s picks %s, which is not a commit of #%d", c, source, v.o.number), v, nil + } + if i <= last { + return fmt.Sprintf("%.12s picks %.12s out of the order of #%d's commits", c, source, v.o.number), v, nil + } + for _, claimed := range claimedSources(msgs[c].message) { + if claimed != source && (claimed == v.o.merge || slices.Contains(v.commits, claimed)) { + return fmt.Sprintf("%.12s claims more than one commit of #%d", c, v.o.number), v, nil + } + } + prev, last = c, i + } + why, err := contained(b, v, u.parents[0], u.commit) + return why, v, err +} + +// contained says why the change from base to tip is not inside the files v +// changes, or "" when it is. +func contained(b *batch, v *verdict, base, tip string) (string, error) { + if err := b.load(v); err != nil { + return "", err + } + changes, err := changedFiles(base, tip) + if err != nil { + return "", err + } + if len(changes) == 0 { + return fmt.Sprintf("%.12s changes nothing", tip), nil + } + allowed := paths(v.changes) + var extra []string + for _, p := range paths(changes) { + if !slices.Contains(allowed, p) { + extra = append(extra, strconv.Quote(p)) + } + } + if len(extra) > 0 { + return fmt.Sprintf("%.12s touches %s, which #%d does not touch", tip, strings.Join(extra, ", "), v.o.number), nil + } + return "", nil +} + +// replayUnit picks a unit of the batch again onto HEAD in its own shape: a +// pushed resolution keeps what it resolved. On a conflict it takes the branch +// back to where it was before the unit and names the commit that failed. +func replayUnit(u unit) (string, []string, error) { + base, err := resolve("HEAD") + if err != nil { + return "", nil, err + } + picks := u.picks + if !u.merge() { + picks = []string{u.commit} + } + for _, c := range picks { + status, files, err := applyPick("-c", "core.hooksPath=/dev/null", "cherry-pick", "--end-of-options", c) + if err == nil && status == "conflict" { + return c, files, resetTo(base) + } + if err != nil { + if rerr := resetTo(base); rerr != nil { + return "", nil, rerr + } + return "", nil, fmt.Errorf("picking %.12s again: %w", c, err) + } + } + tip, err := resolve("HEAD") + if err != nil || !u.merge() || tip == base { + return "", nil, err + } + _, err = commitMerge(base, tip, u.message) + return "", nil, err +} + +// summarize says, for each original the batch head carries in units, how many +// commits it brought and which of them differ from the commit they were picked +// from: a resolution, or a pick git had to fit to the line. +func summarize(b *batch, head string, on []*verdict) (map[*verdict]string, error) { + units, err := batchUnits(head, b.lineHead) + if err != nil { + return nil, err + } + out := map[*verdict]string{} + for _, u := range units { + v := b.unitOriginal(u) + if v == nil || !slices.Contains(on, v) { + continue + } + picks := u.picks + if !u.merge() { + picks = []string{u.commit} + } + msgs, err := commitMessages(picks) + if err != nil { + return nil, err + } + var changed []string + for _, c := range picks { + source := appendedSource(msgs[c].message) + if source == "" { + continue + } + same, err := samePatch(source, c) + if err != nil { + return nil, err + } + if !same { + changed = append(changed, codeSpan(c[:12])) + } + } + s := "1 commit" + if len(picks) != 1 { + s = fmt.Sprintf("%d commits", len(picks)) + } + if len(changed) > 0 { + s += ", changed in the pick: " + strings.Join(changed, ", ") + } + out[v] = s + } + return out, nil +} + +func plural(n int, word string) string { + if n == 1 { + return word + } + return word + "s" +} + +// samePatch says whether two commits make the same change to their first +// parents, by git patch-id of their diffs without context, so that a pick +// that only lands among other lines on the line counts as the same. +func samePatch(a, b string) (bool, error) { + pa, err := patchID(a) + if err != nil { + return false, err + } + pb, err := patchID(b) + return pa == pb, err +} + +func patchID(c string) (string, error) { + diff, err := git("diff", "--no-ext-diff", "--no-textconv", "--unified=0", "--end-of-options", c+"^", c) + if err != nil { + return "", err + } + out, err := gitInput(diff, "patch-id", "--stable") + if err != nil { + return "", err + } + id, _, _ := strings.Cut(strings.TrimSpace(out), " ") + return id, nil +} diff --git a/cmd/backport-batch/shape_test.go b/cmd/backport-batch/shape_test.go new file mode 100644 index 000000000..0ff713a7f --- /dev/null +++ b/cmd/backport-batch/shape_test.go @@ -0,0 +1,372 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "os" + "path/filepath" + "slices" + "strings" + "testing" +) + +// A conflict on a commit in the middle of a pull request takes the batch back +// to where it was before that original: none of its commits stays, the +// conflict is recorded with the commit that failed, and the next original is +// still picked. +func TestRunConflictInAMiddleCommit(t *testing.T) { + f := newFixture(t) + f.lines() + f.onLine("release-1.1", "fix(chart): a line-only value", map[string]string{"chart/values.yaml": "replicas: 3\nimage: app:1\n"}) + mc5 := f.prCommits(5, "fix(chart): five", "", + change{files: map[string]string{"app/five.txt": "5\n"}}, + change{files: map[string]string{"chart/values.yaml": "replicas: 5\nimage: app:1\n"}}, + change{files: map[string]string{"app/five.txt": "5, done\n"}}) + own := strings.Fields(f.git("rev-list", "--reverse", mc5+"^1.."+mc5+"^2")) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(app): six (#6)"}) { + t.Errorf("the batch carries %q, want only #6", got) + } + if log := f.git("log", "--format=%B", "origin/release-1.1..HEAD"); strings.Contains(log, own[0]) || strings.Contains(log, own[1]) { + t.Errorf("a commit of #5 stayed on the batch:\n%s", log) + } + if out := f.git("status", "--porcelain"); out != "" { + t.Errorf("the tree is left dirty:\n%s", out) + } + if _, err := os.Stat(filepath.Join(f.dir, ".git", "CHERRY_PICK_HEAD")); err == nil { + t.Error("a cherry-pick is left in progress") + } + l, _ := parseLine("release-1.1") + deferred, err := parseDeferred(stdout, l) + if err != nil || deferred[5].Commit != own[1] || !slices.Equal(deferred[5].Files, []string{"chart/values.yaml"}) { + t.Errorf("deferred = %+v, %v; want #5 at %s in chart/values.yaml", deferred, err, own[1]) + } + if !strings.Contains(stdout, "`fix(chart): five` at `"+own[1][:12]+"`: `chart/values.yaml`") { + t.Errorf("the body does not name the commit that failed:\n%s", stdout) + } +} + +// What counts as carrying a merged pull request of several commits: picks +// whose appended -x lines name every commit of its own, as a batch merge's do, +// or a pick whose appended -x line names its merge commit; not some of its +// commits. A Backported-from trailer counts only on a commit from before the +// batch-only day, and only in its trailer block. +func TestPresenceOfAMultiCommitOriginal(t *testing.T) { + cases := []struct { + name string + land func(f *fixture, mc string, own []string) + carried string + }{ + { + name: "a batch merge of its commits", + land: func(f *fixture, mc string, own []string) { + f.git("fetch", "-q", "origin") + base := f.git("rev-parse", "origin/release-1.1") + f.git("switch", "-q", "--detach", base) + for _, c := range own { + f.git("cherry-pick", "-x", c) + } + f.git("switch", "-q", "--detach", f.joined(base, f.git("rev-parse", "HEAD"), "fix(app): five (#5)\n\n"+batchTrailer+": "+mc)) + f.git("push", "-q", "origin", "HEAD:refs/heads/release-1.1") + f.git("switch", "-q", "main") + }, + carried: "its 2 commits picked", + }, + { + name: "a pick whose -x line names the merge commit", + land: func(f *fixture, mc string, _ []string) { + f.onLine("release-1.1", "port\n\n(cherry picked from commit "+mc+")", map[string]string{"app/a.txt": "a\n"}) + }, + carried: "picked in", + }, + { + name: "picks whose -x lines name every commit of its own", + land: func(f *fixture, _ string, own []string) { + f.onLine("release-1.1", "a\n\n(cherry picked from commit "+own[0]+")", map[string]string{"app/a.txt": "a\n"}) + f.onLine("release-1.1", "b\n\n(cherry picked from commit "+own[1]+")", map[string]string{"app/b.txt": "b, half\n"}) + }, + carried: "its 2 commits picked", + }, + { + name: "a pick of one of its two commits", + land: func(f *fixture, _ string, own []string) { + f.onLine("release-1.1", "a\n\n(cherry picked from commit "+own[0]+")", map[string]string{"app/a.txt": "a\n"}) + }, + }, + { + name: "the trailer alone, from before the batch-only day", + land: func(f *fixture, mc string, _ []string) { + f.t.Setenv("GIT_COMMITTER_DATE", "2026-10-01T00:00:00Z") + f.onLine("release-1.1", "fix(app): five (#5)\n\n"+batchTrailer+": "+mc, map[string]string{"app/a.txt": "a\n"}) + }, + carried: "named by a " + batchTrailer + " trailer", + }, + { + name: "the trailer alone, after the batch-only day", + land: func(f *fixture, mc string, _ []string) { + f.t.Setenv("GIT_COMMITTER_DATE", "2026-10-27T00:00:00Z") + f.onLine("release-1.1", "fix(app): five (#5)\n\n"+batchTrailer+": "+mc, map[string]string{"app/a.txt": "a\n"}) + }, + }, + { + name: "the trailer in the body, from before the batch-only day", + land: func(f *fixture, mc string, _ []string) { + f.t.Setenv("GIT_COMMITTER_DATE", "2026-10-01T00:00:00Z") + f.onLine("release-1.1", "port\n\n"+batchTrailer+": "+mc+"\n\nMore to say.", map[string]string{"app/a.txt": "a\n"}) + }, + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f := newFixture(t) + f.lines() + mc := f.prCommits(5, "fix(app): five", "", + change{files: map[string]string{"app/a.txt": "a\n"}}, + change{files: map[string]string{"app/b.txt": "b\n"}}) + own := strings.Fields(f.git("rev-list", "--reverse", mc+"^1.."+mc+"^2")) + tc.land(f, mc, own) + v := verdictOn(t, f, nil, "release-1.1", 5) + switch { + case tc.carried == "" && v.rule != 0: + t.Errorf("verdict = rule %d (%s), want it to qualify", v.rule, v.reason) + case tc.carried != "" && (v.rule != ruleOnLine || !strings.HasPrefix(v.reason, tc.carried)): + t.Errorf("verdict = rule %d (%s), want rule 1 with %q", v.rule, v.reason, tc.carried) + } + }) + } +} + +// What a pull request's author writes reaches the line with every commit the +// batch picks and with the merge message it carries. None of it makes another +// original count as carried: not an -x line, a Backported-from trailer or a +// Backport of naming it, not at the end of the message where the line git +// appends lands in the same trailer block, and not an -x line naming one of +// the other original's own commits. The line is read as after the batch-only +// day, as a batch made after it leaves it. +func TestPresenceCannotBeClaimedByText(t *testing.T) { + type world struct { + f *fixture + mc5, c5 string + } + claims := []struct { + name string + claim func(w *world) string + }{ + {"an -x line naming the merge commit", func(w *world) string { return "(cherry picked from commit " + w.mc5 + ")" }}, + {"an -x line naming its own commit", func(w *world) string { return "(cherry picked from commit " + w.c5 + ")" }}, + {"a Backported-from trailer", func(w *world) string { return batchTrailer + ": " + w.mc5 }}, + {"a Backport of", func(w *world) string { return "Backport of #5" }}, + {"all of them after a sign-off", func(w *world) string { + return "Signed-off" + "-by: " + contributor + "\n(cherry picked from commit " + w.mc5 + ")\n(cherry picked from commit " + w.c5 + ")\n" + batchTrailer + ": " + w.mc5 + }}, + } + for _, where := range []string{"commit message", "pull request body"} { + for _, tc := range claims { + t.Run(where+": "+tc.name, func(t *testing.T) { + f := newFixture(t) + f.lines() + // #5 conflicts on the line, so the batch defers it. + f.onLine("release-1.1", "fix(chart): a line-only value", map[string]string{"chart/values.yaml": "replicas: 3\nimage: app:1\n"}) + w := &world{f: f} + w.mc5 = f.pr(5, "fix(chart): five", "", map[string]string{"chart/values.yaml": "replicas: 5\nimage: app:1\n"}) + w.c5 = f.git("rev-parse", w.mc5+"^2") + claim := tc.claim(w) + msg, body := "fix(app): six\n\n"+claim, "" + if where == "pull request body" { + msg, body = "fix(app): six", "Six.\n\n"+claim + } + f.prCommits(6, "fix(app): six", body, change{files: map[string]string{"app/six.txt": "6\n"}, message: msg}) + t.Setenv("GIT_COMMITTER_DATE", "2026-10-27T06:00:00Z") + + // The batch picks #6; on the open batch pull request the text + // claims nothing for #5. + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, gh.bodies[0])} + if v := verdictOn(t, f, gh, "release-1.1", 6); v.onBatch == "" { + t.Fatal("#6 is not on the batch") + } + if v := verdictOn(t, f, gh, "release-1.1", 5); v.onBatch != "" { + t.Errorf("#5 is on the batch in %.12s, want nowhere", v.onBatch) + } + + // Nor on the line, once the line takes the batch. + f.git("push", "-q", "origin", f.remoteHead(batchBranch1)+":refs/heads/release-1.1") + if v := verdictOn(t, f, nil, "release-1.1", 6); v.rule != ruleOnLine { + t.Fatalf("#6 = rule %d (%s), want it on the line", v.rule, v.reason) + } + if v := verdictOn(t, f, nil, "release-1.1", 5); v.rule != 0 { + t.Errorf("#5 = rule %d (%s), want it still to qualify", v.rule, v.reason) + } + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/5 ") { + t.Errorf("check = %d, want #5 missing:\n%s\n%s", code, stdout, stderr) + } + }) + } + } +} + +// verify passes a batch the run built and fails one with an original that is +// not in the batch's shape, reading nothing from GitHub when given the head. +func TestVerify(t *testing.T) { + type world struct { + f *fixture + gh *fakeGH + head string + mc5 string + sq6 string + } + setup := func(t *testing.T) *world { + f := newFixture(t) + f.lines() + w := &world{f: f} + w.mc5 = f.prCommits(5, "fix(app): five", "", + change{files: map[string]string{"app/five.txt": "5\n"}}, + change{files: map[string]string{"app/five.txt": "5, done\n"}}) + w.sq6 = f.squash(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + w.gh = &fakeGH{} + w.head = openBatch(t, f, w.gh) + return w + } + // on adds a commit with message and files on top of the batch head. + on := func(w *world, message string, files map[string]string) string { + w.f.git("switch", "-q", "--detach", w.head) + w.f.apply(files) + w.f.git("commit", "-q", "--allow-empty", "-m", message) + sha := w.f.git("rev-parse", "HEAD") + w.f.git("switch", "-q", "main") + return sha + } + cases := []struct { + name string + head func(w *world) string + refusal string + }{ + {name: "the batch as the run built it", head: func(w *world) string { return w.head }}, + { + name: "an original picked as one commit with -m 1", + head: func(w *world) string { + return on(w, "fix(app): five (#5)\n\n(cherry picked from commit "+w.mc5+")", map[string]string{"app/five.txt": "5, again\n"}) + }, + refusal: "picks the merge commit of #5 as one commit", + }, + { + name: "a commit that is no original", + head: func(w *world) string { + return on(w, "fix(ci): adjust for the line", map[string]string{"ci.txt": "x\n"}) + }, + refusal: "ends in no (cherry picked from commit", + }, + { + name: "an original carried twice", + head: func(w *world) string { + own := strings.Fields(w.f.git("rev-list", "--reverse", w.mc5+"^1.."+w.mc5+"^2")) + p := on(w, "again\n\n(cherry picked from commit "+own[1]+")", map[string]string{"app/five.txt": "5, done again\n"}) + w.f.git("switch", "-q", "--detach", w.head) + w.f.git("merge", "-q", "--no-ff", "-m", "fix(app): five (#5)\n\n"+batchTrailer+": "+w.mc5, p) + sha := w.f.git("rev-parse", "HEAD") + w.f.git("switch", "-q", "main") + return sha + }, + refusal: "carries #5 again", + }, + { + name: "a squash-merged original carried as a merge", + head: func(w *world) string { + p := on(w, "six\n\n(cherry picked from commit "+w.sq6+")", map[string]string{"app/six.txt": "6, again\n"}) + w.f.git("switch", "-q", "--detach", w.head) + w.f.git("merge", "-q", "--no-ff", "-m", "fix(app): six (#6)\n\n"+batchTrailer+": "+w.sq6, p) + sha := w.f.git("rev-parse", "HEAD") + w.f.git("switch", "-q", "main") + return sha + }, + refusal: "which main took as one commit", + }, + { + name: "a merge of the line into the batch", + head: func(w *world) string { + line := w.f.onLine("release-1.1", "fix(chart): on the line", map[string]string{"chart/values.yaml": "replicas: 2\nimage: app:1\n"}) + w.f.git("switch", "-q", "--detach", w.head) + w.f.git("merge", "-q", "--no-ff", "-m", "Merge branch 'release-1.1' into the batch", line) + sha := w.f.git("rev-parse", "HEAD") + w.f.git("switch", "-q", "main") + return sha + }, + refusal: "with 0 " + batchTrailer + " trailers", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + w := setup(t) + head := tc.head(w) + // GitHub is not asked: a failing fake would fail the run. + gh := &fakeGH{err: errString("gh: not to be asked")} + code, stdout, stderr := w.f.exec(gh, "verify", "--line", "release-1.1", "--head", head) + switch { + case tc.refusal == "" && code != 0: + t.Errorf("verify = %d\n%s\n%s", code, stdout, stderr) + case tc.refusal != "" && (code != 1 || !strings.Contains(stdout, tc.refusal)): + t.Errorf("verify = %d, want 1 with %q:\n%s\n%s", code, tc.refusal, stdout, stderr) + } + }) + } + + t.Run("the open batch pull request", func(t *testing.T) { + w := setup(t) + code, stdout, stderr := w.f.exec(w.gh, "verify", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, "3 units at "+w.head[:12]) { + t.Errorf("verify = %d\n%s\n%s", code, stdout, stderr) + } + if code, stdout, _ := w.f.exec(&fakeGH{}, "verify", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, "nothing to verify") { + t.Errorf("verify with no batch open = %d, %q", code, stdout) + } + }) + t.Run("a head that is not here", func(t *testing.T) { + w := setup(t) + if code, _, stderr := w.f.exec(w.gh, "verify", "--line", "release-1.1", "--head", strings.Repeat("ab", 20)); code != 2 || !strings.Contains(stderr, "fetch it first") { + t.Errorf("verify = %d (%s), want 2", code, stderr) + } + }) +} + +// A pick that lands among lines the line changed is the same change, and the +// body does not call it changed; one whose change differs is called so. +func TestSummaryCallsOnlyChangedPicksChanged(t *testing.T) { + f := newFixture(t) + f.release("release-1.0") + f.squash(2, "chore(app): a file to change", "", map[string]string{"app/ctx.txt": "a\nb\nc\nd\ne\n"}) + f.pr(1, "feat(app): something for 1.1", "", map[string]string{"app/feature.txt": "1.1\n"}) + f.release("release-1.1") + f.onLine("release-1.1", "fix(app): a line-only change", map[string]string{"app/ctx.txt": "A\nb\nc\nd\ne\n"}) + f.pr(5, "fix(app): five", "", map[string]string{"app/ctx.txt": "a\nb\nC\nd\ne\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if !strings.Contains(stdout, "`fix(app): five`, 1 commit\n") { + t.Errorf("a pick git only fit to its place is called changed:\n%s", stdout) + } +} diff --git a/cmd/backport-batch/verify.go b/cmd/backport-batch/verify.go new file mode 100644 index 000000000..05023a259 --- /dev/null +++ b/cmd/backport-batch/verify.go @@ -0,0 +1,139 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "strings" +) + +// verify checks every original a batch carries, from the line's head to the +// batch head, against the shape the batch carries it in: what fold checks for +// one resolution, for all of them and for the clean picks too. fold runs only +// where someone resolved a conflict and chose to run it; CI runs verify on +// every push to a batch pull request. +// +// With --head it reads that commit and nothing from GitHub, which is how CI +// runs it: the tool built from main, the batch head fetched as data. +func (a *app) verify(args []string) int { + fs := a.flags("verify") + lineArg := fs.String("line", "", "release line, release-X.Y") + headArg := fs.String("head", "", "the batch head to verify (default: the head of the open batch PR)") + if err := parse(fs, args); err != nil { + return 2 + } + l, err := parseLine(*lineArg) + if err != nil { + return a.fail(err) + } + if *headArg != "" { + if err := checkSHA(*headArg); err != nil { + return a.fail(fmt.Errorf("--head: %w", err)) + } + } + if err := fetch(); err != nil { + return a.fail(err) + } + b, err := shapeOf(l) + if err != nil { + return a.fail(err) + } + head := *headArg + if head == "" { + prs, err := a.gh.openPulls(l.branch()) + if err != nil { + return a.fail(err) + } + pr, warnings, err := batchPull(a.gh, prs, l) + for _, w := range warnings { + a.warn("%s", w) + } + if err != nil { + return a.fail(err) + } + if pr == nil { + fmt.Fprintf(a.stdout, "%s: no batch pull request is open; nothing to verify\n", l.branch()) + return 0 + } + if head, err = batchHead(pr); err != nil { + return a.fail(err) + } + } else if have, err := lookup(head); err != nil { + return a.fail(err) + } else if have == "" { + return a.fail(fmt.Errorf("--head %.12s is not in this repository; fetch it first", head)) + } + units, err := batchUnits(head, b.lineHead) + if err != nil { + return a.fail(err) + } + seen := map[*verdict]string{} + var refused []string + for _, u := range units { + why, v, err := checkUnit(b, u) + if err != nil { + return a.fail(err) + } + if why == "" && seen[v] != "" { + why = fmt.Sprintf("carries #%d again, which %.12s already carries", v.o.number, seen[v]) + } + if why != "" { + refused = append(refused, why) + fmt.Fprintf(a.stdout, "refused %.12s: %s\n", u.commit, why) + continue + } + seen[v] = u.commit + n := len(u.picks) + if !u.merge() { + n = 1 + } + fmt.Fprintf(a.stdout, "ok %.12s #%d, %d %s\n", u.commit, v.o.number, n, plural(n, "commit")) + } + if len(refused) > 0 { + fmt.Fprintf(a.stdout, "%s: %d of %d units at %.12s are not originals in the batch's shape\n", l.branch(), len(refused), len(units), head) + return 1 + } + fmt.Fprintf(a.stdout, "%s: %d %s at %.12s, each one original in the batch's shape\n", l.branch(), len(units), plural(len(units), "unit"), head) + return 0 +} + +// shapeOf reads what verify needs of a line, from git alone: its head, and +// every pull request merged into main since it branched, unjudged. +func shapeOf(l line) (*batch, error) { + b := &batch{line: l} + var err error + if b.lineHead, err = resolve("refs/remotes/" + remote + "/" + l.branch()); err != nil { + return nil, err + } + if b.mainHead, err = resolve("refs/remotes/" + remote + "/main"); err != nil { + return nil, err + } + out, err := git("merge-base", "--end-of-options", b.mainHead, b.lineHead) + if err != nil { + return nil, fmt.Errorf("%s shares no history with main: %w", l.branch(), err) + } + b.base = strings.TrimSpace(out) + merges, other, err := mainMerges(b.base, b.mainHead) + if err != nil { + return nil, err + } + b.other = other + for _, o := range merges { + b.verdicts = append(b.verdicts, &verdict{o: o}) + } + return b, nil +} diff --git a/cmd/backport-batch/workflows_test.go b/cmd/backport-batch/workflows_test.go new file mode 100644 index 000000000..69f402974 --- /dev/null +++ b/cmd/backport-batch/workflows_test.go @@ -0,0 +1,249 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "slices" + "strings" + "testing" +) + +// workflowWorld has #5 change a workflow and #6 change the app and a file +// under .github/ that is no workflow. +func workflowWorld(t *testing.T) *fixture { + t.Helper() + f := newFixture(t) + f.lines() + f.pr(5, "fix(ci): five", "", map[string]string{".github/workflows/five.yaml": "on: push\n", "app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n", ".github/renovate.json": "{}\n"}) + return f +} + +// With --no-workflow-files, as the scheduled batch runs, an original that +// changes a workflow is left to a maintainer, with its workflow files, and is +// neither a conflict nor deferred with a reason. Without the flag it is picked +// like any other. +func TestRunLeavesWorkflowChangesToAMaintainer(t *testing.T) { + l, _ := parseLine("release-1.1") + f := workflowWorld(t) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1", "--no-workflow-files") + if code != 0 { + t.Fatalf("run --no-workflow-files = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(app): six (#6)"}) { + t.Errorf("picked %q, want only #6", got) + } + deferred, err := parseDeferred(stdout, l) + if e := deferred[5]; err != nil || len(deferred) != 1 || !e.Workflows || e.Reason != "" || e.Commit != "" || !slices.Equal(e.Files, []string{".github/workflows/five.yaml"}) { + t.Errorf("deferred = %+v, %v; want #5 waiting with its workflow file", deferred, err) + } + for _, want := range []string{"### Waiting for a maintainer", "[#5](", "`.github/workflows/five.yaml`", "#changes-to-workflows"} { + if !strings.Contains(stdout, want) { + t.Errorf("the body lacks %q:\n%s", want, stdout) + } + } + if strings.Contains(stdout, "### Deferred") { + t.Errorf("the body lists #5 as deferred:\n%s", stdout) + } + if !strings.Contains(stderr, "waiting for a maintainer: https://github.com/cozystack/cozystack/pull/5 fix(ci): five changes .github/workflows/five.yaml") { + t.Errorf("the run does not say #5 waits:\n%s", stderr) + } + + f = workflowWorld(t) + code, stdout, stderr = f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(ci): five (#5)", "fix(app): six (#6)"}) { + t.Errorf("without the flag picked %q, want #5 and #6", got) + } + if strings.Contains(stdout, "Waiting for a maintainer") { + t.Errorf("without the flag the body has something wait:\n%s", stdout) + } +} + +// Waiting is no way off the line: check lists the original until it is on +// it, the scheduled run that continues the batch keeps it waiting rather than +// trying it as a conflict, notify lists it apart from the conflicts, and a +// maintainer's own run picks it onto the open batch. +func TestWorkflowChangesWaitUntilOnTheLine(t *testing.T) { + f := workflowWorld(t) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files"); code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/5 ") { + t.Errorf("check with no batch open = %d, %q; want #5 missing", code, stdout) + } + + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "warning: waiting on #90 for a maintainer, since it changes .github/workflows/: https://github.com/cozystack/cozystack/pull/5 ") { + t.Errorf("check = %d, %q; want 0 and a warning about #5:\n%s", code, stdout, stderr) + } + code, stdout, _ = f.exec(gh, "check", "--line", "release-1.1", "--cut") + if code != 1 || !strings.Contains(stdout, "#5 changes .github/workflows/ and waits on batch PR #90 for a maintainer") { + t.Errorf("check --cut = %d, %q; want #5 to stop the cut", code, stdout) + } + + code, stdout, stderr = f.exec(gh, "notify", "--line", "release-1.1", "--push") + if code != 0 || !strings.Contains(stdout, "1 originals on #90 change .github/workflows/ and wait for a maintainer") || strings.Contains(stdout, "deferred conflicts") { + t.Errorf("notify = %d, %q; want #5 for a maintainer and no conflict\n%s", code, stdout, stderr) + } + if len(gh.issueComments) != 1 || !strings.Contains(gh.issueComments[0].Body, "[#5](") || !strings.Contains(gh.issueComments[0].Body, "#changes-to-workflows") || strings.Contains(gh.issueComments[0].Body, "These conflicts") { + t.Errorf("the comment does not list #5 for a maintainer: %+v", gh.issueComments) + } + + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + code, stdout, stderr = f.exec(gh, "run", "--line", "release-1.1", "--continue", "--no-workflow-files", "--push") + if code != 0 { + t.Fatalf("run --continue = %d\n%s", code, stderr) + } + l, _ := parseLine("release-1.1") + if deferred, err := parseDeferred(stdout, l); err != nil || !deferred[5].Workflows || len(deferred) != 1 { + t.Errorf("the continued body has %+v, %v; want #5 still waiting", deferred, err) + } + if got := f.subjects("origin/release-1.1", f.remoteHead(batchBranch1)); !slices.Equal(got, []string{"fix(app): six (#6)", "fix(app): seven (#7)"}) { + t.Errorf("the continued batch carries %q", got) + } + + f.git("switch", "-q", "main") + reopen(f, gh) + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push"); code != 0 { + t.Fatalf("a maintainer's run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", f.remoteHead(batchBranch1)); !slices.Equal(got, []string{"fix(app): six (#6)", "fix(app): seven (#7)", "fix(ci): five (#5)"}) { + t.Errorf("after a maintainer's run the batch carries %q, want #5 on top", got) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || strings.Contains(stderr, "waiting") { + t.Errorf("check after the maintainer's run = %d, %q\n%s", code, stdout, stderr) + } +} + +// --defer keeps what waits as it is, and takes one out like any other original +// when a maintainer gives the reason. +func TestDeferKeepsWorkflowChangesWaiting(t *testing.T) { + f := workflowWorld(t) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + l, _ := parseLine("release-1.1") + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "7", "--reason", "fails the app e2e", "--push") + if code != 0 { + t.Fatalf("run --defer 7 = %d\n%s", code, stderr) + } + if deferred, err := parseDeferred(stdout, l); err != nil || !deferred[5].Workflows || deferred[7].Reason != "fails the app e2e" || len(deferred) != 2 { + t.Errorf("after --defer 7 the body has %+v, %v; want #5 waiting and #7 deferred", deferred, err) + } + + f.git("switch", "-q", "main") + reopen(f, gh) + code, stdout, stderr = f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "waits for the 1.1.4 workflow rework", "--push") + if code != 0 { + t.Fatalf("run --defer 5 = %d\n%s", code, stderr) + } + if deferred, err := parseDeferred(stdout, l); err != nil || deferred[5].Workflows || deferred[5].Reason == "" { + t.Errorf("after --defer 5 the body has %+v, %v; want #5 deferred with the reason", deferred, err) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); strings.Contains(stdout, "#5 changes") { + t.Errorf("check --cut = %d, %q; a deferred #5 still stops the cut as waiting", code, stdout) + } +} + +// A record that has an original wait and also names a failed commit or a +// reason is refused: it is one or the other. +func TestParseDeferredRefusesAMixedEntry(t *testing.T) { + l, _ := parseLine("release-1.1") + for _, entry := range []string{ + `{"original":5,"files":[],"workflows":true,"reason":"x"}`, + `{"original":5,"files":[],"workflows":true,"commit":"` + strings.Repeat("a", 40) + `"}`, + } { + body := deferredFence + "\n" + `{"line":"release-1.1","deferred":[` + entry + `]}` + "\n```\n" + if _, err := parseDeferred(body, l); err == nil { + t.Errorf("parseDeferred accepted %s", entry) + } + } +} + +// A pull request that changes a workflow in one commit and takes it back in a +// later one changes no workflow in its merge, but its first pick does, and +// the push carries that pick: it waits as well. +func TestRunHoldsAWorkflowChangeTakenBack(t *testing.T) { + f := newFixture(t) + f.lines() + f.prCommits(5, "fix(ci): five", "", + change{files: map[string]string{".github/workflows/five.yaml": "on: push\n"}}, + change{files: map[string]string{".github/workflows/five.yaml": "", "app/five.txt": "5\n"}}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1", "--no-workflow-files") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(app): six (#6)"}) { + t.Errorf("picked %q, want only #6", got) + } + l, _ := parseLine("release-1.1") + if deferred, err := parseDeferred(stdout, l); err != nil || !deferred[5].Workflows || !slices.Equal(deferred[5].Files, []string{".github/workflows/five.yaml"}) { + t.Errorf("deferred = %+v, %v; want #5 waiting on its workflow file", deferred, err) + } +} + +// A rebuild keeps the count of workflow files a waiting original changes +// beyond those the block names. +func TestDeferKeepsTheWaitingFileCount(t *testing.T) { + f := newFixture(t) + f.lines() + files := map[string]string{"app/five.txt": "5\n"} + for i := range filesKept + 2 { + files[fmt.Sprintf(".github/workflows/w%02d.yaml", i)] = "on: push\n" + } + f.pr(5, "fix(ci): five", "", files) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "7", "--reason", "fails the app e2e", "--push") + if code != 0 { + t.Fatalf("run --defer = %d\n%s", code, stderr) + } + l, _ := parseLine("release-1.1") + deferred, err := parseDeferred(stdout, l) + if e := deferred[5]; err != nil || !e.Workflows || len(e.Files) != filesKept || e.More != 2 { + t.Errorf("after the rebuild #5 is %d files and %d more, %v; want %d and 2", len(e.Files), e.More, err, filesKept) + } + if want := fmt.Sprintf("%d more", filesKept+2-filesListed); !strings.Contains(stdout, want) { + t.Errorf("the rebuilt body does not say %q:\n%s", want, stdout) + } +} From 4a750a1e23e5bb071b25b22590503e05bc8714ab Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:40:12 +0000 Subject: [PATCH 160/518] ci(release): run the backport batch twice a week The batch runs on Monday and Thursday from a script a maintainer can also run by hand. It runs as the cozystack-ci App with a token for this repository alone and Contents and Pull requests write, so its pushes start CI on the batch and no person's token is involved. The App may not push a workflow change, so such an original waits for a maintainer. pr-labeler holds every batch pull request to the batch's shape, with the tool built from main and the head read as data. It also closes a pull request into a release line opened by someone with read access or none, since fixes reach the lines through the batch. The Commit trailers check exempts a batch branch of this repository, whose picked messages cannot be rewritten. Promotion now says how to bring its own pipeline to a line through the batch. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .github/workflows/backport-batch.yaml | 107 ++++++++ .github/workflows/pr-labeler.yaml | 83 +++++- .github/workflows/pre-commit.yml | 16 +- .github/workflows/promote-rc.yaml | 29 +- hack/backport-batch-scheduled.bats | 186 +++++++++++++ hack/backport-batch-scheduled.sh | 122 +++++++++ hack/backport-batch-verify-job.bats | 59 +++++ hack/check-commit-trailers.bats | 14 + hack/close-manual-backport.bats | 367 ++++++++++++++++++++++++++ hack/close-manual-backport.sh | 156 +++++++++++ hack/pr-labeler-contract.bats | 7 +- 11 files changed, 1121 insertions(+), 25 deletions(-) create mode 100644 .github/workflows/backport-batch.yaml create mode 100644 hack/backport-batch-scheduled.bats create mode 100755 hack/backport-batch-scheduled.sh create mode 100644 hack/backport-batch-verify-job.bats create mode 100644 hack/close-manual-backport.bats create mode 100755 hack/close-manual-backport.sh diff --git a/.github/workflows/backport-batch.yaml b/.github/workflows/backport-batch.yaml new file mode 100644 index 000000000..8196c7468 --- /dev/null +++ b/.github/workflows/backport-batch.yaml @@ -0,0 +1,107 @@ +name: Backport batch + +# Builds the backport batch for each line `backport-batch lines` names, the +# newest released lines and any line in its rc window, newest line first, as +# docs/release/backports.md describes. The steps live in +# hack/backport-batch-scheduled.sh, which a maintainer can also run by hand. +# +# The batch runs as the cozystack-ci App, with a token minted for this +# repository alone with Contents and Pull requests write. It has no +# workflows permission, so an original that changes .github/workflows/ +# waits on the batch for a maintainer (--no-workflow-files). Its pushes, +# unlike GITHUB_TOKEN's, start CI on the batch pull request. +# +# Repository settings it reads; none is created here: +# secret COZYSTACK_CI_APP_ID, COZYSTACK_CI_PRIVATE_KEY +# the cozystack-ci App, shared with the +# release workflows. Required. +# variable BACKPORT_ALERT_TEAM the team or person the batch's comment +# mentions, org/team or a login. +# +# A batch open since the previous run, deferred conflicts and originals that +# change workflows wait for a maintainer: the batch's comment lists them. A +# failed batch CI waits too, and shows on the batch pull request's checks. + +on: + schedule: + # Monday and Thursday: each run leaves working days to cut an rc, run its + # e2e and promote it before the weekend. 06:23 UTC is after the nightly + # and the daily sweeps. + - cron: '23 6 * * 1,4' + workflow_dispatch: {} + +permissions: {} + +concurrency: + group: backport-batch + cancel-in-progress: false + +jobs: + batch: + name: Backport batch + if: github.repository == 'cozystack/cozystack' + runs-on: ubuntu-latest + # The App token lives an hour from its mint, a few minutes in; a run takes + # minutes, so the job ends well before the token does. + timeout-minutes: 30 + permissions: + contents: read + env: + # The pipe falls back to direct on any proxy error (hack/common-envs.mk). + GOPROXY: "https://proxy.golang.org|direct" + steps: + - name: Require the cozystack-ci App + env: + APP_ID: ${{ secrets.COZYSTACK_CI_APP_ID }} + APP_KEY: ${{ secrets.COZYSTACK_CI_PRIVATE_KEY }} + run: | + if [ -z "$APP_ID" ] || [ -z "$APP_KEY" ]; then + echo "::error::COZYSTACK_CI_APP_ID or COZYSTACK_CI_PRIVATE_KEY is not set. The batch pushes and opens pull requests as the cozystack-ci App only; see docs/release/backports.md#the-scheduled-run." + exit 1 + fi + + - name: Checkout + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + fetch-depth: 0 + persist-credentials: false + + - name: Set up Go + uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5 + with: + go-version-file: go.mod + cache: false + + - name: Build backport-batch + run: go build -o "$RUNNER_TEMP/backport-batch" ./cmd/backport-batch + + - name: Generate GitHub App token + id: app-token + uses: actions/create-github-app-token@d72941d797fd3113feb6b93fd0dec494b13a2547 # v1 + with: + app-id: ${{ secrets.COZYSTACK_CI_APP_ID }} + private-key: ${{ secrets.COZYSTACK_CI_PRIVATE_KEY }} + owner: cozystack + repositories: cozystack + permission-contents: write + permission-pull-requests: write + + # git pushes through gh, which reads the token from the environment of + # each call, so the token is never written into the checkout. + - name: Use the App's bot as git identity and credential + env: + GH_TOKEN: ${{ steps.app-token.outputs.token }} + APP_SLUG: ${{ steps.app-token.outputs.app-slug }} + run: | + gh auth setup-git + id=$(gh api "/users/${APP_SLUG}[bot]" --jq .id) + git config --global user.name "${APP_SLUG}[bot]" + git config --global user.email "${id}+${APP_SLUG}[bot]@users.noreply.github.com" + + - name: Run the batch + env: + BACKPORT_BATCH: ${{ runner.temp }}/backport-batch + BACKPORT_BATCH_TOKEN: ${{ steps.app-token.outputs.token }} + BACKPORT_BATCH_BOT: ${{ steps.app-token.outputs.app-slug }}[bot] + BACKPORT_ALERT_TEAM: ${{ vars.BACKPORT_ALERT_TEAM }} + run: hack/backport-batch-scheduled.sh --no-workflow-files diff --git a/.github/workflows/pr-labeler.yaml b/.github/workflows/pr-labeler.yaml index a241b8169..7aca218f7 100644 --- a/.github/workflows/pr-labeler.yaml +++ b/.github/workflows/pr-labeler.yaml @@ -24,7 +24,7 @@ jobs: with: script: | // Conventional Commits types accepted by Cozystack (per docs/agents/contributing.md): - // feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert + // feat, fix, security, docs, style, refactor, perf, test, build, ci, chore, revert // Mapping below maps a subset to kind/* — types not listed do not produce a kind/*. const typeToKind = { feat: 'kind/feature', @@ -32,7 +32,7 @@ jobs: docs: 'kind/documentation', chore: 'kind/cleanup', refactor: 'kind/cleanup', - // style, perf, test, build, ci, revert — no kind mapping + // security, style, perf, test, build, ci, revert — no kind mapping }; // scope -> area/* mapping. Keys are the scopes observed in cozystack issues @@ -296,3 +296,82 @@ jobs: labels: newLabels, }); core.info(`Applied labels: ${newLabels.join(', ')}`); + + # Backports are automatic (docs/release/backports.md), so a pull request + # against a release branch from outside the team is closed with a pointer to + # the policy. Only when it is opened: a maintainer keeps a fix that exists + # only on a release line by reopening it. + # + # pull_request_target hands a write token to a fork's pull request, so this + # job never touches the head: actions/checkout takes the event's own ref, + # which is the base repository, the script reads only the event JSON and the + # author's permission on the repository, and no `${{ }}` expression reaches + # the shell. The permission read needs repository metadata read access, which + # a GITHUB_TOKEN always carries and `permissions:` has no key for. + close-manual-backport: + if: >- + github.event.action == 'opened' && + startsWith(github.event.pull_request.base.ref, 'release-') + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + contents: read # check out hack/ from the base repository + pull-requests: write # close the pull request with a comment + steps: + - name: Check out the base repository + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + persist-credentials: false + sparse-checkout: hack + - name: Close a backport opened by hand from outside the team + env: + GH_TOKEN: ${{ github.token }} + run: hack/close-manual-backport.sh "$GITHUB_EVENT_PATH" + + # A batch pull request carries each original in the shape cmd/backport-batch + # gives it. fold checks one resolution, and only when whoever resolved the + # conflict runs it; this checks every original on the batch on each push. + # + # pull_request_target takes the workflow from main for a pull request into + # any branch, so the check reaches release-* lines, which do not carry it; a + # pull_request job would run the line's own workflows. The tool is built from + # main, since the lines do not carry it either. Nothing from the pull request + # is checked out or run: its head is fetched as data and read with git, and + # no `${{ }}` expression reaches the shell. Only a branch of this repository + # named like a batch is checked. + backport-batch-verify: + name: Backport batch shape + if: >- + startsWith(github.event.pull_request.base.ref, 'release-') && + startsWith(github.event.pull_request.head.ref, 'backport-batch/') && + github.event.pull_request.head.repo.full_name == github.repository + runs-on: ubuntu-latest + timeout-minutes: 15 + permissions: + contents: read + env: + # The pipe falls back to direct on any proxy error (hack/common-envs.mk). + GOPROXY: "https://proxy.golang.org|direct" + steps: + - name: Check out main + uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 + with: + ref: main + fetch-depth: 0 + persist-credentials: false + - name: Set up Go + uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5 + with: + go-version-file: go.mod + cache: false + - name: Build backport-batch from main + run: go build -o "$RUNNER_TEMP/backport-batch" ./cmd/backport-batch + - name: Fetch the batch head as data + env: + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + run: git fetch --no-tags --quiet origin "$HEAD_SHA" + - name: Verify every original on the batch + env: + BASE_REF: ${{ github.event.pull_request.base.ref }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + run: '"$RUNNER_TEMP/backport-batch" verify --line "$BASE_REF" --head "$HEAD_SHA"' diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml index dbec873fb..c2e98276d 100644 --- a/.github/workflows/pre-commit.yml +++ b/.github/workflows/pre-commit.yml @@ -89,16 +89,14 @@ jobs: runs-on: ubuntu-22.04 permissions: contents: read - # A backport PR is the one shape whose author cannot fix a finding: the - # backport action cherry-picks merged commits verbatim, so their messages + # A backport batch is the one shape whose author cannot fix a finding: + # cmd/backport-batch cherry-picks merged commits verbatim, so their messages # are history, and roughly a quarter of the commits on main predate the - # rule. Matched on the branch name the action generates, - # backport--to-, rather than on the prefix alone; the label that - # asks for a backport sits on the original PR, not on the generated one, so - # a branch name is the only signal a pull_request event carries. That makes - # the exemption a convention rather than something enforced: anyone can name - # a branch this way. Every other PR, fork included, is checked. - if: ${{ !(startsWith(github.head_ref, 'backport-') && contains(github.head_ref, '-to-')) }} + # rule. Matched on the branch name the batch uses, + # backport-batch/-, and only from a branch of this repository, + # which takes write access to push: a fork can name a branch the same way. + # Every other PR, fork included, is checked. + if: ${{ !(startsWith(github.head_ref, 'backport-batch/') && github.event.pull_request.head.repo.full_name == github.repository) }} steps: - name: Checkout head uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 diff --git a/.github/workflows/promote-rc.yaml b/.github/workflows/promote-rc.yaml index fdb57e9e3..d0567018a 100644 --- a/.github/workflows/promote-rc.yaml +++ b/.github/workflows/promote-rc.yaml @@ -327,9 +327,12 @@ jobs: // A maintenance branch created before the candidate pipeline landed // would execute its old pull_request workflows after merge. Refuse // before the first registry write unless the actual target base has - // both trusted verification gates and their helpers. Backport this - // change to an existing release-X.Y line before promoting its next - // patch; the rc-derived head does not need to carry the tooling. + // both trusted verification gates and their helpers. An existing + // release-X.Y line gets them before its next patch is promoted, + // through the backport batch with backport/include on the pull + // request that brought them to main, or through a maintainer's pull + // request against the line; the rc-derived head does not need to + // carry the tooling. // // Every file the base-checked-out verifier reaches for belongs in // the list below, its own sourced libraries included: those are @@ -363,7 +366,7 @@ jobs: }); } catch (e) { if (e.status === 404) { - core.setFailed(`Target base '${baseBranch}' lacks '${path}'. Backport the candidate-aware promotion pipeline before dispatch.`); + core.setFailed(`Target base '${baseBranch}' lacks '${path}'. Bring the candidate-aware promotion pipeline to it before dispatch: backport/include on the pull request that added it to main, or a maintainer pull request against the line.`); return; } throw e; @@ -374,7 +377,7 @@ jobs: } const content = Buffer.from(file.data.content, 'base64').toString('utf8'); if (!content.includes(marker)) { - core.setFailed(`Target base '${baseBranch}' has an old '${path}'. Backport the candidate-aware promotion pipeline before dispatch.`); + core.setFailed(`Target base '${baseBranch}' has an old '${path}'. Bring the candidate-aware promotion pipeline to it before dispatch: backport/include on the pull request that added it to main, or a maintainer pull request against the line.`); return; } } @@ -603,10 +606,11 @@ jobs: tar -xzf "$syft_dir/syft.tar.gz" -C "$syft_dir" syft rm -f "$syft_dir/syft.tar.gz" echo "$syft_dir" >> "$GITHUB_PATH" - # cosign is the version docs/release.md asks users for, so the job checks - # its own signature the way they will. The action verifies that binary - # against cosign's keyless and KMS bundles with its digest-pinned - # bootstrap cosign. Bump with tags.yaml; the contract test holds them equal. + # cosign is the version docs/release/signatures-and-sbom.md asks users + # for, so the job checks its own signature the way they will. The action + # verifies that binary against cosign's keyless and KMS bundles with its + # digest-pinned bootstrap cosign. Bump with tags.yaml; the contract test + # holds them equal. - name: Set up cosign uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 with: @@ -1044,9 +1048,10 @@ jobs: PR_STATE="$(gh pr view "${STABLE_BRANCH}" --json state --jq .state 2>/dev/null || echo "")" if [ "${PR_STATE}" = "OPEN" ]; then # Refresh only while the body is still the one this workflow wrote. - # docs/release.md tells reviewers to paste upstream breaking-change - # notes into this body before merge; a re-dispatch that overwrites it - # would destroy that with no trace and no way to recover it. + # docs/release/pre-release-checklist.md tells reviewers to paste + # upstream breaking-change notes into this body before merge; a + # re-dispatch that overwrites it would destroy that with no trace + # and no way to recover it. CURRENT="$(gh pr view "${STABLE_BRANCH}" --json body --jq .body 2>/dev/null || echo "")" if [ "${CURRENT}" = "${BODY}" ]; then echo "PR already open for ${STABLE_BRANCH} with an identical body; nothing to do." diff --git a/hack/backport-batch-scheduled.bats b/hack/backport-batch-scheduled.bats new file mode 100644 index 000000000..e844869e8 --- /dev/null +++ b/hack/backport-batch-scheduled.bats @@ -0,0 +1,186 @@ +#!/usr/bin/env bats +# Tests for hack/backport-batch-scheduled.sh, the scheduled backport batch, and +# for the workflow that runs it, .github/workflows/backport-batch.yaml. +# +# The backport-batch binary is stubbed through BACKPORT_BATCH: every call is +# recorded with the tokens gh would see, `lines` answers with $STUB_LINES, and +# a subcommand fails for the line named in $STUB_FAIL_. +# +# Run with: bats hack/backport-batch-scheduled.bats + +load test_helper + +SCRIPT="${BATS_TEST_DIRNAME}/backport-batch-scheduled.sh" +WF="${BATS_TEST_DIRNAME}/../.github/workflows/backport-batch.yaml" + +setup() { + strict_setup + export TMPDIR="$BATS_TEST_TMPDIR" + export CALLS="${BATS_TEST_TMPDIR}/calls" + export BACKPORT_BATCH="${BATS_TEST_TMPDIR}/backport-batch" + cat > "$BACKPORT_BATCH" <<'EOF' +#!/usr/bin/env bash +printf 'GH_TOKEN=%s GITHUB_TOKEN=%s %s\n' "${GH_TOKEN-unset}" "${GITHUB_TOKEN-unset}" "$*" >> "$CALLS" +line=${3:-} +case "$1" in + lines) printf '%s\n' $STUB_LINES ;; + run) [ "${STUB_FAIL_run:-}" != "$line" ] || exit 2 ;; + check) [ "${STUB_FAIL_check:-}" != "$line" ] || exit 1 ;; + notify) + [ "${STUB_FAIL_notify:-}" != "$line" ] || exit 2 + echo "$line: something waits" + ;; +esac +EOF + chmod +x "$BACKPORT_BATCH" + export STUB_LINES="release-1.7 release-1.6" + unset STUB_FAIL_run STUB_FAIL_check STUB_FAIL_notify + unset BACKPORT_ALERT_TEAM GITHUB_STEP_SUMMARY GH_TOKEN + export GITHUB_TOKEN=job-token + export BACKPORT_BATCH_TOKEN=batch-token + : > "$CALLS" +} + +# The calls recorded, without the token fields. +calls() { + sed 's/^GH_TOKEN=[^ ]* GITHUB_TOKEN=[^ ]* //' "$CALLS" +} + +@test "runs, checks and reports each line, newest first, pushing with the batch's token alone" { + run "$SCRIPT" + [ "$status" -eq 0 ] + [ "$(calls)" = "lines +run --line release-1.7 --continue --push +check --line release-1.7 +notify --line release-1.7 --push +run --line release-1.6 --continue --push +check --line release-1.6 +notify --line release-1.6 --push" ] + # Every call sees the batch's token, and the job's token is gone. + [ "$(grep -c '^GH_TOKEN=batch-token GITHUB_TOKEN=unset ' "$CALLS")" -eq 7 ] +} + +@test "a dry run pushes, opens and comments nothing, and needs no token" { + unset BACKPORT_BATCH_TOKEN + run "$SCRIPT" --dry-run + [ "$status" -eq 0 ] + if grep -q -- '--push' "$CALLS"; then + echo "a dry run passed --push:"; cat "$CALLS" + false + fi + [ "$(calls | grep -c '^run --line release-.* --continue$')" -eq 2 ] + [ "$(calls | grep -c '^notify --line release-.*[0-9]$')" -eq 2 ] +} + +@test "--no-workflow-files reaches run alone, and an unknown argument stops the script" { + run "$SCRIPT" --no-workflow-files + [ "$status" -eq 0 ] + [ "$(calls | grep -c '^run --line release-.* --continue --no-workflow-files --push$')" -eq 2 ] + if calls | grep -v '^run ' | grep -q -- '--no-workflow-files'; then + echo "the flag reached another command:"; calls + false + fi + + : > "$CALLS" + run "$SCRIPT" --dry-run --no-workflow-files + [ "$status" -eq 0 ] + [ "$(calls | grep -c '^run --line release-.* --continue --no-workflow-files$')" -eq 2 ] + + : > "$CALLS" + run "$SCRIPT" --no-workflows + [ "$status" -eq 2 ] + [ ! -s "$CALLS" ] +} + +@test "without BACKPORT_BATCH_TOKEN it stops before anything, even with the job's token" { + unset BACKPORT_BATCH_TOKEN + run "$SCRIPT" + [ "$status" -eq 2 ] + [[ "$output" == *"BACKPORT_BATCH_TOKEN is empty"* ]] + [ ! -s "$CALLS" ] +} + +@test "outside a worktree of the repository it stops before anything" { + export GIT_CEILING_DIRECTORIES="$BATS_TEST_TMPDIR" + mkdir "$BATS_TEST_TMPDIR/elsewhere" + cd "$BATS_TEST_TMPDIR/elsewhere" + run "$SCRIPT" + [ "$status" -eq 2 ] + [[ "$output" == *"not in a worktree of the repository"* ]] + [ ! -s "$CALLS" ] +} + +@test "the team is mentioned only when BACKPORT_ALERT_TEAM is set" { + export BACKPORT_ALERT_TEAM=cozystack/maintainers + run "$SCRIPT" + [ "$status" -eq 0 ] + [ "$(calls | grep -c '^notify --line release-.* --mention cozystack/maintainers --push$')" -eq 2 ] + + export BACKPORT_ALERT_TEAM= + : > "$CALLS" + run "$SCRIPT" + [ "$status" -eq 0 ] + if grep -q -- '--mention' "$CALLS"; then + echo "an empty team was mentioned:"; cat "$CALLS" + false + fi +} + +@test "a line that fails goes into the summary, the other line still runs, and the job fails" { + export GITHUB_STEP_SUMMARY="${BATS_TEST_TMPDIR}/summary" + export STUB_FAIL_run=release-1.7 STUB_FAIL_check=release-1.6 + run "$SCRIPT" + [ "$status" -eq 1 ] + [ "$(calls | grep -c '^run --line release-1.6 ')" -eq 1 ] + grep -qx -- '- release-1.7: the run failed with exit 2' "$GITHUB_STEP_SUMMARY" + grep -qx -- '- release-1.6: the check exited 1; see its output above' "$GITHUB_STEP_SUMMARY" + grep -qx -- '- release-1.7: something waits' "$GITHUB_STEP_SUMMARY" + + # Each failure fails the job on its own. + export STUB_FAIL_run=release-1.7 STUB_FAIL_check= + run "$SCRIPT" + [ "$status" -eq 1 ] + export STUB_FAIL_run= STUB_FAIL_check=release-1.6 + run "$SCRIPT" + [ "$status" -eq 1 ] + export STUB_FAIL_check= STUB_FAIL_notify=release-1.6 + run "$SCRIPT" + [ "$status" -eq 1 ] + export STUB_FAIL_notify= + run "$SCRIPT" + [ "$status" -eq 0 ] +} + +@test "the workflow runs Monday and Thursday as the cozystack-ci App and never with the job's token" { + [ "$(yq -r '.on.schedule[0].cron' "$WF")" = "23 6 * * 1,4" ] + [ "$(yq -r '.on.schedule | length' "$WF")" -eq 1 ] + [ "$(yq -o=json -I=0 '.permissions' "$WF")" = "{}" ] + [ "$(yq -o=json -I=0 '.jobs.batch.permissions' "$WF")" = '{"contents":"read"}' ] + [ "$(yq -r '.jobs.batch.steps[] | select(.uses == "actions/checkout*") | .with."persist-credentials"' "$WF")" = "false" ] + run_step='.jobs.batch.steps[] | select(.name == "Run the batch")' + [ "$(yq -r "$run_step | .run" "$WF")" = "hack/backport-batch-scheduled.sh --no-workflow-files" ] + [ "$(yq -r "$run_step | .env.BACKPORT_BATCH_TOKEN" "$WF")" = '${{ steps.app-token.outputs.token }}' ] + [ "$(yq -r "$run_step | .env.BACKPORT_BATCH_BOT" "$WF")" = '${{ steps.app-token.outputs.app-slug }}[bot]' ] + + # The App token covers this repository alone, with Contents and Pull + # requests and nothing else: no workflows, no packages. + mint='.jobs.batch.steps[] | select(.id == "app-token")' + [ "$(yq -r "$mint | .uses" "$WF")" = "actions/create-github-app-token@d72941d797fd3113feb6b93fd0dec494b13a2547" ] + [ "$(yq -r "$mint | .with.owner" "$WF")" = "cozystack" ] + [ "$(yq -r "$mint | .with.repositories" "$WF")" = "cozystack" ] + [ "$(yq -r "$mint | .with | to_entries | map(select(.key | test(\"^permission-\"))) | map(.key + \"=\" + .value) | join(\",\")" "$WF")" = "permission-contents=write,permission-pull-requests=write" ] + # A token minted after the checkout lives an hour; the job ends before. + [ "$(yq -r '.jobs.batch."timeout-minutes"' "$WF")" -le 45 ] + + # The commits carry the App's bot, read from the token, not a person. + identity='.jobs.batch.steps[] | select(.name == "Use the App'"'"'s bot as git identity and credential")' + [ "$(yq -r "$identity | .env.APP_SLUG" "$WF")" = '${{ steps.app-token.outputs.app-slug }}' ] + yq -r "$identity | .run" "$WF" | grep -qF 'gh api "/users/${APP_SLUG}[bot]" --jq .id' + yq -r "$identity | .run" "$WF" | grep -qF 'user.email "${id}+${APP_SLUG}[bot]@users.noreply.github.com"' + + # No step reaches for the job's own token or a person's batch token. + if grep -v '^[[:space:]]*#' "$WF" | grep -Eq 'secrets\.GITHUB_TOKEN|github\.token|secrets\.BACKPORT_BATCH_TOKEN|gh api user'; then + echo "the workflow reads the job's own token or a person's" + false + fi +} diff --git a/hack/backport-batch-scheduled.sh b/hack/backport-batch-scheduled.sh new file mode 100755 index 000000000..ddc7f886d --- /dev/null +++ b/hack/backport-batch-scheduled.sh @@ -0,0 +1,122 @@ +#!/usr/bin/env bash +# The scheduled backport batch (docs/release/backports.md): for each line the +# batch runs for, newest first, continue the open batch pull request or open +# one, check the line, and report what waits on the batch. This is what +# .github/workflows/backport-batch.yaml runs; a maintainer can run it by hand +# from a clean worktree of the repository. +# +# Usage: backport-batch-scheduled.sh [--dry-run] [--no-workflow-files] +# +# --dry-run builds and checks each batch in this worktree and prints what it +# would report, but pushes nothing, opens or edits no pull request and writes +# no comment. Its check sees the line without the batch it just built, so it +# can exit 1 where the real run's check passes. Without --dry-run the batch's +# token has to be in BACKPORT_BATCH_TOKEN, and it is the only token gh sees: +# the job's own GITHUB_TOKEN is never a fallback, because a push made with it +# starts no CI on the batch pull request. The workflow passes the cozystack-ci +# App's token; a maintainer running this by hand passes their own. +# +# --no-workflow-files goes to run: an original that changes .github/workflows/ +# is left on the batch pull request for a maintainer instead of picked, for a +# token that may not push such a commit, as the workflow's is. +# +# Environment: +# BACKPORT_BATCH the backport-batch binary (default: built from this tree) +# BACKPORT_BATCH_TOKEN the token the batch pushes, opens and comments with +# BACKPORT_BATCH_BOT the bot the scheduled batch writes as, [bot], +# whose batch pull request and comment are trusted; +# cozystack-ci[bot] when unset +# BACKPORT_ALERT_TEAM the team or person the batch's comment mentions, +# org/team or a login; nobody is mentioned when it is +# unset +# GITHUB_STEP_SUMMARY when set, each finding is appended to it as a line +# +# Exits 0 when every line ran and its check passed, 1 when a line's run, +# check or report failed, and 2 on a usage or setup error. +set -euo pipefail + +die() { + echo "backport-batch-scheduled: $*" >&2 + exit 2 +} + +usage="usage: backport-batch-scheduled.sh [--dry-run] [--no-workflow-files]" +dry_run=false +# An empty array expands under set -u only as ${a[@]+"${a[@]}"} on bash before +# 4.4, which is what macOS ships. +workflows=() +for arg in "$@"; do + case "$arg" in + --dry-run) dry_run=true ;; + --no-workflow-files) workflows=(--no-workflow-files) ;; + *) die "$usage" ;; + esac +done + +push=() +if ! $dry_run; then + [ -n "${BACKPORT_BATCH_TOKEN:-}" ] || + die "BACKPORT_BATCH_TOKEN is empty; the workflow sets it to the cozystack-ci App's token, a run by hand to your own (docs/release/backports.md#the-scheduled-run). The job's GITHUB_TOKEN is not used: its pushes start no CI." + export GH_TOKEN="$BACKPORT_BATCH_TOKEN" + unset GITHUB_TOKEN + push=(--push) +fi + +mention=() +if [ -n "${BACKPORT_ALERT_TEAM:-}" ]; then + mention=(--mention "$BACKPORT_ALERT_TEAM") +fi + +top=$(git rev-parse --show-toplevel) || die "not in a worktree of the repository" +cd "$top" +out=$(mktemp -d) +bin=${BACKPORT_BATCH:-} +if [ -z "$bin" ]; then + # Built before the first run switches this tree to a batch branch, whose + # line may not have the tool at all. + bin="$out/backport-batch" + go build -o "$bin" ./cmd/backport-batch +fi + +report() { + echo "$1" + if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then + printf -- '- %s\n' "$1" >> "$GITHUB_STEP_SUMMARY" + fi +} + +if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then + printf '## Backport batch\n\n' >> "$GITHUB_STEP_SUMMARY" +fi +lines=$("$bin" lines) || die "cannot list the release lines" +[ -n "$lines" ] || die "no release line to run the batch for" + +status=0 +for line in $lines; do + echo "::group::$line" + rc=0 + "$bin" run --line "$line" --continue ${workflows[@]+"${workflows[@]}"} ${push[@]+"${push[@]}"} > "$out/$line.md" || rc=$? + if [ "$rc" -ne 0 ]; then + report "$line: the run failed with exit $rc" + status=1 + elif $dry_run; then + report "$line: the batch is built in this tree, its body in $out/$line.md" + fi + rc=0 + "$bin" check --line "$line" || rc=$? + if [ "$rc" -ne 0 ]; then + report "$line: the check exited $rc; see its output above" + status=1 + fi + rc=0 + notes=$("$bin" notify --line "$line" ${mention[@]+"${mention[@]}"} ${push[@]+"${push[@]}"}) || rc=$? + while IFS= read -r note; do + [ -z "$note" ] || report "$note" + done <<< "$notes" + if [ "$rc" -ne 0 ]; then + report "$line: reporting what waits on the batch failed with exit $rc" + status=1 + fi + echo "::endgroup::" +done +exit "$status" diff --git a/hack/backport-batch-verify-job.bats b/hack/backport-batch-verify-job.bats new file mode 100644 index 000000000..5cc0c1597 --- /dev/null +++ b/hack/backport-batch-verify-job.bats @@ -0,0 +1,59 @@ +#!/usr/bin/env bats +# Contract for the backport-batch-verify job of .github/workflows/pr-labeler.yaml, +# which runs `backport-batch verify` on every push to a batch pull request. +# +# The job runs under pull_request_target, so it holds a token on a pull +# request's events and must never run code from the pull request. These tests +# pin what keeps it that way: the tool is built from main, the head is only +# fetched, the token reads contents and nothing more, and no expression +# reaches a shell. +# +# Run with: bats hack/backport-batch-verify-job.bats + +load test_helper + +WF="${BATS_TEST_DIRNAME}/../.github/workflows/pr-labeler.yaml" +JOB='.jobs.backport-batch-verify' + +@test "the job runs on pull_request_target, so main's copy reaches release lines" { + [ "$(yq -r '.on | keys | .[]' "$WF")" = "pull_request_target" ] + [ "$(yq -r "$JOB.name" "$WF")" = "Backport batch shape" ] +} + +@test "the job runs only for a batch branch of this repository into a release line" { + cond="$(yq -r "$JOB.if" "$WF")" + [[ "$cond" == *"startsWith(github.event.pull_request.base.ref, 'release-')"* ]] + [[ "$cond" == *"startsWith(github.event.pull_request.head.ref, 'backport-batch/')"* ]] + [[ "$cond" == *"github.event.pull_request.head.repo.full_name == github.repository"* ]] + [ "$(grep -o '&&' <<<"$cond" | wc -l)" -eq 2 ] +} + +@test "the token reads contents and nothing else" { + [ "$(yq -o=json -I=0 "$JOB.permissions" "$WF")" = '{"contents":"read"}' ] +} + +@test "main is checked out, without credentials, and the head only fetched" { + [ "$(yq -r "$JOB.steps[0].uses" "$WF")" = "actions/checkout@11d5960a326750d5838078e36cf38b85af677262" ] + [ "$(yq -r "$JOB.steps[0].with.ref" "$WF")" = "main" ] + [ "$(yq -r "$JOB.steps[0].with.persist-credentials" "$WF")" = "false" ] + [ "$(yq -r "[$JOB.steps[] | select(.uses == \"actions/checkout*\")] | length" "$WF")" -eq 1 ] + [ "$(yq -r "$JOB.steps[] | select(.name == \"Build backport-batch from main\") | .run" "$WF")" = 'go build -o "$RUNNER_TEMP/backport-batch" ./cmd/backport-batch' ] + [ "$(yq -r "$JOB.steps[] | select(.name == \"Fetch the batch head as data\") | .run" "$WF")" = 'git fetch --no-tags --quiet origin "$HEAD_SHA"' ] +} + +@test "the tool is built before the head is fetched, and verify gets it through env" { + names="$(yq -r "$JOB.steps[].name" "$WF")" + [ "$(grep -n 'Build backport-batch from main' <<<"$names" | cut -d: -f1)" -lt "$(grep -n 'Fetch the batch head as data' <<<"$names" | cut -d: -f1)" ] + run_line="$(yq -r "$JOB.steps[] | select(.name == \"Verify every original on the batch\") | .run" "$WF")" + [ "$run_line" = '"$RUNNER_TEMP/backport-batch" verify --line "$BASE_REF" --head "$HEAD_SHA"' ] +} + +@test "no expression reaches a shell in the job" { + runs="$(yq -r "$JOB.steps[] | select(has(\"run\")) | .run" "$WF")" + [ -n "$runs" ] + if grep -q '\${{' <<<"$runs"; then + echo "a run: line carries an expression:" + echo "$runs" + false + fi +} diff --git a/hack/check-commit-trailers.bats b/hack/check-commit-trailers.bats index 4818c0981..85603b8d8 100644 --- a/hack/check-commit-trailers.bats +++ b/hack/check-commit-trailers.bats @@ -520,3 +520,17 @@ EOF fi cleanup } + +# The workflow skips this check on backport branches, whose commits are picked +# verbatim. A fork can name a branch backport-batch/anything, so the batch's +# exemption holds only for a branch of this repository. +@test "the workflow exempts a batch branch only from this repository" { + cond="$(yq -r '.jobs.commit-trailers.if' "$REPO_ROOT/.github/workflows/pre-commit.yml")" + case "$cond" in + *"!(startsWith(github.head_ref, 'backport-batch/') && github.event.pull_request.head.repo.full_name == github.repository)"*) ;; + *) + echo "the batch exemption is not bound to this repository: $cond" >&2 + exit 1 + ;; + esac +} diff --git a/hack/close-manual-backport.bats b/hack/close-manual-backport.bats new file mode 100644 index 000000000..93127ebd6 --- /dev/null +++ b/hack/close-manual-backport.bats @@ -0,0 +1,367 @@ +#!/usr/bin/env bats +# Tests for hack/close-manual-backport.sh, which closes a pull request opened by +# hand against a release branch by someone outside the team. +# +# gh is stubbed: every call is recorded, the permission endpoint answers with +# $PERM_JSON (or fails under PERM_FAIL), and any other `gh api` with $PR_JSON. +# A case that expects the pull request to stay open asserts exactly which reads +# gh recorded, so a close that slips past a guard cannot go unseen. +# +# Run with: bats hack/close-manual-backport.bats + +load test_helper + +SCRIPT="${BATS_TEST_DIRNAME}/close-manual-backport.sh" +WF="${BATS_TEST_DIRNAME}/../.github/workflows/pr-labeler.yaml" +POLICY_URL="https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md" +PERM_CALL="api repos/cozystack/cozystack/collaborators/someone/permission" + +# The close-manual-backport job of pr-labeler.yaml, comment lines dropped so a +# setting demoted to a comment cannot satisfy an assertion. +guard_job() { + awk '/^ close-manual-backport:$/ { f = 1; next } f && /^ [^ ]/ { exit } f' "$WF" \ + | grep -v '^[[:space:]]*#' +} + +_stub_gh() { + mkdir -p "$1" + cat > "$1/gh" <<'EOF' +#!/bin/sh +printf '%s\n' "$*" >> "$GH_CALLS" +case "$1 $2" in + "api repos/"*/collaborators/*/permission) + if [ -n "${PERM_FAIL:-}" ]; then + echo 'gh: Not Found (HTTP 404)' >&2 + exit 1 + fi + cat "$PERM_JSON" + ;; + "api "*) cat "$PR_JSON" ;; +esac +EOF + chmod +x "$1/gh" +} + +setup() { + strict_setup + _stub_gh "${BATS_TEST_TMPDIR}/bin" + export PATH="${BATS_TEST_TMPDIR}/bin:$PATH" + export GH_CALLS="${BATS_TEST_TMPDIR}/calls" + export PR_JSON="${BATS_TEST_TMPDIR}/pr.json" + export PERM_JSON="${BATS_TEST_TMPDIR}/perm.json" + unset PERM_FAIL + set_permission read read + EVENT="${BATS_TEST_TMPDIR}/event.json" + # A run inside GitHub Actions has its own event here; never read it. + export GITHUB_EVENT_PATH="$EVENT" + : > "$GH_CALLS" +} + +# The permission endpoint's answer: legacy permission, then role_name. +set_permission() { + printf '{"permission":"%s","role_name":"%s","user":{"login":"someone"}}\n' "$1" "$2" > "$PERM_JSON" +} + +# Writes a pull_request_target event to $EVENT and its pull request object to +# $PR_JSON. Arguments: base ref, author association, author type, labels as a +# JSON array of names, and the author's login, someone by default. "-" leaves +# the association or the type out. +make_event() { + jq -n --arg base "$1" --arg assoc "$2" --arg type "$3" --argjson labels "$4" --arg login "${5:-someone}" ' + ({number: 42, state: "open", + base: {ref: $base, repo: {full_name: "cozystack/cozystack"}}, + user: {login: $login, type: $type}, + author_association: $assoc, + labels: ($labels | map({name: .}))} + | if $assoc == "-" then del(.author_association) else . end + | if $type == "-" then del(.user.type) else . end) as $pr + | {action: "opened", number: 42, pull_request: $pr, + repository: {full_name: "cozystack/cozystack"}}' > "$EVENT" + jq '.pull_request' "$EVENT" > "$PR_JSON" +} + +assert_left_open() { + [ "$status" -eq 0 ] || { echo "FAIL: exit $status: $output"; false; } + if [ -s "$GH_CALLS" ]; then + echo "FAIL: gh was called:" + cat "$GH_CALLS" + false + fi + case "$output" in + *"leaving it open"*) ;; + *) echo "FAIL: no reason given: $output"; false ;; + esac +} + +# Failed after the permission read, which is then the only gh call, with +# nothing closed and why said. +assert_failed_after_permission() { + [ "$status" -eq 1 ] || { echo "FAIL: exit $status: $output"; false; } + [ "$(cat "$GH_CALLS")" = "$PERM_CALL" ] || { echo "FAIL: gh calls:"; cat "$GH_CALLS"; false; } + case "$output" in + *"closing nothing"*) ;; + *) echo "FAIL: no reason given: $output"; false ;; + esac +} + +# Left open after the permission read, which is then the only gh call. +assert_left_open_after_permission() { + [ "$status" -eq 0 ] || { echo "FAIL: exit $status: $output"; false; } + [ "$(cat "$GH_CALLS")" = "$PERM_CALL" ] || { echo "FAIL: gh calls:"; cat "$GH_CALLS"; false; } + case "$output" in + *"leaving it open"*) ;; + *) echo "FAIL: no reason given: $output"; false ;; + esac +} + +@test "closes a pull request to release-1.6 from outside the team, linking the policy" { + for assoc in CONTRIBUTOR FIRST_TIME_CONTRIBUTOR FIRST_TIMER NONE; do + : > "$GH_CALLS" + make_event release-1.6 "$assoc" User '["kind/bug"]' + run "$SCRIPT" "$EVENT" + [ "$status" -eq 0 ] || { echo "FAIL: $assoc: exit $status: $output"; false; } + [ "$(wc -l < "$GH_CALLS" | tr -d ' ')" -eq 2 ] || { echo "FAIL: $assoc:"; cat "$GH_CALLS"; false; } + [ "$(sed -n 1p "$GH_CALLS")" = "$PERM_CALL" ] + sed -n 2p "$GH_CALLS" | grep -q '^pr close 42 --repo cozystack/cozystack --comment Thank you' + grep -qF "$POLICY_URL" "$GH_CALLS" + grep -qF 'a maintainer can reopen it.' "$GH_CALLS" + if grep -qF '—' "$GH_CALLS"; then echo "FAIL: the comment carries an em dash"; false; fi + done +} + +@test "leaves OWNER, MEMBER and COLLABORATOR pull requests open" { + for assoc in OWNER MEMBER COLLABORATOR; do + : > "$GH_CALLS" + make_event release-1.6 "$assoc" User '[]' + run "$SCRIPT" "$EVENT" + assert_left_open + done +} + +@test "leaves a bot's pull request open" { + make_event release-1.6 NONE Bot '[]' + run "$SCRIPT" "$EVENT" + assert_left_open +} + +# The scheduled batch opens its pull requests as the cozystack-ci App, which +# GitHub can report as a contributor once it has commits here. +@test "leaves the batch's own pull request open" { + for assoc in NONE CONTRIBUTOR; do + : > "$GH_CALLS" + make_event release-1.7 "$assoc" Bot '[]' 'cozystack-ci[bot]' + run "$SCRIPT" "$EVENT" + assert_left_open + done +} + +# No label keeps a pull request open: what keeps one is a maintainer +# reopening it, which the guard does not act on. +@test "closes a pull request whatever labels it carries" { + make_event release-1.6 CONTRIBUTOR User '["kind/bug","release-only","backport/include"]' + run "$SCRIPT" "$EVENT" + [ "$status" -eq 0 ] || { echo "FAIL: exit $status: $output"; false; } + sed -n 2p "$GH_CALLS" | grep -q '^pr close 42 --repo cozystack/cozystack --comment ' \ + || { echo "FAIL: not closed:"; cat "$GH_CALLS"; false; } +} + +@test "acts only on an opened event, so a maintainer keeps a fix by reopening it" { + for action in reopened synchronize edited labeled ''; do + : > "$GH_CALLS" + make_event release-1.6 CONTRIBUTOR User '[]' + jq --arg a "$action" 'if $a == "" then del(.action) else .action = $a end' "$EVENT" > "$EVENT.tmp" + mv "$EVENT.tmp" "$EVENT" + run "$SCRIPT" "$EVENT" + assert_left_open + case "$output" in + *"not opened"*) ;; + *) echo "FAIL: reason does not name the event ($action): $output"; false ;; + esac + done +} + +@test "leaves pull requests to main and to other branches open" { + for base in main feature-release-1.6 releases; do + : > "$GH_CALLS" + make_event "$base" CONTRIBUTOR User '[]' + run "$SCRIPT" "$EVENT" + assert_left_open + done +} + +@test "does nothing when the author association is missing" { + make_event release-1.6 - User '[]' + run "$SCRIPT" "$EVENT" + assert_left_open + case "$output" in + *"no author association"*) ;; + *) echo "FAIL: reason does not name the association: $output"; false ;; + esac +} + +@test "does nothing on an unknown association, a missing author type or an odd login" { + make_event release-1.6 SOMETHING_NEW User '[]' + run "$SCRIPT" "$EVENT" + assert_left_open + + : > "$GH_CALLS" + make_event release-1.6 CONTRIBUTOR - '[]' + run "$SCRIPT" "$EVENT" + assert_left_open + + # The login goes into the API path, so only a GitHub login shape is asked about. + : > "$GH_CALLS" + make_event release-1.6 CONTRIBUTOR User '[]' + jq '.pull_request.user.login = "someone/../../orgs/x"' "$EVENT" > "$EVENT.tmp" + mv "$EVENT.tmp" "$EVENT" + run "$SCRIPT" "$EVENT" + assert_left_open +} + +# author_association shows an organization member whose membership is private +# as a CONTRIBUTOR. `permission` alone would also pass a triage member, which +# GitHub folds into read. +@test "a private member reported as CONTRIBUTOR stays open when their access is above read" { + for answer in "write write" "admin admin" "write maintain" "read triage" "read some-custom-role"; do + : > "$GH_CALLS" + set -- $answer + set_permission "$1" "$2" + make_event release-1.6 CONTRIBUTOR User '[]' + run "$SCRIPT" "$EVENT" + assert_left_open_after_permission + case "$output" in + *"permission on cozystack/cozystack is $1 (role $2)"*) ;; + *) echo "FAIL: reason does not name the access ($answer): $output"; false ;; + esac + done +} + +@test "a failed or unreadable permission answer is an error and closes nothing" { + make_event release-1.6 CONTRIBUTOR User '[]' + + export PERM_FAIL=1 + run "$SCRIPT" "$EVENT" + assert_failed_after_permission + case "$output" in + *"cannot read someone's permission"*) ;; + *) echo "FAIL: reason does not name the failed read: $output"; false ;; + esac + unset PERM_FAIL + + for answer in 'not json' '{}' '{"permission":"read"}' '{"role_name":"read"}' '{"permission":"read","role_name":null}' '[]'; do + : > "$GH_CALLS" + printf '%s\n' "$answer" > "$PERM_JSON" + run "$SCRIPT" "$EVENT" + assert_failed_after_permission + case "$output" in + *"does not give both permission and role_name"*) ;; + *) echo "FAIL: reason does not name the unreadable answer ($answer): $output"; false ;; + esac + done +} + +@test "an author with read or no access is closed" { + for answer in "read read" "none none" "read none" "none read"; do + : > "$GH_CALLS" + set -- $answer + set_permission "$1" "$2" + make_event release-1.6 CONTRIBUTOR User '[]' + run "$SCRIPT" "$EVENT" + [ "$status" -eq 0 ] || { echo "FAIL: $answer: exit $status: $output"; false; } + [ "$(sed -n 1p "$GH_CALLS")" = "$PERM_CALL" ] || { echo "FAIL: $answer:"; cat "$GH_CALLS"; false; } + sed -n 2p "$GH_CALLS" | grep -q '^pr close 42 --repo cozystack/cozystack --comment ' \ + || { echo "FAIL: $answer: not closed:"; cat "$GH_CALLS"; false; } + done +} + +@test "--dry-run never calls a write verb" { + make_event release-1.6 CONTRIBUTOR User '[]' + + # From the event the one gh call allowed is the permission read. + run "$SCRIPT" --dry-run "$EVENT" + [ "$status" -eq 0 ] + [ "$(cat "$GH_CALLS")" = "$PERM_CALL" ] || { echo "FAIL: gh calls:"; cat "$GH_CALLS"; false; } + case "$output" in + *"would close PR #42 in cozystack/cozystack"*"$POLICY_URL"*) ;; + *) echo "FAIL: dry run did not report the close: $output"; false ;; + esac + + # By number, the pull request read and the permission read. + : > "$GH_CALLS" + run "$SCRIPT" --dry-run --pr 42 + [ "$status" -eq 0 ] + [ "$(cat "$GH_CALLS")" = "$(printf 'api repos/{owner}/{repo}/pulls/42\n%s' "$PERM_CALL")" ] \ + || { echo "FAIL: gh calls:"; cat "$GH_CALLS"; false; } + case "$output" in + *"would close PR #42"*) ;; + *) echo "FAIL: dry run did not report the close: $output"; false ;; + esac +} + +@test "--pr reads the pull request with gh and closes it" { + make_event release-1.6 CONTRIBUTOR User '[]' + run "$SCRIPT" --pr 42 + [ "$status" -eq 0 ] + [ "$(wc -l < "$GH_CALLS" | tr -d ' ')" -eq 3 ] + [ "$(sed -n 1p "$GH_CALLS")" = 'api repos/{owner}/{repo}/pulls/42' ] + [ "$(sed -n 2p "$GH_CALLS")" = "$PERM_CALL" ] + sed -n 3p "$GH_CALLS" | grep -q '^pr close 42 --repo cozystack/cozystack --comment ' +} + +@test "leaves a pull request that is already closed alone" { + make_event release-1.6 CONTRIBUTOR User '[]' + jq '.state = "closed"' "$PR_JSON" > "$PR_JSON.tmp" + mv "$PR_JSON.tmp" "$PR_JSON" + run "$SCRIPT" --pr 42 + [ "$status" -eq 0 ] + [ "$(cat "$GH_CALLS")" = 'api repos/{owner}/{repo}/pulls/42' ] + case "$output" in + *"is not open"*) ;; + *) echo "FAIL: $output"; false ;; + esac +} + +@test "an unreadable event is an error and closes nothing" { + printf 'not json' > "$EVENT" + run "$SCRIPT" "$EVENT" + [ "$status" -eq 1 ] + if [ -s "$GH_CALLS" ]; then echo "FAIL: gh was called:"; cat "$GH_CALLS"; false; fi +} + +@test "a non-numeric --pr is a usage error" { + run "$SCRIPT" --pr '42; gh pr close 1' + [ "$status" -eq 2 ] + if [ -s "$GH_CALLS" ]; then echo "FAIL: gh was called:"; cat "$GH_CALLS"; false; fi +} + +@test "pr-labeler runs the guard only on opened pull requests to release-*" { + job="$(guard_job)" + [ -n "$job" ] + printf '%s\n' "$job" | grep -qxF " github.event.action == 'opened' &&" + if printf '%s\n' "$job" | grep -qF 'reopened'; then echo "FAIL: the job runs on reopened"; false; fi + printf '%s\n' "$job" | grep -qF "startsWith(github.event.pull_request.base.ref, 'release-')" + printf '%s\n' "$job" | grep -qxF ' run: hack/close-manual-backport.sh "$GITHUB_EVENT_PATH"' +} + +# pull_request_target hands a write token to a fork's pull request. The job is +# safe only while it holds no more than it needs and never reaches the head. +@test "the guard job holds the least token and never checks out the pull request" { + job="$(guard_job)" + [ -n "$job" ] + + perms="$(printf '%s\n' "$job" | awk '/^ permissions:$/ { f = 1; next } f && /^ [^ ]/ { exit } f' \ + | sed 's/[[:space:]]*#.*$//' | sort)" + [ "$perms" = "$(printf ' contents: read\n pull-requests: write')" ] || { + echo "FAIL: permissions are:"; printf '%s\n' "$perms"; false + } + + # actions/checkout without a ref takes the event's own, which on + # pull_request_target is the base repository; a ref is the way to the head. + if printf '%s\n' "$job" | grep -qE '^[[:space:]]+ref:'; then echo "FAIL: the checkout names a ref"; false; fi + printf '%s\n' "$job" | grep -qxF ' persist-credentials: false' + + # The token is the only expression: event fields reach the script through + # the JSON file, never through the shell. + [ "$(printf '%s\n' "$job" | grep -oF '${{' | wc -l | tr -d ' ')" -eq 1 ] + printf '%s\n' "$job" | grep -qxF ' GH_TOKEN: ${{ github.token }}' +} diff --git a/hack/close-manual-backport.sh b/hack/close-manual-backport.sh new file mode 100755 index 000000000..3b1770562 --- /dev/null +++ b/hack/close-manual-backport.sh @@ -0,0 +1,156 @@ +#!/bin/sh +# Close a pull request opened by hand against a release branch by someone +# outside the team. Fixes reach the release lines through the backport batch +# (docs/release/backports.md), so a hand-made backport is work nobody needs. It +# acts only on an `opened` event: a maintainer keeps a fix that exists only on +# a release line by reopening it. +# +# Usage: close-manual-backport.sh [--dry-run] [EVENT_JSON] +# close-manual-backport.sh [--dry-run] --pr NUMBER +# +# EVENT_JSON defaults to $GITHUB_EVENT_PATH and is a pull_request or +# pull_request_target payload. --pr reads the same REST pull request object +# with `gh api`, because `gh pr view --json` has no author association. Only +# JSON is read, that object and the author's repository permission: nothing +# from the pull request is checked out or run. +# +# The pull request is closed only when the event, if there is one, is +# `opened`, the pull request is open and targets release-*, its author's +# association is not OWNER, MEMBER or COLLABORATOR, its author is not a bot, +# and GitHub answers that the author's permission and role on the repository +# are both read or none. A missing field or a value this script does not know +# leaves it open and exits 0. Exits 1, closing nothing, when the input or +# GitHub's answer on the author cannot be read, 2 on a usage error, and with +# gh's status when the close fails. +set -eu + +usage() { + echo "usage: close-manual-backport.sh [--dry-run] [EVENT_JSON | --pr NUMBER]" >&2 + exit 2 +} + +leave() { + echo "close-manual-backport: $*; leaving it open." + exit 0 +} + +dry_run=false +pr_number= +event= +while [ "$#" -gt 0 ]; do + case "$1" in + --dry-run) dry_run=true ;; + --pr) + [ "$#" -ge 2 ] || usage + pr_number=$2 + shift + ;; + -*) usage ;; + *) + [ -z "$event" ] || usage + event=$1 + ;; + esac + shift +done + +command -v jq >/dev/null 2>&1 || { echo "close-manual-backport: jq is required" >&2; exit 1; } + +if [ -n "$pr_number" ]; then + [ -z "$event" ] || usage + case "$pr_number" in '' | *[!0-9]*) usage ;; esac + if ! pr=$(gh api "repos/{owner}/{repo}/pulls/$pr_number"); then + echo "close-manual-backport: cannot read PR #$pr_number with gh" >&2 + exit 1 + fi +else + event=${event:-${GITHUB_EVENT_PATH:-}} + [ -n "$event" ] || usage + if ! pr=$(jq -c '.pull_request' "$event" 2>/dev/null); then + echo "close-manual-backport: cannot read the event JSON at $event" >&2 + exit 1 + fi + action=$(jq -r '.action | strings' "$event" 2>/dev/null || true) + [ "$action" = opened ] || leave "the event is ${action:-missing}, not opened" +fi + +# A filter that fails on an unexpected shape prints nothing, which every check +# below reads as a missing field. +json_field() { + printf '%s' "$1" | jq -r "$2" 2>/dev/null || true +} + +field() { + json_field "$pr" "$1" +} + +[ "$(field 'type')" = object ] || leave "the input carries no pull request" +number=$(field '.number | select(type == "number" and . > 0 and . == floor) | tostring') +[ -n "$number" ] || leave "the pull request carries no number" + +base=$(field '.base.ref | strings') +case "$base" in + release-*) ;; + '') leave "PR #$number carries no base ref" ;; + *) leave "PR #$number targets $base, not a release branch" ;; +esac + +state=$(field '.state | strings') +[ "$state" = open ] || leave "PR #$number is not open (state: ${state:-missing})" + +assoc=$(field '.author_association | strings') +case "$assoc" in + OWNER | MEMBER | COLLABORATOR) leave "PR #$number is from a repository $assoc" ;; + CONTRIBUTOR | FIRST_TIME_CONTRIBUTOR | FIRST_TIMER | NONE) ;; + '') leave "PR #$number carries no author association" ;; + *) leave "PR #$number has an author association this script does not know ($assoc)" ;; +esac + +user_type=$(field '.user.type | strings') +case "$user_type" in + User) ;; + Bot) leave "PR #$number is from a bot" ;; + '') leave "PR #$number carries no author type" ;; + *) leave "PR #$number has an author type this script does not know ($user_type)" ;; +esac + +repo=$(field '.base.repo.full_name | strings') +case "$repo" in + */*/* | *[!A-Za-z0-9._/-]* | /* | */) leave "PR #$number names no usable base repository" ;; + */*) ;; + *) leave "PR #$number names no usable base repository" ;; +esac +login=$(field '.user.login | strings') +case "$login" in + '' | *[!A-Za-z0-9-]*) leave "PR #$number carries no usable author login" ;; +esac + +# author_association reports an organization member whose membership is +# private as a CONTRIBUTOR, so the author's own access to the repository is the +# check that decides. `permission` folds maintain into write and triage into +# read, which is why `role_name` has to agree. An answer that cannot be read +# fails the job: left open in silence, the pull request would look decided. +if ! access=$(gh api "repos/$repo/collaborators/$login/permission"); then + echo "close-manual-backport: PR #$number: cannot read $login's permission on $repo; closing nothing" >&2 + exit 1 +fi +permission=$(json_field "$access" '.permission | strings') +role=$(json_field "$access" '.role_name | strings') +if [ -z "$permission" ] || [ -z "$role" ]; then + echo "close-manual-backport: PR #$number: GitHub's answer on $login's permission on $repo does not give both permission and role_name; closing nothing" >&2 + exit 1 +fi +case "$permission:$role" in + read:read | read:none | none:read | none:none) ;; + *) leave "PR #$number is from $login, whose permission on $repo is $permission (role $role)" ;; +esac + +policy="https://github.com/$repo/blob/main/docs/release/backports.md" +comment="Thank you for the contribution! Backports are automatic: fixes merged to \`main\` reach the release lines the batch runs for on their own, as the [backport policy]($policy) describes, so this pull request is closed. If the fix only makes sense on a release line and has no counterpart on \`main\`, a maintainer can reopen it." + +if [ "$dry_run" = true ]; then + printf 'close-manual-backport: would close PR #%s in %s with this comment:\n%s\n' "$number" "$repo" "$comment" + exit 0 +fi +gh pr close "$number" --repo "$repo" --comment "$comment" +echo "close-manual-backport: closed PR #$number in $repo." diff --git a/hack/pr-labeler-contract.bats b/hack/pr-labeler-contract.bats index 2c2daa140..8c8633f7d 100644 --- a/hack/pr-labeler-contract.bats +++ b/hack/pr-labeler-contract.bats @@ -23,9 +23,12 @@ need_node() { return 1 } -# The `script: |` block, de-indented to plain JavaScript. +# The `script: |` block, de-indented to plain JavaScript. The block ends at the +# first non-blank line indented less than its body, where the next job starts. extract_script() { - awk 'f { print substr($0, 13) } /^ +script: \|$/ { f = 1 }' "$WF" > "$1" + awk 'f && length($0) && substr($0, 1, 12) != " " { exit } + f { print substr($0, 13) } + /^ +script: \|$/ { f = 1 }' "$WF" > "$1" [ -s "$1" ] } From eaa0169ff34bfd2324a55b2a5852ac4dd65697aa Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:40:12 +0000 Subject: [PATCH 161/518] chore(release): retire the backport bot, its labels and backport-audit The batch replaces the label-driven backport workflow and the audit that gated a cut against it. The kind/backport labels, the stale exemption for them, their contract tests and the audit's make target go with them. labels.yml gains a backport/ category instead, with backport/include and backport/skip, the overrides the batch reads off a merged pull request. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .github/labels.yml | 27 +- .github/workflows/backport.yaml | 305 ------ .github/workflows/stale.yaml | 3 +- .gitignore | 2 +- Makefile | 12 +- cmd/backport-audit/README.md | 226 ---- cmd/backport-audit/main.go | 1569 ---------------------------- cmd/backport-audit/main_test.go | 1496 -------------------------- hack/label-workflows-contract.bats | 2 +- hack/release-freeze-contract.bats | 434 +------- 10 files changed, 35 insertions(+), 4041 deletions(-) delete mode 100644 .github/workflows/backport.yaml delete mode 100644 cmd/backport-audit/README.md delete mode 100644 cmd/backport-audit/main.go delete mode 100644 cmd/backport-audit/main_test.go diff --git a/.github/labels.yml b/.github/labels.yml index c71518c43..d00744897 100644 --- a/.github/labels.yml +++ b/.github/labels.yml @@ -22,6 +22,7 @@ # do-not-merge/ PR merge blockers # security/ security-finding severity and status (Cozystack-specific) # size/ PR size (auto-applied) +# backport/ maintainer overrides of the backport batch rules (Cozystack-specific) # # `aliases:` lets EndBug/label-sync rename existing labels without losing # references on already-tagged issues and PRs. @@ -78,18 +79,6 @@ color: 'e11d21' description: Indicates the change introduces a breaking API or behaviour change -# kind/backport-previous has no upstream Kubernetes equivalent (k8s has no -# two-release-line model); it stays under kind/ per the file convention. -- name: kind/backport - color: 'fbca04' - description: Categorizes issue or PR as requiring a backport to the current release line - aliases: ['backport'] - -- name: kind/backport-previous - color: 'fbd876' - description: Categorizes issue or PR as requiring a backport to the previous release line - aliases: ['backport-previous'] - # ────────────────────────────────────────────── # priority/ — urgency # ────────────────────────────────────────────── @@ -411,3 +400,17 @@ - name: security/fixed color: '0e8a16' description: Fix released + +# ────────────────────────────────────────────── +# backport/ — maintainer overrides of the backport batch rules +# ────────────────────────────────────────────── + +# Read by cmd/backport-batch from the merged pull request on main; +# docs/release/backports.md describes both. +- name: backport/include + color: 'c2e0c6' + description: The backport batch takes this PR though rules 2-4 keep it out; it must still pick and pass CI + +- name: backport/skip + color: 'f9d0c4' + description: The backport batch keeps this PR out although the rules take it diff --git a/.github/workflows/backport.yaml b/.github/workflows/backport.yaml deleted file mode 100644 index a2b9b75a9..000000000 --- a/.github/workflows/backport.yaml +++ /dev/null @@ -1,305 +0,0 @@ -name: Automatic Backport - -on: - pull_request_target: - types: [closed, labeled] # fires when PR is closed (merged) or labeled - -# Two mechanisms here, answering two different failures. -# -# A run joins its concurrency group before any job-level `if` is evaluated, so -# `prepare`'s guard below cannot stop a label event from disturbing the run -# already backporting this PR; it only decides what the label run does after the -# damage. Labels set with the default GITHUB_TOKEN start no run, but labels set -# by third-party GitHub Apps do, and they arrive in bursts. Under a plain -# `cancel-in-progress: true` each one cancelled the backport the merge had just -# started, and nothing reports that beyond a cancelled run sitting next to a -# green one. -# -# The delivery survived anyway, because the old guard admitted a run on the PR's -# cumulative labels: the last label of a burst requalified and redid the work. -# `prepare`'s narrowing below removes that accidental safety net, so the two -# changes land together — narrowing alone would turn a noisy delivery into an -# absent one. -# -# So the group key moves a label event that is NOT a backport request aside, and -# `cancel-in-progress` stops one that IS from cancelling rather than queuing. -# Both are load-bearing. `backport` and `backport-previous` are separate requests -# that both have to complete, so they share the main group and must not cancel -# each other; every other label does no work here and only has to stay out of -# the way. -# -# That is also why this file does not simply copy pull-requests.yaml, which -# splits the key and leaves cancellation unconditional: a discarded label run -# there publishes nothing, so isolating it is the whole fix. -# -# Queuing protects a pending run from the run ahead of it and never from the one -# behind: the group holds a single pending run, and a newer one evicts it. With -# the split in place that costs nothing, and for the same reason the fan-out -# below is harmless — the only events left in this group are backport requests, -# the evictor is by definition the newer of the two, and `prepare`'s `Check which -# labels are present` step reads the cumulative label set. So the survivor covers -# a superset of what the evicted run would have delivered, for any sequence that -# only adds labels. Removing one in between is the exception and needs no cover: -# the evictor then carries less because the request was withdrawn. -# -# That is exactly what the split buys. Without it the evictor could be any label -# at all, and a run that does not qualify delivers nothing in place of the -# request it displaced. -# -# `queue: max` raises the single pending slot to a hundred and is the obvious -# thing to reach for instead of the split. Taking it means giving up the -# conditional cancel, because `queue: max` and `cancel-in-progress: true` are a -# validation error together and this key still evaluates to true on a merge. -# The trade is available and was not taken: eviction is what keeps a burst to one -# surviving run rather than a hundred queued ones, and the feature only shipped -# 2026-05-07, on a path that delivers fixes to release lines. -concurrency: - group: backport-${{ github.workflow }}-${{ github.event.pull_request.number }}${{ (github.event.action == 'labeled' && github.event.label.name != 'backport' && github.event.label.name != 'backport-previous' && github.event.label.name != 'kind/backport' && github.event.label.name != 'kind/backport-previous') && '-label' || '' }} - # Stated positively, matching the group key above. `!= 'labeled'` is equivalent - # only while `types:` holds exactly these two, and that form fails open: adding - # `unlabeled` or `reopened` there would hand those events cancellation in the - # main group, which is the behaviour this file exists to remove. - # - # The positive form removes the cancelling, not the occupying. A third type - # still lands in the main group, still takes its single pending slot, and its - # `prepare` still skips — so it can evict a genuine backport request and - # deliver nothing in its place. `types:` is load-bearing for that reason and is - # pinned in hack/release-freeze-contract.bats; widening it needs the group key - # widened with it. - cancel-in-progress: ${{ github.event.action == 'closed' }} - -# Top-level token defaults to read-only. A job-level block replaces this one -# rather than adding to it, so each job restates every scope it needs. -permissions: - contents: read - -jobs: - # Determine which backports are needed - prepare: - # This guard admits each action on the labels it is entitled to. `closed` - # carries no label of its own, so it reads the set the PR ended up with. - # `labeled` reads only the label just added: matching the cumulative set - # here means every later unrelated label on a merged PR that still carries - # `backport` re-enters this job to redo a backport already delivered. - # - # Scope, so this is not read as covering the whole job: the `Check which - # labels are present` step below still reads the cumulative set, so a - # `backport-previous` event on a PR also carrying `backport` fans out to - # both matrix legs. That is deliberate, and each state a re-entered leg can - # find is already accounted for: delivered is skipped by the merged-backport - # guard, still open is skipped by backport-action on the 422, and closed - # unmerged gets a fresh backport PR, which is the retry that guard - # deliberately preserves rather than an accident. What the narrowing here - # decides is whether the job RUNS, not which targets it picks. - if: | - github.event.pull_request.merged == true && - github.event.pull_request.base.ref == 'main' && - ( - (github.event.action == 'closed' && (contains(github.event.pull_request.labels.*.name, 'backport') || contains(github.event.pull_request.labels.*.name, 'backport-previous') || contains(github.event.pull_request.labels.*.name, 'kind/backport') || contains(github.event.pull_request.labels.*.name, 'kind/backport-previous'))) || - (github.event.action == 'labeled' && (github.event.label.name == 'backport' || github.event.label.name == 'backport-previous' || github.event.label.name == 'kind/backport' || github.event.label.name == 'kind/backport-previous')) - ) - # GitHub API calls only — no docker, no repo toolchain: GitHub-hosted. - runs-on: ubuntu-latest - # Queuing gives a stuck run a way to hurt the next one that a cancelling key - # did not: the label event that used to kill it now waits behind it rather - # than clearing it. - # - # This ceiling bounds a stuck JOB, which is the case it can reach. A run can - # also wedge at the RUN level with every job already finished, and no - # job-level key expires that one; it holds the group until GitHub retires the - # run. Rare but real: run 31232645133 has been `in_progress` since - # 2026-08-08 with `prepare` and `backport` both skipped. Nothing here - # reaches that state, so docs/release.md tells an operator to look for a run - # still in flight and clear it before re-applying the label. - # - # What it bounds is EXECUTION, not the wait for a runner. GitHub's limits - # page says "job execution time" for the 6-hour job cap while spelling out - # that the 35-day run cap "includes execution duration, and time spent on - # waiting and approval" — the contrast is deliberate — and - # orgs/community#50926 shows a job sitting six hours on "Waiting for a runner - # to pick up this job" under `timeout-minutes: 5`. So queue time is a third - # unbounded case alongside the run-level wedge, not something this covers. - # - # 30 is therefore generous rather than calculated. Execution on run - # 29844315422 was 6s for `prepare` and 2s and 16s for the two matrix legs, - # against 9m24s, 5m28s and 5m45s of queue for the same three jobs. Every - # execution measured on this workflow has been seconds, and no maximum is - # quoted on purpose: an absolute maximum over a growing set is stale as soon - # as a slower run lands. The value leaves room for a cherry-pick far larger - # than anything seen while replacing the 6-hour default, and nothing here is - # sized against the queue because nothing here can be. - timeout-minutes: 30 - permissions: - contents: read - outputs: - backport_current: ${{ steps.labels.outputs.backport }} - backport_previous: ${{ steps.labels.outputs.backport_previous }} - current_branch: ${{ steps.branches.outputs.current_branch }} - previous_branch: ${{ steps.branches.outputs.previous_branch }} - steps: - - name: Check which labels are present - id: labels - uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7 - with: - script: | - const pr = context.payload.pull_request; - const labels = pr.labels.map(l => l.name); - // TRANSITIONAL: both spellings are accepted so the rename can land - // in any order relative to the org-level dosubot, which still - // applies the legacy names and is today the dominant applier on - // real backports. Once dosubot's PR labelling is switched off and - // the in-repo labeler is the only applier, the legacy arms here, - // in the `if:` gate above and in the concurrency group can all be - // deleted in one pass. - const isBackport = labels.includes('kind/backport') || labels.includes('backport'); - const isBackportPrevious = labels.includes('kind/backport-previous') || labels.includes('backport-previous'); - - core.setOutput('backport', isBackport ? 'true' : 'false'); - core.setOutput('backport_previous', isBackportPrevious ? 'true' : 'false'); - - console.log(`backport label: ${isBackport}, backport-previous label: ${isBackportPrevious}`); - - - name: Determine target branches - id: branches - uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7 - with: - script: | - // Enumerate the real release-X.Y branches rather than deriving them - // from the latest published release. - // - // Since cutting an rc freezes the line (cut-prerelease.yaml), - // release-X.Y is created when the first vX.Y.0-rc.N is cut — before - // vX.Y.0 is ever published. getLatestRelease returns the newest - // published STABLE, so throughout the freeze window it still named - // the previous line: a `backport` on a fix for the release being - // stabilised would have landed on release-X.(Y-1) and missed the - // release it was written for. That window is exactly when backports - // matter most, since the frozen branch is the only way in. - // - // Enumerating branches also drops the contiguous-minor assumption: - // `previous` used to be computed as min-1, which named a - // non-existent branch whenever a minor was skipped. - const branches = await github.paginate(github.rest.repos.listBranches, { - owner: context.repo.owner, - repo: context.repo.repo, - per_page: 100 - }); - - const lines = branches - .map(b => /^release-(\d+)\.(\d+)$/.exec(b.name)) - .filter(Boolean) - .map(m => ({ name: m[0], maj: Number(m[1]), min: Number(m[2]) })) - .sort((a, b) => (b.maj - a.maj) || (b.min - a.min)); - - if (lines.length === 0) { - core.setFailed('No release-X.Y branches found; cannot determine backport target.'); - return; - } - - const currentBranch = lines[0].name; - const previousBranch = lines.length > 1 ? lines[1].name : ''; - - core.setOutput('current_branch', currentBranch); - core.setOutput('previous_branch', previousBranch); - - console.log(`Release lines found: ${lines.map(l => l.name).join(', ')}`); - console.log(`Current branch: ${currentBranch}, Previous branch: ${previousBranch || 'N/A'}`); - - // Both targets come from the enumeration above, so they exist by - // construction — no getBranch probe needed. What CAN still fail is - // asking for a previous line when only one exists. - if (!previousBranch && '${{ steps.labels.outputs.backport_previous }}' === 'true') { - core.setFailed('backport-previous requested, but only one release line exists.'); - return; - } - - backport: - needs: prepare - if: | - github.event.pull_request.merged == true && - github.event.pull_request.base.ref == 'main' && - (needs.prepare.outputs.backport_current == 'true' || needs.prepare.outputs.backport_previous == 'true') - # git cherry-picks via backport-action — no docker, no repo toolchain: GitHub-hosted. - runs-on: ubuntu-latest - # See `prepare` above: a queued request waits on this one, so it needs a - # ceiling rather than the 6-hour default. Bounds this job's execution only; - # the wait for a runner is outside it either way. - timeout-minutes: 30 - permissions: - contents: write - pull-requests: write - strategy: - matrix: - backport_type: [current, previous] - steps: - # 1. Determine target branch based on matrix - - name: Set target branch - id: target - if: | - (matrix.backport_type == 'current' && needs.prepare.outputs.backport_current == 'true') || - (matrix.backport_type == 'previous' && needs.prepare.outputs.backport_previous == 'true') - run: | - if [ "${{ matrix.backport_type }}" == "current" ]; then - echo "branch=${{ needs.prepare.outputs.current_branch }}" >> $GITHUB_OUTPUT - echo "Target branch: ${{ needs.prepare.outputs.current_branch }}" - else - echo "branch=${{ needs.prepare.outputs.previous_branch }}" >> $GITHUB_OUTPUT - echo "Target branch: ${{ needs.prepare.outputs.previous_branch }}" - fi - - # 2. Skip if this target already has a merged backport. - # - # The action's backport branch name is deterministic, - # backport--to-. A merged PR with that head - # means an earlier run already delivered this change to the target — a - # second label event (e.g. backport-previous added after backport - # already merged) would re-enter this job and cherry-pick commits the - # target branch already has. Git reports that as an empty pick (exit 1) - # rather than a conflict, which draft_commit_conflicts below misreads - # as a real conflict and reports as a failed backport on the original - # PR. Skipping here avoids the false report; it does not touch the - # empty-pick behavior itself, which lives in the action. - - name: Check for existing merged backport - id: guard - if: steps.target.outcome == 'success' - uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7 - with: - script: | - const targetBranch = '${{ steps.target.outputs.branch }}'; - const head = `backport-${context.payload.pull_request.number}-to-${targetBranch}`; - const { data: prs } = await github.rest.pulls.list({ - owner: context.repo.owner, - repo: context.repo.repo, - state: 'closed', - head: `${context.repo.owner}:${head}`, - }); - const alreadyMerged = prs.some(p => p.merged_at !== null); - core.setOutput('already_merged', alreadyMerged ? 'true' : 'false'); - if (alreadyMerged) { - console.log(`Skipping backport to ${targetBranch}: ${head} already merged.`); - } - - # 3. Checkout (required by backport‑action) - - name: Checkout repository - if: steps.target.outcome == 'success' && steps.guard.outputs.already_merged != 'true' - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 - - # 4. Create the back‑port pull request - - name: Create back‑port PR - id: backport - if: steps.target.outcome == 'success' && steps.guard.outputs.already_merged != 'true' - uses: korthout/backport-action@0193454f0c5947491d348f33a275c119f30eb736 # v3.2.1 - with: - github_token: ${{ secrets.GITHUB_TOKEN }} - label_pattern: '' # don't read labels for targets - target_branches: ${{ steps.target.outputs.branch }} - merge_commits: skip - # conflict_resolution lives inside the experimental JSON input, not as a - # top-level key. Passed top-level it is silently ignored and the action - # falls back to the experimental default ("fail"), which on a conflict - # creates no PR and only comments. Nesting it here makes a conflicting - # backport open a draft PR with the conflict committed for manual fixup. - experimental: | - { - "conflict_resolution": "draft_commit_conflicts" - } diff --git a/.github/workflows/stale.yaml b/.github/workflows/stale.yaml index 96e495abc..d8b40aadc 100644 --- a/.github/workflows/stale.yaml +++ b/.github/workflows/stale.yaml @@ -39,8 +39,7 @@ jobs: security/triage-needed,security/in-progress, triage/accepted,kind/regression exempt-pr-labels: >- - lifecycle/frozen,do-not-merge/hold, - kind/backport,kind/backport-previous + lifecycle/frozen,do-not-merge/hold days-before-stale: 60 days-before-close: 14 operations-per-run: 100 diff --git a/.gitignore b/.gitignore index db787248d..e265b9fd4 100644 --- a/.gitignore +++ b/.gitignore @@ -11,7 +11,7 @@ _repos # root-level binary waiting to happen — each ~80 MB, and nothing else here # matched them, leaving `git add .` one keystroke from committing one. /api-gate -/backport-audit +/backport-batch /backup-controller /backupstrategy-controller /check-readiness diff --git a/Makefile b/Makefile index 6da0e1037..5a553db08 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: manifests assets prepare-env prepare-env-container unit-tests helm-unit-tests bats-unit-tests bats-unit-files-check bats-posix-compat-tests print-bats-unit-files print-bats-posix-compat-files print-bats-jobs rd-presets-check migrations-target-check test test-controllers test-controllers-envtest test-backport-audit preflight +.PHONY: manifests assets prepare-env prepare-env-container unit-tests helm-unit-tests bats-unit-tests bats-unit-files-check bats-posix-compat-tests print-bats-unit-files print-bats-posix-compat-files print-bats-jobs rd-presets-check migrations-target-check test test-controllers test-controllers-envtest test-backport-batch preflight include hack/common-envs.mk @@ -119,7 +119,7 @@ test: make -C packages/core/testing apply make -C packages/core/testing e2e -unit-tests: helm-unit-tests bats-unit-tests bats-posix-compat-tests go-unit-tests go-module-tests rd-presets-check test-check-readiness test-backport-audit migrations-target-check +unit-tests: helm-unit-tests bats-unit-tests bats-posix-compat-tests go-unit-tests go-module-tests rd-presets-check test-check-readiness test-backport-batch migrations-target-check helm-unit-tests: hack/helm-unit-tests.sh @@ -194,9 +194,9 @@ test-controllers-envtest: $(SETUP_ENVTEST) test-check-readiness: go test ./test/check-readiness/ -count=1 -# ./cmd/... is excluded from go-unit-tests, so the audit needs its own target. -test-backport-audit: - go test ./cmd/backport-audit/ -count=1 +# ./cmd/... is excluded from go-unit-tests, so the batch tool needs its own target. +test-backport-batch: + go test ./cmd/backport-batch/... -count=1 # Unit discovery is one level deep. Live-cluster files need an invocation in # packages/core/testing; hack/bats-runner-coverage.bats checks that boundary. @@ -294,7 +294,7 @@ bats-posix-compat-tests: # own too; hack/unit-lane-no-cluster.bats fails when one is missing. NO_CLUSTER_TARGETS := unit-tests test-controllers helm-unit-tests bats-unit-tests \ bats-posix-compat-tests go-unit-tests go-module-tests rd-presets-check \ - test-check-readiness test-backport-audit migrations-target-check + test-check-readiness test-backport-batch migrations-target-check $(NO_CLUSTER_TARGETS): export KUBECONFIG := /dev/null $(NO_CLUSTER_TARGETS): export KUBERNETES_SERVICE_HOST := diff --git a/cmd/backport-audit/README.md b/cmd/backport-audit/README.md deleted file mode 100644 index da446caba..000000000 --- a/cmd/backport-audit/README.md +++ /dev/null @@ -1,226 +0,0 @@ -# backport-audit - -Answers one question, before a release or an rc is cut: did everything labelled for backport actually land on the release branch? It prints the URLs of the ones that did not, and exits non-zero, so it can gate the cut. - -The manual alternative is `gh pr list --search "is:merged label:kind/backport"`, which lists every PR ever labelled and says nothing about whether the change reached the branch. Answering that by hand means reading the branch history per PR, which is where backports quietly go missing. - -```console -$ go run ./cmd/backport-audit release-1.5 - -=== release-1.5 === 43 candidate PRs: MISSING=2 pending=3 dropped=5 backported=33 - - MISSING -- labelled for this line, no trace of it here (2): - https://github.com/cozystack/cozystack/pull/3449 - #3449 ci(release): move release validation to rc time — e2e lane, changelog button, promote gate, docs at promote - label=kind/backport-previous author=myasnikovdaniil merged=2026-07-29 -- no backport PR, nothing on branch -``` - -The URL is always the **original PR on main** — the thing you decide about. The evidence line names the backport PR when one exists. A labelled PR already on the branch is counted in the header and never listed, so silence under the outstanding headings means done. Two more sections, described in [Backports the labels do not account for](#backports-the-labels-do-not-account-for), start from the backport PRs on the branch instead of from labels: `DUPLICATE`, which is outstanding, and `UNLABELLED`, which is informational. - -## Usage - -```bash -go run ./cmd/backport-audit # the two newest release lines -go run ./cmd/backport-audit release-1.5 # one line -go run ./cmd/backport-audit release-1.6 release-1.5 --json -``` - -Requires `git` and an authenticated `gh` on PATH. The audit does not modify GitHub or working-tree files. Unless `--no-fetch` is set, it updates local Git refs and objects while fetching. - -| Flag | Meaning | -|------|---------| -| `--remote NAME` | Git remote holding the release branches (default `origin`) | -| `--limit N` | Max merged PRs scanned per label spelling (default 1000, which is also the ceiling — see [Exit code](#exit-code)) | -| `--no-fetch` | Skip `git fetch`; both main and release refs must be current so shared history stays out of backport evidence | -| `--json` | Machine-readable output | -| `--no-color` | Disable color (auto-disabled on a non-TTY and under `--json`) | - -Arguments and flags may be given in any order. - -## Exit code - -`0` when nothing is outstanding, `1` when something is, `2` when the audit could not be completed. Outstanding means a `MISSING`, `partial`, `unverified`, `pending` or `dropped` verdict, or any `DUPLICATE` entry; `UNLABELLED` entries never move the exit code. That is the point of the tool, so it holds in `--json` mode too: - -```bash -go build ./cmd/backport-audit -./backport-audit release-1.5 -case $? in - 0) echo "clean" ;; - 1) echo "do not cut yet" ;; - *) echo "the audit itself failed; no answer yet" ;; -esac -``` - -Use the built binary (`go build` as above, or `go install ./cmd/backport-audit`) wherever the exit code matters. `go run` does not preserve it: any non-zero exit of the program comes back as `go run`'s own `1`, after an `exit status N` line on stderr, so under `go run` an audit that could not be completed looks exactly like one that found outstanding work. `go run` is fine for reading the report. - -Exit `2` covers the cases where an answer cannot be trusted rather than merely being bad news, and a saturated listing is one of them: `gh` truncates at `--limit` in silence, and a truncated list does not make the audit partial, it makes it wrong — the PRs past the cut are reported nowhere and the exit code says clean. Each of the three listings is checked against its cap and fails instead, naming the cap to raise. The label listings go through GitHub search, which never returns more than 1000 results however far it is paginated, so they are checked against the lower of `--limit` and 1000: a `--limit` above 1000 cannot make a listing cut at 1000 pass for complete. A git read that a verdict depends on is the other case: when one fails, a merge commit missing from the local repository for instance, the audit stops with `2` instead of judging the PR without it. - -## Statuses - -| Status | Meaning | Action | -|--------|---------|--------| -| `in-branch` | The PR merged before the branch was cut, so its change is already there | none, no backport was ever needed | -| `backported` | Every commit of the PR is on the branch, or, for a PR of a single commit, a backport PR merged | none | -| `confirmed` | Would be `partial` or `unverified`, but a maintainer attested the backport complete on the merged backport PR; the report names who and links the comment | none | -| `partial` | Some of the PR's commits are on the branch and some are not, listed with SHA and subject — a merged bot conflict draft, which stops at the first commit that did not apply, or a backport squashed by hand | backport the missing commits, or [attest the backport complete](#attesting-a-backport-complete) | -| `unverified` | A backport PR merged, but none of the PR's several commits is on the branch under its own SHA, `-x` reference or subject — typically a backport squashed or reworded by hand | check it, then attest it complete or backport what is missing | -| `pending` | A backport PR exists and is still open, including the drafts the bot opens with the conflict committed | merge it, or finish the draft | -| `dropped` | A backport PR was closed unmerged, reported with whatever reason someone left on it | none if the reason still holds | -| `MISSING` | No backport PR ever existed and nothing on the branch matches | cherry-pick it by hand, or drop it deliberately | - -`dropped` is usually healthy — a maintainer deciding a fix does not apply to that line, e.g. because the feature it repairs never shipped there. The recorded reason is printed so the next release does not re-open the same investigation. A `dropped` entry reading `no reason recorded` is the one that needs a human. - -## Backports the labels do not account for - -The verdicts start from labels, so on their own they cannot see a backport of a PR nobody labelled. Two further sections start from the other side: every PR on the release branch, and the originals it names by the bot's head branch or in a `Backport of` phrase (see [How landing is established](#how-landing-is-established)). The header carries their counts after a `|`, e.g. `backported=42 | DUPLICATE=3 unlabelled=13`. - -`UNLABELLED` lists each original that backport PRs on the branch claim although it is not a candidate for the branch, with every backport PR claiming it and that PR's state. Each entry also says what those PRs amount to: `backport PR merged` once one of them merged, `claimed here, not merged` while the only live one is still open, and `claimed here, closed unmerged` when every one was closed. A merged claim is only a merged PR, since an unlabelled original is not checked commit by commit; it reads `backport PR merged, confirmed by @login` when the PR carries the [attestation marker](#attesting-a-backport-complete). Most are hand backports of PRs that never carried a label, merged on purpose. An original that does carry a backport request, one that resolved to a different line at merge time, is listed with its labels. The section is informational and never moves the exit code: the gate answers whether everything labelled landed, and an unlabelled backport can only add to a branch, never leave a labelled change off it. - -`DUPLICATE` lists each original claimed by more than one live backport PR, labelled or not: - -| Kind | Meaning | Action | -|------|---------|--------| -| `open-twice` | Two or more backport PRs are open for it — typically the bot's conflict draft next to a hand backport, or a fork PR next to its reopening from a branch in this repository | merge one, close the rest | -| `open-after-merge` | A backport already merged and another one is still open | close the leftover, or merge it if it is the second half of a split backport | - -Both kinds make the audit exit `1`. Neither reaches a verdict, which judges the commits on the branch and names at most one open backport PR, and in both a human has to decide what happens to the open PR before the cut. A closed PR next to an open one is the normal shape of a redone backport and is not flagged, and neither are two merged ones, which are history the branch already carries. - -```console - DUPLICATE -- one original, more than one live backport PR (3): - https://github.com/cozystack/cozystack/pull/3936 - #3936 fix(rabbitmq): right-size the default resources preset to s1.nano - label=kind/backport -- a backport PR still open after another one merged - backport #4372 OPEN (draft) https://github.com/cozystack/cozystack/pull/4372 - backport #4393 MERGED https://github.com/cozystack/cozystack/pull/4393 - - UNLABELLED -- backport PRs here for originals not labelled for this line (13, informational): - https://github.com/cozystack/cozystack/pull/4280 - #4280 fix(kafka): keep the pre-delete hook's credentials as long as its Job - backport PR merged - backport #4421 CLOSED https://github.com/cozystack/cozystack/pull/4421 - backport #4456 MERGED https://github.com/cozystack/cozystack/pull/4456 -``` - -The titles of unlabelled originals are not in any listing the audit already makes, so they come from one GraphQL request covering the whole run. If it fails, the report goes out without them and says so on stderr; the URLs are derived locally and the verdicts and exit code do not depend on it. - -## How a PR is matched to a release line - -A `kind/backport` label does not name a branch. The audit selects merged PRs that currently carry a backport label, then assigns each to the release line selected at its merge time: `kind/backport` means the current line, and `kind/backport-previous` means the previous line. A label added after merge still makes the PR a candidate. [`backport.yaml`](../../.github/workflows/backport.yaml) also runs when a label is added and selects its target at that time, so a late label can make the audit and the bot disagree; see [Limits](#limits). - -What "the current line" means changed once, so the audit carries both rules and picks by the PR's merge date: - -| PR merged | rule `backport.yaml` was running | a line exists from | -|-----------|----------------------------------|--------------------| -| before 2026-08-03 | `getLatestRelease` | its first non-prerelease release publishing | -| on or after 2026-08-03 | the two newest existing `release-X.Y` branches | its `release-X.Y` branch being pushed | - -The cutover is a single moment because both halves landed in one push: 53a7fc4dc made cutting an rc freeze the line into `release-X.Y`, and 6ff5b98d5 pointed the bot at the branch list. Before that a release branch was created at promote time, so the newest branch and the newest published stable named the same line and the two rules could not disagree. After it they disagree for the length of every freeze window — `release-X.Y` is created when the first `vX.Y.0-rc.N` is cut, potentially weeks before `vX.Y.0` publishes, and everything labelled in between belongs to the line being stabilised. That window is exactly when a backport has to reach the branch, because the frozen branch is the only way into the release. - -The audit estimates branch opening from the earliest `vX.Y.0-rc.N` tagged commit dated on or after the cutover. It ignores earlier tags individually; a line with no remaining tag falls back to its first stable release date. [`cut-prerelease.yaml`](../../.github/workflows/cut-prerelease.yaml) creates the branch at the tagged commit, but the commit timestamp is not the branch-push timestamp. PRs merged in that interval can be assigned to the new line before the bot could target it. The rc release's `publishedAt` is later still, after the build finishes. - -Lines are then ranked by version and never by when they opened, matching the numeric descending sort the workflow applies to its branch list. The two orders come apart as soon as a freeze overlaps the previous line's stabilisation, and version order is also what keeps `release-1.10` above `release-1.9`. - -The labels themselves were namespaced under `kind/` in 7b71053a0, which renamed them in place via label-sync aliases, so every historical PR reads back under the new name. The bare `backport` and `backport-previous` are still queried alongside them, because the org-level dosubot goes on applying the old spelling to new PRs — the same transitional allowance [`backport.yaml`](../../.github/workflows/backport.yaml) makes, and it retires at the same time. A PR found under either spelling is reported under the `kind/` one. - -## How landing is established - -Evidence is weighed per commit, because a commit is what a backport can lose: - -1. **Reachability.** The PR's merge commit is reachable from the release branch, i.e. it merged before the branch was cut (or `main` was later merged in). Nothing was ever needed: `in-branch`. -2. **A linked backport PR.** Found by the bot's head branch `backport--to-release-X.Y`, or by a `Backport of #N` reference in the body, which is what a hand-written backport carries. It never outranks the commits (item 3): a merged backport PR with one of the PR's commits missing is `partial`, because the bot's conflict drafts stop at the first commit that does not apply and drop the rest, so a merged draft looks finished at PR level. `MERGED` settles it alone only for a PR of a single commit — including on a `--no-fetch` run whose local refs predate the backport's merge. For a PR of several commits, none of them found and a merged backport PR is `unverified`. `OPEN` is `pending`; `CLOSED` is `dropped`. - - A backport carrying several changes links to every original its `Backport of` phrase names: a list (`Backport of #3938 and #4280`, `Backport of #1, #2, and #3`), and the `Backport of #4253 to release-1.6, together with #3460` form used for a dependency pulled along. Only that phrase is read, so the issue a backport fixes or a CI run it cites further on is never taken for an original. A list counts in full only when it visibly ends — at the end of the line or the sentence, or where the `to release-X.Y` clause starts. One that runs on into anything else (`Backport of #10, #20 is not included`) may be saying something about its later items, so only its first reference counts. A reference qualified with this repository (`cozystack/cozystack#N`, any case) counts like a bare `#N`, and one qualified with any other repository is skipped, because reading `other/repo#20` as local #20 would let an unrelated backport vouch for whichever local PR carries that number. -3. **The branch's own history, commit by commit.** Every commit the PR contributed has to be on the branch; merge commits and commits that change nothing are skipped, since the backport bot drops both. A commit counts when it is reachable, when an `-x` cherry-pick reference names it, or when a commit with its subject is there. Subject and `-x` evidence comes only from commits on the release branch that are not reachable from `/main`; shared history still counts for direct reachability. Subjects are matched one to one: two commits with the same subject, `fix tests` say, need two branch commits of that name, and a branch commit whose `-x` reference names any other commit than the PR's merge commit is evidence only through that reference. Nothing weaker is taken: not an `-x` reference to the PR's merge commit, which survives a cherry-pick later amended to drop a commit, and not matching lines, which an unrelated line of the same text satisfies. A wrong `backported` is the one answer a release gate must not give, while a false alarm costs a look and can be settled by an attestation. All commits is `backported`, some is `partial`. This is also what catches a hand backport nobody linked. - -### Attesting a backport complete - -Some complete backports cannot be shown commit by commit, most often a fork contributor's backport squashed into one commit that carries only the first commit's subject. A maintainer who has checked such a backport can say so on the merged backport PR, with a comment that consists of this single line and nothing else: - -```text -backport-audit: complete -``` - -That turns the candidate's `partial` or `unverified` into `confirmed`, which does not fail the gate; the report names the author, links the comment, and keeps listing the commits nothing on the branch names. - -The marker is an explicit attestation by whoever wrote it, never something the audit infers. It counts only when it is the whole comment: exactly `backport-audit: complete`, in that case, starting at the beginning of its line, with nothing but blank lines before it and nothing but whitespace after it. Any other text makes the comment something other than an attestation, so a sentence such as "do not post backport-audit: complete yet", a quote or a code block never counts, and neither does the marker with an explanation next to it; post the explanation as a separate comment. It counts only on a backport PR that merged, since an open or closed one put nothing on the branch to vouch for. And it counts only from someone with a hand in the repository: the comment's GitHub author association has to be `OWNER`, `MEMBER` or `COLLABORATOR`, and comments from automation are ignored whatever their association — logins ending in `[bot]`, and the review and dependency bots gh reports without that suffix, such as `coderabbitai`, `dependabot`, `renovate` and `copilot`. A review bot quoting the marker, or a contributor vouching for their own backport, confirms nothing. - -## Machine-readable output - -```bash -# URLs of what never landed, ready to paste into a tracking issue -go run ./cmd/backport-audit --json release-1.5 | jq -r '.[].candidates[] | select(.status=="MISSING") | .url' - -# every open backport PR the audit links, labelled or not, to merge or close, across lines -go run ./cmd/backport-audit --json release-1.6 release-1.5 \ - | jq -r '[.[] | (.candidates[], .unlabelled[]) | .backport_prs[] | select(.state=="OPEN") | .url] | unique[]' - -# dropped items with the reason someone recorded -go run ./cmd/backport-audit --json release-1.4 \ - | jq -r '.[].candidates[] | select(.status=="dropped") | "#\(.number)\t\(.reason)"' - -# commits a partial backport still lacks, one per line -go run ./cmd/backport-audit --json release-1.6 \ - | jq -r '.[].candidates[] | select(.status=="partial") | "#\(.number)\t\(.missing_commits[] | "\(.oid[0:9]) \(.subject)")"' - -# open backport PRs competing for one original, to settle before the cut -go run ./cmd/backport-audit --json release-1.6 \ - | jq -r '.[].duplicates[] | "#\(.number) \(.kind): \([.backport_prs[] | select(.state=="OPEN") | .url] | join(" "))"' -``` - -Output is an object keyed by branch, each holding three arrays: `candidates`, `unlabelled` and `duplicates`, each always present and possibly empty. `partial`, `unverified` and `confirmed` candidates carry `missing_commits`, each an `oid` and a `subject`, and a `confirmed` one also carries `confirmation`: `by`, `url` and `backport_pr`. An `unlabelled` record carries the same `confirmation` when its merged backport PR has the marker. A `candidates` record: - -```json -{ - "number": 3963, - "title": "fix(clickhouse): scheme the backup S3_ENDPOINT on the system-bucket flow", - "url": "https://github.com/cozystack/cozystack/pull/3963", - "author": "androndo", - "mergedAt": "2026-08-28T11:05:15Z", - "label": "kind/backport", - "backport_prs": [{"number": 3987, "state": "OPEN", "url": "..."}], - "status": "pending", - "evidence": "backport PR #3987 open" -} -``` - -An `unlabelled` record, where `labels` lists the backport requests the original carries for other lines and is empty when it carries none: - -```json -{ - "number": 4280, - "title": "fix(kafka): keep the pre-delete hook's credentials as long as its Job", - "url": "https://github.com/cozystack/cozystack/pull/4280", - "labels": [], - "backport_prs": [ - {"number": 4421, "state": "CLOSED", "url": "..."}, - {"number": 4456, "state": "MERGED", "url": "..."} - ] -} -``` - -A `duplicates` record, where `label` is the request the original was audited under on this line and is empty when it is not a candidate here: - -```json -{ - "number": 4254, - "title": "feat(kubevirt): expose migration configuration through platform values", - "url": "https://github.com/cozystack/cozystack/pull/4254", - "label": "kind/backport", - "kind": "open-twice", - "backport_prs": [ - {"number": 4322, "state": "OPEN", "url": "...", "draft": true}, - {"number": 4339, "state": "OPEN", "url": "..."}, - {"number": 4402, "state": "OPEN", "url": "..."} - ] -} -``` - -## Limits - -A `MISSING`, `partial` or `unverified` verdict is a prompt to check, not proof of absence. A hand backport squashed or reworded without `-x` reads as `partial` or `unverified` until someone attests it complete, and as `MISSING` when nothing links it to its original; confirm at diff level before redoing the work. And the audit reports *that* something is missing, never *why* — a run of `MISSING` entries clustered in time usually means the bot itself was failing during that window, which is worth checking before cherry-picking them one by one. - -The audit reads labels as they stand when it runs. A label added afterwards is seen only by the next run, and until then its PR is either absent or, if someone already backported it, listed as `UNLABELLED` rather than audited, so run it right before the cut rather than once at the start of the day. - -A label is always read against the release lines as they stood when the PR merged, however much later the label was added, because nothing the audit lists records when that happened. A PR merged on 2026-07-01 and labelled `kind/backport` a month later, after `v1.6.0` published, is audited as a `release-1.5` candidate. [`backport.yaml`](../../.github/workflows/backport.yaml) runs again when the label is added and resolves its targets at that moment, so it opens the backport against `release-1.6`. When the line has moved on between the merge and the label, the two therefore disagree: the audit expects the change on the merge-time line and reports it `MISSING` there unless it arrived some other way, and lists the bot's backport under `UNLABELLED` on the newer one. `TestCandidateLabel` pins the audit's side of this. - -A backport PR that names its original neither by the bot's head branch nor in a `Backport of` phrase links to nothing, so it is invisible to both cross-checks: it is not listed as `UNLABELLED` and does not count towards a `DUPLICATE`. For a labelled original the branch history can still prove the change landed, through an identical commit subject or an `-x` cherry-pick reference. For an unlabelled one those are not read at all, so a backport of an unlabelled PR that only a cherry-pick trailer points to, or nothing does, appears nowhere in the report. diff --git a/cmd/backport-audit/main.go b/cmd/backport-audit/main.go deleted file mode 100644 index cee7aa791..000000000 --- a/cmd/backport-audit/main.go +++ /dev/null @@ -1,1569 +0,0 @@ -/* -Copyright 2026 The Cozystack Authors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -// Command backport-audit reports outstanding labelled changes before a release -// cut. It reads GitHub and local Git history, fetching refs unless --no-fetch is -// set. The README describes candidate selection, delivery evidence and limits. -package main - -import ( - "encoding/json" - "errors" - "fmt" - "os" - "os/exec" - "regexp" - "slices" - "sort" - "strconv" - "strings" - "time" -) - -const usageText = `Audit whether every backport-labelled PR actually landed on a release branch. - -Exits 0 when nothing is outstanding and 1 when something is, so it can gate a release. -Backports of PRs that carry no label for the line are listed without affecting the -exit code; an original with more than one live backport PR counts as outstanding. - -Usage: - backport-audit [OPTIONS] [release-X.Y ...] - -Arguments: - release-X.Y ... Release lines to audit (default: the two newest) - -Options: - --remote NAME Git remote holding the release branches (default: origin) - --limit N Max merged PRs to scan per label (default and ceiling: 1000) - --no-fetch Trust the local refs as-is, skipping git fetch - --json Machine-readable output - --no-color Disable color output (auto-disabled on non-TTY) - -h, --help Show this help -` - -// A reference to a PR, optionally qualified by its repository, and a list of -// them as prose writes one: "#1 and #2", "#1, #2 and #3", "#1, #2, and #3". -const ( - prRef = `(?:[\w.-]+/[\w.-]+)?#\d+` - prRefList = prRef + `(?:(?:\s*,\s*(?:and\s+)?|\s+and\s+)` + prRef + `)*` -) - -// A release line named in prose, bare or in backticks, as a whole word: -// release-1.6-fixes, release-1.6.1 and release-1.6.fixes start with a line name -// without being one, so punctuation ends the name only when a space or the end -// of the text follows it. -const releaseLineName = `(?:\x60release-\d+\.\d+\x60|release-\d+\.\d+)(?:$|\s|\)|[.,;:](?:\s|$))` - -// upstreamRepo is the repository an unqualified reference is read against when -// a PR's URL does not say which repository it belongs to. -const upstreamRepo = "cozystack/cozystack" - -var ( - releaseLineRE = regexp.MustCompile(`^release-(\d+)\.(\d+)$`) - backportHeadRE = regexp.MustCompile(`^backport-(\d+)-to-(release-\d+\.\d+)$`) - // A hand backport that carries more than one change names all of them in - // its "Backport of" phrase, either as a list or as the "to release-X.Y, - // together with #N" this repository writes for a dependency pulled along. - backportOfRE = regexp.MustCompile(`[Bb]ackport of\s+(` + prRefList + `)` + - `(?:\s+to\s+\x60?release-\d+\.\d+\x60?,?\s+together with\s+(` + prRefList + `))?`) - // What a list of references may be followed by for every item in it to be - // an original: the end of the line, the end of a sentence, or the "to - // release-X.Y" that names the target. Anything else may be the list going - // on to say something about its later items -- "#10, #20 is not included", - // "#10, #20 to follow in a separate PR" -- so an unterminated list keeps - // only its first item, the one the phrase names directly. - prRefListEndRE = regexp.MustCompile(`^(?:[ \t]*(?:\r?\n|$)|\.(?:\s|$)|\s+to\s+` + releaseLineName + `)`) - prRefRE = regexp.MustCompile(`(?:([\w.-]+/[\w.-]+))?#(\d+)`) - cherryPickRE = regexp.MustCompile(`cherry picked from commit ([0-9a-f]{7,40})`) - releaseTagRE = regexp.MustCompile(`^v(\d+)\.(\d+)\.(\d+)$`) - // The tag whose cut creates release-X.Y. cut-prerelease.yaml gates the - // freeze on kind == rc and patch == 0, so alpha, beta and patch-line rcs - // deliberately do not match. - freezeTagRE = regexp.MustCompile(`^v(\d+)\.(\d+)\.0-rc\.\d+$`) -) - -// Caps on the three listings the audit makes. gh truncates at --limit in -// silence, so each one is checked against its cap by truncated() and the -// numbers are set well clear of the current populations rather than close to -// them: at the time of writing, 290 labelled PRs, 217 releases, and at most 54 -// PRs on any one release branch. Raising a cap costs nothing when it is not -// reached -- gh stops as soon as the result set is exhausted -- so the reason -// they are bounded at all is to keep a runaway query from paginating forever. -const ( - releaseListCap = 1000 - branchPRCap = 1000 -) - -// searchResultCap is the most results GitHub search returns for one query, -// however far it is paginated. gh pr list answers a --label filter through -// search, so the candidate listing stops here whatever --limit asks for, and -// gh's --json output does not warn that it did. The other two listings do not -// go through search and are bounded by their own caps alone. -const searchResultCap = 1000 - -// candidateCeiling is the listing length at which a label query with --limit -// limit has to be taken as truncated, and what to do about it. -func candidateCeiling(limit int) (int, string) { - if limit < searchResultCap { - return limit, "re-run with a higher --limit" - } - return searchResultCap, "GitHub search returns at most 1000 results per query; split the label query in cmd/backport-audit, e.g. by merge date" -} - -// freezeContractLandedAt is when cutting an rc started creating release-X.Y, -// and with it when backport.yaml stopped resolving its targets from the last -// published stable. -// -// One moment, not two: 53a7fc4dc ("cutting an rc freezes the line into -// release-X.Y") and 6ff5b98d5 ("target the newest existing release line") -// landed on main in the same push, which is what makes a single cutover -// faithful. Before it a release-X.Y branch was created at promote time, so the -// newest branch and the newest published stable named the same line and the -// two rules could not disagree. After it they disagree for the length of every -// freeze window, which is precisely when a backport has to reach the branch. -var freezeContractLandedAt = time.Date(2026, 8, 3, 16, 31, 59, 0, time.UTC) - -// The two backport requests .github/workflows/backport.yaml acts on, named by -// the label that carries them. These names are what the audit reports. -const ( - labelCurrent = "kind/backport" - labelPrevious = "kind/backport-previous" -) - -// backportLabels maps each request to every label spelling that expresses it. -// -// TRANSITIONAL, matching the same allowance in backport.yaml: the labels were -// namespaced under kind/ in 7b71053a0, carrying the bare names as label-sync -// aliases so the rename applied in place. Historical PRs therefore read back as -// `kind/backport` today, but the org-level dosubot still applies the bare names -// on new PRs, so a candidate can carry either. Both are queried and folded onto -// the canonical name above. When dosubot's PR labelling is switched off, the -// legacy spellings here and the ones in backport.yaml can be dropped together. -var backportLabels = []struct { - canonical string - spellings []string -}{ - {labelCurrent, []string{labelCurrent, "backport"}}, - {labelPrevious, []string{labelPrevious, "backport-previous"}}, -} - -// Statuses, most to least in need of attention. outstanding is the prefix of -// this order that makes the audit exit non-zero. -const ( - statusMissing = "MISSING" - statusPartial = "partial" - statusUnverified = "unverified" - statusPending = "pending" - statusDropped = "dropped" - statusConfirmed = "confirmed" - statusBackported = "backported" - statusInBranch = "in-branch" -) - -var ( - statusOrder = []string{statusMissing, statusPartial, statusUnverified, statusPending, statusDropped, - statusConfirmed, statusBackported, statusInBranch} - outstanding = []string{statusMissing, statusPartial, statusUnverified, statusPending, statusDropped} - headings = map[string]string{ - statusMissing: "MISSING -- labelled for this line, no trace of it here", - statusPartial: "PARTIAL -- some of the PR's commits are here, not all of them", - statusUnverified: "UNVERIFIED -- backport PR merged, none of the PR's commits found here", - statusPending: "PENDING -- backport PR open, not merged", - statusDropped: "DROPPED -- backport PR closed without merging", - } -) - -// The two ways one original can be claimed by more than one live backport PR on -// a branch. Both are outstanding. -const ( - dupOpenTwice = "open-twice" - dupOpenAfterMerge = "open-after-merge" -) - -var dupDescriptions = map[string]string{ - dupOpenTwice: "more than one backport PR open", - dupOpenAfterMerge: "a backport PR still open after another one merged", -} - -type config struct { - remote string - limit int - fetch bool - asJSON bool - useColor bool - branches []string - - red, green, yellow, cyan, dim, bold, reset string -} - -// line is a release line, i.e. a major.minor pair with a release-X.Y branch. -type line struct{ major, minor int } - -func (l line) branch() string { return fmt.Sprintf("release-%d.%d", l.major, l.minor) } -func (l line) less(o line) bool { - if l.major != o.major { - return l.major < o.major - } - return l.minor < o.minor -} - -// mainPR is a merged-to-main PR carrying at least one backport label. -type mainPR struct { - Number int `json:"number"` - Title string `json:"title"` - URL string `json:"url"` - MergedAt time.Time `json:"mergedAt"` - MergeCommit struct { - Oid string `json:"oid"` - } `json:"mergeCommit"` - Author struct { - Login string `json:"login"` - } `json:"author"` - Labels []struct { - Name string `json:"name"` - } `json:"labels"` - - labels map[string]bool -} - -// backportPR is a PR opened against a release branch. -type backportPR struct { - Number int `json:"number"` - Title string `json:"title"` - URL string `json:"url"` - State string `json:"state"` - HeadRefName string `json:"headRefName"` - Body string `json:"body"` - IsDraft bool `json:"isDraft"` - Comments []prComment `json:"comments"` -} - -// prComment is one comment on a backport PR. -type prComment struct { - Author struct { - Login string `json:"login"` - } `json:"author"` - AuthorAssociation string `json:"authorAssociation"` - Body string `json:"body"` - URL string `json:"url"` -} - -// linkedPR is the trimmed view of a backport PR carried in the output. -type linkedPR struct { - Number int `json:"number"` - State string `json:"state"` - URL string `json:"url"` - Draft bool `json:"draft,omitempty"` -} - -func trimPR(b backportPR) linkedPR { - return linkedPR{Number: b.Number, State: b.State, URL: b.URL, Draft: b.IsDraft} -} - -// verdict is one audited candidate. The JSON field names are the tool's -// contract with jq one-liners, so they are spelled explicitly. -type verdict struct { - Number int `json:"number"` - Title string `json:"title"` - URL string `json:"url"` - Author string `json:"author"` - MergedAt string `json:"mergedAt"` - Label string `json:"label"` - BackportPRs []linkedPR `json:"backport_prs"` - Status string `json:"status"` - Evidence string `json:"evidence"` - Reason string `json:"reason,omitempty"` - // MissingCommits are the PR's commits a partial or unverified backport - // shows no trace of. - MissingCommits []prCommit `json:"missing_commits,omitempty"` - Confirmation *confirmation `json:"confirmation,omitempty"` -} - -// prCommit is one commit a PR contributed to main. -type prCommit struct { - Oid string `json:"oid"` - Subject string `json:"subject"` -} - -// unlabelledBackport is an original that backport PRs on the branch claim -// although it is not a candidate for the branch. Labels holds the backport -// requests it does carry, which then resolved to some other line. -type unlabelledBackport struct { - Number int `json:"number"` - Title string `json:"title"` - URL string `json:"url"` - Labels []string `json:"labels"` - BackportPRs []linkedPR `json:"backport_prs"` - Confirmation *confirmation `json:"confirmation,omitempty"` -} - -// duplicateBackport is an original claimed by more than one live backport PR -// on the branch. Label is the request it was audited under here, empty when it -// is not a candidate for the branch. -type duplicateBackport struct { - Number int `json:"number"` - Title string `json:"title"` - URL string `json:"url"` - Label string `json:"label"` - Kind string `json:"kind"` - BackportPRs []linkedPR `json:"backport_prs"` -} - -// branchReport is everything the audit found on one release branch. -type branchReport struct { - Candidates []verdict `json:"candidates"` - Unlabelled []unlabelledBackport `json:"unlabelled"` - Duplicates []duplicateBackport `json:"duplicates"` -} - -// clean reports whether the branch passes the gate. Unlabelled backports do not -// enter into it: they report claims outside the labelled candidate set. -func (r *branchReport) clean() bool { - for _, v := range r.Candidates { - if slices.Contains(outstanding, v.Status) { - return false - } - } - return len(r.Duplicates) == 0 -} - -func main() { os.Exit(run()) } - -func run() int { - cfg, code, done := parseArgs(os.Args[1:]) - if done { - return code - } - - if cfg.fetch { - fmt.Fprintln(os.Stderr, "fetching refs...") - if _, err := capture("git", "fetch", "--quiet", "--tags", cfg.remote); err != nil { - fmt.Fprintf(os.Stderr, "git fetch failed: %v\n", err) - return 2 - } - } - - if len(cfg.branches) == 0 { - lines, err := releaseLines(cfg.remote) - if err != nil { - fmt.Fprintf(os.Stderr, "%v\n", err) - return 2 - } - if len(lines) == 0 { - fmt.Fprintf(os.Stderr, "no release-X.Y branches found on %s\n", cfg.remote) - return 2 - } - // Newest first, at most two. - for i := len(lines) - 1; i >= 0 && len(cfg.branches) < 2; i-- { - cfg.branches = append(cfg.branches, lines[i].branch()) - } - fmt.Fprintf(os.Stderr, "auditing %s\n", strings.Join(cfg.branches, ", ")) - } - - for _, b := range cfg.branches { - if !releaseLineRE.MatchString(b) { - fmt.Fprintf(os.Stderr, "not a release line: %s (expected release-.)\n", b) - return 2 - } - if !gitOK("rev-parse", "--verify", "--quiet", cfg.remote+"/"+b) { - fmt.Fprintf(os.Stderr, "no such branch: %s/%s\n", cfg.remote, b) - return 2 - } - } - - opened, err := lineOpenDates() - if err != nil { - fmt.Fprintf(os.Stderr, "%v\n", err) - return 2 - } - cands, err := candidates(cfg.limit) - if err != nil { - fmt.Fprintf(os.Stderr, "%v\n", err) - return 2 - } - fmt.Fprintf(os.Stderr, "%d merged PRs carry a backport label\n", len(cands)) - - out := map[string]*branchReport{} - clean := true - for _, branch := range cfg.branches { - rep, err := cfg.audit(branch, cands, opened) - if err != nil { - fmt.Fprintf(os.Stderr, "%v\n", err) - return 2 - } - out[branch] = rep - // The exit code is the whole point of running this in a gate, so it is - // computed from the results, not as a side effect of printing them. - if !rep.clean() { - clean = false - } - } - - // A title is decoration, so a failed lookup costs the titles and nothing - // else: the verdicts and the exit code are already settled above. - if err := fillTitles(out); err != nil { - fmt.Fprintf(os.Stderr, "could not look up titles of unlabelled originals: %v\n", err) - } - - if cfg.asJSON { - blob, err := json.MarshalIndent(out, "", " ") - if err != nil { - fmt.Fprintf(os.Stderr, "%v\n", err) - return 2 - } - fmt.Println(string(blob)) - } else { - for _, branch := range cfg.branches { - cfg.report(branch, out[branch]) - } - fmt.Println() - } - - if clean { - return 0 - } - return 1 -} - -func parseArgs(args []string) (*config, int, bool) { - cfg := &config{remote: "origin", limit: 1000, fetch: true, useColor: true} - for i := 0; i < len(args); i++ { - a := args[i] - next := func() (string, bool) { - if i+1 >= len(args) { - fmt.Fprintf(os.Stderr, "%s requires a value\n", a) - return "", false - } - i++ - return args[i], true - } - switch a { - case "-h", "--help": - fmt.Print(usageText) - return nil, 0, true - case "--remote": - v, ok := next() - if !ok { - return nil, 2, true - } - cfg.remote = v - case "--limit": - v, ok := next() - if !ok { - return nil, 2, true - } - n, err := strconv.Atoi(v) - if err != nil || n <= 0 { - fmt.Fprintf(os.Stderr, "invalid --limit value: %s\n", v) - return nil, 2, true - } - cfg.limit = n - case "--no-fetch": - cfg.fetch = false - case "--json": - cfg.asJSON = true - case "--no-color": - cfg.useColor = false - default: - if strings.HasPrefix(a, "-") { - fmt.Fprintf(os.Stderr, "Unknown argument: %s\n", a) - fmt.Fprint(os.Stderr, usageText) - return nil, 2, true - } - cfg.branches = append(cfg.branches, a) - } - } - - // Color is for a human reading a terminal; --json and pipes get none. - if cfg.asJSON { - cfg.useColor = false - } - if fi, err := os.Stdout.Stat(); err != nil || fi.Mode()&os.ModeCharDevice == 0 { - cfg.useColor = false - } - cfg.setColors() - return cfg, 0, false -} - -func (cfg *config) setColors() { - if cfg.useColor { - cfg.red = "\033[0;31m" - cfg.green = "\033[0;32m" - cfg.yellow = "\033[1;33m" - cfg.cyan = "\033[0;36m" - cfg.dim = "\033[2m" - cfg.bold = "\033[1m" - cfg.reset = "\033[0m" - } -} - -// colorFor is the accent a status is reported in. -func (cfg *config) colorFor(status string) string { - switch status { - case statusMissing, statusPartial, statusUnverified: - return cfg.red - case statusPending: - return cfg.yellow - case statusDropped: - return cfg.cyan - case statusConfirmed, statusBackported, statusInBranch: - return cfg.green - } - return "" -} - -func capture(name string, args ...string) (string, error) { - out, err := exec.Command(name, args...).Output() - if err != nil { - if ee, ok := err.(*exec.ExitError); ok && len(ee.Stderr) > 0 { - return "", fmt.Errorf("%s %s: %w: %s", name, strings.Join(args, " "), err, strings.TrimSpace(string(ee.Stderr))) - } - return "", fmt.Errorf("%s %s: %w", name, strings.Join(args, " "), err) - } - return string(out), nil -} - -// git runs a git command whose failure means the audit cannot be trusted. -func git(args ...string) (string, error) { return capture("git", args...) } - -func gitOK(args ...string) bool { - return exec.Command("git", args...).Run() == nil -} - -// truncated turns a saturated `gh ... --limit N` query into an error instead of -// a quietly shorter answer. -// -// Every list this tool makes is a completeness claim -- these are all the -// candidates, this is every backport PR on the branch -- and gh stops at -// --limit without saying so. A saturated query therefore does not make the -// audit partial, it makes its verdicts wrong: the PRs past the cut are -// reported nowhere, and the exit code says clean. That is the same failure -// class the tool exists to catch, so it fails loudly instead. -func truncated(got, limit int, what, remedy string) error { - if got < limit { - return nil - } - return fmt.Errorf("%s hit its cap of %d results, so the list is truncated and the audit would be unsound: %s", what, limit, remedy) -} - -func ghJSON(target any, args ...string) error { - out, err := capture("gh", args...) - if err != nil { - return err - } - return json.Unmarshal([]byte(out), target) -} - -// releaseLines returns the two-component release branches on the remote, -// oldest first. release-1.6.0-rc.1 and the like are staging branches, not -// maintenance lines, so they are excluded. -func releaseLines(remote string) ([]line, error) { - out, err := git("branch", "-r", "--list", remote+"/release-*") - if err != nil { - return nil, err - } - seen := map[line]bool{} - for raw := range strings.SplitSeq(out, "\n") { - name := strings.TrimPrefix(strings.TrimSpace(raw), remote+"/") - if m := releaseLineRE.FindStringSubmatch(name); m != nil { - major, _ := strconv.Atoi(m[1]) - minor, _ := strconv.Atoi(m[2]) - seen[line{major, minor}] = true - } - } - lines := make([]line, 0, len(seen)) - for l := range seen { - lines = append(lines, l) - } - sort.Slice(lines, func(i, j int) bool { return lines[i].less(lines[j]) }) - return lines, nil -} - -type lineOpen struct { - line line - when time.Time -} - -// Stable publication dates cover lines opened before the freeze contract; -// tagged-commit dates estimate openings since then. Both eras are needed when -// resolving a historical backport label against the lines available at merge. -func lineOpenDates() ([]lineOpen, error) { - var releases []struct { - TagName string `json:"tagName"` - PublishedAt time.Time `json:"publishedAt"` - IsPrerelease bool `json:"isPrerelease"` - IsDraft bool `json:"isDraft"` - } - if err := ghJSON(&releases, "release", "list", "--limit", strconv.Itoa(releaseListCap), - "--json", "tagName,publishedAt,isPrerelease,isDraft"); err != nil { - return nil, err - } - if err := truncated(len(releases), releaseListCap, "the release listing", - "raise releaseListCap in cmd/backport-audit"); err != nil { - return nil, err - } - - earliest := map[line]time.Time{} - for _, rel := range releases { - if rel.IsPrerelease || rel.IsDraft { - continue - } - m := releaseTagRE.FindStringSubmatch(rel.TagName) - if m == nil { - continue - } - major, _ := strconv.Atoi(m[1]) - minor, _ := strconv.Atoi(m[2]) - l := line{major, minor} - if prev, ok := earliest[l]; !ok || rel.PublishedAt.Before(prev) { - earliest[l] = rel.PublishedAt - } - } - - frozen, err := freezeDates() - if err != nil { - return nil, err - } - for l, when := range frozen { - earliest[l] = when - } - - opened := make([]lineOpen, 0, len(earliest)) - for l, when := range earliest { - opened = append(opened, lineOpen{l, when}) - } - sort.Slice(opened, func(i, j int) bool { return opened[i].when.Before(opened[j].when) }) - return opened, nil -} - -// freezeDates estimates line opening from the earliest vX.Y.0-rc.N tagged -// commit at or after the freeze-contract cutover. Earlier tags are ignored; -// lines with no remaining tag use the published-stable date in lineOpenDates. -// The tagged commit can predate the branch push; see the README's limits. -func freezeDates() (map[line]time.Time, error) { - out, err := git("for-each-ref", "--format=%(refname:strip=2) %(committerdate:iso-strict)", "refs/tags/v*") - if err != nil { - return nil, err - } - return parseFreezeDates(out) -} - -// parseFreezeDates is the half of freezeDates that does not need a repository: -// ` ` per line, in, freeze moments out. -func parseFreezeDates(out string) (map[line]time.Time, error) { - frozen := map[line]time.Time{} - for raw := range strings.SplitSeq(out, "\n") { - name, date, ok := strings.Cut(strings.TrimSpace(raw), " ") - if !ok { - continue - } - m := freezeTagRE.FindStringSubmatch(name) - if m == nil { - continue - } - when, err := time.Parse(time.RFC3339, date) - if err != nil { - return nil, fmt.Errorf("tag %s: unparseable date %q: %w", name, date, err) - } - if when.Before(freezeContractLandedAt) { - continue - } - major, _ := strconv.Atoi(m[1]) - minor, _ := strconv.Atoi(m[2]) - l := line{major, minor} - if prev, ok := frozen[l]; !ok || when.Before(prev) { - frozen[l] = when - } - } - return frozen, nil -} - -// targetsAt reports the current and previous release lines as of when, which is -// what the `kind/backport` and `kind/backport-previous` labels meant at that -// moment. -// -// The lines that exist at that point are ranked by version, not by when they -// opened, matching the numeric descending sort backport.yaml applies to its -// branch list. The two orders normally coincide, and stop coinciding as soon -// as a freeze overlaps the previous line's stabilisation: cut vX.(Y+1).0-rc.1 -// while vX.Y.0 is still unpublished and the newer line opens first in time -// while still being the newer line. Ranking by version is also what keeps -// release-1.10 above release-1.9. -func targetsAt(opened []lineOpen, when time.Time) (current, previous *line) { - var live []line - for _, o := range opened { - if !o.when.After(when) { - live = append(live, o.line) - } - } - sort.Slice(live, func(i, j int) bool { return live[j].less(live[i]) }) - if n := len(live); n > 0 { - current = &live[0] - if n > 1 { - previous = &live[1] - } - } - return current, previous -} - -// candidates returns the merged-to-main PRs carrying a backport label, keyed by -// PR number so a PR matched under two spellings of the same request, or under -// both requests, is one entry carrying both canonical names. -func candidates(limit int) (map[int]*mainPR, error) { - found := map[int]*mainPR{} - for _, req := range backportLabels { - for _, spelling := range req.spellings { - var prs []mainPR - if err := ghJSON(&prs, "pr", "list", "--base", "main", "--state", "merged", - "--label", spelling, "--limit", strconv.Itoa(limit), - "--json", "number,title,url,mergedAt,mergeCommit,labels,author"); err != nil { - return nil, err - } - ceiling, remedy := candidateCeiling(limit) - if err := truncated(len(prs), ceiling, "the merged-PR query for label "+spelling, remedy); err != nil { - return nil, err - } - for i := range prs { - pr := prs[i] - existing, ok := found[pr.Number] - if !ok { - pr.labels = map[string]bool{} - found[pr.Number] = &pr - existing = &pr - } - existing.labels[req.canonical] = true - } - } - } - return found, nil -} - -type branchHistory struct { - branch string - ref string - commits map[string]bool - // Only commits outside main supply subject and -x evidence. Shared - // ancestors can have the same subject without carrying the backport. - subjects map[string][]string - refs map[string][]string -} - -func newBranchHistory(remote, branch string) (*branchHistory, error) { - h := &branchHistory{ - branch: branch, - ref: remote + "/" + branch, - commits: map[string]bool{}, - subjects: map[string][]string{}, - refs: map[string][]string{}, - } - out, err := git("rev-list", h.ref) - if err != nil { - return nil, err - } - for oid := range strings.FieldsSeq(out) { - h.commits[oid] = true - } - out, err = git("log", "--format=%x00%H%x01%s%x01%b", h.ref, "--not", remote+"/main") - if err != nil { - return nil, err - } - for entry := range strings.SplitSeq(out, "\x00") { - if strings.TrimSpace(entry) == "" { - continue - } - parts := strings.SplitN(entry, "\x01", 3) - for len(parts) < 3 { - parts = append(parts, "") - } - oid, subject := strings.TrimSpace(parts[0]), strings.TrimSpace(parts[1]) - h.subjects[subject] = append(h.subjects[subject], oid) - for _, m := range cherryPickRE.FindAllStringSubmatch(parts[2], -1) { - h.refs[oid] = append(h.refs[oid], m[1]) - } - } - return h, nil -} - -func (h *branchHistory) hasSubject(subject string) bool { - return len(h.subjects[strings.TrimSpace(subject)]) > 0 -} - -func (h *branchHistory) contains(commit string) bool { - return commit != "" && h.commits[commit] -} - -// sameCommit reports whether two object names can be the same commit: an -x -// reference may abbreviate, so either may be a prefix of the other. -func sameCommit(a, b string) bool { - return a != "" && b != "" && (strings.HasPrefix(a, b) || strings.HasPrefix(b, a)) -} - -// missingFrom returns the commits of the PR merged as mergeCommit that the -// branch does not carry. A commit is carried when it is on the branch itself, -// when a branch commit's -x reference names it, or when a branch commit with -// its subject is left over for it. -// -// Subjects are spent one to one: two commits both called "fix tests" need two -// branch commits of that name, not the one they share. One of the PR's own -// commits on the branch is spent on itself, and a branch commit whose -x -// reference names anything but the PR's merge commit -- one of the PR's -// commits included -- is evidence only through that reference: it has said -// what it is a copy of. -func (h *branchHistory) missingFrom(commits []prCommit, mergeCommit string) []prCommit { - spent := map[string]bool{} - carried := make([]bool, len(commits)) - for i, c := range commits { - if h.contains(c.Oid) { - carried[i], spent[c.Oid] = true, true - } - for _, refs := range h.refs { - for _, r := range refs { - if sameCommit(c.Oid, r) { - carried[i] = true - } - } - } - } - var missing []prCommit - for i, c := range commits { - for _, b := range h.subjects[strings.TrimSpace(c.Subject)] { - if carried[i] { - break - } - if !spent[b] && h.namesOnly(b, mergeCommit) { - carried[i], spent[b] = true, true - } - } - if !carried[i] { - missing = append(missing, c) - } - } - return missing -} - -// namesOnly reports whether every -x reference branch commit b carries, if it -// carries any, names commit. -func (h *branchHistory) namesOnly(b, commit string) bool { - for _, r := range h.refs[b] { - if !sameCommit(r, commit) { - return false - } - } - return true -} - -// backportPRsFor indexes every PR on branch by the main PRs it claims to -// backport; see claimedOrigins. -func backportPRsFor(branch string) (map[int][]backportPR, error) { - var prs []backportPR - if err := ghJSON(&prs, "pr", "list", "--base", branch, "--state", "all", "--limit", strconv.Itoa(branchPRCap), - "--json", "number,title,url,state,headRefName,body,isDraft,comments"); err != nil { - return nil, err - } - if err := truncated(len(prs), branchPRCap, "the PR listing for "+branch, - "raise branchPRCap in cmd/backport-audit"); err != nil { - return nil, err - } - return indexByOrigin(prs, branch), nil -} - -func indexByOrigin(prs []backportPR, branch string) map[int][]backportPR { - index := map[int][]backportPR{} - for _, pr := range prs { - for _, n := range claimedOrigins(pr, branch) { - index[n] = append(index[n], pr) - } - } - return index -} - -// claimedOrigins reads the main PRs a PR on branch says it backports: the one -// in the bot's head branch name, and every one in the body's "Backport of" -// phrases, which is what a hand-written backport carries. Hand backports often -// reuse the bot's head naming, so the two usually agree and are counted once. -func claimedOrigins(pr backportPR, branch string) []int { - var origins []int - if m := backportHeadRE.FindStringSubmatch(pr.HeadRefName); m != nil && m[2] == branch { - n, _ := strconv.Atoi(m[1]) - origins = append(origins, n) - } - for _, n := range bodyOrigins(pr.Body, repoOf(pr.URL)) { - if !slices.Contains(origins, n) { - origins = append(origins, n) - } - } - return origins -} - -// bodyOrigins returns the PR numbers named in a body's "Backport of" phrases, in -// order of appearance and without repeats. -// -// A reference qualified with any repository but repo is skipped rather than -// read as a local number: "other/repo#20" linked as #20 would let an unrelated -// backport vouch for whatever local PR happens to carry that number, and a -// merged one would turn its MISSING verdict into backported. -func bodyOrigins(body, repo string) []int { - var origins []int - for _, m := range backportOfRE.FindAllStringSubmatchIndex(body, -1) { - for g := 1; 2*g+1 < len(m); g++ { - start, end := m[2*g], m[2*g+1] - if start < 0 { - continue - } - refs := prRefRE.FindAllStringSubmatch(body[start:end], -1) - if !prRefListEndRE.MatchString(body[end:]) { - refs = refs[:1] - } - for _, ref := range refs { - if ref[1] != "" && !strings.EqualFold(ref[1], repo) { - continue - } - n, _ := strconv.Atoi(ref[2]) - if !slices.Contains(origins, n) { - origins = append(origins, n) - } - } - } - } - return origins -} - -// repoOf is the owner/name of the repository a PR URL, as gh prints it, points -// into. -func repoOf(prURL string) string { - i := strings.LastIndex(prURL, "/pull/") - if i < 0 { - return upstreamRepo - } - parts := strings.Split(prURL[:i], "/") - if len(parts) < 2 { - return upstreamRepo - } - return parts[len(parts)-2] + "/" + parts[len(parts)-1] -} - -// botAuthor reports whether a comment was written by automation rather than by -// a person: the backport bot, a review bot, a dependency bot. gh reports some -// of them without the "[bot]" suffix a GitHub App login otherwise carries, so -// those are named. -func botAuthor(login string) bool { - l := strings.ToLower(login) - if strings.HasSuffix(l, "[bot]") { - return true - } - switch l { - case "github-actions", "coderabbitai", "dependabot", "renovate", "copilot", - "copilot-pull-request-reviewer", "dosubot", "gemini-code-assist": - return true - } - return false -} - -// lastHumanComment is the last comment on a PR not written by the backport bot. -func lastHumanComment(pr backportPR) string { - for i := len(pr.Comments) - 1; i >= 0; i-- { - login := pr.Comments[i].Author.Login - if botAuthor(login) { - continue - } - text := strings.Join(strings.Fields(pr.Comments[i].Body), " ") - if text == "" { - continue - } - if len(text) > 200 { - text = text[:200] - } - return login + ": " + text - } - return "no reason recorded" -} - -// confirmMarker is what a maintainer posts, as the whole of a comment on a -// merged backport PR, to attest that it carries the whole original although -// the audit cannot show that commit by commit, as with a backport squashed into -// one commit. The audit never infers it. -const confirmMarker = "backport-audit: complete" - -// confirmation is that attestation: who wrote it, where, on which backport PR. -type confirmation struct { - By string `json:"by"` - URL string `json:"url"` - BackportPR int `json:"backport_pr"` -} - -// confirmedBy returns the first attestation among prs, or nil. Only a backport -// PR that merged counts, since an open or closed one put nothing on the branch -// to vouch for, and only a comment by someone with a hand in the repository: -// its owner, a member of its organisation or a collaborator. A review bot -// quoting the marker, or a drive-by contributor, confirms nothing. -func confirmedBy(prs []backportPR) *confirmation { - for _, b := range prs { - if b.State != "MERGED" { - continue - } - for _, c := range b.Comments { - if !botAuthor(c.Author.Login) && maintains(c.AuthorAssociation) && markerComment(c.Body) { - return &confirmation{By: c.Author.Login, URL: c.URL, BackportPR: b.Number} - } - } - } - return nil -} - -// markerComment reports whether body is the marker and nothing else: only blank -// lines before it, only whitespace after it, and nothing before it on its own -// line. A comment that holds anything more -- a sentence mentioning it ("do not -// post backport-audit: complete yet"), a quote, a code block -- is talk about -// attesting, not an attestation, and no reading of markdown is needed to tell -// the two apart. Leading spaces or a tab on the marker's line are rejected -// rather than trimmed, since enough of them turn the line into code. -func markerComment(body string) bool { - before, found := strings.CutSuffix(strings.TrimRight(body, " \t\r\n"), confirmMarker) - return found && strings.Trim(before, "\r\n") == "" -} - -// maintains reports whether a comment's GitHub author association is one that -// may vouch for a backport. -func maintains(association string) bool { - switch association { - case "OWNER", "MEMBER", "COLLABORATOR": - return true - } - return false -} - -// commitSource lists the commits a PR contributed: gitSource in the audit, -// fixtures in tests. -type commitSource interface { - // commits lists, oldest first, the commits a PR contributed that a - // backport has to carry. A read that fails is an error, never an empty - // list, since a verdict built on it would be a guess. - commits(mergeCommit string) ([]prCommit, error) -} - -// gitSource reads commits through run, which is git in the audit. -type gitSource struct { - run func(args ...string) (string, error) -} - -// commits reads the PR's side of its merge. main takes PRs as merge commits, so -// that is ^1..^2; a squashed or rebased merge has a single parent, and then the -// commit itself is the change. -// -// Merge commits and commits that change nothing are left out: the backport bot -// drops both (merge_commits: skip, and an empty cherry-pick has nothing to -// apply), so a backport without them is complete, and a "re-trigger CI" commit -// must not make one look partial. -func (g gitSource) commits(mergeCommit string) ([]prCommit, error) { - if mergeCommit == "" { - return nil, errors.New("no merge commit recorded") - } - parents, err := g.run("rev-list", "--parents", "-n", "1", mergeCommit) - if err != nil { - return nil, err - } - var rng string - switch f := strings.Fields(parents); len(f) { - case 2: - rng = mergeCommit + "^.." + mergeCommit - case 3: - rng = mergeCommit + "^1.." + mergeCommit + "^2" - default: - return nil, fmt.Errorf("merge commit %s has %d parents", mergeCommit, len(f)-1) - } - out, err := g.run("log", "--reverse", "--no-merges", "--name-only", "--format=%x00%H%x01%s", rng) - if err != nil { - return nil, err - } - return parsePRCommits(out), nil -} - -// parsePRCommits is the half of gitSource.commits that does not need a -// repository: `git log --name-only` output in, the commits that touched a file -// out. -func parsePRCommits(out string) []prCommit { - var commits []prCommit - for entry := range strings.SplitSeq(out, "\x00") { - head, files, _ := strings.Cut(entry, "\n") - oid, subject, ok := strings.Cut(head, "\x01") - if !ok || strings.TrimSpace(files) == "" { - continue - } - commits = append(commits, prCommit{Oid: oid, Subject: subject}) - } - return commits -} - -// candidateLabel is the backport request under which pr is a candidate for the -// line want, or "" when it is not one. -// -// The request is read against the lines as they stood when the PR merged, even -// for a label added later: nothing the audit lists records when a label was -// added. backport.yaml, which runs again when a label is added, resolves its -// targets at that moment instead, so the two disagree for a label added after -// the line moved on. -func candidateLabel(pr *mainPR, opened []lineOpen, want line) string { - current, previous := targetsAt(opened, pr.MergedAt) - switch { - case pr.labels[labelCurrent] && current != nil && *current == want: - return labelCurrent - case pr.labels[labelPrevious] && previous != nil && *previous == want: - return labelPrevious - } - return "" -} - -func (cfg *config) audit(branch string, cands map[int]*mainPR, opened []lineOpen) (*branchReport, error) { - hist, err := newBranchHistory(cfg.remote, branch) - if err != nil { - return nil, err - } - backports, err := backportPRsFor(branch) - if err != nil { - return nil, err - } - - m := releaseLineRE.FindStringSubmatch(branch) - major, _ := strconv.Atoi(m[1]) - minor, _ := strconv.Atoi(m[2]) - want := line{major, minor} - - numbers := make([]int, 0, len(cands)) - for n := range cands { - numbers = append(numbers, n) - } - sort.Ints(numbers) - - results := []verdict{} - audited := map[int]string{} - for _, n := range numbers { - pr := cands[n] - label := candidateLabel(pr, opened, want) - if label == "" { - continue - } - audited[n] = label - v, err := classify(pr, label, hist, backports[pr.Number], gitSource{run: git}) - if err != nil { - return nil, err - } - results = append(results, v) - } - unlabelled, duplicates := crossCheck(backports, audited, cands) - return &branchReport{Candidates: results, Unlabelled: unlabelled, Duplicates: duplicates}, nil -} - -// crossCheck reads the backport PRs on a branch from the side of the originals -// they claim, which is the only side an unlabelled backport can be seen from. -// audited maps each candidate for the branch to the request it was audited -// under; cands supplies the title and labels of any original that carries a -// backport label at all. -func crossCheck(index map[int][]backportPR, audited map[int]string, cands map[int]*mainPR) ([]unlabelledBackport, []duplicateBackport) { - numbers := make([]int, 0, len(index)) - for n := range index { - numbers = append(numbers, n) - } - sort.Ints(numbers) - - unlabelled := []unlabelledBackport{} - duplicates := []duplicateBackport{} - for _, n := range numbers { - claims := index[n] - linked := make([]linkedPR, 0, len(claims)) - for _, b := range claims { - linked = append(linked, trimPR(b)) - } - sort.Slice(linked, func(i, j int) bool { return linked[i].Number < linked[j].Number }) - - // The URL is derived rather than looked up, so it is there even when the - // title lookup fails; GitHub redirects /pull/N to the issue when N is one. - title, url, labels := "", originURL(claims[0].URL, n), []string{} - if pr, ok := cands[n]; ok { - title, url = pr.Title, pr.URL - for _, req := range backportLabels { - if pr.labels[req.canonical] { - labels = append(labels, req.canonical) - } - } - } - - label, isCandidate := audited[n] - if !isCandidate { - unlabelled = append(unlabelled, unlabelledBackport{ - Number: n, Title: title, URL: url, Labels: labels, BackportPRs: linked, Confirmation: confirmedBy(claims), - }) - } - if kind := duplicateKind(claims); kind != "" { - duplicates = append(duplicates, duplicateBackport{ - Number: n, Title: title, URL: url, Label: label, Kind: kind, BackportPRs: linked, - }) - } - } - return unlabelled, duplicates -} - -// duplicateKind classifies the backport PRs claiming one original, returning -// "" unless more than one of them is live. A closed PR next to an open one is -// the normal shape of a conflicting bot backport redone by hand, and two merged -// ones are history the branch already carries, so neither is flagged. -func duplicateKind(claims []backportPR) string { - open, merged := 0, 0 - for _, b := range claims { - switch b.State { - case "OPEN": - open++ - case "MERGED": - merged++ - } - } - switch { - case open > 0 && merged > 0: - return dupOpenAfterMerge - case open > 1: - return dupOpenTwice - } - return "" -} - -// originURL is the URL of PR n in the repository that sibling, the URL of -// another PR, belongs to. -func originURL(sibling string, n int) string { - i := strings.LastIndex(sibling, "/pull/") - if i < 0 { - return "" - } - return sibling[:i] + "/pull/" + strconv.Itoa(n) -} - -// fillTitles sets the title of every unlabelled or duplicate original that no -// listing carried, with one request covering every branch. -func fillTitles(out map[string]*branchReport) error { - var missing []int - for _, rep := range out { - for _, u := range rep.Unlabelled { - if u.Title == "" && !slices.Contains(missing, u.Number) { - missing = append(missing, u.Number) - } - } - for _, d := range rep.Duplicates { - if d.Title == "" && !slices.Contains(missing, d.Number) { - missing = append(missing, d.Number) - } - } - } - if len(missing) == 0 { - return nil - } - sort.Ints(missing) - titles, err := titlesOf(missing) - for _, rep := range out { - for i := range rep.Unlabelled { - if t, ok := titles[rep.Unlabelled[i].Number]; ok && rep.Unlabelled[i].Title == "" { - rep.Unlabelled[i].Title = t - } - } - for i := range rep.Duplicates { - if t, ok := titles[rep.Duplicates[i].Number]; ok && rep.Duplicates[i].Title == "" { - rep.Duplicates[i].Title = t - } - } - } - return err -} - -// titlesOf fetches the titles of the given issue or PR numbers in a single -// GraphQL request, one alias per number, rather than one gh call each. -// -// gh exits non-zero when any one number fails to resolve -- a reference to a -// deleted PR, say -- but still prints the partial response, so stdout is read -// whatever the exit status and every title that did resolve is kept. -func titlesOf(numbers []int) (map[int]string, error) { - var q strings.Builder - q.WriteString("query($owner:String!,$name:String!){repository(owner:$owner,name:$name){") - for _, n := range numbers { - fmt.Fprintf(&q, "n%d:issueOrPullRequest(number:%d){...on PullRequest{title} ...on Issue{title}}", n, n) - } - q.WriteString("}}") - - stdout, runErr := exec.Command("gh", "api", "graphql", - "-F", "owner={owner}", "-F", "name={repo}", "-f", "query="+q.String()).Output() - if ee, ok := runErr.(*exec.ExitError); ok && len(ee.Stderr) > 0 { - runErr = fmt.Errorf("gh api graphql: %w: %s", runErr, strings.TrimSpace(string(ee.Stderr))) - } - var resp struct { - Data struct { - Repository map[string]*struct { - Title string `json:"title"` - } `json:"repository"` - } `json:"data"` - } - if err := json.Unmarshal(stdout, &resp); err != nil { - if runErr != nil { - return nil, runErr - } - return nil, err - } - titles := map[int]string{} - for alias, node := range resp.Data.Repository { - if n, err := strconv.Atoi(strings.TrimPrefix(alias, "n")); err == nil && node != nil { - titles[n] = node.Title - } - } - if runErr != nil && len(titles) < len(numbers) { - return titles, fmt.Errorf("%d of %d did not resolve: %w", len(numbers)-len(titles), len(numbers), runErr) - } - return titles, nil -} - -// A merged conflict draft can carry only a prefix of the original commits, -// so a merged backport alone settles only a single-commit original. Partial -// delivery still needs the missing commits or a maintainer's attestation. -func classify(pr *mainPR, label string, hist *branchHistory, linked []backportPR, src commitSource) (verdict, error) { - v := verdict{ - Number: pr.Number, - Title: pr.Title, - URL: pr.URL, - Author: pr.Author.Login, - MergedAt: pr.MergedAt.Format(time.RFC3339), - Label: label, - BackportPRs: []linkedPR{}, - } - for _, b := range linked { - v.BackportPRs = append(v.BackportPRs, trimPR(b)) - } - - mergeCommit := pr.MergeCommit.Oid - if hist.contains(mergeCommit) { - v.Status, v.Evidence = statusInBranch, "merged before branch cut" - return v, nil - } - - var merged, open *backportPR - for i := range linked { - switch b := &linked[i]; { - case b.State == "MERGED" && merged == nil: - merged = b - case b.State == "OPEN" && open == nil: - open = b - } - } - openNote := "" - if open != nil { - openNote = fmt.Sprintf("backport PR #%d open", open.Number) - if open.IsDraft { - openNote += " (draft/conflict)" - } - } - prMerged := "" - switch { - case merged != nil: - prMerged = fmt.Sprintf("backport PR #%d merged", merged.Number) - case hist.hasSubject(fmt.Sprintf("[Backport %s] %s", hist.branch, pr.Title)): - prMerged = "bot merge commit on branch" - } - - commits, err := src.commits(mergeCommit) - if err != nil { - return verdict{}, fmt.Errorf("reading the commits of #%d: %w", pr.Number, err) - } - missing := hist.missingFrom(commits, mergeCommit) - evidence := fmt.Sprintf("%d of %s on branch", len(commits)-len(missing), commitCount(len(commits))) - if prMerged != "" { - evidence = prMerged + ", " + evidence - } - - switch { - case len(commits) > 0 && len(missing) == 0: - v.Status, v.Evidence = statusBackported, evidence - return v, nil - case len(commits) <= 1 && prMerged != "": - v.Status, v.Evidence = statusBackported, prMerged - if len(commits) == 1 { - v.Evidence = evidence - } - return v, nil - case len(missing) < len(commits): // some carried, not all - v.Status, v.Evidence, v.MissingCommits = statusPartial, evidence, missing - case prMerged != "": - v.Status, v.Evidence, v.MissingCommits = statusUnverified, evidence, missing - case open != nil: - v.Status, v.Evidence = statusPending, openNote - return v, nil - default: - for _, b := range linked { - if b.State == "CLOSED" { - // A closed backport is usually a deliberate "not needed on this - // line". Carry the reason someone left, so the next release does - // not re-open the same investigation from scratch. - v.Status = statusDropped - v.Evidence = fmt.Sprintf("backport PR #%d closed unmerged", b.Number) - v.Reason = lastHumanComment(b) - return v, nil - } - } - v.Status, v.Evidence = statusMissing, "no backport PR, nothing on branch" - return v, nil - } - - // Partial or unverified from here on. - if openNote != "" { - v.Evidence += "; " + openNote - } - if c := confirmedBy(linked); c != nil { - v.Status, v.Confirmation = statusConfirmed, c - v.Evidence = fmt.Sprintf("confirmed by @%s on backport PR #%d; %s", c.By, c.BackportPR, v.Evidence) - } - return v, nil -} - -func commitCount(n int) string { - if n == 1 { - return "1 commit" - } - return fmt.Sprintf("%d commits", n) -} - -func (cfg *config) report(branch string, rep *branchReport) { - results := rep.Candidates - byStatus := map[string][]verdict{} - for _, r := range results { - byStatus[r.Status] = append(byStatus[r.Status], r) - } - - var counts []string - for _, s := range statusOrder { - if n := len(byStatus[s]); n > 0 { - counts = append(counts, fmt.Sprintf("%s%s=%d%s", cfg.colorFor(s), s, n, cfg.reset)) - } - } - summary := strings.Join(counts, " ") - if summary == "" { - summary = "no candidates" - } - var others []string - if n := len(rep.Duplicates); n > 0 { - others = append(others, fmt.Sprintf("%sDUPLICATE=%d%s", cfg.red, n, cfg.reset)) - } - if n := len(rep.Unlabelled); n > 0 { - others = append(others, fmt.Sprintf("unlabelled=%d", n)) - } - if len(others) > 0 { - summary += " | " + strings.Join(others, " ") - } - fmt.Printf("\n%s=== %s ===%s %d candidate PRs: %s\n", - cfg.bold, branch, cfg.reset, len(results), summary) - - for _, status := range outstanding { - rows := byStatus[status] - if len(rows) == 0 { - continue - } - accent := cfg.colorFor(status) - fmt.Printf("\n %s%s%s (%d):\n", accent, headings[status], cfg.reset, len(rows)) - for _, r := range rows { - cfg.printVerdict(accent, r) - } - } - - if len(rep.Duplicates) > 0 { - fmt.Printf("\n %sDUPLICATE -- one original, more than one live backport PR%s (%d):\n", - cfg.red, cfg.reset, len(rep.Duplicates)) - for _, d := range rep.Duplicates { - label := d.Label - if label == "" { - label = "none for this line" - } - cfg.printOrigin(cfg.red, d.URL, d.Number, d.Title) - fmt.Printf(" %slabel=%s -- %s%s\n", cfg.dim, label, dupDescriptions[d.Kind], cfg.reset) - cfg.printBackportPRs(d.BackportPRs) - } - } - - if rep.clean() { - fmt.Printf(" %snothing outstanding: every candidate is on this branch%s\n", cfg.green, cfg.reset) - } - - if rows := byStatus[statusConfirmed]; len(rows) > 0 { - fmt.Printf("\n %sCONFIRMED -- attested complete by a person where the commits fall short%s (%d, informational):\n", - cfg.green, cfg.reset, len(rows)) - for _, r := range rows { - cfg.printVerdict(cfg.green, r) - fmt.Printf(" %sconfirmation: %s%s\n", cfg.dim, r.Confirmation.URL, cfg.reset) - } - } - - if len(rep.Unlabelled) > 0 { - fmt.Printf("\n %sUNLABELLED -- backport PRs here for originals not labelled for this line%s (%d, informational):\n", - cfg.bold, cfg.reset, len(rep.Unlabelled)) - for _, u := range rep.Unlabelled { - cfg.printOrigin("", u.URL, u.Number, u.Title) - fmt.Printf(" %s%s%s\n", cfg.dim, claimState(u.BackportPRs, u.Confirmation), cfg.reset) - if len(u.Labels) > 0 { - fmt.Printf(" %slabels=%s, which did not target this line at merge%s\n", - cfg.dim, strings.Join(u.Labels, ","), cfg.reset) - } - cfg.printBackportPRs(u.BackportPRs) - } - } -} - -// claimState says what the backport PRs claiming an original have done on the -// branch, so an open claim does not read as a delivered one. A merged one is -// only a merged PR: an unlabelled original is never checked commit by commit, -// so nothing here says it arrived whole unless a person attested it. -func claimState(prs []linkedPR, conf *confirmation) string { - state := "claimed here, closed unmerged" - for _, b := range prs { - switch b.State { - case "MERGED": - if conf != nil { - return "backport PR merged, confirmed by @" + conf.By - } - return "backport PR merged" - case "OPEN": - state = "claimed here, not merged" - } - } - return state -} - -func (cfg *config) printVerdict(accent string, r verdict) { - fmt.Printf(" %s%s%s\n", accent, r.URL, cfg.reset) - fmt.Printf(" %s#%d%s %s\n", cfg.bold, r.Number, cfg.reset, r.Title) - fmt.Printf(" %slabel=%s author=%s merged=%s -- %s%s\n", - cfg.dim, r.Label, r.Author, r.MergedAt[:10], r.Evidence, cfg.reset) - if r.Reason != "" { - fmt.Printf(" %sreason: %s%s\n", cfg.dim, r.Reason, cfg.reset) - } - for _, c := range r.MissingCommits { - fmt.Printf(" %smissing: %.9s %s%s\n", cfg.dim, c.Oid, c.Subject, cfg.reset) - } -} - -func (cfg *config) printOrigin(accent, url string, number int, title string) { - fmt.Printf(" %s%s%s\n", accent, url, cfg.reset) - fmt.Printf(" %s#%d%s %s\n", cfg.bold, number, cfg.reset, title) -} - -func (cfg *config) printBackportPRs(prs []linkedPR) { - for _, b := range prs { - draft := "" - if b.Draft { - draft = " (draft)" - } - fmt.Printf(" %sbackport #%d %s%s %s%s\n", cfg.dim, b.Number, b.State, draft, b.URL, cfg.reset) - } -} diff --git a/cmd/backport-audit/main_test.go b/cmd/backport-audit/main_test.go deleted file mode 100644 index 7d4009437..000000000 --- a/cmd/backport-audit/main_test.go +++ /dev/null @@ -1,1496 +0,0 @@ -/* -Copyright 2026 The Cozystack Authors. - -Licensed under the Apache License, Version 2.0 (the "License"); -you may not use this file except in compliance with the License. -You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, software -distributed under the License is distributed on an "AS IS" BASIS, -WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -See the License for the specific language governing permissions and -limitations under the License. -*/ - -package main - -import ( - "errors" - "fmt" - "os" - "os/exec" - "slices" - "strings" - "testing" - "time" -) - -func ts(t *testing.T, s string) time.Time { - t.Helper() - when, err := time.Parse(time.RFC3339, s) - if err != nil { - t.Fatalf("bad timestamp %q: %v", s, err) - } - return when -} - -func name(l *line) string { - if l == nil { - return "" - } - return l.branch() -} - -// The real dates, so a change to freezeContractLandedAt or to the era rules -// shows up here rather than on a release cut. -func liveTimeline(t *testing.T) []lineOpen { - t.Helper() - return []lineOpen{ - {line{1, 4}, ts(t, "2026-05-19T13:10:45Z")}, // v1.4.0 published; branch cut earlier, pre-contract - {line{1, 5}, ts(t, "2026-06-22T15:45:38Z")}, // v1.5.0 published - {line{1, 6}, ts(t, "2026-07-22T15:21:48Z")}, // v1.6.0 published - } -} - -func TestTargetsAt(t *testing.T) { - live := liveTimeline(t) - - // A freeze after the cutover opens the line at the rc, not at a stable - // it does not have yet. - frozen17 := append(append([]lineOpen{}, live...), lineOpen{line{1, 7}, ts(t, "2026-09-10T09:00:00Z")}) - - cases := []struct { - name string - opened []lineOpen - merged string - current, previous string - }{ - { - name: "before the cutover, the published-stable rule is in force", - opened: live, merged: "2026-07-01T00:00:00Z", - current: "release-1.5", previous: "release-1.4", - }, - { - name: "after v1.6.0 publishes, still under the old rule", - opened: live, merged: "2026-07-30T00:00:00Z", - current: "release-1.6", previous: "release-1.5", - }, - { - name: "after the cutover with no freeze since, the branch set is unchanged", - opened: live, merged: "2026-09-01T00:00:00Z", - current: "release-1.6", previous: "release-1.5", - }, - { - name: "the instant before a 1.7 freeze still targets 1.6", - opened: frozen17, merged: "2026-09-10T08:59:59Z", - current: "release-1.6", previous: "release-1.5", - }, - { - name: "once release-1.7 exists it is the target, before v1.7.0 ever publishes", - opened: frozen17, merged: "2026-09-10T09:00:01Z", - current: "release-1.7", previous: "release-1.6", - }, - { - name: "only one line exists: no previous", - opened: live[:1], merged: "2026-06-01T00:00:00Z", - current: "release-1.4", previous: "", - }, - { - name: "merged before any line opened: no target at all", - opened: live, merged: "2026-01-01T00:00:00Z", - current: "", previous: "", - }, - } - - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - current, previous := targetsAt(tc.opened, ts(t, tc.merged)) - if got := name(current); got != tc.current { - t.Errorf("current = %s, want %s", got, tc.current) - } - if got := name(previous); got != tc.previous { - t.Errorf("previous = %s, want %s", got, tc.previous) - } - }) - } -} - -// A label is read against the lines at merge time however late it was added: -// the audit has no label timestamp. A PR merged on 2026-07-01 and labelled -// after v1.6.0 published is still a release-1.5 candidate, even though -// backport.yaml, running at label time, would have targeted release-1.6. -func TestCandidateLabel(t *testing.T) { - live := liveTimeline(t) - pr := func(merged string, labels ...string) *mainPR { - p := &mainPR{MergedAt: ts(t, merged), labels: map[string]bool{}} - for _, l := range labels { - p.labels[l] = true - } - return p - } - cases := []struct { - name string - pr *mainPR - want map[line]string - }{ - { - name: "merged in the 1.5 era: kind/backport means release-1.5 for good", - pr: pr("2026-07-01T00:00:00Z", labelCurrent), - want: map[line]string{{1, 4}: "", {1, 5}: labelCurrent, {1, 6}: ""}, - }, - { - name: "merged in the 1.5 era: kind/backport-previous means release-1.4 for good", - pr: pr("2026-07-01T00:00:00Z", labelPrevious), - want: map[line]string{{1, 4}: labelPrevious, {1, 5}: "", {1, 6}: ""}, - }, - { - name: "merged in the 1.6 era, both requests", - pr: pr("2026-09-01T00:00:00Z", labelCurrent, labelPrevious), - want: map[line]string{{1, 4}: "", {1, 5}: labelPrevious, {1, 6}: labelCurrent}, - }, - { - name: "no backport label: a candidate nowhere", - pr: pr("2026-09-01T00:00:00Z"), - want: map[line]string{{1, 4}: "", {1, 5}: "", {1, 6}: ""}, - }, - } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - for want, label := range tc.want { - if got := candidateLabel(tc.pr, live, want); got != label { - t.Errorf("%s: candidateLabel = %q, want %q", want.branch(), got, label) - } - } - }) - } -} - -// backport.yaml sorts its branch list numerically descending, so the rank is by -// version and never by when a line opened. The two disagree as soon as a freeze -// overlaps the previous line's stabilisation, and 1.10-versus-1.9 is where a -// lexicographic sort would disagree with both. -func TestTargetsAtRanksByVersionNotByOpenTime(t *testing.T) { - // v1.7.0-rc.1 is cut while v1.6.0 is still unpublished, so the newer line - // opens first in time. - opened := []lineOpen{ - {line{1, 7}, ts(t, "2026-09-10T09:00:00Z")}, - {line{1, 6}, ts(t, "2026-09-20T09:00:00Z")}, - } - current, previous := targetsAt(opened, ts(t, "2026-09-21T00:00:00Z")) - if got := name(current); got != "release-1.7" { - t.Errorf("current = %s, want release-1.7", got) - } - if got := name(previous); got != "release-1.6" { - t.Errorf("previous = %s, want release-1.6", got) - } - - opened = []lineOpen{ - {line{1, 9}, ts(t, "2026-09-10T09:00:00Z")}, - {line{1, 10}, ts(t, "2026-10-10T09:00:00Z")}, - } - current, previous = targetsAt(opened, ts(t, "2026-10-11T00:00:00Z")) - if got := name(current); got != "release-1.10" { - t.Errorf("current = %s, want release-1.10", got) - } - if got := name(previous); got != "release-1.9" { - t.Errorf("previous = %s, want release-1.9", got) - } -} - -func TestParseFreezeDates(t *testing.T) { - // Tag names and dates as `git for-each-ref` prints them. - const out = `v1.4.0 2026-05-19T18:10:45+05:00 -v1.4.0-rc.1 2026-05-13T19:10:58+05:00 -v1.4.0-rc.2 2026-05-14T10:15:38+05:00 -v1.6.0-rc.1 2026-07-07T17:46:06+05:00 -v1.6.1-rc.1 2026-08-04T18:52:49+05:00 -v1.7.0-rc.2 2026-09-12T12:00:00+05:00 -v1.7.0-rc.1 2026-09-10T14:00:00+05:00 -v1.8.0-beta.1 2026-10-01T12:00:00+05:00 -v1.8.0-alpha.1 2026-10-02T12:00:00+05:00 -` - frozen, err := parseFreezeDates(out) - if err != nil { - t.Fatalf("parseFreezeDates: %v", err) - } - - // 1.4 and 1.6 are pre-cutover: their branches were not created by a freeze, - // so the rc tag says nothing about when they appeared. - for _, l := range []line{{1, 4}, {1, 6}} { - if when, ok := frozen[l]; ok { - t.Errorf("%s: pre-cutover rc must not set a freeze date, got %s", l.branch(), when) - } - } - // A patch-line rc is cut from a branch that already exists, and alpha/beta - // do not freeze at all. - for _, l := range []line{{1, 8}} { - if _, ok := frozen[l]; ok { - t.Errorf("%s: alpha/beta must not freeze a line", l.branch()) - } - } - // 1.7 freezes at rc.1, not at whichever rc the listing happens to print first. - want := ts(t, "2026-09-10T14:00:00+05:00") - got, ok := frozen[line{1, 7}] - if !ok { - t.Fatalf("release-1.7: expected a freeze date") - } - if !got.Equal(want) { - t.Errorf("release-1.7 froze at %s, want %s", got, want) - } - if len(frozen) != 1 { - t.Errorf("expected exactly one frozen line, got %d: %v", len(frozen), frozen) - } -} - -// The label listing goes through GitHub search, which stops at 1000 results -// whatever --limit says, so a limit above that must not let a listing cut at -// 1000 pass for a complete one. -func TestCandidateCeiling(t *testing.T) { - cases := []struct { - limit, got int - truncated bool - }{ - {limit: 400, got: 399, truncated: false}, - {limit: 400, got: 400, truncated: true}, - {limit: 1000, got: 999, truncated: false}, - {limit: 1000, got: 1000, truncated: true}, - {limit: 2000, got: 999, truncated: false}, - {limit: 2000, got: 1000, truncated: true}, - } - for _, tc := range cases { - ceiling, remedy := candidateCeiling(tc.limit) - err := truncated(tc.got, ceiling, "the listing", remedy) - if (err != nil) != tc.truncated { - t.Errorf("--limit %d, %d results: truncated = %v, want %v", tc.limit, tc.got, err != nil, tc.truncated) - } - } -} - -// Bodies are the ones real backport PRs carry, trimmed, so a change to the -// grammar is judged against how people actually write them. -func TestBodyOrigins(t *testing.T) { - cases := []struct { - name string - body string - want []int - }{ - { - name: "the plain form", - body: "Backport of #3742 to `release-1.6`.\n\nCherry-picked clean with `-x`.", - want: []int{3742}, - }, - { - name: "two originals in one phrase, and an issue further on that is not one", - body: "Backport of #3938 and #4280 to `release-1.6`, together, because #3938 alone breaks every Kafka deletion (#4276) and #4280 is what fixes that.", - want: []int{3938, 4280}, - }, - { - name: "a list without the serial comma", - body: "Backport of #1, #2 and #3 to `release-1.6`.", - want: []int{1, 2, 3}, - }, - { - name: "a list with the serial comma", - body: "Backport of #1, #2, and #3.", - want: []int{1, 2, 3}, - }, - { - name: "a list ending the line", - body: "Backport of #1, #2 and #3\n\nAll cherry-picked with `-x`.", - want: []int{1, 2, 3}, - }, - { - name: "a list ending the body", - body: "Backport of #1 and #2", - want: []int{1, 2}, - }, - { - name: "a list running on into a clause about its items keeps only the first", - body: "Backport of #10, #20 is not included.", - want: []int{10}, - }, - { - name: "a list running into a parenthesis keeps only the first", - body: "Backport of #10 and #20 (the second only in part).", - want: []int{10}, - }, - { - name: "a to that does not name a release line does not close a list", - body: "Backport of #10, #20 to follow in a separate PR.", - want: []int{10}, - }, - { - name: "an unquoted release line closes a list, whichever line it names", - body: "Backport of #1 and #2 to release-1.5, both clean.", - want: []int{1, 2}, - }, - { - name: "an unquoted release line before a full stop closes a list", - body: "Backport of #1 and #2 to release-1.5.", - want: []int{1, 2}, - }, - { - name: "an unquoted release line ending the body closes a list", - body: "Backport of #1 and #2 to release-1.5", - want: []int{1, 2}, - }, - { - name: "a branch that only starts with a release line does not close a list", - body: "Backport of #10, #20 to release-1.6-fixes will follow later.", - want: []int{10}, - }, - { - name: "a patch version is not a release line", - body: "Backport of #10, #20 to release-1.6.1 later.", - want: []int{10}, - }, - { - name: "a dotted suffix is not a release line", - body: "Backport of #10, #20 to release-1.6.fixes will follow later.", - want: []int{10}, - }, - { - name: "a full stop and a space after the release line close a list", - body: "Backport of #10, #20 to release-1.6. Next, the tests.", - want: []int{10, 20}, - }, - { - name: "a quoted release line running on into a word does not close a list", - body: "Backport of #10, #20 to `release-1.6`-ish branches.", - want: []int{10}, - }, - { - name: "an ampersand is not a separator", - body: "Backport of #1 & #2.", - want: []int{1}, - }, - { - name: "repository-qualified references, alone and in a list", - body: "Backport of cozystack/cozystack#3471 and #3472 to `release-1.6` (clean cherry-pick).", - want: []int{3471, 3472}, - }, - { - name: "this repository's qualifier matches case-insensitively", - body: "Backport of CozyStack/Cozystack#7.", - want: []int{7}, - }, - { - name: "another repository's reference in a list is skipped", - body: "Backport of #10 and other/repo#20.", - want: []int{10}, - }, - { - name: "another repository's reference alone is skipped", - body: "Backport of cozystack/website#20 to `release-1.6`.", - want: nil, - }, - { - name: "another repository's reference first does not hide the rest of a closed list", - body: "Backport of other/repo#20 and #10.", - want: []int{10}, - }, - { - name: "a dependency pulled along with together-with, as #4328 writes it", - body: "Backport of #4253 to `release-1.6`, together with #3460 which it depends on. On 1.6 kube-ovn #99 differs.", - want: []int{4253, 3460}, - }, - { - name: "the same as #4377 writes it", - body: "Backport of #4333 to `release-1.6`, together with #4231 which it depends on. 1.6 has no in-tree chart.", - want: []int{4333, 4231}, - }, - { - name: "the same as #4421 writes it", - body: "## What this PR does\n\nManual backport of #3938 to `release-1.6`, together with #4280.\n\n#3938 cannot go to 1.6 on its own.", - want: []int{3938, 4280}, - }, - { - name: "together-with and an unquoted branch", - body: "Manual backport of #3938 to release-1.6, together with #4280.", - want: []int{3938, 4280}, - }, - { - name: "a closed together-with list keeps every item", - body: "Backport of #10 to `release-1.6`, together with #20 and #30.", - want: []int{10, 20, 30}, - }, - { - name: "a together-with list running on into a clause keeps only its first", - body: "Backport of #10 to release-1.6, together with #20, #30 is tracked separately.", - want: []int{10, 20}, - }, - { - name: "a together-with list running into a to that is not a release line keeps only its first", - body: "Backport of #10 to `release-1.6`, together with #20, #30 to follow later.", - want: []int{10, 20}, - }, - { - name: "a together-with list running into a branch that only starts with a release line keeps only its first", - body: "Backport of #10 to `release-1.6`, together with #20, #30 to release-1.6-fixes later.", - want: []int{10, 20}, - }, - { - name: "a together-with list running into a patch version keeps only its first", - body: "Backport of #10 to `release-1.6`, together with #20, #30 to release-1.6.1 later.", - want: []int{10, 20}, - }, - { - name: "a together-with list running into a dotted suffix keeps only its first", - body: "Backport of #10 to `release-1.6`, together with #20, #30 to release-1.6.fixes later.", - want: []int{10, 20}, - }, - { - name: "a together-with list closed by a release line, a full stop and a space keeps every item", - body: "Backport of #10 to `release-1.6`, together with #20, #30 to release-1.6. Next, the tests.", - want: []int{10, 20, 30}, - }, - { - name: "a together-with list running into a quoted release line and a word keeps only its first", - body: "Backport of #10 to `release-1.6`, together with #20, #30 to `release-1.6`-ish branches.", - want: []int{10, 20}, - }, - { - name: "a together-with list closed by a quoted release line keeps every item", - body: "Backport of #10 to `release-1.6`, together with #20 and #30 to `release-1.5` as well.", - want: []int{10, 20, 30}, - }, - { - name: "together-with in a later sentence is not part of the phrase", - body: "Backport of #7 to `release-1.6`. Together with #8 it fixes the flake.", - want: []int{7}, - }, - { - name: "a hyphenated prefix still anchors", - body: "Hand-backport of #3034 to `release-1.5`. Fixes #12 and #13.", - want: []int{3034}, - }, - { - name: "a list that turns into prose stops at the prose", - body: "Backport of #1, which fixes #2, and #3.", - want: []int{1}, - }, - { - name: "and followed by prose stops at the prose", - body: "Backport of #1 and the follow-up to #2.", - want: []int{1}, - }, - { - name: "every phrase in the body counts, each number once", - body: "Backport of #10 to `release-1.6`.\n\nThe second commit is a backport of #11 and #10.", - want: []int{10, 11}, - }, - { - name: "an article between the phrase and the number is not a reference", - body: "Hand-redone backport of the **kube-ovn-webhook half** of #3997.", - want: nil, - }, - { - name: "no phrase at all", - body: "Fixes #4276 on this branch.", - want: nil, - }, - } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - if got := bodyOrigins(tc.body, "cozystack/cozystack"); !slices.Equal(got, tc.want) { - t.Errorf("bodyOrigins = %v, want %v", got, tc.want) - } - }) - } -} - -func TestClaimedOrigins(t *testing.T) { - cases := []struct { - name string - pr backportPR - want []int - }{ - { - name: "a hand backport reusing the bot's head naming is counted once", - pr: backportPR{HeadRefName: "backport-3938-to-release-1.6", Body: "Backport of #3938 and #4280 to `release-1.6`."}, - want: []int{3938, 4280}, - }, - { - name: "a head naming another branch says nothing about this one", - pr: backportPR{HeadRefName: "backport-3455-to-release-1.5"}, - want: nil, - }, - { - name: "a head off the bot's pattern leaves only the body", - pr: backportPR{HeadRefName: "backport-3938-release-1.6", Body: "Manual backport of #3938 to `release-1.6`, together with #4280."}, - want: []int{3938, 4280}, - }, - { - name: "qualifiers are read against the repository the PR is in", - pr: backportPR{URL: "https://github.com/example/fork/pull/5", - Body: "Backport of example/fork#3 and cozystack/cozystack#4."}, - want: []int{3}, - }, - } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - if got := claimedOrigins(tc.pr, "release-1.6"); !slices.Equal(got, tc.want) { - t.Errorf("claimedOrigins = %v, want %v", got, tc.want) - } - }) - } -} - -func TestRepoOf(t *testing.T) { - cases := map[string]string{ - "https://github.com/cozystack/cozystack/pull/4431": "cozystack/cozystack", - "https://github.com/example/fork/pull/5": "example/fork", - "": upstreamRepo, - } - for url, want := range cases { - if got := repoOf(url); got != want { - t.Errorf("repoOf(%q) = %q, want %q", url, got, want) - } - } -} - -func bp(n int, state, head, body string) backportPR { - return backportPR{ - Number: n, - State: state, - URL: originURL("https://github.com/cozystack/cozystack/pull/1", n), - HeadRefName: head, - Body: body, - } -} - -func numbersOf[T any](records []T, number func(T) int) []int { - out := []int{} - for _, r := range records { - out = append(out, number(r)) - } - return out -} - -// Modelled on release-1.6: hand backports of main PRs that carry no backport -// label, one of them bundling two originals, next to a closed fork PR that had -// claimed the same pair, and a labelled candidate whose backport merged. -func release16(state4421, state4456 string) []backportPR { - return []backportPR{ - bp(4431, "MERGED", "backport-3455-to-release-1.6", "Backport of #3455 to `release-1.6`."), - bp(4437, "MERGED", "backport-4020-to-release-1.6", "Backport of #4020 to `release-1.6`, without its product changes."), - bp(4456, state4456, "backport-3938-to-release-1.6", - "Backport of #3938 and #4280 to `release-1.6`, together, because #3938 alone breaks every Kafka deletion (#4276)."), - bp(4421, state4421, "backport-3938-release-1.6", - "## What this PR does\n\nManual backport of #3938 to `release-1.6`, together with #4280."), - bp(4474, "MERGED", "backport-4291-to-release-1.6", "Backport of #4291 to `release-1.6`."), - } -} - -func TestCrossCheck(t *testing.T) { - labelled := &mainPR{Number: 4291, Title: "fix(cluster-api): live migration before host eviction", - URL: "https://github.com/cozystack/cozystack/pull/4291", labels: map[string]bool{labelCurrent: true}} - // Labelled, but for another line: it is not audited here, so a backport of - // it on this branch is exactly as unaccounted for as an unlabelled one. - elsewhere := &mainPR{Number: 3455, Title: "fix(e2e): disable guest fsync on ephemeral CI VM disks", - URL: "https://github.com/cozystack/cozystack/pull/3455", labels: map[string]bool{labelPrevious: true}} - cands := map[int]*mainPR{4291: labelled, 3455: elsewhere} - audited := map[int]string{4291: labelCurrent} - - t.Run("merged unlabelled backports are listed, the candidate is not", func(t *testing.T) { - index := indexByOrigin(release16("CLOSED", "MERGED"), "release-1.6") - unlabelled, duplicates := crossCheck(index, audited, cands) - - got := numbersOf(unlabelled, func(u unlabelledBackport) int { return u.Number }) - if want := []int{3455, 3938, 4020, 4280}; !slices.Equal(got, want) { - t.Fatalf("unlabelled originals = %v, want %v", got, want) - } - if len(duplicates) != 0 { - t.Errorf("a closed PR next to a merged one is not a duplicate, got %+v", duplicates) - } - - byNumber := map[int]unlabelledBackport{} - for _, u := range unlabelled { - byNumber[u.Number] = u - } - if u := byNumber[4280]; u.URL != "https://github.com/cozystack/cozystack/pull/4280" || u.Title != "" || len(u.Labels) != 0 { - t.Errorf("#4280: want a derived URL, no title yet and no labels, got %+v", u) - } - if u := byNumber[3455]; u.Title != elsewhere.Title || !slices.Equal(u.Labels, []string{labelPrevious}) { - t.Errorf("#3455: want the candidate's title and its label, got %+v", u) - } - if got := numbersOf(byNumber[3938].BackportPRs, func(l linkedPR) int { return l.Number }); !slices.Equal(got, []int{4421, 4456}) { - t.Errorf("#3938: backport PRs = %v, want both claims sorted", got) - } - }) - - t.Run("two open claims on the same pair are both duplicates", func(t *testing.T) { - index := indexByOrigin(release16("OPEN", "OPEN"), "release-1.6") - _, duplicates := crossCheck(index, audited, cands) - got := numbersOf(duplicates, func(d duplicateBackport) int { return d.Number }) - if want := []int{3938, 4280}; !slices.Equal(got, want) { - t.Fatalf("duplicate originals = %v, want %v", got, want) - } - for _, d := range duplicates { - if d.Kind != dupOpenTwice || d.Label != "" { - t.Errorf("#%d: want kind %s and no label, got %+v", d.Number, dupOpenTwice, d) - } - } - }) - - t.Run("an open claim after a merged one is a leftover", func(t *testing.T) { - index := indexByOrigin(release16("OPEN", "MERGED"), "release-1.6") - _, duplicates := crossCheck(index, audited, cands) - got := numbersOf(duplicates, func(d duplicateBackport) int { return d.Number }) - if want := []int{3938, 4280}; !slices.Equal(got, want) { - t.Fatalf("duplicate originals = %v, want %v", got, want) - } - for _, d := range duplicates { - if d.Kind != dupOpenAfterMerge { - t.Errorf("#%d: want kind %s, got %s", d.Number, dupOpenAfterMerge, d.Kind) - } - } - }) - - t.Run("a labelled candidate claimed twice is a duplicate carrying its label", func(t *testing.T) { - prs := append(release16("CLOSED", "MERGED"), - bp(4480, "OPEN", "fix-migration-again", "Backport of #4291 to `release-1.6`.")) - unlabelled, duplicates := crossCheck(indexByOrigin(prs, "release-1.6"), audited, cands) - if len(duplicates) != 1 || duplicates[0].Number != 4291 || duplicates[0].Label != labelCurrent || - duplicates[0].Kind != dupOpenAfterMerge || duplicates[0].Title != labelled.Title { - t.Fatalf("want one open-after-merge duplicate for #4291 under %s, got %+v", labelCurrent, duplicates) - } - for _, u := range unlabelled { - if u.Number == 4291 { - t.Errorf("an audited candidate must not also be listed as unlabelled") - } - } - }) -} - -func TestDuplicateKind(t *testing.T) { - cases := []struct { - states []string - want string - }{ - {[]string{"OPEN"}, ""}, - {[]string{"MERGED"}, ""}, - {[]string{"CLOSED", "OPEN"}, ""}, - {[]string{"MERGED", "MERGED"}, ""}, - {[]string{"CLOSED", "CLOSED", "MERGED"}, ""}, - {[]string{"OPEN", "OPEN"}, dupOpenTwice}, - {[]string{"OPEN", "CLOSED", "OPEN"}, dupOpenTwice}, - {[]string{"MERGED", "OPEN"}, dupOpenAfterMerge}, - {[]string{"OPEN", "MERGED", "OPEN"}, dupOpenAfterMerge}, - } - for _, tc := range cases { - var claims []backportPR - for i, s := range tc.states { - claims = append(claims, backportPR{Number: i + 1, State: s}) - } - if got := duplicateKind(claims); got != tc.want { - t.Errorf("duplicateKind(%v) = %q, want %q", tc.states, got, tc.want) - } - } -} - -// The gate answers whether everything labelled landed. Unlabelled backports -// can only add to a branch, so they never fail it; a duplicate needs a human -// before the cut, so it always does. -func TestBranchReportClean(t *testing.T) { - landed := []verdict{{Number: 1, Status: statusBackported}, {Number: 2, Status: statusInBranch}} - unlabelled := []unlabelledBackport{{Number: 3455, BackportPRs: []linkedPR{{Number: 4431, State: "OPEN"}}}} - duplicate := []duplicateBackport{{Number: 3938, Kind: dupOpenTwice}} - - cases := []struct { - name string - rep branchReport - want bool - }{ - {"nothing at all", branchReport{}, true}, - {"everything landed", branchReport{Candidates: landed}, true}, - {"only unlabelled backports, even an open one", branchReport{Candidates: landed, Unlabelled: unlabelled}, true}, - {"a duplicate", branchReport{Candidates: landed, Duplicates: duplicate}, false}, - {"a missing candidate", branchReport{Candidates: append(landed, verdict{Number: 3, Status: statusMissing})}, false}, - {"a pending candidate", branchReport{Candidates: append(landed, verdict{Number: 3, Status: statusPending})}, false}, - {"a dropped candidate", branchReport{Candidates: append(landed, verdict{Number: 3, Status: statusDropped})}, false}, - } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - if got := tc.rep.clean(); got != tc.want { - t.Errorf("clean() = %v, want %v", got, tc.want) - } - }) - } -} - -func TestOriginURL(t *testing.T) { - if got := originURL("https://github.com/cozystack/cozystack/pull/4431", 3455); got != "https://github.com/cozystack/cozystack/pull/3455" { - t.Errorf("originURL = %q", got) - } - if got := originURL("not a PR URL", 3455); got != "" { - t.Errorf("originURL of a non-PR URL = %q, want empty", got) - } -} - -// A real listing, of the PR merged in f1f383626: its third commit is an empty -// "re-trigger CI" commit, which the backport bot drops. -const prLogWithEmptyCommit = "\x00fea40d6314424a4105d31c7ee77f818062f5bc1a\x01fix(ci): only overlay current-main images on main-based PRs\n\n" + - ".github/workflows/pull-requests.yaml\nhack/overlay-main-images_test.bats\n" + - "\x002330d6f3ca42306da92190b5e9dddf0bc1054fcd\x01fix(ci): overlay images from the PR base branch, and publish per-line artifacts\n\n" + - ".github/workflows/build-release.yaml\n.github/workflows/pull-requests.yaml\nhack/overlay-main-images_test.bats\n" + - "\x00007d0b1a22ec1aadca95960645d8232a5b202fcb\x01chore(ci): re-trigger CI after a label event produced a no-op run\n" - -func TestParsePRCommits(t *testing.T) { - got := parsePRCommits(prLogWithEmptyCommit) - want := []prCommit{ - {"fea40d6314424a4105d31c7ee77f818062f5bc1a", "fix(ci): only overlay current-main images on main-based PRs"}, - {"2330d6f3ca42306da92190b5e9dddf0bc1054fcd", "fix(ci): overlay images from the PR base branch, and publish per-line artifacts"}, - } - if !slices.Equal(got, want) { - t.Errorf("parsePRCommits = %+v, want %+v", got, want) - } -} - -// fakeSource serves a PR's commits from a fixture, or fails the read. -type fakeSource struct { - commitsOf func(string) []prCommit - err error -} - -func (f fakeSource) commits(mergeCommit string) ([]prCommit, error) { - if f.err != nil { - return nil, f.err - } - return f.commitsOf(mergeCommit), nil -} - -// branchCommit is one commit on a release branch in a test's history. -type branchCommit struct { - oid, subject string - picks []string -} - -// branchOf is release-1.6 holding exactly these commits. -func branchOf(cs ...branchCommit) *branchHistory { - h := &branchHistory{branch: "release-1.6", ref: "origin/release-1.6", - commits: map[string]bool{}, subjects: map[string][]string{}, refs: map[string][]string{}} - for _, c := range cs { - h.commits[c.oid] = true - h.subjects[c.subject] = append(h.subjects[c.subject], c.oid) - if len(c.picks) > 0 { - h.refs[c.oid] = c.picks - } - } - return h -} - -// history is a release branch with one commit per given subject, one per -x -// reference, and the given commits themselves. -func history(subjects, picked, reachable []string) *branchHistory { - var cs []branchCommit - for i, s := range subjects { - cs = append(cs, branchCommit{oid: fmt.Sprintf("s%039d", i), subject: s}) - } - for i, p := range picked { - cs = append(cs, branchCommit{oid: fmt.Sprintf("p%039d", i), subject: "cherry-pick " + p, picks: []string{p}}) - } - for _, r := range reachable { - cs = append(cs, branchCommit{oid: r, subject: "reachable " + r}) - } - return branchOf(cs...) -} - -func comment(login, association, body, url string) prComment { - c := prComment{AuthorAssociation: association, Body: body, URL: url} - c.Author.Login = login - return c -} - -const ( - firstHalf = "1111111111111111111111111111111111111111" - secondHalf = "2222222222222222222222222222222222222222" - emptyOne = "3333333333333333333333333333333333333333" - squashOne = "4444444444444444444444444444444444444444" - mergeOf = "9999999999999999999999999999999999999999" -) - -// A two-commit PR, #4399, plus an empty commit that must never count. -func twoHalves(string) []prCommit { - return parsePRCommits("\x00" + firstHalf + "\x01fix(x): first half\n\nx.go\n" + - "\x00" + secondHalf + "\x01fix(x): second half\n\ny.go\n" + - "\x00" + emptyOne + "\x01chore(ci): re-trigger CI\n") -} - -func pr4399() *mainPR { - pr := &mainPR{Number: 4399, Title: "fix(x): two halves"} - pr.MergeCommit.Oid = mergeOf - return pr -} - -// The bot's conflict drafts stop at the first commit that does not apply and -// drop the rest, so one merged draft reads as a finished backport to anything -// that looks at the PR, or at any single commit, rather than at every commit. -func TestClassifyJudgesEveryCommit(t *testing.T) { - oneCommit := func(string) []prCommit { - return []prCommit{{squashOne, "fix(y): the whole change (#4398)"}} - } - onlyEmpty := func(string) []prCommit { - return parsePRCommits("\x00" + emptyOne + "\x01chore(ci): re-trigger CI\n") - } - bpMerged := backportPR{Number: 4400, State: "MERGED"} - bpOpen := backportPR{Number: 4401, State: "OPEN", IsDraft: true} - botSubject := "[Backport release-1.6] fix(x): two halves" - - cases := []struct { - name string - hist *branchHistory - linked []backportPR - commitsOf func(string) []prCommit - status string - evidence string - missing []string - }{ - { - name: "only the first commit carried, by -x reference: partial", - hist: history(nil, []string{firstHalf}, nil), - commitsOf: twoHalves, - status: statusPartial, evidence: "1 of 2 commits on branch", missing: []string{secondHalf}, - }, - { - name: "both carried, one by -x reference and one by subject; the empty commit is not needed", - hist: history([]string{"fix(x): second half"}, []string{firstHalf}, nil), - commitsOf: twoHalves, - status: statusBackported, evidence: "2 of 2 commits on branch", - }, - { - name: "reworded on the branch, so only an abbreviated -x reference proves it, and the other is reachable", - hist: history(nil, []string{firstHalf[:7]}, []string{secondHalf}), - commitsOf: twoHalves, - status: statusBackported, evidence: "2 of 2 commits on branch", - }, - { - name: "a merged backport PR does not make a partial backport whole", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{bpMerged}, - commitsOf: twoHalves, - status: statusPartial, evidence: "backport PR #4400 merged, 1 of 2 commits on branch", missing: []string{secondHalf}, - }, - { - name: "partial with the rest still open is partial, and says so", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{bpOpen}, - commitsOf: twoHalves, - status: statusPartial, evidence: "1 of 2 commits on branch; backport PR #4401 open (draft/conflict)", missing: []string{secondHalf}, - }, - { - name: "a merged backport PR with none of several commits found proves only that something merged: unverified", - hist: history(nil, nil, nil), - linked: []backportPR{bpMerged}, - commitsOf: twoHalves, - status: statusUnverified, evidence: "backport PR #4400 merged, 0 of 2 commits on branch", missing: []string{firstHalf, secondHalf}, - }, - { - name: "the bot's merge subject alone is no better for several commits", - hist: history([]string{botSubject}, nil, nil), - commitsOf: twoHalves, - status: statusUnverified, evidence: "bot merge commit on branch, 0 of 2 commits on branch", missing: []string{firstHalf, secondHalf}, - }, - { - name: "an -x reference to the merge commit credits no commit by itself", - hist: history(nil, []string{mergeOf[:12]}, nil), - linked: []backportPR{bpMerged}, - commitsOf: twoHalves, - status: statusUnverified, evidence: "backport PR #4400 merged, 0 of 2 commits on branch", missing: []string{firstHalf, secondHalf}, - }, - { - name: "nor does it lift a partial one", - hist: history(nil, []string{mergeOf[:12], firstHalf}, nil), - commitsOf: twoHalves, - status: statusPartial, evidence: "1 of 2 commits on branch", missing: []string{secondHalf}, - }, - { - name: "no commit found and only an open backport PR: pending", - hist: history(nil, nil, nil), - linked: []backportPR{bpOpen}, - commitsOf: twoHalves, - status: statusPending, evidence: "backport PR #4401 open (draft/conflict)", - }, - { - name: "a squashed PR's one commit carried: backported", - hist: history([]string{"fix(y): the whole change (#4398)"}, nil, nil), - commitsOf: oneCommit, - status: statusBackported, evidence: "1 of 1 commit on branch", - }, - { - name: "a squashed PR's commit not found but its backport PR merged: backported", - hist: history(nil, nil, nil), - linked: []backportPR{bpMerged}, - commitsOf: oneCommit, - status: statusBackported, evidence: "backport PR #4400 merged, 0 of 1 commit on branch", - }, - { - name: "and the bot's merge subject settles it the same way", - hist: history([]string{botSubject}, nil, nil), - commitsOf: oneCommit, - status: statusBackported, evidence: "bot merge commit on branch, 0 of 1 commit on branch", - }, - { - name: "nothing a backport has to carry: the merged backport PR decides", - hist: history(nil, nil, nil), - linked: []backportPR{bpMerged}, - commitsOf: onlyEmpty, - status: statusBackported, evidence: "backport PR #4400 merged", - }, - { - name: "nothing anywhere: MISSING", - hist: history(nil, nil, nil), - commitsOf: twoHalves, - status: statusMissing, evidence: "no backport PR, nothing on branch", - }, - { - name: "merged before the cut: in-branch, without reading its commits", - hist: history(nil, nil, []string{mergeOf}), - commitsOf: func(string) []prCommit { - t.Fatalf("commits read for a PR whose merge commit is on the branch") - return nil - }, - status: statusInBranch, evidence: "merged before branch cut", - }, - } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - v, err := classify(pr4399(), labelCurrent, tc.hist, tc.linked, fakeSource{commitsOf: tc.commitsOf}) - if err != nil { - t.Fatalf("classify: %v", err) - } - if v.Status != tc.status || v.Evidence != tc.evidence { - t.Errorf("got %s %q, want %s %q", v.Status, v.Evidence, tc.status, tc.evidence) - } - var missing []string - for _, c := range v.MissingCommits { - missing = append(missing, c.Oid) - } - if !slices.Equal(missing, tc.missing) { - t.Errorf("missing commits = %v, want %v", missing, tc.missing) - } - // Partial and unverified are outstanding, like MISSING, so they fail - // the gate. - rep := branchReport{Candidates: []verdict{v}} - if want := v.Status == statusBackported || v.Status == statusInBranch; rep.clean() != want { - t.Errorf("clean() = %v for a %s verdict, want %v", rep.clean(), v.Status, want) - } - }) - } -} - -// Only a person's explicit word, on the backport PR that merged, lifts partial -// or unverified; nothing is inferred, and nothing else will do. -func TestClassifyConfirmation(t *testing.T) { - const marker = "backport-audit: complete" - const url = "https://github.com/cozystack/cozystack/pull/4400#issuecomment-1" - onPR := func(state string, comments ...prComment) backportPR { - return backportPR{Number: 4400, State: state, Comments: comments} - } - cases := []struct { - name string - hist *branchHistory - linked []backportPR - status string - by string - }{ - { - name: "partial, attested by a person on the merged backport PR: confirmed", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("coderabbit-reviewer", "NONE", "looks fine", ""), comment("alice", "MEMBER", marker, url))}, - status: statusConfirmed, by: "alice", - }, - { - name: "unverified, attested: confirmed", - hist: history(nil, nil, nil), - linked: []backportPR{onPR("MERGED", comment("alice", "MEMBER", marker, url))}, - status: statusConfirmed, by: "alice", - }, - { - name: "the marker from the backport bot is ignored", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("github-actions", "MEMBER", marker, url))}, - status: statusPartial, - }, - { - name: "also under the bot's [bot] login", - hist: history(nil, nil, nil), - linked: []backportPR{onPR("MERGED", comment("github-actions[bot]", "MEMBER", marker, url))}, - status: statusUnverified, - }, - { - name: "an owner's attestation confirms", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("alice", "OWNER", marker, url))}, - status: statusConfirmed, by: "alice", - }, - { - name: "a collaborator's attestation confirms", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("alice", "COLLABORATOR", marker, url))}, - status: statusConfirmed, by: "alice", - }, - { - name: "a contributor's marker is ignored", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("alice", "CONTRIBUTOR", marker, url))}, - status: statusPartial, - }, - { - name: "a first-time contributor's marker is ignored", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("alice", "FIRST_TIME_CONTRIBUTOR", marker, url))}, - status: statusPartial, - }, - { - name: "a marker from someone with no association is ignored", - hist: history(nil, nil, nil), - linked: []backportPR{onPR("MERGED", comment("alice", "NONE", marker, url))}, - status: statusUnverified, - }, - { - name: "a review bot is ignored even as a member with a well-formed marker", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("coderabbitai", "MEMBER", marker, url))}, - status: statusPartial, - }, - { - name: "the marker with an explanation above it attests nothing", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("alice", "MEMBER", "Squashed by the contributor; both commits are in.\n\n"+marker, url))}, - status: statusPartial, - }, - { - name: "a member who only mentions the marker in a sentence attests nothing", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("alice", "MEMBER", "Do not post backport-audit: complete yet; the second commit is missing.", url))}, - status: statusPartial, - }, - { - name: "a maintainer's attestation counts even after a contributor's", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", - comment("bob", "CONTRIBUTOR", marker, "https://example.invalid/bob"), comment("alice", "MEMBER", marker, url))}, - status: statusConfirmed, by: "alice", - }, - { - name: "the marker on an open backport PR is ignored", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("OPEN", comment("alice", "MEMBER", marker, url))}, - status: statusPartial, - }, - { - name: "the marker on a backport PR closed unmerged is ignored, even next to a merged one", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED"), {Number: 4402, State: "CLOSED", Comments: []prComment{comment("alice", "MEMBER", marker, url)}}}, - status: statusPartial, - }, - { - name: "anything short of the literal marker is not an attestation", - hist: history(nil, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("alice", "MEMBER", "backport-audit complete, I think", url))}, - status: statusPartial, - }, - { - name: "complete commit evidence needs no attestation", - hist: history([]string{"fix(x): second half"}, []string{firstHalf}, nil), - linked: []backportPR{onPR("MERGED", comment("alice", "MEMBER", marker, url))}, - status: statusBackported, - }, - } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - v, err := classify(pr4399(), labelCurrent, tc.hist, tc.linked, fakeSource{commitsOf: twoHalves}) - if err != nil { - t.Fatalf("classify: %v", err) - } - if v.Status != tc.status { - t.Fatalf("status = %s (%s), want %s", v.Status, v.Evidence, tc.status) - } - if tc.by == "" { - if v.Confirmation != nil { - t.Errorf("unexpected confirmation %+v", v.Confirmation) - } - return - } - want := confirmation{By: tc.by, URL: url, BackportPR: 4400} - if v.Confirmation == nil || *v.Confirmation != want { - t.Errorf("confirmation = %+v, want %+v", v.Confirmation, want) - } - if !strings.HasPrefix(v.Evidence, "confirmed by @"+tc.by+" on backport PR #4400; ") { - t.Errorf("evidence %q does not name who confirmed it", v.Evidence) - } - if rep := (branchReport{Candidates: []verdict{v}}); !rep.clean() { - t.Errorf("a confirmed candidate must not fail the gate") - } - if len(v.MissingCommits) == 0 { - t.Errorf("a confirmed candidate keeps the commits nothing on the branch names") - } - }) - } -} - -// A read of the PR's commits that fails must end the audit, not feed a verdict: -// with a merged backport PR linked, falling back to it would say backported on -// the strength of nothing. -func TestClassifyFailedReadIsAnError(t *testing.T) { - linked := []backportPR{{Number: 4400, State: "MERGED"}} - _, err := classify(pr4399(), labelCurrent, history(nil, nil, nil), linked, - fakeSource{err: errors.New("fatal: bad object 9999999")}) - if err == nil || !strings.Contains(err.Error(), "bad object") || !strings.Contains(err.Error(), "#4399") { - t.Fatalf("err = %v, want the failed read of #4399", err) - } -} - -func TestGitSourceCommits(t *testing.T) { - const m = mergeOf - boom := errors.New("fatal: bad object") - cases := []struct { - name string - merge string - parents string - parentErr error - logErr error - wantRange string - wantN int - wantErr bool - }{ - {name: "a merge commit: its second parent's side", merge: m, parents: m + " a b\n", wantRange: m + "^1.." + m + "^2", wantN: 2}, - {name: "a squashed PR: the commit itself", merge: m, parents: m + " a\n", wantRange: m + "^.." + m, wantN: 2}, - {name: "the merge commit is not in the repository", merge: m, parentErr: boom, wantErr: true}, - {name: "the log fails", merge: m, parents: m + " a b\n", logErr: boom, wantRange: m + "^1.." + m + "^2", wantErr: true}, - {name: "an octopus merge is not a PR merge", merge: m, parents: m + " a b c\n", wantErr: true}, - {name: "no merge commit recorded", merge: "", wantErr: true}, - } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - gotRange := "" - src := gitSource{run: func(args ...string) (string, error) { - switch args[0] { - case "rev-list": - return tc.parents, tc.parentErr - case "log": - gotRange = args[len(args)-1] - return prLogWithEmptyCommit, tc.logErr - } - return "", fmt.Errorf("unexpected git %v", args) - }} - commits, err := src.commits(tc.merge) - if (err != nil) != tc.wantErr { - t.Fatalf("err = %v, want error %v", err, tc.wantErr) - } - if gotRange != tc.wantRange { - t.Errorf("log range = %q, want %q", gotRange, tc.wantRange) - } - if err != nil && commits != nil { - t.Errorf("a failed read must not return commits, got %v", commits) - } - if !tc.wantErr && len(commits) != tc.wantN { - t.Errorf("got %d commits, want %d", len(commits), tc.wantN) - } - }) - } -} - -// An unlabelled original is never checked commit by commit, so the only thing -// that can make its merged backport read as confirmed is the same attestation. -func TestCrossCheckCarriesConfirmation(t *testing.T) { - attested := bp(4456, "MERGED", "backport-3938-to-release-1.6", "Backport of #3938 to `release-1.6`.") - attested.Comments = []prComment{comment("alice", "MEMBER", "backport-audit: complete", "https://example.invalid/c/1")} - plain := bp(4431, "MERGED", "backport-3455-to-release-1.6", "Backport of #3455 to `release-1.6`.") - unlabelled, _ := crossCheck(indexByOrigin([]backportPR{attested, plain}, "release-1.6"), nil, nil) - got := map[int]*confirmation{} - for _, u := range unlabelled { - got[u.Number] = u.Confirmation - } - if c := got[3938]; c == nil || c.By != "alice" || c.BackportPR != 4456 { - t.Errorf("#3938: confirmation = %+v, want alice on #4456", c) - } - if c := got[3455]; c != nil { - t.Errorf("#3455: confirmation = %+v, want none", c) - } -} - -func TestSameCommit(t *testing.T) { - full := "0123456789abcdef0123456789abcdef01234567" - cases := []struct { - name string - a, b string - want bool - }{ - {"identical", full, full, true}, - {"the reference abbreviates", full, full[:7], true}, - {"the other side abbreviates", full[:7], full, true}, - {"different commits", full, "fff1234000000000000000000000000000000000", false}, - {"an empty name is nobody's commit", full, "", false}, - {"nor is it on the other side", "", full, false}, - } - for _, tc := range cases { - if got := sameCommit(tc.a, tc.b); got != tc.want { - t.Errorf("%s: sameCommit(%q, %q) = %v, want %v", tc.name, tc.a, tc.b, got, tc.want) - } - } -} - -func TestBranchHistoryEvidence(t *testing.T) { - for _, evidence := range []string{"subject", "cherry-pick reference"} { - t.Run(evidence, func(t *testing.T) { - t.Chdir(t.TempDir()) - // Git hooks can export paths into the caller's repository. - for _, key := range []string{"GIT_DIR", "GIT_WORK_TREE", "GIT_INDEX_FILE", "GIT_COMMON_DIR", "GIT_OBJECT_DIRECTORY", "GIT_ALTERNATE_OBJECT_DIRECTORIES"} { - t.Setenv(key, "") - if err := os.Unsetenv(key); err != nil { - t.Fatal(err) - } - } - t.Setenv("GIT_CONFIG_GLOBAL", os.DevNull) - t.Setenv("GIT_CONFIG_NOSYSTEM", "1") - run := func(args ...string) string { - t.Helper() - args = append([]string{"-c", "user.name=Audit Test", "-c", "user.email=audit@example.invalid", "-c", "commit.gpgsign=false"}, args...) - out, err := exec.Command("git", args...).CombinedOutput() - if err != nil { - t.Fatalf("git %v: %v\n%s", args, err, out) - } - return strings.TrimSpace(string(out)) - } - commit := func(file, message string) string { - t.Helper() - if err := os.WriteFile(file, []byte(file+"\n"), 0600); err != nil { - t.Fatal(err) - } - run("add", file) - run("commit", "-m", message) - return run("rev-parse", "HEAD") - } - run("init", "-b", "main") - commit("seed", "chore: seed") - run("switch", "-c", "feature") - first := commit("first", "feat(x): first half") - second := commit("second", "fix tests") - run("switch", "main") - message := "fix tests" - if evidence == "cherry-pick reference" { - message = "chore: unrelated change\n\n(cherry picked from commit " + second + ")" - } - shared := commit("shared", message) - run("branch", "release-1.6") - run("merge", "--no-ff", "feature", "-m", "feat(x): two halves") - merge := run("rev-parse", "HEAD") - run("update-ref", "refs/remotes/origin/main", merge) - run("switch", "release-1.6") - run("cherry-pick", "-x", first) - run("commit", "--amend", "-m", "fix(x): adapted first half\n\n(cherry picked from commit "+first+")") - run("update-ref", "refs/remotes/origin/release-1.6", "HEAD") - - pr := &mainPR{Number: 4399, Title: "feat(x): two halves"} - pr.MergeCommit.Oid = merge - h, err := newBranchHistory("origin", "release-1.6") - if err != nil { - t.Fatal(err) - } - commits := []prCommit{{first, "feat(x): first half"}, {second, "fix tests"}} - if got := h.missingFrom(commits, merge); !slices.Equal(got, commits[1:]) { - t.Errorf("missingFrom = %v, want only the second commit", got) - } - v, err := classify(pr, labelCurrent, h, nil, gitSource{run: git}) - if err != nil { - t.Fatal(err) - } - if v.Status != statusPartial || (&branchReport{Candidates: []verdict{v}}).clean() { - t.Errorf("incomplete backport = %s (%s), want partial and a failing gate", v.Status, v.Evidence) - } - sharedPR := &mainPR{} - sharedPR.MergeCommit.Oid = shared - v, err = classify(sharedPR, labelCurrent, h, nil, gitSource{run: git}) - if err != nil || v.Status != statusInBranch { - t.Errorf("shared history = %s, %v; want in-branch", v.Status, err) - } - - // A branch-only copy without -x must still supply subject evidence. - run("cherry-pick", second) - run("update-ref", "refs/remotes/origin/release-1.6", "HEAD") - h, err = newBranchHistory("origin", "release-1.6") - if err != nil { - t.Fatal(err) - } - v, err = classify(pr, labelCurrent, h, nil, gitSource{run: git}) - if err != nil || v.Status != statusBackported || !(&branchReport{Candidates: []verdict{v}}).clean() { - t.Errorf("complete backport = %s, %v; want backported and a passing gate", v.Status, err) - } - run("update-ref", "-d", "refs/remotes/origin/main") - if _, err := newBranchHistory("origin", "release-1.6"); err == nil { - t.Error("missing main ref must fail the audit") - } - }) - } -} - -// One branch commit cannot supply subject evidence for two original commits. -func TestMissingFrom(t *testing.T) { - const ( - a = "aaaa000000000000000000000000000000000000" - b = "bbbb000000000000000000000000000000000000" - merge = "eeee000000000000000000000000000000000000" - other = "ffff000000000000000000000000000000000000" - ) - pr := []prCommit{{a, "fix tests"}, {b, "fix tests"}} - cases := []struct { - name string - hist *branchHistory - missing []string - }{ - { - name: "one copy of a shared subject carries one commit, not both", - hist: branchOf(branchCommit{oid: "c1", subject: "fix tests"}), - missing: []string{b}, - }, - { - name: "two copies carry both", - hist: branchOf(branchCommit{oid: "c1", subject: "fix tests"}, branchCommit{oid: "c2", subject: "fix tests"}), - missing: nil, - }, - { - name: "a copy that names the first commit with -x is spent on it, not on the second", - hist: branchOf(branchCommit{oid: "c1", subject: "fix tests", picks: []string{a[:9]}}), - missing: []string{b}, - }, - { - name: "the first commit on the branch itself is spent on it, not on the second", - hist: branchOf(branchCommit{oid: a, subject: "fix tests"}), - missing: []string{b}, - }, - { - name: "a copy whose -x names the PR's merge commit can still carry one commit by subject", - hist: branchOf(branchCommit{oid: "c1", subject: "fix tests", picks: []string{merge[:9]}}), - missing: []string{b}, - }, - { - name: "a copy whose -x names some other commit carries nothing here by its subject", - hist: branchOf(branchCommit{oid: "c1", subject: "fix tests", picks: []string{other}}), - missing: []string{a, b}, - }, - { - name: "an -x reference carries its commit whatever the subject", - hist: branchOf(branchCommit{oid: "c1", subject: "reworded", picks: []string{a}}, branchCommit{oid: "c2", subject: "fix tests"}), - missing: nil, - }, - } - for _, tc := range cases { - t.Run(tc.name, func(t *testing.T) { - var got []string - for _, c := range tc.hist.missingFrom(pr, merge) { - got = append(got, c.Oid) - } - if !slices.Equal(got, tc.missing) { - t.Errorf("missing = %v, want %v", got, tc.missing) - } - }) - } -} - -// A shared subject with only one copy on the branch leaves the candidate -// partial, so the gate stays shut. -func TestClassifySharedSubject(t *testing.T) { - pr := pr4399() - commits := func(string) []prCommit { - return []prCommit{{firstHalf, "fix tests"}, {secondHalf, "fix tests"}} - } - v, err := classify(pr, labelCurrent, history([]string{"fix tests"}, nil, nil), nil, fakeSource{commitsOf: commits}) - if err != nil { - t.Fatal(err) - } - if v.Status != statusPartial || v.Evidence != "1 of 2 commits on branch" { - t.Errorf("got %s %q, want partial 1 of 2", v.Status, v.Evidence) - } - if (&branchReport{Candidates: []verdict{v}}).clean() { - t.Errorf("a partial candidate must fail the gate") - } -} - -// Quoted markers, code blocks and surrounding prose do not attest completeness. -func TestMarkerComment(t *testing.T) { - cases := []struct { - name string - body string - want bool - }{ - {"the bare marker", "backport-audit: complete", true}, - {"with blank lines before it and whitespace after it", "\n\nbackport-audit: complete \t\n\n", true}, - {"with CRLF around it", "\r\nbackport-audit: complete\r\n", true}, - {"a sentence that says not to", "Do not post backport-audit: complete yet; the second commit is missing.", false}, - {"mid-sentence", "I think backport-audit: complete applies here.", false}, - {"with words after it", "backport-audit: complete, except the docs", false}, - {"in another case", "Backport-Audit: Complete", false}, - {"quoted", "> backport-audit: complete", false}, - {"quoted, then a reply", "> backport-audit: complete\n\nAgreed?", false}, - {"inside a backtick fence", "```\nbackport-audit: complete\n```", false}, - {"inside a tilde fence", "~~~\nbackport-audit: complete\n~~~", false}, - {"inside nested tilde fences", "~~~~\n~~~\nbackport-audit: complete\n~~~\n~~~~", false}, - {"as an indented code block", "Post this once checked:\n\n backport-audit: complete", false}, - {"indented by a space and a tab after a blank line", "\n\n \tbackport-audit: complete", false}, - {"indented by a tab", "\tbackport-audit: complete", false}, - {"indented by two spaces", " backport-audit: complete", false}, - {"after a line of spaces", " \nbackport-audit: complete", false}, - {"with text glued in front", "xbackport-audit: complete", false}, - {"with an explanation above it", "Checked every commit by hand.\n\nbackport-audit: complete", false}, - {"with a note below it", "backport-audit: complete\n\nthanks!", false}, - {"twice", "backport-audit: complete\nbackport-audit: complete", false}, - {"empty", "", false}, - } - for _, tc := range cases { - if got := markerComment(tc.body); got != tc.want { - t.Errorf("%s: markerComment(%q) = %v, want %v", tc.name, tc.body, got, tc.want) - } - } -} - -func TestBotAuthor(t *testing.T) { - for login, want := range map[string]bool{ - "github-actions": true, - "github-actions[bot]": true, - "coderabbitai": true, - "CodeRabbitAI": true, - "coderabbitai[bot]": true, - "dependabot": true, - "renovate": true, - "copilot": true, - "some-new-app[bot]": true, - "myasnikovdaniil": false, - "alice": false, - "renovate-fan": false, - "bot-enthusiast": false, - } { - if got := botAuthor(login); got != want { - t.Errorf("botAuthor(%q) = %v, want %v", login, got, want) - } - } -} - -func TestClaimState(t *testing.T) { - alice := &confirmation{By: "alice"} - cases := []struct { - states []string - conf *confirmation - want string - }{ - {[]string{"MERGED"}, nil, "backport PR merged"}, - {[]string{"CLOSED", "MERGED"}, nil, "backport PR merged"}, - {[]string{"OPEN", "MERGED"}, nil, "backport PR merged"}, - {[]string{"MERGED"}, alice, "backport PR merged, confirmed by @alice"}, - {[]string{"OPEN"}, nil, "claimed here, not merged"}, - {[]string{"CLOSED", "OPEN"}, nil, "claimed here, not merged"}, - {[]string{"CLOSED"}, nil, "claimed here, closed unmerged"}, - } - for _, tc := range cases { - var prs []linkedPR - for i, s := range tc.states { - prs = append(prs, linkedPR{Number: i + 1, State: s}) - } - if got := claimState(prs, tc.conf); got != tc.want { - t.Errorf("claimState(%v, %v) = %q, want %q", tc.states, tc.conf, got, tc.want) - } - } -} diff --git a/hack/label-workflows-contract.bats b/hack/label-workflows-contract.bats index 32454ffe4..5e80293ac 100644 --- a/hack/label-workflows-contract.bats +++ b/hack/label-workflows-contract.bats @@ -214,7 +214,7 @@ assert_labels_defined() { count="$(stale_exempt_list exempt-issue-labels | wc -l | tr -d ' ')" [ "$count" -eq 12 ] count="$(stale_exempt_list exempt-pr-labels | wc -l | tr -d ' ')" - [ "$count" -eq 4 ] + [ "$count" -eq 2 ] # The renewal path: frozen exempts both; an accepted issue or a held PR is # deliberate, not stale. diff --git a/hack/release-freeze-contract.bats b/hack/release-freeze-contract.bats index 197734a2f..71e907f60 100644 --- a/hack/release-freeze-contract.bats +++ b/hack/release-freeze-contract.bats @@ -1,28 +1,22 @@ #!/usr/bin/env bats -# Contract for the rc freeze, the backport target resolution that depends on it, -# the triggering and concurrency rules that decide whether a backport runs at -# all, and the agreement between those rules and cmd/backport-audit, which -# gates a cut by reproducing them. Like promote-gate-contract.bats, these tests pin executable/structural -# workflow lines rather than prose: a commented-out gate or a guard demoted to a -# comment must never satisfy the contract. +# Contract for the rc freeze and the release-line build it dispatches. Like +# promote-gate-contract.bats, these tests pin executable/structural workflow +# lines rather than prose: a commented-out gate or a guard demoted to a comment +# must never satisfy the contract. # -# Neither workflow can be exercised by a PR lane. cut-prerelease.yaml runs only -# on workflow_dispatch from main or a release line, and its whole purpose is a -# one-way, write-once side effect — the first vX.Y.0-rc.N freezes the line, and -# there is no dry-run mode and no undo short of deleting the branch by hand. -# backport.yaml runs only on a merged main-targeted PR. So every invariant below -# is one that first reports a slip at a real release, on the one commit nobody -# wants to be debugging: getting the freeze condition wrong re-opens a frozen -# line silently, and getting the comparator wrong aims every backport at the -# wrong branch. Cheap to pin here, expensive to discover there. +# No PR lane exercises them. cut-prerelease.yaml runs only on workflow_dispatch +# from main or a release line, and its whole purpose is a one-way, write-once +# side effect — the first vX.Y.0-rc.N freezes the line, and there is no dry-run +# mode and no undo short of deleting the branch by hand. So every invariant +# below is one that first reports a slip at a real release, on the one commit +# nobody wants to be debugging: getting the freeze condition wrong re-opens a +# frozen line silently. Cheap to pin here, expensive to discover there. load test_helper REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)" CUT="$REPO_ROOT/.github/workflows/cut-prerelease.yaml" -BACKPORT="$REPO_ROOT/.github/workflows/backport.yaml" -AUDIT="$REPO_ROOT/cmd/backport-audit/main.go" # Strip YAML comments. POSIX `grep` only: the unit-test runner has no ripgrep. # grep exits 1 when nothing is selected (legitimate for an all-comment block) @@ -44,24 +38,6 @@ code_lines() { [ "$rc" -le 1 ] } -# Strip JavaScript comments. backport.yaml's logic lives in a github-script -# `script: |` block, where comments start with `//` and survive code_lines -# untouched — and that block's comments name the very APIs these tests assert -# are gone, so filtering on `#` alone would let a comment satisfy a pin. -script_lines() { - local rc=0 - grep -v '^[[:space:]]*//' || rc=$? - [ "$rc" -le 1 ] -} - -# Body of one job, up to the next top-level job key. -job_block() { - awk -v job=" $1:" ' - $0 == job { inside = 1; next } - /^ [a-z0-9_-]+:$/ { inside = 0 } - inside' "$2" -} - # Body of one `- name:` step, up to the next step at the same indent. step_block() { awk -v want=" - name: $1" ' @@ -78,7 +54,6 @@ step_line() { @test "workflows under contract exist" { [ -f "$CUT" ] - [ -f "$BACKPORT" ] } # ── the freeze condition ───────────────────────────────────────────────────── @@ -274,390 +249,3 @@ step_line() { [ -n "$guard" ] && [ -n "$checkout" ] [ "$guard" -lt "$checkout" ] } - -# ── backport target resolution ─────────────────────────────────────────────── -# Both assertions below run over script_lines: this logic sits in a -# github-script block whose comments discuss getLatestRelease and min-1 by name. - -@test "backport targets come from enumerated branches, not the latest release" { - block="$(job_block prepare "$BACKPORT")" - [ -n "$block" ] - - # Enumerate real branches. getLatestRelease returns the newest published - # STABLE, which throughout a freeze window still names the PREVIOUS line — so - # a `backport` on a fix for the release being stabilised would land on the - # wrong branch and miss the release it was written for. - printf '%s\n' "$block" | script_lines | grep -qF 'github.paginate(github.rest.repos.listBranches' - count="$(printf '%s\n' "$block" | script_lines | grep -cF 'getLatestRelease' || true)" - [ "${count:-0}" -eq 0 ] - - # Only branches matching release-. qualify, so per-release - # staging branches (release-1.6.1) and rc branches never become targets. - printf '%s\n' "$block" | script_lines | grep -qF 'release-(\d+)\.(\d+)$' - - # previous is the second-newest EXISTING line, never an arithmetic guess. The - # old min-1 derivation named a non-existent branch whenever a minor was - # skipped. - printf '%s\n' "$block" | script_lines | grep -qF 'lines.length > 1 ? lines[1].name' - count="$(printf '%s\n' "$block" | script_lines | grep -cE 'parseInt\(min\)[[:space:]]*-[[:space:]]*1|min[[:space:]]*-[[:space:]]*1' || true)" - [ "${count:-0}" -eq 0 ] - - # Asking for a previous line when only one exists must fail, not silently - # backport nowhere. - printf '%s\n' "$block" | script_lines | grep -qF 'core.setFailed' -} - -@test "release lines sort numerically descending so 1.10 outranks 1.9" { - block="$(job_block prepare "$BACKPORT")" - [ -n "$block" ] - - # Numeric descending on major then minor is the ONLY reason release-1.10 - # ranks above release-1.9. A slip to the default lexicographic sort compares - # the names as strings, puts "release-1.9" first, and aims every backport at - # a stale line — with nothing going red. - printf '%s\n' "$block" | script_lines | grep -qF '.sort((a, b) => (b.maj - a.maj) || (b.min - a.min))' - - # The comparands must be numbers, not the captured strings. - printf '%s\n' "$block" | script_lines | grep -qF 'maj: Number(m[1]), min: Number(m[2])' - - # And the head of that sort is what `backport` uses. - printf '%s\n' "$block" | script_lines | grep -qF 'lines[0].name' -} - -# ── skip a target whose backport already merged ───────────────────────────── -# A second label event on a PR whose first backport already merged (e.g. -# backport-previous added after backport merged) re-enters this job for the -# already-delivered target. The cherry-pick is then empty, which the action's -# draft_commit_conflicts handling misreads as a real conflict and reports as a -# failed backport on the original PR — this guard skips the re-run instead. - -@test "the guard checks merged, not just closed, PRs on the deterministic branch" { - block="$(job_block backport "$BACKPORT")" - [ -n "$block" ] - - # The branch name the action itself would have used. Checking anything else - # answers a different question than "did THIS backport already land". - printf '%s\n' "$block" | script_lines | grep -qF 'backport-${context.payload.pull_request.number}-to-${targetBranch}' - - # Closed alone is not enough: a closed-without-merge PR is a stale or - # abandoned attempt, not proof the change reached the target. Loosening this - # to any closed PR would skip a real retry after a conflict was closed - # unmerged. - printf '%s\n' "$block" | script_lines | grep -qF "state: 'closed'" - printf '%s\n' "$block" | script_lines | grep -qF 'p.merged_at !== null' -} - -@test "both the checkout and the backport-action step are gated on the guard" { - block="$(job_block backport "$BACKPORT")" - [ -n "$block" ] - - # Losing the gate on either step still runs the action against a target that - # already has the change, reproducing the false comment the guard exists to - # prevent. - count="$(printf '%s\n' "$block" | code_lines | grep -cF "steps.guard.outputs.already_merged != 'true'" || true)" - [ "${count:-0}" -eq 2 ] -} - -@test "the guard runs before the checkout and the backport-action step" { - guard="$(step_line 'Check for existing merged backport' "$BACKPORT")" - checkout="$(step_line 'Checkout repository' "$BACKPORT")" - create="$(step_line 'Create back‑port PR' "$BACKPORT")" - [ -n "$guard" ] && [ -n "$checkout" ] && [ -n "$create" ] - - [ "$guard" -lt "$checkout" ] - [ "$guard" -lt "$create" ] -} - -# Mirrors release-changelog-behaviour.bats's convention for a github-script -# snippet: extract the decision expression verbatim in shape and execute it -# under node, because a grep for "merged_at" cannot tell an `.some()` from an -# `.every()` or a flipped comparison. Two cases are the ones that matter and -# are NOT interchangeable: the first-ever run (branch never existed, so the -# API returns an empty list) and a real retry (someone closed a conflicting -# backport PR without merging it). Both must NOT skip, or either the very -# first backport attempt or a legitimate re-creation after a manual close -# never runs. -@test "the guard's merged decision is correct on empty, closed-unmerged and closed-merged lists" { - command -v node >/dev/null || { echo "node unavailable; skipping"; return 0; } - tmp="$(mktemp -d)" - - cat > "$tmp/guard.js" <<'JS' -function alreadyMerged(prs) { - return prs.some(p => p.merged_at !== null); -} -const cases = { - 'first-run-empty': [], - 'closed-unmerged': [{ merged_at: null }], - 'closed-merged': [{ merged_at: '2026-01-01T00:00:00Z' }], -}; -const lines = Object.entries(cases).map(([name, prs]) => name + '=' + alreadyMerged(prs)); -require('fs').writeFileSync(1, lines.join('\n') + '\n'); -JS - - node "$tmp/guard.js" > "$tmp/out.txt" 2>"$tmp/err.txt" || { - echo "node failed:" >&2; cat "$tmp/err.txt" >&2; exit 1; } - - want="first-run-empty=false -closed-unmerged=false -closed-merged=true" - got="$(cat "$tmp/out.txt")" - [ "$got" = "$want" ] || { - echo "guard decision table is wrong." >&2 - echo "want:" >&2; printf '%s\n' "$want" | sed 's/^/ /' >&2 - echo "got:" >&2; printf '%s\n' "$got" | sed 's/^/ /' >&2 - exit 1 - } - - # The mirror must match the real guard's exact decision expression, not just - # the presence of "merged_at" somewhere in the step. Wrapped like the other - # tests in this file's final assertion: a bare failing command here, under a - # trap-installed EXIT handler, has been observed to abort the whole bats run - # instead of just this test on macOS. - block="$(job_block backport "$BACKPORT")" - [ -n "$block" ] - printf '%s\n' "$block" | script_lines | grep -qF 'prs.some(p => p.merged_at !== null)' || { - echo "backport.yaml's guard no longer uses prs.some(p => p.merged_at !== null)." >&2; exit 1; } - - rm -rf "$tmp" -} - -# ── backport concurrency ───────────────────────────────────────────────────── -# A run joins its concurrency group before `prepare`'s guard is evaluated, so -# nothing in that guard can keep a label event from disturbing the run already -# backporting the PR. The group key and `cancel-in-progress` close different -# halves of that and are pinned separately: either one alone still leaves a -# live failure, and neither reads load-bearing on its own. - -@test "the trigger surface the concurrency key is built for does not widen" { - # Both halves of the key are written against one trigger delivering `closed` or - # `labeled`, and neither degrades safely if anything else arrives. The group key - # routes only `labeled` aside, so a new event lands in the main group; - # `cancel-in-progress` is positive, so it does not cancel there; and `prepare` - # does not qualify it. The result occupies the single pending slot, evicts a - # genuine backport request, and delivers nothing. - # - # Pinned as the whole surface rather than just the types list, because a SIBLING - # trigger reaches the same harm without touching `types:` at all: anything - # carrying a `pull_request` payload resolves the PR number and so lands in the - # same per-PR group. `pull_request_review` is the realistic one. - # - # Deliberately strict: this also fails on a widening that would be harmless, - # `workflow_dispatch` for instance, whose payload carries no `pull_request` so - # the key degenerates to a shared group holding no genuine request. Separating - # the harmless case needs a list of which triggers carry that payload, and that - # list drifts. A red here means read the block above, not that the pin is wrong. - triggers="$(awk '/^on:/{inside=1;next} /^[a-z]/{inside=0} inside && /^ [a-z_]+:/{print $1}' "$BACKPORT")" - [ -n "$triggers" ] - [ "$(printf '%s\n' "$triggers" | wc -l | tr -d ' ')" -eq 1 ] - printf '%s\n' "$triggers" | grep -qF 'pull_request_target:' - - printf '%s\n' "$(code_lines < "$BACKPORT")" | grep -qF 'types: [closed, labeled]' -} - -@test "a label event that requests no backport gets its own concurrency group" { - line="$(code_lines < "$BACKPORT" | grep '^ group: backport-')" - [ -n "$line" ] - - # The per-PR discriminator, first: without it every PR's backport shares one - # group, and since a merge still cancels in that group, merging one PR kills - # another's in-flight backport. The assertions below all describe how a label - # event is separated from a merge WITHIN one PR, and every one of them stays - # green with the PR number deleted, so the widest failure of the three is the - # one nothing else here would notice. - printf '%s\n' "$line" | grep -qF 'backport-${{ github.workflow }}-${{ github.event.pull_request.number }}${{' - - count="$(printf '%s\n' "$line" | grep -o "github\.event\.action == 'labeled'" | wc -l | tr -d ' ')" - [ "${count:-0}" -eq 1 ] - - # The split is on the label NAMES, not on the action alone. Routing EVERY label - # event aside would move `backport` and `backport-previous` out of the group - # holding the run they have to queue behind, and one request could then evict - # the other. - # - # Pinned as the literal names, not as a count of `label.name != '`. That count - # measures the operator and says nothing about the operands, so renaming one - # name here and not in `prepare` — the half-finished rename — left it at two - # and stayed green, while a real `backport-previous` event took the `-label` - # branch and became free to cherry-pick alongside the merge run. - # Including the conjunction that joins the action check to the name checks. - # `&&` binds tighter than `||` in GitHub expressions, so flipping this one - # operator reads as `action == 'labeled' || (name != … && name != …)`, and on a - # `closed` event `github.event.label` is absent, both name checks are true, and - # EVERY event takes the suffix. A constant suffix is exactly as vacuous as no - # split at all. Every operand here was pinned before this line existed; the - # operator joining them was not. - # - # TRANSITIONAL: four names, not the original pair, because backport.yaml - # accepts both spellings while the org-level dosubot still applies the legacy - # ones. The namespaced names are genuine backport requests, so they belong on - # this side of the split exactly as the legacy ones do — routing them aside - # would hand a `kind/backport` event its own group and let it cherry-pick - # alongside the merge run, which is the failure the pin above describes. When - # dosubot's PR labelling is switched off and backport.yaml drops its legacy - # arms, this literal and the count below go back to the pair. - printf '%s\n' "$line" | grep -qF "github.event.action == 'labeled' && github.event.label.name != 'backport' && github.event.label.name != 'backport-previous' && github.event.label.name != 'kind/backport' && github.event.label.name != 'kind/backport-previous'" - - # And exactly four of them. The literal above is a substring check, so appending - # a fifth exclusion satisfies it unchanged — and a fifth exclusion is not - # cosmetic: the named label stops being routed aside, lands in the main group, - # takes its single pending slot, skips in `prepare`, and evicts a genuine - # backport request while delivering nothing. Same harm as widening `types:`. - # Presence and count answer different questions; this test needs both. - count="$(printf '%s\n' "$line" | grep -o "github\.event\.label\.name != '" | wc -l | tr -d ' ')" - [ "${count:-0}" -eq 4 ] - - # The suffix the condition selects, not only the condition. Asserting the - # operands alone leaves the whole split deletable — collapsing the tail to - # `&& '' || ''` puts every label back in the main group with the condition - # still reading correctly above it. Inverting it to `&& '' || '-label'` is - # worse than deleting it, because that moves the two backport requests aside - # and leaves the irrelevant labels sharing the group with the run in flight, - # which is the arrangement this file exists to prevent. Both survive every - # other assertion here. - printf '%s\n' "$line" | grep -qF "&& '-label' || ''" -} - -@test "a labeled event queues behind the backport in flight instead of cancelling it" { - # Labels set with the default GITHUB_TOKEN start no run, but labels set by - # third-party GitHub Apps do, and they arrive in bursts. Under a plain `true` - # each one killed the backport the merge had just started, leaving a cancelled - # run next to a green one and the work repeated by whichever label came last. - # With `prepare` now narrowed, no burst label requalifies to repeat it, - # so dropping this arm does not restore the old noisy delivery — it loses the - # backport. `cancel-in-progress: true` is the obvious thing for the next reader - # to normalise this back to, so pin its absence too. - count="$(code_lines < "$BACKPORT" | grep -cF " cancel-in-progress: \${{ github.event.action == 'closed' }}" || true)" - [ "${count:-0}" -eq 1 ] - - # Positively, not as `!= 'labeled'`. The two are equivalent only while `types:` - # holds exactly `closed` and `labeled`; the negative form hands cancellation to - # any third type added later, in the main group, which is the behaviour this - # arm removes. - count="$(code_lines < "$BACKPORT" | grep -c "cancel-in-progress:.*!= 'labeled'" || true)" - [ "${count:-0}" -eq 0 ] - - count="$(code_lines < "$BACKPORT" | grep -c '^ cancel-in-progress: true$' || true)" - [ "${count:-0}" -eq 0 ] -} - -@test "both backport jobs bound how long a queued request can wait" { - # These ceilings exist because of the queuing above, not for their own sake. A - # cancelling key disposed of a stuck run by killing it; queuing makes the next - # genuine request wait behind it instead, and an unbounded job on the 6-hour - # default turns one wedged job into a six-hour hole in the release line. - # Nothing else in this file makes them look load-bearing, so a tidy-up that - # drops them reads as harmless. - # - # Scope, since the test name says "how long a queued request can wait": what is - # bounded here is a stuck JOB. A run can wedge at the RUN level with every job - # already finished, and `timeout-minutes` is job-level with no run-level - # equivalent, so that state holds the group until the run is cancelled. These - # ceilings do not reach it and are not meant to. - # Asserted as a bound, not as presence. The hazard is the SIZE of the wait, so - # a pin that only checks the key exists is satisfied by `timeout-minutes: 355`, - # which restores the hole to within five minutes of the default it was written - # against. - # - # Bounded on BOTH sides, because both directions break it. Too high restores - # the hole; too low cancels a real backport that was only waiting for a runner, - # and an upper-bound-only assertion stays green at `timeout-minutes: 1`. - # - # The floor is judgement, not measurement, and says so. `timeout-minutes` - # bounds execution, and every execution measured on this workflow has been - # seconds, so no observation argues for any particular floor. No maximum is - # quoted deliberately: a superlative over a growing set goes stale on the next - # slower run. What the floor guards is a ceiling set low enough to cancel a - # cherry-pick much larger than anything seen yet; ten minutes is far above - # every leg measured. Per JOB, not per run, and not over the runner queue, - # which no `timeout-minutes` reaches. - for job in prepare backport; do - block="$(job_block "$job" "$BACKPORT" | code_lines)" - [ -n "$block" ] - minutes="$(printf '%s\n' "$block" | grep -o '^ timeout-minutes: [0-9][0-9]*' | grep -o '[0-9][0-9]*$')" - [ -n "$minutes" ] - [ "$minutes" -ge 10 ] - [ "$minutes" -le 30 ] - done -} - -@test "each backport trigger reads only the labels it is entitled to" { - block="$(job_block prepare "$BACKPORT" | code_lines)" - [ -n "$block" ] - - # The cumulative label set answers for the merge and for nothing else. Left - # ungated it re-enters this job on every later unrelated label — an automated - # size/* or kind/* — for a merged PR still carrying `backport`, redoing a - # backport that has already been delivered. - # - # TRANSITIONAL: four reads, not the original pair. backport.yaml accepts the - # namespaced and the legacy spelling of each label while the org-level dosubot - # still applies the legacy ones; when its PR labelling is switched off and the - # legacy arms are dropped, this literal and the count below go back to two. - printf '%s\n' "$block" | grep -qF "(github.event.action == 'closed' && (contains(github.event.pull_request.labels.*.name, 'backport') || contains(github.event.pull_request.labels.*.name, 'backport-previous') || contains(github.event.pull_request.labels.*.name, 'kind/backport') || contains(github.event.pull_request.labels.*.name, 'kind/backport-previous')))" - - # And the guard reads that set exactly four times, all of them inside the gated - # disjunct above, so it cannot be joined by an ungated one that quietly - # restores the old behaviour while the assertion above stays green. - # - # Counted as OCCURRENCES, not lines. `grep -c` counts matching lines, and all - # four reads live on one line here, so it answers 1 and keeps answering 1 when a - # fifth read is appended to that same line — which is exactly the restoration - # this is meant to catch. Only an append on a NEW line would have moved it. - count="$(printf '%s\n' "$block" | grep -o 'contains(github\.event\.pull_request\.labels' | wc -l | tr -d ' ')" - [ "${count:-0}" -eq 4 ] - - # A label event answers for the label it carries. Both spellings of each name, - # for the transitional reason given above. - printf '%s\n' "$block" | grep -qF "(github.event.action == 'labeled' && (github.event.label.name == 'backport' || github.event.label.name == 'backport-previous' || github.event.label.name == 'kind/backport' || github.event.label.name == 'kind/backport-previous'))" - - # The two conjuncts the label logic hangs off, which the label assertions above - # would not miss. `base.ref == 'main'` is what stops the bot backporting its own - # output: its PRs target release-X.Y, so they cannot satisfy it however often a - # labeler re-applies `backport` to a `[Backport release-X.Y]` title. docs/release.md - # calls that architectural protection, which is only true while the line is here. - # With their trailing conjunctions, for the reason the group-key test spells - # out: flipping either `&&` to `||` turns the guard into a disjunction that - # fires on every merge to main and on labels applied to open PRs. That one does - # not produce a wrong backport, because the `backport` job re-checks both - # independently, but it runs the job when nothing asked for it. - # In BOTH jobs, because docs/release.md rests the architectural protection on - # both `if:` blocks. Losing either copy alone is survivable (the other still - # skips, and `backport` needs `prepare`), so this pins a duplicate rather than - # a hole — but an untested duplicate is how a duplicate stops being one. - for job in prepare backport; do - b="$(job_block "$job" "$BACKPORT" | code_lines)" - [ -n "$b" ] - printf '%s\n' "$b" | grep -qF "github.event.pull_request.base.ref == 'main' &&" - printf '%s\n' "$b" | grep -qF 'github.event.pull_request.merged == true &&' - done -} - -@test "the audit accepts exactly the backport labels the bot acts on" { - # cmd/backport-audit answers "has everything labelled for this line landed on - # it" and exits non-zero while anything has not, so docs/release.md gates a - # patch cut on it. It reaches its candidates with `gh pr list --label`, which - # matches a label by exact name. - # - # Pin the two sets equal rather than pinning either one's contents: the - # audit must not miss a spelling the bot honours, and must not keep one the - # bot has retired. Both directions matter, because the legacy names are - # transitional and the two files have to drop them together. - [ -f "$AUDIT" ] - - # Every name the bot tests for, from the label-detection script. - bot="$(grep -o "labels\.includes('[^']*')" "$BACKPORT" | sed "s/.*('//;s/')//" | sort -u)" - [ -n "$bot" ] - - # Every backport label spelled as a Go string literal in the audit. Its own - # prose quotes these names with backticks, not double quotes, so a comment - # cannot satisfy this. - audit="$(grep -o '"\(kind/\)\{0,1\}backport\(-previous\)\{0,1\}"' "$AUDIT" | tr -d '"' | sort -u)" - [ -n "$audit" ] - - [ "$bot" = "$audit" ] - - # Both requests are represented, so the sets cannot agree by both having lost - # the same half. - printf '%s\n' "$audit" | grep -qx 'kind/backport' - printf '%s\n' "$audit" | grep -qx 'kind/backport-previous' -} From 1ceaeb8b6e0dc6f7ebc260040971ea8f7918e638 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:40:12 +0000 Subject: [PATCH 162/518] docs(security): support the two latest minor releases The security policy promised three supported lines and said no rule decides which of them a fix reaches. The backport batch runs for the two newest lines with a stable release, by the rules on the backport policy page, and the maintainers agreed to make that the support window. SECURITY.md, the incident-response summary and the technical review now name the two latest minor releases, say that the older one drops out when a new minor ships, and send a reader to the backport policy for which fixes reach them. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- GENERAL_TECHNICAL_REVIEW.md | 2 +- SECURITY.md | 10 +++++----- docs/security/incident-response.md | 6 +----- 3 files changed, 7 insertions(+), 11 deletions(-) diff --git a/GENERAL_TECHNICAL_REVIEW.md b/GENERAL_TECHNICAL_REVIEW.md index 9e7818b55..81b83fb8a 100644 --- a/GENERAL_TECHNICAL_REVIEW.md +++ b/GENERAL_TECHNICAL_REVIEW.md @@ -88,7 +88,7 @@ ### Rollout, Upgrade and Rollback Planning -* **Compatibility with Kubernetes & frequency.** Cozystack ships a curated, tested Kubernetes version per release; tenant clusters are Certified Kubernetes following upstream skew/support. Platform upgrades are delivered as new Cozystack releases. The authoritative list of what has shipped is the [GitHub Releases page](https://github.com/cozystack/cozystack/releases); which release lines are still receiving security fixes is defined as a rolling rule in [SECURITY.md](https://github.com/cozystack/cozystack/blob/main/SECURITY.md): the latest stable minor and the two preceding minors are supported, older `1.x` minors receive critical and upgrade-blocking fixes at maintainer discretion, `v0.41.x` is end of life and anything earlier is unsupported. Specific version numbers are deliberately not restated here, because they drift with every release. Note that being in a supported line does not entail receiving every fix: the project has no documented rule for which supported lines a given fix is backported to, and that is recorded in `SECURITY.md` rather than presented as a policy. +* **Compatibility with Kubernetes & frequency.** Cozystack ships a curated, tested Kubernetes version per release; tenant clusters are Certified Kubernetes following upstream skew/support. Platform upgrades are delivered as new Cozystack releases. The authoritative list of what has shipped is the [GitHub Releases page](https://github.com/cozystack/cozystack/releases); which release lines are still receiving security fixes is defined as a rolling rule in [SECURITY.md](https://github.com/cozystack/cozystack/blob/main/SECURITY.md): the two latest minor releases are supported, older `1.x` minors receive critical and upgrade-blocking fixes at maintainer discretion, `v0.41.x` is end of life and anything earlier is unsupported. Specific version numbers are deliberately not restated here, because they drift with every release. Note that being in a supported line does not entail receiving every fix: which fixes reach a supported line is the [backport policy](https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md), which keeps breaking changes, platform migrations and the like for the next minor. * **Upgrade procedure** ([upgrade docs](https://cozystack.io/docs/v1.6/operations/cluster/upgrade/)): run the troubleshooting/health checklist; annotate the `cozy-system` namespace and `cozystack-version` ConfigMap with `helm.sh/resource-policy=keep`; `helm upgrade cozystack oci://ghcr.io/cozystack/cozystack/cozy-installer --version X.Y.Z` (**do not** use `--reuse-values` — the chart pins OCI repo/package versions); verify pods and `kubectl get hr -A`. * **Rollback.** GitOps/Helm-managed with versioned releases; roll back by targeting the previous release (previous HelmRelease revisions) via the same mechanism. Stateful data (databases, VM disks) persists on replicated storage across control-plane rollbacks. **No dedicated rollback runbook is published**, and version-skip constraints for downgrades are not documented; the [upgrade docs](https://cozystack.io/docs/v1.6/operations/cluster/upgrade/) currently cover the forward path only. * **How a rollout/rollback can fail & impact.** Risks: incompatible CRD schema change, storage/CNI component upgrades requiring node reboots (on the Talos path), or a database-operator major bump. Impact is scoped to the component; running tenant workloads generally continue, though storage/CNI upgrades can transiently affect networking or volume attachment during node rolls. `[maintainers: confirm whether upgrades trigger Talos node reboots]` diff --git a/SECURITY.md b/SECURITY.md index 8db8eb742..552ed1950 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -8,7 +8,7 @@ Cozystack integrates and ships many upstream cloud native components. If you bel ## Supported Versions -The Cozystack project maintains three release lines at a time: **the latest stable minor and the two minors preceding it.** That is a rolling rule rather than a list of version numbers, so it does not go stale between releases — read it against the GitHub Releases page to see which concrete lines it currently names: +The Cozystack project supports two release lines at a time: **the two latest minor releases**, the latest stable minor and the one before it. When a new minor ships its first stable release it becomes the latest, and the older of the two drops out. That is a rolling rule rather than a list of version numbers, so it does not go stale between releases — read it against the GitHub Releases page to see which concrete lines it currently names: @@ -16,16 +16,16 @@ Note that the rule is keyed to release lines, not to the existence of a `release | Version line | Status | Notes | | --- | --- | --- | -| Latest stable minor | Supported | Current stable release line. Security fixes land here first, and patch releases are cut from it. | -| The two minors preceding the latest | Supported | Receive security and important maintenance fixes, backported from the current line. | +| Latest stable minor | Supported | Current stable release line, the first supported line a fix reaches. Patch releases are cut from it. | +| The minor before the latest | Supported | Receives the same security and maintenance fixes, backported once the latest line carries them. | | Older `1.x` minors | Limited support | Critical and upgrade-blocking fixes may be backported at maintainer discretion. Adopters are encouraged to move to a supported line. | | `v0.41.x` | End of life | The final pre-v1 line. It received patch releases through March 2026 during the v0 to v1 transition and no longer does; no further fixes are planned. | | Everything before `v0.41` | Not supported | Long superseded. Upgrade to a `1.x` line. | | `alpha`, `beta`, `rc` releases | Not supported | Pre-release builds are for testing and evaluation only. | -Being in a supported row does not mean a line receives every fix. **The project does not have a documented rule for choosing which supported lines a given fix is backported to** — that decision is taken per fix by the maintainers, weighing severity against the risk and effort of the backport. This is stated rather than dressed up as a policy because no such policy exists; adopters who need a specific fix on a specific line should ask. +Being in a supported row does not mean a line receives every fix. Fixes reach the supported lines through the backport batch, by the rules in [`docs/release/backports.md`](docs/release/backports.md): a fix merged into `main` with a `fix` or `security` title goes onto both lines, unless the policy keeps it for the next minor, a breaking change or a platform migration for instance, and a maintainer can override that for a single fix. Adopters who need a specific fix on a specific line should ask. -Reporting a vulnerability against a line with limited support or at end of life is still welcome. We will confirm whether the issue also affects a supported line and fix it there; whether the fix is additionally backported to the older line is a per-fix maintainer decision on the same basis. +Reporting a vulnerability against a line with limited support or at end of life is still welcome. We will confirm whether the issue also affects a supported line and fix it there; whether the fix is additionally backported to the older line is a per-fix maintainer decision. ## Reporting a Vulnerability diff --git a/docs/security/incident-response.md b/docs/security/incident-response.md index 62469519e..156ce4f2f 100644 --- a/docs/security/incident-response.md +++ b/docs/security/incident-response.md @@ -40,11 +40,7 @@ One person may hold more than one role on a small incident. 4. **Private fix development.** The fix is developed under a **GitHub Security Advisory** (private fork) so that code and discussion stay confidential until release. -5. **Coordinated release.** The fix is shipped through the normal patch-release - and backport process. Which supported lines receive the backport is decided - per fix, weighing severity against backport risk — see the *Supported - Versions* table in `SECURITY.md`, which also records that no documented rule - governs that choice. +5. **Coordinated release.** The fix is shipped through the normal patch-release and backport process: the backport batch carries it onto the supported lines, the two latest minor releases, newest first, by the rules in [`docs/release/backports.md`](../release/backports.md), which a maintainer can override for a single fix. See the *Supported Versions* table in `SECURITY.md`. 6. **Disclosure.** A GHSA — and, when applicable, a CVE — is published. Public disclosure follows the outbound channels: GitHub Releases and release notes, the changelog, and the GHSA itself. The reporter is credited unless anonymity From 62b4213784b4552acd8170bb356ab248b23424ef Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 15:56:51 +0300 Subject: [PATCH 163/518] fix(seaweedfs): keep bucket lifecycle enforcement off unless enabled The daily walk that expires objects sends its deletes without an object identity, and the s3 server skips its compare-and-swap when none is given (walker_dispatcher.go and identityMatches, unchanged from 4.45 through 4.52). It does not re-check the object's age either, so a key overwritten between the walk's listing and the delete loses its new content. Backup buckets write unique keys and barely meet that window; a tenant bucket that rewrites a fixed key does. Turning enforcement on by default would also start applying, on upgrade, every rule that was stored and ignored until now. Both are retention changes an operator should choose, so the admin server and the worker now run only where lifecycle.enabled is set. On by default can return once the walk carries identity. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backups/kafka/92-scenario-user-restore.md | 2 +- .../backups/nats/92-scenario-user-restore.md | 2 +- .../psmdbstrategy_controller.go | 6 ++-- packages/extra/seaweedfs/README.md | 2 +- .../extra/seaweedfs/tests/lifecycle_test.yaml | 36 ++++++++++++++----- packages/extra/seaweedfs/values.schema.json | 4 +-- packages/extra/seaweedfs/values.yaml | 11 ++++-- .../seaweedfs-rd/cozyrds/seaweedfs.yaml | 2 +- 8 files changed, 46 insertions(+), 19 deletions(-) diff --git a/examples/backups/kafka/92-scenario-user-restore.md b/examples/backups/kafka/92-scenario-user-restore.md index 5416500ed..d637a6c9f 100644 --- a/examples/backups/kafka/92-scenario-user-restore.md +++ b/examples/backups/kafka/92-scenario-user-restore.md @@ -71,6 +71,6 @@ The script: had already replayed. The Pod log names each topic before its replay and again once it landed; empty or delete every topic it shows as started, not only the one the refusal names, then re-run. -- The tarball in object storage is not managed by the Cozystack `Backup` lifecycle: deleting the `Backup` CR removes the reference but leaves the S3 object in place. Tenants who need to reclaim space should delete the object, or set a lifecycle rule on the bucket. The Cozystack SeaweedFS applies such rules once a day while its `lifecycle.enabled` is on, which is the default; another object store applies them only if it implements bucket lifecycle. +- The tarball in object storage is not managed by the Cozystack `Backup` lifecycle: deleting the `Backup` CR removes the reference but leaves the S3 object in place. Tenants who need to reclaim space should delete the object, or set a lifecycle rule on the bucket. A Simple or MultiZone Cozystack SeaweedFS applies such rules once a day only while its `lifecycle.enabled` is on, which is not the default; a Client instance applies none itself, so the SeaweedFS it connects to must have `lifecycle.enabled` on. Another object store applies them only if it implements bucket lifecycle. - Consumer-group offsets, topic configs, ACLs and users are not part of the backup; consumers start from the restored topic's fresh offsets. diff --git a/examples/backups/nats/92-scenario-user-restore.md b/examples/backups/nats/92-scenario-user-restore.md index 98c51a5cc..5aefface0 100644 --- a/examples/backups/nats/92-scenario-user-restore.md +++ b/examples/backups/nats/92-scenario-user-restore.md @@ -55,6 +55,6 @@ The script: - Both flows rely on the strategy template branching on `.Mode`. See `01-create-strategy.sh`. -- The tarball in object storage is not managed by the Cozystack `Backup` lifecycle: deleting the `Backup` CR removes the reference but leaves the S3 object in place. Tenants who need to reclaim space should delete the object, or set a lifecycle rule on the bucket. The Cozystack SeaweedFS applies such rules once a day while its `lifecycle.enabled` is on, which is the default; another object store applies them only if it implements bucket lifecycle. +- The tarball in object storage is not managed by the Cozystack `Backup` lifecycle: deleting the `Backup` CR removes the reference but leaves the S3 object in place. Tenants who need to reclaim space should delete the object, or set a lifecycle rule on the bucket. A Simple or MultiZone Cozystack SeaweedFS applies such rules once a day only while its `lifecycle.enabled` is on, which is not the default; a Client instance applies none itself, so the SeaweedFS it connects to must have `lifecycle.enabled` on. Another object store applies them only if it implements bucket lifecycle. - `nats stream restore` recreates the stream's configuration from the snapshot; consumers are restored as captured at backup time. diff --git a/internal/backupcontroller/psmdbstrategy_controller.go b/internal/backupcontroller/psmdbstrategy_controller.go index 819274f71..9caa218a7 100644 --- a/internal/backupcontroller/psmdbstrategy_controller.go +++ b/internal/backupcontroller/psmdbstrategy_controller.go @@ -1376,9 +1376,9 @@ func (r *BackupJobReconciler) createMongoDBBackupArtifact( // annotation (honoured before any apiserver read, so it still frees a Backup // whose operator CR has become unreadable) and a going-away namespace (the psmdb // delete-backup exec cannot run there) each make the driver strip its own -// finalizer and release, leaving the object to a lifecycle rule on the bucket, -// if one is set, or to manual cleanup. This bounds only what the driver -// contributes: a Get that fails with something other than +// finalizer and release, leaving the object to manual cleanup, or to a +// lifecycle rule on the bucket where the object store applies one. This bounds +// only what the driver contributes: a Get that fails with something other than // NotFound/NoMatchError still requeues (below), and the psmdb operator's own // release-lock finalizer keeps the CR Terminating while the operator is down, // whatever the driver strips. A down operator in a live namespace is likewise diff --git a/packages/extra/seaweedfs/README.md b/packages/extra/seaweedfs/README.md index c0866722b..fa500de37 100644 --- a/packages/extra/seaweedfs/README.md +++ b/packages/extra/seaweedfs/README.md @@ -79,5 +79,5 @@ | `s3.resources.memory` | Amount of memory allocated. | `quantity` | `""` | | `s3.resourcesPreset` | Default sizing preset used when `resources` is omitted. | `string` | `t1.small` | | `lifecycle` | S3 bucket lifecycle configuration. | `object` | `{}` | -| `lifecycle.enabled` | Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. | `bool` | `true` | +| `lifecycle.enabled` | Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new. | `bool` | `false` | diff --git a/packages/extra/seaweedfs/tests/lifecycle_test.yaml b/packages/extra/seaweedfs/tests/lifecycle_test.yaml index e8d3c0842..2d74d3272 100644 --- a/packages/extra/seaweedfs/tests/lifecycle_test.yaml +++ b/packages/extra/seaweedfs/tests/lifecycle_test.yaml @@ -14,7 +14,10 @@ release: namespace: tenant-root tests: - - it: runs the lifecycle admin and worker by default + # The upstream walker deletes without an object identity, so a key + # overwritten between its listing and the delete RPC goes too. Applying + # stored rules is therefore an operator's decision, not an upgrade's. + - it: keeps the lifecycle admin and worker off by default template: templates/seaweedfs.yaml set: &platform _namespace: @@ -26,6 +29,23 @@ tests: documentSelector: path: metadata.name value: seaweedfs-system + asserts: + - equal: + path: spec.values.seaweedfs.admin.enabled + value: false + - equal: + path: spec.values.seaweedfs.worker.enabled + value: false + + - it: runs the lifecycle admin and worker when lifecycle is enabled + template: templates/seaweedfs.yaml + set: &enabled + <<: *platform + lifecycle: + enabled: true + documentSelector: + path: metadata.name + value: seaweedfs-system asserts: - equal: path: spec.values.seaweedfs.admin.enabled @@ -59,7 +79,7 @@ tests: - it: puts the admin volume on the instance's storage class template: templates/seaweedfs.yaml set: - <<: *platform + <<: *enabled volume: storageClass: replicated documentSelector: @@ -72,7 +92,7 @@ tests: - it: leaves the admin volume on the cluster default when the instance names none template: templates/seaweedfs.yaml - set: *platform + set: *enabled documentSelector: path: metadata.name value: seaweedfs-system @@ -82,7 +102,7 @@ tests: - it: monitors the admin and the worker while lifecycle is on template: templates/seaweedfs.yaml - set: *platform + set: *enabled asserts: - containsDocument: kind: WorkloadMonitor @@ -133,7 +153,7 @@ tests: - it: generates admin credentials for a new instance template: templates/admin-credentials.yaml - set: *platform + set: *enabled asserts: - isKind: of: Secret @@ -151,7 +171,7 @@ tests: # one, since it reads the Secret into its environment once at start. - it: keeps the password of an existing Secret template: templates/admin-credentials.yaml - set: *platform + set: *enabled kubernetesProvider: scheme: "v1/Secret": @@ -188,7 +208,7 @@ tests: - it: renders no credentials for a Client instance template: templates/admin-credentials.yaml set: - <<: *platform + <<: *enabled topology: Client filer: grpcHost: seaweedfs-filer.tenant-other.svc @@ -222,7 +242,7 @@ tests: # monitor missing from it never shows up for the tenant. - it: lets the dashboard read the admin and worker monitors template: templates/dashboard-resourcemap.yaml - set: *platform + set: *enabled documentSelector: path: kind value: Role diff --git a/packages/extra/seaweedfs/values.schema.json b/packages/extra/seaweedfs/values.schema.json index 9ce4b184a..882ed5195 100644 --- a/packages/extra/seaweedfs/values.schema.json +++ b/packages/extra/seaweedfs/values.schema.json @@ -875,9 +875,9 @@ "default": {}, "properties": { "enabled": { - "description": "Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to.", + "description": "Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new.", "type": "boolean", - "default": true + "default": false } } } diff --git a/packages/extra/seaweedfs/values.yaml b/packages/extra/seaweedfs/values.yaml index 4f47c4fcc..4ccf78584 100644 --- a/packages/extra/seaweedfs/values.yaml +++ b/packages/extra/seaweedfs/values.yaml @@ -194,8 +194,15 @@ s3: resourcesPreset: "t1.small" ## @typedef {struct} Lifecycle - Enforcement of S3 bucket lifecycle rules. -## @field {bool} [enabled] - Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. +## @field {bool} [enabled] - Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new. ## @param {Lifecycle} [lifecycle] - S3 bucket lifecycle configuration. lifecycle: - enabled: true + # Off by default. The daily walk of SeaweedFS (4.45 through 4.52) sends its + # expiry deletes without an object identity, which the s3 server takes as + # "skip the check" (weed/s3api/s3lifecycle/dailyrun/walker_dispatcher.go, + # identityMatches in weed/s3api/s3api_internal_lifecycle.go), and it does not + # re-check the object's age. A key overwritten between the walk's listing and + # the delete loses its new content, which a tenant bucket that rewrites a + # fixed key can hit. On by default only once that path carries identity. + enabled: false diff --git a/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml b/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml index ddf5834e0..3afeed1c7 100644 --- a/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml +++ b/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml @@ -8,7 +8,7 @@ spec: singular: seaweedfs plural: seaweedfses openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"host":{"description":"The hostname used to access SeaweedFS externally (defaults to 's3' subdomain for the tenant host).","type":"string","default":""},"topology":{"description":"The topology of the SeaweedFS cluster.","type":"string","default":"Simple","enum":["Simple","MultiZone","Client"]},"replicationFactor":{"description":"Replication factor: number of replicas for each volume in the SeaweedFS cluster.","type":"integer","default":2},"db":{"description":"Database configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of database replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""}}},"master":{"description":"Master service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of master replicas.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"volumeSizeLimitMB":{"description":"Size in MB at which a volume stops taking writes, and the divisor that sets how many buckets a volume server can serve. Lower it to fit more buckets on the same disk, at the cost of more volume files for the same stored data; on a large volume it is worth raising instead. Changing it on an instance that already holds data costs something either way: lowering it stops writes to every bucket whose volumes all reach the new value until that bucket can grow, and raising it takes the room new buckets grow into, so run the check in docs/operations/seaweedfs-lowering-the-volume-size-limit.md first. Must be between 1 and 30000. A value close to or above `volume.size` leaves each volume server a single slot rather than failing, since the server floors its slot count at one.","type":"integer","default":400,"maximum":30000,"minimum":1}}},"filer":{"description":"Filer service configuration.","type":"object","default":{},"properties":{"grpcHost":{"description":"The hostname used to expose or access the filer service externally.","type":"string","default":""},"grpcPort":{"description":"The port used to access the filer service externally.","type":"integer","default":443},"replicas":{"description":"Number of filer replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"whitelist":{"description":"A list of IP addresses or CIDR ranges that are allowed to access the filer service.","type":"array","default":[],"items":{"type":"string"}}}},"volume":{"description":"Volume service configuration.","type":"object","default":{},"properties":{"diskType":{"description":"SeaweedFS disk type tag for the default volume servers (e.g., \"hdd\", \"ssd\").","type":"string","default":""},"pools":{"description":"A map of storage pools. Each pool creates a separate Volume StatefulSet with its own disk type.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of volume replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""},"zones":{"description":"A map of zones for MultiZone topology. Each zone can have its own number of replicas and size.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"dataCenter":{"description":"SeaweedFS data center name for this zone. Defaults to the zone name.","type":"string"},"nodeSelector":{"description":"YAML nodeSelector for this zone (default: topology.kubernetes.io/zone: ).","type":"string"},"pools":{"description":"A map of storage pools for this zone. Each pool creates a separate Volume StatefulSet per zone.","type":"object","additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of replicas in the zone.","type":"integer"},"size":{"description":"Zone storage size.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store zone data. Defaults to volume.storageClass.","type":"string"}}}}}},"s3":{"description":"S3 service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of S3 replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"lifecycle":{"description":"S3 bucket lifecycle configuration.","type":"object","default":{},"properties":{"enabled":{"description":"Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to.","type":"boolean","default":true}}}}} + {"title":"Chart Values","type":"object","properties":{"host":{"description":"The hostname used to access SeaweedFS externally (defaults to 's3' subdomain for the tenant host).","type":"string","default":""},"topology":{"description":"The topology of the SeaweedFS cluster.","type":"string","default":"Simple","enum":["Simple","MultiZone","Client"]},"replicationFactor":{"description":"Replication factor: number of replicas for each volume in the SeaweedFS cluster.","type":"integer","default":2},"db":{"description":"Database configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of database replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""}}},"master":{"description":"Master service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of master replicas.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"volumeSizeLimitMB":{"description":"Size in MB at which a volume stops taking writes, and the divisor that sets how many buckets a volume server can serve. Lower it to fit more buckets on the same disk, at the cost of more volume files for the same stored data; on a large volume it is worth raising instead. Changing it on an instance that already holds data costs something either way: lowering it stops writes to every bucket whose volumes all reach the new value until that bucket can grow, and raising it takes the room new buckets grow into, so run the check in docs/operations/seaweedfs-lowering-the-volume-size-limit.md first. Must be between 1 and 30000. A value close to or above `volume.size` leaves each volume server a single slot rather than failing, since the server floors its slot count at one.","type":"integer","default":400,"maximum":30000,"minimum":1}}},"filer":{"description":"Filer service configuration.","type":"object","default":{},"properties":{"grpcHost":{"description":"The hostname used to expose or access the filer service externally.","type":"string","default":""},"grpcPort":{"description":"The port used to access the filer service externally.","type":"integer","default":443},"replicas":{"description":"Number of filer replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"whitelist":{"description":"A list of IP addresses or CIDR ranges that are allowed to access the filer service.","type":"array","default":[],"items":{"type":"string"}}}},"volume":{"description":"Volume service configuration.","type":"object","default":{},"properties":{"diskType":{"description":"SeaweedFS disk type tag for the default volume servers (e.g., \"hdd\", \"ssd\").","type":"string","default":""},"pools":{"description":"A map of storage pools. Each pool creates a separate Volume StatefulSet with its own disk type.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of volume replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""},"zones":{"description":"A map of zones for MultiZone topology. Each zone can have its own number of replicas and size.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"dataCenter":{"description":"SeaweedFS data center name for this zone. Defaults to the zone name.","type":"string"},"nodeSelector":{"description":"YAML nodeSelector for this zone (default: topology.kubernetes.io/zone: ).","type":"string"},"pools":{"description":"A map of storage pools for this zone. Each pool creates a separate Volume StatefulSet per zone.","type":"object","additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of replicas in the zone.","type":"integer"},"size":{"description":"Zone storage size.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store zone data. Defaults to volume.storageClass.","type":"string"}}}}}},"s3":{"description":"S3 service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of S3 replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"lifecycle":{"description":"S3 bucket lifecycle configuration.","type":"object","default":{},"properties":{"enabled":{"description":"Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new.","type":"boolean","default":false}}}}} release: prefix: "" labels: From 55f1fd33a8dd8b73c196d4b45bb328e4c1791a74 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 16:04:04 +0300 Subject: [PATCH 164/518] fix(backups): let the controller delete operator MariaDB Backups Deleting a MariaDB Backup now deletes the k8s.mariadb.com/Backup that wrote its dump, once the dump itself is gone, so the controller needs the delete verb on that resource. Restores stay create-only. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../templates/rbac.yaml | 11 ++++---- .../tests/mariadb_cleanup_rbac_test.yaml | 28 +++++++++++++++++++ 2 files changed, 34 insertions(+), 5 deletions(-) create mode 100644 packages/system/backupstrategy-controller/tests/mariadb_cleanup_rbac_test.yaml diff --git a/packages/system/backupstrategy-controller/templates/rbac.yaml b/packages/system/backupstrategy-controller/templates/rbac.yaml index 8d90980a8..e9e35b066 100644 --- a/packages/system/backupstrategy-controller/templates/rbac.yaml +++ b/packages/system/backupstrategy-controller/templates/rbac.yaml @@ -146,15 +146,16 @@ rules: # MariaDB strategy: read the operator-side MariaDB CR to surface "not yet # rendered" preconditions, create one-shot k8s.mariadb.com/Backup CRs on # the BackupJob path, and create k8s.mariadb.com/Restore CRs on the -# RestoreJob path. The driver only ever Creates/Lists/Gets these objects - -# it does not patch or delete them; their lifecycle is owned by -# mariadb-operator and (for cleanup) by tenants / explicit teardown flows, -# never by this controller. +# RestoreJob path. A Backup is deleted once the dump it wrote is, when its +# Cozystack Backup is deleted; a Restore is never deleted. - apiGroups: ["k8s.mariadb.com"] resources: ["mariadbs"] verbs: ["get", "list", "watch"] - apiGroups: ["k8s.mariadb.com"] - resources: ["backups", "restores"] + resources: ["backups"] + verbs: ["get", "list", "watch", "create", "delete"] +- apiGroups: ["k8s.mariadb.com"] + resources: ["restores"] verbs: ["get", "list", "watch", "create"] # MongoDB strategy: read the operator-side PerconaServerMongoDB CR to gate on # the backup preconditions, create one-shot PerconaServerMongoDBBackup CRs on the diff --git a/packages/system/backupstrategy-controller/tests/mariadb_cleanup_rbac_test.yaml b/packages/system/backupstrategy-controller/tests/mariadb_cleanup_rbac_test.yaml new file mode 100644 index 000000000..21ac6e807 --- /dev/null +++ b/packages/system/backupstrategy-controller/tests/mariadb_cleanup_rbac_test.yaml @@ -0,0 +1,28 @@ +suite: RBAC of the MariaDB dump cleanup + +# Deleting a MariaDB Backup deletes its operator Backup once the dump is gone. +# The Go unit tests run with no authorization, so a missing verb only shows up +# on a cluster, where every such Backup would stay Terminating. + +templates: + - templates/rbac.yaml + +release: + name: backupstrategy-controller + namespace: cozy-backup-controller + +tests: + - it: "ClusterRole can delete operator Backups but not Restores" + asserts: + - contains: + path: rules + content: + apiGroups: ["k8s.mariadb.com"] + resources: ["backups"] + verbs: ["get", "list", "watch", "create", "delete"] + - contains: + path: rules + content: + apiGroups: ["k8s.mariadb.com"] + resources: ["restores"] + verbs: ["get", "list", "watch", "create"] From 5cbb3a633e2231eef57da36ac5a1371304154b14 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 15:57:38 +0300 Subject: [PATCH 165/518] fix(seaweedfs): give the lifecycle admin volume its own storage class volume.storageClass is picked for volume server data, and in a MultiZone instance it can name a class whose volumes exist in one zone or on one node. Reusing it for the admin volume would tie the admin to that zone or node, so losing it would stop lifecycle enforcement for the whole instance, a placement nobody chose for the admin. The admin volume now takes lifecycle.storageClass, and the cluster default when that is empty. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- packages/extra/seaweedfs/README.md | 1 + .../extra/seaweedfs/templates/seaweedfs.yaml | 2 +- .../extra/seaweedfs/tests/lifecycle_test.yaml | 29 +++++++++++++++---- packages/extra/seaweedfs/values.schema.json | 5 ++++ packages/extra/seaweedfs/values.yaml | 2 ++ .../seaweedfs-rd/cozyrds/seaweedfs.yaml | 4 +-- 6 files changed, 35 insertions(+), 8 deletions(-) diff --git a/packages/extra/seaweedfs/README.md b/packages/extra/seaweedfs/README.md index fa500de37..0540db4af 100644 --- a/packages/extra/seaweedfs/README.md +++ b/packages/extra/seaweedfs/README.md @@ -80,4 +80,5 @@ | `s3.resourcesPreset` | Default sizing preset used when `resources` is omitted. | `string` | `t1.small` | | `lifecycle` | S3 bucket lifecycle configuration. | `object` | `{}` | | `lifecycle.enabled` | Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new. | `bool` | `false` | +| `lifecycle.storageClass` | StorageClass of the admin server's 1Gi job-state volume. Empty uses the cluster default; `volume.storageClass` is not used, because in MultiZone it can be local to one zone. Prefer a replicated class: on a node-local one, losing the node stops lifecycle enforcement. Set it before enabling lifecycle: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. To move the volume, disable lifecycle, delete the `admin-data-seaweedfs-admin-0` PVC of the instance, change the class and enable lifecycle again. | `string` | `""` | diff --git a/packages/extra/seaweedfs/templates/seaweedfs.yaml b/packages/extra/seaweedfs/templates/seaweedfs.yaml index a2cf02c5b..9990a450c 100644 --- a/packages/extra/seaweedfs/templates/seaweedfs.yaml +++ b/packages/extra/seaweedfs/templates/seaweedfs.yaml @@ -385,7 +385,7 @@ spec: enabled: {{ .Values.lifecycle.enabled }} secret: existingSecret: {{ .Release.Name }}-admin-credentials - {{- with .Values.volume.storageClass }} + {{- with .Values.lifecycle.storageClass }} data: storageClass: {{ . }} {{- end }} diff --git a/packages/extra/seaweedfs/tests/lifecycle_test.yaml b/packages/extra/seaweedfs/tests/lifecycle_test.yaml index 2d74d3272..7af101faf 100644 --- a/packages/extra/seaweedfs/tests/lifecycle_test.yaml +++ b/packages/extra/seaweedfs/tests/lifecycle_test.yaml @@ -74,13 +74,15 @@ tests: path: spec.values.seaweedfs.worker.enabled value: false - # The admin's volume follows volume.storageClass when the instance sets one. - # Zone storage classes are not followed: the admin is not pinned to a zone. - - it: puts the admin volume on the instance's storage class + # volume.storageClass is chosen for volume data, and in MultiZone it can be + # a class local to one zone, while the admin is not pinned to any zone. Its + # volume therefore takes its own class, and the cluster default otherwise. + - it: puts the admin volume on the lifecycle storage class template: templates/seaweedfs.yaml set: <<: *enabled - volume: + lifecycle: + enabled: true storageClass: replicated documentSelector: path: metadata.name @@ -90,7 +92,24 @@ tests: path: spec.values.seaweedfs.admin.data.storageClass value: replicated - - it: leaves the admin volume on the cluster default when the instance names none + - it: does not put the admin volume on the volume storage class + template: templates/seaweedfs.yaml + set: + <<: *enabled + topology: MultiZone + replicationFactor: 1 + volume: + storageClass: zone-a-local + zones: + a: {} + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - notExists: + path: spec.values.seaweedfs.admin.data + + - it: leaves the admin volume on the cluster default when lifecycle names none template: templates/seaweedfs.yaml set: *enabled documentSelector: diff --git a/packages/extra/seaweedfs/values.schema.json b/packages/extra/seaweedfs/values.schema.json index 882ed5195..000755ffe 100644 --- a/packages/extra/seaweedfs/values.schema.json +++ b/packages/extra/seaweedfs/values.schema.json @@ -878,6 +878,11 @@ "description": "Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new.", "type": "boolean", "default": false + }, + "storageClass": { + "description": "StorageClass of the admin server's 1Gi job-state volume. Empty uses the cluster default; `volume.storageClass` is not used, because in MultiZone it can be local to one zone. Prefer a replicated class: on a node-local one, losing the node stops lifecycle enforcement. Set it before enabling lifecycle: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. To move the volume, disable lifecycle, delete the `admin-data-seaweedfs-admin-0` PVC of the instance, change the class and enable lifecycle again.", + "type": "string", + "default": "" } } } diff --git a/packages/extra/seaweedfs/values.yaml b/packages/extra/seaweedfs/values.yaml index 4ccf78584..3e5f37953 100644 --- a/packages/extra/seaweedfs/values.yaml +++ b/packages/extra/seaweedfs/values.yaml @@ -195,6 +195,7 @@ s3: ## @typedef {struct} Lifecycle - Enforcement of S3 bucket lifecycle rules. ## @field {bool} [enabled] - Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new. +## @field {string} [storageClass] - StorageClass of the admin server's 1Gi job-state volume. Empty uses the cluster default; `volume.storageClass` is not used, because in MultiZone it can be local to one zone. Prefer a replicated class: on a node-local one, losing the node stops lifecycle enforcement. Set it before enabling lifecycle: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. To move the volume, disable lifecycle, delete the `admin-data-seaweedfs-admin-0` PVC of the instance, change the class and enable lifecycle again. ## @param {Lifecycle} [lifecycle] - S3 bucket lifecycle configuration. lifecycle: @@ -206,3 +207,4 @@ lifecycle: # the delete loses its new content, which a tenant bucket that rewrites a # fixed key can hit. On by default only once that path carries identity. enabled: false + storageClass: "" diff --git a/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml b/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml index 3afeed1c7..f2efca2f8 100644 --- a/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml +++ b/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml @@ -8,7 +8,7 @@ spec: singular: seaweedfs plural: seaweedfses openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"host":{"description":"The hostname used to access SeaweedFS externally (defaults to 's3' subdomain for the tenant host).","type":"string","default":""},"topology":{"description":"The topology of the SeaweedFS cluster.","type":"string","default":"Simple","enum":["Simple","MultiZone","Client"]},"replicationFactor":{"description":"Replication factor: number of replicas for each volume in the SeaweedFS cluster.","type":"integer","default":2},"db":{"description":"Database configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of database replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""}}},"master":{"description":"Master service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of master replicas.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"volumeSizeLimitMB":{"description":"Size in MB at which a volume stops taking writes, and the divisor that sets how many buckets a volume server can serve. Lower it to fit more buckets on the same disk, at the cost of more volume files for the same stored data; on a large volume it is worth raising instead. Changing it on an instance that already holds data costs something either way: lowering it stops writes to every bucket whose volumes all reach the new value until that bucket can grow, and raising it takes the room new buckets grow into, so run the check in docs/operations/seaweedfs-lowering-the-volume-size-limit.md first. Must be between 1 and 30000. A value close to or above `volume.size` leaves each volume server a single slot rather than failing, since the server floors its slot count at one.","type":"integer","default":400,"maximum":30000,"minimum":1}}},"filer":{"description":"Filer service configuration.","type":"object","default":{},"properties":{"grpcHost":{"description":"The hostname used to expose or access the filer service externally.","type":"string","default":""},"grpcPort":{"description":"The port used to access the filer service externally.","type":"integer","default":443},"replicas":{"description":"Number of filer replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"whitelist":{"description":"A list of IP addresses or CIDR ranges that are allowed to access the filer service.","type":"array","default":[],"items":{"type":"string"}}}},"volume":{"description":"Volume service configuration.","type":"object","default":{},"properties":{"diskType":{"description":"SeaweedFS disk type tag for the default volume servers (e.g., \"hdd\", \"ssd\").","type":"string","default":""},"pools":{"description":"A map of storage pools. Each pool creates a separate Volume StatefulSet with its own disk type.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of volume replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""},"zones":{"description":"A map of zones for MultiZone topology. Each zone can have its own number of replicas and size.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"dataCenter":{"description":"SeaweedFS data center name for this zone. Defaults to the zone name.","type":"string"},"nodeSelector":{"description":"YAML nodeSelector for this zone (default: topology.kubernetes.io/zone: ).","type":"string"},"pools":{"description":"A map of storage pools for this zone. Each pool creates a separate Volume StatefulSet per zone.","type":"object","additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of replicas in the zone.","type":"integer"},"size":{"description":"Zone storage size.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store zone data. Defaults to volume.storageClass.","type":"string"}}}}}},"s3":{"description":"S3 service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of S3 replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"lifecycle":{"description":"S3 bucket lifecycle configuration.","type":"object","default":{},"properties":{"enabled":{"description":"Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new.","type":"boolean","default":false}}}}} + {"title":"Chart Values","type":"object","properties":{"host":{"description":"The hostname used to access SeaweedFS externally (defaults to 's3' subdomain for the tenant host).","type":"string","default":""},"topology":{"description":"The topology of the SeaweedFS cluster.","type":"string","default":"Simple","enum":["Simple","MultiZone","Client"]},"replicationFactor":{"description":"Replication factor: number of replicas for each volume in the SeaweedFS cluster.","type":"integer","default":2},"db":{"description":"Database configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of database replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""}}},"master":{"description":"Master service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of master replicas.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"volumeSizeLimitMB":{"description":"Size in MB at which a volume stops taking writes, and the divisor that sets how many buckets a volume server can serve. Lower it to fit more buckets on the same disk, at the cost of more volume files for the same stored data; on a large volume it is worth raising instead. Changing it on an instance that already holds data costs something either way: lowering it stops writes to every bucket whose volumes all reach the new value until that bucket can grow, and raising it takes the room new buckets grow into, so run the check in docs/operations/seaweedfs-lowering-the-volume-size-limit.md first. Must be between 1 and 30000. A value close to or above `volume.size` leaves each volume server a single slot rather than failing, since the server floors its slot count at one.","type":"integer","default":400,"maximum":30000,"minimum":1}}},"filer":{"description":"Filer service configuration.","type":"object","default":{},"properties":{"grpcHost":{"description":"The hostname used to expose or access the filer service externally.","type":"string","default":""},"grpcPort":{"description":"The port used to access the filer service externally.","type":"integer","default":443},"replicas":{"description":"Number of filer replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"whitelist":{"description":"A list of IP addresses or CIDR ranges that are allowed to access the filer service.","type":"array","default":[],"items":{"type":"string"}}}},"volume":{"description":"Volume service configuration.","type":"object","default":{},"properties":{"diskType":{"description":"SeaweedFS disk type tag for the default volume servers (e.g., \"hdd\", \"ssd\").","type":"string","default":""},"pools":{"description":"A map of storage pools. Each pool creates a separate Volume StatefulSet with its own disk type.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of volume replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""},"zones":{"description":"A map of zones for MultiZone topology. Each zone can have its own number of replicas and size.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"dataCenter":{"description":"SeaweedFS data center name for this zone. Defaults to the zone name.","type":"string"},"nodeSelector":{"description":"YAML nodeSelector for this zone (default: topology.kubernetes.io/zone: ).","type":"string"},"pools":{"description":"A map of storage pools for this zone. Each pool creates a separate Volume StatefulSet per zone.","type":"object","additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of replicas in the zone.","type":"integer"},"size":{"description":"Zone storage size.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store zone data. Defaults to volume.storageClass.","type":"string"}}}}}},"s3":{"description":"S3 service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of S3 replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"lifecycle":{"description":"S3 bucket lifecycle configuration.","type":"object","default":{},"properties":{"enabled":{"description":"Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new.","type":"boolean","default":false},"storageClass":{"description":"StorageClass of the admin server's 1Gi job-state volume. Empty uses the cluster default; `volume.storageClass` is not used, because in MultiZone it can be local to one zone. Prefer a replicated class: on a node-local one, losing the node stops lifecycle enforcement. Set it before enabling lifecycle: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. To move the volume, disable lifecycle, delete the `admin-data-seaweedfs-admin-0` PVC of the instance, change the class and enable lifecycle again.","type":"string","default":""}}}}} release: prefix: "" labels: @@ -26,7 +26,7 @@ spec: description: Seaweedfs module: true icon: PHN2ZyB3aWR0aD0iMTQ0IiBoZWlnaHQ9IjE0NCIgdmlld0JveD0iMCAwIDE0NCAxNDQiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CjxyZWN0IHdpZHRoPSIxNDQiIGhlaWdodD0iMTQ0IiByeD0iMjQiIGZpbGw9InVybCgjcGFpbnQwX2xpbmVhcl82NzlfMTkxMCkiLz4KPHBhdGggZD0iTTEzOC42ODUgMTIxLjA1N0MxMzguNjg1IDEyNi42NTIgMTM2LjQ2MiAxMzIuMDE3IDEzMi41MDQgMTM1Ljk3M0MxMjguNTQ3IDEzOS45MjkgMTIzLjE3OSAxNDIuMTUxIDExNy41ODIgMTQyLjE1MUMxMTEuOTg1IDE0Mi4xNTEgMTA2LjYxOCAxMzkuOTI5IDEwMi42NiAxMzUuOTczQzk4LjcwMjggMTMyLjAxNyA5Ni40Nzk1IDEyNi42NTIgOTYuNDc5NSAxMjEuMDU3Qzk2LjQ3OTUgMTE4LjI4NyA5Ny4wMjUzIDExNS41NDQgOTguMDg1OCAxMTIuOTg1Qzk5LjE0NjMgMTEwLjQyNSAxMDAuNzAxIDEwOC4xIDEwMi42NiAxMDYuMTQxQzEwNC42MiAxMDQuMTgyIDEwNi45NDYgMTAyLjYyOSAxMDkuNTA3IDEwMS41NjlDMTEyLjA2NyAxMDAuNTA5IDExNC44MTEgOTkuOTYyOSAxMTcuNTgyIDk5Ljk2MjlDMTIwLjM1MyA5OS45NjI5IDEyMy4wOTggMTAwLjUwOSAxMjUuNjU4IDEwMS41NjlDMTI4LjIxOCAxMDIuNjI5IDEzMC41NDQgMTA0LjE4MiAxMzIuNTA0IDEwNi4xNDFDMTM0LjQ2NCAxMDguMSAxMzYuMDE4IDExMC40MjUgMTM3LjA3OSAxMTIuOTg1QzEzOC4xMzkgMTE1LjU0NCAxMzguNjg1IDExOC4yODcgMTM4LjY4NSAxMjEuMDU3WiIgZmlsbD0idXJsKCNwYWludDFfcmFkaWFsXzY3OV8xOTEwKSIvPgo8cGF0aCBkPSJNMTEwLjMzNiAxMjYuMTQ3SDEyMy42OFYxMzYuODIzSDExMC4zMzZWMTI2LjE0N1oiIGZpbGw9IiMwRjVFOUMiLz4KPHBhdGggZD0iTTExOS4yNyAxMTIuMjk0QzExOS4yNyAxMTIuNzE0IDExOS4xNzggMTEzLjEzMSAxMTkgMTEzLjUxOUMxMTguODIxIDExMy45MDggMTE4LjU2IDExNC4yNjEgMTE4LjIzIDExNC41NThDMTE3LjkwMSAxMTQuODU1IDExNy41MDkgMTE1LjA5MSAxMTcuMDc5IDExNS4yNTJDMTE2LjY0OCAxMTUuNDEzIDExNi4xODYgMTE1LjQ5NiAxMTUuNzIgMTE1LjQ5NkMxMTQuNzc4IDExNS40OTYgMTEzLjg3NSAxMTUuMTU4IDExMy4yMSAxMTQuNTU4QzExMi41NDQgMTEzLjk1NyAxMTIuMTcgMTEzLjE0MyAxMTIuMTcgMTEyLjI5NEMxMTIuMTcgMTExLjQ0NSAxMTIuNTQ0IDExMC42MyAxMTMuMjEgMTEwLjAzQzExMy44NzUgMTA5LjQyOSAxMTQuNzc4IDEwOS4wOTIgMTE1LjcyIDEwOS4wOTJDMTE2LjE4NiAxMDkuMDkyIDExNi42NDggMTA5LjE3NSAxMTcuMDc5IDEwOS4zMzZDMTE3LjUwOSAxMDkuNDk2IDExNy45MDEgMTA5LjczMiAxMTguMjMgMTEwLjAzQzExOC41NiAxMTAuMzI3IDExOC44MjEgMTEwLjY4IDExOSAxMTEuMDY4QzExOS4xNzggMTExLjQ1NyAxMTkuMjcgMTExLjg3MyAxMTkuMjcgMTEyLjI5NFoiIGZpbGw9IiM1OTY4NkYiLz4KPHBhdGggZD0iTTEyOC4xMSAxMTQuMTAzQzEyOC4xMSAxMTQuOTUzIDEyNy43MzYgMTE1Ljc2NyAxMjcuMDcgMTE2LjM2OEMxMjYuNDA0IDExNi45NjggMTI1LjUwMSAxMTcuMzA1IDEyNC41NiAxMTcuMzA1QzEyNC4wOTQgMTE3LjMwNSAxMjMuNjMyIDExNy4yMjMgMTIzLjIwMSAxMTcuMDYyQzEyMi43NzEgMTE2LjkwMSAxMjIuMzc5IDExNi42NjUgMTIyLjA1IDExNi4zNjhDMTIxLjcyIDExNi4wNyAxMjEuNDU4IDExNS43MTcgMTIxLjI4IDExNS4zMjlDMTIxLjEwMiAxMTQuOTQgMTIxLjAxIDExNC41MjQgMTIxLjAxIDExNC4xMDNDMTIxLjAxIDExMy42ODMgMTIxLjEwMiAxMTMuMjY2IDEyMS4yOCAxMTIuODc4QzEyMS40NTggMTEyLjQ5IDEyMS43MiAxMTIuMTM3IDEyMi4wNSAxMTEuODM5QzEyMi4zNzkgMTExLjU0MiAxMjIuNzcxIDExMS4zMDYgMTIzLjIwMSAxMTEuMTQ1QzEyMy42MzIgMTEwLjk4NCAxMjQuMDk0IDExMC45MDEgMTI0LjU2IDExMC45MDFDMTI1LjUwMSAxMTAuOTAxIDEyNi40MDQgMTExLjIzOSAxMjcuMDcgMTExLjgzOUMxMjcuNzM2IDExMi40NCAxMjguMTEgMTEzLjI1NCAxMjguMTEgMTE0LjEwM1oiIGZpbGw9IiM1OTY4NkYiLz4KPHBhdGggZD0iTTEyMi4zMzMgMTE4Ljk3NkMxMjIuMzMzIDExOS44MjYgMTIxLjk1OCAxMjAuNjQgMTIxLjI5MyAxMjEuMjQxQzEyMC42MjcgMTIxLjg0MSAxMTkuNzI0IDEyMi4xNzggMTE4Ljc4MiAxMjIuMTc4QzExOC4zMTYgMTIyLjE3OCAxMTcuODU1IDEyMi4wOTYgMTE3LjQyNCAxMjEuOTM1QzExNi45OTMgMTIxLjc3NCAxMTYuNjAyIDEyMS41MzggMTE2LjI3MiAxMjEuMjQxQzExNS45NDMgMTIwLjk0MyAxMTUuNjgxIDEyMC41OSAxMTUuNTAzIDEyMC4yMDJDMTE1LjMyNCAxMTkuODEzIDExNS4yMzIgMTE5LjM5NyAxMTUuMjMyIDExOC45NzZDMTE1LjIzMiAxMTguNTU2IDExNS4zMjQgMTE4LjE0IDExNS41MDMgMTE3Ljc1MUMxMTUuNjgxIDExNy4zNjMgMTE1Ljk0MyAxMTcuMDEgMTE2LjI3MiAxMTYuNzEyQzExNi42MDIgMTE2LjQxNSAxMTYuOTkzIDExNi4xNzkgMTE3LjQyNCAxMTYuMDE4QzExNy44NTUgMTE1Ljg1NyAxMTguMzE2IDExNS43NzQgMTE4Ljc4MiAxMTUuNzc0QzExOS43MjQgMTE1Ljc3NCAxMjAuNjI3IDExNi4xMTIgMTIxLjI5MyAxMTYuNzEyQzEyMS45NTggMTE3LjMxMyAxMjIuMzMzIDExOC4xMjcgMTIyLjMzMyAxMTguOTc2WiIgZmlsbD0iIzU5Njg2RiIvPgo8cGF0aCBkPSJNMTE1LjMwOCAxMjEuOTA1QzExMy43MzUgMTIxLjQyNiAxMTUuNzA3IDEyMC42OCAxMTUuNDI5IDEyMC41NzNDMTE0LjY1MyAxMjAuMjc2IDExNS43MyAxMTkuMzMzIDExNi43NCAxMTguNTM5QzExNy42MjggMTE3Ljg0MSAxMTcuNjU5IDExNy44MzkgMTE3Ljg5MiAxMTguNDY4QzExOC4wMjQgMTE4LjgyMyAxMTguMzcxIDExOS4yMDYgMTE4LjY2NSAxMTkuMzE5QzExOS4yODkgMTE5LjU1OCAxMjAuOTUxIDExOS4yNjkgMTIwLjk1MSAxMTguODM1QzEyMC45NTEgMTE4LjIyMyAxMjEuNDE1IDExOC42OTkgMTIxLjc5NCAxMTkuNTMxQzEyMi40NTcgMTIwLjk4NyAxMjIuNDM3IDEyMi40NzIgMTIxLjQ1IDEyMi40NzJDMTIwLjg1OSAxMjIuNDcyIDEyMC4zMSAxMjIuNjkxIDEyMC4xOSAxMjMuMTQ3QzExNS4wNDYgMTI0LjYgMTE2LjQ3MSAxMjMuMjg0IDExNS4zMDggMTIxLjkwNVpNMTIzLjA5NCAxMTguMDU0QzEyMS42MDkgMTE3LjQ2NiAxMjAuNTQ3IDExNC44MzggMTIwLjU0NyAxMTMuMzA5QzEyMC41NDcgMTExLjMyMyAxMjIuNTQxIDEwOS4wOTUgMTI0LjMxNyAxMDkuMDk1QzEyOC4zMTUgMTA5LjA5NSAxMzAuNjg0IDExMi4yNjEgMTI4LjgzOCAxMTguNDA5QzEyOC4zODUgMTE5LjkxOSAxMjMuOTMzIDExOC4zODcgMTIzLjA5NCAxMTguMDU0Wk0xMjUuODY2IDExNS42NDlDMTI3LjU0NCAxMTQuNDc0IDEyNS44NTcgMTExLjc1NSAxMjMuOTgxIDExMi42MUMxMjIuNDc5IDExMy4yOTQgMTIzLjExOSAxMTYuMTE1IDEyNC43NyAxMTYuMTY0QzEyNC45NjEgMTE2LjE2OSAxMjUuNDU0IDExNS45MzggMTI1Ljg2NiAxMTUuNjQ5Wk0xMjQuMzM5IDExNC41NzlDMTIzLjc3MSAxMTQuMzgxIDEyMy44MDMgMTEzLjI1NiAxMjQuMzgzIDExMy4wMzRDMTI0LjkwMiAxMTIuODM1IDEyNS42MDQgMTEzLjM5OCAxMjUuNjA0IDExNC4wMTRDMTI1LjYwNCAxMTQuNDg5IDEyNC45MzYgMTE0Ljc4NyAxMjQuMzM5IDExNC41NzlaTTExMi4zMTkgMTE2Ljk4QzEwOS45NiAxMTUuNjcgMTEwLjI4NiAxMTEuMDA2IDExMi40MTcgMTA4Ljk2NUMxMTQuMzk2IDEwNy4wNjkgMTE4Ljc3OCAxMDcuMTAzIDExOS43NjQgMTA5LjQ2MkMxMjAuNDE1IDExMS4wMjEgMTIwLjIyOCAxMTIuNiAxMTkuMzk4IDExNC4wNzdDMTE4LjE5NCAxMTYuMjE5IDExNC4yNDIgMTE4LjA0OSAxMTIuMzE5IDExNi45OFpNMTE2LjU2IDExMy41OTNDMTE3LjMyNSAxMTIuOTAxIDExNy4yOTcgMTEyLjA2IDExNi41NzIgMTExLjI1OUMxMTUuNzc3IDExMC4zNzkgMTE0LjY5MiAxMTAuMzQ1IDExNC4wMzUgMTExLjM0N0MxMTMuNTI0IDExMi4xMjcgMTEzLjQzMSAxMTIuNTI4IDExMy45NDMgMTEzLjMwOUMxMTQuNTggMTE0LjI4IDExNS42NyAxMTQuMzk5IDExNi41NiAxMTMuNTkzWk0xMTQuMzc0IDExMi4zNEMxMTQuMTI4IDExMS42OTggMTE0Ljk4NSAxMTAuOTgxIDExNS42OCAxMTEuMjQ3QzExNS45NzQgMTExLjM2IDExNi4xNjQgMTExLjcxOCAxMTYuMTAyIDExMi4wNDRDMTE1Ljk2MSAxMTIuNzg1IDExNC42MzIgMTEzLjAxMiAxMTQuMzc0IDExMi4zNFoiIGZpbGw9IiNEM0Q2REEiLz4KPHBhdGggZD0iTTExOC40NjMgMTIxLjAwOEwxMTcuOTQ1IDEyMC43OEMxMTcuNTEgMTIwLjY4NSAxMTcuMjMxIDEyMS4xMDcgMTE3LjEzNiAxMjEuNTQzTDExNi44ODEgMTIyLjcwOUMxMTYuNzg2IDEyMy4xNDUgMTE3LjA2MSAxMjMuNTcxIDExNy40OTYgMTIzLjY2NkwxMTguNDQ3IDEyMy44NzRDMTE4Ljg4MiAxMjMuOTY5IDExOS4zMTEgMTIzLjY5NCAxMTkuNDA2IDEyMy4yNTlMMTE5LjY4NSAxMjEuNTU2QzExOS43OCAxMjEuMTIgMTE5LjQ3OSAxMjEuMjI4IDExOS4wNDMgMTIxLjEzM0wxMTguNjI4IDEyMS4wNDNDMTE4LjU3MyAxMjEuMzIxIDExOC41MTYgMTIxLjYxMyAxMTguNDcxIDEyMS44MzNDMTE4LjQzOCAxMjEuOTk4IDExOC40MDcgMTIyLjE0OCAxMTguMzc5IDEyMi4yODJDMTE4LjM1MSAxMjIuNDE1IDExOC4zMjcgMTIyLjUzMyAxMTguMzA1IDEyMi42MzVDMTE4LjI4MiAxMjIuNzM4IDExOC4yNjIgMTIyLjgyNSAxMTguMjQ1IDEyMi44OTdDMTE4LjIyOCAxMjIuOTY5IDExOC4yMTQgMTIzLjAyNSAxMTguMjAyIDEyMy4wNjhDMTE4LjE5NiAxMjMuMDg5IDExOC4xOTMgMTIzLjEwNyAxMTguMTg4IDEyMy4xMjFDMTE4LjE4MyAxMjMuMTM1IDExOC4xNzggMTIzLjE0NSAxMTguMTc1IDEyMy4xNTJDMTE4LjE3MyAxMjMuMTU1IDExOC4xNzIgMTIzLjE1NiAxMTguMTcxIDEyMy4xNThDMTE4LjE3IDEyMy4xNTkgMTE4LjE2OCAxMjMuMTYgMTE4LjE2NyAxMjMuMTZMMTE4LjE2NSAxMjMuMTU4QzExOC4xNjQgMTIzLjE1NiAxMTguMTYzIDEyMy4xNTIgMTE4LjE2MyAxMjMuMTQ4QzExOC4xNjIgMTIzLjE0IDExOC4xNjIgMTIzLjEyOSAxMTguMTYzIDEyMy4xMTVDMTE4LjE2MyAxMjMuMSAxMTguMTYzIDEyMy4wODMgMTE4LjE2NSAxMjMuMDYxQzExOC4xNjggMTIzLjAxOCAxMTguMTc1IDEyMi45NjEgMTE4LjE4MyAxMjIuODkxQzExOC4xOTIgMTIyLjgyIDExOC4yMDMgMTIyLjczNiAxMTguMjE2IDEyMi42NEMxMTguMjI5IDEyMi41NDMgMTE4LjI0NiAxMjIuNDMxIDExOC4yNjQgMTIyLjMwOEMxMTguMjgxIDEyMi4xODUgMTE4LjMwMSAxMjIuMDUxIDExOC4zMjMgMTIxLjkwM0MxMTguMzQ2IDEyMS43NTUgMTE4LjM3IDEyMS41OTIgMTE4LjM5NyAxMjEuNDE5QzExOC40MTYgMTIxLjI5OSAxMTguNDQyIDEyMS4xNCAxMTguNDYzIDEyMS4wMDhaIiBmaWxsPSIjOThDNkQ4Ii8+CjxwYXRoIGQ9Ik0xMDMuNTgxIDExNi4zNjdDMTAzLjQ3OSAxMTYuMTAzIDEwMy42MjIgMTE1LjY5OSAxMDMuODk4IDExNS40NjlDMTA0LjMwMyAxMTUuMTMzIDEwNC41MDQgMTE1LjE1NSAxMDQuOTMgMTE1LjU4MUMxMDUuMjIgMTE1Ljg3MSAxMDUuMzU1IDExNi4yNzYgMTA1LjIzIDExNi40NzlDMTA0LjkwMiAxMTcuMDA5IDEwMy43OTggMTE2LjkzNCAxMDMuNTgxIDExNi4zNjdaIiBmaWxsPSIjOThDNkQ4Ii8+CjxwYXRoIGQ9Ik0xMDYuMDM4IDExMi4yODFDMTA1LjY4IDExMS44NDkgMTA1LjcwMiAxMTEuNjYgMTA2LjE2NSAxMTEuMTk3QzEwNi42ODkgMTEwLjY3NCAxMDYuNzY0IDExMC42NzQgMTA3LjI4NyAxMTEuMTk3QzEwNy43NTEgMTExLjY2IDEwNy43NzMgMTExLjg0OSAxMDcuNDE1IDExMi4yODFDMTA3LjE3NiAxMTIuNTY4IDEwNi44NjYgMTEyLjgwMyAxMDYuNzI2IDExMi44MDNDMTA2LjU4NiAxMTIuODAzIDEwNi4yNzcgMTEyLjU2OCAxMDYuMDM4IDExMi4yODFaIiBmaWxsPSIjOThDNkQ4Ii8+CjxwYXRoIGQ9Ik0xMDYuMDM4IDEwNy4yMjRDMTA1LjY4MiAxMDYuNzk1IDEwNS42OTQgMTA2LjYxMiAxMDYuMTA2IDEwNi4yMDFDMTA2LjY1IDEwNS42NTcgMTA3LjczOCAxMDUuODggMTA3LjczOCAxMDYuNTM2QzEwNy43MzggMTA2Ljk3IDEwNy4wNzIgMTA3Ljc0NyAxMDYuNyAxMDcuNzQ3QzEwNi41NzUgMTA3Ljc0NyAxMDYuMjc3IDEwNy41MTIgMTA2LjAzOCAxMDcuMjI0WiIgZmlsbD0iIzk4QzZEOCIvPgo8cGF0aCBkPSJNMTE3Ljk1NSAxMzYuNTQxQzExNC41NzggMTM2LjExMSAxMTAuMDg3IDEzNC44ODEgMTA5LjQxIDEzNC4yNzRDMTA4LjEyMyAxMzMuMTE5IDEwOC45NDIgMTI3Ljg3OSAxMTAuNDY2IDEyNi41NEMxMTEuMzg3IDEyNS43MzEgMTE1LjAzOSAxMjQuNjYzIDExNy4xODIgMTI1LjE2OUMxMTkuNjQyIDEyNS43NDkgMTIzLjkzMiAxMjguNjQ4IDEyNC4yNSAxMjkuNTA4QzEyNC42MjUgMTMwLjUyMiAxMjQuMDQ4IDEzNC41NDEgMTIzLjMyMyAxMzUuNTM2QzEyMi42OTQgMTM2LjM5OSAxMjAuMzI1IDEzNi44NDMgMTE3Ljk1NSAxMzYuNTQxWk0xMjAuNTQgMTM0LjA4NEMxMjEuMjk3IDEzMy4zOTkgMTIxLjM0MiAxMzEuODQyIDEyMC42MjcgMTMxLjEyNkMxMTkuNDkgMTI5Ljk4OSAxMTcuMTEyIDEzMC44NjkgMTE3LjExMiAxMzIuNDI4QzExNy4xMTIgMTM0LjMwNCAxMTkuMTg5IDEzNS4zMDcgMTIwLjU0IDEzNC4wODRaTTExOC41ODIgMTMzLjEyNUMxMTguMzExIDEzMi40MTkgMTE4LjY4IDEzMS42MDggMTE5LjI3MiAxMzEuNjA4QzEyMCAxMzEuNjA4IDEyMC4yMjggMTMyLjE3MyAxMTkuODEyIDEzMi45NDlDMTE5LjM4MSAxMzMuNzU1IDExOC44NTMgMTMzLjgzIDExOC41ODIgMTMzLjEyNVpNMTE2LjQyMiAxMzMuMTQzQzExNy4wOTQgMTMyLjMzMyAxMTYuNzExIDEzMS4xNzYgMTE1LjY4MSAxMzAuOTAyQzExNC41ODEgMTMwLjYxIDExNC4wNTIgMTMxLjE4MyAxMTQuODY5IDEzMS43OEMxMTUuNjgzIDEzMi4zNzUgMTE1LjU1MSAxMzIuNjc5IDExNC41MzMgMTMyLjU1N0MxMTMuMzI3IDEzMi40MTMgMTEyLjgzMyAxMzEuNTY2IDExMy4yNTQgMTMwLjM2MkMxMTMuNjgxIDEyOS4xNDIgMTE1LjE5NyAxMjguODYzIDExNS43NTMgMTI5LjkwMkMxMTYuMTkzIDEzMC43MjUgMTE2LjYyNyAxMzAuNzkzIDExNi45IDEzMC4wODNDMTE3LjIyOSAxMjkuMjI0IDExNS45MTQgMTI4LjIzNyAxMTQuNDQgMTI4LjIzN0MxMTEuOTkzIDEyOC4yMzcgMTEwLjgxNiAxMzEuMDc0IDExMi41NDYgMTMyLjgwM0MxMTMuNSAxMzMuNzU4IDExNS43NDkgMTMzLjk1NSAxMTYuNDIyIDEzMy4xNDNaIiBmaWxsPSIjN0JBOUI5Ii8+CjxwYXRoIGQ9Ik0xMjAuMTE4IDEzOC41OTJDMTE4Ljc1MSAxMzcuMjk3IDEyMS45MTYgMTM2LjA5NiAxMjMuNjA2IDEzOC4yODNDMTI0LjQ2OSAxMzkuMzIyIDEyMi44NTMgMTM5Ljk3NiAxMjAuMTE4IDEzOC41OTJaTTEwNi4xMiAxMzUuNjU4QzEwNC43NTEgMTM0LjI4OSAxMDYuOTYzIDEzMy45MDQgMTA4LjU4MSAxMzUuMjNMMTA5Ljk0MSAxMzUuOTJMMTA4LjA1OSAxMzYuMDYxQzEwNy4yMTUgMTM2LjA2MiAxMDYuMzQzIDEzNS44ODEgMTA2LjEyIDEzNS42NThWMTM1LjY1OFpNMTI1LjM2MyAxMjcuODY2QzEyNS4wODIgMTI3LjQxIDEyOC4zMzMgMTI2LjI2NyAxMjguNjg4IDEyNi40ODZDMTI4Ljg0NiAxMjYuNTg0IDEyOC45NzUgMTI3LjMzNyAxMjguOTc1IDEyOC4xNjFDMTI4Ljk3NSAxMjkuMjM5IDEyOC44NCAxMjkuNjU4IDEyOC40OSAxMjkuNjU4QzEyOC4yMjIgMTI5LjY1OCAxMjUuNDc2IDEyOC4wNDcgMTI1LjM2MyAxMjcuODY2Wk0xMTguNjQxIDEyMS4wOTVDMTE0Ljg3MSAxMjAuNzUzIDExNS4zNzggMTE5LjMyNyAxMTYuMzYxIDExOC40OTlDMTE2LjkyOCAxMTguMDIxIDExNy4zNCAxMTcuNzc5IDExNy45MTMgMTE3LjY2N0MxMTcuOTEzIDExNy42NjcgMTE3Ljg3MiAxMTkuMDQ2IDExOC42NjYgMTE5LjMxOUMxMTkuMjk3IDExOS41MzYgMTIwLjI3OCAxMTkuMDY0IDEyMC40NzEgMTE4LjY3NUMxMjIuMTg4IDExNS4yMDEgMTIxLjQxNSAxMTguNjk5IDEyMS43OTQgMTE5LjUzMkMxMjMuMjM4IDEyNC4wMDcgMTIwLjE4OCAxMjEuNzMgMTE4LjY0MSAxMjEuMDk1Wk0xMDQuOTEyIDEyMS4yM0MxMDQuMjQyIDEyMC40ODcgMTAzLjY5MyAxMTkuNzI5IDEwMy42OTMgMTE5LjU0NEMxMDMuNjkzIDExOS4xMTYgMTA0Ljc0NiAxMTkuMTEyIDEwNS44NjMgMTE5LjUzN0MxMDYuNTkzIDExOS44MTQgMTEwLjM0NyAxMjAuMDA2IDExMC4zNDcgMTIxLjE1M0MxMTAuMzQ3IDEyMS44OTkgMTA4Ljg5IDEyMy43NjIgMTA4LjcyNiAxMjMuNzYyQzEwOC41NjMgMTIzLjc2MiAxMDUuNTgzIDEyMS45NzIgMTA0LjkxMiAxMjEuMjNaIiBmaWxsPSIjQTZCM0MyIi8+CjxwYXRoIGQ9Ik0xMTguNDYzIDEyMS4wMDhDMTE4LjQ0MiAxMjEuMTQgMTE4LjQxNiAxMjEuMjk5IDExOC4zOTcgMTIxLjQxOUMxMTguMzcgMTIxLjU5MiAxMTguMzQ1IDEyMS43NTUgMTE4LjMyMyAxMjEuOTAzQzExOC4zIDEyMi4wNTEgMTE4LjI4MSAxMjIuMTg1IDExOC4yNjMgMTIyLjMwOEMxMTguMjQ1IDEyMi40MzEgMTE4LjIyOSAxMjIuNTQyIDExOC4yMTYgMTIyLjYzOUMxMTguMjAzIDEyMi43MzYgMTE4LjE5MSAxMjIuODIgMTE4LjE4MyAxMjIuODlDMTE4LjE3NSAxMjIuOTYxIDExOC4xNjggMTIzLjAxOCAxMTguMTY1IDEyMy4wNjFDMTE4LjE2MyAxMjMuMDgzIDExOC4xNjQgMTIzLjEgMTE4LjE2MyAxMjMuMTE1QzExOC4xNjIgMTIzLjEyOSAxMTguMTYyIDEyMy4xNCAxMTguMTYzIDEyMy4xNDhDMTE4LjE2MyAxMjMuMTUyIDExOC4xNjQgMTIzLjE1NiAxMTguMTY1IDEyMy4xNThMMTE4LjE2NyAxMjMuMTZDMTE4LjE2OCAxMjMuMTYgMTE4LjE3IDEyMy4xNTkgMTE4LjE3MSAxMjMuMTU4QzExOC4xNzIgMTIzLjE1NyAxMTguMTczIDEyMy4xNTUgMTE4LjE3NSAxMjMuMTUyQzExOC4xNzggMTIzLjE0NSAxMTguMTgyIDEyMy4xMzUgMTE4LjE4NyAxMjMuMTIxQzExOC4xOTIgMTIzLjEwNyAxMTguMTk2IDEyMy4wODkgMTE4LjIwMiAxMjMuMDY3QzExOC4yMTMgMTIzLjAyNSAxMTguMjI4IDEyMi45NjkgMTE4LjI0NSAxMjIuODk3QzExOC4yNjIgMTIyLjgyNSAxMTguMjgyIDEyMi43MzggMTE4LjMwNSAxMjIuNjM1QzExOC4zMjcgMTIyLjUzMyAxMTguMzUxIDEyMi40MTUgMTE4LjM3OSAxMjIuMjgxQzExOC40MDYgMTIyLjE0OCAxMTguNDM4IDEyMS45OTggMTE4LjQ3MSAxMjEuODMzQzExOC41MTYgMTIxLjYxMyAxMTguNTczIDEyMS4zMjEgMTE4LjYyOCAxMjEuMDQzTDExOC40NjMgMTIxLjAwOFoiIGZpbGw9IiM0QzlDQkIiLz4KPHBhdGggZD0iTTMwLjk5ODYgMTQyLjY3M0MyNi40NDg5IDE0MC45MTYgMjEuNDY2NCAxMzYuODA1IDE4Ljc5MjEgMTMyLjYwM0MxNS40MDUzIDEyNy4yODEgMTQuNDA1MiAxMjMuNDE5IDE1LjA2MzYgMTE4LjIwM0MxNS42ODA2IDExMy4zMTUgMTcuMzU1OCAxMTAuNTU3IDIyLjIyMzIgMTA2LjQxM0MyOS4wNDE3IDEwMC42MDggMzQuNDg0NiA5Ny45Nzk5IDQwLjU5NDcgOTcuNTQxNUM0OS44MjEyIDk2Ljg3OTQgNTUuMDk3OCA5OS44MTA5IDU3LjY0NjUgMTA3LjAxNUM1OC41MDQzIDEwOS40MzkgNTguNjg1NiAxMTUuNTcgNTguMDU0NCAxMjAuODA5QzU3LjgzMTEgMTIyLjY2MyA1Ny40Nzg5IDEyNi4wNTUgNTcuMjcxOCAxMjguMzQ4QzU2LjU0NjcgMTM2LjM3NSA1NC40MTQ3IDE0MC40MTcgNDkuNjI1MyAxNDIuODQzQzQ4LjI5NzkgMTQzLjUxNiA0Ny43MTk5IDE0My41NjEgNDAuNjkyNCAxNDMuNTM5QzMzLjUxNTEgMTQzLjUxNiAzMy4wODEgMTQzLjQ3OCAzMC45OTg2IDE0Mi42NzNaTTAuMDc5NTc2NSAxMDQuOTY1QzAuMDgxNTcyMyAxMDMuODUzIDAuMTQ0NDAzIDEwMy40MzggMC4yMTk0MjcgMTA0LjA0NEMwLjI5NDQ1MSAxMDQuNjUgMC4yOTI4OTkgMTA1LjU2IDAuMjE1OTk2IDEwNi4wNjdDMC4xMzkwNzkgMTA2LjU3MyAwLjA3NzY4MzggMTA2LjA3OCAwLjA3OTU3NjUgMTA0Ljk2NVpNMC4wMDE3MzM1MSAxMDIuMjRDMC4wMTc4OTkyIDEwMS44NDggMC4wOTc3ODk3IDEwMS43NjggMC4yMDUzOTggMTAyLjAzN0MwLjMwMjc3MiAxMDIuMjggMC4yOTA3OTcgMTAyLjU3MSAwLjE3ODc4NyAxMDIuNjgzQzAuMDY2Nzc0OCAxMDIuNzk1IC0wLjAxMjkwMjEgMTAyLjU5NiAwLjAwMTczMzUxIDEwMi4yNFpNMC4wMjgzNDQ0IDc1LjEzMjZDMC4wMjgzNDQ0IDc0LjY2OTEgMC4xMDQ4NTcgNzQuNDc5NSAwLjE5ODM2OSA3NC43MTEyQzAuMjkxODg0IDc0Ljk0MyAwLjI5MTg4NCA3NS4zMjIyIDAuMTk4MzY5IDc1LjU1MzlDMC4xMDQ4NTcgNzUuNzg1NyAwLjAyODM0NDQgNzUuNTk2MSAwLjAyODM0NDQgNzUuMTMyNloiIGZpbGw9InVybCgjcGFpbnQyX3JhZGlhbF82NzlfMTkxMCkiLz4KPHBhdGggZD0iTTM1LjMxMzMgMTEyLjU0NEwzNy45IDExMS4wOTFMMzkuMzUxNyAxMjEuNDZMMzkuMjQ4IDEyOC45MjZMMzUuMzcwMyAxMzAuODUxTDMyLjkyMjkgMTI2LjQzN0wzNS4zMTMzIDExMi41NDRaIiBmaWxsPSIjMzA2MEFEIi8+CjxwYXRoIGQ9Ik0zOC43Mjk1IDExNi42OUwzOS4zNTE2IDEyNC41NzFMNDMuMjkxOSAxMjUuOTE5TDQzLjM5NTYgMTA5LjIyNUw0MC4xODEyIDExMC4wNTRMMzguNzI5NSAxMTYuNjlaIiBmaWxsPSIjNjA2MzY4Ii8+CjxwYXRoIGQ9Ik0zNS4zNjk5IDEzMC44NTFDMzMuMDMyNCAxMjkuNzMgMjkuNzMwMSAxMjYuNDA4IDI4LjY2MjEgMTI0LjEwM0MyOC4wMzMyIDEyMi43NDUgMjcuNjM5NyAxMjIuMzEgMjYuODkyMyAxMjIuMTQ2QzI1Ljc4NzkgMTIxLjkwMyAyNS44MDg0IDEyMS45NzEgMjYuMzIxOCAxMjAuMjU4QzI2LjU5MzYgMTE5LjM1IDI2LjkwMzIgMTE4Ljk1NCAyNy4zNDA2IDExOC45NTRDMjcuNjg2MiAxMTguOTU0IDI5LjE1OTcgMTE3Ljg0MSAzMC42MTQ5IDExNi40OEMzMi4wNzAyIDExNS4xMTkgMzMuNzIyNiAxMTMuNjc3IDM0LjI4NjkgMTEzLjI3NUwzNS4zMTI5IDExMi41NDRMMzUuNjc2NyAxMTQuMjQxQzM3LjA0NDMgMTIwLjYxOCAzNy4wMjcgMTI0LjI2NiAzNS42MTk4IDEyNi4yNDNDMzQuNTUyMSAxMjcuNzQ0IDM0Ljg0MTcgMTI4LjM5MyAzNi41NzkxIDEyOC4zOTNDMzcuMzk3IDEyOC4zOTMgMzcuODIyMiAxMjguMTg1IDM4LjEzMDUgMTI3LjYzNEMzOC42NzQ4IDEyNi42NjMgMzguNjMxNSAxMjAuOTY4IDM4LjA0NTQgMTE2LjQyNkMzNy4zNDQ2IDExMC45OTUgMzcuMzEyNyAxMTEuMTY3IDM5LjE1NzggMTEwLjM5OEM0MC4wMzYgMTEwLjAzMSA0MC44MTggMTA5Ljc5NyA0MC44OTU1IDEwOS44NzdDNDAuOTczMSAxMDkuOTU2IDQwLjgxOTEgMTExLjIzNSA0MC41NTM0IDExMi43MThDMzkuNjA2NyAxMTguMDAzIDM5LjkzMTUgMTIzLjIzOCA0MS4yNzUyIDEyNC4zNTNDNDEuNjg1MSAxMjQuNjkzIDQxLjcyMzMgMTIzLjk2OSA0MS41NDUyIDExOS4yMzJDNDEuMzE1NiAxMTMuMTIzIDQxLjUxNzQgMTEwLjkxOSA0Mi40MDc1IDEwOS44MkM0Mi45MTk4IDEwOS4xODggNDMuMjEzIDEwOS4xMTYgNDQuNDk1OSAxMDkuMzFDNDYuOTU2NiAxMDkuNjgyIDQ2Ljk1MDYgMTA5LjY2OCA0Ni4yMDk5IDExMy4yNTRDNDUuODQ5NiAxMTQuOTk5IDQ1LjQ0ODMgMTE3LjIyNCA0NS4zMTggMTE4LjJMNDUuMDgxMiAxMTkuOTczTDQ1Ljk0NDYgMTE5LjY0NUM0Ni40MTk1IDExOS40NjUgNDcuNDQxMyAxMTguOTk0IDQ4LjIxNTMgMTE4LjU5OUw0OS42MjI2IDExNy44ODFMNDkuMzg5MyAxMTguNjdDNDkuMjYxIDExOS4xMDUgNDkuMDY3MiAxMjAuNzAxIDQ4Ljk1ODcgMTIyLjIxOEw0OC43NjE0IDEyNC45NzZMNDcuMTA5NSAxMjQuMzI1QzQ2LjIwMSAxMjMuOTY3IDQ1LjMxMTYgMTIzLjY3NCA0NS4xMzMxIDEyMy42NzRDNDQuOTQxIDEyMy42NzQgNDQuODA4NSAxMjUuNTMyIDQ0LjgwODUgMTI4LjIyNEM0NC44MDg1IDEzMS44OTEgNDQuNzE3MSAxMzIuNzc1IDQ0LjMzODYgMTMyLjc3NUM0MS4wMTk4IDEzMi4yOTUgMzguNzg0MiAxMzIuNDU4IDM1LjM2OTkgMTMwLjg1MVoiIGZpbGw9IiNFRUMyM0IiLz4KPHBhdGggZD0iTTQxLjE2MzkgMTMyLjQxOEM0MS4xNjM5IDEzMi4yODMgNDEuMzQwOCAxMzAuNTE1IDQxLjY2OTIgMTI4LjE0NUM0MS45OTc3IDEyNS43NzUgNDIuMzU3NyAxMjIuOTYxIDQyLjQ2OTIgMTIxLjg5M0M0Mi41ODA4IDEyMC44MjQgNDIuNzk2OSAxMjAuMDI3IDQyLjk0OTUgMTIwLjEyMUM0My42NDQzIDEyMC41NTEgNDQuMjkzOCAxMjkuMzc1IDQ0LjA0NzQgMTMxLjQyQzQzLjg5MDkgMTMyLjcxOSA0My42NTk1IDEzMi43MzcgNDMuMzAxOSAxMzIuNjc3QzQyLjY2MDIgMTMyLjU3IDQxLjkxMDEgMTMyLjUxMyA0MS4xNjM5IDEzMi40MThaTTMyLjU5NjkgMTIyLjczNUMzMS42ODEzIDEyMi41ODMgMzAuODI0MiAxMjEuNDQyIDMwLjgyNDIgMTIwLjM3NEMzMC44MjQyIDExOS40MDcgMzIuMjM4MSAxMTcuOTM3IDMzLjE2ODUgMTE3LjkzN0MzNC42NjY2IDExNy45MzcgMzUuODM5MiAxMjAuNTEzIDM0Ljk0ODQgMTIxLjg0N0MzNC41NTE3IDEyMi40NDEgMzMuNDE0NyAxMjIuODcgMzIuNTk2OSAxMjIuNzM1WiIgZmlsbD0id2hpdGUiLz4KPHBhdGggZD0iTTExNC4wMjEgMTM3LjQ5OEMxMDcuMzE4IDEzNS43NCAxMDYuMjczIDEzNC4xMDQgMTA2LjYwOSAxMjUuODkxQzEwNi43MTYgMTIzLjI4MSAxMDYuODg1IDEyMC41MzkgMTA2Ljk4NiAxMTkuNzk3QzEwNy41NDUgMTE1LjY2NCAxMDguMzQyIDExMi4xNzMgMTA5LjA4NiAxMTAuNTk5QzEwOS43NDQgMTA5LjIwNiAxMDkuODQyIDEwOC43MTIgMTA5LjUzNSAxMDguMzQyQzEwOC40NSAxMDcuMDM1IDExMC4zNzEgMTA0Ljk5NSAxMTEuOTEgMTA1LjgxOUMxMTIuNDU3IDEwNi4xMTEgMTEyLjkxOCAxMDYuMDc3IDExMy45NDUgMTA1LjY2NUMxMTcuMDYyIDEwNC40MTkgMTIzLjExNyAxMDUuNTcxIDEyNi44MTQgMTA4LjExNUMxMjguMTQ5IDEwOS4wMzMgMTI4LjQ4NSAxMDkuMTM2IDEyOS4wMzggMTA4Ljc5MUMxMzAuNTIzIDEwNy44NjMgMTMyLjQ1MSAxMTAuNTczIDEzMS4yNSAxMTEuOUMxMzAuNzg3IDExMi40MTIgMTMwLjc1NSAxMTIuNzA1IDEzMS4wNjUgMTEzLjU5NkMxMzEuODUgMTE1Ljg0NiAxMzEuNzk0IDExNi4xMDQgMTI4LjAwNiAxMjcuNzE5QzEyNS41OCAxMzUuMTYgMTI0LjU0MyAxMzcuMzY1IDEyMy4yNjQgMTM3LjgxMUMxMjEuNjUzIDEzOC4zNzMgMTE2LjcyMyAxMzguMjA2IDExNC4wMjEgMTM3LjQ5OFpNMTIyLjU0MSAxMzUuMjI5QzEyMy4wMDMgMTM0LjU0MSAxMjMuMzU5IDEzMy4zMTMgMTIzLjUxMyAxMzEuODc4QzEyMy43OTQgMTI5LjI1IDEyMy44MTkgMTI5LjI4MyAxMjAuMzE3IDEyNy42MDFDMTE3LjY4NSAxMjYuMzM4IDExMy44MzMgMTI1Ljk3MiAxMTEuODU1IDEyNi43OThDMTEwLjgyOSAxMjcuMjI3IDExMC41OTUgMTI3LjU1NiAxMTAuMDU5IDEyOS4zMjdDMTA5LjExNiAxMzIuNDQzIDEwOS43MDMgMTM0LjMwOCAxMTEuODMyIDEzNC45NThDMTEzLjEzOSAxMzUuMzU3IDEyMC4wOSAxMzYuNDMgMTIwLjk4IDEzNi4zN0MxMjEuNTA0IDEzNi4zMzUgMTIyLjA4MiAxMzUuOTEyIDEyMi41NDEgMTM1LjIyOVpNMTE5LjQ1MSAxMjIuNDJDMTE5LjYzIDEyMi4wODEgMTE5LjY4NSAxMjEuNTU2IDExOS42ODUgMTIxLjU1NkMxMjAuMTU2IDEyMS42NTQgMTIwLjQ4IDEyMS42ODMgMTIwLjcyNiAxMjEuNzE5QzEyMS4xMDUgMTIxLjc3NCAxMjEuNTY0IDEyMS41ODcgMTIxLjc0NSAxMjEuMzAzQzEyMi4xNDkgMTIwLjY3MSAxMjAuNDc0IDExNy43MDEgMTE5LjU0OCAxMTcuNDA4QzExOC42NjggMTE3LjEyOCAxMTUuOTM1IDExOC44MjcgMTE1LjkzNSAxMTkuNjUzQzExNS45MzUgMTIwLjAyMiAxMTYuNTUzIDEyMC40MTEgMTE2LjgzMSAxMjAuNTE4QzExNi45ODIgMTIwLjU3NiAxMTcuMjUyIDEyMC43MzggMTE3LjU3NiAxMjAuOTM4QzExNy4zOCAxMjEuMzUxIDExNy4zNDYgMTIxLjQ3OCAxMTcuMzEyIDEyMS43MjRDMTE3LjIwNCAxMjIuNDk4IDExNy40MzUgMTIyLjY4NiAxMTguNDE2IDEyMi45MTJDMTE5LjE1MSAxMjIuOTkxIDExOS4yNSAxMjIuODQ1IDExOS40NTEgMTIyLjQyWk0xMjcuMjY1IDExNi44MzZDMTMwLjkzMyAxMTMuNDc1IDEyNy4wNzQgMTA3LjQxIDEyMi43MDUgMTA5LjY3QzEyMS42MjYgMTEwLjIyOCAxMjAuNjU0IDExMi4wMTkgMTIwLjY1NCAxMTMuNDQ5QzEyMC42NTQgMTE1LjY2NCAxMjIuODMxIDExNy45MzcgMTI0Ljk1NyAxMTcuOTQxQzEyNS43MzMgMTE3Ljk0MyAxMjYuNDE0IDExNy42MTcgMTI3LjI2NSAxMTYuODM2Wk0xMTcuMTQxIDExNS42MjJDMTE5LjAzNSAxMTQuNDY4IDExOS44MDQgMTExLjc5NSAxMTguODAyIDEwOS44NTdDMTE4LjA4MiAxMDguNDY0IDExNy4wMDggMTA3LjgzIDExNS4zNjYgMTA3LjgzQzExMS41MDggMTA3LjgzIDEwOS40ODUgMTEzLjIzNCAxMTIuNDY1IDExNS41NzhDMTEzLjU3IDExNi40NDggMTE1Ljc1NCAxMTYuNDY4IDExNy4xNDEgMTE1LjYyMloiIGZpbGw9IiMwOTk2RDEiLz4KPHBhdGggb3BhY2l0eT0iMC45IiBkPSJNMzMuMTc1NyAyNkwzMi4zODEgMjcuMjE5QzMxLjk0NDkgMjcuODg5IDMxLjI4MzYgMjkuMzI0OCAzMC45MDkxIDMwLjQwODhDMzAuNTM0NyAzMS40OTI4IDI5Ljk5MjIgMzIuOTcwMSAyOS43MDQzIDMzLjY5MjhDMjkuNDE2MyAzNC40MTU0IDI5LjE3MzMgMzYuMTE1IDI5LjE2MzggMzcuNDdDMjkuMTQzNiA0MC4zNTMgMjguNzQ5NCA0MS44ODg4IDI3LjYzNjQgNDMuNDIyNUMyNi45MjEyIDQ0LjQwOCAyNi42NzUxIDQ0LjUyMjMgMjUuNDUzMiA0NC40NDFDMjMuODk5NSA0NC4zMzc3IDIyLjgxNTMgNDUuMDgxNSAyMS4yMTcgNDcuMzQ4MUMyMC43MTY1IDQ4LjA1OCAyMC4xMzY3IDQ4LjYzOTMgMTkuOTI4OSA0OC42MzkzQzE5LjcyMTEgNDguNjM5MyAxOS4xODA1IDQ4LjIzMTkgMTguNzI2MiA0Ny43MzUxQzE4LjI3MTkgNDcuMjM4MiAxNy45OTQ0IDQ3LjAxMTUgMTguMTExIDQ3LjIyOThDMTguNDkgNDcuOTM5NiAxNy41OTA0IDUzLjI2MTUgMTYuODM3OCA1NC43NjQyQzE1Ljk2NyA1Ni41MDI4IDE0LjM4NzIgNTcuNzIzMyAxMi41MzU0IDU4LjA4NDNDMTEuMDA1OSA1OC4zODI2IDExLjIyODMgNTguMDkgOC43MTc5NSA2My4wOTQ2QzYuNjYzMzQgNjcuMTkwNyA1Ljk2OTM1IDY3LjgyNjQgMi42NTUzMyA2OC42NTQyQzEuNDk3MTcgNjguOTQzNCAwLjQ1ODc1OSA2OS43MTQyIDAgNzAuNTI4N1Y3Mi44MjYzQzAuMzU2MTkzIDcyLjc0NzMgMC44Mjg3MjggNzIuNjIwMyAxLjYxNDk5IDcyLjM4NzNDMi45OTQ4NiA3MS45NzgyIDQuODIzNDggNzEuNDgwNiA1LjY3ODEgNzEuMjgwNkM3LjcyOTM0IDcwLjgwMDYgOS4zOTM1MSA2OS4yMTk1IDkuOTIyNzkgNjcuMjQ2N0MxMC4yNDI0IDY2LjA1NTQgMTAuNzAzMyA2NS40MjQxIDExLjk5OTIgNjQuNDA1OEMxMy4yODQgNjMuMzk2MiAxMy43NTY1IDYyLjc1NDYgMTQuMDY3MSA2MS41OTY5QzE0LjM3MzkgNjAuNDUzMiAxNC43MDExIDYwLjAwNCAxNS40NDA3IDU5LjcwNjNDMTcuMDE1MyA1OS4wNzI0IDIwLjY0NDQgNTUuODY1NSAyMS4yMzYzIDU0LjU4NThDMjEuNTM3MyA1My45MzQ5IDIyLjExOTggNTIuNTM5OSAyMi41Mjg3IDUxLjQ4NjJDMjMuMjYzNCA0OS41OTMyIDI1LjAxOTUgNDcuMzI2MSAyNS43NTAxIDQ3LjMyNjFDMjUuOTU3NCA0Ny4zMjYxIDI3LjEwOSA0Ni43MjQ4IDI4LjMwOTMgNDUuOTkwOEMzMS4yMzc2IDQ0LjIwMDMgMzEuNTk5MSA0My40MTM2IDMxLjU5OTEgMzguODYxNEMzMS41OTkxIDM1LjY4MTQgMzEuNjg3IDM1LjE1NTcgMzIuMzQ2OCAzNC4zNjg0QzMzLjAwMzcgMzMuNTg0NyAzMy4wOTk3IDMzLjAyNDkgMzMuMTM1MSAyOS43MzkxTDMzLjE3NTcgMjZaTTI0LjMzMzggNjMuNTk1OEMyNC4yMDM3IDYzLjYxMzUgMjQuMDczOCA2My42NjE4IDIzLjkyNTggNjMuNzM2MkMyMi45Mzk3IDY0LjIzMTUgMjIuNjEzNiA2NC45MTAyIDIxLjU5NzMgNjguNTg2QzIxLjAxNzkgNzAuNjgxNSAyMC40MjEyIDcyLjA4MTkgMTkuODc5OCA3Mi42MTc4QzE5LjQyMzUgNzMuMDY5NSAxOC45MzgzIDczLjg4MjMgMTguODAxIDc0LjQyNDJDMTguMTgxIDc2Ljg3MTUgMTYuNzQ1NCA3OC42MDczIDE0LjQ0OTUgNzkuNjg3MUMxMi40NTU1IDgwLjYyNDkgMTAuNjQwNyA4Mi4xMTc5IDkuMTExMDIgODQuMDc5OEM4LjMyMTA5IDg1LjA5MjkgNy40ODc1MiA4Ni4wMjkgNy4yNTY3OCA4Ni4xNjA5QzYuNzYxNjQgODYuNDQ0IDUuNzA4OTkgODguODk0MSA0LjYwNzg1IDkyLjMyOEMzLjcxMDkyIDk1LjEyNSAxLjcxMDY1IDk3LjQ2ODIgMC40Nzg1MTQgOTcuMTY1OEMwLjI1NDkzMSA5Ny4xMTA5IDAuMTA2OTExIDk3LjExMzcgMCA5Ny4yOTAxVjEwNC40OTJDMC41NTY0MTEgMTAzLjY2NCAxLjIzNDIgMTAyLjYyOSAyLjU2MzQ3IDEwMC41NEM0LjM0NTExIDk3LjczOTcgNi40NjM4NSA5NC40ODY3IDcuMjcxNzMgOTMuMzEyNEM4LjA3OTYgOTIuMTM4MSA5LjM1Njg3IDkwLjE0ODggMTAuMTA4NiA4OC44OTE2QzExLjIzNDIgODcuMDA5MiAxMi4xMDE2IDg2LjEyNDMgMTUuMDI4NCA4My44ODMzTDE4LjU4MzEgODEuMTYyN0wyMC40MDUzIDc1Ljk4ODFDMjMuMTg5NSA2OC4wODE3IDIzLjk1ODUgNjcuMTk5NyAyNy43NjI0IDY3LjU1MTVDMjkuNDYwNSA2Ny43MDg1IDMwLjYzMzQgNjcuNjE5NiAzMi4wNzM0IDY3LjIyNjdDMzQuMTgyNiA2Ni42NTEyIDM0LjQ3OTkgNjYuMTg3MiAzMy4zNTUxIDY1LjIzMThDMzIuNjU1NCA2NC42Mzc2IDMxLjMxNTMgNjQuNjQgMjguMjM4OCA2NS4yNDE4QzI3LjMzOTcgNjUuNDE3NyAyNi45MzQzIDY1LjI3NTcgMjUuODkzMiA2NC40MTc4QzI1LjEwNTEgNjMuNzY4NCAyNC43MjQxIDYzLjU0MjYgMjQuMzMzOCA2My41OTU4Wk0zMS43MTY2IDgzLjI1OThDMzEuNjYwNSA4My4yNzQ5IDMxLjU5MjEgODMuMzEyNiAzMS41MTE1IDgzLjM3MjFDMzEuMjcwOSA4My41NSAzMC42NTM2IDgzLjc5IDMwLjEzNzkgODMuOTA1NEMyOS4zODc2IDg0LjA3MzQgMjkuMDQ5OCA4NC40OTMyIDI4LjQ0ODIgODYuMDA0NUMyOC4wMzQ3IDg3LjA0MzIgMjcuNDkxNSA4OC4xMTUzIDI3LjI0MTIgODguMzg2M0MyNi45OTA5IDg4LjY1NzMgMjYuNDc1IDg5LjU1MSAyNi4wOTQxIDkwLjM3MzJDMjQuOTQyMiA5Mi44NTkyIDIzLjUyODIgOTQuMDYxOCAxOS45MDExIDk1LjY0MjFDMTYuNjEwNyA5Ny4wNzU2IDE2LjU2NyA5Ny4xMDg2IDE1LjY2NzEgOTguOTgwMkMxNS4xMDI3IDEwMC4xNTQgMTQuNjc5MiAxMDEuNzU3IDE0LjU0OTkgMTAzLjIwNUMxNC4yNzMxIDEwNi4zMDIgMTMuNTk2MiAxMDcuMDMxIDkuNTIxMTggMTA4LjYxNkM1LjkxNDk3IDExMC4wMTkgMy41NzcwMiAxMTEuNzQ1IDMuNDE1ODMgMTEzLjEyM0MzLjI0MDQxIDExNC42MjMgMi41OTgxMyAxMTUuMDg0IDAuOTI5MjYgMTE0LjkwN0MwLjUzOTUyOSAxMTQuODY2IDAuMjQ0MzMyIDExNC44NTEgMCAxMTQuODgxVjEyMUMxLjAyNjU4IDExOS42NTggMi43MTkyMiAxMTguMzYxIDYuMjI0OTcgMTE2LjIwNkMxMC4zNjA0IDExMy42NjUgMTIuNzA2NSAxMTEuNTc1IDEzLjU3NzkgMTA5LjY1NEMxMy45MjYgMTA4Ljg4NyAxNC42NTQ3IDEwNy4zOTMgMTUuMTk3MSAxMDYuMzM0QzE1LjczOTUgMTA1LjI3NSAxNi4yODQ1IDEwMy45MDEgMTYuNDA4NCAxMDMuMjgxQzE2LjczMjQgMTAxLjY2IDE4LjMwMzIgMTAwLjM0OSAyMC42NjU5IDk5LjcyNkMyNS4xNzcgOTguNTM3MiAyNS43NDYxIDk4LjAxNjMgMjYuMzUyNSA5NC41MzU0QzI2Ljg1NjEgOTEuNjQ1MiAyOC42MTIgODguMDY2MSAzMC41MjY3IDg2LjAyNDZDMzEuMzA5NSA4NS4xOSAzMS45NDk1IDg0LjE4IDMxLjk0OTUgODMuNzc5MUMzMS45NDk1IDgzLjM3NjYgMzEuODg0OSA4My4yMTQ2IDMxLjcxNjYgODMuMjU5OFoiIGZpbGw9IiMzNTkxMzYiLz4KPHBhdGggZD0iTTguMDgxNzcgNzQuNDA4M0M2LjAzNDE5IDczLjg3MDUgNS44OTQ0MyA3My43MzAyIDYuMDQyNzEgNzIuMzYxNUM2LjEzMDI1IDcxLjU1MzMgNi4yNDAxNyA3MC44NTc1IDYuMjg2OTYgNzAuODE1NEM2LjMzMzc2IDcwLjc3MzMgNy43MDQ0MyA3MS4wNjg3IDkuMzMyOTIgNzEuNDcxOUMxMS45NzY4IDcyLjEyNjQgMTIuNDg2NiA3Mi4xMjM4IDE0LjA5NDcgNzEuNDQ3NEMxNi40MDQxIDcwLjQ3NiAxNy4yMzgzIDY4Ljc3NTMgMTYuNDAyMiA2Ni43NDMxQzE1Ljk0MjQgNjUuNjI1NyAxNC45MjA0IDY0LjgyODEgMTIuMjM4NCA2My40OTRDNy42NDE3IDYxLjIwNzUgNi41MzA2MiA1OS45OTkgNi4yNzQ2NSA1Ny4wMDc2QzUuODkzMzcgNTIuNTUxOSA4Ljg0Njg5IDQ5Ljk4MzYgMTQuMzUyMiA0OS45ODM2QzE1Ljk3MzMgNDkuOTgzNiAxNy44NDg3IDUwLjE5MzcgMTguNTE5NiA1MC40NTA1QzE5LjYzNjggNTAuODc4MSAxOS42OTM3IDUxLjAzOTkgMTkuMTk0NSA1Mi4zNzEyQzE4LjcxODIgNTMuNjQxNiAxOC41MDI3IDUzLjc2ODggMTcuNDg2MSA1My4zNzk2QzE1LjUxOTkgNTIuNjI3IDEyLjE5NDggNTIuODU1NCAxMS4wMDU4IDUzLjgyNDdDOS43OTA1OCA1NC44MTUzIDkuNTU3MzYgNTYuOTI1MiAxMC41MzI0IDU4LjEwNzlDMTAuODcyNyA1OC41MjA3IDEyLjkwODcgNTkuNzc5IDE1LjA1NjkgNjAuOTA0MUMxOC40NjI4IDYyLjY4ODEgMTkuMDgyIDYzLjIyMTQgMTkuODk1MSA2NS4wNzE5QzIxLjExNTIgNjcuODQ4NyAyMC42Nzg4IDcwLjM0MzcgMTguNjA3OCA3Mi40MzIzQzE2LjE2MDcgNzQuOTAwMiAxMi41NDEzIDc1LjU3OTYgOC4wODE3NyA3NC40MDgzWk0yOC45MzI4IDc0LjM4MjNDMjUuMjM2NSA3Mi45OTU4IDIzLjcxMzggNzAuNTQwNSAyMy43MTM4IDY1Ljk2NjdDMjMuNzEzOCA2Mi41MDM0IDI0LjU1ODggNjAuNDY5NyAyNi44MzMzIDU4LjQ1OTNDMjguMzk2NCA1Ny4wNzc2IDI4Ljk4NjcgNTYuODY4NiAzMS4zMjY4IDU2Ljg2ODZDMzIuODE2OCA1Ni44Njg2IDM0LjY4MjEgNTcuMjEyOSAzNS41MDA1IDU3LjYzODlDMzcuMzE2OSA1OC41ODQ2IDM4LjgzMTkgNjEuNzQyNiAzOC44NDY2IDY0LjYxNDNMMzguODU3MyA2Ni43MDQ0SDMyLjk0MjNIMjcuMDI3M0wyNy4zNDIgNjguMDU2OUMyNy41MTUgNjguODAwNyAyOC4zMzU0IDcwLjAzOTEgMjkuMTY1IDcwLjgwOUMzMC41NzczIDcyLjExOTYgMzAuODk0OSA3Mi4xOTc5IDM0LjE1NDcgNzIuMDM4NUMzNy4zOTM4IDcxLjg4MDEgMzcuNjQ2MyA3MS45NDA5IDM3Ljc4NCA3Mi45MTIyQzM3Ljk2MjYgNzQuMTcyNSAzNy4xODgyIDc0LjUyMzYgMzMuNTI0MiA3NC44NDM0QzMxLjgwODUgNzQuOTkzMiAzMC4xMDU2IDc0LjgyMjIgMjguOTMyOCA3NC4zODIzWk0zNS4wODgxIDYyLjUyNDJDMzQuOTU1OCA2MC42NzMgMzMuMzY0NCA1OS4zMjc2IDMxLjMwNzIgNTkuMzI3NkMyOS43MDg2IDU5LjMyNzYgMjguNzI4MSA2MC4wOTA5IDI3LjY2MiA2Mi4xNjU1QzI2Ljc0NTcgNjMuOTQ4NiAyNy40MTIgNjQuMjk2NSAzMS40NDg4IDY0LjE0MjVMMzUuMTkzNiA2My45OTk2TDM1LjA4ODEgNjIuNTI0MlpNNDQuMzkxNCA3NC4zNzA4QzQyLjU3MTEgNzMuNTQ2OCA0MS43MDU5IDcyLjEyODYgNDEuNjQyMSA2OS44NjQ3QzQxLjU0MTYgNjYuMjk0NCA0NC4xMTEgNjQuMTQ2OCA0OS4yNTc2IDYzLjQ5OTRDNTEuODQxNCA2My4xNzQ0IDUyLjA0OTUgNjMuMDU3NyA1MS43NTkzIDYyLjA5NjlDNTAuOTY1IDU5LjQ2NjYgNDguMjQyMyA1OC41NjQ0IDQ0LjkyNzYgNTkuODMzMUM0My40MzAzIDYwLjQwNjMgNDMuNDg2OCA2MC40MzEzIDQzLjA1MTQgNTkuMDAzMkM0Mi43ODAxIDU4LjExMzUgNDIuOTk4MiA1Ny44NTExIDQ0LjM5NSA1Ny4zODdDNDYuOTM2NCA1Ni41NDI3IDUxLjM3NDUgNTYuNjM0MiA1Mi43Nzc5IDU3LjU2QzU0LjkzOTIgNTguOTg1NyA1NS4zNzUxIDYwLjYwMTkgNTUuNTQ4NyA2Ny44MzI4TDU1LjcxMDYgNzQuNTc0NUw1NC4xMjMgNzQuNTc0QzUzLjAzNzcgNzQuNTczMyA1Mi41MzUzIDc0LjMzOTYgNTIuNTM1MyA3My44MzU0QzUyLjUzNTMgNzIuOTA2NiA1Mi4wNTI4IDcyLjkwMjQgNTAuNzUyIDczLjgxOTZDNDkuMjAxMyA3NC45MTMgNDYuMTY5NyA3NS4xNzU3IDQ0LjM5MTQgNzQuMzcwOFpNNTAuOTQ3NyA3MC44MTc1QzUxLjcwOTkgNjkuOTczNSA1Mi4wNDY4IDY4Ljk4NiA1Mi4wNDY4IDY3LjU5NjZWNjUuNTkyOUw0OS44MTgyIDY1Ljg0NThDNDUuNzE5IDY2LjMxMSA0My44NzQ5IDY5LjUxMTggNDYuNTA1OCA3MS41OTUzQzQ4LjAzNzEgNzIuODA3OSA0OS4zNTk0IDcyLjU3NjMgNTAuOTQ3NyA3MC44MTc1Wk05Mi4wMDIgNzQuNTc4OEM4OC40MDI0IDczLjQ4NjQgODYuMjQxOSA3MC4yNDY2IDg2LjI0MTkgNjUuOTQxQzg2LjI0MTkgNjAuNDA0MiA4OS41MjExIDU2LjgzMjUgOTQuNjQxIDU2Ljc5MjVDOTguNjk4OSA1Ni43NjA5IDEwMS4yNjMgNTkuNDY3OSAxMDEuNTAzIDY0LjAzNjZMMTAxLjYzIDY2LjQ1ODVMOTYuMjU2MiA2Ni41ODg1QzkzLjMwMDcgNjYuNjYgOTAuNjE1OCA2Ni44MTYgOTAuMjg5NyA2Ni45MzUxQzg5LjM4MTggNjcuMjY2OCA5MC40Nzg4IDY5Ljk0NyA5Mi4wMzA2IDcxLjE4ODZDOTMuMTU5OSA3Mi4wOTIxIDkzLjc3MSA3Mi4xOTEyIDk2LjgxNzggNzEuOTY1QzEwMC4xOTMgNzEuNzE0NCAxMDAuMzE4IDcxLjc0NDkgMTAwLjUzMSA3Mi44Njg0QzEwMC43MyA3My45MTMzIDEwMC41MDkgNzQuMDgzNSA5OC4zNTc5IDc0LjU0OEM5NS42MTAzIDc1LjE0MTIgOTMuODgyOSA3NS4xNDk2IDkyLjAwMiA3NC41Nzg4Wk05Ny45NjU5IDYzLjA4MjNDOTcuOTY1OSA2MS4xMjMxIDk2LjE4NTggNTkuMzI3NiA5NC4yNDM0IDU5LjMyNzZDOTIuMjQ5NCA1OS4zMjc2IDkwLjE0OTkgNjEuMjU2MiA5MC4xNDk5IDYzLjA4NzhDOTAuMTQ5OSA2NC4yMDcgOTAuMjc5NyA2NC4yNDU1IDk0LjA1NzkgNjQuMjQ1NUM5Ny44NDE0IDY0LjI0NTUgOTcuOTY1OSA2NC4yMDg0IDk3Ljk2NTkgNjMuMDgyM1pNMTEwLjA0MiA3NC40N0MxMDYuMTQ5IDczLjQzIDEwMy45MyA2OS40MDI5IDEwNC41NCA2NC40ODYyQzEwNS4xNjYgNTkuNDQ0MiAxMDcuOTUxIDU2Ljg3MzYgMTEyLjc5MiA1Ni44NzA1QzExNS4zNzQgNTYuODY5IDExNS44MDUgNTcuMDI0OSAxMTcuMjYgNTguNDlDMTE4Ljg3OCA2MC4xMTg1IDExOS45NDEgNjIuODM0MyAxMTkuOTQ2IDY1LjM1MkwxMTkuOTQ5IDY2LjcwNDRIMTE0LjA4N0MxMDguNDg4IDY2LjcwNDQgMTA4LjIyNSA2Ni43NTAxIDEwOC4yMjUgNjcuNzIwM0MxMDguMjI1IDcwLjcxMjIgMTEwLjk1MiA3Mi4zNDQyIDExNS4zNTcgNzEuOTg4MkMxMTguMTM2IDcxLjc2MzcgMTE4LjQyMyA3MS44MzE1IDExOC42NjcgNzIuNzY3NEMxMTguOTk4IDc0LjA0NDMgMTE4Ljk2OCA3NC4wNzIgMTE2LjY5IDc0LjYwNDNDMTE0LjM2NyA3NS4xNDcgMTEyLjQzIDc1LjEwNzggMTEwLjA0MiA3NC40N1pNMTE2LjExOSA2Mi43NzAxQzExNS44NjQgNjAuODc4NyAxMTQuMTU5IDU5LjMyNzYgMTEyLjMzNSA1OS4zMjc2QzExMC41NjYgNTkuMzI3NiAxMDguMjI0IDYxLjYwMTQgMTA4LjIyNCA2My4zMTk1QzEwOC4yMjQgNjQuMjEyOCAxMDguNTI3IDY0LjI3NDYgMTEyLjI1NSA2NC4xNDI0QzExNi4yNDEgNjQuMDAxMSAxMTYuMjgzIDYzLjk4NjIgMTE2LjExOSA2Mi43NzAxWk0xMjYuODk0IDc0LjI0N0MxMjQuODI2IDczLjIyMDggMTIzLjUzMiA3MS4zNTg5IDEyMi44OTQgNjguNDg4OUMxMjEuOTc1IDY0LjM2MDUgMTIzLjA3NCA2MC40OTE0IDEyNS44MyA1OC4xNTY3QzEyNy44MjcgNTYuNDY1NyAxMzEuODM4IDU2LjMzMDUgMTMzLjgwNSA1Ny44ODc5TDEzNS4wOTIgNTguOTA3MlY1My45NTM2VjQ5SDEzNy4wNDZIMTM5VjYxLjc4NjVWNzQuNTczMUgxMzcuMjlDMTM1Ljc2MiA3NC41NzMxIDEzNS41OCA3NC40MzMxIDEzNS41OCA3My4yNTc5VjcxLjk0MjdMMTM0LjQ4MSA3Mi45NzkyQzEzMi4yMzMgNzUuMDk5OCAxMjkuNTI1IDc1LjU1MjMgMTI2Ljg5NCA3NC4yNDdaTTEzMy42NjQgNzAuNjc2NkMxMzQuOTc2IDY5LjM1NTcgMTM1LjA5MiA2OC45NTM3IDEzNS4wOTIgNjUuNzIwOUMxMzUuMDkyIDYyLjQ1NzIgMTM0Ljk4NSA2Mi4wOTQ1IDEzMy42MDcgNjAuNzA3NkMxMzIuMzY1IDU5LjQ1NzUgMTMxLjg2NSA1OS4yNjIyIDEzMC41NTQgNTkuNTE1NEMxMjYuMTQ0IDYwLjM2NzIgMTI0LjgzNSA2OC4xNDIzIDEyOC41OTggNzEuMTIyOUMxMzAuMzQ4IDcyLjUwODEgMTMxLjk4OCA3Mi4zNjM1IDEzMy42NjQgNzAuNjc2NlpNNjEuNDEwNyA2Ni4wNTg0QzYwLjAyMiA2MS4zNzU0IDU4Ljg4NTkgNTcuMzkxOSA1OC44ODU5IDU3LjIwNjJDNTguODg1OSA1Ny4wMjA2IDU5LjcyNzQgNTYuODY4NiA2MC43NTU5IDU2Ljg2ODZINjIuNjI1OUw2NC4yNTUxIDYzLjg3NjZDNjUuMTUxMiA2Ny43MzEgNjUuOTU2IDcwLjY2MzMgNjYuMDQzNyA3MC4zOTI5QzY2LjEzMTMgNzAuMTIyNCA2Ny4wNTU3IDY3LjAyNDEgNjguMDk3OCA2My41MDc4TDY5Ljk5MjYgNTcuMTE0NUg3MS41OTY5SDczLjIwMTJMNzQuODg2NSA2My4yNjE5Qzc1LjgxMzQgNjYuNjQzIDc2LjY4MzIgNjkuNzQxMiA3Ni44MTk0IDcwLjE0N0M3Ni45NTU3IDcwLjU1MjcgNzcuODQxOSA2Ny43MzEgNzguNzg5IDYzLjg3NjZMODAuNTEwOSA1Ni44Njg2SDgyLjQ1MTdDODMuOTUxNyA1Ni44Njg2IDg0LjMzMDMgNTcuMDMxNiA4NC4xMTkgNTcuNTg2QzgzLjk2ODYgNTcuOTgwNiA4Mi42OTU2IDYxLjk2NDEgODEuMjkwMiA2Ni40MzgyTDc4LjczNDkgNzQuNTczMUg3Ni45NjM1SDc1LjE5MjFMNzMuNTgxMiA2OC43OTQ1QzcyLjY5NTIgNjUuNjE2MyA3MS44NTg1IDYyLjY4NDEgNzEuNzIxNyA2Mi4yNzgzQzcxLjQ0NjUgNjEuNDYxNiA3MS41OTExIDYxLjA3MzMgNjkuMTExNCA2OS4yODYzTDY3LjUxNTIgNzQuNTczMUg2NS43MjU0SDYzLjkzNTZMNjEuNDEwNyA2Ni4wNTg0WiIgZmlsbD0id2hpdGUiLz4KPHBhdGggZD0iTTE5Ljk3NDYgMTE5LjI3NEMxOS4wMTU4IDExOC4xMTkgMjAuMzM4MSAxMTYuMTY2IDIxLjU2MDQgMTE2LjkzMkMyMi43NjMyIDExNy42ODYgMjIuMzMwNSAxMTkuNzYxIDIwLjk3MDQgMTE5Ljc2MUMyMC42NDUgMTE5Ljc2MSAyMC4xOTY5IDExOS41NDIgMTkuOTc0NiAxMTkuMjc0WiIgZmlsbD0idXJsKCNwYWludDNfcmFkaWFsXzY3OV8xOTEwKSIvPgo8cGF0aCBkPSJNMjMuODM2IDExMi42M0MyMi4xMTUyIDExMS42ODkgMjEuODAyOSAxMDkuNzk3IDIzLjE0NjUgMTA4LjQ1M0MyNC4wODU3IDEwNy41MTQgMjUuMjEgMTA3LjM5NCAyNi4yNTA3IDEwOC4xMjNDMjcuODU3MSAxMDkuMjQ5IDI3LjQ2NTUgMTEyLjAxMyAyNS42MDc5IDExMi42NkMyNS42MDc5IDExMi42NiAyNC40NzQ1IDExMi45NzMgMjMuODM2IDExMi42M1oiIGZpbGw9InVybCgjcGFpbnQ0X3JhZGlhbF82NzlfMTkxMCkiLz4KPHBhdGggZD0iTTMzLjc4NTUgMTIwLjAzNkMzMy43ODU1IDEyMC4xODggMzMuNzU1NyAxMjAuMzM4IDMzLjY5NzYgMTIwLjQ3OUMzMy42Mzk1IDEyMC42MTkgMzMuNTU0NCAxMjAuNzQ2IDMzLjQ0NzEgMTIwLjg1M0MzMy4zMzk4IDEyMC45NjEgMzMuMjEyNCAxMjEuMDQ2IDMzLjA3MjMgMTIxLjEwNEMzMi45MzIxIDEyMS4xNjIgMzIuNzgxOCAxMjEuMTkyIDMyLjYzMDEgMTIxLjE5MkMzMi40NzgzIDEyMS4xOTIgMzIuMzI4MSAxMjEuMTYyIDMyLjE4NzkgMTIxLjEwNEMzMi4wNDc3IDEyMS4wNDYgMzEuOTIwMyAxMjAuOTYxIDMxLjgxMyAxMjAuODUzQzMxLjcwNTcgMTIwLjc0NiAzMS42MjA2IDEyMC42MTkgMzEuNTYyNiAxMjAuNDc5QzMxLjUwNDUgMTIwLjMzOCAzMS40NzQ2IDEyMC4xODggMzEuNDc0NiAxMjAuMDM2QzMxLjQ3NDYgMTE5LjczIDMxLjU5NjMgMTE5LjQzNiAzMS44MTMgMTE5LjIxOUMzMi4wMjk3IDExOS4wMDMgMzIuMzIzNiAxMTguODgxIDMyLjYzMDEgMTE4Ljg4MUMzMi45MzY1IDExOC44ODEgMzMuMjMwNCAxMTkuMDAzIDMzLjQ0NzEgMTE5LjIxOUMzMy42NjM4IDExOS40MzYgMzMuNzg1NSAxMTkuNzMgMzMuNzg1NSAxMjAuMDM2WiIgZmlsbD0iYmxhY2siLz4KPHBhdGggZD0iTTEyMC43MjEgMTE4LjI3NkMxMjAuNzIxIDExOC42MjMgMTIwLjU2NSAxMTguOTU2IDEyMC4yODcgMTE5LjIwMUMxMjAuMDA5IDExOS40NDYgMTE5LjYzMiAxMTkuNTg0IDExOS4yMzkgMTE5LjU4NEMxMTguODQ2IDExOS41ODQgMTE4LjQ2OSAxMTkuNDQ2IDExOC4xOTEgMTE5LjIwMUMxMTcuOTEzIDExOC45NTYgMTE3Ljc1NyAxMTguNjIzIDExNy43NTcgMTE4LjI3NkMxMTcuNzU3IDExOC4xMDUgMTE3Ljc5NSAxMTcuOTM1IDExNy44NyAxMTcuNzc2QzExNy45NDQgMTE3LjYxNyAxMTguMDUzIDExNy40NzMgMTE4LjE5MSAxMTcuMzUyQzExOC4zMjkgMTE3LjIzIDExOC40OTIgMTE3LjEzNCAxMTguNjcyIDExNy4wNjhDMTE4Ljg1MSAxMTcuMDAzIDExOS4wNDQgMTE2Ljk2OSAxMTkuMjM5IDExNi45NjlDMTE5LjQzMyAxMTYuOTY5IDExOS42MjYgMTE3LjAwMyAxMTkuODA2IDExNy4wNjhDMTE5Ljk4NiAxMTcuMTM0IDEyMC4xNDkgMTE3LjIzIDEyMC4yODcgMTE3LjM1MkMxMjAuNDI0IDExNy40NzMgMTIwLjUzMyAxMTcuNjE3IDEyMC42MDggMTE3Ljc3NkMxMjAuNjgyIDExNy45MzUgMTIwLjcyMSAxMTguMTA1IDEyMC43MjEgMTE4LjI3NloiIGZpbGw9IiM1OTY4NkYiLz4KPGRlZnM+CjxsaW5lYXJHcmFkaWVudCBpZD0icGFpbnQwX2xpbmVhcl82NzlfMTkxMCIgeDE9IjExIiB5MT0iLTQuNzc3NjhlLTA3IiB4Mj0iMTY0LjUiIHkyPSIxNTAuNSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiPgo8c3RvcCBzdG9wLWNvbG9yPSIjNjRCN0ZGIi8+CjxzdG9wIG9mZnNldD0iMSIgc3RvcC1jb2xvcj0iIzAwNURBRCIvPgo8L2xpbmVhckdyYWRpZW50Pgo8cmFkaWFsR3JhZGllbnQgaWQ9InBhaW50MV9yYWRpYWxfNjc5XzE5MTAiIGN4PSIwIiBjeT0iMCIgcj0iMSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiIGdyYWRpZW50VHJhbnNmb3JtPSJ0cmFuc2xhdGUoMTI0LjkwNCAxMjguNjM2KSByb3RhdGUoLTAuMDQ4NTUyKSBzY2FsZSgzNi4yODAyIDMxLjQwMzIpIj4KPHN0b3Agc3RvcC1jb2xvcj0iIzExNzdDRSIvPgo8c3RvcCBvZmZzZXQ9IjEiIHN0b3AtY29sb3I9IiM3RUM3RkYiIHN0b3Atb3BhY2l0eT0iMCIvPgo8L3JhZGlhbEdyYWRpZW50Pgo8cmFkaWFsR3JhZGllbnQgaWQ9InBhaW50Ml9yYWRpYWxfNjc5XzE5MTAiIGN4PSIwIiBjeT0iMCIgcj0iMSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiIGdyYWRpZW50VHJhbnNmb3JtPSJ0cmFuc2xhdGUoMzkuMzUxMiAxMjEuNDYpIHJvdGF0ZSgtMC4wNTczKSBzY2FsZSg1MC4yMjc2IDUxLjMwOTEpIj4KPHN0b3Agc3RvcC1jb2xvcj0iIzExNzdDRSIvPgo8c3RvcCBvZmZzZXQ9IjEiIHN0b3AtY29sb3I9IiM3RUM3RkYiIHN0b3Atb3BhY2l0eT0iMCIvPgo8L3JhZGlhbEdyYWRpZW50Pgo8cmFkaWFsR3JhZGllbnQgaWQ9InBhaW50M19yYWRpYWxfNjc5XzE5MTAiIGN4PSIwIiBjeT0iMCIgcj0iMSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiIGdyYWRpZW50VHJhbnNmb3JtPSJ0cmFuc2xhdGUoMjAuOTU0OCAxMTguMjYpIHNjYWxlKDEuMzA5MjkgMS41MDEyNykiPgo8c3RvcCBzdG9wLWNvbG9yPSIjODhBM0QwIi8+CjxzdG9wIG9mZnNldD0iMSIgc3RvcC1jb2xvcj0iIzVEODNCRiIvPgo8L3JhZGlhbEdyYWRpZW50Pgo8cmFkaWFsR3JhZGllbnQgaWQ9InBhaW50NF9yYWRpYWxfNjc5XzE5MTAiIGN4PSIwIiBjeT0iMCIgcj0iMSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiIGdyYWRpZW50VHJhbnNmb3JtPSJ0cmFuc2xhdGUoMjQuNzg0MyAxMTAuMjIxKSBzY2FsZSgyLjQ3Mjc3IDIuNTY5NTQpIj4KPHN0b3Agc3RvcC1jb2xvcj0iIzg4QTNEMCIvPgo8c3RvcCBvZmZzZXQ9IjEiIHN0b3AtY29sb3I9IiM1RDgzQkYiLz4KPC9yYWRpYWxHcmFkaWVudD4KPC9kZWZzPgo8L3N2Zz4K - keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "host"], ["spec", "topology"], ["spec", "replicationFactor"], ["spec", "db"], ["spec", "db", "replicas"], ["spec", "db", "size"], ["spec", "db", "storageClass"], ["spec", "db", "resources"], ["spec", "db", "resourcesPreset"], ["spec", "master"], ["spec", "master", "replicas"], ["spec", "master", "resources"], ["spec", "master", "resourcesPreset"], ["spec", "master", "volumeSizeLimitMB"], ["spec", "filer"], ["spec", "filer", "replicas"], ["spec", "filer", "resources"], ["spec", "filer", "resourcesPreset"], ["spec", "filer", "grpcHost"], ["spec", "filer", "grpcPort"], ["spec", "filer", "whitelist"], ["spec", "volume"], ["spec", "volume", "replicas"], ["spec", "volume", "size"], ["spec", "volume", "storageClass"], ["spec", "volume", "diskType"], ["spec", "volume", "resources"], ["spec", "volume", "resourcesPreset"], ["spec", "volume", "zones"], ["spec", "volume", "pools"], ["spec", "s3"], ["spec", "s3", "replicas"], ["spec", "s3", "resources"], ["spec", "s3", "resourcesPreset"], ["spec", "lifecycle"], ["spec", "lifecycle", "enabled"]] + keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "host"], ["spec", "topology"], ["spec", "replicationFactor"], ["spec", "db"], ["spec", "db", "replicas"], ["spec", "db", "size"], ["spec", "db", "storageClass"], ["spec", "db", "resources"], ["spec", "db", "resourcesPreset"], ["spec", "master"], ["spec", "master", "replicas"], ["spec", "master", "resources"], ["spec", "master", "resourcesPreset"], ["spec", "master", "volumeSizeLimitMB"], ["spec", "filer"], ["spec", "filer", "replicas"], ["spec", "filer", "resources"], ["spec", "filer", "resourcesPreset"], ["spec", "filer", "grpcHost"], ["spec", "filer", "grpcPort"], ["spec", "filer", "whitelist"], ["spec", "volume"], ["spec", "volume", "replicas"], ["spec", "volume", "size"], ["spec", "volume", "storageClass"], ["spec", "volume", "diskType"], ["spec", "volume", "resources"], ["spec", "volume", "resourcesPreset"], ["spec", "volume", "zones"], ["spec", "volume", "pools"], ["spec", "s3"], ["spec", "s3", "replicas"], ["spec", "s3", "resources"], ["spec", "s3", "resourcesPreset"], ["spec", "lifecycle"], ["spec", "lifecycle", "enabled"], ["spec", "lifecycle", "storageClass"]] secrets: exclude: [] include: [] From dfd7cbed36d87a5ca40639df476beb72431b6951 Mon Sep 17 00:00:00 2001 From: Kirill Ilin Date: Fri, 6 Nov 2026 18:28:45 +0500 Subject: [PATCH 166/518] fix(operator): reject a Git cert secret before installing CRDs and Flux CRDs and Flux are installed before the platform source, so refusing a cert secret on a Git source only when the source is written exits the operator after a partial install. The check now runs right after flag parsing, and the source install reuses it. The guard runs on every start, so a Git source without a cert secret is now covered over https and ssh: a guard that misfired there would crash-loop the operator for every Git platform source. The update test also renames the cert Secret, as a CA rotation does. Assisted-by: LLM Signed-off-by: Kirill Ilin --- cmd/cozystack-operator/main.go | 22 ++++++- cmd/cozystack-operator/main_test.go | 90 ++++++++++++++++++++++++----- 2 files changed, 94 insertions(+), 18 deletions(-) diff --git a/cmd/cozystack-operator/main.go b/cmd/cozystack-operator/main.go index 1942237ac..e11290aed 100644 --- a/cmd/cozystack-operator/main.go +++ b/cmd/cozystack-operator/main.go @@ -191,6 +191,11 @@ func main() { os.Exit(1) } + if err := validatePlatformSourceCertSecret(platformSourceURL, platformSourceCertSecret); err != nil { + setupLog.Error(err, "invalid platform source flags") + os.Exit(1) + } + config := ctrl.GetConfigOrDie() // Create a direct client (without cache) for pre-start operations @@ -439,8 +444,8 @@ func installPlatformSourceResource(ctx context.Context, k8sClient client.Client, return fmt.Errorf("failed to generate OCIRepository: %w", err) } case "git": - if certSecretName != "" { - return fmt.Errorf("--platform-source-cert-secret is only supported for OCI sources; for a Git source put ca.crt into the --platform-source-secret Secret") + if err := validatePlatformSourceCertSecret(sourceURL, certSecretName); err != nil { + return err } obj, err = generateGitRepository(resourceName, repoURL, refMap, secretName) if err != nil { @@ -537,6 +542,19 @@ func parseSystemNamespaceMemory(rawLimit, rawRequest string) (limit, request res return limit, request, nil } +// validatePlatformSourceCertSecret rejects --platform-source-cert-secret for a +// Git source: GitRepository has no certSecretRef, and Flux reads a Git CA from +// the ca.crt key of the --platform-source-secret Secret instead. +func validatePlatformSourceCertSecret(sourceURL, certSecretName string) error { + if certSecretName == "" { + return nil + } + if sourceType, _, err := parsePlatformSourceURL(sourceURL); err == nil && sourceType == "git" { + return fmt.Errorf("--platform-source-cert-secret is only supported for OCI sources; for a Git source put ca.crt into the --platform-source-secret Secret") + } + return nil +} + // parsePlatformSourceURL parses the source URL and returns the source type and repository URL. // Supports formats: // - oci://registry.example.com/repo diff --git a/cmd/cozystack-operator/main_test.go b/cmd/cozystack-operator/main_test.go index 3b38ace1a..136db4603 100644 --- a/cmd/cozystack-operator/main_test.go +++ b/cmd/cozystack-operator/main_test.go @@ -612,26 +612,31 @@ func TestGenerateOCIRepository_WithCertSecret(t *testing.T) { } } -// The operator rewrites the whole platform OCIRepository on every start, so a -// certSecretRef patched onto the live object by hand is lost on the next -// restart and source-controller then fails TLS against a registry behind a -// private CA. The CA has to come from the operator's own arguments and survive -// that update path. -func TestInstallPlatformSourceResource_UpdateKeepsCertSecret(t *testing.T) { +func newSourceTestClient(t *testing.T, objs ...client.Object) client.Client { + t.Helper() scheme := newTestScheme() if err := sourcev1.AddToScheme(scheme); err != nil { t.Fatalf("add source scheme: %v", err) } + return fake.NewClientBuilder().WithScheme(scheme).WithObjects(objs...).Build() +} + +// The operator rewrites the whole platform OCIRepository on every start, so a +// certSecretRef patched onto the live object by hand is lost on the next +// restart and source-controller then fails TLS against a registry behind a +// private CA. The CA has to come from the operator's own arguments and survive +// that update path, including a rename of the Secret when the CA is rotated. +func TestInstallPlatformSourceResource_UpdateKeepsCertSecret(t *testing.T) { existing := &sourcev1.OCIRepository{ ObjectMeta: metav1.ObjectMeta{Name: "cozystack-platform", Namespace: "cozy-system"}, Spec: sourcev1.OCIRepositorySpec{URL: "oci://ghcr.io/cozystack/cozystack/cozystack-packages"}, } - k8sClient := fake.NewClientBuilder().WithScheme(scheme).WithObjects(existing).Build() + k8sClient := newSourceTestClient(t, existing) - for range 2 { + for _, certSecret := range []string{"registry-ca", "registry-ca-next"} { if err := installPlatformSourceResource(context.Background(), k8sClient, "oci://registry.internal:8083/cozystack/cozystack/cozystack-packages", "cozystack-platform", - "digest=sha256:abc123", "registry-creds", "registry-ca"); err != nil { + "digest=sha256:abc123", "registry-creds", certSecret); err != nil { t.Fatalf("installPlatformSourceResource: %v", err) } } @@ -640,8 +645,8 @@ func TestInstallPlatformSourceResource_UpdateKeepsCertSecret(t *testing.T) { if err := k8sClient.Get(context.Background(), client.ObjectKey{Namespace: "cozy-system", Name: "cozystack-platform"}, got); err != nil { t.Fatalf("get OCIRepository: %v", err) } - if got.Spec.CertSecretRef == nil || got.Spec.CertSecretRef.Name != "registry-ca" { - t.Errorf("expected CertSecretRef registry-ca after update, got %+v", got.Spec.CertSecretRef) + if got.Spec.CertSecretRef == nil || got.Spec.CertSecretRef.Name != "registry-ca-next" { + t.Errorf("expected CertSecretRef registry-ca-next after update, got %+v", got.Spec.CertSecretRef) } if got.Spec.SecretRef == nil || got.Spec.SecretRef.Name != "registry-creds" { t.Errorf("expected SecretRef registry-creds after update, got %+v", got.Spec.SecretRef) @@ -655,11 +660,7 @@ func TestInstallPlatformSourceResource_UpdateKeepsCertSecret(t *testing.T) { // of the secretRef Secret. Accepting the flag for a Git source would silently // drop the CA, so it must fail before anything is written. func TestInstallPlatformSourceResource_GitRejectsCertSecret(t *testing.T) { - scheme := newTestScheme() - if err := sourcev1.AddToScheme(scheme); err != nil { - t.Fatalf("add source scheme: %v", err) - } - k8sClient := fake.NewClientBuilder().WithScheme(scheme).Build() + k8sClient := newSourceTestClient(t) err := installPlatformSourceResource(context.Background(), k8sClient, "https://git.internal/cozystack/cozystack", "cozystack-platform", "branch=main", "git-creds", "git-ca") @@ -679,6 +680,63 @@ func TestInstallPlatformSourceResource_GitRejectsCertSecret(t *testing.T) { } } +// The Git guard runs on every start; one that misfires without a cert secret +// would crash-loop the operator for every Git platform source. +func TestInstallPlatformSourceResource_GitWithoutCertSecret(t *testing.T) { + for _, sourceURL := range []string{ + "https://git.internal/cozystack/cozystack", + "ssh://git@git.internal/cozystack/cozystack", + } { + t.Run(sourceURL, func(t *testing.T) { + k8sClient := newSourceTestClient(t) + + if err := installPlatformSourceResource(context.Background(), k8sClient, + sourceURL, "cozystack-platform", "branch=main", "git-creds", ""); err != nil { + t.Fatalf("installPlatformSourceResource: %v", err) + } + + got := &sourcev1.GitRepository{} + if err := k8sClient.Get(context.Background(), client.ObjectKey{Namespace: "cozy-system", Name: "cozystack-platform"}, got); err != nil { + t.Fatalf("get GitRepository: %v", err) + } + if got.Spec.URL != sourceURL { + t.Errorf("expected URL %q, got %q", sourceURL, got.Spec.URL) + } + if got.Spec.SecretRef == nil || got.Spec.SecretRef.Name != "git-creds" { + t.Errorf("expected SecretRef git-creds, got %+v", got.Spec.SecretRef) + } + }) + } +} + +// Checked right after flag parsing: CRDs and Flux are installed before the +// platform source, so a combination rejected only there would exit after a +// partial install. +func TestValidatePlatformSourceCertSecret(t *testing.T) { + tests := []struct { + name string + sourceURL string + certSecret string + wantErr bool + }{ + {name: "oci with cert secret", sourceURL: "oci://registry.internal:8083/cozystack-packages", certSecret: "registry-ca"}, + {name: "oci without cert secret", sourceURL: "oci://registry.internal:8083/cozystack-packages"}, + {name: "https git without cert secret", sourceURL: "https://git.internal/cozystack/cozystack"}, + {name: "ssh git without cert secret", sourceURL: "ssh://git@git.internal/cozystack/cozystack"}, + {name: "no source url", certSecret: "registry-ca"}, + {name: "https git with cert secret", sourceURL: "https://git.internal/cozystack/cozystack", certSecret: "git-ca", wantErr: true}, + {name: "ssh git with cert secret", sourceURL: "ssh://git@git.internal/cozystack/cozystack", certSecret: "git-ca", wantErr: true}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + err := validatePlatformSourceCertSecret(tt.sourceURL, tt.certSecret) + if (err != nil) != tt.wantErr { + t.Errorf("validatePlatformSourceCertSecret(%q, %q) error = %v, wantErr %v", tt.sourceURL, tt.certSecret, err, tt.wantErr) + } + }) + } +} + func TestGenerateGitRepository(t *testing.T) { refMap := map[string]string{"branch": "main", "commit": "abc1234def5678"} obj, err := generateGitRepository("my-repo", "https://github.com/user/repo", refMap, "") From fbfc270c712c8eb2dac240c0e6643a2b88bcdbda Mon Sep 17 00:00:00 2001 From: Kirill Ilin Date: Fri, 6 Nov 2026 18:29:18 +0500 Subject: [PATCH 167/518] test(installer): tie each platform source flag to its own value Both secrets were only ever set together, so a cert flag keyed on platformSourceSecret stayed green. A pull-through cache behind a private CA usually needs the CA without credentials, which is exactly the case such a slip breaks. The absence checks match the flag by prefix, since a whole-element comparison misses the flag rendered with any value. Assisted-by: LLM Signed-off-by: Kirill Ilin --- .../installer/tests/platform_source_test.yaml | 44 ++++++++++++++++--- 1 file changed, 39 insertions(+), 5 deletions(-) diff --git a/packages/core/installer/tests/platform_source_test.yaml b/packages/core/installer/tests/platform_source_test.yaml index 5d2db4033..a925754c1 100644 --- a/packages/core/installer/tests/platform_source_test.yaml +++ b/packages/core/installer/tests/platform_source_test.yaml @@ -1,12 +1,47 @@ suite: cozystack-operator platform source arguments +# Each flag must follow its own value: a pull-through cache behind a private CA +# often needs the CA without credentials, so a cert flag keyed on the wrong +# value would bring back the TLS failure while every other case stays green. + templates: - templates/cozystack-operator.yaml tests: - - it: passes platformSourceCertSecret to the operator + - it: passes platformSourceCertSecret alone, without credentials + set: + cozystackOperator.platformSourceCertSecret: registry-ca + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --platform-source-cert-secret=registry-ca + documentSelector: + path: kind + value: Deployment + - notExists: + path: spec.template.spec.containers[0].args[?(@ =~ /^--platform-source-secret=/)] + documentSelector: + path: kind + value: Deployment + + - it: passes platformSourceSecret alone, without the cert flag + set: + cozystackOperator.platformSourceSecret: registry-creds + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --platform-source-secret=registry-creds + documentSelector: + path: kind + value: Deployment + - notExists: + path: spec.template.spec.containers[0].args[?(@ =~ /^--platform-source-cert-secret=/)] + documentSelector: + path: kind + value: Deployment + + - it: passes both secrets together set: - cozystackOperator.platformSourceUrl: oci://registry.internal:8083/cozystack/cozystack/cozystack-packages cozystackOperator.platformSourceSecret: registry-creds cozystackOperator.platformSourceCertSecret: registry-ca asserts: @@ -25,9 +60,8 @@ tests: - it: omits the cert secret flag by default asserts: - - notContains: - path: spec.template.spec.containers[0].args - content: --platform-source-cert-secret= + - notExists: + path: spec.template.spec.containers[0].args[?(@ =~ /^--platform-source-cert-secret=/)] documentSelector: path: kind value: Deployment From 0a448963e3be7055d45bf0aec9e604a6bb384085 Mon Sep 17 00:00:00 2001 From: Kirill Ilin Date: Fri, 6 Nov 2026 18:29:18 +0500 Subject: [PATCH 168/518] docs(installer): say a CA-only cert secret must be Opaque kubernetes.io/tls requires both tls.crt and tls.key, so a Secret that carries only ca.crt is rejected under that type. Assisted-by: LLM Signed-off-by: Kirill Ilin --- packages/core/installer/values.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/packages/core/installer/values.yaml b/packages/core/installer/values.yaml index 6224b5955..55f19c0d9 100644 --- a/packages/core/installer/values.yaml +++ b/packages/core/installer/values.yaml @@ -65,8 +65,9 @@ cozystackOperator: platformSourceSecret: '' # Name of the Secret in cozy-system namespace with TLS material for an OCI # platform source: `ca.crt` for a registry whose certificate is issued by a - # private CA, and/or `tls.crt`+`tls.key` for client certificate auth (type - # Opaque or kubernetes.io/tls). When non-empty, sets spec.certSecretRef on the + # private CA, and/or `tls.crt`+`tls.key` for client certificate auth. A + # CA-only Secret must be type Opaque: kubernetes.io/tls requires `tls.crt` + # and `tls.key`. When non-empty, sets spec.certSecretRef on the # generated OCIRepository; the operator rewrites that object on every start, # so a certSecretRef patched onto it by hand does not survive a restart. # The cozystack-packages source copies the spec, so it inherits the CA too. From 41507e36aa2d20b8d5a55cab6aeddbb030329842 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 6 Nov 2026 18:56:09 +0500 Subject: [PATCH 169/518] docs(kubernetes): say where worker registry mirrors are configured The air-gapped note said in-guest image pulls honour no per-tenant mirror until a Phase 2 that has landed, so an operator setting up an air-gapped cluster would read that mirroring is unavailable and file an issue. It now says the Secret and the platform registries value are removed, that a Package still setting the value keeps installing, and that worker mirrors are set per pool through talos.registryMirrors of the KubernetesNodes resource. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- packages/apps/kubernetes/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/packages/apps/kubernetes/README.md b/packages/apps/kubernetes/README.md index 17e2a2822..2ac5cfc45 100644 --- a/packages/apps/kubernetes/README.md +++ b/packages/apps/kubernetes/README.md @@ -87,7 +87,7 @@ See the reference for components utilized in this service: - **`ephemeralStorage` renamed to `diskSize`** (v1.4): The `nodeGroups[name].ephemeralStorage` field has been renamed to `nodeGroups[name].diskSize`. Existing clusters are migrated transparently by platform migration 41 during the pre-upgrade hook — no manual action is required. Newly written values should use `diskSize`. Existing VMs are rolling-updated via CAPI when the new values are applied. With the Talos worker rollover in this release the field now sizes the **single system disk** (Talos OS image streamed from `factory.talos.dev`, kubelet state, containerd image cache, local-path PVCs) — the pre-Talos `disk-kubelet` PVC layout was removed. State on that single disk persists across same-VM reboots (virt-launcher restart, guest reboot, node failure); VM replacement by CAPI (e.g. nodeGroup field change, MachineHealthCheck remediation) provisions a fresh disk. -- **Air-gapped tenant workers (Phase 1 Talos rollover)**: tenant worker bootstrap moves from Ubuntu containerDisk + kubeadm to a Talos OS disk image fetched over HTTP by CDI and a `.../installer/...` installer reference. Both sources are now overridable — set `talos.imageFactoryURL` (default `https://factory.talos.dev`) to a self-hosted Image Factory, a caching mirror, or an internal HTTP file server, and `talos.installerRepository` (default `factory.talos.dev/installer`) to a mirrored registry. The Helm-rendered `*-patch-containerd` Secret that previously plumbed the platform-wide `registries.mirrors` config to tenant workers (via the kubeadm template's containerd certs.d mount) has no consumer in the Talos machineconfig and was removed in this release; **in-guest container-image pulls** (pods pulling from registries inside the tenant cluster) still honour no per-tenant `registries.mirrors` override until Phase 2 restores it via `machine.registries.mirrors` knobs. So a strict-egress tenant can now mirror the Talos OS image and installer, but rate-limited in-guest registry pulls remain a Phase 2 follow-up — file an issue if you depend on this and it is not yet landed. +- **Air-gapped tenant workers (Phase 1 Talos rollover)**: tenant worker bootstrap moves from Ubuntu containerDisk + kubeadm to a Talos OS disk image fetched over HTTP by CDI and a `.../installer/...` installer reference. Both sources are now overridable — set `talos.imageFactoryURL` (default `https://factory.talos.dev`) to a self-hosted Image Factory, a caching mirror, or an internal HTTP file server, and `talos.installerRepository` (default `factory.talos.dev/installer`) to a mirrored registry. The Helm-rendered `*-patch-containerd` Secret that previously plumbed the platform-wide `registries.mirrors` config to tenant workers (via the kubeadm template's containerd certs.d mount) has no consumer in the Talos machineconfig, so the Secret is removed together with the platform `registries` value that fed it. A Package that still sets `registries` keeps installing, the key is ignored. Registry mirrors for tenant workers, **in-guest container-image pulls** (pods pulling from registries inside the tenant cluster) included, are configured per worker pool through `talos.registryMirrors` of the `KubernetesNodes` resource. - **Worker MachineHealthCheck remediation is now enabled by default** (Phase 1 Talos rollover): the worker MHC `maxUnhealthy` moved from a hard-coded `0` (remediation effectively **off**) to a configurable `nodeHealthCheck.maxUnhealthy` defaulting to `"50%"`. CAPI now auto-remediates (deletes and replaces) unhealthy worker Machines while at least 50% of a group stays healthy. `"50%"` deliberately leaves headroom for transient unhealthy nodes during the kubeadm→Talos rollover and slow first boots from `factory.talos.dev`; set `nodeHealthCheck.maxUnhealthy: "0%"` to keep the previous remediation-off behaviour until your fleet is stable on Talos workers. From f147b22d86343eb25aa8ff88651efc5ce7993a95 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 16:04:11 +0300 Subject: [PATCH 170/518] fix(backups): delete a MariaDB dump when its Backup is deleted Deleting a MariaDB Backup left its dump in the bucket for good: mariadb-operator's maxRetention prunes only from inside the next run of the same prefix, and a Plan never deletes a Backup, so on the platform bucket nothing else ever removes it. Every run of an application writes under one prefix, so the key deleted has to be proven to be this Backup's. It is the dump the Backup records, or, for a Backup taken before dumps were recorded, the one the existing ownership match attributes to its operator Backup. That match is recorded on the Backup before the delete, so a retry deletes the same key by name instead of matching again against a bucket that no longer holds it. Without proof the Backup is released with a Warning and nothing is deleted, the operator Backup included, since any later attribution starts from it. The operator Backup is deleted only after its dump, and the delete waits while a RestoreJob still restores from the Backup. The delete runs in the controller through the same minio client the size probe uses, so it verifies TLS the way mariadb-operator did when it wrote the dump. A dump on the platform bucket, which the operator Backup names by its bucket and by the projected credentials Secret, is reached through the platform's current endpoint and keys, still trusting a CA the strategy pinned, so a Backup taken before the endpoint moved is still deleted. Platform credentials that cannot be read fall back to the ones the operator Backup names. A failed delete holds the Backup and retries with a Warning naming the skip-artifact-cleanup annotation, except in a terminating namespace, where the Backup is released with a Warning so it does not hold the namespace. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/artifact_size.go | 38 +- .../backupcontroller/backup_controller.go | 33 +- .../backup_controller_test.go | 7 +- .../backupcontroller/mariadb_artifact_size.go | 15 +- .../mariadbstrategy_cleanup.go | 268 ++++++ .../mariadbstrategy_cleanup_test.go | 886 ++++++++++++++++++ 6 files changed, 1223 insertions(+), 24 deletions(-) create mode 100644 internal/backupcontroller/mariadbstrategy_cleanup.go create mode 100644 internal/backupcontroller/mariadbstrategy_cleanup_test.go diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 9197f07be..40d827c62 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -38,6 +38,11 @@ type s3ObjectLister interface { ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) } +type s3ObjectRemover interface { + // RemoveObject deletes key; a key that does not exist is not an error. + RemoveObject(ctx context.Context, bucket, key string) error +} + // s3Connection is what a driver resolves from its rendered strategy to // read the bucket with the same identity the backup was written with. type s3Connection struct { @@ -63,8 +68,10 @@ type objectStore struct { // reader reads Secrets, which the manager's client fetches uncached, // and lists operator Backups, which it serves from the cache. reader client.Reader - // newLister overrides the S3 client; nil means a real minio client. - newLister func(conn s3Connection) (s3ObjectLister, error) + // newLister and newRemover override the S3 client; nil means a real + // minio client. + newLister func(conn s3Connection) (s3ObjectLister, error) + newRemover func(conn s3Connection) (s3ObjectRemover, error) } func (s objectStore) lister(conn s3Connection) (s3ObjectLister, error) { @@ -74,6 +81,17 @@ func (s objectStore) lister(conn s3Connection) (s3ObjectLister, error) { return newMinioObjectLister(conn) } +func (s objectStore) remover(conn s3Connection) (s3ObjectRemover, error) { + if s.newRemover != nil { + return s.newRemover(conn) + } + c, err := newMinioObjectClient(conn) + if err != nil { + return nil, err + } + return c, nil +} + func (s objectStore) secretValue(ctx context.Context, namespace, name, key string) (string, error) { secret := &corev1.Secret{} if err := s.reader.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, secret); err != nil { @@ -94,6 +112,10 @@ func (r *RestoreJobReconciler) objectStore() objectStore { return objectStore{reader: r.Client, newLister: r.newS3ObjectLister} } +func (r *BackupReconciler) objectStore() objectStore { + return objectStore{reader: r.Client, newLister: r.newS3ObjectLister, newRemover: r.newS3ObjectRemover} +} + func (r *BackupJobReconciler) s3Lister(conn s3Connection) (s3ObjectLister, error) { return r.objectStore().lister(conn) } @@ -103,6 +125,14 @@ type minioObjectLister struct { } func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { + c, err := newMinioObjectClient(conn) + if err != nil { + return nil, err + } + return c, nil +} + +func newMinioObjectClient(conn s3Connection) (*minioObjectLister, error) { opts := &minio.Options{ Creds: credentials.NewStaticV4(conn.AccessKey, conn.SecretKey, conn.SessionToken), Region: conn.Region, @@ -152,6 +182,10 @@ func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix stri return collectObjects(ctx, l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: recursive})) } +func (l *minioObjectLister) RemoveObject(ctx context.Context, bucket, key string) error { + return l.client.RemoveObject(ctx, bucket, key, minio.RemoveObjectOptions{}) +} + // collectObjects drains a minio listing. minio-go ends a listing whose // context is cancelled between pages by closing the channel without an // error item, so a cancelled context is checked once the channel closes: diff --git a/internal/backupcontroller/backup_controller.go b/internal/backupcontroller/backup_controller.go index 1a8ec0b40..f817e696b 100644 --- a/internal/backupcontroller/backup_controller.go +++ b/internal/backupcontroller/backup_controller.go @@ -39,9 +39,16 @@ type BackupReconciler struct { client.Client Scheme *runtime.Scheme Recorder record.EventRecorder - // CredentialsConfig lets the Rabbitmq cleanup path project cozy-backups-creds - // into the Backup's namespace before spawning its best-effort delete Job. + // CredentialsConfig locates the platform backup credentials: the Job-based + // cleanups project them into the Backup's namespace, and the MariaDB + // cleanup reads them to reach the platform bucket. CredentialsConfig BackupCredentialsConfig + + // newS3ObjectLister and newS3ObjectRemover override the S3 client the + // MariaDB cleanup deletes a dump with; nil means a real minio client. + // Set by unit tests. + newS3ObjectLister func(conn s3Connection) (s3ObjectLister, error) + newS3ObjectRemover func(conn s3Connection) (s3ObjectRemover, error) } func (r *BackupReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { @@ -126,22 +133,18 @@ func (r *BackupReconciler) cleanupOnDelete(ctx context.Context, backup *backupsv // orchestration" for tenant-facing guidance. return ctrl.Result{}, nil case strategyv1alpha1.MariaDBStrategyKind: - // Cozystack Backup deletion does NOT delete the operator-side - // k8s.mariadb.com/Backup CR or the backing S3/PVC archive. - // Lifecycle of that CR is owned by mariadb-operator's - // MaxRetention setting and by tenants / explicit teardown flows; - // the controller's RBAC has no delete verb on - // k8s.mariadb.com/backups for that reason. Surfacing this as an - // explicit branch (rather than falling through to the Velero - // default) keeps the contract honest: the MariaDB driver does - // not own the archive, so it does not delete it on CR removal. - // Mirrors the Altinity branch's "we do not own S3" stance. - return ctrl.Result{}, nil + // mariadb-operator's maxRetention prunes only from inside the next + // backup run of the same prefix and a Plan never deletes a Backup, so + // a dump on the platform bucket is deleted here or never. A dump in + // a tenant's own bucket is deleted too: the Backup proves it owns + // it, and the tenant deleting the Backup is asking for exactly that + // (see cleanupMariaDBBackup). + return r.cleanupMariaDBBackup(ctx, backup) case strategyv1alpha1.MongoDBStrategyKind: // Split by flow (see cleanupMongoDBBackup). A legacy backup writes to the // tenant's own bucket and its lifecycle stays the tenant's / psmdb // operator's — that branch is a no-op, the same "we do not own the - // archive" contract as Altinity / MariaDB / FDB. A useSystemBucket backup + // archive" contract as Altinity / FDB. A useSystemBucket backup // lands in the shared cozy-backups bucket with no psmdb task retention to // prune it, so the driver OWNS it: this deletes the operator // PerconaServerMongoDBBackup CR and waits for its delete-backup finalizer @@ -154,7 +157,7 @@ func (r *BackupReconciler) cleanupOnDelete(ctx context.Context, backup *backupsv // is governed by the next BackupJob's stopOtherFoundationDBBackups // call, by manual operator action, or by // examples/backups/foundationdb/cleanup.sh. Same "we do not own - // the archive" contract as Altinity / MariaDB — the explicit + // the archive" contract as Altinity — the explicit // branch guards the seam against a future driver refactor that // might incidentally stamp velero.io/backup-name onto FDB // driverMetadata via a shared helper. diff --git a/internal/backupcontroller/backup_controller_test.go b/internal/backupcontroller/backup_controller_test.go index 579cb5fe6..d77f5d208 100644 --- a/internal/backupcontroller/backup_controller_test.go +++ b/internal/backupcontroller/backup_controller_test.go @@ -176,11 +176,8 @@ func TestBackupCleanup_Altinity_DoesNotFallThroughToVelero(t *testing.T) { // TestBackupCleanup_MariaDB_DoesNotFallThroughToVelero locks in the same // invariant as the Altinity test above for the MariaDB strategy. The -// dispatcher must explicitly route MariaDB-strategy Backups to a no-op -// cleanup rather than fall through to the Velero default - the MariaDB -// driver does not own the operator-side k8s.mariadb.com/Backup CR or the -// S3/PVC archive behind it (rbac.yaml grants no delete verb on -// k8s.mariadb.com/backups for that reason). The test seeds the Velero +// dispatcher must route MariaDB-strategy Backups to their own cleanup +// rather than fall through to the Velero default. The test seeds the Velero // metadata keys plus a matching velero.io/Backup; with the explicit branch // no DeleteBackupRequest is synthesised, and the Velero Backup survives. // Without the branch the default falls through to cleanupVeleroBackup, diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index a61ec4a43..f9b9973ad 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -60,11 +60,22 @@ func (r *BackupJobReconciler) attributedMariaDBDump(ctx context.Context, mdbBack // from can have been edited since, and pointing at another bucket now says // nothing about where this dump went. func findMariaDBDump(ctx context.Context, store objectStore, mdbBackup *mariadbtypes.Backup, completedAt time.Time) (mariadbDump, error) { - lister, s3, err := mariadbStorage(ctx, store, mdbBackup) + lister, _, err := mariadbStorage(ctx, store, mdbBackup) if err != nil { return mariadbDump{}, err } - others, err := finishedMariaDBRuns(ctx, store.reader, mdbBackup) + return findMariaDBDumpWith(ctx, lister, store.reader, mdbBackup, completedAt) +} + +// findMariaDBDumpWith is findMariaDBDump reading the bucket through lister, +// for a caller that reaches the bucket by another endpoint than the one +// mdbBackup names. +func findMariaDBDumpWith(ctx context.Context, lister s3ObjectLister, reader client.Reader, mdbBackup *mariadbtypes.Backup, completedAt time.Time) (mariadbDump, error) { + s3 := mdbBackup.Spec.Storage.S3 + if s3 == nil { + return mariadbDump{}, errors.New("the operator Backup does not store to S3") + } + others, err := finishedMariaDBRuns(ctx, reader, mdbBackup) if err != nil { return mariadbDump{}, fmt.Errorf("list operator Backups: %w", err) } diff --git a/internal/backupcontroller/mariadbstrategy_cleanup.go b/internal/backupcontroller/mariadbstrategy_cleanup.go new file mode 100644 index 000000000..483ded502 --- /dev/null +++ b/internal/backupcontroller/mariadbstrategy_cleanup.go @@ -0,0 +1,268 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "errors" + "fmt" + "strings" + "time" + + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + apimeta "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" + "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" +) + +const ( + mariadbSkipArtifactCleanupAnnotation = "backups.cozystack.io/skip-artifact-cleanup" + + // Retry cadence while a dump cannot be deleted or a restore from the + // Backup is still running. + mariadbCleanupRetryInterval = 30 * time.Second +) + +// mariadbDumpCleanupTimeout bounds the object-store round trips of one +// cleanup attempt. The Backup controller has a single worker, so an +// unreachable endpoint stalls every other Backup for this long. A variable +// so tests can shorten it. +var mariadbDumpCleanupTimeout = 10 * time.Second + +// cleanupMariaDBBackup deletes the dump a MariaDB Backup was taken as, then +// the operator Backup that wrote it. Every run of an application writes +// under one prefix, so the only key deleted is one proven to be this +// Backup's: the dump it records, or, for a Backup taken before dumps were +// recorded, the one findMariaDBDump attributes to its operator Backup. When +// no dump is proven the Backup is released with a Warning and nothing is +// deleted, the operator Backup included, since it is what any later +// attribution would start from. A failed delete holds the Backup and +// retries, as the Redis and Rabbitmq cleanups do. +func (r *BackupReconciler) cleanupMariaDBBackup(ctx context.Context, backup *backupsv1alpha1.Backup) (ctrl.Result, error) { + logger := getLogger(ctx) + if backup.Annotations[mariadbSkipArtifactCleanupAnnotation] == "true" { + logger.Debug("skipping MariaDB artifact cleanup per annotation", "backup", backup.Name, "annotation", mariadbSkipArtifactCleanupAnnotation) + return ctrl.Result{}, nil + } + + opName := backup.Spec.DriverMetadata[mariadbBackupNameKey] + if opName == "" { + return r.releaseMariaDBCleanup(ctx, backup, "the dump", "the Backup records no k8s.mariadb.com/Backup"), nil + } + + // The operator Restore reads the dump through the operator Backup, so + // neither may go while a restore from this Backup runs. + restore, err := r.activeRestoreOf(ctx, backup) + if err != nil { + return ctrl.Result{}, err + } + if restore != "" { + logger.Debug("holding MariaDB Backup cleanup while a restore from it runs", "backup", backup.Name, "restorejob", restore) + if r.Recorder != nil { + r.Recorder.Eventf(backup, corev1.EventTypeNormal, "ArtifactCleanupWaiting", + "RestoreJob %s is restoring from this Backup; its dump is deleted once the restore finishes. Annotate the Backup %s=true to release it now and leave the dump", + restore, mariadbSkipArtifactCleanupAnnotation) + } + return ctrl.Result{RequeueAfter: mariadbCleanupRetryInterval}, nil + } + + op := &mariadbtypes.Backup{} + if err := r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: opName}, op); err != nil { + if apierrors.IsNotFound(err) || apimeta.IsNoMatchError(err) { + return r.releaseMariaDBCleanup(ctx, backup, "the dump", + fmt.Sprintf("k8s.mariadb.com/Backup %s is gone, so the storage and credentials its dump was written with cannot be read", opName)), nil + } + return ctrl.Result{}, err + } + + s3 := op.Spec.Storage.S3 + if s3 == nil { + return r.releaseMariaDBCleanup(ctx, backup, fmt.Sprintf("the dump on the storage volume of k8s.mariadb.com/Backup %s", opName), + "PVC and Volume storage is not reachable from the controller"), nil + } + prefix := mariadbOperatorS3Prefix(s3.Prefix) + location := fmt.Sprintf("s3://%s/%s", s3.Bucket, prefix) + + // A failed Job is Complete=True too, with reason JobFailed, and wrote no + // dump: whatever its window holds is another run's. + if cond := apimeta.FindStatusCondition(op.Status.Conditions, mariadbtypes.ConditionTypeComplete); cond == nil || + cond.Status != metav1.ConditionTrue || cond.Reason != mariadbtypes.ConditionReasonJobComplete { + return r.releaseMariaDBCleanup(ctx, backup, location, + fmt.Sprintf("k8s.mariadb.com/Backup %s did not complete successfully, so no dump is its own", opName)), nil + } + + name := backup.Spec.DriverMetadata[mariadbDumpFileKey] + if name != "" { + if _, ok := parseMariaDBLogicalBackupFile(name); !ok { + return r.releaseMariaDBCleanup(ctx, backup, location, + fmt.Sprintf("driverMetadata %s=%q is not the name of a mariadb-operator dump", mariadbDumpFileKey, name)), nil + } + } + + conn, err := r.mariadbCleanupConnection(ctx, op) + if err != nil { + return r.retryMariaDBCleanup(ctx, backup, location, fmt.Errorf("resolve S3 credentials: %w", err)) + } + + s3ctx, cancel := context.WithTimeout(ctx, mariadbDumpCleanupTimeout) + defer cancel() + store := r.objectStore() + if name == "" { + lister, err := store.lister(conn) + if err != nil { + return r.retryMariaDBCleanup(ctx, backup, location, fmt.Errorf("build S3 client: %w", err)) + } + dump, err := findMariaDBDumpWith(s3ctx, lister, r.Client, op, backup.Spec.TakenAt.Time) + if errors.Is(err, errMariaDBDumpNotFound) || errors.Is(err, errMariaDBDumpAmbiguous) { + return r.releaseMariaDBCleanup(ctx, backup, location, + fmt.Sprintf("the Backup records no dump and none is proven to be the one k8s.mariadb.com/Backup %s wrote (%v); the operator Backup is kept", opName, err)), nil + } + if err != nil { + return r.retryMariaDBCleanup(ctx, backup, location, err) + } + name = dump.Name + // Recorded first: once the dump is gone a retry could not match it + // again, and could match a neighbour's instead. + base := backup.DeepCopy() + backup.Spec.DriverMetadata[mariadbDumpFileKey] = name + backup.Spec.DriverMetadata[mariadbDumpTimeKey] = dump.Timestamp.UTC().Format(time.RFC3339) + if err := r.Patch(ctx, backup, client.MergeFrom(base)); err != nil { + return ctrl.Result{}, fmt.Errorf("record dump %s on Backup %s: %w", name, backup.Name, err) + } + } + + remover, err := store.remover(conn) + if err != nil { + return r.retryMariaDBCleanup(ctx, backup, location, fmt.Errorf("build S3 client: %w", err)) + } + if err := remover.RemoveObject(s3ctx, s3.Bucket, prefix+name); err != nil { + return r.retryMariaDBCleanup(ctx, backup, location+name, err) + } + policy := metav1.DeletePropagationBackground + if err := r.Delete(ctx, op, &client.DeleteOptions{PropagationPolicy: &policy}); err != nil && !apierrors.IsNotFound(err) { + return ctrl.Result{}, fmt.Errorf("delete k8s.mariadb.com/Backup %s/%s: %w", op.Namespace, op.Name, err) + } + logger.Info("deleted MariaDB dump and its operator Backup", "backup", backup.Name, "dump", location+name, "operatorBackup", opName) + return ctrl.Result{}, nil +} + +// activeRestoreOf returns the name of a RestoreJob that restores from +// backup and has not finished, or "". +func (r *BackupReconciler) activeRestoreOf(ctx context.Context, backup *backupsv1alpha1.Backup) (string, error) { + list := &backupsv1alpha1.RestoreJobList{} + if err := r.List(ctx, list, client.InNamespace(backup.Namespace)); err != nil { + return "", fmt.Errorf("list RestoreJobs: %w", err) + } + for i := range list.Items { + rj := &list.Items[i] + if rj.Spec.BackupRef.Name != backup.Name { + continue + } + if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded && rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + return rj.Name, nil + } + } + return "", nil +} + +// mariadbCleanupConnection returns how to reach the bucket op wrote to. A +// dump on the platform bucket, which op names by its bucket and by the +// Secret the platform credentials are projected to, is reached as the +// platform reaches it now, so a Backup taken before the endpoint moved is +// still deleted. Any other bucket, or the platform bucket while the +// platform credentials cannot be read, is reached as op recorded it; a +// delete that fails there is handled by retryMariaDBCleanup like any other. +func (r *BackupReconciler) mariadbCleanupConnection(ctx context.Context, op *mariadbtypes.Backup) (s3Connection, error) { + if r.CredentialsConfig.IsEnabled() { + conn, bucket, err := r.platformS3Connection(ctx) + switch { + case err != nil: + getLogger(ctx).Info("cannot read the platform backup credentials; using the operator Backup's own", "operatorBackup", op.Name, "error", err) + case bucket == op.Spec.Storage.S3.Bucket && op.Spec.Storage.S3.AccessKeyIdSecretKeyRef.Name == r.CredentialsConfig.TargetSecretName: + // A strategy can pin a private CA for the platform bucket; the + // platform credentials carry none. + if tls := op.Spec.Storage.S3.TLS; conn.Secure && tls != nil && tls.CASecretKeyRef != nil { + ca, err := r.objectStore().secretValue(ctx, op.Namespace, tls.CASecretKeyRef.Name, tls.CASecretKeyRef.Key) + if err != nil { + return s3Connection{}, err + } + conn.CAPEM = []byte(ca) + } + return conn, nil + } + } + return mariadbS3Connection(ctx, r.objectStore(), op.Namespace, op.Spec.Storage.S3) +} + +// platformS3Connection reads the platform's backup bucket from the Secret +// cozy-backups-creds is projected from, and its TLS from the scheme of +// BACKUP_STORAGE_ENDPOINT, which is how the default strategy sets +// tls.enabled. It reads the source rather than projecting it: projecting +// writes a Secret, which a terminating namespace refuses. +func (r *BackupReconciler) platformS3Connection(ctx context.Context) (s3Connection, string, error) { + cfg := r.CredentialsConfig + src := &corev1.Secret{} + if err := r.Get(ctx, types.NamespacedName{Namespace: cfg.SourceNamespace, Name: cfg.SourceSecretName}, src); err != nil { + if apierrors.IsNotFound(err) { + return s3Connection{}, "", &ProjectionError{Reason: ReasonSourceMissing, + Message: fmt.Sprintf("source credentials Secret %s/%s not found", cfg.SourceNamespace, cfg.SourceSecretName)} + } + return s3Connection{}, "", &ProjectionError{Reason: ReasonAPIError, Message: fmt.Sprintf("read source credentials Secret: %v", err)} + } + creds, err := parseSourceSecret(src) + if err != nil { + return s3Connection{}, "", err + } + if creds.endpoint == "" { + creds.endpoint = stripScheme(cfg.Endpoint) + } + if creds.endpoint == "" { + return s3Connection{}, "", &ProjectionError{Reason: ReasonSourceMalformed, + Message: fmt.Sprintf("endpoint is empty in both source Secret %s/%s and BACKUP_STORAGE_ENDPOINT", cfg.SourceNamespace, cfg.SourceSecretName)} + } + if creds.region == "" { + creds.region = cfg.Region + } + return s3Connection{ + Endpoint: creds.endpoint, + Secure: strings.HasPrefix(cfg.Endpoint, "https://"), + Region: creds.region, + AccessKey: creds.accessKey, + SecretKey: creds.secretKey, + }, creds.bucket, nil +} + +// retryMariaDBCleanup holds the Backup after a failed attempt and says so, +// unless its namespace is going away, where nothing it needs can be read +// any longer and holding it would hold the namespace. +func (r *BackupReconciler) retryMariaDBCleanup(ctx context.Context, backup *backupsv1alpha1.Backup, location string, cause error) (ctrl.Result, error) { + terminating, err := r.namespaceTerminating(ctx, backup.Namespace) + if err != nil { + return ctrl.Result{}, err + } + if terminating { + return r.releaseMariaDBCleanup(ctx, backup, location, fmt.Sprintf("namespace is terminating and the dump cannot be deleted: %v", cause)), nil + } + getLogger(ctx).Info("cannot delete MariaDB dump; retrying", "backup", backup.Name, "location", location, "error", cause) + if r.Recorder != nil { + r.Recorder.Eventf(backup, corev1.EventTypeWarning, "ArtifactCleanupFailed", + "cannot delete the dump under %s: %v; retrying. Annotate the Backup %s=true to release it and leave the dump", + location, cause, mariadbSkipArtifactCleanupAnnotation) + } + return ctrl.Result{RequeueAfter: mariadbCleanupRetryInterval}, nil +} + +// releaseMariaDBCleanup lets the Backup go without deleting anything further +// and records a Warning naming what was left, so the give-up is visible. +func (r *BackupReconciler) releaseMariaDBCleanup(ctx context.Context, backup *backupsv1alpha1.Backup, location, reason string) ctrl.Result { + getLogger(ctx).Info("releasing Backup without deleting its dump", "backup", backup.Name, "location", location, "reason", reason) + if r.Recorder != nil { + r.Recorder.Eventf(backup, corev1.EventTypeWarning, "ArtifactNotDeleted", "left %s behind: %s", location, reason) + } + return ctrl.Result{} +} diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go new file mode 100644 index 000000000..745f7aef4 --- /dev/null +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -0,0 +1,886 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "errors" + "fmt" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + apimeta "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/tools/record" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + + strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" + "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" +) + +// fakeS3Bucket stands in for the object store on both the listing and the +// delete path, and records what was asked of it. +type fakeS3Bucket struct { + objects []s3ObjectInfo + listErr error + removeErr error + + conns []s3Connection + removed []string +} + +func (f *fakeS3Bucket) ListObjects(_ context.Context, _, _ string, _ bool) ([]s3ObjectInfo, error) { + return f.objects, f.listErr +} + +func (f *fakeS3Bucket) RemoveObject(_ context.Context, bucket, key string) error { + if f.removeErr != nil { + return f.removeErr + } + f.removed = append(f.removed, bucket+"/"+key) + return nil +} + +var ( + cleanupCreated = time.Date(2026, 10, 21, 10, 0, 0, 0, time.UTC) + cleanupCompleted = cleanupCreated.Add(3 * time.Minute) + cleanupDumpName = "backup." + cleanupCreated.Add(time.Minute).Format(time.RFC3339) + ".gzip.sql" +) + +const cleanupPrefix = "tenant/db/" + +// cleanupOperatorBackup is the operator Backup a MariaDB Backup was made of: +// it stored to bkt under tenant/db with the tenant's creds Secret, and its +// Job succeeded. +func cleanupOperatorBackup(t *testing.T) *mariadbtypes.Backup { + t.Helper() + s3 := newRenderedMariaDBTemplate().Storage.S3 + s3.Prefix = "tenant/db" + op := operatorBackupStoringTo(t, s3, cleanupCreated) + op.Status.Conditions = []metav1.Condition{mariadbBackupSucceeded(metav1.NewTime(cleanupCompleted))} + return op +} + +// cleanupBackup is a MariaDB Backup being deleted, made of operator Backup +// op-bk, recording dumpFile when it is not empty. +func cleanupBackup(dumpFile string) *backupsv1alpha1.Backup { + apiGroup := strategyv1alpha1.GroupVersion.Group + md := map[string]string{ + mariadbBackupNameKey: "op-bk", + mariadbBackupNamespaceKey: "tenant", + } + if dumpFile != "" { + md[mariadbDumpFileKey] = dumpFile + } + now := metav1.Now() + return &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", + Name: "bk", + DeletionTimestamp: &now, + Finalizers: []string{backupFinalizer}, + }, + Spec: backupsv1alpha1.BackupSpec{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: &apiGroup, Kind: strategyv1alpha1.MariaDBStrategyKind, Name: "cozy-default-mariadb", + }, + TakenAt: metav1.NewTime(cleanupCompleted), + DriverMetadata: md, + }, + } +} + +func tenantNamespace() *corev1.Namespace { + return &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: "tenant"}} +} + +func cleanupReconciler(t *testing.T, bucket *fakeS3Bucket, objs ...client.Object) (*BackupReconciler, *record.FakeRecorder) { + t.Helper() + return cleanupReconcilerWithClient(newMariaDBStrategyTestClient(t, objs...), bucket) +} + +func cleanupReconcilerWithClient(c client.Client, bucket *fakeS3Bucket) (*BackupReconciler, *record.FakeRecorder) { + rec := record.NewFakeRecorder(20) + return &BackupReconciler{ + Client: c, + Scheme: c.Scheme(), + Recorder: rec, + newS3ObjectLister: func(conn s3Connection) (s3ObjectLister, error) { + bucket.conns = append(bucket.conns, conn) + return bucket, nil + }, + newS3ObjectRemover: func(conn s3Connection) (s3ObjectRemover, error) { + bucket.conns = append(bucket.conns, conn) + return bucket, nil + }, + }, rec +} + +func operatorBackupExists(t *testing.T, c client.Client) bool { + t.Helper() + err := c.Get(context.Background(), client.ObjectKey{Namespace: "tenant", Name: "op-bk"}, &mariadbtypes.Backup{}) + if err != nil && !apierrors.IsNotFound(err) { + t.Fatalf("get operator Backup: %v", err) + } + return err == nil +} + +func requeues(res ctrl.Result) bool { + return res.Requeue || res.RequeueAfter > 0 +} + +func hasEvent(events []string, prefix string) bool { + for _, e := range events { + if strings.HasPrefix(e, prefix) { + return true + } + } + return false +} + +// assertReleasedUntouched checks a Backup that is let go with nothing +// deleted: no S3 delete, the operator Backup kept, and a Warning saying so. +func assertReleasedUntouched(t *testing.T, r *BackupReconciler, rec *record.FakeRecorder, bucket *fakeS3Bucket, res ctrl.Result, err error) { + t.Helper() + if err != nil || requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup released", res, err) + } + if len(bucket.removed) != 0 { + t.Errorf("removed %v, want nothing deleted from the bucket", bucket.removed) + } + if !operatorBackupExists(t, r.Client) { + t.Error("operator Backup deleted, want it kept") + } + if events := drainEvents(rec); !hasEvent(events, "Warning ArtifactNotDeleted") { + t.Errorf("events = %v, want a Warning ArtifactNotDeleted", events) + } +} + +func TestCleanupMariaDBBackup_DeletesRecordedDumpThenOperatorBackup(t *testing.T) { + bucket := &fakeS3Bucket{} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup released", res, err) + } + if want := []string{"bkt/" + cleanupPrefix + cleanupDumpName}; fmt.Sprint(bucket.removed) != fmt.Sprint(want) { + t.Errorf("removed %v, want %v", bucket.removed, want) + } + if operatorBackupExists(t, r.Client) { + t.Error("operator Backup kept, want it deleted once its dump is") + } + if len(bucket.conns) == 0 || bucket.conns[0].Endpoint != "s3.example:9000" || bucket.conns[0].AccessKey != "AKID" { + t.Errorf("connections = %+v, want the operator Backup's endpoint and Secret", bucket.conns) + } + if events := drainEvents(rec); len(events) != 0 { + t.Errorf("events = %v, want none", events) + } +} + +func TestCleanupMariaDBBackup_FailedDeleteKeepsOperatorBackupAndRetries(t *testing.T) { + bucket := &fakeS3Bucket{removeErr: errors.New("503 SlowDown")} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil { + t.Fatalf("err = %v", err) + } + if !requeues(res) { + t.Errorf("res = %+v, want the Backup held and retried", res) + } + if !operatorBackupExists(t, r.Client) { + t.Error("operator Backup deleted although its dump was not") + } + events := drainEvents(rec) + if !hasEvent(events, "Warning ArtifactCleanupFailed") || !strings.Contains(strings.Join(events, "\n"), mariadbSkipArtifactCleanupAnnotation) { + t.Errorf("events = %v, want a Warning ArtifactCleanupFailed naming the skip annotation", events) + } +} + +func TestCleanupMariaDBBackup_FailedOperatorBackupDeleteRetries(t *testing.T) { + bucket := &fakeS3Bucket{} + c := interceptor.NewClient(newMariaDBStrategyTestClient(t, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()).(client.WithWatch), interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + if _, ok := obj.(*mariadbtypes.Backup); ok { + return errors.New("apiserver unavailable") + } + return c.Delete(ctx, obj, opts...) + }, + }) + r, _ := cleanupReconcilerWithClient(c, bucket) + + if _, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)); err == nil { + t.Fatal("err = nil, want the failed operator Backup delete returned so the Backup is held") + } +} + +// A legacy Backup records the dump it proved before deleting it, so a retry +// after a failed operator Backup delete finds the key by name instead of +// matching again against a bucket that no longer holds it. +func TestBackupReconcile_LegacyBackupRetryAfterAFailedOperatorBackupDelete(t *testing.T) { + key := cleanupPrefix + cleanupDumpName + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: key, LastModified: cleanupCreated.Add(2 * time.Minute)}}} + backup := cleanupBackup("") + failDelete := true + c := interceptor.NewClient(newMariaDBStrategyTestClient(t, backup, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()).(client.WithWatch), interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + if _, ok := obj.(*mariadbtypes.Backup); ok && failDelete { + return errors.New("apiserver unavailable") + } + return c.Delete(ctx, obj, opts...) + }, + }) + r, rec := cleanupReconcilerWithClient(c, bucket) + req := ctrl.Request{NamespacedName: client.ObjectKeyFromObject(backup)} + + if _, err := r.Reconcile(context.Background(), req); err == nil { + t.Fatal("first pass: err = nil, want the failed operator Backup delete returned") + } + got := &backupsv1alpha1.Backup{} + if err := r.Get(context.Background(), req.NamespacedName, got); err != nil { + t.Fatalf("Backup gone after a failed operator Backup delete: %v", err) + } + if got.Spec.DriverMetadata[mariadbDumpFileKey] != cleanupDumpName { + t.Errorf("driverMetadata = %v, want the proven dump recorded before its delete", got.Spec.DriverMetadata) + } + + // The dump is gone now, as the first pass deleted it. + bucket.objects = nil + failDelete = false + if _, err := r.Reconcile(context.Background(), req); err != nil { + t.Fatalf("second pass: %v", err) + } + if operatorBackupExists(t, r.Client) { + t.Error("operator Backup kept, want it deleted on the retry") + } + if err := r.Get(context.Background(), req.NamespacedName, &backupsv1alpha1.Backup{}); !apierrors.IsNotFound(err) { + t.Errorf("Backup still present (err=%v), want its finalizer removed", err) + } + if events := drainEvents(rec); hasEvent(events, "Warning ArtifactNotDeleted") { + t.Errorf("events = %v, want no claim that the deleted dump was left behind", events) + } +} + +func TestCleanupMariaDBBackup_LegacyBackupThatCannotRecordItsDumpDeletesNothing(t *testing.T) { + key := cleanupPrefix + cleanupDumpName + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: key, LastModified: cleanupCreated.Add(2 * time.Minute)}}} + backup := cleanupBackup("") + c := interceptor.NewClient(newMariaDBStrategyTestClient(t, backup, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()).(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, c client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + if _, ok := obj.(*backupsv1alpha1.Backup); ok { + return errors.New("apiserver unavailable") + } + return c.Patch(ctx, obj, patch, opts...) + }, + }) + r, _ := cleanupReconcilerWithClient(c, bucket) + + if _, err := r.cleanupMariaDBBackup(context.Background(), backup); err == nil { + t.Fatal("err = nil, want the failed record returned so the Backup is held") + } + if len(bucket.removed) != 0 || !operatorBackupExists(t, r.Client) { + t.Errorf("removed=%v operatorBackupExists=%v, want nothing deleted before the dump is recorded", bucket.removed, operatorBackupExists(t, r.Client)) + } +} + +func TestCleanupMariaDBBackup_RejectsARecordedNameThatIsNotADump(t *testing.T) { + for _, name := range []string{"../other/backup.2026-10-21T10:01:00Z.sql", "x.sql", "sub/" + cleanupDumpName} { + t.Run(name, func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(name)) + assertReleasedUntouched(t, r, rec, bucket, res, err) + }) + } +} + +func TestCleanupMariaDBBackup_LegacyBackupDeletesTheProvenDump(t *testing.T) { + key := cleanupPrefix + cleanupDumpName + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{ + {Key: dumpKey(cleanupPrefix, cleanupCreated.Add(-24*time.Hour), "gzip"), LastModified: cleanupCreated.Add(-24 * time.Hour)}, + {Key: key, LastModified: cleanupCreated.Add(2 * time.Minute)}, + }} + backup := cleanupBackup("") + r, rec := cleanupReconciler(t, bucket, backup, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), backup) + if err != nil || requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup released", res, err) + } + if want := []string{"bkt/" + key}; fmt.Sprint(bucket.removed) != fmt.Sprint(want) { + t.Errorf("removed %v, want %v", bucket.removed, want) + } + if operatorBackupExists(t, r.Client) { + t.Error("operator Backup kept, want it deleted once its dump is") + } + if events := drainEvents(rec); len(events) != 0 { + t.Errorf("events = %v, want none", events) + } + got := &backupsv1alpha1.Backup{} + if err := r.Get(context.Background(), client.ObjectKeyFromObject(backup), got); err != nil { + t.Fatal(err) + } + if got.Spec.DriverMetadata[mariadbDumpFileKey] != cleanupDumpName || got.Spec.DriverMetadata[mariadbDumpTimeKey] != cleanupCreated.Add(time.Minute).Format(time.RFC3339) { + t.Errorf("driverMetadata = %v, want the proven dump and its time recorded", got.Spec.DriverMetadata) + } +} + +// The window is the operator Backup's lifetime, closed by the Backup's +// takenAt: a dump stamped after takenAt plus the skew is not this run's. +func TestCleanupMariaDBBackup_LegacyWindowEndsAtTakenAt(t *testing.T) { + late := cleanupCompleted.Add(mariadbSizeProbeClockSkew + time.Minute) + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{ + {Key: dumpKey(cleanupPrefix, late, "gzip"), LastModified: late}, + }} + op := cleanupOperatorBackup(t) + // The operator's own Complete transition is not what bounds the window. + op.Status.Conditions = []metav1.Condition{mariadbBackupSucceeded(metav1.NewTime(late.Add(time.Hour)))} + r, rec := cleanupReconciler(t, bucket, op, s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup("")) + assertReleasedUntouched(t, r, rec, bucket, res, err) +} + +func TestCleanupMariaDBBackup_LegacyBackupWithoutAProvenDumpIsReleased(t *testing.T) { + inWindow := cleanupCreated.Add(time.Minute) + cases := map[string][]s3ObjectInfo{ + "no dump in its lifetime": { + {Key: dumpKey(cleanupPrefix, cleanupCreated.Add(-24*time.Hour), "gzip"), LastModified: cleanupCreated.Add(-24 * time.Hour)}, + }, + "two dumps in its lifetime": { + {Key: dumpKey(cleanupPrefix, inWindow, "gzip"), LastModified: inWindow}, + {Key: dumpKey(cleanupPrefix, inWindow.Add(30*time.Second), "gzip"), LastModified: inWindow.Add(30 * time.Second)}, + }, + "a dump placed by one clock only": { + {Key: dumpKey(cleanupPrefix, inWindow, "gzip"), LastModified: inWindow.Add(time.Hour)}, + }, + } + for name, objects := range cases { + t.Run(name, func(t *testing.T) { + bucket := &fakeS3Bucket{objects: objects} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup("")) + assertReleasedUntouched(t, r, rec, bucket, res, err) + }) + } +} + +func TestCleanupMariaDBBackup_LegacyBackupWhoseDumpANeighbourCouldOwnIsReleased(t *testing.T) { + inWindow := cleanupCreated.Add(time.Minute) + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: dumpKey(cleanupPrefix, inWindow, "gzip"), LastModified: inWindow}}} + neighbour := cleanupOperatorBackup(t) + neighbour.Name = "op-neighbour" + neighbour.CreationTimestamp = metav1.NewTime(cleanupCreated.Add(30 * time.Second)) + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), neighbour, s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup("")) + assertReleasedUntouched(t, r, rec, bucket, res, err) +} + +func TestCleanupMariaDBBackup_LegacyListingFailureRetries(t *testing.T) { + bucket := &fakeS3Bucket{listErr: errors.New("connection refused")} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup("")) + if err != nil || !requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup held and retried", res, err) + } + if !operatorBackupExists(t, r.Client) { + t.Error("operator Backup deleted, want it kept") + } + if events := drainEvents(rec); !hasEvent(events, "Warning ArtifactCleanupFailed") { + t.Errorf("events = %v, want a Warning ArtifactCleanupFailed", events) + } +} + +func TestCleanupMariaDBBackup_ReleasesWhatItCannotAttribute(t *testing.T) { + t.Run("operator Backup is gone", func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, rec := cleanupReconciler(t, bucket, s3CredsSecret("tenant", "creds"), tenantNamespace()) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 0 { + t.Fatalf("got res=%+v err=%v removed=%v, want released with nothing deleted", res, err, bucket.removed) + } + if events := drainEvents(rec); !hasEvent(events, "Warning ArtifactNotDeleted") { + t.Errorf("events = %v, want a Warning ArtifactNotDeleted", events) + } + }) + t.Run("Backup records no operator Backup", func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), tenantNamespace()) + backup := cleanupBackup(cleanupDumpName) + delete(backup.Spec.DriverMetadata, mariadbBackupNameKey) + res, err := r.cleanupMariaDBBackup(context.Background(), backup) + assertReleasedUntouched(t, r, rec, bucket, res, err) + }) + t.Run("operator Backup's Job failed", func(t *testing.T) { + bucket := &fakeS3Bucket{} + op := cleanupOperatorBackup(t) + op.Status.Conditions[0].Reason = mariadbtypes.ConditionReasonJobFailed + r, rec := cleanupReconciler(t, bucket, op, s3CredsSecret("tenant", "creds"), tenantNamespace()) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + assertReleasedUntouched(t, r, rec, bucket, res, err) + }) + t.Run("operator Backup has no Complete condition", func(t *testing.T) { + bucket := &fakeS3Bucket{} + op := cleanupOperatorBackup(t) + op.Status.Conditions = nil + r, rec := cleanupReconciler(t, bucket, op, s3CredsSecret("tenant", "creds"), tenantNamespace()) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + assertReleasedUntouched(t, r, rec, bucket, res, err) + }) + t.Run("operator Backup never completed", func(t *testing.T) { + bucket := &fakeS3Bucket{} + op := cleanupOperatorBackup(t) + op.Status.Conditions[0].Status = metav1.ConditionFalse + r, rec := cleanupReconciler(t, bucket, op, s3CredsSecret("tenant", "creds"), tenantNamespace()) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + assertReleasedUntouched(t, r, rec, bucket, res, err) + }) + t.Run("dump is on a volume", func(t *testing.T) { + bucket := &fakeS3Bucket{} + op := cleanupOperatorBackup(t) + op.Spec.Storage = mariadbtypes.BackupStorage{} + r, rec := cleanupReconciler(t, bucket, op, tenantNamespace()) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + assertReleasedUntouched(t, r, rec, bucket, res, err) + }) +} + +func TestCleanupMariaDBBackup_SkipAnnotationDeletesNothing(t *testing.T) { + bucket := &fakeS3Bucket{} + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + backup := cleanupBackup(cleanupDumpName) + backup.Annotations = map[string]string{mariadbSkipArtifactCleanupAnnotation: "true"} + + res, err := r.cleanupMariaDBBackup(context.Background(), backup) + if err != nil || requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup released", res, err) + } + if len(bucket.removed) != 0 || !operatorBackupExists(t, r.Client) { + t.Errorf("removed=%v operatorBackupExists=%v, want nothing deleted", bucket.removed, operatorBackupExists(t, r.Client)) + } +} + +func restoreJobOf(name, backupName string, phase backupsv1alpha1.RestoreJobPhase) *backupsv1alpha1.RestoreJob { + return &backupsv1alpha1.RestoreJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: name}, + Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backupName}}, + Status: backupsv1alpha1.RestoreJobStatus{Phase: phase}, + } +} + +func TestCleanupMariaDBBackup_WaitsForARestoreFromIt(t *testing.T) { + for _, phase := range []backupsv1alpha1.RestoreJobPhase{backupsv1alpha1.RestoreJobPhaseEmpty, backupsv1alpha1.RestoreJobPhasePending, backupsv1alpha1.RestoreJobPhaseRunning} { + t.Run(string(phase), func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), + restoreJobOf("rj", "bk", phase)) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || !requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup held while the restore runs", res, err) + } + if len(bucket.removed) != 0 || !operatorBackupExists(t, r.Client) { + t.Errorf("removed=%v operatorBackupExists=%v, want nothing deleted during the restore", bucket.removed, operatorBackupExists(t, r.Client)) + } + if events := drainEvents(rec); !hasEvent(events, "Normal ArtifactCleanupWaiting") || !strings.Contains(strings.Join(events, "\n"), mariadbSkipArtifactCleanupAnnotation) { + t.Errorf("events = %v, want a Normal ArtifactCleanupWaiting naming the skip annotation", events) + } + }) + } + t.Run("finished restores and restores of other Backups do not hold it", func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), + restoreJobOf("done", "bk", backupsv1alpha1.RestoreJobPhaseSucceeded), + restoreJobOf("failed", "bk", backupsv1alpha1.RestoreJobPhaseFailed), + restoreJobOf("other", "other-bk", backupsv1alpha1.RestoreJobPhaseRunning)) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 1 { + t.Fatalf("got res=%+v err=%v removed=%v, want the dump deleted", res, err, bucket.removed) + } + }) +} + +// platformCredsConfig points the reconciler at the platform's credentials +// Secret, the source the per-tenant Secret is projected from. The projected +// Secret is "creds", the one cleanupOperatorBackup names, as the default +// strategy names cozy-backups-creds. +func platformCredsConfig(endpoint string) BackupCredentialsConfig { + return BackupCredentialsConfig{ + SourceNamespace: "cozy-system", + SourceSecretName: "cozy-backups-credentials", + TargetSecretName: "creds", + Endpoint: endpoint, + Region: "us-east-1", + } +} + +func platformCredsSecret(bucketName, endpoint string) *corev1.Secret { + return &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "cozy-system", Name: "cozy-backups-credentials"}, + Data: map[string][]byte{ + "accessKey": []byte("PLATFORM"), + "secretKey": []byte("platform-secret"), + "endpoint": []byte(endpoint), + "bucketName": []byte(bucketName), + }, + } +} + +func TestCleanupMariaDBBackup_SystemBucketUsesTheCurrentEndpoint(t *testing.T) { + bucket := &fakeS3Bucket{} + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), + platformCredsSecret("bkt", "s3.new.example")) + r.CredentialsConfig = platformCredsConfig("https://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 1 { + t.Fatalf("got res=%+v err=%v removed=%v, want the dump deleted", res, err, bucket.removed) + } + for _, conn := range bucket.conns { + if conn.Endpoint != "s3.new.example" || !conn.Secure || conn.AccessKey != "PLATFORM" || conn.Region != "us-east-1" { + t.Errorf("connection = %+v, want the platform's current endpoint over TLS with its keys", conn) + } + } +} + +func TestCleanupMariaDBBackup_OtherBucketUsesTheOperatorBackupsEndpoint(t *testing.T) { + bucket := &fakeS3Bucket{} + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), + platformCredsSecret("another-bucket", "s3.new.example")) + r.CredentialsConfig = platformCredsConfig("https://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 1 { + t.Fatalf("got res=%+v err=%v removed=%v, want the dump deleted", res, err, bucket.removed) + } + for _, conn := range bucket.conns { + if conn.Endpoint != "s3.example:9000" || conn.AccessKey != "AKID" { + t.Errorf("connection = %+v, want the operator Backup's endpoint and Secret", conn) + } + } +} + +func TestCleanupMariaDBBackup_UnreadablePlatformCredentialsFallBackToTheOperatorBackup(t *testing.T) { + for _, bucketName := range []string{"bkt", "tenant-own-bucket"} { + t.Run(bucketName, func(t *testing.T) { + bucket := &fakeS3Bucket{} + op := cleanupOperatorBackup(t) + op.Spec.Storage.S3.Bucket = bucketName + // No platform Secret: the projection source is missing. + r, _ := cleanupReconciler(t, bucket, op, s3CredsSecret("tenant", "creds"), tenantNamespace()) + r.CredentialsConfig = platformCredsConfig("https://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 1 { + t.Fatalf("got res=%+v err=%v removed=%v, want the dump deleted with the operator Backup's own credentials", res, err, bucket.removed) + } + if bucket.conns[0].Endpoint != "s3.example:9000" || bucket.conns[0].AccessKey != "AKID" { + t.Errorf("connection = %+v, want the operator Backup's endpoint and Secret", bucket.conns[0]) + } + }) + } +} + +func TestCleanupMariaDBBackup_UnreadablePlatformCredentialsAndAFailingDeleteHold(t *testing.T) { + bucket := &fakeS3Bucket{removeErr: errors.New("dial tcp: no such host")} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + r.CredentialsConfig = platformCredsConfig("https://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || !requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup held", res, err) + } + if !operatorBackupExists(t, r.Client) { + t.Error("operator Backup deleted although its dump was not") + } + events := drainEvents(rec) + if hasEvent(events, "Warning ArtifactNotDeleted") || !hasEvent(events, "Warning ArtifactCleanupFailed") { + t.Errorf("events = %v, want a Warning ArtifactCleanupFailed and the Backup not released", events) + } +} + +func TestCleanupMariaDBBackup_UnreadablePlatformCredentialsAndAFailingDeleteReleaseInATerminatingNamespace(t *testing.T) { + bucket := &fakeS3Bucket{removeErr: errors.New("dial tcp: no such host")} + ns := tenantNamespace() + now := metav1.Now() + ns.DeletionTimestamp = &now + ns.Finalizers = []string{"kubernetes"} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), ns) + r.CredentialsConfig = platformCredsConfig("https://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + assertReleasedUntouched(t, r, rec, bucket, res, err) +} + +func TestCleanupMariaDBBackup_SystemBucketKeepsTheOperatorBackupsCA(t *testing.T) { + bucket := &fakeS3Bucket{} + op := cleanupOperatorBackup(t) + op.Spec.Storage.S3.TLS = &mariadbtypes.S3TLS{Enabled: true, CASecretKeyRef: &mariadbtypes.SecretKeySelector{Name: "ca", Key: "ca.crt"}} + ca := &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "ca"}, Data: map[string][]byte{"ca.crt": []byte("PEM")}} + r, _ := cleanupReconciler(t, bucket, op, ca, s3CredsSecret("tenant", "creds"), tenantNamespace(), platformCredsSecret("bkt", "s3.new.example")) + r.CredentialsConfig = platformCredsConfig("https://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 1 { + t.Fatalf("got res=%+v err=%v removed=%v, want the dump deleted", res, err, bucket.removed) + } + if string(bucket.conns[0].CAPEM) != "PEM" || bucket.conns[0].Endpoint != "s3.new.example" { + t.Errorf("connection = %+v, want the current endpoint trusting the operator Backup's CA", bucket.conns[0]) + } +} + +func TestCleanupMariaDBBackup_SystemBucketWithAnUnreadableCAIsHeld(t *testing.T) { + bucket := &fakeS3Bucket{} + op := cleanupOperatorBackup(t) + op.Spec.Storage.S3.TLS = &mariadbtypes.S3TLS{Enabled: true, CASecretKeyRef: &mariadbtypes.SecretKeySelector{Name: "ca", Key: "ca.crt"}} + r, rec := cleanupReconciler(t, bucket, op, s3CredsSecret("tenant", "creds"), tenantNamespace(), platformCredsSecret("bkt", "s3.new.example")) + r.CredentialsConfig = platformCredsConfig("https://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || !requeues(res) || len(bucket.removed) != 0 { + t.Fatalf("got res=%+v err=%v removed=%v, want the Backup held with nothing deleted", res, err, bucket.removed) + } + if events := drainEvents(rec); !hasEvent(events, "Warning ArtifactCleanupFailed") { + t.Errorf("events = %v, want a Warning ArtifactCleanupFailed", events) + } +} + +func TestCleanupMariaDBBackup_PlainHTTPPlatformEndpointNeedsNoCA(t *testing.T) { + bucket := &fakeS3Bucket{} + op := cleanupOperatorBackup(t) + op.Spec.Storage.S3.TLS = &mariadbtypes.S3TLS{Enabled: true, CASecretKeyRef: &mariadbtypes.SecretKeySelector{Name: "ca", Key: "ca.crt"}} + r, _ := cleanupReconciler(t, bucket, op, s3CredsSecret("tenant", "creds"), tenantNamespace(), platformCredsSecret("bkt", "s3.new.example")) + r.CredentialsConfig = platformCredsConfig("http://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 1 { + t.Fatalf("got res=%+v err=%v removed=%v, want the dump deleted without reading a CA plain HTTP does not use", res, err, bucket.removed) + } +} + +func TestCleanupMariaDBBackup_PlatformEndpointFallsBackToTheConfiguredOne(t *testing.T) { + bucket := &fakeS3Bucket{} + platform := platformCredsSecret("bkt", "") + delete(platform.Data, "endpoint") + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), platform) + r.CredentialsConfig = platformCredsConfig("https://s3.configured.example") + + if _, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)); err != nil { + t.Fatal(err) + } + if bucket.conns[0].Endpoint != "s3.configured.example" { + t.Errorf("connection = %+v, want BACKUP_STORAGE_ENDPOINT without its scheme", bucket.conns[0]) + } +} + +// blockingBucket answers only when its context ends, as an unreachable +// endpoint does. +type blockingBucket struct{ fakeS3Bucket } + +func (b *blockingBucket) RemoveObject(ctx context.Context, _, _ string) error { + <-ctx.Done() + return ctx.Err() +} + +func TestCleanupMariaDBBackup_BoundsAnUnreachableEndpoint(t *testing.T) { + prev := mariadbDumpCleanupTimeout + mariadbDumpCleanupTimeout = 50 * time.Millisecond + t.Cleanup(func() { mariadbDumpCleanupTimeout = prev }) + bucket := &blockingBucket{} + c := newMariaDBStrategyTestClient(t, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + r := &BackupReconciler{Client: c, Scheme: c.Scheme(), Recorder: record.NewFakeRecorder(20), + newS3ObjectRemover: func(s3Connection) (s3ObjectRemover, error) { return bucket, nil }} + + done := make(chan ctrl.Result, 1) + go func() { + res, _ := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + done <- res + }() + select { + case res := <-done: + if !requeues(res) { + t.Errorf("res = %+v, want the Backup held and retried", res) + } + case <-time.After(5 * time.Second): + t.Fatal("cleanup did not give up on an unreachable endpoint") + } +} + +func TestCleanupMariaDBBackup_ReleasesWhenTheOperatorCRDIsGone(t *testing.T) { + bucket := &fakeS3Bucket{} + c := interceptor.NewClient(newMariaDBStrategyTestClient(t, tenantNamespace()).(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*mariadbtypes.Backup); ok { + return &apimeta.NoKindMatchError{GroupKind: mariadbtypes.GroupVersion.WithKind("Backup").GroupKind()} + } + return c.Get(ctx, key, obj, opts...) + }, + }) + r, rec := cleanupReconcilerWithClient(c, bucket) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 0 { + t.Fatalf("got res=%+v err=%v removed=%v, want released with nothing deleted", res, err, bucket.removed) + } + if events := drainEvents(rec); !hasEvent(events, "Warning ArtifactNotDeleted") { + t.Errorf("events = %v, want a Warning ArtifactNotDeleted", events) + } +} + +func TestCleanupMariaDBBackup_OperatorBackupAlreadyDeletedIsDone(t *testing.T) { + bucket := &fakeS3Bucket{} + c := interceptor.NewClient(newMariaDBStrategyTestClient(t, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()).(client.WithWatch), interceptor.Funcs{ + Delete: func(ctx context.Context, c client.WithWatch, obj client.Object, opts ...client.DeleteOption) error { + if _, ok := obj.(*mariadbtypes.Backup); ok { + return apierrors.NewNotFound(mariadbtypes.GroupVersion.WithResource("backups").GroupResource(), obj.GetName()) + } + return c.Delete(ctx, obj, opts...) + }, + }) + r, _ := cleanupReconcilerWithClient(c, bucket) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup released", res, err) + } +} + +// A strategy of its own that happens to use the platform bucket's name, as +// an admin-chosen name on external S3 can, is reached as recorded: only the +// projected Secret marks a Backup as written to the platform bucket. +func TestCleanupMariaDBBackup_SameBucketNameWithOwnCredentialsUsesTheOperatorBackupsEndpoint(t *testing.T) { + bucket := &fakeS3Bucket{} + op := cleanupOperatorBackup(t) + op.Spec.Storage.S3.AccessKeyIdSecretKeyRef.Name = "own-creds" + op.Spec.Storage.S3.SecretAccessKeySecretKeyRef.Name = "own-creds" + r, _ := cleanupReconciler(t, bucket, op, s3CredsSecret("tenant", "own-creds"), tenantNamespace(), + platformCredsSecret("bkt", "s3.new.example")) + r.CredentialsConfig = platformCredsConfig("https://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 1 { + t.Fatalf("got res=%+v err=%v removed=%v, want the dump deleted", res, err, bucket.removed) + } + if bucket.conns[0].Endpoint != "s3.example:9000" || bucket.conns[0].AccessKey != "AKID" { + t.Errorf("connection = %+v, want the operator Backup's endpoint and Secret", bucket.conns[0]) + } +} + +// With no endpoint anywhere the platform credentials are unusable, which +// falls back like any other unreadable credentials. +func TestCleanupMariaDBBackup_PlatformCredentialsWithoutAnEndpointFallBack(t *testing.T) { + bucket := &fakeS3Bucket{} + platform := platformCredsSecret("bkt", "") + delete(platform.Data, "endpoint") + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), platform) + r.CredentialsConfig = platformCredsConfig("") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 1 { + t.Fatalf("got res=%+v err=%v removed=%v, want the dump deleted", res, err, bucket.removed) + } + if bucket.conns[0].Endpoint != "s3.example:9000" { + t.Errorf("connection = %+v, want the operator Backup's own endpoint", bucket.conns[0]) + } +} + +func TestCleanupMariaDBBackup_MalformedPlatformCredentialsFallBackToTheOperatorBackup(t *testing.T) { + bucket := &fakeS3Bucket{} + malformed := platformCredsSecret("bkt", "s3.new.example") + delete(malformed.Data, "secretKey") + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), malformed) + r.CredentialsConfig = platformCredsConfig("https://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 1 { + t.Fatalf("got res=%+v err=%v removed=%v, want the dump deleted", res, err, bucket.removed) + } + if bucket.conns[0].Endpoint != "s3.example:9000" { + t.Errorf("connection = %+v, want the operator Backup's own endpoint", bucket.conns[0]) + } +} + +func TestCleanupMariaDBBackup_TerminatingNamespaceReleasesOnS3Failure(t *testing.T) { + bucket := &fakeS3Bucket{removeErr: errors.New("connection refused")} + ns := tenantNamespace() + now := metav1.Now() + ns.DeletionTimestamp = &now + ns.Finalizers = []string{"kubernetes"} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), ns) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + assertReleasedUntouched(t, r, rec, bucket, res, err) +} + +// Reconcile keeps the finalizer while the dump cannot be deleted, and drops +// it once both the dump and the operator Backup are gone. +func TestBackupReconcile_MariaDBFinalizerFollowsTheDumpDelete(t *testing.T) { + bucket := &fakeS3Bucket{removeErr: errors.New("503")} + backup := cleanupBackup(cleanupDumpName) + r, _ := cleanupReconciler(t, bucket, backup, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + req := ctrl.Request{NamespacedName: client.ObjectKeyFromObject(backup)} + + if _, err := r.Reconcile(context.Background(), req); err != nil { + t.Fatalf("Reconcile: %v", err) + } + if err := r.Get(context.Background(), req.NamespacedName, &backupsv1alpha1.Backup{}); err != nil { + t.Fatalf("Backup gone while its dump could not be deleted: %v", err) + } + + bucket.removeErr = nil + if _, err := r.Reconcile(context.Background(), req); err != nil { + t.Fatalf("Reconcile: %v", err) + } + if err := r.Get(context.Background(), req.NamespacedName, &backupsv1alpha1.Backup{}); !apierrors.IsNotFound(err) { + t.Errorf("Backup still present (err=%v), want its finalizer removed", err) + } +} + +// The minio adapter deletes the exact key, with the colons of the dump's +// timestamp escaped in the path; SeaweedFS refuses them raw. +func TestMinioObjectRemover_DeletesTheEscapedKey(t *testing.T) { + var method, rawPath, auth string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { + method, rawPath, auth = req.Method, req.URL.EscapedPath(), req.Header.Get("Authorization") + w.WriteHeader(http.StatusNoContent) + })) + defer srv.Close() + + remover, err := newMinioObjectLister(s3Connection{Endpoint: strings.TrimPrefix(srv.URL, "http://"), Region: "us-east-1", AccessKey: "AKID", SecretKey: "s"}) + if err != nil { + t.Fatal(err) + } + if err := remover.(s3ObjectRemover).RemoveObject(context.Background(), "bkt", "tenant#a/db/backup.2026-10-21T10:01:00Z.gzip.sql"); err != nil { + t.Fatalf("RemoveObject: %v", err) + } + if method != http.MethodDelete || rawPath != "/bkt/tenant%23a/db/backup.2026-10-21T10%3A01%3A00Z.gzip.sql" { + t.Errorf("request %s %s, want DELETE of the escaped key", method, rawPath) + } + if !strings.Contains(auth, "Credential=AKID/") { + t.Errorf("request not signed with the access key: %q", auth) + } +} + +func TestMinioObjectRemover_ReturnsTheServersError(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/xml") + w.WriteHeader(http.StatusForbidden) + fmt.Fprint(w, `AccessDeniedAccess Denied`) + })) + defer srv.Close() + + remover, err := newMinioObjectLister(s3Connection{Endpoint: strings.TrimPrefix(srv.URL, "http://"), Region: "us-east-1", AccessKey: "AKID", SecretKey: "s"}) + if err != nil { + t.Fatal(err) + } + if err := remover.(s3ObjectRemover).RemoveObject(context.Background(), "bkt", "k"); err == nil { + t.Fatal("RemoveObject = nil, want the AccessDenied error") + } +} From 9376ba0078714440416a4a0a19d2955a61bcee30 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 17:56:59 +0300 Subject: [PATCH 171/518] fix(backups): hand every driver the same forcePathStyle backupStorage.forcePathStyle reaches drivers that read it differently. The controller, Velero and the ClickHouse sidecar parse it with strconv.ParseBool, the etcd and MongoDB strategies write it into a boolean field that Helm decodes as YAML 1.1, and the Kafka and RabbitMQ scripts take anything but the string "false" as path style. So "False" meant path style to Kafka and RabbitMQ only; "yes" meant path style to etcd, MongoDB, Kafka and RabbitMQ and was refused by Velero and the ClickHouse sidecar; "T" and "1" could not be applied to the boolean fields. The chart now accepts the spellings of strconv.ParseBool and of YAML 1.1 booleans, a superset of the boolean spellings the etcd and MongoDB strategies accepted, and passes every consumer "true" or "false", so they agree. Unset stays unset, so each driver keeps its own default. Any other value fails the render with a message naming it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../templates/_helpers.tpl | 25 + .../templates/deployment.yaml | 2 +- .../templates/strategy-etcd-default.yaml | 2 +- .../templates/strategy-kafka-default.yaml | 2 +- .../templates/strategy-mongodb-default.yaml | 2 +- .../templates/strategy-rabbitmq-default.yaml | 2 +- .../templates/velero-bsl.yaml | 2 +- .../tests/deployment_test.yaml | 9 + .../tests/force_path_style_test.yaml | 501 ++++++++++++++++++ 9 files changed, 541 insertions(+), 6 deletions(-) create mode 100644 packages/system/backupstrategy-controller/tests/force_path_style_test.yaml diff --git a/packages/system/backupstrategy-controller/templates/_helpers.tpl b/packages/system/backupstrategy-controller/templates/_helpers.tpl index 6665cc639..106d65b35 100644 --- a/packages/system/backupstrategy-controller/templates/_helpers.tpl +++ b/packages/system/backupstrategy-controller/templates/_helpers.tpl @@ -187,3 +187,28 @@ true {{- end -}} {{- end -}} {{- end -}} + +{{/* + backupStorage.forcePathStyle as "true", "false", or "" when unset. Its + consumers read it differently: the controller and Velero parse it with + strconv.ParseBool, the etcd and MongoDB strategies write it into a boolean + field that Helm decodes as YAML 1.1 (so "yes" or "off" is a boolean there), + and the Kafka and RabbitMQ scripts compare the string with "false". + Normalising it here means a spelling cannot mean path style to some drivers + and not to others. The spellings accepted are those of strconv.ParseBool and + of YAML 1.1 booleans, a superset of the boolean spellings the etcd and + MongoDB strategies accepted; any other value fails the render. +*/}} +{{- define "backupstrategy-controller.forcePathStyle" -}} +{{- $value := .Values.backupStorage.forcePathStyle -}} +{{- if not (kindIs "invalid" $value) -}} +{{- $raw := toString $value -}} +{{- if has $raw (list "1" "t" "T" "true" "True" "TRUE" "y" "Y" "yes" "Yes" "YES" "on" "On" "ON") -}} +true +{{- else if has $raw (list "0" "f" "F" "false" "False" "FALSE" "n" "N" "no" "No" "NO" "off" "Off" "OFF") -}} +false +{{- else if ne $raw "" -}} +{{- fail (printf "backupStorage.forcePathStyle must be a boolean or unset, got %q" $raw) -}} +{{- end -}} +{{- end -}} +{{- end -}} diff --git a/packages/system/backupstrategy-controller/templates/deployment.yaml b/packages/system/backupstrategy-controller/templates/deployment.yaml index f35e97661..7d5c32fc4 100644 --- a/packages/system/backupstrategy-controller/templates/deployment.yaml +++ b/packages/system/backupstrategy-controller/templates/deployment.yaml @@ -47,7 +47,7 @@ spec: - name: BACKUP_STORAGE_REGION value: {{ .Values.backupStorage.region | quote }} - name: BACKUP_STORAGE_FORCE_PATH_STYLE - value: {{ .Values.backupStorage.forcePathStyle | quote }} + value: {{ include "backupstrategy-controller.forcePathStyle" . | quote }} - name: BACKUP_STORAGE_SYSTEM_NAMESPACES value: {{ .Values.backupStorage.systemNamespaces | join "," | quote }} {{- if .Values.backupStorage.reconcileDefaultObjects }} diff --git a/packages/system/backupstrategy-controller/templates/strategy-etcd-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-etcd-default.yaml index f335f04cf..8d4b069a3 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-etcd-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-etcd-default.yaml @@ -13,7 +13,7 @@ spec: endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} key: {{ printf "{{ .Application.metadata.namespace }}/{{ .Application.metadata.name }}/" | quote }} region: {{ .Values.backupStorage.region | quote }} - forcePathStyle: {{ .Values.backupStorage.forcePathStyle }} + forcePathStyle: {{ include "backupstrategy-controller.forcePathStyle" . }} # The etcd-operator reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY # from this Secret by default (no explicit key fields like the # MariaDB strategy's *SecretKeyRef). The projector writes exactly diff --git a/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml index c65462295..93607fd32 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml @@ -67,7 +67,7 @@ spec: - name: S3_REGION value: {{ .Values.backupStorage.region | quote }} - name: S3_FORCE_PATH_STYLE - value: {{ .Values.backupStorage.forcePathStyle | quote }} + value: {{ include "backupstrategy-controller.forcePathStyle" . | quote }} command: ["/usr/bin/bash", "-c"] # The driver script is kept as a standalone shell file and # embedded verbatim so hack/kafka-backup-script.bats can execute diff --git a/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml index 7509f4bf7..12cc1d160 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml @@ -74,6 +74,6 @@ spec: prefix: {{ printf "{{ .Application.metadata.namespace }}/{{ .Application.metadata.name }}" | quote }} region: {{ .Values.backupStorage.region | quote }} credentialsSecret: cozy-backups-creds - forcePathStyle: {{ .Values.backupStorage.forcePathStyle }} + forcePathStyle: {{ include "backupstrategy-controller.forcePathStyle" . }} insecureSkipTLSVerify: {{ .Values.backupStorage.insecureSkipTLSVerify }} {{- end -}} diff --git a/packages/system/backupstrategy-controller/templates/strategy-rabbitmq-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-rabbitmq-default.yaml index 9c303a6d3..5f1e85db3 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-rabbitmq-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-rabbitmq-default.yaml @@ -96,7 +96,7 @@ spec: - name: S3_REGION value: {{ .Values.backupStorage.region | quote }} - name: S3_FORCE_PATH_STYLE - value: {{ .Values.backupStorage.forcePathStyle | quote }} + value: {{ include "backupstrategy-controller.forcePathStyle" . | quote }} command: ["/bin/sh", "-c"] args: - | diff --git a/packages/system/backupstrategy-controller/templates/velero-bsl.yaml b/packages/system/backupstrategy-controller/templates/velero-bsl.yaml index e0ba05ad8..0621ebf0a 100644 --- a/packages/system/backupstrategy-controller/templates/velero-bsl.yaml +++ b/packages/system/backupstrategy-controller/templates/velero-bsl.yaml @@ -29,7 +29,7 @@ spec: key: cloud config: region: {{ .Values.backupStorage.region | quote }} - s3ForcePathStyle: {{ .Values.backupStorage.forcePathStyle | quote }} + s3ForcePathStyle: {{ include "backupstrategy-controller.forcePathStyle" . | quote }} s3Url: {{ include "backupstrategy-controller.endpoint" . | quote }} {{- end }} {{- end }} diff --git a/packages/system/backupstrategy-controller/tests/deployment_test.yaml b/packages/system/backupstrategy-controller/tests/deployment_test.yaml index 6c17f5f58..f92455a8d 100644 --- a/packages/system/backupstrategy-controller/tests/deployment_test.yaml +++ b/packages/system/backupstrategy-controller/tests/deployment_test.yaml @@ -54,3 +54,12 @@ tests: name: BACKUP_STORAGE_SECRET_NAMESPACE value: tenant-root any: true + + # A value no consumer took as a boolean is refused once, at render time; see + # tests/force_path_style_test.yaml for what each consumer receives. + - it: "forcePathStyle that is no boolean spelling fails the render and names the value" + set: + backupStorage.forcePathStyle: "maybe" + asserts: + - failedTemplate: + errorPattern: 'backupStorage\.forcePathStyle must be a boolean or unset, got "maybe"' diff --git a/packages/system/backupstrategy-controller/tests/force_path_style_test.yaml b/packages/system/backupstrategy-controller/tests/force_path_style_test.yaml new file mode 100644 index 000000000..eda742951 --- /dev/null +++ b/packages/system/backupstrategy-controller/tests/force_path_style_test.yaml @@ -0,0 +1,501 @@ +suite: forcePathStyle — one normalised boolean for every consumer + +# backupStorage.forcePathStyle reaches drivers that read it differently: the +# controller and Velero parse it with strconv.ParseBool, the etcd and MongoDB +# strategies write it into a boolean field Helm decodes as YAML 1.1, and the +# Kafka and RabbitMQ scripts compare the string with "false". The chart accepts +# the spellings of both strconv.ParseBool and YAML 1.1, hands each consumer +# "true" or "false", and refuses anything else, so a spelling cannot mean path +# style to one driver and not to another. + +templates: + - templates/deployment.yaml + - templates/strategy-etcd-default.yaml + - templates/strategy-mongodb-default.yaml + - templates/strategy-kafka-default.yaml + - templates/strategy-rabbitmq-default.yaml + - templates/velero-bsl.yaml + +release: + name: t + namespace: cozy-backup-controller + +set: + backupStorage.bucketNameOverride: test-bucket + +tests: + - it: "forcePathStyle '1' reaches the controller as true" + set: + backupStorage.forcePathStyle: "1" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 't' reaches the controller as true" + set: + backupStorage.forcePathStyle: "t" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 'T' reaches every consumer as true" + set: + backupStorage.forcePathStyle: "T" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + - equal: + path: spec.template.destination.s3.forcePathStyle + value: true + template: templates/strategy-etcd-default.yaml + - equal: + path: spec.template.s3.forcePathStyle + value: true + template: templates/strategy-mongodb-default.yaml + - contains: + path: spec.template.spec.containers[0].env + content: + name: S3_FORCE_PATH_STYLE + value: "true" + template: templates/strategy-kafka-default.yaml + - contains: + path: spec.template.spec.containers[0].env + content: + name: S3_FORCE_PATH_STYLE + value: "true" + template: templates/strategy-rabbitmq-default.yaml + - equal: + path: spec.config.s3ForcePathStyle + value: "true" + template: templates/velero-bsl.yaml + + - it: "forcePathStyle 'true' reaches the controller as true" + set: + backupStorage.forcePathStyle: "true" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 'True' reaches the controller as true" + set: + backupStorage.forcePathStyle: "True" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 'TRUE' reaches the controller as true" + set: + backupStorage.forcePathStyle: "TRUE" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 'y' reaches the controller as true" + set: + backupStorage.forcePathStyle: "y" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 'Y' reaches the controller as true" + set: + backupStorage.forcePathStyle: "Y" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 'yes' reaches every consumer as true" + set: + backupStorage.forcePathStyle: "yes" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + - equal: + path: spec.template.destination.s3.forcePathStyle + value: true + template: templates/strategy-etcd-default.yaml + - equal: + path: spec.template.s3.forcePathStyle + value: true + template: templates/strategy-mongodb-default.yaml + - contains: + path: spec.template.spec.containers[0].env + content: + name: S3_FORCE_PATH_STYLE + value: "true" + template: templates/strategy-kafka-default.yaml + - contains: + path: spec.template.spec.containers[0].env + content: + name: S3_FORCE_PATH_STYLE + value: "true" + template: templates/strategy-rabbitmq-default.yaml + - equal: + path: spec.config.s3ForcePathStyle + value: "true" + template: templates/velero-bsl.yaml + + - it: "forcePathStyle 'Yes' reaches the controller as true" + set: + backupStorage.forcePathStyle: "Yes" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 'YES' reaches the controller as true" + set: + backupStorage.forcePathStyle: "YES" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 'on' reaches the controller as true" + set: + backupStorage.forcePathStyle: "on" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 'On' reaches the controller as true" + set: + backupStorage.forcePathStyle: "On" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle 'ON' reaches the controller as true" + set: + backupStorage.forcePathStyle: "ON" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + + - it: "forcePathStyle '0' reaches the controller as false" + set: + backupStorage.forcePathStyle: "0" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'f' reaches the controller as false" + set: + backupStorage.forcePathStyle: "f" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'F' reaches the controller as false" + set: + backupStorage.forcePathStyle: "F" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'false' reaches the controller as false" + set: + backupStorage.forcePathStyle: "false" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'False' reaches every consumer as false" + set: + backupStorage.forcePathStyle: "False" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + - equal: + path: spec.template.destination.s3.forcePathStyle + value: false + template: templates/strategy-etcd-default.yaml + - equal: + path: spec.template.s3.forcePathStyle + value: false + template: templates/strategy-mongodb-default.yaml + - contains: + path: spec.template.spec.containers[0].env + content: + name: S3_FORCE_PATH_STYLE + value: "false" + template: templates/strategy-kafka-default.yaml + - contains: + path: spec.template.spec.containers[0].env + content: + name: S3_FORCE_PATH_STYLE + value: "false" + template: templates/strategy-rabbitmq-default.yaml + - equal: + path: spec.config.s3ForcePathStyle + value: "false" + template: templates/velero-bsl.yaml + + - it: "forcePathStyle 'FALSE' reaches the controller as false" + set: + backupStorage.forcePathStyle: "FALSE" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'n' reaches the controller as false" + set: + backupStorage.forcePathStyle: "n" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'N' reaches the controller as false" + set: + backupStorage.forcePathStyle: "N" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'no' reaches the controller as false" + set: + backupStorage.forcePathStyle: "no" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'No' reaches the controller as false" + set: + backupStorage.forcePathStyle: "No" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'NO' reaches the controller as false" + set: + backupStorage.forcePathStyle: "NO" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'off' reaches every consumer as false" + set: + backupStorage.forcePathStyle: "off" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + - equal: + path: spec.template.destination.s3.forcePathStyle + value: false + template: templates/strategy-etcd-default.yaml + - equal: + path: spec.template.s3.forcePathStyle + value: false + template: templates/strategy-mongodb-default.yaml + - contains: + path: spec.template.spec.containers[0].env + content: + name: S3_FORCE_PATH_STYLE + value: "false" + template: templates/strategy-kafka-default.yaml + - contains: + path: spec.template.spec.containers[0].env + content: + name: S3_FORCE_PATH_STYLE + value: "false" + template: templates/strategy-rabbitmq-default.yaml + - equal: + path: spec.config.s3ForcePathStyle + value: "false" + template: templates/velero-bsl.yaml + + - it: "forcePathStyle 'Off' reaches the controller as false" + set: + backupStorage.forcePathStyle: "Off" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle 'OFF' reaches the controller as false" + set: + backupStorage.forcePathStyle: "OFF" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "false" + template: templates/deployment.yaml + + - it: "forcePathStyle as a YAML boolean (the default) reaches every consumer as true" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "true" + template: templates/deployment.yaml + - equal: + path: spec.template.destination.s3.forcePathStyle + value: true + template: templates/strategy-etcd-default.yaml + + - it: "forcePathStyle as a YAML boolean false reaches the scripts as false" + set: + backupStorage.forcePathStyle: false + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: S3_FORCE_PATH_STYLE + value: "false" + template: templates/strategy-kafka-default.yaml + - equal: + path: spec.template.s3.forcePathStyle + value: false + template: templates/strategy-mongodb-default.yaml + + - it: "forcePathStyle as the number 1 reaches the boolean fields as true" + set: + backupStorage.forcePathStyle: 1 + asserts: + - equal: + path: spec.template.destination.s3.forcePathStyle + value: true + template: templates/strategy-etcd-default.yaml + + # Unset leaves each driver its own default, as before. + - it: "forcePathStyle null stays unset for every consumer" + set: + backupStorage.forcePathStyle: null + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "" + template: templates/deployment.yaml + - equal: + path: spec.template.destination.s3.forcePathStyle + value: null + template: templates/strategy-etcd-default.yaml + - equal: + path: spec.template.s3.forcePathStyle + value: null + template: templates/strategy-mongodb-default.yaml + + - it: "forcePathStyle empty stays unset for every consumer" + set: + backupStorage.forcePathStyle: "" + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: BACKUP_STORAGE_FORCE_PATH_STYLE + value: "" + template: templates/deployment.yaml + - equal: + path: spec.template.destination.s3.forcePathStyle + value: null + template: templates/strategy-etcd-default.yaml From 477540b8f163d791d245a8c0a0e48cdb36b33623 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 16:02:11 +0300 Subject: [PATCH 172/518] fix(backups): keep projecting credentials when forcePathStyle cannot be parsed The credentials projection carries every backup driver's keys, but only the ClickHouse sidecar's file needs forcePathStyle parsed as a boolean. A value that failed the parse failed the whole projection as a malformed source Secret, a terminal reason pointing at the wrong object, so CNPG, MariaDB, etcd and the other drivers lost their backups over a parse that serves one file. The value is now left out of that file and the reason is logged at error level with the namespace; the rest of the projection proceeds. The chart refuses such a value at render time, so this covers an environment set around it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backupcontroller/credentials_projector.go | 30 +++++++---- .../credentials_projector_test.go | 53 ++++++++++++++++--- 2 files changed, 67 insertions(+), 16 deletions(-) diff --git a/internal/backupcontroller/credentials_projector.go b/internal/backupcontroller/credentials_projector.go index ddf5ff8ef..576991d86 100644 --- a/internal/backupcontroller/credentials_projector.go +++ b/internal/backupcontroller/credentials_projector.go @@ -18,6 +18,7 @@ import ( "k8s.io/apimachinery/pkg/types" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil" + "sigs.k8s.io/controller-runtime/pkg/log" "sigs.k8s.io/yaml" ) @@ -88,8 +89,8 @@ type BackupCredentialsConfig struct { TargetSecretName string Endpoint string Region string - // ForcePathStyle is the string "true"/"false" from - // backupStorage.forcePathStyle, projected into the credentials Secret + // ForcePathStyle is backupStorage.forcePathStyle as the chart + // normalises it ("true", "false", or "" when unset), projected into the credentials Secret // under the forcePathStyle key and into clickhouse-backup.yml. The app // charts cannot read backupStorage.* directly, so the ClickHouse // sidecar takes it from there. @@ -181,7 +182,14 @@ func ProjectBackupCredentials(ctx context.Context, c client.Client, cfg BackupCr } cloud := buildVeleroCredentialsFile(creds.accessKey, creds.secretKey) - clickHouseConfig, err := buildClickHouseBackupConfig(creds, cfg.ForcePathStyle) + clickHouseConfig, pathStyleErr, err := buildClickHouseBackupConfig(creds, cfg.ForcePathStyle) + if pathStyleErr != nil { + // The chart refuses such a value at render time, so only an env set + // around it gets here. It concerns the ClickHouse file alone, and + // failing the projection would take every other driver's + // credentials down with it. + log.FromContext(ctx).Error(pathStyleErr, "left force_path_style out of "+clickHouseBackupConfigKey, "namespace", targetNamespace) + } if err != nil { return &ProjectionError{ Reason: ReasonSourceMalformed, @@ -449,8 +457,9 @@ const clickHouseBackupConfigKey = "clickhouse-backup.yml" // buildClickHouseBackupConfig renders that file. The projected endpoint is // a bare host, and clickhouse-backup's S3 client rejects one without a -// scheme. -func buildClickHouseBackupConfig(creds parsedCreds, forcePathStyle string) ([]byte, error) { +// scheme. A forcePathStyle that is not a boolean is left out of the file +// and reported in pathStyleErr; err is a failure to render the file at all. +func buildClickHouseBackupConfig(creds parsedCreds, forcePathStyle string) (config []byte, pathStyleErr, err error) { s3 := map[string]any{ "bucket": creds.bucket, "endpoint": "https://" + creds.endpoint, @@ -463,13 +472,14 @@ func buildClickHouseBackupConfig(creds parsedCreds, forcePathStyle string) ([]by s3["region"] = creds.region } if forcePathStyle != "" { - pathStyle, err := strconv.ParseBool(forcePathStyle) - if err != nil { - return nil, fmt.Errorf("forcePathStyle %q is not a boolean", forcePathStyle) + if pathStyle, perr := strconv.ParseBool(forcePathStyle); perr != nil { + pathStyleErr = fmt.Errorf("forcePathStyle %q is not a boolean", forcePathStyle) + } else { + s3["force_path_style"] = pathStyle } - s3["force_path_style"] = pathStyle } - return yaml.Marshal(map[string]any{"s3": s3}) + config, err = yaml.Marshal(map[string]any{"s3": s3}) + return config, pathStyleErr, err } // stripScheme returns the endpoint stripped of any leading scheme prefix diff --git a/internal/backupcontroller/credentials_projector_test.go b/internal/backupcontroller/credentials_projector_test.go index 78ffb6c94..dd3cf131b 100644 --- a/internal/backupcontroller/credentials_projector_test.go +++ b/internal/backupcontroller/credentials_projector_test.go @@ -5,15 +5,19 @@ import ( "encoding/json" "errors" "fmt" + "strings" "testing" "time" + "github.com/go-logr/logr/funcr" + corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/log" "sigs.k8s.io/yaml" ) @@ -746,14 +750,51 @@ func TestProject_ClickHouseBackupConfigOmitsAnEmptyRegion(t *testing.T) { } } -func TestProject_RejectsMalformedForcePathStyle(t *testing.T) { +// The parse only serves the ClickHouse file, while the same projection +// carries every other driver's credentials. An unparseable platform value +// must cost ClickHouse its path-style setting and say so, not fail the +// projection for all of them. +func TestProject_MalformedForcePathStyleSkipsOnlyTheClickHouseSetting(t *testing.T) { c := newFakeClient(flatSourceSecret()) + var logged []string + ctx := log.IntoContext(context.Background(), funcr.New(func(prefix, args string) { + logged = append(logged, args) + }, funcr.Options{})) cfg := defaultCfg() - cfg.ForcePathStyle = "yes please" - err := ProjectBackupCredentials(context.Background(), c, cfg, "tenant-acme") - var perr *ProjectionError - if !errors.As(err, &perr) || perr.Reason != ReasonSourceMalformed { - t.Fatalf("err = %v, want a %s ProjectionError", err, ReasonSourceMalformed) + cfg.ForcePathStyle = "yes" + if err := ProjectBackupCredentials(ctx, c, cfg, "tenant-acme"); err != nil { + t.Fatalf("projection failed for every driver over a ClickHouse-only setting: %v", err) + } + got := &corev1.Secret{} + if err := c.Get(ctx, types.NamespacedName{Namespace: "tenant-acme", Name: "cozy-backups-creds"}, got); err != nil { + t.Fatalf("projected Secret not found: %v", err) + } + if string(got.Data["AWS_ACCESS_KEY_ID"]) != "AK" { + t.Errorf("AWS_ACCESS_KEY_ID = %q, want the projected key", got.Data["AWS_ACCESS_KEY_ID"]) + } + if string(got.Data["forcePathStyle"]) != "yes" { + t.Errorf("forcePathStyle = %q, want the platform value carried verbatim", got.Data["forcePathStyle"]) + } + var doc struct { + S3 map[string]any `json:"s3"` + } + if err := yaml.Unmarshal(got.Data[clickHouseBackupConfigKey], &doc); err != nil { + t.Fatalf("%s is not YAML: %v", clickHouseBackupConfigKey, err) + } + if doc.S3["access_key"] != "AK" { + t.Errorf("s3.access_key = %v, want the rest of the file written", doc.S3["access_key"]) + } + if pathStyle, set := doc.S3["force_path_style"]; set { + t.Errorf("s3.force_path_style = %v, want it left out", pathStyle) + } + found := false + for _, line := range logged { + if strings.Contains(line, `"error"`) && strings.Contains(line, `\"yes\"`) && strings.Contains(line, "force_path_style") { + found = true + } + } + if !found { + t.Errorf("no error naming the rejected value and the skipped setting was logged; got %q", logged) } } From c76080cc073d4282c4c968cc191c397e95be8e0a Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 17:04:28 +0300 Subject: [PATCH 173/518] test(clickhouse): pin the archive name across a credentials wait A backup's archive is named from the BackupJob's persisted StartedAt, and the restore of a Backup that records no archive picks one stamped no later than its takenAt. The credentials wait holds the Job after StartedAt is set and writes the job's status while it holds. Should a held reconcile restamp StartedAt, the Pod would write an archive the earlier reconciles never named; nothing caught that. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../altinitystrategy_controller_test.go | 76 +++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index 942957b7d..48592ac54 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -1495,3 +1495,79 @@ func TestReconcileAltinityRestore_UnreadableHoldStateDoesNotStartTheJob(t *testi }) } } + +// A backup held for the credentials keeps the archive name it was given from +// its persisted StartedAt: the held reconciles write the job's status, and the +// Pod created after the hold, and the Backup recorded after it, must still +// name the same archive, which is stamped no later than the Backup's takenAt. +func TestReconcileAltinity_CredentialsWaitKeepsTheArchiveName(t *testing.T) { + startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC)) + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + BackupClassName: "clickhouse-backup", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt, Phase: backupsv1alpha1.BackupJobPhaseRunning}, + } + secret := projectedCredentials("tenant-test", time.Now()) + r, _, _ := newAltinityTestEnv(t, newAltinityClickHouseApp("ch-test", "tenant-test"), + clientfake.NewClientBuilder().WithObjects(backupJob, newAltinityArchiveStrategy("clickhouse-strategy"), secret)) + r.CredentialsConfig.TargetSecretName = "cozy-backups-creds" + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + } + ctx := context.Background() + const want = "clickhouse-ch-test-20260304T050607Z" + reconcile := func(step string) { + t.Helper() + j := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), j); err != nil { + t.Fatalf("%s: get BackupJob: %v", step, err) + } + if _, err := r.reconcileAltinity(ctx, j, resolved); err != nil { + t.Fatalf("%s: reconcileAltinity: %v", step, err) + } + } + + reconcile("held") + job := &batchv1.Job{} + if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}, job); !apierrors.IsNotFound(err) { + t.Fatalf("the Job exists while the credentials wait should hold it: %v", err) + } + + if err := r.Get(ctx, client.ObjectKeyFromObject(secret), secret); err != nil { + t.Fatalf("get projected Secret: %v", err) + } + secret.Annotations[credentialsUpdatedAtAnnotation] = time.Now().Add(-altinityCredentialsPropagation - time.Second).UTC().Format(time.RFC3339) + if err := r.Update(ctx, secret); err != nil { + t.Fatalf("age the credentials stamp: %v", err) + } + reconcile("after the wait") + if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}, job); err != nil { + t.Fatalf("get backup Job: %v", err) + } + if got := job.Spec.Template.Spec.Containers[0].Args[1]; got != "--archive="+want { + t.Fatalf("backup Pod was rendered with %q after the wait, want --archive=%s", got, want) + } + + job.Status.Conditions = []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}} + if err := r.Status().Update(ctx, job); err != nil { + t.Fatalf("mark backup Job complete: %v", err) + } + reconcile("completed") + created := &backupsv1alpha1.Backup{} + if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: backupJob.Name}, created); err != nil { + t.Fatalf("get Backup: %v", err) + } + if got := created.Spec.DriverMetadata[altinityArchiveMetaKey]; got != want { + t.Errorf("Backup records archive %q, want %q", got, want) + } + if stamp := created.Spec.TakenAt.UTC().Format(altinityArchiveTimeLayout); stamp < "20260304T050607Z" { + t.Errorf("Backup takenAt %s is earlier than its archive's stamp", stamp) + } +} From 6280d06d010e8c1c829cad13aec74e8630f1cf48 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 17:47:51 +0300 Subject: [PATCH 174/518] test(clickhouse): describe the round-trip's guards by the archive-recording restore Restores now take the archive the Backup recorded rather than the newest one, and a leftover Backup of the BackupJob's name that records another archive fails the new BackupJob instead of being adopted. The comments justifying the sentinel row and the Backup cleanup, in the round-trip and in its unit test, still gave the earlier reasons, which would send the next reader after behaviour the driver no longer has. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/clickhouse-system-bucket-shared-state.bats | 6 +++--- hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml | 3 ++- hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/hack/clickhouse-system-bucket-shared-state.bats b/hack/clickhouse-system-bucket-shared-state.bats index e76cc0a1d..ec4900cca 100644 --- a/hack/clickhouse-system-bucket-shared-state.bats +++ b/hack/clickhouse-system-bucket-shared-state.bats @@ -92,9 +92,9 @@ restore() { jq -e '.data.Corefile == ".:53 {\n errors\n}\n"' "$STUB/replaced.json" >/dev/null } -# The Altinity driver names the Backup after the BackupJob and adopts one that -# already exists under that name, so a Backup left behind would stand in for -# the next run's backup. It is reclaimed by name, whether or not the BackupJob +# The Altinity driver names the Backup after the BackupJob, and a Backup left +# behind under that name records an earlier run's archive, which fails the next +# run's BackupJob. It is reclaimed by name, whether or not the BackupJob # recorded it in status.backupRef. @test "the round-trip reclaims its Backup by name before and after the run" { suite=hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index c48e32f24..a4c4aef40 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -452,7 +452,8 @@ spec: fi # Fixed names in a persistent namespace: a leftover Succeeded job # would satisfy the waits in the run step without a new attempt, and - # the driver adopts a leftover Backup of the BackupJob's name. + # a leftover Backup of the BackupJob's name, recording an earlier + # run's archive, fails the new BackupJob. kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s kubectl -n "$NAMESPACE" delete backup sysbucket-backup --ignore-not-found --wait=true --timeout=60s diff --git a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh index 73d2f0e63..c1c216389 100755 --- a/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh +++ b/hack/e2e-chainsaw/clickhouse/system-bucket-roundtrip.sh @@ -120,8 +120,8 @@ sidecar_before=$(sidecar_identity) [[ "$sidecar_before" =~ ^[0-9a-f-]+\ [0-9]+$ ]] \ || { log_error "could not read the ClickHouse Pod uid and clickhouse-backup restart count (got '${sidecar_before}')"; exit 1; } -# Restore picks the newest remote backup of the release, so only a row no -# earlier run wrote shows that the archive restored is this run's. +# Restore takes the archive this run's Backup recorded; a row no earlier run +# wrote is what shows the data restored is that archive's. sentinel="before-backup-$(date -u +%Y%m%dT%H%M%SZ)-${RANDOM}" log_step "Writing the sentinel row ${sentinel}" clickhouse_query "$CLICKHOUSE_NAME" "CREATE TABLE IF NOT EXISTS default.sentinel (id UInt32, name String) ENGINE = MergeTree ORDER BY id" From 291b1fa3c042ddc90c37b531211ebe23657b1da4 Mon Sep 17 00:00:00 2001 From: Yasar Rahman Date: Tue, 13 Oct 2026 19:20:34 +0500 Subject: [PATCH 175/518] fix(vpa): shorten the admission webhook timeout to 5s The VPA admission controller registers its Pod CREATE webhook in every namespace with failurePolicy Ignore and a 30s timeout. When its pod is down and the Service IP blackholes (a cold start after a full cluster shutdown, before the CNI is back), each Pod creation waits the full 30s here on top of the other webhooks and fails with "context deadline exceeded". That includes the CNI's own DaemonSet pods, so the cluster cannot recover without deleting the webhook configuration by hand. A mutation takes milliseconds; 5s keeps slow responses working and lets Ignore actually ignore. Uses the admission controller's documented --webhook-timeout-seconds flag through the chart's extraArgs. Refs #3125 Assisted-by: LLM Signed-off-by: Yasar Rahman --- packages/system/vertical-pod-autoscaler/values.yaml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/packages/system/vertical-pod-autoscaler/values.yaml b/packages/system/vertical-pod-autoscaler/values.yaml index 46678903e..b870049be 100644 --- a/packages/system/vertical-pod-autoscaler/values.yaml +++ b/packages/system/vertical-pod-autoscaler/values.yaml @@ -37,6 +37,15 @@ vertical-pod-autoscaler: prometheus-cadvisor-job-name: cadvisor admissionController: + extraArgs: + # The admission controller registers its mutating webhook for Pod CREATE in every + # namespace with failurePolicy Ignore and, by default, a 30s timeout. When its pod is + # down and the Service IP blackholes (e.g. a cold start after a full cluster shutdown, + # before the CNI is back), every Pod creation waits the full 30s here on top of the + # other webhooks and fails with "context deadline exceeded" - including the CNI's own + # DaemonSet pods, so the cluster cannot recover by itself (#3125). A mutation takes + # milliseconds; 5s keeps a slow response working and lets Ignore actually ignore. + webhook-timeout-seconds: 5 resources: limits: memory: 256Mi From d1b472319a24e95ad9b844151a5d41aa29222ff7 Mon Sep 17 00:00:00 2001 From: Yasar Rahman Date: Tue, 13 Oct 2026 19:45:30 +0500 Subject: [PATCH 176/518] fix(vpa): explain that the webhook waits add up within one request failurePolicy Ignore applies only after the timeout; the Pod CREATE webhooks are called sequentially within one request, so their waits accumulate and can pass the request deadline. Assisted-by: LLM Signed-off-by: Yasar Rahman --- .../system/vertical-pod-autoscaler/values.yaml | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/packages/system/vertical-pod-autoscaler/values.yaml b/packages/system/vertical-pod-autoscaler/values.yaml index b870049be..a828ee95b 100644 --- a/packages/system/vertical-pod-autoscaler/values.yaml +++ b/packages/system/vertical-pod-autoscaler/values.yaml @@ -39,12 +39,14 @@ vertical-pod-autoscaler: admissionController: extraArgs: # The admission controller registers its mutating webhook for Pod CREATE in every - # namespace with failurePolicy Ignore and, by default, a 30s timeout. When its pod is - # down and the Service IP blackholes (e.g. a cold start after a full cluster shutdown, - # before the CNI is back), every Pod creation waits the full 30s here on top of the - # other webhooks and fails with "context deadline exceeded" - including the CNI's own - # DaemonSet pods, so the cluster cannot recover by itself (#3125). A mutation takes - # milliseconds; 5s keeps a slow response working and lets Ignore actually ignore. + # namespace with failurePolicy Ignore and, by default, a 30s timeout. Ignore only + # applies after the timeout: when its pod is down and the Service IP blackholes (e.g. a + # cold start after a full cluster shutdown, before the CNI is back), the API server + # still waits the full 30s. The Pod CREATE webhooks are called one after another within + # one request, so their waits add up; past the request's deadline the creation fails + # with "context deadline exceeded", including for the CNI's own DaemonSet pods (#3125). + # A mutation takes milliseconds; 5s keeps slow responses working and keeps the total + # wait well inside the deadline. webhook-timeout-seconds: 5 resources: limits: From d0f47f7dd5a7f57147fd1c67347ba2a9994ae49f Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:20:29 +0300 Subject: [PATCH 177/518] fix(backups): compare the credentials endpoint only for a Secret derived from COSI backupStorage.systemSecretName may name a Secret the bucket chart does not render from a COSI Secret: the raw COSI Secret itself, which the projector accepts, or any Secret without the -credentials suffix. The COSI Secret name was then the source name unchanged, so the gate compared a Secret with itself and either reported it stale on every tick or failed the check, and stopped repairing missing default objects as well. Skip the comparison when there is no distinct COSI Secret or no endpoint key to compare. The gate compares the annotation with what the chart's endpoint helper renders, and the helper renders a present but blank endpoint as a bare https:// instead of falling back, so the gate has to compute the same value or its expectation and the chart disagree for that Secret. The gate's docs, metric help and log lines named only missing objects, so they did not cover a stale endpoint they now report. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backupcontroller/default_objects_gate.go | 41 ++++++++++------ .../default_objects_gate_test.go | 49 +++++++++++++++++++ .../templates/deployment.yaml | 2 + .../tests/default_objects_gate_test.yaml | 13 +++++ .../backupstrategy-controller/values.yaml | 6 +-- 5 files changed, 92 insertions(+), 19 deletions(-) diff --git a/internal/backupcontroller/default_objects_gate.go b/internal/backupcontroller/default_objects_gate.go index efcd970d5..d0ed29954 100644 --- a/internal/backupcontroller/default_objects_gate.go +++ b/internal/backupcontroller/default_objects_gate.go @@ -51,15 +51,16 @@ var defaultObjectsMissing = prometheus.NewGaugeVec( ) // defaultObjectsForceReconciles counts the forced Helm upgrades the gate -// issued to materialise those objects. A counter that keeps climbing means -// the forced render is not producing the objects (e.g. a CRD is missing), +// issued to materialise those objects or to refresh their S3 endpoint. A +// counter that keeps climbing means the forced render is not producing the +// objects (e.g. a CRD is missing) or not changing their endpoint, // which is a different failure than the install-time race this gate closes. // A suspended release is NOT counted here: it is skipped before the patch, // so a paused release cannot masquerade as a render that keeps failing. var defaultObjectsForceReconciles = prometheus.NewCounterVec( prometheus.CounterOpts{ Name: "cozystack_backup_default_objects_force_reconciles_total", - Help: "Number of forced HelmRelease reconciles issued to create missing platform backup objects.", + Help: "Number of forced HelmRelease reconciles issued to create missing platform backup objects or refresh their S3 endpoint.", }, []string{"namespace", "name"}, ) @@ -125,8 +126,9 @@ var backupStorageLocationGVR = schema.GroupVersionResource{ // is resolvable — read from the same platform credentials Secret the // projector already consumes, so no new dependency and no `lookup` — it // checks that every object the platform BackupClass routes to actually -// exists, and forces one Helm upgrade when any is missing. A forced -// upgrade re-runs the lookups, so the gated templates materialise. +// exists, and forces one Helm upgrade when any is missing or carries a stale +// S3 endpoint. A forced upgrade re-runs the lookups, so the gated templates +// materialise with the current endpoint. // // It deliberately does not create the objects itself: their bodies are // values-driven (endpoint, region, forcePathStyle, the Altinity strategy's @@ -317,8 +319,9 @@ func (g *DefaultObjectsGate) checkAndLog(ctx context.Context, logger logr.Logger } // forcedReleaseFor reports which release renders the given missing set, for -// logging only. reconcileCredentials returns the credentials Secret as the -// sole missing object, and it is never mixed with the routed objects — +// logging only. reconcileCredentials and staleCredentials return the +// credentials Secret as the sole entry, and it is never mixed with the routed +// objects — // resolving the bucket name is a precondition for checking those at all. func (g *DefaultObjectsGate) forcedReleaseFor(missing []string) types.NamespacedName { credsKey := fmt.Sprintf("Secret/%s", g.Config.SourceSecretName) @@ -329,9 +332,10 @@ func (g *DefaultObjectsGate) forcedReleaseFor(missing []string) types.Namespaced } // Check resolves the platform bucket name, verifies that every object the -// BackupClass routes to exists, and forces one Helm upgrade when any is -// missing. It returns the missing objects (formatted for logs) and whether -// a forced upgrade was issued on this call. +// BackupClass routes to exists and carries the current S3 endpoint, and +// forces one Helm upgrade when any is missing or stale. It returns those +// objects (formatted for logs) and whether a forced upgrade was issued on this +// call. // // While the source Secret is absent or carries no bucket name, the gate // cannot evaluate the routed objects — their templates gate on the same @@ -412,11 +416,16 @@ func (g *DefaultObjectsGate) Check(ctx context.Context) ([]string, bool, error) // its endpoint differs from the one in the COSI Secret it is rendered from, // comparing against what packages/system/bucket/templates/user-credentials.yaml // writes for it. The COSI Secret being absent (the BucketAccess is being -// re-granted) leaves nothing to compare against and is not staleness. +// re-granted) leaves nothing to compare against and is not staleness, and +// neither is a source Secret the bucket chart does not render: the COSI Secret +// itself (the projector accepts one) or one without the endpoint key. func (g *DefaultObjectsGate) staleCredentials(ctx context.Context, src *corev1.Secret) (string, error) { if g.CredentialsCOSISecretName == "" || g.CredentialsHelmRelease.Name == "" || g.CredentialsHelmRelease.Namespace == "" { return "", nil } + if _, ok := src.Data["endpoint"]; !ok || g.CredentialsCOSISecretName == g.Config.SourceSecretName { + return "", nil + } cosi := &corev1.Secret{} key := types.NamespacedName{Namespace: g.Config.SourceNamespace, Name: g.CredentialsCOSISecretName} if err := g.Client.Get(ctx, key, cosi); err != nil { @@ -446,13 +455,13 @@ func (g *DefaultObjectsGate) staleCredentials(ctx context.Context, src *corev1.S // renderedEndpointFor mirrors the "backupstrategy-controller.endpoint" helper // for a provisioned bucket: the credentials Secret's endpoint, trimmed, with // any scheme replaced by https://. It is the value the chart stamps as -// endpointAnnotation. An empty endpoint makes the helper fall back to -// backupStorage.endpoint, which the gate cannot see, so it returns "". +// endpointAnnotation. An absent or empty endpoint makes the helper fall back +// to backupStorage.endpoint, which the gate cannot see, so it returns "". func renderedEndpointFor(raw []byte) string { - host := strings.TrimSpace(string(raw)) - if host == "" { + if len(raw) == 0 { return "" } + host := strings.TrimSpace(string(raw)) return "https://" + strings.TrimPrefix(strings.TrimPrefix(host, "https://"), "http://") } @@ -511,7 +520,7 @@ func (g *DefaultObjectsGate) force(ctx context.Context, target types.NamespacedN // annotations would be ignored, so skip the patch entirely and // leave the force counter alone — a climbing counter must keep // meaning "the forced render is not producing the objects". - log.FromContext(ctx).Info("skipped forcing a suspended HelmRelease; missing default backup objects will not be repaired until it is resumed", + log.FromContext(ctx).Info("skipped forcing a suspended HelmRelease; missing or stale default backup objects will not be repaired until it is resumed", "helmRelease", target.String(), "missing", missing) *last = now return missing, false, nil diff --git a/internal/backupcontroller/default_objects_gate_test.go b/internal/backupcontroller/default_objects_gate_test.go index dc1c0c9f0..cff6187cd 100644 --- a/internal/backupcontroller/default_objects_gate_test.go +++ b/internal/backupcontroller/default_objects_gate_test.go @@ -1099,6 +1099,9 @@ func TestRenderedEndpointMatchesTheChart(t *testing.T) { // No endpoint in the Secret: the chart renders its values fallback, // which the gate cannot reproduce, so it compares nothing. {"", ""}, + // Present but blank: the helper's lookup sees a non-empty value and + // renders the bare scheme. + {"\n", "https://"}, } { if got := renderedEndpointFor([]byte(tc.in)); got != tc.want { t.Errorf("renderedEndpointFor(%q) = %q, want %q", tc.in, got, tc.want) @@ -1125,3 +1128,49 @@ func TestForcedReleaseForNamesTheStaleCredentialsRelease(t *testing.T) { } } } + +// TestCheckSkipsCredentialsComparisonWithoutDerivedSecret covers a +// systemSecretName the bucket chart does not render: the same name as the COSI +// Secret (the projector also reads a raw COSI Secret) or a Secret without the +// endpoint key. Comparing it would report it stale on every tick, or fail the +// check, and the missing objects would never be repaired. +func TestCheckSkipsCredentialsComparisonWithoutDerivedSecret(t *testing.T) { + for _, tc := range []struct { + name string + apply func(g *DefaultObjectsGate, src *corev1.Secret) + }{ + {"COSI Secret is the source", func(g *DefaultObjectsGate, src *corev1.Secret) { + g.CredentialsCOSISecretName = g.Config.SourceSecretName + src.Data["BucketInfo"] = cosiSecret("https://s3.new.example.com").Data["BucketInfo"] + }}, + {"source without endpoint", func(_ *DefaultObjectsGate, src *corev1.Secret) { + delete(src.Data, "endpoint") + }}, + } { + t.Run(tc.name, func(t *testing.T) { + src := sourceSecret("bucket-1a2b") + var objs []runtime.Object + for _, o := range allDefaultObjects() { + if u := o.(*unstructured.Unstructured); u.GetName() != "cozy-default-cnpg" { + objs = append(objs, u) + } + } + objs = append(objs, helmReleaseObject(), credentialsHelmReleaseObject()) + g, dyn := newGate(t, nil, objs...) + tc.apply(g, src) + g.Client = fake.NewClientBuilder().WithScheme(gateScheme()). + WithObjects(src, cosiSecret("https://s3.new.example.com"), cozyDefaultBackupClass()).Build() + + missing, forced, err := g.Check(context.Background()) + if err != nil { + t.Fatalf("Check: %v", err) + } + if !forced || len(missing) != 1 || missing[0] != "CNPG/cozy-default-cnpg" { + t.Fatalf("missing = %v, forced = %v; want the missing strategy repaired", missing, forced) + } + if ann := credentialsForceAnnotations(t, dyn); len(ann) != 0 { + t.Errorf("bucket release forced: %v", ann) + } + }) + } +} diff --git a/packages/system/backupstrategy-controller/templates/deployment.yaml b/packages/system/backupstrategy-controller/templates/deployment.yaml index adc09fa2a..1278bec8a 100644 --- a/packages/system/backupstrategy-controller/templates/deployment.yaml +++ b/packages/system/backupstrategy-controller/templates/deployment.yaml @@ -92,9 +92,11 @@ spec: # "-credentials" to its name). The COSI sidecar writes it once per # BucketAccess, so the gate compares the two endpoints and forces the # release when the credentials Secret lags behind. + {{- if hasSuffix "-credentials" .Values.backupStorage.systemSecretName }} - name: BACKUP_CREDENTIALS_COSI_SECRET_NAME value: {{ trimSuffix "-credentials" .Values.backupStorage.systemSecretName | quote }} {{- end }} + {{- end }} {{- if .Values.velero.bslEnabled }} - name: BACKUP_DEFAULT_OBJECTS_VELERO_NAMESPACE value: {{ .Values.velero.namespace | quote }} diff --git a/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml b/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml index ed616ce79..446bbdcfa 100644 --- a/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml +++ b/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml @@ -128,6 +128,19 @@ tests: name: BACKUP_CREDENTIALS_COSI_SECRET_NAME value: bucket-cozy-backups-system + # A systemSecretName that is not -credentials is not rendered by + # the bucket chart from a COSI Secret, so there is nothing to compare it to. + - it: "systemSecretName without the -credentials suffix: no COSI Secret to compare against" + template: templates/deployment.yaml + set: + backupStorage.systemSecretName: bucket-cozy-backups-system + asserts: + - notContains: + path: spec.template.spec.containers[0].env + any: true + content: + name: BACKUP_CREDENTIALS_COSI_SECRET_NAME + - it: "velero.bslEnabled=false: the BSL is not rendered, so the gate must not look for it" template: templates/deployment.yaml set: diff --git a/packages/system/backupstrategy-controller/values.yaml b/packages/system/backupstrategy-controller/values.yaml index d87c978e6..67f986e47 100644 --- a/packages/system/backupstrategy-controller/values.yaml +++ b/packages/system/backupstrategy-controller/values.yaml @@ -170,9 +170,9 @@ backupStorage: # release. The COSI Secret itself keeps the endpoint it was granted with, so # after a publishing.host change that endpoint follows only once the # BucketAccess is re-granted. Both are throttled, independently, and a no-op - # in the steady state. A suspended release is skipped rather than re-stamped forever. The - # gate does NOT create the objects itself: their bodies are values-driven - # and Helm stays their single owner. + # in the steady state. A suspended release is skipped rather than + # re-stamped forever. The gate does NOT create the objects itself: their + # bodies are values-driven and Helm stays their single owner. # # Turn this off only if you manage the default Strategy CRs out of band and # accept that a missed install-time render is never repaired. Watch From ef7626a951a84e084b32efe539c4b9059e1229e9 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 19:31:46 +0300 Subject: [PATCH 178/518] fix(backups): keep repairing missing default objects while credentials lag The credentials comparison returned early when the credentials Secret was stale or the COSI Secret could not be read, so a missing default Strategy or BSL was no longer created on that tick, nor on any later one while the bucket release stayed suspended, kept failing, or the COSI Secret stayed unreadable. Repairing missing objects is what the gate did before the comparison existed, and it does not depend on the answer. Run the object check on every tick and force each release for what it renders, reporting a comparison error alongside instead of in place of the repair. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backupcontroller/default_objects_gate.go | 97 +++++++------ .../default_objects_gate_test.go | 131 ++++++++++++++++-- 2 files changed, 177 insertions(+), 51 deletions(-) diff --git a/internal/backupcontroller/default_objects_gate.go b/internal/backupcontroller/default_objects_gate.go index d0ed29954..8fb58c65e 100644 --- a/internal/backupcontroller/default_objects_gate.go +++ b/internal/backupcontroller/default_objects_gate.go @@ -194,10 +194,11 @@ type DefaultObjectsGate struct { // MinForceInterval for no reason. lastForce time.Time lastCredentialsForce time.Time - // renderForced records that a forced render of HelmRelease went through. - // That render stamps endpointAnnotation on every object the chart - // renders, so from then on an object without it (one rendered by a chart - // that predates the annotation) is not counted as stale again. + // renderForced records that a forced render of HelmRelease was requested + // (the annotations were patched; whether the upgrade succeeds is not + // observed). That render stamps endpointAnnotation on every object the + // chart renders, so from then on an object without it (one rendered by a + // chart that predates the annotation) is not counted as stale again. renderForced bool } @@ -302,33 +303,41 @@ func (g *DefaultObjectsGate) checkAndLog(ctx context.Context, logger logr.Logger // Info, not Error: every branch here is either transient (the // bucket is still being provisioned) or already exposed as a // metric, and this runs every minute for the life of the cluster. - logger.Info("default backup objects check failed", "error", err.Error()) + logger.Info("default backup objects check failed", "error", err.Error(), + "missing", missing, "forced", forced) case len(missing) > 0 && forced: - // The release named here is whichever one renders what is missing: + // The releases named here are the ones that render what is missing: // the credentials Secret comes from the bucket's -system // release, everything else from this chart's own. logger.Info("forced a Helm upgrade to create missing or stale default backup objects", - "helmRelease", g.forcedReleaseFor(missing).String(), "missing", missing) + "helmReleases", g.forcedReleasesFor(missing), "missing", missing) case len(missing) > 0: logger.Info("default backup objects still missing or stale, no force issued on this tick", - "helmRelease", g.forcedReleaseFor(missing).String(), "missing", missing, + "helmReleases", g.forcedReleasesFor(missing), "missing", missing, "minForceInterval", g.MinForceInterval.String()) default: logger.V(1).Info("all default backup objects present") } } -// forcedReleaseFor reports which release renders the given missing set, for -// logging only. reconcileCredentials and staleCredentials return the -// credentials Secret as the sole entry, and it is never mixed with the routed -// objects — -// resolving the bucket name is a precondition for checking those at all. -func (g *DefaultObjectsGate) forcedReleaseFor(missing []string) types.NamespacedName { +// forcedReleasesFor reports which releases render the given entries, for +// logging only: the credentials Secret comes from CredentialsHelmRelease, +// everything else from HelmRelease. +func (g *DefaultObjectsGate) forcedReleasesFor(missing []string) []string { credsKey := fmt.Sprintf("Secret/%s", g.Config.SourceSecretName) - if len(missing) == 1 && (missing[0] == credsKey || strings.HasPrefix(missing[0], credsKey+" ")) && g.CredentialsHelmRelease.Name != "" { - return g.CredentialsHelmRelease + var out []string + seen := map[string]bool{} + for _, m := range missing { + release := g.HelmRelease.String() + if (m == credsKey || strings.HasPrefix(m, credsKey+" ")) && g.CredentialsHelmRelease.Name != "" { + release = g.CredentialsHelmRelease.String() + } + if !seen[release] { + seen[release] = true + out = append(out, release) + } } - return g.HelmRelease + return out } // Check resolves the platform bucket name, verifies that every object the @@ -369,21 +378,17 @@ func (g *DefaultObjectsGate) Check(ctx context.Context) ([]string, bool, error) return g.reconcileCredentials(ctx) } - // The objects below are rendered from this Secret, so while it is stale - // they would only be re-rendered with the same stale endpoint. - stale, err := g.staleCredentials(ctx, src) - if err != nil { + staleCreds, credsErr := g.staleCredentials(ctx, src) + if credsErr != nil { defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() - return nil, false, err - } - if stale != "" { - return g.force(ctx, g.CredentialsHelmRelease, &g.lastCredentialsForce, []string{stale}) } backupClass := &backupsv1alpha1.BackupClass{} if err := g.Client.Get(ctx, client.ObjectKey{Name: g.BackupClassName}, backupClass); err != nil { - defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() - return nil, false, fmt.Errorf("get BackupClass %s: %w", g.BackupClassName, err) + if credsErr == nil { + defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() + } + return nil, false, errors.Join(credsErr, fmt.Errorf("get BackupClass %s: %w", g.BackupClassName, err)) } // With provisionBucket=false (no bucket release) the chart renders @@ -395,21 +400,33 @@ func (g *DefaultObjectsGate) Check(ctx context.Context) ([]string, bool, error) } state, err := g.inspectObjects(ctx, backupClass, want) if err != nil { - defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() - return nil, false, err + if credsErr == nil { + defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() + } + return nil, false, errors.Join(credsErr, err) } defaultObjectsMissing.WithLabelValues(g.BackupClassName).Set(float64(len(state.missing))) - needed := append(state.missing, state.stale...) - if len(needed) == 0 { - return nil, false, nil - } - sort.Strings(needed) - needed, forced, err := g.force(ctx, g.HelmRelease, &g.lastForce, needed) - if forced { - g.renderForced = true + var needed []string + var forced bool + errs := []error{credsErr} + if staleCreds != "" { + _, credsForced, err := g.force(ctx, g.CredentialsHelmRelease, &g.lastCredentialsForce, []string{staleCreds}) + needed = append(needed, staleCreds) + forced = credsForced + errs = append(errs, err) } - return needed, forced, err + if objects := append(state.missing, state.stale...); len(objects) > 0 { + sort.Strings(objects) + _, objectsForced, err := g.force(ctx, g.HelmRelease, &g.lastForce, objects) + needed = append(needed, objects...) + if objectsForced { + forced = true + g.renderForced = true + } + errs = append(errs, err) + } + return needed, forced, errors.Join(errs...) } // staleCredentials reports the credentials Secret, formatted for logs, when @@ -559,8 +576,8 @@ func (g *DefaultObjectsGate) staleEndpoint(state *defaultObjectsState, key strin // inspectObjects returns a sorted, de-duplicated list of the // "/" objects the BackupClass references that do not exist, -// plus the Velero BSL when a Velero namespace is configured, and the existing -// ones whose endpointAnnotation differs from want. +// plus the Velero BSL when a Velero namespace is configured, and, unsorted, +// the existing ones whose endpointAnnotation differs from want. // // An unmappable group/kind (the CRD is not installed) is NOT reported as // missing: no Helm re-render can create an object whose CRD is absent, and diff --git a/internal/backupcontroller/default_objects_gate_test.go b/internal/backupcontroller/default_objects_gate_test.go index cff6187cd..fac76e132 100644 --- a/internal/backupcontroller/default_objects_gate_test.go +++ b/internal/backupcontroller/default_objects_gate_test.go @@ -3,6 +3,7 @@ package backupcontroller import ( "context" "errors" + "slices" "strings" "testing" "time" @@ -1109,24 +1110,32 @@ func TestRenderedEndpointMatchesTheChart(t *testing.T) { } } -// TestForcedReleaseForNamesTheStaleCredentialsRelease pins the release the -// log line names: a stale credentials Secret is repaired by the bucket -// release, like a missing one, and everything else by this chart's own. -func TestForcedReleaseForNamesTheStaleCredentialsRelease(t *testing.T) { +// TestForcedReleasesForNamesEachRenderingRelease pins the releases the log +// line names: the credentials Secret, missing or stale, is repaired by the +// bucket release, everything else by this chart's own, and both are named when +// one check forced both. +func TestForcedReleasesForNamesEachRenderingRelease(t *testing.T) { g, _ := newGate(t, nil) + creds, own := g.CredentialsHelmRelease.String(), g.HelmRelease.String() for _, tc := range []struct { missing []string - want types.NamespacedName + want []string }{ - {[]string{"Secret/bucket-cozy-backups-system-credentials"}, g.CredentialsHelmRelease}, - {[]string{`Secret/bucket-cozy-backups-system-credentials (endpoint "s3.old", COSI Secret has "s3.new")`}, g.CredentialsHelmRelease}, - {[]string{"Secret/bucket-cozy-backups-system-credentials-other"}, g.HelmRelease}, - {[]string{`CNPG/cozy-default-cnpg (endpoint "https://s3.old", want "https://s3.new")`}, g.HelmRelease}, + {[]string{"Secret/bucket-cozy-backups-system-credentials"}, []string{creds}}, + {[]string{`Secret/bucket-cozy-backups-system-credentials (endpoint "s3.old", COSI Secret has "s3.new")`}, []string{creds}}, + {[]string{"Secret/bucket-cozy-backups-system-credentials-other"}, []string{own}}, + {[]string{`CNPG/cozy-default-cnpg (endpoint "https://s3.old", want "https://s3.new")`, "Etcd/cozy-default-etcd"}, []string{own}}, + {[]string{`Secret/bucket-cozy-backups-system-credentials (endpoint "s3.old", COSI Secret has "s3.new")`, "CNPG/cozy-default-cnpg"}, []string{creds, own}}, } { - if got := g.forcedReleaseFor(tc.missing); got != tc.want { - t.Errorf("forcedReleaseFor(%v) = %v, want %v", tc.missing, got, tc.want) + if got := g.forcedReleasesFor(tc.missing); !slices.Equal(got, tc.want) { + t.Errorf("forcedReleasesFor(%v) = %v, want %v", tc.missing, got, tc.want) } } + // External S3: no bucket release, so nothing but this chart's own. + g.CredentialsHelmRelease = types.NamespacedName{} + if got := g.forcedReleasesFor([]string{"Secret/bucket-cozy-backups-system-credentials"}); !slices.Equal(got, []string{own}) { + t.Errorf("forcedReleasesFor without a bucket release = %v, want [%s]", got, own) + } } // TestCheckSkipsCredentialsComparisonWithoutDerivedSecret covers a @@ -1174,3 +1183,103 @@ func TestCheckSkipsCredentialsComparisonWithoutDerivedSecret(t *testing.T) { }) } } + +// TestCheckRepairsMissingObjectsWhileCredentialsUnresolved pins that a stale +// or unreadable credentials comparison does not cost the repair of missing +// objects the gate made before it existed. The comparison only withholds the +// endpoint check, whose answer would be rendered from the same stale Secret. +func TestCheckRepairsMissingObjectsWhileCredentialsUnresolved(t *testing.T) { + malformed := cosiSecret("") + malformed.Data["BucketInfo"] = []byte("{not json") + for _, tc := range []struct { + name string + cosi *corev1.Secret + credsHR *unstructured.Unstructured + wantErr bool + wantCredsHRAt bool + }{ + {"stale credentials", cosiSecret("https://s3.new.example.com"), credentialsHelmReleaseObject(), false, true}, + {"stale credentials, bucket release suspended", cosiSecret("https://s3.new.example.com"), suspended(credentialsHelmReleaseObject()), false, false}, + {"malformed COSI Secret", malformed, credentialsHelmReleaseObject(), true, false}, + } { + t.Run(tc.name, func(t *testing.T) { + defaultObjectsCheckErrors.Reset() + defaultObjectsMissing.Reset() + var objs []runtime.Object + for _, o := range allDefaultObjects() { + if u := o.(*unstructured.Unstructured); u.GetName() != "cozy-default-cnpg" { + objs = append(objs, u) + } + } + objs = append(objs, helmReleaseObject(), tc.credsHR) + g, dyn := newGate(t, []client.Object{sourceSecret("bucket-1a2b"), tc.cosi, cozyDefaultBackupClass()}, objs...) + + missing, forced, err := g.Check(context.Background()) + if (err != nil) != tc.wantErr { + t.Fatalf("err = %v, want error %v", err, tc.wantErr) + } + if !forced { + t.Error("forced = false, want this chart's release forced for the missing strategy") + } + if ann := forceAnnotations(t, dyn); ann["reconcile.fluxcd.io/forceAt"] == "" { + t.Errorf("own HelmRelease not forced for a missing strategy: %v", ann) + } + if got := credentialsForceAnnotations(t, dyn)["reconcile.fluxcd.io/forceAt"] != ""; got != tc.wantCredsHRAt { + t.Errorf("bucket release forced = %v, want %v", got, tc.wantCredsHRAt) + } + found := false + for _, m := range missing { + found = found || m == "CNPG/cozy-default-cnpg" + } + if !found { + t.Errorf("missing = %v, want CNPG/cozy-default-cnpg reported", missing) + } + if got := testutil.ToFloat64(defaultObjectsMissing.WithLabelValues("cozy-default")); got != 1 { + t.Errorf("missing gauge = %v, want 1", got) + } + wantErrs := 0.0 + if tc.wantErr { + wantErrs = 1 + } + if got := testutil.ToFloat64(defaultObjectsCheckErrors.WithLabelValues("cozy-default")); got != wantErrs { + t.Errorf("check errors = %v, want %v", got, wantErrs) + } + }) + } +} + +// TestCheckCountsOneErrorPerCheck pins that a check failing in two places is +// still one check that did not reach a conclusion. +func TestCheckCountsOneErrorPerCheck(t *testing.T) { + defaultObjectsCheckErrors.Reset() + bad := cosiSecret("") + bad.Data["BucketInfo"] = []byte("{not json") + g, _ := newGate(t, []client.Object{sourceSecret("bucket-1a2b"), bad}, helmReleaseObject(), credentialsHelmReleaseObject()) + + if _, _, err := g.Check(context.Background()); err == nil { + t.Fatal("Check succeeded with a malformed COSI Secret and no BackupClass") + } + if got := testutil.ToFloat64(defaultObjectsCheckErrors.WithLabelValues("cozy-default")); got != 1 { + t.Fatalf("check errors = %v, want 1", got) + } +} + +// TestCheckSurfacesCredentialsPatchFailure pins that a failed force of the +// bucket release is reported, not recorded as done. +func TestCheckSurfacesCredentialsPatchFailure(t *testing.T) { + objs := append([]runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()}, allDefaultObjects()...) + g, dyn := newGate(t, + []client.Object{sourceSecret("bucket-1a2b"), cosiSecret("https://s3.new.example.com"), cozyDefaultBackupClass()}, + objs..., + ) + dyn.PrependReactor("patch", "helmreleases", func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewForbidden(schema.GroupResource{Group: "helm.toolkit.fluxcd.io", Resource: "helmreleases"}, "bucket-cozy-backups-system", nil) + }) + + if _, forced, err := g.Check(context.Background()); err == nil || forced { + t.Fatalf("forced = %v, err = %v; want an error and forced=false", forced, err) + } + if !g.lastCredentialsForce.IsZero() { + t.Error("lastCredentialsForce advanced despite the patch failing") + } +} From 399ccd3f4aaf2c1f7d64fccc76173021104f0717 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:02:16 +0300 Subject: [PATCH 179/518] fix(backups): refresh stale credentials even when the routed objects are unreadable The credentials endpoint comparison needs only the credentials Secret and the COSI Secret it is rendered from, yet its forced render was issued only after the BackupClass and every routed object had been read. A missing BackupClass or one unreadable Strategy therefore kept the old endpoint in the credentials Secret, which the projector hands to every driver, not just the one whose object failed. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backupcontroller/default_objects_gate.go | 25 ++++++---- .../default_objects_gate_test.go | 50 +++++++++++++++++++ 2 files changed, 64 insertions(+), 11 deletions(-) diff --git a/internal/backupcontroller/default_objects_gate.go b/internal/backupcontroller/default_objects_gate.go index 8fb58c65e..b99f67b41 100644 --- a/internal/backupcontroller/default_objects_gate.go +++ b/internal/backupcontroller/default_objects_gate.go @@ -378,17 +378,29 @@ func (g *DefaultObjectsGate) Check(ctx context.Context) ([]string, bool, error) return g.reconcileCredentials(ctx) } + // The credentials comparison needs only the two Secrets, so its force goes + // out before the object reads: an unreadable BackupClass or Strategy must + // not keep the stale endpoint in the Secret every driver reads. + var needed []string + var forced bool staleCreds, credsErr := g.staleCredentials(ctx, src) + errs := []error{credsErr} if credsErr != nil { defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() } + if staleCreds != "" { + _, credsForced, err := g.force(ctx, g.CredentialsHelmRelease, &g.lastCredentialsForce, []string{staleCreds}) + needed = append(needed, staleCreds) + forced = credsForced + errs = append(errs, err) + } backupClass := &backupsv1alpha1.BackupClass{} if err := g.Client.Get(ctx, client.ObjectKey{Name: g.BackupClassName}, backupClass); err != nil { if credsErr == nil { defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() } - return nil, false, errors.Join(credsErr, fmt.Errorf("get BackupClass %s: %w", g.BackupClassName, err)) + return needed, forced, errors.Join(append(errs, fmt.Errorf("get BackupClass %s: %w", g.BackupClassName, err))...) } // With provisionBucket=false (no bucket release) the chart renders @@ -403,19 +415,10 @@ func (g *DefaultObjectsGate) Check(ctx context.Context) ([]string, bool, error) if credsErr == nil { defaultObjectsCheckErrors.WithLabelValues(g.BackupClassName).Inc() } - return nil, false, errors.Join(credsErr, err) + return needed, forced, errors.Join(append(errs, err)...) } defaultObjectsMissing.WithLabelValues(g.BackupClassName).Set(float64(len(state.missing))) - var needed []string - var forced bool - errs := []error{credsErr} - if staleCreds != "" { - _, credsForced, err := g.force(ctx, g.CredentialsHelmRelease, &g.lastCredentialsForce, []string{staleCreds}) - needed = append(needed, staleCreds) - forced = credsForced - errs = append(errs, err) - } if objects := append(state.missing, state.stale...); len(objects) > 0 { sort.Strings(objects) _, objectsForced, err := g.force(ctx, g.HelmRelease, &g.lastForce, objects) diff --git a/internal/backupcontroller/default_objects_gate_test.go b/internal/backupcontroller/default_objects_gate_test.go index fac76e132..befaedabe 100644 --- a/internal/backupcontroller/default_objects_gate_test.go +++ b/internal/backupcontroller/default_objects_gate_test.go @@ -1283,3 +1283,53 @@ func TestCheckSurfacesCredentialsPatchFailure(t *testing.T) { t.Error("lastCredentialsForce advanced despite the patch failing") } } + +// TestCheckForcesStaleCredentialsWhenObjectsUnreadable pins that the +// credentials comparison needs only the two Secrets: a BackupClass or a routed +// object the gate cannot read must not keep the stale endpoint in the +// credentials Secret, which every other driver reads through the projector. +func TestCheckForcesStaleCredentialsWhenObjectsUnreadable(t *testing.T) { + for _, tc := range []struct { + name string + backupClass bool + forbidCNPG bool + }{ + {"BackupClass absent", false, false}, + {"routed Strategy unreadable", true, true}, + } { + t.Run(tc.name, func(t *testing.T) { + defaultObjectsCheckErrors.Reset() + ctrlObjs := []client.Object{sourceSecret("bucket-1a2b"), cosiSecret("https://s3.new.example.com")} + if tc.backupClass { + ctrlObjs = append(ctrlObjs, cozyDefaultBackupClass()) + } + objs := append([]runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()}, allDefaultObjects()...) + g, dyn := newGate(t, ctrlObjs, objs...) + if tc.forbidCNPG { + dyn.PrependReactor("get", "cnpgs", func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewForbidden(schema.GroupResource{Group: strategyAPIGroup, Resource: "cnpgs"}, "cozy-default-cnpg", nil) + }) + } + + missing, forced, err := g.Check(context.Background()) + if err == nil { + t.Fatal("Check succeeded although the routed objects could not be evaluated") + } + if !forced { + t.Error("forced = false, want the bucket release forced for its stale credentials Secret") + } + if ann := credentialsForceAnnotations(t, dyn); ann["reconcile.fluxcd.io/forceAt"] == "" { + t.Errorf("bucket release not forced: %v", ann) + } + if ann := forceAnnotations(t, dyn); len(ann) != 0 { + t.Errorf("own HelmRelease forced without a conclusive object check: %v", ann) + } + if len(missing) != 1 || !strings.HasPrefix(missing[0], "Secret/bucket-cozy-backups-system-credentials") { + t.Errorf("missing = %v, want the credentials Secret", missing) + } + if got := testutil.ToFloat64(defaultObjectsCheckErrors.WithLabelValues("cozy-default")); got != 1 { + t.Errorf("check errors = %v, want 1", got) + } + }) + } +} From ccac0e17e9cccdde4492c8331316e79f17863565 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 14:29:44 +0300 Subject: [PATCH 180/518] fix(mariadb): fail backup and restore jobs whose operator Job failed mariadb-operator reports a failed backing Job as Complete=True with reason JobFailed, and Complete=True alone was read as success. A backup whose dump or upload failed therefore ended Succeeded and left a Ready Backup pointing at an archive that does not exist, and a scheduled Plan kept producing such Backups; a failed restore ended Succeeded too. Success now requires reason JobComplete, and JobFailed fails the job whatever the status. Signed-off-by: Alexey Artamonov Assisted-by: LLM --- .../mariadbstrategy_controller.go | 43 ++--- .../mariadbstrategy_controller_test.go | 181 ++++++++++++++++++ .../backupcontroller/mariadbtypes/types.go | 12 +- 3 files changed, 207 insertions(+), 29 deletions(-) diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index a5bd77094..48abd3644 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -204,7 +204,10 @@ func (r *BackupJobReconciler) reconcileMariaDB(ctx context.Context, j *backupsv1 cond := apimeta.FindStatusCondition(mdbBackup.Status.Conditions, mariadbtypes.ConditionTypeComplete) switch { - case cond != nil && cond.Status == metav1.ConditionTrue: + case cond != nil && cond.Reason == mariadbtypes.ConditionReasonJobFailed: + return r.markBackupJobFailed(ctx, j, mariadbJobFailedMessage("Backup", mdbBackup.Namespace, mdbBackup.Name, cond)) + + case cond != nil && cond.Status == metav1.ConditionTrue && cond.Reason == mariadbtypes.ConditionReasonJobComplete: // Backup completed successfully. Materialise the Cozystack artifact // and finalise the BackupJob. Idempotent: if the artifact already // exists (a previous reconcile created it and then raced on the @@ -231,22 +234,12 @@ func (r *BackupJobReconciler) reconcileMariaDB(ctx context.Context, j *backupsv1 } return ctrl.Result{}, nil - case cond != nil && cond.Status == metav1.ConditionFalse && cond.Reason == mariadbtypes.ConditionReasonJobFailed: - // Terminal failure. The operator marks Complete=False/JobFailed only - // after the underlying Job exhausts its backoff budget, so we can - // fail the BackupJob immediately rather than waiting for the - // driver-side deadline. - message := cond.Message - if message == "" { - message = "k8s.mariadb.com Backup terminal failure" - } - return r.markBackupJobFailed(ctx, j, message) - default: // Still running (Complete=False/JobRunning, Complete=False/JobNotComplete, - // JobSuspended, or no Complete condition yet). Apply a wall-clock - // deadline so a permanently-stuck Backup eventually fails the - // BackupJob instead of pinning it Running forever. + // JobSuspended, a reason this driver does not know, or no Complete + // condition yet). Apply a wall-clock deadline so a permanently-stuck + // Backup eventually fails the BackupJob instead of pinning it Running + // forever. if mariadbBackupDeadlineExceeded(j.Status.StartedAt) { detail := "no Complete condition observed" if cond != nil { @@ -259,6 +252,14 @@ func (r *BackupJobReconciler) reconcileMariaDB(ctx context.Context, j *backupsv1 } } +// mariadbJobFailedMessage names the operator object whose Job failed. The +// operator's own condition message is only "Failed"; the cause is in the +// logs of that object's Job. +func mariadbJobFailedMessage(kind, namespace, name string, cond *metav1.Condition) string { + return fmt.Sprintf("k8s.mariadb.com/%s %s/%s failed (Complete=%s/%s); the cause is in the logs of its Job", + kind, namespace, name, cond.Status, cond.Reason) +} + // mariadbBackupDeadlineExceeded reports whether enough wall-clock time has // elapsed since the BackupJob started that we should give up on a stuck // operator Backup. Returns false when StartedAt is nil so the very first @@ -570,7 +571,10 @@ func (r *RestoreJobReconciler) reconcileMariaDBRestore(ctx context.Context, rest cond := apimeta.FindStatusCondition(mdbRestore.Status.Conditions, mariadbtypes.ConditionTypeComplete) switch { - case cond != nil && cond.Status == metav1.ConditionTrue: + case cond != nil && cond.Reason == mariadbtypes.ConditionReasonJobFailed: + return r.markRestoreJobFailed(ctx, restoreJob, mariadbJobFailedMessage("Restore", mdbRestore.Namespace, mdbRestore.Name, cond)) + + case cond != nil && cond.Status == metav1.ConditionTrue && cond.Reason == mariadbtypes.ConditionReasonJobComplete: now := metav1.Now() restoreJob.Status.CompletedAt = &now restoreJob.Status.Phase = backupsv1alpha1.RestoreJobPhaseSucceeded @@ -585,13 +589,6 @@ func (r *RestoreJobReconciler) reconcileMariaDBRestore(ctx context.Context, rest } return ctrl.Result{}, nil - case cond != nil && cond.Status == metav1.ConditionFalse && cond.Reason == mariadbtypes.ConditionReasonJobFailed: - message := cond.Message - if message == "" { - message = fmt.Sprintf("k8s.mariadb.com/Restore %s/%s terminal failure", mdbRestore.Namespace, mdbRestore.Name) - } - return r.markRestoreJobFailed(ctx, restoreJob, message) - default: deadline := options.effectiveRestoreDeadline() if restoreJob.Status.StartedAt != nil && time.Since(restoreJob.Status.StartedAt.Time) > deadline { diff --git a/internal/backupcontroller/mariadbstrategy_controller_test.go b/internal/backupcontroller/mariadbstrategy_controller_test.go index 0e4f63b5e..307301223 100644 --- a/internal/backupcontroller/mariadbstrategy_controller_test.go +++ b/internal/backupcontroller/mariadbstrategy_controller_test.go @@ -867,6 +867,187 @@ func TestReconcileMariaDBRestore_KindCheckedBeforeSrcBackupGet(t *testing.T) { } } +// The operator reports a failed backing Job as Complete=True with reason +// JobFailed (SetCompleteWithJob in mariadb-operator 25.10.2), so the +// Complete status alone does not tell success from failure. +var mariadbCompleteOutcomeCases = []struct { + name string + status metav1.ConditionStatus + reason string + wantFailed bool + wantDone bool +}{ + {name: "Complete=True/JobComplete succeeds", status: metav1.ConditionTrue, reason: mariadbtypes.ConditionReasonJobComplete, wantDone: true}, + {name: "Complete=True/JobFailed fails", status: metav1.ConditionTrue, reason: mariadbtypes.ConditionReasonJobFailed, wantFailed: true}, + {name: "Complete=False/JobFailed fails", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobFailed, wantFailed: true}, + {name: "Complete=True with an unknown reason keeps waiting", status: metav1.ConditionTrue, reason: "SomethingElse"}, + {name: "Complete=False/JobComplete keeps waiting", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobComplete}, + {name: "Complete=False/JobRunning keeps waiting", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobRunning}, +} + +func mariadbCompleteCondition(status metav1.ConditionStatus, reason string) []metav1.Condition { + return []metav1.Condition{{ + Type: mariadbtypes.ConditionTypeComplete, + Status: status, + Reason: reason, + Message: "Failed", + LastTransitionTime: metav1.Now(), + }} +} + +func TestReconcileMariaDBBackup_CompleteConditionOutcome(t *testing.T) { + strategyAPIGroup := strategyv1alpha1.GroupVersion.Group + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: &strategyAPIGroup, + Kind: strategyv1alpha1.MariaDBStrategyKind, + Name: "mdb-strategy", + }, + } + + for _, tc := range mariadbCompleteOutcomeCases { + t.Run(tc.name, func(t *testing.T) { + now := metav1.Now() + seed := newMariaDBBackupJob("bj", "tenant") + seed.Status.StartedAt = &now + strategy := &strategyv1alpha1.MariaDB{ + ObjectMeta: metav1.ObjectMeta{Name: "mdb-strategy"}, + Spec: strategyv1alpha1.MariaDBSpec{Template: *newRenderedMariaDBTemplate()}, + } + operatorMariaDB := &mariadbtypes.MariaDB{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: mariadbNameForApp(seed.Spec.ApplicationRef.Name)}, + } + operatorBackup := &mariadbtypes.Backup{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", + Name: "bj-op", + Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: seed.Name, + backupsv1alpha1.OwningJobNamespaceLabel: seed.Namespace, + }, + }, + Status: mariadbtypes.BackupStatus{Conditions: mariadbCompleteCondition(tc.status, tc.reason)}, + } + c := newMariaDBStrategyTestClient(t, seed, newMariaDBApp(seed.Spec.ApplicationRef.Name, "tenant"), + strategy, operatorMariaDB, operatorBackup) + r := &BackupJobReconciler{Client: c, Scheme: c.Scheme(), Recorder: record.NewFakeRecorder(10)} + + bj := &backupsv1alpha1.BackupJob{} + if err := c.Get(context.Background(), client.ObjectKeyFromObject(seed), bj); err != nil { + t.Fatalf("seed Get: %v", err) + } + res, err := r.reconcileMariaDB(context.Background(), bj, resolved) + if err != nil { + t.Fatalf("reconcileMariaDB: %v", err) + } + + artifacts := &backupsv1alpha1.BackupList{} + if err := c.List(context.Background(), artifacts, client.InNamespace("tenant")); err != nil { + t.Fatalf("list Backups: %v", err) + } + switch { + case tc.wantFailed: + if bj.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed { + t.Errorf("expected Phase=Failed, got %q", bj.Status.Phase) + } + if !strings.Contains(bj.Status.Message, "bj-op") { + t.Errorf("failure message must name the operator Backup, got %q", bj.Status.Message) + } + if bj.Status.BackupRef != nil || len(artifacts.Items) != 0 { + t.Errorf("a failed run must not produce a Backup: ref=%v artifacts=%d", bj.Status.BackupRef, len(artifacts.Items)) + } + case tc.wantDone: + if bj.Status.Phase != backupsv1alpha1.BackupJobPhaseSucceeded { + t.Errorf("expected Phase=Succeeded, got %q (message %q)", bj.Status.Phase, bj.Status.Message) + } + if bj.Status.BackupRef == nil || len(artifacts.Items) != 1 { + t.Errorf("a successful run must produce exactly one Backup: ref=%v artifacts=%d", bj.Status.BackupRef, len(artifacts.Items)) + } + default: + if bj.Status.Phase != backupsv1alpha1.BackupJobPhaseRunning { + t.Errorf("expected Phase=Running, got %q", bj.Status.Phase) + } + if res.RequeueAfter == 0 { + t.Errorf("expected a requeue while the run is in progress, got %+v", res) + } + if len(artifacts.Items) != 0 { + t.Errorf("an unfinished run must not produce a Backup, got %d", len(artifacts.Items)) + } + } + }) + } +} + +func TestReconcileMariaDBRestore_CompleteConditionOutcome(t *testing.T) { + apps := mariadbapp.GroupName + + for _, tc := range mariadbCompleteOutcomeCases { + t.Run(tc.name, func(t *testing.T) { + now := metav1.Now() + cozyBackup := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "src-cozy-bk"}, + Spec: backupsv1alpha1.BackupSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "src", APIGroup: &apps}, + DriverMetadata: map[string]string{mariadbBackupNameKey: "src-op-bk"}, + }, + } + seed := &backupsv1alpha1.RestoreJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "rj"}, + Spec: backupsv1alpha1.RestoreJobSpec{ + BackupRef: corev1.LocalObjectReference{Name: cozyBackup.Name}, + TargetApplicationRef: &corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "target", APIGroup: &apps}, + }, + Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning}, + } + operatorRestore := &mariadbtypes.Restore{ + ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", + Name: "rj-op", + Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: seed.Name, + backupsv1alpha1.OwningJobNamespaceLabel: seed.Namespace, + }, + }, + Status: mariadbtypes.RestoreStatus{Conditions: mariadbCompleteCondition(tc.status, tc.reason)}, + } + c := newMariaDBStrategyTestClient(t, cozyBackup, seed, operatorRestore, + &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "src-op-bk"}}, + &mariadbtypes.MariaDB{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: mariadbNameForApp("target")}}) + r := &RestoreJobReconciler{Client: c, Recorder: record.NewFakeRecorder(10)} + + rj := &backupsv1alpha1.RestoreJob{} + if err := c.Get(context.Background(), client.ObjectKeyFromObject(seed), rj); err != nil { + t.Fatalf("seed Get: %v", err) + } + res, err := r.reconcileMariaDBRestore(context.Background(), rj, cozyBackup) + if err != nil { + t.Fatalf("reconcileMariaDBRestore: %v", err) + } + + switch { + case tc.wantFailed: + if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Errorf("expected Phase=Failed, got %q", rj.Status.Phase) + } + if !strings.Contains(rj.Status.Message, "rj-op") { + t.Errorf("failure message must name the operator Restore, got %q", rj.Status.Message) + } + case tc.wantDone: + if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded { + t.Errorf("expected Phase=Succeeded, got %q (message %q)", rj.Status.Phase, rj.Status.Message) + } + default: + if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseRunning { + t.Errorf("expected Phase=Running, got %q", rj.Status.Phase) + } + if res.RequeueAfter == 0 { + t.Errorf("expected a requeue while the run is in progress, got %+v", res) + } + } + }) + } +} + // --------------------------------------------------------------------------- // Test helpers // --------------------------------------------------------------------------- diff --git a/internal/backupcontroller/mariadbtypes/types.go b/internal/backupcontroller/mariadbtypes/types.go index 7bc5b03a1..d9c09f30a 100644 --- a/internal/backupcontroller/mariadbtypes/types.go +++ b/internal/backupcontroller/mariadbtypes/types.go @@ -27,14 +27,14 @@ const ( GroupName = "k8s.mariadb.com" Version = "v1alpha1" - // Operator-side condition type set on Backup and Restore once the run - // terminates. Status=True means success; Status=False with Reason - // explains the failure or in-progress state. + // Operator-side condition type set on Backup and Restore. Status=True + // means the backing Job terminated, successfully or not; only the + // Reason tells the two apart (SetCompleteWithJob in mariadb-operator). ConditionTypeComplete = "Complete" // Reason values the upstream operator emits on the Complete condition. // JobComplete -> Status=True (run succeeded). - // JobFailed -> Status=False (terminal failure of the backing Job). + // JobFailed -> Status=True (terminal failure of the backing Job). // JobRunning -> Status=False (still in progress). // JobSuspended -> Status=False (admin-suspended via spec). ConditionReasonJobComplete = "JobComplete" @@ -183,7 +183,7 @@ type S3TLS struct { } // BackupStatus mirrors Backup.status. The operator expresses success -// via condition type "Complete" with status=True. +// via condition type "Complete" with reason JobComplete. type BackupStatus struct { Conditions []metav1.Condition `json:"conditions,omitempty"` } @@ -217,7 +217,7 @@ type RestoreSpec struct { } // RestoreStatus mirrors Restore.status. Same condition convention as -// Backup: type=Complete, Status=True on success. +// Backup: type=Complete, reason JobComplete on success. type RestoreStatus struct { Conditions []metav1.Condition `json:"conditions,omitempty"` } From bd2b6c4b49ece9741db01d3013bbe6f8f837d3de Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:02:51 +0300 Subject: [PATCH 181/518] fix(backups): leave the credentials Secret alone when BucketInfo has no endpoint A BucketInfo without an endpoint differs from every credentials Secret that has one, so the gate forced the bucket release on every throttle window. That render cannot converge: an empty endpoint would be written into the Secret every driver reads, and with the key absent the template's trimPrefix fails and parks the release in Failed. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/default_objects_gate.go | 6 ++++-- internal/backupcontroller/default_objects_gate_test.go | 4 ++++ 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/internal/backupcontroller/default_objects_gate.go b/internal/backupcontroller/default_objects_gate.go index b99f67b41..9b6fc0ede 100644 --- a/internal/backupcontroller/default_objects_gate.go +++ b/internal/backupcontroller/default_objects_gate.go @@ -438,7 +438,9 @@ func (g *DefaultObjectsGate) Check(ctx context.Context) ([]string, bool, error) // writes for it. The COSI Secret being absent (the BucketAccess is being // re-granted) leaves nothing to compare against and is not staleness, and // neither is a source Secret the bucket chart does not render: the COSI Secret -// itself (the projector accepts one) or one without the endpoint key. +// itself (the projector accepts one) or one without the endpoint key. A +// BucketInfo without an endpoint is not a target either: rendering from it +// writes an empty endpoint, or fails the release when the key is absent. func (g *DefaultObjectsGate) staleCredentials(ctx context.Context, src *corev1.Secret) (string, error) { if g.CredentialsCOSISecretName == "" || g.CredentialsHelmRelease.Name == "" || g.CredentialsHelmRelease.Namespace == "" { return "", nil @@ -466,7 +468,7 @@ func (g *DefaultObjectsGate) staleCredentials(ctx context.Context, src *corev1.S } want := strings.TrimPrefix(strings.TrimPrefix(info.Spec.SecretS3.Endpoint, "https://"), "http://") have := string(src.Data["endpoint"]) - if have == want { + if want == "" || have == want { return "", nil } return fmt.Sprintf("Secret/%s (endpoint %q, COSI Secret has %q)", g.Config.SourceSecretName, have, want), nil diff --git a/internal/backupcontroller/default_objects_gate_test.go b/internal/backupcontroller/default_objects_gate_test.go index befaedabe..bed60053f 100644 --- a/internal/backupcontroller/default_objects_gate_test.go +++ b/internal/backupcontroller/default_objects_gate_test.go @@ -851,6 +851,10 @@ func TestCheckCredentialsEndpointComparison(t *testing.T) { {"bare host", "s3.example.com", false}, {"other host", "https://s3.new.example.com", true}, {"other port", "https://s3.example.com:8443", true}, + // A forced render from a BucketInfo with no endpoint would write an + // empty endpoint into the Secret every driver reads (or, with the key + // absent, fail the bucket release's render), never a working one. + {"no endpoint in BucketInfo", "", false}, } { t.Run(tc.name, func(t *testing.T) { objs := append([]runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()}, allDefaultObjects()...) From 7e1fc7173c628fdcbf5a1f1c810e0eb4edf92d28 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:03:14 +0300 Subject: [PATCH 182/518] test(mariadb): cover the absent Complete condition and expired deadlines Every first reconcile after the operator Backup or Restore is created sees no Complete condition, and the guards that keep that case from dereferencing nil could be dropped with the outcome tables staying green. The tables now also cover the deadline failure for a run that never reports a known reason, and read the job status back from the API, so a lost status write fails them instead of passing on the in-memory object. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadbstrategy_controller_test.go | 122 ++++++++++++------ 1 file changed, 79 insertions(+), 43 deletions(-) diff --git a/internal/backupcontroller/mariadbstrategy_controller_test.go b/internal/backupcontroller/mariadbstrategy_controller_test.go index 307301223..0c706574c 100644 --- a/internal/backupcontroller/mariadbstrategy_controller_test.go +++ b/internal/backupcontroller/mariadbstrategy_controller_test.go @@ -871,30 +871,64 @@ func TestReconcileMariaDBRestore_KindCheckedBeforeSrcBackupGet(t *testing.T) { // JobFailed (SetCompleteWithJob in mariadb-operator 25.10.2), so the // Complete status alone does not tell success from failure. var mariadbCompleteOutcomeCases = []struct { - name string - status metav1.ConditionStatus - reason string - wantFailed bool - wantDone bool + name string + noCondition bool + status metav1.ConditionStatus + reason string + message string + expired bool + wantFailed bool + wantDone bool + wantMessage []string }{ - {name: "Complete=True/JobComplete succeeds", status: metav1.ConditionTrue, reason: mariadbtypes.ConditionReasonJobComplete, wantDone: true}, - {name: "Complete=True/JobFailed fails", status: metav1.ConditionTrue, reason: mariadbtypes.ConditionReasonJobFailed, wantFailed: true}, - {name: "Complete=False/JobFailed fails", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobFailed, wantFailed: true}, - {name: "Complete=True with an unknown reason keeps waiting", status: metav1.ConditionTrue, reason: "SomethingElse"}, - {name: "Complete=False/JobComplete keeps waiting", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobComplete}, - {name: "Complete=False/JobRunning keeps waiting", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobRunning}, + {name: "Complete=True/JobComplete succeeds", status: metav1.ConditionTrue, reason: mariadbtypes.ConditionReasonJobComplete, message: "Success", wantDone: true}, + {name: "Complete=True/JobFailed fails", status: metav1.ConditionTrue, reason: mariadbtypes.ConditionReasonJobFailed, message: "Failed", wantFailed: true, + wantMessage: []string{"tenant/op"}}, + {name: "Complete=False/JobFailed fails", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobFailed, message: "Failed", wantFailed: true, + wantMessage: []string{"tenant/op"}}, + {name: "Complete=True with an unknown reason keeps waiting", status: metav1.ConditionTrue, reason: "SomethingElse", message: "Something"}, + {name: "Complete=False/JobComplete keeps waiting", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobComplete, message: "Success"}, + {name: "Complete=False/JobRunning keeps waiting", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobRunning, message: "Running"}, + {name: "no Complete condition keeps waiting", noCondition: true}, + {name: "no Complete condition past the deadline fails", noCondition: true, expired: true, wantFailed: true, + wantMessage: []string{"no Complete condition observed"}}, + {name: "Complete=True with an unknown reason past the deadline fails", status: metav1.ConditionTrue, reason: "SomethingElse", message: "Something", expired: true, wantFailed: true, + wantMessage: []string{"Complete=True/SomethingElse"}}, } -func mariadbCompleteCondition(status metav1.ConditionStatus, reason string) []metav1.Condition { +func mariadbCompleteConditions(noCondition bool, status metav1.ConditionStatus, reason, message string) []metav1.Condition { + if noCondition { + return nil + } return []metav1.Condition{{ Type: mariadbtypes.ConditionTypeComplete, Status: status, Reason: reason, - Message: "Failed", + Message: message, LastTransitionTime: metav1.Now(), }} } +// mariadbOutcomeStartedAt returns a StartedAt that is fresh, or older than +// both the backup and the default restore deadline. +func mariadbOutcomeStartedAt(expired bool) *metav1.Time { + if !expired { + now := metav1.Now() + return &now + } + past := metav1.NewTime(time.Now().Add(-2 * max(mariadbDefaultBackupDeadline, mariadbDefaultRestoreDeadline))) + return &past +} + +func assertMariaDBFailureMessage(t *testing.T, message string, want []string) { + t.Helper() + for _, w := range want { + if !strings.Contains(message, w) { + t.Errorf("failure message must contain %q, got %q", w, message) + } + } +} + func TestReconcileMariaDBBackup_CompleteConditionOutcome(t *testing.T) { strategyAPIGroup := strategyv1alpha1.GroupVersion.Group resolved := &ResolvedBackupConfig{ @@ -907,9 +941,8 @@ func TestReconcileMariaDBBackup_CompleteConditionOutcome(t *testing.T) { for _, tc := range mariadbCompleteOutcomeCases { t.Run(tc.name, func(t *testing.T) { - now := metav1.Now() seed := newMariaDBBackupJob("bj", "tenant") - seed.Status.StartedAt = &now + seed.Status.StartedAt = mariadbOutcomeStartedAt(tc.expired) strategy := &strategyv1alpha1.MariaDB{ ObjectMeta: metav1.ObjectMeta{Name: "mdb-strategy"}, Spec: strategyv1alpha1.MariaDBSpec{Template: *newRenderedMariaDBTemplate()}, @@ -920,13 +953,13 @@ func TestReconcileMariaDBBackup_CompleteConditionOutcome(t *testing.T) { operatorBackup := &mariadbtypes.Backup{ ObjectMeta: metav1.ObjectMeta{ Namespace: "tenant", - Name: "bj-op", + Name: "op", Labels: map[string]string{ backupsv1alpha1.OwningJobNameLabel: seed.Name, backupsv1alpha1.OwningJobNamespaceLabel: seed.Namespace, }, }, - Status: mariadbtypes.BackupStatus{Conditions: mariadbCompleteCondition(tc.status, tc.reason)}, + Status: mariadbtypes.BackupStatus{Conditions: mariadbCompleteConditions(tc.noCondition, tc.status, tc.reason, tc.message)}, } c := newMariaDBStrategyTestClient(t, seed, newMariaDBApp(seed.Spec.ApplicationRef.Name, "tenant"), strategy, operatorMariaDB, operatorBackup) @@ -941,31 +974,33 @@ func TestReconcileMariaDBBackup_CompleteConditionOutcome(t *testing.T) { t.Fatalf("reconcileMariaDB: %v", err) } + stored := &backupsv1alpha1.BackupJob{} + if err := c.Get(context.Background(), client.ObjectKeyFromObject(seed), stored); err != nil { + t.Fatalf("persisted Get: %v", err) + } artifacts := &backupsv1alpha1.BackupList{} if err := c.List(context.Background(), artifacts, client.InNamespace("tenant")); err != nil { t.Fatalf("list Backups: %v", err) } switch { case tc.wantFailed: - if bj.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed { - t.Errorf("expected Phase=Failed, got %q", bj.Status.Phase) + if stored.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed { + t.Errorf("expected Phase=Failed, got %q", stored.Status.Phase) } - if !strings.Contains(bj.Status.Message, "bj-op") { - t.Errorf("failure message must name the operator Backup, got %q", bj.Status.Message) - } - if bj.Status.BackupRef != nil || len(artifacts.Items) != 0 { - t.Errorf("a failed run must not produce a Backup: ref=%v artifacts=%d", bj.Status.BackupRef, len(artifacts.Items)) + assertMariaDBFailureMessage(t, stored.Status.Message, tc.wantMessage) + if stored.Status.BackupRef != nil || len(artifacts.Items) != 0 { + t.Errorf("a failed run must not produce a Backup: ref=%v artifacts=%d", stored.Status.BackupRef, len(artifacts.Items)) } case tc.wantDone: - if bj.Status.Phase != backupsv1alpha1.BackupJobPhaseSucceeded { - t.Errorf("expected Phase=Succeeded, got %q (message %q)", bj.Status.Phase, bj.Status.Message) + if stored.Status.Phase != backupsv1alpha1.BackupJobPhaseSucceeded { + t.Errorf("expected Phase=Succeeded, got %q (message %q)", stored.Status.Phase, stored.Status.Message) } - if bj.Status.BackupRef == nil || len(artifacts.Items) != 1 { - t.Errorf("a successful run must produce exactly one Backup: ref=%v artifacts=%d", bj.Status.BackupRef, len(artifacts.Items)) + if stored.Status.BackupRef == nil || len(artifacts.Items) != 1 { + t.Errorf("a successful run must produce exactly one Backup: ref=%v artifacts=%d", stored.Status.BackupRef, len(artifacts.Items)) } default: - if bj.Status.Phase != backupsv1alpha1.BackupJobPhaseRunning { - t.Errorf("expected Phase=Running, got %q", bj.Status.Phase) + if stored.Status.Phase != backupsv1alpha1.BackupJobPhaseRunning { + t.Errorf("expected Phase=Running, got %q", stored.Status.Phase) } if res.RequeueAfter == 0 { t.Errorf("expected a requeue while the run is in progress, got %+v", res) @@ -983,7 +1018,6 @@ func TestReconcileMariaDBRestore_CompleteConditionOutcome(t *testing.T) { for _, tc := range mariadbCompleteOutcomeCases { t.Run(tc.name, func(t *testing.T) { - now := metav1.Now() cozyBackup := &backupsv1alpha1.Backup{ ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "src-cozy-bk"}, Spec: backupsv1alpha1.BackupSpec{ @@ -997,18 +1031,18 @@ func TestReconcileMariaDBRestore_CompleteConditionOutcome(t *testing.T) { BackupRef: corev1.LocalObjectReference{Name: cozyBackup.Name}, TargetApplicationRef: &corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "target", APIGroup: &apps}, }, - Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &now, Phase: backupsv1alpha1.RestoreJobPhaseRunning}, + Status: backupsv1alpha1.RestoreJobStatus{StartedAt: mariadbOutcomeStartedAt(tc.expired), Phase: backupsv1alpha1.RestoreJobPhaseRunning}, } operatorRestore := &mariadbtypes.Restore{ ObjectMeta: metav1.ObjectMeta{ Namespace: "tenant", - Name: "rj-op", + Name: "op", Labels: map[string]string{ backupsv1alpha1.OwningJobNameLabel: seed.Name, backupsv1alpha1.OwningJobNamespaceLabel: seed.Namespace, }, }, - Status: mariadbtypes.RestoreStatus{Conditions: mariadbCompleteCondition(tc.status, tc.reason)}, + Status: mariadbtypes.RestoreStatus{Conditions: mariadbCompleteConditions(tc.noCondition, tc.status, tc.reason, tc.message)}, } c := newMariaDBStrategyTestClient(t, cozyBackup, seed, operatorRestore, &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "src-op-bk"}}, @@ -1024,21 +1058,23 @@ func TestReconcileMariaDBRestore_CompleteConditionOutcome(t *testing.T) { t.Fatalf("reconcileMariaDBRestore: %v", err) } + stored := &backupsv1alpha1.RestoreJob{} + if err := c.Get(context.Background(), client.ObjectKeyFromObject(seed), stored); err != nil { + t.Fatalf("persisted Get: %v", err) + } switch { case tc.wantFailed: - if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { - t.Errorf("expected Phase=Failed, got %q", rj.Status.Phase) - } - if !strings.Contains(rj.Status.Message, "rj-op") { - t.Errorf("failure message must name the operator Restore, got %q", rj.Status.Message) + if stored.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Errorf("expected Phase=Failed, got %q", stored.Status.Phase) } + assertMariaDBFailureMessage(t, stored.Status.Message, tc.wantMessage) case tc.wantDone: - if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded { - t.Errorf("expected Phase=Succeeded, got %q (message %q)", rj.Status.Phase, rj.Status.Message) + if stored.Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded { + t.Errorf("expected Phase=Succeeded, got %q (message %q)", stored.Status.Phase, stored.Status.Message) } default: - if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseRunning { - t.Errorf("expected Phase=Running, got %q", rj.Status.Phase) + if stored.Status.Phase != backupsv1alpha1.RestoreJobPhaseRunning { + t.Errorf("expected Phase=Running, got %q", stored.Status.Phase) } if res.RequeueAfter == 0 { t.Errorf("expected a requeue while the run is in progress, got %+v", res) From dc909c778c8585c5acb4cadcd456fa4a0d13d5cd Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:03:16 +0300 Subject: [PATCH 183/518] fix(backups): compare the credentials endpoint only for the forced bucket's Secret The gate forces bucket--system, which renders the credentials Secret of that bucket's system user and nothing else. The COSI Secret name was derived from systemSecretName instead, so a value naming another bucket's Secret made a lagging endpoint there force the platform bucket release every throttle window without ever refreshing the Secret that was compared. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../templates/deployment.yaml | 9 ++++++--- .../tests/default_objects_gate_test.yaml | 15 +++++++++++++++ 2 files changed, 21 insertions(+), 3 deletions(-) diff --git a/packages/system/backupstrategy-controller/templates/deployment.yaml b/packages/system/backupstrategy-controller/templates/deployment.yaml index 1278bec8a..887b7e64e 100644 --- a/packages/system/backupstrategy-controller/templates/deployment.yaml +++ b/packages/system/backupstrategy-controller/templates/deployment.yaml @@ -91,10 +91,13 @@ spec: # (packages/system/bucket/templates/user-credentials.yaml appends # "-credentials" to its name). The COSI sidecar writes it once per # BucketAccess, so the gate compares the two endpoints and forces the - # release when the credentials Secret lags behind. - {{- if hasSuffix "-credentials" .Values.backupStorage.systemSecretName }} + # release when the credentials Secret lags behind. Only the system + # user's Secret of this very bucket is rendered by that release; any + # other systemSecretName would be forced without ever changing. + {{- $cosiSecretName := printf "bucket-%s-system" .Values.backupStorage.bucketName }} + {{- if eq .Values.backupStorage.systemSecretName (printf "%s-credentials" $cosiSecretName) }} - name: BACKUP_CREDENTIALS_COSI_SECRET_NAME - value: {{ trimSuffix "-credentials" .Values.backupStorage.systemSecretName | quote }} + value: {{ $cosiSecretName | quote }} {{- end }} {{- end }} {{- if .Values.velero.bslEnabled }} diff --git a/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml b/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml index 446bbdcfa..a222d66ee 100644 --- a/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml +++ b/packages/system/backupstrategy-controller/tests/default_objects_gate_test.yaml @@ -141,6 +141,21 @@ tests: content: name: BACKUP_CREDENTIALS_COSI_SECRET_NAME + # The forced release renders only -credentials for its own system + # user. A systemSecretName naming another bucket's Secret is not refreshed by + # forcing bucket--system, so comparing it would force that + # release every MinForceInterval and change nothing. + - it: "systemSecretName of another bucket: no COSI Secret to compare against" + template: templates/deployment.yaml + set: + backupStorage.systemSecretName: bucket-other-system-credentials + asserts: + - notContains: + path: spec.template.spec.containers[0].env + any: true + content: + name: BACKUP_CREDENTIALS_COSI_SECRET_NAME + - it: "velero.bslEnabled=false: the BSL is not rendered, so the gate must not look for it" template: templates/deployment.yaml set: From 9f51b9db041f2d8753d0d81f2c0ad447827a11d7 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:05:36 +0300 Subject: [PATCH 184/518] fix(mariadb): report the operator Job's own failure on JobFailed The operator's JobFailed condition message is only "Failed", and with the operator's default restartPolicy OnFailure the Job controller deletes the pod once the backoff limit is spent, before the operator sets JobFailed. Pointing the reader at the Job's logs therefore sent them to logs that no longer exist. The Job, which the operator names after the Backup or Restore, keeps its Failed condition with the reason and message, so the BackupJob or RestoreJob now carries those instead; a Job that cannot be read is named in the message rather than dropped. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadbstrategy_controller.go | 29 +++++-- .../mariadbstrategy_controller_test.go | 81 +++++++++++++++++-- 2 files changed, 94 insertions(+), 16 deletions(-) diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 48abd3644..c724e2dab 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -7,6 +7,7 @@ import ( "fmt" "time" + batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" apimeta "k8s.io/apimachinery/pkg/api/meta" @@ -205,7 +206,7 @@ func (r *BackupJobReconciler) reconcileMariaDB(ctx context.Context, j *backupsv1 switch { case cond != nil && cond.Reason == mariadbtypes.ConditionReasonJobFailed: - return r.markBackupJobFailed(ctx, j, mariadbJobFailedMessage("Backup", mdbBackup.Namespace, mdbBackup.Name, cond)) + return r.markBackupJobFailed(ctx, j, mariadbJobFailedMessage(ctx, r.Client, "Backup", mdbBackup.Namespace, mdbBackup.Name, cond)) case cond != nil && cond.Status == metav1.ConditionTrue && cond.Reason == mariadbtypes.ConditionReasonJobComplete: // Backup completed successfully. Materialise the Cozystack artifact @@ -252,12 +253,24 @@ func (r *BackupJobReconciler) reconcileMariaDB(ctx context.Context, j *backupsv1 } } -// mariadbJobFailedMessage names the operator object whose Job failed. The -// operator's own condition message is only "Failed"; the cause is in the -// logs of that object's Job. -func mariadbJobFailedMessage(kind, namespace, name string, cond *metav1.Condition) string { - return fmt.Sprintf("k8s.mariadb.com/%s %s/%s failed (Complete=%s/%s); the cause is in the logs of its Job", - kind, namespace, name, cond.Status, cond.Reason) +// mariadbJobFailedMessage names the operator object whose Job failed and +// carries that Job's own Failed condition, because the operator's condition +// message is only "Failed". The operator names the Job after the object. +func mariadbJobFailedMessage(ctx context.Context, c client.Reader, kind, namespace, name string, cond *metav1.Condition) string { + message := fmt.Sprintf("k8s.mariadb.com/%s %s/%s failed (Complete=%s/%s)", kind, namespace, name, cond.Status, cond.Reason) + job := &batchv1.Job{} + if err := c.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, job); err != nil { + if apierrors.IsNotFound(err) { + return fmt.Sprintf("%s; Job %s/%s not found", message, namespace, name) + } + return fmt.Sprintf("%s; reading Job %s/%s: %v", message, namespace, name, err) + } + for _, jc := range job.Status.Conditions { + if jc.Type == batchv1.JobFailed && jc.Status == corev1.ConditionTrue { + return fmt.Sprintf("%s; Job %s/%s: %s: %s", message, namespace, name, jc.Reason, jc.Message) + } + } + return fmt.Sprintf("%s; Job %s/%s reports no Failed condition", message, namespace, name) } // mariadbBackupDeadlineExceeded reports whether enough wall-clock time has @@ -572,7 +585,7 @@ func (r *RestoreJobReconciler) reconcileMariaDBRestore(ctx context.Context, rest cond := apimeta.FindStatusCondition(mdbRestore.Status.Conditions, mariadbtypes.ConditionTypeComplete) switch { case cond != nil && cond.Reason == mariadbtypes.ConditionReasonJobFailed: - return r.markRestoreJobFailed(ctx, restoreJob, mariadbJobFailedMessage("Restore", mdbRestore.Namespace, mdbRestore.Name, cond)) + return r.markRestoreJobFailed(ctx, restoreJob, mariadbJobFailedMessage(ctx, r.Client, "Restore", mdbRestore.Namespace, mdbRestore.Name, cond)) case cond != nil && cond.Status == metav1.ConditionTrue && cond.Reason == mariadbtypes.ConditionReasonJobComplete: now := metav1.Now() diff --git a/internal/backupcontroller/mariadbstrategy_controller_test.go b/internal/backupcontroller/mariadbstrategy_controller_test.go index 0c706574c..2ace7329e 100644 --- a/internal/backupcontroller/mariadbstrategy_controller_test.go +++ b/internal/backupcontroller/mariadbstrategy_controller_test.go @@ -4,12 +4,15 @@ package backupcontroller import ( "context" "encoding/json" + "errors" "strings" "testing" "time" + batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" apiequality "k8s.io/apimachinery/pkg/api/equality" + apierrors "k8s.io/apimachinery/pkg/api/errors" apimeta "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" @@ -17,6 +20,7 @@ import ( "k8s.io/client-go/tools/record" "sigs.k8s.io/controller-runtime/pkg/client" clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" @@ -877,15 +881,18 @@ var mariadbCompleteOutcomeCases = []struct { reason string message string expired bool + failedJob bool wantFailed bool wantDone bool wantMessage []string }{ {name: "Complete=True/JobComplete succeeds", status: metav1.ConditionTrue, reason: mariadbtypes.ConditionReasonJobComplete, message: "Success", wantDone: true}, - {name: "Complete=True/JobFailed fails", status: metav1.ConditionTrue, reason: mariadbtypes.ConditionReasonJobFailed, message: "Failed", wantFailed: true, - wantMessage: []string{"tenant/op"}}, - {name: "Complete=False/JobFailed fails", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobFailed, message: "Failed", wantFailed: true, - wantMessage: []string{"tenant/op"}}, + {name: "Complete=True/JobFailed fails with the Job's failure", status: metav1.ConditionTrue, reason: mariadbtypes.ConditionReasonJobFailed, message: "Failed", failedJob: true, wantFailed: true, + wantMessage: []string{"tenant/op", "BackoffLimitExceeded", "Job has reached the specified backoff limit"}}, + {name: "Complete=False/JobFailed fails with the Job's failure", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobFailed, message: "Failed", failedJob: true, wantFailed: true, + wantMessage: []string{"tenant/op", "BackoffLimitExceeded", "Job has reached the specified backoff limit"}}, + {name: "Complete=True/JobFailed with its Job gone fails", status: metav1.ConditionTrue, reason: mariadbtypes.ConditionReasonJobFailed, message: "Failed", wantFailed: true, + wantMessage: []string{"tenant/op", "Job tenant/op not found"}}, {name: "Complete=True with an unknown reason keeps waiting", status: metav1.ConditionTrue, reason: "SomethingElse", message: "Something"}, {name: "Complete=False/JobComplete keeps waiting", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobComplete, message: "Success"}, {name: "Complete=False/JobRunning keeps waiting", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobRunning, message: "Running"}, @@ -920,8 +927,29 @@ func mariadbOutcomeStartedAt(expired bool) *metav1.Time { return &past } +// mariadbOutcomeObjects adds the operator's Job, named after the operator +// Backup or Restore, failed the way the Job controller fails it once the +// backoff limit is spent. +func mariadbOutcomeObjects(failedJob bool, objs ...client.Object) []client.Object { + if !failedJob { + return objs + } + return append(objs, &batchv1.Job{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "op"}, + Status: batchv1.JobStatus{Conditions: []batchv1.JobCondition{{ + Type: batchv1.JobFailed, + Status: corev1.ConditionTrue, + Reason: batchv1.JobReasonBackoffLimitExceeded, + Message: "Job has reached the specified backoff limit", + }}}, + }) +} + func assertMariaDBFailureMessage(t *testing.T, message string, want []string) { t.Helper() + if strings.Contains(message, "logs") { + t.Errorf("failure message must not send the reader to Job logs, which are gone with the deleted pod: %q", message) + } for _, w := range want { if !strings.Contains(message, w) { t.Errorf("failure message must contain %q, got %q", w, message) @@ -961,8 +989,8 @@ func TestReconcileMariaDBBackup_CompleteConditionOutcome(t *testing.T) { }, Status: mariadbtypes.BackupStatus{Conditions: mariadbCompleteConditions(tc.noCondition, tc.status, tc.reason, tc.message)}, } - c := newMariaDBStrategyTestClient(t, seed, newMariaDBApp(seed.Spec.ApplicationRef.Name, "tenant"), - strategy, operatorMariaDB, operatorBackup) + c := newMariaDBStrategyTestClient(t, mariadbOutcomeObjects(tc.failedJob, seed, newMariaDBApp(seed.Spec.ApplicationRef.Name, "tenant"), + strategy, operatorMariaDB, operatorBackup)...) r := &BackupJobReconciler{Client: c, Scheme: c.Scheme(), Recorder: record.NewFakeRecorder(10)} bj := &backupsv1alpha1.BackupJob{} @@ -1013,6 +1041,43 @@ func TestReconcileMariaDBBackup_CompleteConditionOutcome(t *testing.T) { } } +func TestMariaDBJobFailedMessage_JobUnreadable(t *testing.T) { + cond := &metav1.Condition{Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionTrue, Reason: mariadbtypes.ConditionReasonJobFailed} + readErr := apierrors.NewForbidden(batchv1.Resource("jobs"), "op", errors.New("rbac")) + c := interceptor.NewClient(newMariaDBStrategyTestClient(t).(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*batchv1.Job); ok { + return readErr + } + return cl.Get(ctx, key, obj, opts...) + }, + }) + + got := mariadbJobFailedMessage(context.Background(), c, "Backup", "tenant", "op", cond) + for _, want := range []string{"k8s.mariadb.com/Backup tenant/op failed (Complete=True/JobFailed)", "reading Job tenant/op", readErr.Error()} { + if !strings.Contains(got, want) { + t.Errorf("message must contain %q, got %q", want, got) + } + } +} + +func TestMariaDBJobFailedMessage_JobWithoutFailedCondition(t *testing.T) { + cond := &metav1.Condition{Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionTrue, Reason: mariadbtypes.ConditionReasonJobFailed} + c := newMariaDBStrategyTestClient(t, &batchv1.Job{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "op"}, + Status: batchv1.JobStatus{Conditions: []batchv1.JobCondition{ + {Type: batchv1.JobFailureTarget, Status: corev1.ConditionTrue, Reason: batchv1.JobReasonPodFailurePolicy}, + {Type: batchv1.JobFailed, Status: corev1.ConditionFalse, Reason: batchv1.JobReasonBackoffLimitExceeded}, + }}, + }) + + got := mariadbJobFailedMessage(context.Background(), c, "Restore", "tenant", "op", cond) + if !strings.Contains(got, "Job tenant/op reports no Failed condition") || + strings.Contains(got, batchv1.JobReasonBackoffLimitExceeded) || strings.Contains(got, batchv1.JobReasonPodFailurePolicy) { + t.Errorf("a Failed condition that is not True must not be reported as the cause, got %q", got) + } +} + func TestReconcileMariaDBRestore_CompleteConditionOutcome(t *testing.T) { apps := mariadbapp.GroupName @@ -1044,9 +1109,9 @@ func TestReconcileMariaDBRestore_CompleteConditionOutcome(t *testing.T) { }, Status: mariadbtypes.RestoreStatus{Conditions: mariadbCompleteConditions(tc.noCondition, tc.status, tc.reason, tc.message)}, } - c := newMariaDBStrategyTestClient(t, cozyBackup, seed, operatorRestore, + c := newMariaDBStrategyTestClient(t, mariadbOutcomeObjects(tc.failedJob, cozyBackup, seed, operatorRestore, &mariadbtypes.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "src-op-bk"}}, - &mariadbtypes.MariaDB{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: mariadbNameForApp("target")}}) + &mariadbtypes.MariaDB{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: mariadbNameForApp("target")}})...) r := &RestoreJobReconciler{Client: c, Recorder: record.NewFakeRecorder(10)} rj := &backupsv1alpha1.RestoreJob{} From ee5e51ee6ce272302bc47932ba58eadb4c7c6dc0 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:04:25 +0300 Subject: [PATCH 185/518] fix(backups): keep forcing objects that still lack the endpoint annotation An object without backups.cozystack.io/endpoint stopped counting as stale as soon as one forced render had been requested, whether or not that render succeeded. A failed upgrade left the objects unannotated and the gate silent until the controller restarted, which also hid the climbing force counter meant to report exactly that failure. Such an object now stays stale until a render stamps it; MinForceInterval already bounds the cost. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/backupstrategy-crd-gate.bats | 6 +-- .../backupcontroller/default_objects_gate.go | 19 +++------- .../default_objects_gate_test.go | 38 ++++++------------- 3 files changed, 19 insertions(+), 44 deletions(-) diff --git a/hack/backupstrategy-crd-gate.bats b/hack/backupstrategy-crd-gate.bats index 0f68a1ca4..fac48f35e 100644 --- a/hack/backupstrategy-crd-gate.bats +++ b/hack/backupstrategy-crd-gate.bats @@ -137,9 +137,9 @@ $tpl: $obj does not gate on crdEstablished for $crd" # The controller's DefaultObjectsGate tells an object rendered with a stale S3 # endpoint (publishing.host changed after the render) by the # backups.cozystack.io/endpoint annotation, because the endpoint itself sits at -# a different path in every kind. An object the chart renders without it is -# forced once and then never compared again, so a new template that forgets it -# would silently keep a stale endpoint. +# a different path in every kind. An object the chart renders without it +# counts as stale on every check, so a new template that forgets it would +# force this release every MinForceInterval forever. @test "every rendered Strategy CR and the Velero BSL carry the rendered endpoint annotation" { tmp=$(mktemp -d) diff --git a/internal/backupcontroller/default_objects_gate.go b/internal/backupcontroller/default_objects_gate.go index 9b6fc0ede..9aea96f31 100644 --- a/internal/backupcontroller/default_objects_gate.go +++ b/internal/backupcontroller/default_objects_gate.go @@ -194,12 +194,6 @@ type DefaultObjectsGate struct { // MinForceInterval for no reason. lastForce time.Time lastCredentialsForce time.Time - // renderForced records that a forced render of HelmRelease was requested - // (the annotations were patched; whether the upgrade succeeds is not - // observed). That render stamps endpointAnnotation on every object the - // chart renders, so from then on an object without it (one rendered by a - // chart that predates the annotation) is not counted as stale again. - renderForced bool } // errHelmReleaseSuspended reports that the target release has @@ -423,10 +417,7 @@ func (g *DefaultObjectsGate) Check(ctx context.Context) ([]string, bool, error) sort.Strings(objects) _, objectsForced, err := g.force(ctx, g.HelmRelease, &g.lastForce, objects) needed = append(needed, objects...) - if objectsForced { - forced = true - g.renderForced = true - } + forced = forced || objectsForced errs = append(errs, err) } return needed, forced, errors.Join(errs...) @@ -563,7 +554,9 @@ type defaultObjectsState struct { } // staleEndpoint reports an existing object rendered with another endpoint -// than want, formatted for logs. An empty want disables the comparison. +// than want, formatted for logs. An empty want disables the comparison. An +// object without the annotation (rendered by a chart that predates it) stays +// stale until a render stamps it: requesting one does not prove it landed. func (g *DefaultObjectsGate) staleEndpoint(state *defaultObjectsState, key string, obj *unstructured.Unstructured, want string) { if want == "" { return @@ -571,9 +564,7 @@ func (g *DefaultObjectsGate) staleEndpoint(state *defaultObjectsState, key strin have, ok := obj.GetAnnotations()[endpointAnnotation] switch { case !ok: - if !g.renderForced { - state.stale = append(state.stale, key+" (no endpoint annotation)") - } + state.stale = append(state.stale, key+" (no endpoint annotation)") case have != want: state.stale = append(state.stale, fmt.Sprintf("%s (endpoint %q, want %q)", key, have, want)) } diff --git a/internal/backupcontroller/default_objects_gate_test.go b/internal/backupcontroller/default_objects_gate_test.go index bed60053f..e287c9872 100644 --- a/internal/backupcontroller/default_objects_gate_test.go +++ b/internal/backupcontroller/default_objects_gate_test.go @@ -1037,13 +1037,13 @@ func TestCheckIgnoresObjectEndpointWithoutSecretEndpoint(t *testing.T) { } } -// TestCheckForcesOnceForUnannotatedObjects covers objects rendered by a chart -// that predates the endpoint annotation. Their endpoint is unknown, so they -// are treated as stale until one forced render has gone through, which stamps -// the annotation. A force that did not go through (throttled) does not count, -// and once one has, an object still without the annotation (one the chart -// does not render) must not force again every MinForceInterval. -func TestCheckForcesOnceForUnannotatedObjects(t *testing.T) { +// TestCheckKeepsForcingForUnannotatedObjects covers objects rendered by a +// chart that predates the endpoint annotation. Their endpoint is unknown, so +// they count as stale until a render stamps the annotation. Having requested +// a render proves nothing: one that failed leaves them unannotated, and the +// next unthrottled check must force again rather than fall silent, so the +// force counter keeps reporting the render that does not land. +func TestCheckKeepsForcingForUnannotatedObjects(t *testing.T) { base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) objs := []runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()} for _, o := range allDefaultObjects() { @@ -1065,27 +1065,11 @@ func TestCheckForcesOnceForUnannotatedObjects(t *testing.T) { g.now = func() time.Time { return base.Add(time.Hour) } missing, forced, err := g.Check(context.Background()) - if err != nil || forced || len(missing) != 0 { - t.Fatalf("after the forced render: missing = %v, forced = %v, err = %v; want a no-op", missing, forced, err) + if err != nil || !forced || len(missing) != 7 { + t.Fatalf("after a render that left them unannotated: missing = %v, forced = %v, err = %v; want 7 reported and forced again", missing, forced, err) } - if again := forceAnnotations(t, dyn)["reconcile.fluxcd.io/forceAt"]; again != first { - t.Errorf("forced again for unannotated objects: %q, then %q", first, again) - } -} - -// TestCheckStillForcesStaleAfterUnannotatedForce pins that the once-only -// treatment covers the missing annotation alone: an annotation that names the -// wrong endpoint still forces after that. -func TestCheckStillForcesStaleAfterUnannotatedForce(t *testing.T) { - objs := []runtime.Object{helmReleaseObject(), credentialsHelmReleaseObject()} - for _, o := range allDefaultObjects() { - objs = append(objs, withEndpoint(o.(*unstructured.Unstructured), "https://s3.old.example.com")) - } - g, _ := newGate(t, []client.Object{sourceSecret("bucket-1a2b"), cozyDefaultBackupClass()}, objs...) - g.renderForced = true - - if _, forced, err := g.Check(context.Background()); err != nil || !forced { - t.Fatalf("forced = %v, err = %v; want forced", forced, err) + if again := forceAnnotations(t, dyn)["reconcile.fluxcd.io/forceAt"]; again == first { + t.Errorf("not forced again for unannotated objects: forceAt still %q", first) } } From 69cea9ccbdd3799be27e465db6479d3313e4dcda Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:06:27 +0300 Subject: [PATCH 186/518] fix(mariadb): keep the operator's message in deadline failures The operator sets Complete=False with reason Failed and its own cause in the message ("Error creating Job", "Backup not complete") when a run fails before or outside its Job. Such a run waits out the deadline, and the deadline failure kept only the status and reason, so the BackupJob or RestoreJob ended without the one line that says why. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadbstrategy_controller.go | 26 ++++++++++++------- .../mariadbstrategy_controller_test.go | 17 ++++++++++++ 2 files changed, 33 insertions(+), 10 deletions(-) diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index c724e2dab..f17d6fdaf 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -242,12 +242,8 @@ func (r *BackupJobReconciler) reconcileMariaDB(ctx context.Context, j *backupsv1 // Backup eventually fails the BackupJob instead of pinning it Running // forever. if mariadbBackupDeadlineExceeded(j.Status.StartedAt) { - detail := "no Complete condition observed" - if cond != nil { - detail = fmt.Sprintf("Complete=%s/%s", cond.Status, cond.Reason) - } return r.markBackupJobFailed(ctx, j, fmt.Sprintf( - "k8s.mariadb.com Backup did not complete within %s (%s)", mariadbDefaultBackupDeadline, detail)) + "k8s.mariadb.com Backup did not complete within %s (%s)", mariadbDefaultBackupDeadline, mariadbCompleteDetail(cond))) } return ctrl.Result{RequeueAfter: mariadbPollInterval}, nil } @@ -273,6 +269,20 @@ func mariadbJobFailedMessage(ctx context.Context, c client.Reader, kind, namespa return fmt.Sprintf("%s; Job %s/%s reports no Failed condition", message, namespace, name) } +// mariadbCompleteDetail describes the last Complete condition for a deadline +// failure. The message matters for reason Failed, which the operator sets +// with its own cause ("Error creating Job", "Backup not complete") rather +// than from a Job. +func mariadbCompleteDetail(cond *metav1.Condition) string { + if cond == nil { + return "no Complete condition observed" + } + if cond.Message == "" { + return fmt.Sprintf("Complete=%s/%s", cond.Status, cond.Reason) + } + return fmt.Sprintf("Complete=%s/%s: %s", cond.Status, cond.Reason, cond.Message) +} + // mariadbBackupDeadlineExceeded reports whether enough wall-clock time has // elapsed since the BackupJob started that we should give up on a stuck // operator Backup. Returns false when StartedAt is nil so the very first @@ -605,13 +615,9 @@ func (r *RestoreJobReconciler) reconcileMariaDBRestore(ctx context.Context, rest default: deadline := options.effectiveRestoreDeadline() if restoreJob.Status.StartedAt != nil && time.Since(restoreJob.Status.StartedAt.Time) > deadline { - detail := "no Complete condition observed" - if cond != nil { - detail = fmt.Sprintf("Complete=%s/%s", cond.Status, cond.Reason) - } return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf( "k8s.mariadb.com/Restore did not complete within %s (%s; override via spec.options.restoreTimeoutSeconds)", - deadline, detail)) + deadline, mariadbCompleteDetail(cond))) } return ctrl.Result{RequeueAfter: mariadbPollInterval}, nil } diff --git a/internal/backupcontroller/mariadbstrategy_controller_test.go b/internal/backupcontroller/mariadbstrategy_controller_test.go index 2ace7329e..a23351584 100644 --- a/internal/backupcontroller/mariadbstrategy_controller_test.go +++ b/internal/backupcontroller/mariadbstrategy_controller_test.go @@ -901,6 +901,8 @@ var mariadbCompleteOutcomeCases = []struct { wantMessage: []string{"no Complete condition observed"}}, {name: "Complete=True with an unknown reason past the deadline fails", status: metav1.ConditionTrue, reason: "SomethingElse", message: "Something", expired: true, wantFailed: true, wantMessage: []string{"Complete=True/SomethingElse"}}, + {name: "Complete=False/Failed past the deadline carries the operator's message", status: metav1.ConditionFalse, reason: "Failed", message: "Error creating Job", expired: true, wantFailed: true, + wantMessage: []string{"Complete=False/Failed: Error creating Job"}}, } func mariadbCompleteConditions(noCondition bool, status metav1.ConditionStatus, reason, message string) []metav1.Condition { @@ -1078,6 +1080,21 @@ func TestMariaDBJobFailedMessage_JobWithoutFailedCondition(t *testing.T) { } } +func TestMariaDBCompleteDetail(t *testing.T) { + for _, tc := range []struct { + cond *metav1.Condition + want string + }{ + {nil, "no Complete condition observed"}, + {&metav1.Condition{Status: metav1.ConditionFalse, Reason: "Failed", Message: "Backup not complete"}, "Complete=False/Failed: Backup not complete"}, + {&metav1.Condition{Status: metav1.ConditionFalse, Reason: "Failed"}, "Complete=False/Failed"}, + } { + if got := mariadbCompleteDetail(tc.cond); got != tc.want { + t.Errorf("mariadbCompleteDetail(%+v) = %q, want %q", tc.cond, got, tc.want) + } + } +} + func TestReconcileMariaDBRestore_CompleteConditionOutcome(t *testing.T) { apps := mariadbapp.GroupName From c6c107fd08fd62e3b48ef8f905c4e299ff25cf00 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:14:32 +0300 Subject: [PATCH 187/518] docs(backups): state when the gate compares endpoints in the values reference The values reference read as if the credentials endpoint were always compared and an object without the endpoint annotation never forced a render. Operators sizing the force counter or choosing a custom systemSecretName need both conditions spelled out. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- packages/system/backupstrategy-controller/values.yaml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/packages/system/backupstrategy-controller/values.yaml b/packages/system/backupstrategy-controller/values.yaml index 67f986e47..590d7aca2 100644 --- a/packages/system/backupstrategy-controller/values.yaml +++ b/packages/system/backupstrategy-controller/values.yaml @@ -164,10 +164,12 @@ backupStorage: # cozy-default routes to exists and forces THIS release when any is # missing. It repairs a stale S3 endpoint the same way: when the # credentials Secret's endpoint differs from the COSI Secret it is rendered - # from it forces the bucket's -system release, and when the + # from it forces the bucket's -system release (only while + # systemSecretName is bucket--system-credentials, the one + # Secret that release renders), and when the # backups.cozystack.io/endpoint annotation on the Velero BSL or a default - # Strategy differs from the credentials Secret's endpoint it forces THIS - # release. The COSI Secret itself keeps the endpoint it was granted with, so + # Strategy differs from the credentials Secret's endpoint, or is absent, it + # forces THIS release. The COSI Secret itself keeps the endpoint it was granted with, so # after a publishing.host change that endpoint follows only once the # BucketAccess is re-granted. Both are throttled, independently, and a no-op # in the steady state. A suspended release is skipped rather than From 5f0da5a09aab36000e4e1004065ce73e83e3f142 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 22:11:54 +0300 Subject: [PATCH 188/518] docs(mariadb): list the operator's Failed reason on the Complete condition The reason list read as everything the operator writes on Complete, yet mariadb-operator also sets Complete=False with reason Failed when a run fails before or outside its Job. A reader taking the list as exhaustive would treat Failed as an unknown reason, or make it terminal, while the operator retries those failures. A table row pins that the driver keeps waiting on it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/mariadbstrategy_controller_test.go | 1 + internal/backupcontroller/mariadbtypes/types.go | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/internal/backupcontroller/mariadbstrategy_controller_test.go b/internal/backupcontroller/mariadbstrategy_controller_test.go index a23351584..f03b67d94 100644 --- a/internal/backupcontroller/mariadbstrategy_controller_test.go +++ b/internal/backupcontroller/mariadbstrategy_controller_test.go @@ -896,6 +896,7 @@ var mariadbCompleteOutcomeCases = []struct { {name: "Complete=True with an unknown reason keeps waiting", status: metav1.ConditionTrue, reason: "SomethingElse", message: "Something"}, {name: "Complete=False/JobComplete keeps waiting", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobComplete, message: "Success"}, {name: "Complete=False/JobRunning keeps waiting", status: metav1.ConditionFalse, reason: mariadbtypes.ConditionReasonJobRunning, message: "Running"}, + {name: "Complete=False/Failed keeps waiting", status: metav1.ConditionFalse, reason: "Failed", message: "Error creating Job"}, {name: "no Complete condition keeps waiting", noCondition: true}, {name: "no Complete condition past the deadline fails", noCondition: true, expired: true, wantFailed: true, wantMessage: []string{"no Complete condition observed"}}, diff --git a/internal/backupcontroller/mariadbtypes/types.go b/internal/backupcontroller/mariadbtypes/types.go index d9c09f30a..61fc71019 100644 --- a/internal/backupcontroller/mariadbtypes/types.go +++ b/internal/backupcontroller/mariadbtypes/types.go @@ -37,6 +37,10 @@ const ( // JobFailed -> Status=True (terminal failure of the backing Job). // JobRunning -> Status=False (still in progress). // JobSuspended -> Status=False (admin-suspended via spec). + // The operator also writes reason Failed with Status=False and its own + // message ("Error creating Job", "Backup not complete") when a run fails + // outside its Job. It retries those, so the driver keeps waiting on + // Failed until its deadline rather than failing the run at once. ConditionReasonJobComplete = "JobComplete" ConditionReasonJobFailed = "JobFailed" ConditionReasonJobRunning = "JobRunning" From e66b764734a505a85009922c36629a4d674512c6 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Sun, 8 Nov 2026 20:21:22 +0300 Subject: [PATCH 189/518] fix(backups): let an unfinished MariaDB run block a legacy dump delete A Backup taken before dumps were recorded finds its dump by window, and only finished neighbour runs were allowed to contest a candidate. A run uploads its dump and prunes older ones through maxRetention before its operator Backup turns Complete, so in that gap the pruned run's cleanup saw the newer run's dump as the only candidate and deleted it, leaving that Backup Ready with nothing to restore. The size probe and the restore keep ignoring unfinished runs, where a wrong guess costs a size or a refused restore. Deletion now treats a run that has not completed as one whose window is still open, so the dump is left behind with a Warning, as for any other unproven attribution. Holding the Backup would not help: once that run completes its window still covers the candidate, and the attribution stays ambiguous. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backupcontroller/mariadb_artifact_size.go | 37 ++++++++----- .../mariadbstrategy_cleanup.go | 2 +- .../mariadbstrategy_cleanup_test.go | 52 +++++++++++++++++++ 3 files changed, 77 insertions(+), 14 deletions(-) diff --git a/internal/backupcontroller/mariadb_artifact_size.go b/internal/backupcontroller/mariadb_artifact_size.go index f9b9973ad..4e0525517 100644 --- a/internal/backupcontroller/mariadb_artifact_size.go +++ b/internal/backupcontroller/mariadb_artifact_size.go @@ -64,18 +64,19 @@ func findMariaDBDump(ctx context.Context, store objectStore, mdbBackup *mariadbt if err != nil { return mariadbDump{}, err } - return findMariaDBDumpWith(ctx, lister, store.reader, mdbBackup, completedAt) + return findMariaDBDumpWith(ctx, lister, store.reader, mdbBackup, completedAt, false) } // findMariaDBDumpWith is findMariaDBDump reading the bucket through lister, // for a caller that reaches the bucket by another endpoint than the one -// mdbBackup names. -func findMariaDBDumpWith(ctx context.Context, lister s3ObjectLister, reader client.Reader, mdbBackup *mariadbtypes.Backup, completedAt time.Time) (mariadbDump, error) { +// mdbBackup names. withUnfinished lets runs that have not completed contest +// the dump too, as neighbourMariaDBRuns describes. +func findMariaDBDumpWith(ctx context.Context, lister s3ObjectLister, reader client.Reader, mdbBackup *mariadbtypes.Backup, completedAt time.Time, withUnfinished bool) (mariadbDump, error) { s3 := mdbBackup.Spec.Storage.S3 if s3 == nil { return mariadbDump{}, errors.New("the operator Backup does not store to S3") } - others, err := finishedMariaDBRuns(ctx, reader, mdbBackup) + others, err := neighbourMariaDBRuns(ctx, reader, mdbBackup, withUnfinished) if err != nil { return mariadbDump{}, fmt.Errorf("list operator Backups: %w", err) } @@ -136,6 +137,10 @@ type mariadbRun struct { lower, upper time.Time } +// mariadbUnfinishedRunEnd closes the window of a run that has not completed, +// which can still write its dump at any time. +var mariadbUnfinishedRunEnd = time.Date(9999, time.December, 31, 0, 0, 0, 0, time.UTC) + func newMariaDBRun(b *mariadbtypes.Backup, created, completed time.Time) mariadbRun { return mariadbRun{ name: b.Namespace + "/" + b.Name, @@ -161,8 +166,8 @@ func (r mariadbRun) placesOnEitherClock(d mariadbDump) bool { // dump in the window on one clock only may be this run's own, misplaced by // a skewed clock, which leaves any other candidate unproven. A neighbour's // dump is still taken if this run's own is missing from the listing, or -// misplaced on both clocks, and the neighbour's operator Backup is gone or -// has not completed. +// misplaced on both clocks, and the neighbour's operator Backup is gone or, +// unless others includes unfinished runs, has not completed. func matchMariaDBDump(objects []s3ObjectInfo, prefix string, own mariadbRun, others []mariadbRun) (mariadbDump, error) { var near []mariadbDump for _, obj := range objects { @@ -194,14 +199,17 @@ func matchMariaDBDump(objects []s3ObjectInfo, prefix string, own mariadbRun, oth return d, nil } -// finishedMariaDBRuns returns the runs of the other operator Backups that +// neighbourMariaDBRuns returns the runs of the other operator Backups that // write under the same bucket and prefix as b and have completed, whether -// their Job succeeded or failed. One still running is left out: its window -// has no end yet, and an operator Backup stuck short of Complete, as one -// whose Job pod never starts is, would otherwise contest every later run of -// the application for good. The endpoint is not compared, since one has -// several spellings, and a false neighbour only leaves a size unset. -func finishedMariaDBRuns(ctx context.Context, reader client.Reader, b *mariadbtypes.Backup) ([]mariadbRun, error) { +// their Job succeeded or failed. One still running is left out unless +// withUnfinished is set: its window has no end yet, and an operator Backup +// stuck short of Complete, as one whose Job pod never starts is, would +// otherwise contest every later run of the application for good. That is +// the price of deleting a dump, since a run can upload its dump and prune +// older ones through maxRetention before it completes. The endpoint is not +// compared, since one has several spellings, and a false neighbour only +// keeps a dump unattributed. +func neighbourMariaDBRuns(ctx context.Context, reader client.Reader, b *mariadbtypes.Backup, withUnfinished bool) ([]mariadbRun, error) { list := &mariadbtypes.BackupList{} if err := reader.List(ctx, list); err != nil { return nil, err @@ -218,6 +226,9 @@ func finishedMariaDBRuns(ctx context.Context, reader client.Reader, b *mariadbty } cond := apimeta.FindStatusCondition(other.Status.Conditions, mariadbtypes.ConditionTypeComplete) if cond == nil || cond.Status != metav1.ConditionTrue { + if withUnfinished { + runs = append(runs, newMariaDBRun(other, other.CreationTimestamp.Time, mariadbUnfinishedRunEnd)) + } continue } runs = append(runs, newMariaDBRun(other, other.CreationTimestamp.Time, cond.LastTransitionTime.Time)) diff --git a/internal/backupcontroller/mariadbstrategy_cleanup.go b/internal/backupcontroller/mariadbstrategy_cleanup.go index 483ded502..0a2fc1e34 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup.go @@ -117,7 +117,7 @@ func (r *BackupReconciler) cleanupMariaDBBackup(ctx context.Context, backup *bac if err != nil { return r.retryMariaDBCleanup(ctx, backup, location, fmt.Errorf("build S3 client: %w", err)) } - dump, err := findMariaDBDumpWith(s3ctx, lister, r.Client, op, backup.Spec.TakenAt.Time) + dump, err := findMariaDBDumpWith(s3ctx, lister, r.Client, op, backup.Spec.TakenAt.Time, true) if errors.Is(err, errMariaDBDumpNotFound) || errors.Is(err, errMariaDBDumpAmbiguous) { return r.releaseMariaDBCleanup(ctx, backup, location, fmt.Sprintf("the Backup records no dump and none is proven to be the one k8s.mariadb.com/Backup %s wrote (%v); the operator Backup is kept", opName, err)), nil diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index 745f7aef4..9b41f4701 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -385,6 +385,58 @@ func TestCleanupMariaDBBackup_LegacyBackupWhoseDumpANeighbourCouldOwnIsReleased( assertReleasedUntouched(t, r, rec, bucket, res, err) } +// A run started before this one completed can upload its dump and prune +// this run's through maxRetention before its own operator Backup completes. +// Its dump is then the only one in this run's window, and only its open +// window says it may be its own. +func TestCleanupMariaDBBackup_LegacyBackupWhoseDumpAnUnfinishedRunCouldOwnIsReleased(t *testing.T) { + started := cleanupCreated.Add(2 * time.Minute) + cases := map[string][]metav1.Condition{ + "running": {{ + Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionFalse, + Reason: mariadbtypes.ConditionReasonJobRunning, Message: "Running", LastTransitionTime: metav1.NewTime(started), + }}, + "not yet reconciled": nil, + } + for name, conditions := range cases { + t.Run(name, func(t *testing.T) { + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{ + {Key: dumpKey(cleanupPrefix, started.Add(90*time.Second), "gzip"), LastModified: started.Add(100 * time.Second)}, + }} + running := cleanupOperatorBackup(t) + running.Name = "op-running" + running.CreationTimestamp = metav1.NewTime(started) + running.Status.Conditions = conditions + backup := cleanupBackup("") + r, rec := cleanupReconciler(t, bucket, backup, cleanupOperatorBackup(t), running, s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), backup) + assertReleasedUntouched(t, r, rec, bucket, res, err) + }) + } +} + +// An unfinished run contests only what it could have written: a dump +// that predates it, on both clocks, is still this run's. +func TestCleanupMariaDBBackup_LegacyBackupDeletesItsDumpDespiteALaterUnfinishedRun(t *testing.T) { + key := cleanupPrefix + cleanupDumpName + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: key, LastModified: cleanupCreated.Add(2 * time.Minute)}}} + running := cleanupOperatorBackup(t) + running.Name = "op-running" + running.CreationTimestamp = metav1.NewTime(cleanupCompleted.Add(time.Hour)) + running.Status.Conditions = nil + backup := cleanupBackup("") + r, _ := cleanupReconciler(t, bucket, backup, cleanupOperatorBackup(t), running, s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), backup) + if err != nil || requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup released", res, err) + } + if want := []string{"bkt/" + key}; fmt.Sprint(bucket.removed) != fmt.Sprint(want) { + t.Errorf("removed %v, want %v", bucket.removed, want) + } +} + func TestCleanupMariaDBBackup_LegacyListingFailureRetries(t *testing.T) { bucket := &fakeS3Bucket{listErr: errors.New("connection refused")} r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) From 1003cdb1b67d7a8ce2ee8ec6f6c627dadd638185 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Sun, 8 Nov 2026 20:22:35 +0300 Subject: [PATCH 190/518] test(clickhouse): find the round-trip's Ingress Secret by its owner label alone kubectl refuses a resource name together with a label selector, and the restore asked for both with its stderr discarded, so the lookup always came back empty and the Secret the Test serves under the platform's S3 Ingress name was never deleted. The next run then stops on its own precondition that the Secret must not exist. A lookup that does fail now fails the restore, since an empty answer from a failed read is what kept this hidden. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- ...clickhouse-system-bucket-shared-state.bats | 58 ++++++++++++++++++- .../clickhouse/system-bucket-shared-state.sh | 8 ++- 2 files changed, 64 insertions(+), 2 deletions(-) diff --git a/hack/clickhouse-system-bucket-shared-state.bats b/hack/clickhouse-system-bucket-shared-state.bats index ec4900cca..876e90954 100644 --- a/hack/clickhouse-system-bucket-shared-state.bats +++ b/hack/clickhouse-system-bucket-shared-state.bats @@ -18,8 +18,12 @@ setup() { metadata: {name: "coredns", namespace: "kube-system", resourceVersion: "7"}, data: {Corefile: ".:53 {\n # === BEGIN e2e-clickhouse-system-bucket ===\n rewrite name exact s3.example.org root-ingress-controller.tenant-root.svc.cozy.local\n # === END e2e-clickhouse-system-bucket ===\n errors\n}\n"}}' \ >"$STUB/coredns.json" + # The secret lookup follows kubectl's contract: a name together with a + # selector is refused, and a selector-only lookup prints the names in + # OWNED_SECRETS, the Secrets carrying the owner label. cat >"$STUB/kubectl" <<'EOF' #!/bin/bash +echo "$*" >>"$STUB/calls" case "$*" in *"get configmap coredns -o json"*) cat "$STUB/coredns.json" ;; replace*) @@ -29,7 +33,29 @@ case "$*" in exit 1 fi ;; - *"get secret"*) ;; + *"get secret"*) + # Positionals are "get", "secret" and then any names. + words=0 selector= + while [ $# -gt 0 ]; do + case "$1" in + -l|--selector) selector=$2; shift 2 ;; + -n|--namespace|-o|--output) shift 2 ;; + -*) shift ;; + *) words=$((words + 1)); shift ;; + esac + done + if [ -n "${LOOKUP_FAILS:-}" ]; then + echo 'Unable to connect to the server: dial tcp: i/o timeout' >&2 + exit 1 + fi + if [ -n "$selector" ] && [ "$words" -gt 2 ]; then + echo 'error: name cannot be provided when a selector is specified' >&2 + exit 1 + fi + [ "$selector" = cozystack.io/e2e-owner=clickhouse-system-bucket ] || exit 0 + for s in ${OWNED_SECRETS:-}; do echo "secret/$s"; done + ;; + *"delete secret "*) ;; *) echo "unexpected kubectl $*" >&2; exit 2 ;; esac EOF @@ -74,6 +100,36 @@ restore() { [ "$(jq -r '.metadata.resourceVersion' "$STUB/replaced.json")" = 7 ] } +@test "the restore deletes the Ingress TLS Secret the Test created" { + OWNED_SECRETS=seaweedfs-s3-ingress-tls restore + grep -qE '^-n tenant-root delete secret seaweedfs-s3-ingress-tls( |$)' "$STUB/calls" || { + echo "restore_shared_state did not delete the Secret carrying the owner label; kubectl calls:" >&2 + cat "$STUB/calls" >&2 + false + } +} + +@test "the restore leaves an Ingress TLS Secret without the owner label alone" { + for owned in '' e2e-other seaweedfs-s3-ingress-tls-old; do + : >"$STUB/calls" + OWNED_SECRETS=$owned restore + if grep -q 'delete secret' "$STUB/calls"; then + echo "restore_shared_state deleted a Secret although the labelled ones are '$owned'; kubectl calls:" >&2 + cat "$STUB/calls" >&2 + false + fi + done +} + +@test "a Secret lookup that fails fails the restore" { + err=$(mktemp) + if LOOKUP_FAILS=1 restore 2>"$err"; then + echo "restore_shared_state returned 0 although it could not tell whether the Secret is left over" >&2 + cat "$err" >&2 + false + fi +} + @test "the run step's finally restores the shared objects" { # When the step ceiling fires, Chainsaw kills the shell and the EXIT trap # of system-bucket-roundtrip.sh never runs; the step's finally still does. diff --git a/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh b/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh index 8275e8c13..acb577d27 100644 --- a/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh +++ b/hack/e2e-chainsaw/clickhouse/system-bucket-shared-state.sh @@ -47,7 +47,13 @@ remove_dns_block() { restore_shared_state() { local rc=0 remove_dns_block || rc=1 - if kubectl -n "$NAMESPACE" get secret "$INGRESS_SECRET" -l "$OWNER_LABEL" -o name 2>/dev/null | grep -q .; then + # kubectl refuses a name together with a selector, so the label is + # looked up on its own and the name matched in what comes back. + local owned + if ! owned=$(kubectl -n "$NAMESPACE" get secret -l "$OWNER_LABEL" -o name); then + log_error "could not list the Secrets labelled ${OWNER_LABEL} in ${NAMESPACE}; delete ${INGRESS_SECRET} by hand if it carries that label" + rc=1 + elif printf '%s\n' "$owned" | grep -qxF "secret/$INGRESS_SECRET"; then kubectl -n "$NAMESPACE" delete secret "$INGRESS_SECRET" --wait=true --timeout=60s || rc=1 fi return "$rc" From 6e23bd5629da28edf63f60666fe2243c285b4ec1 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Sun, 8 Nov 2026 20:22:43 +0300 Subject: [PATCH 191/518] test(clickhouse): delete the round-trip's Certificate before its Secret cert-manager issues the Secret of a Certificate that still exists again as soon as it is deleted, and the Certificate is only removed by Chainsaw's own cleanup after the step's finally has run. Deleting the Secret first therefore left a freshly issued copy behind in tenant-root, carrying no owner reference to take it down with the Certificate. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- ...clickhouse-system-bucket-shared-state.bats | 23 +++++++++++++++++++ .../clickhouse/chainsaw-test.yaml | 7 ++++-- 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/hack/clickhouse-system-bucket-shared-state.bats b/hack/clickhouse-system-bucket-shared-state.bats index 876e90954..d03434531 100644 --- a/hack/clickhouse-system-bucket-shared-state.bats +++ b/hack/clickhouse-system-bucket-shared-state.bats @@ -179,3 +179,26 @@ STUBEOF } done } + +# cert-manager issues the Secret of a Certificate that still exists again as +# soon as it is deleted, so the Certificate has to go first or its Secret +# outlives the run. +@test "the round-trip cleanup deletes the Certificate before its Secret" { + script=$(yq 'select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[] + | select(.name == "run") | .finally[] + | select(.description == "system-bucket round-trip cleanup") | .script.content' \ + hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml) + [ -n "$script" ] || { echo "the run step has no system-bucket round-trip cleanup" >&2; false; } + cat >"$STUB/kubectl" <<'STUBEOF' +#!/bin/bash +echo "$*" >>"$STUB/calls" +STUBEOF + PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$script" + cert=$(grep -nE '^-n tenant-root delete certificate e2e-system-bucket-s3( |$)' "$STUB/calls" | cut -d: -f1) + secret=$(grep -nE '^-n tenant-root delete secret e2e-system-bucket-s3-tls( |$)' "$STUB/calls" | cut -d: -f1) + [ -n "$cert" ] && [ -n "$secret" ] && [ "$cert" -lt "$secret" ] || { + echo "the cleanup does not delete Certificate e2e-system-bucket-s3 before Secret e2e-system-bucket-s3-tls; kubectl calls:" >&2 + cat "$STUB/calls" >&2 + false + } +} diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index a4c4aef40..774941ced 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -559,10 +559,12 @@ spec: ' # Chainsaw did not apply these: the jobs come from the script and the # Backup from the controller, and deleting a BackupJob reclaims no Backup. - # The Certificate's Secret outlives the Certificate Chainsaw deletes. + # The Certificate's Secret outlives the Certificate, and cert-manager + # issues it again while the Certificate exists, so the Certificate goes + # first rather than at Chainsaw's cleanup after this finally. - description: system-bucket round-trip cleanup script: - timeout: 3m + timeout: 4m content: | set -eu kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s @@ -570,4 +572,5 @@ spec: # The driver names the Backup after the BackupJob; by name it is # found even when status.backupRef was never written. kubectl -n "$NAMESPACE" delete backup sysbucket-backup --ignore-not-found --wait=true --timeout=60s + kubectl -n "$NAMESPACE" delete certificate e2e-system-bucket-s3 --ignore-not-found --wait=true --timeout=30s kubectl -n "$NAMESPACE" delete secret e2e-system-bucket-s3-tls --ignore-not-found --wait=true --timeout=30s From 28f36134505e75d64adc5b3636fda198630412e9 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Sun, 8 Nov 2026 20:28:53 +0300 Subject: [PATCH 192/518] test(clickhouse): delete the round-trip's TLS Secret in its own step's cleanup Chainsaw applied the Certificate, so the scripted delete of it in the run step's finally was hand-rolled teardown of a resource Chainsaw already reclaims. Chainsaw deletes a step's applied resources, waiting for them to go, before it runs that step's cleanup operations, which puts the Secret's delete after the Certificate's without any ordering written into a script. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- ...clickhouse-system-bucket-shared-state.bats | 36 +++++++++---------- .../clickhouse/chainsaw-test.yaml | 17 +++++---- 2 files changed, 29 insertions(+), 24 deletions(-) diff --git a/hack/clickhouse-system-bucket-shared-state.bats b/hack/clickhouse-system-bucket-shared-state.bats index d03434531..574eba84f 100644 --- a/hack/clickhouse-system-bucket-shared-state.bats +++ b/hack/clickhouse-system-bucket-shared-state.bats @@ -181,24 +181,24 @@ STUBEOF } # cert-manager issues the Secret of a Certificate that still exists again as -# soon as it is deleted, so the Certificate has to go first or its Secret -# outlives the run. -@test "the round-trip cleanup deletes the Certificate before its Secret" { - script=$(yq 'select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[] - | select(.name == "run") | .finally[] - | select(.description == "system-bucket round-trip cleanup") | .script.content' \ - hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml) - [ -n "$script" ] || { echo "the run step has no system-bucket round-trip cleanup" >&2; false; } - cat >"$STUB/kubectl" <<'STUBEOF' -#!/bin/bash -echo "$*" >>"$STUB/calls" -STUBEOF - PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$script" - cert=$(grep -nE '^-n tenant-root delete certificate e2e-system-bucket-s3( |$)' "$STUB/calls" | cut -d: -f1) - secret=$(grep -nE '^-n tenant-root delete secret e2e-system-bucket-s3-tls( |$)' "$STUB/calls" | cut -d: -f1) - [ -n "$cert" ] && [ -n "$secret" ] && [ "$cert" -lt "$secret" ] || { - echo "the cleanup does not delete Certificate e2e-system-bucket-s3 before Secret e2e-system-bucket-s3-tls; kubectl calls:" >&2 - cat "$STUB/calls" >&2 +# soon as it is deleted. Chainsaw deletes the Certificate it applied, and waits +# for it, before it runs the cleanup ops of the same step, so the Secret is +# deleted there and nowhere earlier. +@test "the Certificate's Secret is deleted after Chainsaw deletes the Certificate" { + test='select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[]' + suite=hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml + applies=$(yq "$test | select(.name == \"create-certificate-and-probe\") | .try[] | select(has(\"apply\")) | .apply.file" "$suite") + [ "$applies" = system-bucket-tls.yaml ] || { + echo "step create-certificate-and-probe no longer applies system-bucket-tls.yaml alone: $applies" >&2 false } + deletes=$(yq -o=json -I=0 "$test | select(.name == \"create-certificate-and-probe\") | .cleanup" "$suite") + [ "$deletes" = '[{"delete":{"ref":{"apiVersion":"v1","kind":"Secret","name":"e2e-system-bucket-s3-tls"}}}]' ] || { + echo "the cleanup of step create-certificate-and-probe is not the one delete of Secret e2e-system-bucket-s3-tls: $deletes" >&2 + false + } + if yq "$test | (.try[], .finally[]?) | .script.content // \"\"" "$suite" | grep -qE 'delete +(secret +e2e-system-bucket-s3-tls|certificate +e2e-system-bucket-s3)( |$)'; then + echo "a step script deletes the Certificate or its Secret while Chainsaw still holds the Certificate" >&2 + false + fi } diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index 774941ced..b71728720 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -485,6 +485,16 @@ spec: status: (conditions[?type == 'Ready']): - status: "True" + # The Certificate's Secret carries no owner reference, so it outlives the + # Certificate, and cert-manager issues it again while the Certificate + # exists. Chainsaw runs a step's cleanup ops after this step's cleaner has + # deleted the Certificate and waited for it. + cleanup: + - delete: + ref: + apiVersion: v1 + kind: Secret + name: e2e-system-bucket-s3-tls - name: create-clickhouse try: @@ -559,12 +569,9 @@ spec: ' # Chainsaw did not apply these: the jobs come from the script and the # Backup from the controller, and deleting a BackupJob reclaims no Backup. - # The Certificate's Secret outlives the Certificate, and cert-manager - # issues it again while the Certificate exists, so the Certificate goes - # first rather than at Chainsaw's cleanup after this finally. - description: system-bucket round-trip cleanup script: - timeout: 4m + timeout: 3m content: | set -eu kubectl -n "$NAMESPACE" delete restorejob sysbucket-restore --ignore-not-found --wait=true --timeout=30s @@ -572,5 +579,3 @@ spec: # The driver names the Backup after the BackupJob; by name it is # found even when status.backupRef was never written. kubectl -n "$NAMESPACE" delete backup sysbucket-backup --ignore-not-found --wait=true --timeout=60s - kubectl -n "$NAMESPACE" delete certificate e2e-system-bucket-s3 --ignore-not-found --wait=true --timeout=30s - kubectl -n "$NAMESPACE" delete secret e2e-system-bucket-s3-tls --ignore-not-found --wait=true --timeout=30s From d20b26d2ed9cbaabcb610825f085caeef8b92887 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Sun, 8 Nov 2026 20:34:28 +0300 Subject: [PATCH 193/518] test(clickhouse): pin that a refused Secret delete fails the round-trip restore The owned Ingress Secret's delete only became reachable once its lookup stopped asking for a name and a selector together, and nothing failed the stub's delete, so dropping the delete's status from the restore's result passed every test. The CoreDNS half of the restore already has this guarantee pinned. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/clickhouse-system-bucket-shared-state.bats | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/hack/clickhouse-system-bucket-shared-state.bats b/hack/clickhouse-system-bucket-shared-state.bats index 574eba84f..d09fa0260 100644 --- a/hack/clickhouse-system-bucket-shared-state.bats +++ b/hack/clickhouse-system-bucket-shared-state.bats @@ -55,7 +55,12 @@ case "$*" in [ "$selector" = cozystack.io/e2e-owner=clickhouse-system-bucket ] || exit 0 for s in ${OWNED_SECRETS:-}; do echo "secret/$s"; done ;; - *"delete secret "*) ;; + *"delete secret "*) + if [ -n "${DELETE_FAILS:-}" ]; then + echo 'Error from server (Forbidden): secrets "seaweedfs-s3-ingress-tls" is forbidden' >&2 + exit 1 + fi + ;; *) echo "unexpected kubectl $*" >&2; exit 2 ;; esac EOF @@ -121,6 +126,15 @@ restore() { done } +@test "a Secret delete the API server refuses fails the restore" { + err=$(mktemp) + if OWNED_SECRETS=seaweedfs-s3-ingress-tls DELETE_FAILS=1 restore 2>"$err"; then + echo "restore_shared_state returned 0 although kubectl delete secret was refused" >&2 + cat "$err" >&2 + false + fi +} + @test "a Secret lookup that fails fails the restore" { err=$(mktemp) if LOOKUP_FAILS=1 restore 2>"$err"; then From 4345a346ccdce59245777d93c0451a96aa1fd33f Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Sun, 8 Nov 2026 20:37:38 +0300 Subject: [PATCH 194/518] test(clickhouse): check why the round-trip restore failed, not only that it did The refused-delete and failed-lookup cases asserted a non-zero status alone, which any earlier failure in the restore also produces, so a stub drifting from the script would have kept them green for the wrong reason. They now also require the delete to have been attempted and the lookup failure to be named. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/clickhouse-system-bucket-shared-state.bats | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/hack/clickhouse-system-bucket-shared-state.bats b/hack/clickhouse-system-bucket-shared-state.bats index d09fa0260..10a869af6 100644 --- a/hack/clickhouse-system-bucket-shared-state.bats +++ b/hack/clickhouse-system-bucket-shared-state.bats @@ -133,6 +133,11 @@ restore() { cat "$err" >&2 false fi + grep -qE '^-n tenant-root delete secret seaweedfs-s3-ingress-tls( |$)' "$STUB/calls" || { + echo "restore_shared_state failed before it reached the Secret delete; kubectl calls:" >&2 + cat "$STUB/calls" "$err" >&2 + false + } } @test "a Secret lookup that fails fails the restore" { @@ -142,6 +147,11 @@ restore() { cat "$err" >&2 false fi + grep -q 'could not list the Secrets labelled' "$err" || { + echo "restore_shared_state failed without naming the Secret lookup:" >&2 + cat "$err" >&2 + false + } } @test "the run step's finally restores the shared objects" { From 55d6c52fabc8e7ac080674757e97f7881368af14 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Wed, 21 Oct 2026 12:01:16 +0200 Subject: [PATCH 195/518] fix(kubernetes, monitoring): send tenant cluster metrics to the tenant's vminsert The monitoring-agents chart reads vmagent.remoteWrite.urls as a list and takes the tenant label from global.target, while the kubernetes app still passed a single remoteWrite.url and vmagent.externalLabels.tenant. Both keys were dropped in silence, so the vmagent of every tenant cluster fell back to the chart defaults: it wrote to vminsert services in cozy-monitoring that do not exist inside the tenant cluster, queued the failed writes on the node disk, and labelled every series tenant="cozy-monitoring". No tenant cluster metric reached the tenant's Monitoring. The tenant label has to name the tenant that owns the cluster, which is not always the tenant whose Monitoring receives the data: a tenant without its own Monitoring inherits a parent's. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../helmreleases/monitoring-agents.yaml | 3 +- .../tests/monitoring_agents_test.yaml | 42 ++++++++++++ .../monitoring-agents/templates/vmagent.yaml | 2 +- .../tests/vmagent_remote_write_test.yaml | 65 +++++++++++++++++++ 4 files changed, 110 insertions(+), 2 deletions(-) create mode 100644 packages/apps/kubernetes/tests/monitoring_agents_test.yaml create mode 100644 packages/system/monitoring-agents/tests/vmagent_remote_write_test.yaml diff --git a/packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml b/packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml index 2ec0bc3ca..7e5c45e64 100644 --- a/packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml +++ b/packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml @@ -45,7 +45,8 @@ spec: cluster: {{ .Release.Name }} tenant: {{ .Release.Namespace }} remoteWrite: - url: http://vminsert-shortterm.{{ $targetTenant }}.svc.{{ $clusterDomain }}:8480/insert/0/prometheus + urls: + - http://vminsert-shortterm.{{ $targetTenant }}.svc.{{ $clusterDomain }}:8480/insert/0/prometheus fluent-bit: readinessProbe: httpGet: diff --git a/packages/apps/kubernetes/tests/monitoring_agents_test.yaml b/packages/apps/kubernetes/tests/monitoring_agents_test.yaml new file mode 100644 index 000000000..1dc21181a --- /dev/null +++ b/packages/apps/kubernetes/tests/monitoring_agents_test.yaml @@ -0,0 +1,42 @@ +suite: monitoring-agents HelmRelease values match the monitoring-agents chart + +# The values below are consumed by packages/system/monitoring-agents, which +# reads vmagent.remoteWrite.urls as a list and the tenant label from +# vmagent.externalLabels.tenant. A key the chart does not read is dropped in +# silence, and the chart then falls back to cozy-monitoring defaults that do +# not resolve inside a tenant cluster. Its counterpart is +# packages/system/monitoring-agents/tests/vmagent_remote_write_test.yaml. +templates: + - templates/helmreleases/monitoring-agents.yaml + +values: + - values/common.yaml + +release: + name: kubernetes-clusterprod + namespace: tenant-a + +set: + addons.monitoringAgents.enabled: true + _namespace.monitoring: tenant-root + _cluster: + cluster-domain: cozy.local + +tests: + - it: sends metrics to the shortterm vminsert of the monitoring tenant as a urls list + asserts: + - equal: + path: spec.values.vmagent.remoteWrite + value: + urls: + - http://vminsert-shortterm.tenant-root.svc.cozy.local:8480/insert/0/prometheus + + - it: labels series with the tenant that owns the cluster, not the monitoring tenant + asserts: + - equal: + path: spec.values.vmagent.externalLabels + value: + cluster: kubernetes-clusterprod + tenant: tenant-a + - notExists: + path: spec.values.global diff --git a/packages/system/monitoring-agents/templates/vmagent.yaml b/packages/system/monitoring-agents/templates/vmagent.yaml index 00ab63c10..7a87fa5b5 100644 --- a/packages/system/monitoring-agents/templates/vmagent.yaml +++ b/packages/system/monitoring-agents/templates/vmagent.yaml @@ -10,7 +10,7 @@ spec: shardCount: 1 externalLabels: cluster: {{ .Values.vmagent.externalLabels.cluster }} - tenant: {{ .Values.global.target }} + tenant: {{ .Values.vmagent.externalLabels.tenant | default .Values.global.target | quote }} extraArgs: {{- toYaml (deepCopy .Values.vmagent.extraArgs | mergeOverwrite (fromYaml (include "monitoring-agents.vmagent.defaultExtraArgs" .))) | nindent 4 }} remoteWrite: diff --git a/packages/system/monitoring-agents/tests/vmagent_remote_write_test.yaml b/packages/system/monitoring-agents/tests/vmagent_remote_write_test.yaml new file mode 100644 index 000000000..894a1c053 --- /dev/null +++ b/packages/system/monitoring-agents/tests/vmagent_remote_write_test.yaml @@ -0,0 +1,65 @@ +suite: vmagent remote write targets and tenant label + +# The root bundle sets only global.target, and the chart defaults derive both +# the remote write URLs and the tenant label from it. A tenant Kubernetes +# cluster gets this chart through packages/apps/kubernetes instead, which +# passes vmagent.remoteWrite.urls and vmagent.externalLabels.tenant and leaves +# global.target at its default. If either value is dropped, vmagent writes to +# cozy-monitoring services that do not exist in the tenant cluster, or labels +# every series tenant="cozy-monitoring". +templates: + - templates/vmagent.yaml +release: + name: cozy-monitoring-agents + namespace: cozy-monitoring +tests: + - it: root install writes to both stores of global.target and labels series with it + set: + global.target: tenant-root + asserts: + - equal: + path: spec.remoteWrite + value: + - url: http://vminsert-shortterm.tenant-root.svc:8480/insert/0/prometheus + - url: http://vminsert-longterm.tenant-root.svc:8480/insert/0/prometheus + - equal: + path: spec.externalLabels.tenant + value: tenant-root + + - it: tenant cluster writes only to the URLs its parent app passes + set: + vmagent: + externalLabels: + cluster: kubernetes-clusterprod + tenant: tenant-a + remoteWrite: + urls: + - http://vminsert-shortterm.tenant-a.svc.cozy.local:8480/insert/0/prometheus + asserts: + - equal: + path: spec.remoteWrite + value: + - url: http://vminsert-shortterm.tenant-a.svc.cozy.local:8480/insert/0/prometheus + + - it: tenant cluster labels series with the owning tenant, not global.target + set: + vmagent: + externalLabels: + cluster: kubernetes-clusterprod + tenant: tenant-a + asserts: + - equal: + path: spec.externalLabels + value: + cluster: kubernetes-clusterprod + tenant: tenant-a + + - it: tenant label stays a string when the value reads as a YAML scalar + set: + vmagent: + externalLabels: + tenant: "true" + asserts: + - equal: + path: spec.externalLabels.tenant + value: "true" From fd9245f00804a1884a50883000c42ff2993b3191 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Sun, 8 Nov 2026 19:34:49 +0100 Subject: [PATCH 196/518] test(lineage): reject CAPK labels on objects that are not KubeVirt VMs The bridge from a VirtualMachine to its KubevirtMachine is guarded by the object's API version and kind, but no test exercised that guard: weakening it to match on either field alone still passed, and let a labelled kubevirt.io Cluster or a VirtualMachine of another group resolve to the Kubernetes application. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- pkg/lineage/capk_test.go | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/pkg/lineage/capk_test.go b/pkg/lineage/capk_test.go index 606647eed..0c17300d2 100644 --- a/pkg/lineage/capk_test.go +++ b/pkg/lineage/capk_test.go @@ -81,6 +81,34 @@ func TestWalkResolvesCAPKNodeVMToKubernetesApp(t *testing.T) { } } +func TestWalkFollowsCAPKLabelsOnlyFromKubeVirtVMs(t *testing.T) { + // The CAPK labels name a valid KubevirtMachine in the same namespace in + // every case: only the kind and API group of the labelled object differ, so + // each case fails solely on the guard that restricts the bridge to + // kubevirt.io/v1 VirtualMachines. + capkLabels := map[string]string{ + KubevirtMachineNameLabel: "kubernetes-cluster-a-md0-x-y", + KubevirtMachineNamespaceLabel: capkNamespace, + } + for _, tc := range []struct { + name, apiVersion, kind string + }{ + {name: "other kind in the kubevirt.io group", apiVersion: VMAPIVersion, kind: "Cluster"}, + {name: "VirtualMachineInstance", apiVersion: VMAPIVersion, kind: "VirtualMachineInstance"}, + {name: "VirtualMachine of another group", apiVersion: "example.com/v1", kind: VMKind}, + {name: "VirtualMachine of another kubevirt.io version", apiVersion: "kubevirt.io/v1alpha3", kind: VMKind}, + } { + t.Run(tc.name, func(t *testing.T) { + _, objs, mapper := capkNodeChain(nil) + impostor := capkObject(tc.apiVersion, tc.kind, "impostor", capkLabels, nil) + dyn := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme(), objs...) + if got := WalkOwnershipGraph(context.Background(), dyn, mapper, kubernetesAppMapper{}, impostor); len(got) != 0 { + t.Fatalf("WalkOwnershipGraph(%s %s) = %+v, want no ancestors", tc.apiVersion, tc.kind, got) + } + }) + } +} + func TestWalkIgnoresKubevirtMachineOfAnotherNamespace(t *testing.T) { // A VM pointing at a KubevirtMachine outside its own namespace (an external // infrastructure cluster, or a forged label) must not be resolved through it. From a7f84242d1b82398f5dd478452bc8e0ae8f891cd Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Sun, 8 Nov 2026 19:36:44 +0100 Subject: [PATCH 197/518] fix(redis): keep three Sentinels for a zero or negative replica count The operator runs three Redis pods when the count is zero or negative, but the chart treated any count up to 1 as the single-pod case. Such a cluster got one Sentinel and no Sentinel PDB, while three Redis pods ran under it. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- packages/apps/redis/templates/_resources.tpl | 2 +- .../redis/tests/sentinel_pdb_hook_test.yaml | 20 +++++++++++++ .../redis/tests/sentinel_replicas_test.yaml | 28 +++++++++++++++++++ 3 files changed, 49 insertions(+), 1 deletion(-) diff --git a/packages/apps/redis/templates/_resources.tpl b/packages/apps/redis/templates/_resources.tpl index 98119dbb2..45399c958 100644 --- a/packages/apps/redis/templates/_resources.tpl +++ b/packages/apps/redis/templates/_resources.tpl @@ -49,5 +49,5 @@ These presets are for basic testing and not meant to be used in production {{- end -}} {{- define "redis.sentinelReplicas" -}} -{{- ternary 1 3 (le (int .Values.replicas) 1) -}} +{{- ternary 1 3 (eq (int .Values.replicas) 1) -}} {{- end -}} diff --git a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml index bc6aea706..5676f8d31 100644 --- a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml +++ b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml @@ -77,6 +77,26 @@ tests: - hasDocuments: count: 0 + # The operator runs three Redis pods for a zero or negative count, so their + # Sentinel PDB stays. + - it: renders nothing for a zero replica count + release: + upgrade: true + set: + replicas: 0 + asserts: + - hasDocuments: + count: 0 + + - it: renders nothing for a negative replica count + release: + upgrade: true + set: + replicas: -1 + asserts: + - hasDocuments: + count: 0 + # The pods' job-name label takes 63 characters at most; a 53-character # release name plus -sentinel-pdb would pass it and Kubernetes would refuse # the Job, leaving the stale PDB in place. diff --git a/packages/apps/redis/tests/sentinel_replicas_test.yaml b/packages/apps/redis/tests/sentinel_replicas_test.yaml index 012f58247..fc60d52b0 100644 --- a/packages/apps/redis/tests/sentinel_replicas_test.yaml +++ b/packages/apps/redis/tests/sentinel_replicas_test.yaml @@ -54,6 +54,34 @@ tests: - notExists: path: spec.sentinel.disablePodDisruptionBudget + # The operator turns a zero or negative count into three Redis pods, so these + # keep three Sentinels and their PDB rather than the single-pod case. + - it: "keeps three Sentinels and their PDB for a zero replica count" + set: + replicas: 0 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 3 + - notExists: + path: spec.sentinel.disablePodDisruptionBudget + + - it: "keeps three Sentinels and their PDB for a negative replica count" + set: + replicas: -1 + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.sentinel.replicas + value: 3 + - notExists: + path: spec.sentinel.disablePodDisruptionBudget + - it: "keeps three Sentinels for three Redis pods" set: replicas: 3 From e92bc8664b84d9dd605c4f2321d42e493b792f95 Mon Sep 17 00:00:00 2001 From: Marian Koreniuk Date: Thu, 5 Nov 2026 22:57:00 +0100 Subject: [PATCH 198/518] fix(tests): keep two unit tests green after the release image restamp The release build rewrites packages/*/values.yaml: the kube-ovn image tag becomes the Cozystack version and the traces collector becomes the first-party otelcol-tracing build. Two unit tests read exactly those fields, so every PR against release-1.7 has failed Unit & controller tests since the first rc, and release-1.8 would repeat it. Same change as #5179 on release-1.7. Assisted-by: LLM Signed-off-by: Marian Koreniuk --- hack/cni-plugins-staging-contract.bats | 12 +++++++----- .../monitoring/tests/vtraces_collector_test.yaml | 9 ++++++++- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/hack/cni-plugins-staging-contract.bats b/hack/cni-plugins-staging-contract.bats index 2f3acb2f1..ce63e45d2 100644 --- a/hack/cni-plugins-staging-contract.bats +++ b/hack/cni-plugins-staging-contract.bats @@ -22,7 +22,10 @@ extract_has() { } DOCKERFILE="$PWD/packages/system/multus/images/multus-cni/Dockerfile" -KUBEOVN_VALUES="$PWD/packages/system/kubeovn/values.yaml" +# The chart's appVersion, not the values.yaml image tag: a release build +# restamps that tag to the Cozystack version (v1.7.0), the chart keeps the +# kube-ovn release it vendors. +KUBEOVN_CHART="$PWD/packages/system/kubeovn/charts/kube-ovn/Chart.yaml" DOC="$PWD/packages/system/multus/README.md" PLATFORM_VALUES="$PWD/packages/core/platform/values.yaml" TEMPLATE="$PWD/packages/system/multus/templates/multus-daemonset-thick.yml" @@ -347,13 +350,12 @@ PATCHFILE="$PWD/packages/system/multus/patches/customize-deployment.patch" checked=$(sed -n 's|^# kube-ovn-cni-plugins-checked-against: *\([^ ]*\) *$|\1|p' "$DOCKERFILE") [ -n "$checked" ] - vendored=$(yq -r '.global.images.kubeovn.tag' "$KUBEOVN_VALUES" | sed 's|@.*||') - [ -n "$vendored" ] - [ "$vendored" != "null" ] + vendored=$(awk '$1 == "appVersion:" {gsub(/"/, "", $2); print "v" $2}' "$KUBEOVN_CHART") + [ "$vendored" != "v" ] if [ "$checked" != "$vendored" ]; then echo "the multus CNI plugin pin was checked against kube-ovn $checked," >&2 - echo "but packages/system/kubeovn/values.yaml now vendors $vendored." >&2 + echo "but packages/system/kubeovn/charts/kube-ovn/Chart.yaml now vendors $vendored." >&2 echo >&2 echo "Read CNI_PLUGINS_VERSION in kube-ovn $vendored's base image" >&2 echo "(dist/images/Dockerfile.base). If it still matches the" >&2 diff --git a/packages/system/monitoring/tests/vtraces_collector_test.yaml b/packages/system/monitoring/tests/vtraces_collector_test.yaml index b450bf520..eaa5921cb 100644 --- a/packages/system/monitoring/tests/vtraces_collector_test.yaml +++ b/packages/system/monitoring/tests/vtraces_collector_test.yaml @@ -473,11 +473,18 @@ tests: path: spec.template.spec.containers[0].securityContext.capabilities.drop content: ALL + # The image name is not asserted: a release build swaps the upstream + # collector for the first-party otelcol-tracing one and restamps the tag. + # What must hold is that the value goes through cozy-lib.image (the cluster + # images-registry lands in front of it) and stays digest-pinned. - it: pins the collector image through the cozy-lib.image helper + set: + _cluster: + images-registry: mirror.example.com documentSelector: path: kind value: Deployment asserts: - matchRegex: path: spec.template.spec.containers[0].image - pattern: "opentelemetry-collector-contrib:.*@sha256:" + pattern: '^mirror\.example\.com/.+@sha256:[0-9a-f]{64}$' From 525e4161341a4fee333576d3f3adbb9747452bda Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Sun, 8 Nov 2026 19:39:07 +0100 Subject: [PATCH 199/518] fix(kubernetes-nodes): let ComputePlane pools set maxUnavailable and bound it to int32 The ComputePlane wrapper forwards only the pool fields it names, and maxUnavailable was not one of them. A ComputePlane pool without room for one extra VM therefore had no way back to the drain-first rollout, and its rollouts stalled. CAPI decodes maxUnavailable as an int32, so a value above 2147483647 passed the chart schema and was then refused by the MachineDeployment decoder. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- api/apps/v1alpha1/kubernetesnodes/types.go | 1 + .../tests/rollout_strategy_test.yaml | 20 +++++++++++ .../apps/kubernetes-nodes/values.schema.json | 1 + packages/apps/kubernetes-nodes/values.yaml | 1 + packages/extra/computeplane/README.md | 1 + .../extra/computeplane/templates/cluster.yaml | 2 +- .../computeplane/tests/cluster_test.yaml | 35 +++++++++++++++++++ .../extra/computeplane/values.schema.json | 6 ++++ packages/extra/computeplane/values.yaml | 3 ++ .../computeplane-rd/cozyrds/computeplane.yaml | 2 +- .../cozyrds/kubernetes-nodes.yaml | 2 +- 11 files changed, 71 insertions(+), 3 deletions(-) diff --git a/api/apps/v1alpha1/kubernetesnodes/types.go b/api/apps/v1alpha1/kubernetesnodes/types.go index 81cca0bf9..592cb5f4f 100644 --- a/api/apps/v1alpha1/kubernetesnodes/types.go +++ b/api/apps/v1alpha1/kubernetesnodes/types.go @@ -53,6 +53,7 @@ type ConfigSpec struct { Gpus []GPU `json:"gpus,omitempty"` // Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. // +kubebuilder:validation:Minimum=0 + // +kubebuilder:validation:Maximum=2.147483647e+09 MaxUnavailable *int `json:"maxUnavailable,omitempty"` // Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. KernelModules []KernelModule `json:"kernelModules,omitempty"` diff --git a/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml b/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml index 210afb068..17e732d33 100644 --- a/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/rollout_strategy_test.yaml @@ -114,3 +114,23 @@ tests: maxUnavailable: -1 asserts: - failedTemplate: {} + + # CAPI decodes the field as an int32, so a larger value would render and + # then be refused by the MachineDeployment decoder. + - it: refuses a maxUnavailable above the int32 range (schema maximum) + set: + maxUnavailable: 2147483648 + asserts: + - failedTemplate: + errorPattern: "maxUnavailable" + + - it: accepts the int32 maximum + set: + maxUnavailable: 2147483647 + documentSelector: + path: kind + value: MachineDeployment + asserts: + - equal: + path: spec.strategy.rollingUpdate.maxUnavailable + value: 2147483647 diff --git a/packages/apps/kubernetes-nodes/values.schema.json b/packages/apps/kubernetes-nodes/values.schema.json index 53de10b95..8f1068dce 100644 --- a/packages/apps/kubernetes-nodes/values.schema.json +++ b/packages/apps/kubernetes-nodes/values.schema.json @@ -149,6 +149,7 @@ "maxUnavailable": { "description": "Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.", "type": "integer", + "maximum": 2147483647, "minimum": 0 }, "kernelModules": { diff --git a/packages/apps/kubernetes-nodes/values.yaml b/packages/apps/kubernetes-nodes/values.yaml index f5d6c7dcf..9c0ab999d 100644 --- a/packages/apps/kubernetes-nodes/values.yaml +++ b/packages/apps/kubernetes-nodes/values.yaml @@ -79,6 +79,7 @@ gpus: [] ## @param {*int} [maxUnavailable] - Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever. ## @minimum 0 +## @maximum 2147483647 ## @param {[]KernelModule} [kernelModules] - Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. diff --git a/packages/extra/computeplane/README.md b/packages/extra/computeplane/README.md index fe6c1af67..15779168f 100644 --- a/packages/extra/computeplane/README.md +++ b/packages/extra/computeplane/README.md @@ -59,4 +59,5 @@ See the full design in | `nodeGroups[name].kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` | | `nodeGroups[name].kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` | | `nodeGroups[name].kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` | +| `nodeGroups[name].maxUnavailable` | Workers a rollout may remove before their replacement is up. Unset, the `kubernetes-nodes` pool chart uses 0, so each replacement joins before an old worker is drained (1 on a pool with `gpus`). Set 1 on a pool without room for one extra VM, whose rollout would otherwise stall with the new Machine Pending. | `*int` | `null` | diff --git a/packages/extra/computeplane/templates/cluster.yaml b/packages/extra/computeplane/templates/cluster.yaml index 82ecdc452..9c598d1af 100644 --- a/packages/extra/computeplane/templates/cluster.yaml +++ b/packages/extra/computeplane/templates/cluster.yaml @@ -211,7 +211,7 @@ spec: clone or repoint itself at another Image Factory, both of which are operator decisions about the platform's own images. Keep this list in sync with the kubernetes-nodes values schema's pool-shape params. */}} - {{- with pick $group "minReplicas" "maxReplicas" "instanceType" "diskSize" "storageClass" "roles" "resources" "gpus" "kubelet" "logSerialConsole" "maxUnhealthy" "nodeStartupTimeout" "readyUnknownTimeout" "readyFalseTimeout" "podCpuLimit" "podCpuRequest" "maxNodeProvisionTime" }} + {{- with pick $group "minReplicas" "maxReplicas" "instanceType" "diskSize" "storageClass" "roles" "resources" "gpus" "kubelet" "logSerialConsole" "maxUnhealthy" "nodeStartupTimeout" "readyUnknownTimeout" "readyFalseTimeout" "podCpuLimit" "podCpuRequest" "maxNodeProvisionTime" "maxUnavailable" }} {{- toYaml . | nindent 4 }} {{- end }} valuesFrom: diff --git a/packages/extra/computeplane/tests/cluster_test.yaml b/packages/extra/computeplane/tests/cluster_test.yaml index 9d1a6768f..d56e21d30 100644 --- a/packages/extra/computeplane/tests/cluster_test.yaml +++ b/packages/extra/computeplane/tests/cluster_test.yaml @@ -280,6 +280,41 @@ tests: - notExists: path: spec.values.nameOverride + # kubernetes-nodes drains a worker only once its replacement is up; a pool + # without room for one extra VM needs maxUnavailable 1 to finish a rollout, + # so the allowlist must forward it. + - it: forwards maxUnavailable to the pool chart + documentSelector: + path: metadata.name + value: kubernetes-nodes-computeplane-cluster-md0 + set: + nodeGroups: + md0: + minReplicas: 1 + maxReplicas: 3 + instanceType: u1.large + diskSize: 32Gi + roles: ["ingress-nginx"] + maxUnavailable: 1 + asserts: + - equal: + path: spec.values.maxUnavailable + value: 1 + + - it: refuses a maxUnavailable above the int32 range CAPI decodes + set: + nodeGroups: + md0: + minReplicas: 1 + maxReplicas: 3 + instanceType: u1.large + diskSize: 32Gi + roles: ["ingress-nginx"] + maxUnavailable: 2147483648 + asserts: + - failedTemplate: + errorPattern: "maxUnavailable" + # Helm caps a HelmRelease name at 53 chars and each pool renders a 38-char fixed # prefix, so a pool key over 15 chars would create a release the apiserver # accepts but Flux never reconciles. The render must refuse instead (the same diff --git a/packages/extra/computeplane/values.schema.json b/packages/extra/computeplane/values.schema.json index 41db84758..fa640f3d9 100644 --- a/packages/extra/computeplane/values.schema.json +++ b/packages/extra/computeplane/values.schema.json @@ -93,6 +93,12 @@ "type": "integer", "default": 10 }, + "maxUnavailable": { + "description": "Workers a rollout may remove before their replacement is up. Unset, the `kubernetes-nodes` pool chart uses 0, so each replacement joins before an old worker is drained (1 on a pool with `gpus`). Set 1 on a pool without room for one extra VM, whose rollout would otherwise stall with the new Machine Pending.", + "type": "integer", + "maximum": 2147483647, + "minimum": 0 + }, "minReplicas": { "description": "Minimum number of replicas.", "type": "integer", diff --git a/packages/extra/computeplane/values.yaml b/packages/extra/computeplane/values.yaml index 8ab49ea95..2e568ddf3 100644 --- a/packages/extra/computeplane/values.yaml +++ b/packages/extra/computeplane/values.yaml @@ -30,6 +30,9 @@ ## @field {Resources} [resources] - Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Set both `cpu` and `memory` together or neither. ## @field {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). ## @field {Kubelet} [kubelet] - Kubelet resource reservations for this node group. +## @field {*int} [maxUnavailable] - Workers a rollout may remove before their replacement is up. Unset, the `kubernetes-nodes` pool chart uses 0, so each replacement joins before an old worker is drained (1 on a pool with `gpus`). Set 1 on a pool without room for one extra VM, whose rollout would otherwise stall with the new Machine Pending. +## @minimum 0 +## @maximum 2147483647 ## @param {map[string]NodeGroup} nodeGroups - Worker node groups of the ComputePlane cluster (cluster shape only — the security posture is fixed by the module and cannot be overridden through this knob: the module forwards only the documented per-pool fields to the `kubernetes-nodes` pool chart, so any other `NodeGroup` field, e.g. `talos`/`version`/`images`, is silently dropped — accepted but inert). Same structure as the `kubernetes` application's `nodeGroups`. When left empty, the module itself materialises a default scale-from-zero `md0` pool with `roles: [ingress-nginx]` (the wrapped `kubernetes` chart no longer injects one after the Phase 2 split), so the cluster sits idle until a pending Pod (e.g. the ingress-nginx controller) triggers the cluster-autoscaler. Supplied groups are authoritative (not merged with the default), so at least one of them must carry `roles: [ingress-nginx]` — enforced at render time. Each supplied pool key must be at most 15 characters so its `kubernetes-nodes-computeplane-cluster-` HelmRelease name fits Helm's 53-character limit — also enforced at render time. nodeGroups: {} diff --git a/packages/system/computeplane-rd/cozyrds/computeplane.yaml b/packages/system/computeplane-rd/cozyrds/computeplane.yaml index c84d39b78..1a2b68a60 100644 --- a/packages/system/computeplane-rd/cozyrds/computeplane.yaml +++ b/packages/system/computeplane-rd/cozyrds/computeplane.yaml @@ -8,7 +8,7 @@ spec: singular: computeplane plural: computeplanes openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"nodeGroups":{"description":"Worker node groups of the ComputePlane cluster (cluster shape only — the security posture is fixed by the module and cannot be overridden through this knob: the module forwards only the documented per-pool fields to the `kubernetes-nodes` pool chart, so any other `NodeGroup` field, e.g. `talos`/`version`/`images`, is silently dropped — accepted but inert). Same structure as the `kubernetes` application's `nodeGroups`. When left empty, the module itself materialises a default scale-from-zero `md0` pool with `roles: [ingress-nginx]` (the wrapped `kubernetes` chart no longer injects one after the Phase 2 split), so the cluster sits idle until a pending Pod (e.g. the ingress-nginx controller) triggers the cluster-autoscaler. Supplied groups are authoritative (not merged with the default), so at least one of them must carry `roles: [ingress-nginx]` — enforced at render time. Each supplied pool key must be at most 15 characters so its `kubernetes-nodes-computeplane-cluster-` HelmRelease name fits Helm's 53-character limit — also enforced at render time.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskSize","instanceType","maxReplicas","minReplicas"],"properties":{"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"kubelet":{"description":"Kubelet resource reservations for this node group.","type":"object","properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"maxReplicas":{"description":"Maximum number of replicas.","type":"integer","default":10},"minReplicas":{"description":"Minimum number of replicas.","type":"integer","default":0},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Set both `cpu` and `memory` together or neither.","type":"object","properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"roles":{"description":"List of node roles. At least one group must carry `ingress-nginx` — the module pins the ingress-nginx addon on, and its controller only schedules onto nodes with that role (enforced at render time).","type":"array","items":{"type":"string"}},"storageClass":{"description":"StorageClass for worker node persistent disks. When empty, the `kubernetes-nodes` pool chart applies its own default (`replicated`).","type":"string","x-cozystack-options":{"source":"storageclass"}}}}}}} + {"title":"Chart Values","type":"object","properties":{"nodeGroups":{"description":"Worker node groups of the ComputePlane cluster (cluster shape only — the security posture is fixed by the module and cannot be overridden through this knob: the module forwards only the documented per-pool fields to the `kubernetes-nodes` pool chart, so any other `NodeGroup` field, e.g. `talos`/`version`/`images`, is silently dropped — accepted but inert). Same structure as the `kubernetes` application's `nodeGroups`. When left empty, the module itself materialises a default scale-from-zero `md0` pool with `roles: [ingress-nginx]` (the wrapped `kubernetes` chart no longer injects one after the Phase 2 split), so the cluster sits idle until a pending Pod (e.g. the ingress-nginx controller) triggers the cluster-autoscaler. Supplied groups are authoritative (not merged with the default), so at least one of them must carry `roles: [ingress-nginx]` — enforced at render time. Each supplied pool key must be at most 15 characters so its `kubernetes-nodes-computeplane-cluster-` HelmRelease name fits Helm's 53-character limit — also enforced at render time.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskSize","instanceType","maxReplicas","minReplicas"],"properties":{"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"kubelet":{"description":"Kubelet resource reservations for this node group.","type":"object","properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"maxReplicas":{"description":"Maximum number of replicas.","type":"integer","default":10},"maxUnavailable":{"description":"Workers a rollout may remove before their replacement is up. Unset, the `kubernetes-nodes` pool chart uses 0, so each replacement joins before an old worker is drained (1 on a pool with `gpus`). Set 1 on a pool without room for one extra VM, whose rollout would otherwise stall with the new Machine Pending.","type":"integer","maximum":2147483647,"minimum":0},"minReplicas":{"description":"Minimum number of replicas.","type":"integer","default":0},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Set both `cpu` and `memory` together or neither.","type":"object","properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"roles":{"description":"List of node roles. At least one group must carry `ingress-nginx` — the module pins the ingress-nginx addon on, and its controller only schedules onto nodes with that role (enforced at render time).","type":"array","items":{"type":"string"}},"storageClass":{"description":"StorageClass for worker node persistent disks. When empty, the `kubernetes-nodes` pool chart applies its own default (`replicated`).","type":"string","x-cozystack-options":{"source":"storageclass"}}}}}}} release: prefix: "" labels: diff --git a/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml b/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml index 0e5a196fd..df883809c 100644 --- a/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml +++ b/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml @@ -21,7 +21,7 @@ spec: singular: kubernetesnodes plural: kubernetesnodeses openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool name (the part after `-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `\"010\"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `\"010\"` reserves what `\"10\"` does.","type":"object","default":{},"properties":{"cpu":{"description":"Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"podCpuLimit":{"description":"CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"podCpuRequest":{"description":"CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"maxUnavailable":{"description":"Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.","type":"integer","minimum":0},"kernelModules":{"description":"Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Module name as `modprobe` takes it, e.g. `nvidia_uvm`.","type":"string"},"parameters":{"description":"Module parameters, each as a bare `key=value` string.","type":"array","items":{"type":"string"}}}}},"kubelet":{"description":"Kubelet resource reservations for this pool.","type":"object","default":{},"properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.","type":"boolean","default":false},"maxUnhealthy":{"description":"Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer (\"0\", \"1\", ...) or a percentage (\"0%\", \"50%\"); bare numeric strings are rejected, so the safer default is a percentage. Drop to \"0%\" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At \"50%\" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.","type":"string","default":"50%"},"nodeStartupTimeout":{"description":"Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.","type":"string","default":"20m"},"readyUnknownTimeout":{"description":"How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"5m"},"readyFalseTimeout":{"description":"How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"10m"},"maxNodeProvisionTime":{"description":"How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted. Keep in sync with the parent kubernetes chart.","type":"string","default":"30m"},"version":{"description":"Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`.","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"osImage":{"description":"Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool.","type":"object","default":{},"properties":{"builtin":{"description":"Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled.","type":"object","properties":{"schematicID":{"description":"Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version of the golden to clone. Defaults to `talos.version`.","type":"string"}}},"factory":{"description":"Import from an Image Factory / mirror (default).","type":"object","properties":{"imageFactoryURL":{"description":"Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.","type":"string"},"schematicID":{"description":"Talos schematic ID. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version. Defaults to `talos.version`.","type":"string"}}}}},"talos":{"description":"Talos worker image configuration. Keep in sync with the parent cluster's `talos`.","type":"object","default":{},"required":["imageFactoryURL","installerRepository","registryMirrors","schematicID","version"],"properties":{"imageFactoryCA":{"description":"Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.","type":"string","default":""},"imageFactoryURL":{"description":"Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.","type":"string","default":"https://factory.talos.dev"},"installerRepository":{"description":"OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.","type":"string","default":"factory.talos.dev/installer"},"registryMirrors":{"description":"Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true},"schematicID":{"description":"Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID.","type":"string","default":"ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"},"version":{"description":"Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.","type":"string","default":"v1.13.6"}}},"substrate":{"description":"Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment's template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["network"],"properties":{"dnsServers":{"description":"Nameservers written into the worker machineconfig. On kubevirt the workers use the management cluster's CoreDNS, which they reach over the pod network; an off-cluster Proxmox worker cannot, so it needs reachable resolvers of its own. Required when `substrate` is `proxmox`. Keep in sync with the parent kubernetes chart's `proxmox.dnsServers`.","type":"array","default":[],"items":{"type":"string"}},"full":{"description":"Full clone rather than linked. Default is a linked clone: it costs kilobytes at creation instead of the whole disk, because a ZFS-backed linked clone writes only its own increment (measured: 8K against 10.2G for the same worker full-cloned onto a `sparse 0` pool). The cost is that the clone holds the template's base snapshot, so the template cannot be rotated while any linked clone still references it — which is what the consolidation CronJob resolves 24h after creation. Set true for a pool that must be independent of the template from the first second.","type":"boolean","default":false},"network":{"description":"NIC configuration.","type":"object","default":{},"required":["bridge"],"properties":{"bridge":{"description":"Proxmox bridge the VM NIC attaches to (e.g. `vmbr0`). Required when `substrate` is `proxmox`.","type":"string","default":""},"mtu":{"description":"NIC MTU. Omitted when unset.","type":"integer"},"vlan":{"description":"L2 VLAN tag. Omitted when unset.","type":"integer"}}},"pool":{"description":"Proxmox resource pool the worker VMs are added to. Set the same pool on every node pool of a tenant cluster and grant the tenant's tokens their privileges on `/pool/`: that is the one ACL path that follows every VM capmox creates, replacements included. The pool must exist on the hypervisor; capmox does not create it. Empty adds the workers to no pool.","type":"string","default":""},"storage":{"description":"Proxmox storage for the cloned disk; needs `full: true`. A target storage is a full-clone parameter — a linked clone always lives on the template's storage, and Proxmox refuses the pair — so the render refuses it too. Empty keeps the template's storage.","type":"string","default":""},"templateTags":{"description":"Tags identifying the Talos VM template to clone. capmox matches a template when its tag set is EQUAL to this list, not when it contains it, so every tag on the template must be listed here — a missing one yields `found 0 VM templates with tags ...` while the tags are plainly present. The template has to carry this pool's `talos.schematicID` at `talos.version` in the nocloud flavour, with the Proxmox guest agent enabled for the `qemu-guest-agent` extension of the default schematic; `hack/proxmox-talos-template.sh` builds it from these values and prints the tags to list here. Do not build it from the platform release's `nocloud-amd64.raw.xz`: that is the bare-metal node image, whose `zfs` extension keeps a VM from finishing its boot sequence and reboots it at the Talos boot timeout. Required when `substrate` is `proxmox`.","type":"array","default":[],"items":{"type":"string"}}}}}} + {"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool name (the part after `-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `\"010\"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `\"010\"` reserves what `\"10\"` does.","type":"object","default":{},"properties":{"cpu":{"description":"Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"podCpuLimit":{"description":"CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"podCpuRequest":{"description":"CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"maxUnavailable":{"description":"Workers a rollout (instance type, image or version change) may remove before their replacement is up. Unset, it is 0, so each replacement joins before an old worker is drained, and 1 on a pool with `gpus`, whose replacement may need the very GPU the old worker holds. With 0, a rollout needs room for one extra VM: a pool short of quota or capacity for it stalls, with the new Machine Pending, until room is made or `maxUnavailable` is set to 1. Set 1 on a Proxmox pool whose template passes a GPU through: the chart cannot see that GPU, and with 0 the rollout would wait for it forever.","type":"integer","maximum":2147483647,"minimum":0},"kernelModules":{"description":"Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Module name as `modprobe` takes it, e.g. `nvidia_uvm`.","type":"string"},"parameters":{"description":"Module parameters, each as a bare `key=value` string.","type":"array","items":{"type":"string"}}}}},"kubelet":{"description":"Kubelet resource reservations for this pool.","type":"object","default":{},"properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.","type":"boolean","default":false},"maxUnhealthy":{"description":"Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer (\"0\", \"1\", ...) or a percentage (\"0%\", \"50%\"); bare numeric strings are rejected, so the safer default is a percentage. Drop to \"0%\" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At \"50%\" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.","type":"string","default":"50%"},"nodeStartupTimeout":{"description":"Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.","type":"string","default":"20m"},"readyUnknownTimeout":{"description":"How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"5m"},"readyFalseTimeout":{"description":"How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"10m"},"maxNodeProvisionTime":{"description":"How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted. Keep in sync with the parent kubernetes chart.","type":"string","default":"30m"},"version":{"description":"Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`.","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"osImage":{"description":"Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool.","type":"object","default":{},"properties":{"builtin":{"description":"Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled.","type":"object","properties":{"schematicID":{"description":"Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version of the golden to clone. Defaults to `talos.version`.","type":"string"}}},"factory":{"description":"Import from an Image Factory / mirror (default).","type":"object","properties":{"imageFactoryURL":{"description":"Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.","type":"string"},"schematicID":{"description":"Talos schematic ID. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version. Defaults to `talos.version`.","type":"string"}}}}},"talos":{"description":"Talos worker image configuration. Keep in sync with the parent cluster's `talos`.","type":"object","default":{},"required":["imageFactoryURL","installerRepository","registryMirrors","schematicID","version"],"properties":{"imageFactoryCA":{"description":"Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.","type":"string","default":""},"imageFactoryURL":{"description":"Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.","type":"string","default":"https://factory.talos.dev"},"installerRepository":{"description":"OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.","type":"string","default":"factory.talos.dev/installer"},"registryMirrors":{"description":"Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true},"schematicID":{"description":"Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID.","type":"string","default":"ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"},"version":{"description":"Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.","type":"string","default":"v1.13.6"}}},"substrate":{"description":"Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment's template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["network"],"properties":{"dnsServers":{"description":"Nameservers written into the worker machineconfig. On kubevirt the workers use the management cluster's CoreDNS, which they reach over the pod network; an off-cluster Proxmox worker cannot, so it needs reachable resolvers of its own. Required when `substrate` is `proxmox`. Keep in sync with the parent kubernetes chart's `proxmox.dnsServers`.","type":"array","default":[],"items":{"type":"string"}},"full":{"description":"Full clone rather than linked. Default is a linked clone: it costs kilobytes at creation instead of the whole disk, because a ZFS-backed linked clone writes only its own increment (measured: 8K against 10.2G for the same worker full-cloned onto a `sparse 0` pool). The cost is that the clone holds the template's base snapshot, so the template cannot be rotated while any linked clone still references it — which is what the consolidation CronJob resolves 24h after creation. Set true for a pool that must be independent of the template from the first second.","type":"boolean","default":false},"network":{"description":"NIC configuration.","type":"object","default":{},"required":["bridge"],"properties":{"bridge":{"description":"Proxmox bridge the VM NIC attaches to (e.g. `vmbr0`). Required when `substrate` is `proxmox`.","type":"string","default":""},"mtu":{"description":"NIC MTU. Omitted when unset.","type":"integer"},"vlan":{"description":"L2 VLAN tag. Omitted when unset.","type":"integer"}}},"pool":{"description":"Proxmox resource pool the worker VMs are added to. Set the same pool on every node pool of a tenant cluster and grant the tenant's tokens their privileges on `/pool/`: that is the one ACL path that follows every VM capmox creates, replacements included. The pool must exist on the hypervisor; capmox does not create it. Empty adds the workers to no pool.","type":"string","default":""},"storage":{"description":"Proxmox storage for the cloned disk; needs `full: true`. A target storage is a full-clone parameter — a linked clone always lives on the template's storage, and Proxmox refuses the pair — so the render refuses it too. Empty keeps the template's storage.","type":"string","default":""},"templateTags":{"description":"Tags identifying the Talos VM template to clone. capmox matches a template when its tag set is EQUAL to this list, not when it contains it, so every tag on the template must be listed here — a missing one yields `found 0 VM templates with tags ...` while the tags are plainly present. The template has to carry this pool's `talos.schematicID` at `talos.version` in the nocloud flavour, with the Proxmox guest agent enabled for the `qemu-guest-agent` extension of the default schematic; `hack/proxmox-talos-template.sh` builds it from these values and prints the tags to list here. Do not build it from the platform release's `nocloud-amd64.raw.xz`: that is the bare-metal node image, whose `zfs` extension keeps a VM from finishing its boot sequence and reboots it at the Talos boot timeout. Required when `substrate` is `proxmox`.","type":"array","default":[],"items":{"type":"string"}}}}}} release: labels: sharding.fluxcd.io/key: tenants From 9fba8af69cd2083fbfbeb21e14551f21ad47a73d Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Sun, 8 Nov 2026 19:45:18 +0100 Subject: [PATCH 200/518] test(redis): pin the release name in the Sentinel PDB hook for the longest release The long-name case checked only that the Job name fits in 63 characters and that the Job targets the right PDB. A Job name that dropped the release, letting two releases' hooks replace each other, or a Role granting a different PDB would both have passed. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../redis/tests/sentinel_pdb_hook_test.yaml | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml index 5676f8d31..03f3da02d 100644 --- a/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml +++ b/packages/apps/redis/tests/sentinel_pdb_hook_test.yaml @@ -116,10 +116,30 @@ tests: - matchRegex: path: metadata.name pattern: '[a-z0-9]$' + # The full release name stays in the Job name, so the hooks of two + # releases in one namespace never replace each other's Job. + - matchRegex: + path: metadata.name + pattern: ^redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa- - equal: path: spec.template.spec.containers[0].command[4] value: rfs-redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + - it: lets the hook delete the Sentinel PDB of the longest release name + release: + name: redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + upgrade: true + set: + replicas: 1 + documentSelector: + path: kind + value: Role + asserts: + - equal: + path: rules[0].resourceNames + value: + - rfs-redis-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + - it: keeps the full Job name for a short release name release: upgrade: true From 2e5a3bd6a7bcbd85f77e1043f782951ce37411cd Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Mon, 9 Nov 2026 15:47:34 +0700 Subject: [PATCH 201/518] fix(dashboard): keep the tenant logs page off application URLs Every console path of one or two segments belongs to an application list or detail page, so /console/observability/logs took the Overview of an application named logs whose kind has the plural observability. Kubernetes names must start with an alphanumeric character, so the reserved "-" segment cannot collide with any application. Assisted-by: LLM Signed-off-by: ZverGuy --- .../src/routes/ConsolePage.routing.test.tsx | 18 ++++++++++++++++-- .../apps/console/src/routes/ConsolePage.tsx | 2 +- .../src/routes/sidebar-sections.test.tsx | 2 +- .../console/src/routes/sidebar-sections.tsx | 2 +- 4 files changed, 19 insertions(+), 5 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx index cf6982823..8e71ff98a 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx @@ -71,12 +71,12 @@ describe("ConsolePage routing", () => { expect(screen.queryByText(/Cluster-scoped capacity/i)).toBeNull() }) - it("serves the tenant logs page under observability/", async () => { + it("serves the tenant logs page under the reserved -/ segment", async () => { renderWithK8sProvider( , - { client: makeClient(), initialRoute: "/observability/logs" }, + { client: makeClient(), initialRoute: "/-/observability/logs" }, ) expect(await screen.findByText(/^Every application in/)).toBeInTheDocument() @@ -94,4 +94,18 @@ describe("ConsolePage routing", () => { await waitFor(() => expect(client.list).toHaveBeenCalled()) expect(screen.queryByText(/^Every application in/)).toBeNull() }) + + // No ApplicationDefinition plural or application name can be "-", so only + // routes under it are safe from shadowing an application's own pages. + it("leaves /observability/logs to an application named logs of kind observability", async () => { + renderWithK8sProvider( + + + , + { client: makeClient(), initialRoute: "/observability/logs" }, + ) + + expect(await screen.findByText("Loading…")).toBeInTheDocument() + expect(screen.queryByText(/^Every application in/)).toBeNull() + }) }) diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx index 2b2aed3e4..c78ae619e 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.tsx @@ -97,7 +97,7 @@ export function ConsolePage() { path="migration/vmimporttasks/:name" element={} /> - } /> + } /> } /> } /> } /> diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx index 1fcc3cde4..e678c7dd4 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.test.tsx @@ -117,7 +117,7 @@ describe("useConsoleSidebarSections — admin areas moved out", () => { wrapper: makeWrapper(client), }) await waitFor(() => expect(result.current.length).toBeGreaterThan(0)) - expect(findItem(result.current, "Logs")?.to).toBe("/console/observability/logs") + expect(findItem(result.current, "Logs")?.to).toBe("/console/-/observability/logs") expect(result.current.find((s) => s.title === "Observability")?.items).toHaveLength(1) }) }) diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx index 0cf1ce089..51c9ea00a 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/sidebar-sections.tsx @@ -121,7 +121,7 @@ export function useConsoleSidebarSections(): SidebarSection[] { const observabilitySection: SidebarSection = { title: "Observability", - items: [{ label: "Logs", to: "/console/observability/logs", icon: ScrollText }], + items: [{ label: "Logs", to: "/console/-/observability/logs", icon: ScrollText }], } return [...categorySections, backupsSection, migrationSection, observabilitySection] From 8e35ff521c5cd960b9605e681d35ab49defe4b9a Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:11:54 +0700 Subject: [PATCH 202/518] feat(observability-proxy): parse the four tenant observability roots The proxy derives its single SubjectAccessReview from the request path, so the path has to be split and validated before anything else runs. Names are DNS-1123 checked because they end up quoted inside LogsQL and PromQL scopes, and percent-encoding is refused so a decoded slash cannot shift the segments authorization is built from. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/route.go | 146 ++++++++++++++++++++++ internal/observabilityproxy/route_test.go | 124 ++++++++++++++++++ 2 files changed, 270 insertions(+) create mode 100644 internal/observabilityproxy/route.go create mode 100644 internal/observabilityproxy/route_test.go diff --git a/internal/observabilityproxy/route.go b/internal/observabilityproxy/route.go new file mode 100644 index 000000000..7c692c0ae --- /dev/null +++ b/internal/observabilityproxy/route.go @@ -0,0 +1,146 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "fmt" + "net/http" + "strings" + + authorizationv1 "k8s.io/api/authorization/v1" + "k8s.io/apimachinery/pkg/util/validation" +) + +// RouteKind is one of the four Kubernetes-shaped roots the proxy serves. +type RouteKind int + +const ( + NamespaceLogs RouteKind = iota + 1 + NamespaceMetrics + ApplicationLog + ApplicationMetrics +) + +func (k RouteKind) String() string { + switch k { + case NamespaceLogs: + return "namespace-logs" + case NamespaceMetrics: + return "namespace-metrics" + case ApplicationLog: + return "application-log" + case ApplicationMetrics: + return "application-metrics" + } + return "unknown" +} + +const ( + coreGroup = "cozystack.io" + appsGroup = "apps.cozystack.io" +) + +// Route is a request path split into the scope it names and the upstream +// endpoint below the root. Plural and Name are set only on application roots. +type Route struct { + Kind RouteKind + Namespace string + Plural string + Name string + Endpoint string +} + +// HTTPError carries the status code the proxy answers with. +type HTTPError struct { + Code int + Message string +} + +func (e *HTTPError) Error() string { return fmt.Sprintf("%d: %s", e.Code, e.Message) } + +func errorf(code int, format string, args ...any) *HTTPError { + return &HTTPError{Code: code, Message: fmt.Sprintf(format, args...)} +} + +// ParseRoute takes the escaped request path. Percent-encoding is refused +// outright: a decoded "%2F" would shift the segments the authorization is +// derived from. +func ParseRoute(escapedPath string) (Route, *HTTPError) { + notFound := errorf(http.StatusNotFound, "not found") + if strings.Contains(escapedPath, "%") { + return Route{}, notFound + } + segs := strings.Split(strings.TrimPrefix(escapedPath, "/"), "/") + if len(segs) > 0 && segs[len(segs)-1] == "" { + segs = segs[:len(segs)-1] + } + for _, s := range segs { + if s == "" || s == "." || s == ".." { + return Route{}, notFound + } + } + if len(segs) < 6 || segs[0] != "apis" || segs[2] != "v1alpha1" || segs[3] != "namespaces" { + return Route{}, notFound + } + + var r Route + var rest []string + switch { + case segs[1] == coreGroup && segs[5] == "logs": + r.Kind, rest = NamespaceLogs, segs[6:] + case segs[1] == coreGroup && segs[5] == "metrics": + r.Kind, rest = NamespaceMetrics, segs[6:] + case segs[1] == appsGroup && len(segs) >= 8 && segs[7] == "log": + r.Kind, rest = ApplicationLog, segs[8:] + case segs[1] == appsGroup && len(segs) >= 8 && segs[7] == "metrics": + r.Kind, rest = ApplicationMetrics, segs[8:] + default: + return Route{}, notFound + } + + r.Namespace = segs[4] + if errs := validation.IsDNS1123Label(r.Namespace); len(errs) > 0 { + return Route{}, errorf(http.StatusBadRequest, "invalid namespace %q: %s", r.Namespace, strings.Join(errs, "; ")) + } + if r.Kind == ApplicationLog || r.Kind == ApplicationMetrics { + r.Plural, r.Name = segs[5], segs[6] + if errs := validation.IsDNS1123Label(r.Plural); len(errs) > 0 { + return Route{}, errorf(http.StatusBadRequest, "invalid resource %q: %s", r.Plural, strings.Join(errs, "; ")) + } + if errs := validation.IsDNS1123Subdomain(r.Name); len(errs) > 0 { + return Route{}, errorf(http.StatusBadRequest, "invalid name %q: %s", r.Name, strings.Join(errs, "; ")) + } + } + r.Endpoint = strings.Join(rest, "/") + return r, nil +} + +// ResourceAttributes is the single SubjectAccessReview a route requires. It +// depends on the path only, so a POST query authorizes exactly like a GET. +func (r Route) ResourceAttributes() authorizationv1.ResourceAttributes { + switch r.Kind { + case NamespaceLogs: + return authorizationv1.ResourceAttributes{Verb: "list", Group: coreGroup, Resource: "logs", Namespace: r.Namespace} + case NamespaceMetrics: + return authorizationv1.ResourceAttributes{Verb: "list", Group: coreGroup, Resource: "metrics", Namespace: r.Namespace} + case ApplicationLog: + return authorizationv1.ResourceAttributes{Verb: "get", Group: appsGroup, Resource: r.Plural, Subresource: "log", Name: r.Name, Namespace: r.Namespace} + case ApplicationMetrics: + return authorizationv1.ResourceAttributes{Verb: "get", Group: appsGroup, Resource: r.Plural, Subresource: "metrics", Name: r.Name, Namespace: r.Namespace} + } + return authorizationv1.ResourceAttributes{} +} diff --git a/internal/observabilityproxy/route_test.go b/internal/observabilityproxy/route_test.go new file mode 100644 index 000000000..73d8e04cd --- /dev/null +++ b/internal/observabilityproxy/route_test.go @@ -0,0 +1,124 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "net/http" + "testing" + + authorizationv1 "k8s.io/api/authorization/v1" +) + +func TestParseRoute(t *testing.T) { + tests := []struct { + name string + path string + wantCode int + want Route + }{ + { + name: "namespace logs", + path: "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/select/logsql/query", + want: Route{Kind: NamespaceLogs, Namespace: "tenant-a", Endpoint: "select/logsql/query"}, + }, + { + name: "namespace logs root without endpoint", + path: "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/", + want: Route{Kind: NamespaceLogs, Namespace: "tenant-a"}, + }, + { + name: "namespace metrics", + path: "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/api/v1/query", + want: Route{Kind: NamespaceMetrics, Namespace: "tenant-a", Endpoint: "api/v1/query"}, + }, + { + name: "application log", + path: "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db.main/log/select/logsql/tail", + want: Route{Kind: ApplicationLog, Namespace: "tenant-a", Plural: "postgreses", Name: "db.main", Endpoint: "select/logsql/tail"}, + }, + { + name: "application metrics", + path: "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/metrics/api/v1/query_range", + want: Route{Kind: ApplicationMetrics, Namespace: "tenant-a", Plural: "postgreses", Name: "db", Endpoint: "api/v1/query_range"}, + }, + {name: "unknown group", path: "/apis/core.cozystack.io/v1alpha1/namespaces/a/logs/x", wantCode: http.StatusNotFound}, + {name: "unknown version", path: "/apis/cozystack.io/v1/namespaces/a/logs/x", wantCode: http.StatusNotFound}, + {name: "unknown namespace resource", path: "/apis/cozystack.io/v1alpha1/namespaces/a/events/x", wantCode: http.StatusNotFound}, + {name: "unknown app subresource", path: "/apis/apps.cozystack.io/v1alpha1/namespaces/a/postgreses/db/exec/x", wantCode: http.StatusNotFound}, + {name: "core api", path: "/api/v1/namespaces/a/pods", wantCode: http.StatusNotFound}, + {name: "truncated", path: "/apis/cozystack.io/v1alpha1/namespaces/a", wantCode: http.StatusNotFound}, + {name: "empty segment", path: "/apis/cozystack.io/v1alpha1/namespaces/a/logs/select//logsql/query", wantCode: http.StatusNotFound}, + {name: "dot dot segment", path: "/apis/cozystack.io/v1alpha1/namespaces/a/logs/../metrics/x", wantCode: http.StatusNotFound}, + {name: "percent encoded", path: "/apis/cozystack.io/v1alpha1/namespaces/a%2Fb/logs/x", wantCode: http.StatusNotFound}, + {name: "uppercase namespace", path: "/apis/cozystack.io/v1alpha1/namespaces/Tenant/logs/x", wantCode: http.StatusBadRequest}, + {name: "namespace with dot", path: "/apis/cozystack.io/v1alpha1/namespaces/a.b/logs/x", wantCode: http.StatusBadRequest}, + {name: "namespace with quote", path: `/apis/cozystack.io/v1alpha1/namespaces/a"b/logs/x`, wantCode: http.StatusBadRequest}, + {name: "name with logsql metacharacters", path: `/apis/apps.cozystack.io/v1alpha1/namespaces/a/postgreses/db"|union/log/x`, wantCode: http.StatusBadRequest}, + {name: "name with braces", path: "/apis/apps.cozystack.io/v1alpha1/namespaces/a/postgreses/{db}/log/x", wantCode: http.StatusBadRequest}, + {name: "plural uppercase", path: "/apis/apps.cozystack.io/v1alpha1/namespaces/a/Postgreses/db/log/x", wantCode: http.StatusBadRequest}, + {name: "plural with dot", path: "/apis/apps.cozystack.io/v1alpha1/namespaces/a/postgres.es/db/log/x", wantCode: http.StatusBadRequest}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, err := ParseRoute(tt.path) + if tt.wantCode != 0 { + if err == nil { + t.Fatalf("ParseRoute(%q) = %+v, want error with code %d", tt.path, got, tt.wantCode) + } + if err.Code != tt.wantCode { + t.Fatalf("ParseRoute(%q) code = %d (%s), want %d", tt.path, err.Code, err.Message, tt.wantCode) + } + return + } + if err != nil { + t.Fatalf("ParseRoute(%q) error = %v", tt.path, err) + } + if got != tt.want { + t.Fatalf("ParseRoute(%q) = %+v, want %+v", tt.path, got, tt.want) + } + }) + } +} + +func TestRouteResourceAttributes(t *testing.T) { + tests := []struct { + route Route + want authorizationv1.ResourceAttributes + }{ + { + route: Route{Kind: NamespaceLogs, Namespace: "a"}, + want: authorizationv1.ResourceAttributes{Verb: "list", Group: "cozystack.io", Resource: "logs", Namespace: "a"}, + }, + { + route: Route{Kind: NamespaceMetrics, Namespace: "a"}, + want: authorizationv1.ResourceAttributes{Verb: "list", Group: "cozystack.io", Resource: "metrics", Namespace: "a"}, + }, + { + route: Route{Kind: ApplicationLog, Namespace: "a", Plural: "postgreses", Name: "db"}, + want: authorizationv1.ResourceAttributes{Verb: "get", Group: "apps.cozystack.io", Resource: "postgreses", Subresource: "log", Name: "db", Namespace: "a"}, + }, + { + route: Route{Kind: ApplicationMetrics, Namespace: "a", Plural: "postgreses", Name: "db"}, + want: authorizationv1.ResourceAttributes{Verb: "get", Group: "apps.cozystack.io", Resource: "postgreses", Subresource: "metrics", Name: "db", Namespace: "a"}, + }, + } + for _, tt := range tests { + if got := tt.route.ResourceAttributes(); got != tt.want { + t.Errorf("%v.ResourceAttributes() = %+v, want %+v", tt.route.Kind, got, tt.want) + } + } +} From 448cd2b5bd23c45671734884cffaa0384cd703cc Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:13:08 +0700 Subject: [PATCH 203/518] feat(observability-proxy): authenticate and authorize with cached reviews The proxy holds no credentials of its own: kube-apiserver answers who a bearer token belongs to and whether that identity may read a scope, the same way an aggregated API server would ask through delegated auth. Both answers are cached briefly so a dashboard polling several panels does not turn into a review per request, while transport errors are left uncached so an apiserver blip does not lock users out for the TTL. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/auth.go | 164 +++++++++++++++++++++ internal/observabilityproxy/auth_test.go | 173 +++++++++++++++++++++++ 2 files changed, 337 insertions(+) create mode 100644 internal/observabilityproxy/auth.go create mode 100644 internal/observabilityproxy/auth_test.go diff --git a/internal/observabilityproxy/auth.go b/internal/observabilityproxy/auth.go new file mode 100644 index 000000000..90756bdad --- /dev/null +++ b/internal/observabilityproxy/auth.go @@ -0,0 +1,164 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "sync" + "time" + + authenticationv1 "k8s.io/api/authentication/v1" + authorizationv1 "k8s.io/api/authorization/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes" +) + +// ErrUnauthenticated means the token is missing or kube-apiserver rejected it. +var ErrUnauthenticated = errors.New("unauthenticated") + +// Authorizer asks kube-apiserver who a token belongs to and whether that +// identity may read a scope. Answers, including negative ones, are cached +// for the TTL; transport errors are not, so an apiserver blip is retried on +// the next request instead of locking a user out for the TTL. +type Authorizer struct { + client kubernetes.Interface + now func() time.Time + tokens *ttlCache[*authenticationv1.UserInfo] + decisions *ttlCache[bool] +} + +func NewAuthorizer(client kubernetes.Interface, ttl time.Duration) *Authorizer { + return &Authorizer{ + client: client, + now: time.Now, + tokens: newTTLCache[*authenticationv1.UserInfo](ttl), + decisions: newTTLCache[bool](ttl), + } +} + +func (a *Authorizer) Authenticate(ctx context.Context, token string) (*authenticationv1.UserInfo, error) { + if token == "" { + return nil, ErrUnauthenticated + } + sum := sha256.Sum256([]byte(token)) + key := hex.EncodeToString(sum[:]) + if u, ok := a.tokens.get(key, a.now()); ok { + if u == nil { + return nil, ErrUnauthenticated + } + return u, nil + } + tr, err := a.client.AuthenticationV1().TokenReviews().Create(ctx, + &authenticationv1.TokenReview{Spec: authenticationv1.TokenReviewSpec{Token: token}}, metav1.CreateOptions{}) + if err != nil { + return nil, fmt.Errorf("token review: %w", err) + } + var u *authenticationv1.UserInfo + if tr.Status.Authenticated && tr.Status.User.Username != "" { + u = tr.Status.User.DeepCopy() + } + a.tokens.set(key, u, a.now()) + if u == nil { + return nil, ErrUnauthenticated + } + return u, nil +} + +func (a *Authorizer) Authorize(ctx context.Context, user *authenticationv1.UserInfo, attrs authorizationv1.ResourceAttributes) (bool, error) { + extra := make(map[string]authorizationv1.ExtraValue, len(user.Extra)) + for k, v := range user.Extra { + extra[k] = authorizationv1.ExtraValue(v) + } + spec := authorizationv1.SubjectAccessReviewSpec{ + ResourceAttributes: &attrs, + User: user.Username, + UID: user.UID, + Groups: user.Groups, + Extra: extra, + } + raw, err := json.Marshal(spec) + if err != nil { + return false, err + } + key := string(raw) + if allowed, ok := a.decisions.get(key, a.now()); ok { + return allowed, nil + } + sar, err := a.client.AuthorizationV1().SubjectAccessReviews().Create(ctx, + &authorizationv1.SubjectAccessReview{Spec: spec}, metav1.CreateOptions{}) + if err != nil { + return false, fmt.Errorf("subject access review: %w", err) + } + allowed := sar.Status.Allowed && !sar.Status.Denied + a.decisions.set(key, allowed, a.now()) + return allowed, nil +} + +// ttlCacheMaxEntries bounds memory against a flood of distinct tokens; once +// reached, expired entries are swept and, if that is not enough, the cache +// starts over, which costs only extra reviews. +const ttlCacheMaxEntries = 10000 + +type ttlEntry[V any] struct { + value V + expires time.Time +} + +type ttlCache[V any] struct { + mu sync.Mutex + ttl time.Duration + entries map[string]ttlEntry[V] +} + +func newTTLCache[V any](ttl time.Duration) *ttlCache[V] { + return &ttlCache[V]{ttl: ttl, entries: map[string]ttlEntry[V]{}} +} + +func (c *ttlCache[V]) get(key string, now time.Time) (V, bool) { + c.mu.Lock() + defer c.mu.Unlock() + e, ok := c.entries[key] + if !ok || !now.Before(e.expires) { + var zero V + return zero, false + } + return e.value, true +} + +func (c *ttlCache[V]) set(key string, v V, now time.Time) { + if c.ttl <= 0 { + return + } + c.mu.Lock() + defer c.mu.Unlock() + if len(c.entries) >= ttlCacheMaxEntries { + for k, e := range c.entries { + if !now.Before(e.expires) { + delete(c.entries, k) + } + } + if len(c.entries) >= ttlCacheMaxEntries { + c.entries = map[string]ttlEntry[V]{} + } + } + c.entries[key] = ttlEntry[V]{value: v, expires: now.Add(c.ttl)} +} diff --git a/internal/observabilityproxy/auth_test.go b/internal/observabilityproxy/auth_test.go new file mode 100644 index 000000000..b57b80cf1 --- /dev/null +++ b/internal/observabilityproxy/auth_test.go @@ -0,0 +1,173 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "context" + "errors" + "sync/atomic" + "testing" + "time" + + authenticationv1 "k8s.io/api/authentication/v1" + authorizationv1 "k8s.io/api/authorization/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes/fake" + k8stesting "k8s.io/client-go/testing" +) + +// fakeKube answers TokenReviews for the tokens in users and allows a +// SubjectAccessReview when allow returns true; it counts both calls. +type fakeKube struct { + client *fake.Clientset + reviews atomic.Int32 + sars atomic.Int32 + lastSAR atomic.Pointer[authorizationv1.SubjectAccessReviewSpec] + reviewFn func() error +} + +func newFakeKube(users map[string]string, allow func(user string, attrs *authorizationv1.ResourceAttributes) bool) *fakeKube { + f := &fakeKube{client: fake.NewClientset()} + f.client.PrependReactor("create", "tokenreviews", func(action k8stesting.Action) (bool, runtime.Object, error) { + f.reviews.Add(1) + if f.reviewFn != nil { + if err := f.reviewFn(); err != nil { + return true, nil, err + } + } + tr := action.(k8stesting.CreateAction).GetObject().(*authenticationv1.TokenReview).DeepCopy() + if name, ok := users[tr.Spec.Token]; ok { + tr.Status = authenticationv1.TokenReviewStatus{Authenticated: true, User: authenticationv1.UserInfo{Username: name, Groups: []string{"tenant-a-view"}}} + } + return true, tr, nil + }) + f.client.PrependReactor("create", "subjectaccessreviews", func(action k8stesting.Action) (bool, runtime.Object, error) { + f.sars.Add(1) + sar := action.(k8stesting.CreateAction).GetObject().(*authorizationv1.SubjectAccessReview).DeepCopy() + spec := sar.Spec + f.lastSAR.Store(&spec) + sar.Status.Allowed = allow(sar.Spec.User, sar.Spec.ResourceAttributes) + return true, sar, nil + }) + return f +} + +func allowNamespace(ns string) func(string, *authorizationv1.ResourceAttributes) bool { + return func(_ string, a *authorizationv1.ResourceAttributes) bool { return a != nil && a.Namespace == ns } +} + +func TestAuthenticate(t *testing.T) { + kube := newFakeKube(map[string]string{"good": "alice"}, allowNamespace("tenant-a")) + a := NewAuthorizer(kube.client, 10*time.Second) + ctx := context.Background() + + u, err := a.Authenticate(ctx, "good") + if err != nil || u.Username != "alice" { + t.Fatalf("Authenticate(good) = %+v, %v", u, err) + } + if _, err := a.Authenticate(ctx, "good"); err != nil { + t.Fatal(err) + } + if n := kube.reviews.Load(); n != 1 { + t.Fatalf("TokenReviews = %d, want 1 (second call cached)", n) + } + if _, err := a.Authenticate(ctx, "bad"); !errors.Is(err, ErrUnauthenticated) { + t.Fatalf("Authenticate(bad) err = %v, want ErrUnauthenticated", err) + } + if _, err := a.Authenticate(ctx, ""); !errors.Is(err, ErrUnauthenticated) { + t.Fatalf("Authenticate(\"\") err = %v, want ErrUnauthenticated", err) + } +} + +func TestAuthenticateCacheExpires(t *testing.T) { + kube := newFakeKube(map[string]string{"good": "alice"}, allowNamespace("tenant-a")) + a := NewAuthorizer(kube.client, 10*time.Second) + now := time.Unix(1000, 0) + a.now = func() time.Time { return now } + ctx := context.Background() + if _, err := a.Authenticate(ctx, "good"); err != nil { + t.Fatal(err) + } + now = now.Add(11 * time.Second) + if _, err := a.Authenticate(ctx, "good"); err != nil { + t.Fatal(err) + } + if n := kube.reviews.Load(); n != 2 { + t.Fatalf("TokenReviews = %d, want 2 after TTL", n) + } +} + +func TestAuthenticateErrorNotCached(t *testing.T) { + kube := newFakeKube(map[string]string{"good": "alice"}, allowNamespace("tenant-a")) + kube.reviewFn = func() error { return errors.New("apiserver down") } + a := NewAuthorizer(kube.client, 10*time.Second) + ctx := context.Background() + if _, err := a.Authenticate(ctx, "good"); err == nil || errors.Is(err, ErrUnauthenticated) { + t.Fatalf("err = %v, want a transport error", err) + } + kube.reviewFn = nil + if _, err := a.Authenticate(ctx, "good"); err != nil { + t.Fatalf("err = %v after recovery", err) + } +} + +func TestAuthorize(t *testing.T) { + kube := newFakeKube(map[string]string{"good": "alice"}, allowNamespace("tenant-a")) + a := NewAuthorizer(kube.client, 10*time.Second) + ctx := context.Background() + user := &authenticationv1.UserInfo{Username: "alice", UID: "u1", Groups: []string{"g"}, Extra: map[string]authenticationv1.ExtraValue{"scope": {"x"}}} + + allowed := Route{Kind: NamespaceLogs, Namespace: "tenant-a"} + denied := Route{Kind: NamespaceLogs, Namespace: "tenant-b"} + + ok, err := a.Authorize(ctx, user, allowed.ResourceAttributes()) + if err != nil || !ok { + t.Fatalf("allow: %v, %v", ok, err) + } + spec := kube.lastSAR.Load() + if spec.User != "alice" || spec.UID != "u1" || len(spec.Groups) != 1 || spec.Extra["scope"][0] != "x" { + t.Fatalf("SAR identity not forwarded: %+v", spec) + } + if ra := spec.ResourceAttributes; ra.Verb != "list" || ra.Group != "cozystack.io" || ra.Resource != "logs" || ra.Namespace != "tenant-a" { + t.Fatalf("SAR attributes = %+v", ra) + } + + ok, err = a.Authorize(ctx, user, denied.ResourceAttributes()) + if err != nil || ok { + t.Fatalf("deny: %v, %v", ok, err) + } + + for range 3 { + if ok, _ := a.Authorize(ctx, user, allowed.ResourceAttributes()); !ok { + t.Fatal("cached allow flipped") + } + if ok, _ := a.Authorize(ctx, user, denied.ResourceAttributes()); ok { + t.Fatal("cached deny flipped") + } + } + if n := kube.sars.Load(); n != 2 { + t.Fatalf("SubjectAccessReviews = %d, want 2 (rest cached)", n) + } + + other := &authenticationv1.UserInfo{Username: "alice", UID: "u1", Groups: []string{"g", "tenant-b-view"}} + if _, err := a.Authorize(ctx, other, denied.ResourceAttributes()); err != nil { + t.Fatal(err) + } + if n := kube.sars.Load(); n != 3 { + t.Fatalf("a different group set must not hit another identity's cache entry, SARs = %d", n) + } +} From e02d0fa16ed972e7c9423fe366076c55efe50eb2 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:13:36 +0700 Subject: [PATCH 204/518] feat(observability-proxy): resolve a namespace to its log store Fluent Bit writes a namespace's logs to the store its monitoring label names, falling back to the root tenant, so exactly one store answers for a namespace. Reading with any other rule would show an empty log view or the wrong tenant's store, so the resolver repeats the collector's rule and is tested against the same namespace set as its chart test. It reads a namespace lister, keeping store lookups off the apiserver. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/store.go | 55 ++++++++++++++++ internal/observabilityproxy/store_test.go | 78 +++++++++++++++++++++++ 2 files changed, 133 insertions(+) create mode 100644 internal/observabilityproxy/store.go create mode 100644 internal/observabilityproxy/store_test.go diff --git a/internal/observabilityproxy/store.go b/internal/observabilityproxy/store.go new file mode 100644 index 000000000..ba0234ec3 --- /dev/null +++ b/internal/observabilityproxy/store.go @@ -0,0 +1,55 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + apierrors "k8s.io/apimachinery/pkg/api/errors" + corev1listers "k8s.io/client-go/listers/core/v1" +) + +// monitoringLabel is the label the tenant chart sets on every tenant +// namespace and the collector routes logs by +// (packages/system/monitoring-agents/templates/_tenant-log-routes.tpl). +const monitoringLabel = "namespace.cozystack.io/monitoring" + +// StoreResolver maps a namespace to the tenant namespace whose store holds +// its logs, by the same rule the collector writes with. +type StoreResolver struct { + namespaces corev1listers.NamespaceLister + root string +} + +func NewStoreResolver(namespaces corev1listers.NamespaceLister, rootTenant string) *StoreResolver { + return &StoreResolver{namespaces: namespaces, root: rootTenant} +} + +// Resolve answers the root tenant for a namespace that no longer exists: +// its label is gone with it, and the collector's default output is the +// root store, so that is where any lines written without a route are. +func (r *StoreResolver) Resolve(namespace string) (string, error) { + ns, err := r.namespaces.Get(namespace) + if apierrors.IsNotFound(err) { + return r.root, nil + } + if err != nil { + return "", err + } + if target := ns.Labels[monitoringLabel]; target != "" { + return target, nil + } + return r.root, nil +} diff --git a/internal/observabilityproxy/store_test.go b/internal/observabilityproxy/store_test.go new file mode 100644 index 000000000..a87bdf0f6 --- /dev/null +++ b/internal/observabilityproxy/store_test.go @@ -0,0 +1,78 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "testing" + + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + corev1listers "k8s.io/client-go/listers/core/v1" + "k8s.io/client-go/tools/cache" +) + +// The cases mirror packages/system/monitoring-agents/tests/ +// fluent-bit_tenant_log_routing_test.yaml: reads must resolve to the store +// the collector wrote to. +func TestStoreResolver(t *testing.T) { + indexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) + add := func(name string, labels map[string]string) { + if err := indexer.Add(&corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: name, Labels: labels}}); err != nil { + t.Fatal(err) + } + } + add("cozy-system", nil) + add("tenant-empty", map[string]string{monitoringLabel: ""}) + for ns, target := range map[string]string{ + "tenant-root": "tenant-root", + "tenant-inherits-root": "tenant-root", + "tenant-a": "tenant-a", + "tenant-a-child": "tenant-a", + "tenant-b": "tenant-b", + } { + add(ns, map[string]string{monitoringLabel: target}) + } + lister := corev1listers.NewNamespaceLister(indexer) + + tests := []struct { + root, ns, want string + }{ + {"tenant-root", "cozy-system", "tenant-root"}, + {"tenant-root", "tenant-empty", "tenant-root"}, + {"tenant-root", "tenant-root", "tenant-root"}, + {"tenant-root", "tenant-inherits-root", "tenant-root"}, + {"tenant-root", "tenant-a", "tenant-a"}, + {"tenant-root", "tenant-a-child", "tenant-a"}, + {"tenant-root", "tenant-b", "tenant-b"}, + {"tenant-root", "tenant-gone", "tenant-root"}, + {"tenant-a", "tenant-a", "tenant-a"}, + {"tenant-a", "tenant-a-child", "tenant-a"}, + {"tenant-a", "tenant-root", "tenant-root"}, + {"tenant-a", "tenant-inherits-root", "tenant-root"}, + {"tenant-a", "cozy-system", "tenant-a"}, + } + for _, tt := range tests { + r := NewStoreResolver(lister, tt.root) + got, err := r.Resolve(tt.ns) + if err != nil { + t.Fatalf("root=%s Resolve(%s) error: %v", tt.root, tt.ns, err) + } + if got != tt.want { + t.Errorf("root=%s Resolve(%s) = %s, want %s", tt.root, tt.ns, got, tt.want) + } + } +} From eb0d6e5957bbf7c853624469174245bcff140b79 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:15:34 +0700 Subject: [PATCH 205/518] feat(observability-proxy): rebuild logs queries from an allowlist vlselect has no notion of a tenant: whatever filter reaches it decides what is returned. The upstream parameters are therefore rebuilt from a per-endpoint allowlist with the namespace scope as the only extra_stream_filters value, so a client cannot widen it through extra_filters, tenant headers or an alternate parameter spelling. VictoriaLogs ANDs extra_stream_filters into every subquery, so union and in() queries pass through unparsed. What the query can still do is cost: a missing start scans the whole retention, a tiny step returns a bucket per millisecond, query has no default limit, and ignore_global_time_filter lifts the window altogether. Each of those is bounded or refused here, and a tail resume point becomes a capped start_offset. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 314 ++++++++++++++++++ .../observabilityproxy/logsparams_test.go | 272 +++++++++++++++ 2 files changed, 586 insertions(+) create mode 100644 internal/observabilityproxy/logsparams.go create mode 100644 internal/observabilityproxy/logsparams_test.go diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go new file mode 100644 index 000000000..cca53e62d --- /dev/null +++ b/internal/observabilityproxy/logsparams.go @@ -0,0 +1,314 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "fmt" + "net/http" + "net/url" + "regexp" + "strconv" + "strings" + "time" +) + +// LogsLimits bound what one logs request may make vlselect do. +type LogsLimits struct { + DefaultRange time.Duration + MaxRange time.Duration + MaxBuckets int + DefaultLimit int + MaxLimit int + MaxTailResume time.Duration +} + +// tailOffset is vlselect's default tail offset. The proxy never forwards a +// client offset, so the resume arithmetic can rely on this value. +const tailOffset = 5 * time.Second + +const maxHitsFields = 10 + +var logsEndpoints = map[string]bool{ + "query": true, "hits": true, "stats_query_range": true, "tail": true, + "field_values": true, "stream_field_values": true, "facets": true, +} + +// BuildLogsParams rebuilds the upstream parameters from scratch: only the +// allowlisted parameters of the endpoint are read from the client, and the +// namespace scope is added as the only extra_stream_filters value. +// VictoriaLogs ANDs extra_stream_filters into the query and every subquery +// (union, join, in()), which is what lets the query itself pass through +// unparsed. +func BuildLogsParams(endpoint string, in url.Values, namespace string, now time.Time, lim LogsLimits) (url.Values, *HTTPError) { + if !logsEndpoints[endpoint] { + return nil, errorf(http.StatusNotFound, "not found") + } + query := in.Get("query") + if query == "" { + query = "*" + } + // The option lifts the time window this function enforces, and on tail + // it rescans the whole retention every tick. + if strings.Contains(strings.ToLower(query), "ignore_global_time_filter") { + return nil, errorf(http.StatusBadRequest, "ignore_global_time_filter is not allowed") + } + out := url.Values{} + out.Set("query", query) + out.Set("extra_stream_filters", "{kubernetes_namespace_name="+strconv.Quote(namespace)+"}") + + if endpoint == "tail" { + return out, tailParams(in, out, now, lim) + } + + start, end, herr := window(in, now, lim) + if herr != nil { + return nil, herr + } + out.Set("start", start.Format(time.RFC3339Nano)) + out.Set("end", end.Format(time.RFC3339Nano)) + + switch endpoint { + case "query": + herr = setLimit(out, in, "limit", lim.DefaultLimit, lim.MaxLimit) + case "hits": + if herr = setStep(out, in, end.Sub(start), lim); herr != nil { + return nil, herr + } + fields := in["field"] + if len(fields) > maxHitsFields { + return nil, errorf(http.StatusBadRequest, "at most %d field parameters are allowed", maxHitsFields) + } + for _, f := range fields { + if f != "" { + out.Add("field", f) + } + } + if in.Has("fields_limit") { + herr = setLimit(out, in, "fields_limit", 0, lim.MaxLimit) + } + case "stats_query_range": + herr = setStep(out, in, end.Sub(start), lim) + case "field_values", "stream_field_values": + if in.Get("field") == "" { + return nil, errorf(http.StatusBadRequest, "field is required") + } + out.Set("field", in.Get("field")) + if f := in.Get("filter"); f != "" { + out.Set("filter", f) + } + herr = setLimit(out, in, "limit", lim.DefaultLimit, lim.MaxLimit) + case "facets": + for _, k := range []string{"limit", "max_values_per_field", "max_value_len"} { + if k == "limit" || in.Has(k) { + if herr = setLimit(out, in, k, lim.DefaultLimit, lim.MaxLimit); herr != nil { + return nil, herr + } + } + } + if v := in.Get("keep_const_fields"); v == "1" || v == "true" { + out.Set("keep_const_fields", "1") + } + } + if herr != nil { + return nil, herr + } + return out, nil +} + +// window derives a finite [start, end]: VictoriaLogs otherwise starts at +// the oldest stored entry and scans the whole retention. +func window(in url.Values, now time.Time, lim LogsLimits) (time.Time, time.Time, *HTTPError) { + end := now + if v := in.Get("end"); v != "" { + t, err := parseTime(v, now) + if err != nil { + return time.Time{}, time.Time{}, errorf(http.StatusBadRequest, "invalid end: %v", err) + } + end = t + } + start := end.Add(-lim.DefaultRange) + if v := in.Get("start"); v != "" { + t, err := parseTime(v, now) + if err != nil { + return time.Time{}, time.Time{}, errorf(http.StatusBadRequest, "invalid start: %v", err) + } + start = t + } + if start.After(end) { + return time.Time{}, time.Time{}, errorf(http.StatusBadRequest, "start is after end") + } + if end.Sub(start) > lim.MaxRange { + start = end.Add(-lim.MaxRange) + } + return start.UTC(), end.UTC(), nil +} + +// setStep raises step so that hits and stats_query_range, which return one +// bucket per step, return at most MaxBuckets buckets over the window. +func setStep(out, in url.Values, span time.Duration, lim LogsLimits) *HTTPError { + minStep := max((span+time.Duration(lim.MaxBuckets)-1)/time.Duration(lim.MaxBuckets), time.Second) + step := minStep + if v := in.Get("step"); v != "" { + d, err := parseDuration(v) + if err != nil || d <= 0 { + return errorf(http.StatusBadRequest, "invalid step %q", v) + } + step = max(d, minStep) + } + out.Set("step", formatDuration(step)) + return nil +} + +func setLimit(out, in url.Values, key string, def, maxv int) *HTTPError { + n := def + if v := in.Get(key); v != "" { + parsed, err := strconv.Atoi(v) + if err != nil || parsed <= 0 { + return errorf(http.StatusBadRequest, "invalid %s %q", key, v) + } + n = parsed + } + out.Set(key, strconv.Itoa(min(n, maxv))) + return nil +} + +// tailParams turns a resume point into a capped start_offset. A reconnect +// older than the cap resumes at the cap; the gap needs a historical query. +func tailParams(in, out url.Values, now time.Time, lim LogsLimits) *HTTPError { + var start, end time.Time + for _, k := range []string{"start", "end"} { + v := in.Get(k) + if v == "" { + continue + } + t, err := parseTime(v, now) + if err != nil { + return errorf(http.StatusBadRequest, "invalid %s: %v", k, err) + } + out.Set(k, t.UTC().Format(time.RFC3339Nano)) + if k == "start" { + start = t + } else { + end = t + } + } + if !start.IsZero() && !end.IsZero() && start.After(end) { + return errorf(http.StatusBadRequest, "start is after end") + } + + since, offset := in.Get("since"), in.Get("start_offset") + var d time.Duration + switch { + case since != "" && offset != "": + return errorf(http.StatusBadRequest, "since and start_offset are mutually exclusive") + case since != "": + t, err := parseTime(since, now) + if err != nil { + return errorf(http.StatusBadRequest, "invalid since: %v", err) + } + d = now.Add(-tailOffset).Sub(t) + case offset != "": + parsed, err := parseDuration(offset) + if err != nil { + return errorf(http.StatusBadRequest, "invalid start_offset %q", offset) + } + d = parsed + default: + return nil + } + out.Set("start_offset", formatDuration(min(max(d, 0), lim.MaxTailResume))) + return nil +} + +// formatDuration writes whole milliseconds, a form every VictoriaLogs +// duration parser accepts. +func formatDuration(d time.Duration) string { + return fmt.Sprintf("%dms", d.Milliseconds()) +} + +var durationToken = regexp.MustCompile(`^(\d+(?:\.\d+)?)(ms|s|m|h|d|w|y)`) + +var durationUnits = map[string]time.Duration{ + "ms": time.Millisecond, "s": time.Second, "m": time.Minute, "h": time.Hour, + "d": 24 * time.Hour, "w": 7 * 24 * time.Hour, "y": 365 * 24 * time.Hour, +} + +// parseDuration accepts the LogsQL duration forms (1h30m, 1d, 2w) and bare +// seconds; Go's parser knows neither d nor w. +func parseDuration(s string) (time.Duration, error) { + if f, err := strconv.ParseFloat(s, 64); err == nil { + if f < 0 { + return 0, fmt.Errorf("negative duration %q", s) + } + return time.Duration(f * float64(time.Second)), nil + } + if s == "" { + return 0, fmt.Errorf("empty duration") + } + var total time.Duration + for rest := s; rest != ""; { + m := durationToken.FindStringSubmatch(rest) + if m == nil { + return 0, fmt.Errorf("invalid duration %q", s) + } + f, _ := strconv.ParseFloat(m[1], 64) + total += time.Duration(f * float64(durationUnits[m[2]])) + rest = rest[len(m[0]):] + } + return total, nil +} + +// parseTime accepts "now", unix seconds with an optional fraction, RFC3339, +// and a duration meaning that long before now (VictoriaLogs' "start=3h"). +// Anything else is refused rather than guessed, since a misread timestamp +// silently moves the window. +func parseTime(s string, now time.Time) (time.Time, error) { + if s == "now" { + return now, nil + } + if sec, frac, ok := strings.Cut(s, "."); isDigits(sec) && (!ok || isDigits(frac)) { + secs, err := strconv.ParseInt(sec, 10, 64) + if err != nil { + return time.Time{}, err + } + var nanos int64 + if ok { + frac = (frac + "000000000")[:9] + nanos, _ = strconv.ParseInt(frac, 10, 64) + } + return time.Unix(secs, nanos), nil + } + if t, err := time.Parse(time.RFC3339Nano, s); err == nil { + return t, nil + } + if d, err := parseDuration(s); err == nil { + return now.Add(-d), nil + } + return time.Time{}, fmt.Errorf("unsupported time %q", s) +} + +func isDigits(s string) bool { + if s == "" { + return false + } + for _, r := range s { + if r < '0' || r > '9' { + return false + } + } + return true +} diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go new file mode 100644 index 000000000..1ebc03fd9 --- /dev/null +++ b/internal/observabilityproxy/logsparams_test.go @@ -0,0 +1,272 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "net/http" + "net/url" + "sort" + "strings" + "testing" + "time" +) + +var testNow = time.Date(2026, 11, 4, 12, 0, 0, 0, time.UTC) + +func testLimits() LogsLimits { + return LogsLimits{ + DefaultRange: time.Hour, + MaxRange: 7 * 24 * time.Hour, + MaxBuckets: 1000, + DefaultLimit: 1000, + MaxLimit: 5000, + MaxTailResume: 15 * time.Minute, + } +} + +const wantScope = `{kubernetes_namespace_name="tenant-a"}` + +func build(t *testing.T, endpoint string, in url.Values) url.Values { + t.Helper() + out, err := BuildLogsParams(endpoint, in, "tenant-a", testNow, testLimits()) + if err != nil { + t.Fatalf("BuildLogsParams(%s, %v) error: %v", endpoint, in, err) + } + return out +} + +func buildErr(t *testing.T, endpoint string, in url.Values) *HTTPError { + t.Helper() + out, err := BuildLogsParams(endpoint, in, "tenant-a", testNow, testLimits()) + if err == nil { + t.Fatalf("BuildLogsParams(%s, %v) = %v, want error", endpoint, in, out) + } + return err +} + +func TestBuildLogsParamsAllowlist(t *testing.T) { + allowed := map[string][]string{ + "query": {"query", "start", "end", "limit"}, + "hits": {"query", "start", "end", "step", "field", "fields_limit"}, + "stats_query_range": {"query", "start", "end", "step"}, + "field_values": {"query", "start", "end", "field", "limit", "filter"}, + "stream_field_values": {"query", "start", "end", "field", "limit", "filter"}, + "facets": {"query", "start", "end", "limit", "max_values_per_field", "max_value_len", "keep_const_fields"}, + "tail": {"query", "start", "end", "start_offset"}, + } + hostile := url.Values{ + "extra_filters": {`{kubernetes_namespace_name="tenant-b"}`}, + "extra_filters[]": {`kubernetes_namespace_name:~".*"`}, + "extra_stream_filters": {`{kubernetes_namespace_name=~".*"}`}, + "AccountID": {"7"}, + "ProjectID": {"7"}, + "tenant": {"tenant-b"}, + "offset": {"1h"}, + "refresh_interval": {"1ms"}, + "ignore_pipes": {"1"}, + "format": {"csv"}, + "timeout": {"1h"}, + "query": {"error"}, + "field": {"kubernetes_pod_name"}, + "filter": {"api"}, + "keep_const_fields": {"1"}, + "since": {"2026-11-04T11:59:00Z"}, + } + for endpoint, keys := range allowed { + t.Run(endpoint, func(t *testing.T) { + out := build(t, endpoint, hostile) + want := map[string]bool{"extra_stream_filters": true} + for _, k := range keys { + want[k] = true + } + for k := range out { + if !want[k] { + t.Errorf("forwarded non-allowlisted parameter %q", k) + } + } + if got := out["extra_stream_filters"]; len(got) != 1 || got[0] != wantScope { + t.Errorf("extra_stream_filters = %q, want exactly [%s]", got, wantScope) + } + if _, ok := out["extra_filters"]; ok { + t.Error("extra_filters forwarded") + } + }) + } +} + +func TestBuildLogsParamsRejects(t *testing.T) { + tests := []struct { + name string + endpoint string + in url.Values + code int + }{ + {"unknown endpoint", "tenant_ids", url.Values{}, http.StatusNotFound}, + {"stats_query is not proxied", "stats_query", url.Values{}, http.StatusNotFound}, + {"ignore_global_time_filter", "query", url.Values{"query": {`* options(ignore_global_time_filter=true)`}}, http.StatusBadRequest}, + {"ignore_global_time_filter in subquery", "hits", url.Values{"query": {`* | union (options(ignore_global_time_filter=true) error)`}}, http.StatusBadRequest}, + {"ignore_global_time_filter on tail", "tail", url.Values{"query": {`options(IGNORE_GLOBAL_TIME_FILTER=true) *`}}, http.StatusBadRequest}, + {"inverted range", "query", url.Values{"start": {"2026-11-04T11:00:00Z"}, "end": {"2026-11-04T10:00:00Z"}}, http.StatusBadRequest}, + {"inverted range on tail", "tail", url.Values{"start": {"2026-11-04T11:00:00Z"}, "end": {"2026-11-04T10:00:00Z"}}, http.StatusBadRequest}, + {"bad start", "query", url.Values{"start": {"yesterday"}}, http.StatusBadRequest}, + {"bad limit", "query", url.Values{"limit": {"-1"}}, http.StatusBadRequest}, + {"zero limit", "query", url.Values{"limit": {"0"}}, http.StatusBadRequest}, + {"bad step", "hits", url.Values{"step": {"0s"}}, http.StatusBadRequest}, + {"field required", "field_values", url.Values{}, http.StatusBadRequest}, + {"too many hits fields", "hits", url.Values{"field": strings.Fields("a b c d e f g h i j k")}, http.StatusBadRequest}, + {"since and start_offset", "tail", url.Values{"since": {"2026-11-04T11:59:00Z"}, "start_offset": {"1m"}}, http.StatusBadRequest}, + {"bad since", "tail", url.Values{"since": {"soon"}}, http.StatusBadRequest}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if err := buildErr(t, tt.endpoint, tt.in); err.Code != tt.code { + t.Fatalf("code = %d (%s), want %d", err.Code, err.Message, tt.code) + } + }) + } +} + +// VictoriaLogs propagates extra_stream_filters into every subquery +// (docs.victoriametrics.com/victorialogs/querying/), so union and in() +// queries are forwarded verbatim with the scope beside them. +func TestBuildLogsParamsSubqueriesKeepScope(t *testing.T) { + for _, q := range []string{ + `* | union ({kubernetes_namespace_name="tenant-b"})`, + `_stream_id:in({kubernetes_namespace_name="tenant-b"} | fields _stream_id)`, + } { + out := build(t, "query", url.Values{"query": {q}}) + if out.Get("query") != q || out.Get("extra_stream_filters") != wantScope { + t.Errorf("query %q forwarded as %v", q, out) + } + } +} + +func TestBuildLogsParamsWindow(t *testing.T) { + ts := func(s string) string { + v, err := time.Parse(time.RFC3339, s) + if err != nil { + t.Fatal(err) + } + return v.UTC().Format(time.RFC3339Nano) + } + tests := []struct { + name string + in url.Values + wantStart, wantEnd string + }{ + {"defaults", url.Values{}, ts("2026-11-04T11:00:00Z"), ts("2026-11-04T12:00:00Z")}, + {"start only", url.Values{"start": {"2026-11-04T09:00:00Z"}}, ts("2026-11-04T09:00:00Z"), ts("2026-11-04T12:00:00Z")}, + {"end only", url.Values{"end": {"2026-11-03T00:00:00Z"}}, ts("2026-11-02T23:00:00Z"), ts("2026-11-03T00:00:00Z")}, + {"unix seconds", url.Values{"start": {"1793786400"}, "end": {"1793788200.5"}}, "2026-11-04T10:00:00Z", "2026-11-04T10:30:00.5Z"}, + {"relative", url.Values{"start": {"3h"}, "end": {"now"}}, ts("2026-11-04T09:00:00Z"), ts("2026-11-04T12:00:00Z")}, + {"relative days", url.Values{"start": {"1d"}}, ts("2026-11-03T12:00:00Z"), ts("2026-11-04T12:00:00Z")}, + {"over-wide clamped", url.Values{"start": {"2026-01-01T00:00:00Z"}}, ts("2026-10-28T12:00:00Z"), ts("2026-11-04T12:00:00Z")}, + {"empty range allowed", url.Values{"start": {"2026-11-04T10:00:00Z"}, "end": {"2026-11-04T10:00:00Z"}}, ts("2026-11-04T10:00:00Z"), ts("2026-11-04T10:00:00Z")}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + for _, endpoint := range []string{"query", "hits", "stats_query_range", "facets"} { + out := build(t, endpoint, tt.in) + if out.Get("start") != tt.wantStart || out.Get("end") != tt.wantEnd { + t.Errorf("%s: window = [%s, %s], want [%s, %s]", endpoint, out.Get("start"), out.Get("end"), tt.wantStart, tt.wantEnd) + } + } + }) + } +} + +func TestBuildLogsParamsStepAndLimit(t *testing.T) { + day := url.Values{"start": {"2026-11-03T12:00:00Z"}} + with := func(base url.Values, k, v string) url.Values { + c := url.Values{} + for key, vals := range base { + c[key] = vals + } + c.Set(k, v) + return c + } + // 24h over at most 1000 buckets is 86.4s. + if got := build(t, "hits", with(day, "step", "1s")).Get("step"); got != "86400ms" { + t.Errorf("tiny step = %s, want clamped to 86400ms", got) + } + if got := build(t, "stats_query_range", with(day, "step", "1h")).Get("step"); got != "3600000ms" { + t.Errorf("1h step = %s", got) + } + if got := build(t, "stats_query_range", with(day, "step", "1w")).Get("step"); got != "604800000ms" { + t.Errorf("1w step = %s", got) + } + if got := build(t, "hits", day).Get("step"); got != "86400ms" { + t.Errorf("omitted step = %s, want the bucket-bounded minimum", got) + } + if got := build(t, "hits", url.Values{"start": {"2026-11-04T11:59:59Z"}}).Get("step"); got != "1000ms" { + t.Errorf("omitted step on a tiny window = %s, want 1000ms floor", got) + } + + if got := build(t, "query", url.Values{}).Get("limit"); got != "1000" { + t.Errorf("default limit = %s", got) + } + if got := build(t, "query", url.Values{"limit": {"10"}}).Get("limit"); got != "10" { + t.Errorf("limit = %s", got) + } + for _, endpoint := range []string{"query", "field_values", "facets"} { + in := url.Values{"limit": {"1000000"}, "field": {"f"}} + if got := build(t, endpoint, in).Get("limit"); got != "5000" { + t.Errorf("%s: limit = %s, want clamped to 5000", endpoint, got) + } + } + facets := build(t, "facets", url.Values{"max_values_per_field": {"99999"}, "max_value_len": {"99999"}}) + if facets.Get("max_values_per_field") != "5000" || facets.Get("max_value_len") != "5000" { + t.Errorf("facets caps not clamped: %v", facets) + } + hits := build(t, "hits", url.Values{"field": {"b", "a"}, "fields_limit": {"99999"}}) + fields := hits["field"] + sort.Strings(fields) + if strings.Join(fields, ",") != "a,b" || hits.Get("fields_limit") != "5000" { + t.Errorf("hits fields = %v", hits) + } +} + +func TestBuildLogsParamsTail(t *testing.T) { + tests := []struct { + name string + in url.Values + want string + }{ + {"live only", url.Values{}, ""}, + // now - 5s tail offset - since = 55s + {"since within cap", url.Values{"since": {"2026-11-04T11:59:00Z"}}, "55000ms"}, + {"since older than cap", url.Values{"since": {"2026-11-04T08:00:00Z"}}, "900000ms"}, + {"since in the future", url.Values{"since": {"2026-11-04T13:00:00Z"}}, "0ms"}, + {"start_offset within cap", url.Values{"start_offset": {"2m"}}, "120000ms"}, + {"start_offset over cap", url.Values{"start_offset": {"30d"}}, "900000ms"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + out := build(t, "tail", tt.in) + if got := out.Get("start_offset"); got != tt.want { + t.Errorf("start_offset = %q, want %q", got, tt.want) + } + if out.Has("start") || out.Has("end") || out.Has("limit") { + t.Errorf("tail got a derived window or limit: %v", out) + } + }) + } + out := build(t, "tail", url.Values{"start": {"2026-11-04T11:00:00Z"}}) + if out.Get("start") != "2026-11-04T11:00:00Z" { + t.Errorf("tail start = %q, want the client value ANDed into each tick", out.Get("start")) + } +} From 61275df240933a92a4426d44faa6a0c9d7e6d562 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:16:29 +0700 Subject: [PATCH 206/518] feat(observability-proxy): cap per-user concurrency, rate and open tails The log stores are shared by every namespace routed to them, and the root store also serves platform Grafana, so one tenant's dashboard must not be able to monopolise vlselect. Live tails are counted apart from request concurrency because they hold their slot for minutes and would otherwise starve the same user's panels. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/limits.go | 146 +++++++++++++++++++++ internal/observabilityproxy/limits_test.go | 104 +++++++++++++++ 2 files changed, 250 insertions(+) create mode 100644 internal/observabilityproxy/limits.go create mode 100644 internal/observabilityproxy/limits_test.go diff --git a/internal/observabilityproxy/limits.go b/internal/observabilityproxy/limits.go new file mode 100644 index 000000000..588a4f29e --- /dev/null +++ b/internal/observabilityproxy/limits.go @@ -0,0 +1,146 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "sync" + "time" +) + +// LimiterConfig holds the per-user and total caps. A zero cap disables it. +type LimiterConfig struct { + MaxConcurrentPerUser int + RatePerUser float64 + RateBurst int + MaxTailsPerUser int + MaxTails int +} + +// limiter keeps open tails apart from the request concurrency budget: a +// tail holds its slot for minutes, and counting it there would starve the +// same user's dashboard panels. +type limiter struct { + cfg LimiterConfig + now func() time.Time + mu sync.Mutex + active map[string]int + tails map[string]int + total int + buckets map[string]*bucket +} + +type bucket struct { + tokens float64 + last time.Time +} + +func newLimiter(cfg LimiterConfig) *limiter { + return &limiter{ + cfg: cfg, + now: time.Now, + active: map[string]int{}, + tails: map[string]int{}, + buckets: map[string]*bucket{}, + } +} + +func (l *limiter) allowRate(user string) bool { + if l.cfg.RatePerUser <= 0 { + return true + } + l.mu.Lock() + defer l.mu.Unlock() + now := l.now() + burst := float64(max(l.cfg.RateBurst, 1)) + b, ok := l.buckets[user] + if !ok { + if len(l.buckets) >= ttlCacheMaxEntries { + l.sweepBuckets(now, burst) + } + b = &bucket{tokens: burst, last: now} + l.buckets[user] = b + } + b.tokens = min(burst, b.tokens+now.Sub(b.last).Seconds()*l.cfg.RatePerUser) + b.last = now + if b.tokens < 1 { + return false + } + b.tokens-- + return true +} + +// sweepBuckets drops buckets that have refilled completely; recreating one +// later starts it full, which is the same state. +func (l *limiter) sweepBuckets(now time.Time, burst float64) { + for user, b := range l.buckets { + if b.tokens+now.Sub(b.last).Seconds()*l.cfg.RatePerUser >= burst { + delete(l.buckets, user) + } + } +} + +func (l *limiter) acquire(user string) (func(), bool) { + l.mu.Lock() + defer l.mu.Unlock() + if l.cfg.MaxConcurrentPerUser > 0 && l.active[user] >= l.cfg.MaxConcurrentPerUser { + return nil, false + } + l.active[user]++ + return l.releaser(func() { decrement(l.active, user) }), true +} + +func (l *limiter) acquireTail(user string) (func(), bool) { + l.mu.Lock() + defer l.mu.Unlock() + if l.cfg.MaxTailsPerUser > 0 && l.tails[user] >= l.cfg.MaxTailsPerUser { + return nil, false + } + if l.cfg.MaxTails > 0 && l.total >= l.cfg.MaxTails { + return nil, false + } + l.tails[user]++ + l.total++ + return l.releaser(func() { + decrement(l.tails, user) + l.total-- + }), true +} + +func (l *limiter) openTails() int { + l.mu.Lock() + defer l.mu.Unlock() + return l.total +} + +func (l *limiter) releaser(release func()) func() { + var once sync.Once + return func() { + once.Do(func() { + l.mu.Lock() + defer l.mu.Unlock() + release() + }) + } +} + +func decrement(m map[string]int, user string) { + if m[user] <= 1 { + delete(m, user) + return + } + m[user]-- +} diff --git a/internal/observabilityproxy/limits_test.go b/internal/observabilityproxy/limits_test.go new file mode 100644 index 000000000..167598971 --- /dev/null +++ b/internal/observabilityproxy/limits_test.go @@ -0,0 +1,104 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "testing" + "time" +) + +func TestLimiterConcurrency(t *testing.T) { + l := newLimiter(LimiterConfig{MaxConcurrentPerUser: 2}) + r1, ok1 := l.acquire("alice") + _, ok2 := l.acquire("alice") + if !ok1 || !ok2 { + t.Fatal("first two requests refused") + } + if _, ok := l.acquire("alice"); ok { + t.Fatal("third concurrent request admitted") + } + if _, ok := l.acquire("bob"); !ok { + t.Fatal("another user is limited by alice's requests") + } + r1() + r1() + if _, ok := l.acquire("alice"); !ok { + t.Fatal("slot not returned on release") + } + if _, ok := l.acquire("alice"); ok { + t.Fatal("double release freed a second slot") + } +} + +func TestLimiterTails(t *testing.T) { + l := newLimiter(LimiterConfig{MaxConcurrentPerUser: 1, MaxTailsPerUser: 2, MaxTails: 3}) + if _, ok := l.acquire("alice"); !ok { + t.Fatal("request refused") + } + a1, ok1 := l.acquireTail("alice") + _, ok2 := l.acquireTail("alice") + if !ok1 || !ok2 { + t.Fatal("tails share the request concurrency budget") + } + if _, ok := l.acquireTail("alice"); ok { + t.Fatal("per-user tail cap not enforced") + } + if _, ok := l.acquireTail("bob"); !ok { + t.Fatal("bob refused below the total cap") + } + if _, ok := l.acquireTail("carol"); ok { + t.Fatal("total tail cap not enforced") + } + if got := l.openTails(); got != 3 { + t.Fatalf("openTails = %d, want 3", got) + } + a1() + if _, ok := l.acquireTail("carol"); !ok { + t.Fatal("total slot not returned on release") + } +} + +func TestLimiterRate(t *testing.T) { + now := time.Unix(1000, 0) + l := newLimiter(LimiterConfig{RatePerUser: 2, RateBurst: 2}) + l.now = func() time.Time { return now } + if !l.allowRate("alice") || !l.allowRate("alice") { + t.Fatal("burst refused") + } + if l.allowRate("alice") { + t.Fatal("request over the burst admitted") + } + if !l.allowRate("bob") { + t.Fatal("bob limited by alice's rate") + } + now = now.Add(500 * time.Millisecond) + if !l.allowRate("alice") { + t.Fatal("token not refilled after 1/rate") + } + if l.allowRate("alice") { + t.Fatal("refilled more than elapsed*rate") + } +} + +func TestLimiterRateDisabled(t *testing.T) { + l := newLimiter(LimiterConfig{}) + for range 100 { + if !l.allowRate("alice") { + t.Fatal("rate limit applied while disabled") + } + } +} From 689405127d78d904532c224bc91c2de0167f050c Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:31:28 +0700 Subject: [PATCH 207/518] feat(observability-proxy): serve the namespace logs root Tenants get a way to read their namespace's logs without Grafana and without reaching vlselect directly, which would give them neither an RBAC check nor a forced scope. The handler authenticates, authorizes from the path, rebuilds the upstream request with fixed tenant headers and no client headers, and returns VictoriaLogs' native response. A live tail is flushed after every line, is not subject to the request timeout, and is closed once idle, so it survives far past the apiserver request timeout without leaving forgotten browser tabs holding streams open. An unreachable store answers 502 so the dashboard can tell an outage from an empty result. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 270 +++++++++++++++ internal/observabilityproxy/server_test.go | 364 +++++++++++++++++++++ 2 files changed, 634 insertions(+) create mode 100644 internal/observabilityproxy/server.go create mode 100644 internal/observabilityproxy/server_test.go diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go new file mode 100644 index 000000000..a0e2757c5 --- /dev/null +++ b/internal/observabilityproxy/server.go @@ -0,0 +1,270 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "bufio" + "context" + "errors" + "fmt" + "io" + "log/slog" + "net/http" + "strconv" + "strings" + "time" + + "github.com/prometheus/client_golang/prometheus" +) + +// maxFormBytes bounds a POST body; LogsQL queries are a few KiB at most. +const maxFormBytes = 1 << 20 + +// Config is the runtime configuration of the proxy handler. +type Config struct { + ClusterDomain string + // UpstreamURL maps a store namespace to the vlselect base URL; nil + // means the in-cluster Service of the "generic" logs storage. + UpstreamURL func(store string) string + Logs LogsLimits + Limiter LimiterConfig + RequestTimeout time.Duration + TailIdleTimeout time.Duration +} + +// StoreLookup resolves a request namespace to the namespace of its store. +type StoreLookup interface { + Resolve(namespace string) (string, error) +} + +// Server serves the tenant observability roots. +type Server struct { + cfg Config + auth *Authorizer + stores StoreLookup + client *http.Client + limits *limiter + now func() time.Time + requests *prometheus.CounterVec +} + +func NewServer(cfg Config, auth *Authorizer, stores StoreLookup, reg prometheus.Registerer) *Server { + if cfg.UpstreamURL == nil { + domain := cfg.ClusterDomain + cfg.UpstreamURL = func(store string) string { + return fmt.Sprintf("http://vlselect-generic.%s.svc.%s:9471", store, domain) + } + } + s := &Server{ + cfg: cfg, + auth: auth, + stores: stores, + // No client timeout: it would cut live tails. Non-tail requests are + // bounded by a context deadline instead. + client: &http.Client{Transport: http.DefaultTransport.(*http.Transport).Clone()}, + limits: newLimiter(cfg.Limiter), + now: time.Now, + requests: prometheus.NewCounterVec(prometheus.CounterOpts{ + Name: "observability_proxy_requests_total", + Help: "Requests served, by route kind and status code.", + }, []string{"route", "code"}), + } + if reg != nil { + reg.MustRegister(s.requests, prometheus.NewGaugeFunc(prometheus.GaugeOpts{ + Name: "observability_proxy_open_tails", + Help: "Live tails currently open.", + }, func() float64 { return float64(s.limits.openTails()) })) + } + return s +} + +type statusRecorder struct { + http.ResponseWriter + code int +} + +func (r *statusRecorder) WriteHeader(code int) { + if r.code == 0 { + r.code = code + } + r.ResponseWriter.WriteHeader(code) +} + +func (r *statusRecorder) Write(b []byte) (int, error) { + if r.code == 0 { + r.code = http.StatusOK + } + return r.ResponseWriter.Write(b) +} + +func (r *statusRecorder) Unwrap() http.ResponseWriter { return r.ResponseWriter } + +func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { + rec := &statusRecorder{ResponseWriter: w} + route, herr := ParseRoute(r.URL.EscapedPath()) + if herr == nil { + herr = s.serve(rec, r, route) + } + if herr != nil { + writeError(rec, herr) + } + s.requests.WithLabelValues(route.Kind.String(), strconv.Itoa(rec.code)).Inc() +} + +func writeError(w http.ResponseWriter, e *HTTPError) { + if e.Code == http.StatusUnauthorized { + w.Header().Set("WWW-Authenticate", `Bearer realm="cozystack"`) + } + http.Error(w, e.Message, e.Code) +} + +// serve answers an unimplemented root before authenticating, so later +// waves only add a case here. +func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTTPError { + if route.Kind != NamespaceLogs { + return errorf(http.StatusNotFound, "not found") + } + user, herr := s.authorize(r, route) + if herr != nil { + return herr + } + if r.Method != http.MethodGet && r.Method != http.MethodPost { + w.Header().Set("Allow", "GET, POST") + return errorf(http.StatusMethodNotAllowed, "method not allowed") + } + if !s.limits.allowRate(user) { + return errorf(http.StatusUnprocessableEntity, "request rate limit exceeded") + } + endpoint, ok := strings.CutPrefix(route.Endpoint, "select/logsql/") + if !ok { + return errorf(http.StatusNotFound, "not found") + } + r.Body = http.MaxBytesReader(w, r.Body, maxFormBytes) + if err := r.ParseForm(); err != nil { + return errorf(http.StatusBadRequest, "invalid form: %v", err) + } + params, herr := BuildLogsParams(endpoint, r.Form, route.Namespace, s.now(), s.cfg.Logs) + if herr != nil { + return herr + } + + tail := endpoint == "tail" + acquire := s.limits.acquire + if tail { + acquire = s.limits.acquireTail + } + release, ok := acquire(user) + if !ok { + return errorf(http.StatusUnprocessableEntity, "too many concurrent requests") + } + defer release() + + store, err := s.stores.Resolve(route.Namespace) + if err != nil { + slog.Error("resolving log store", "namespace", route.Namespace, "error", err) + return errorf(http.StatusServiceUnavailable, "cannot resolve the log store") + } + + ctx, cancel := context.WithCancel(r.Context()) + defer cancel() + if !tail && s.cfg.RequestTimeout > 0 { + ctx, cancel = context.WithTimeout(ctx, s.cfg.RequestTimeout) + defer cancel() + } + req, err := http.NewRequestWithContext(ctx, http.MethodPost, + s.cfg.UpstreamURL(store)+"/select/logsql/"+endpoint, strings.NewReader(params.Encode())) + if err != nil { + return errorf(http.StatusInternalServerError, "building upstream request: %v", err) + } + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + req.Header.Set("AccountID", "0") + req.Header.Set("ProjectID", "0") + resp, err := s.client.Do(req) + if err != nil { + slog.Warn("log store unreachable", "store", store, "error", err) + return errorf(http.StatusBadGateway, "log store unreachable") + } + defer func() { _ = resp.Body.Close() }() + + if ct := resp.Header.Get("Content-Type"); ct != "" { + w.Header().Set("Content-Type", ct) + } + w.WriteHeader(resp.StatusCode) + rc := http.NewResponseController(w) + _ = rc.Flush() + if tail { + s.streamLines(w, rc, resp.Body, cancel) + return nil + } + if _, err := io.Copy(w, resp.Body); err != nil && !errors.Is(err, context.Canceled) { + slog.Debug("copying upstream response", "error", err) + } + return nil +} + +func (s *Server) authorize(r *http.Request, route Route) (string, *HTTPError) { + token, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ") + if !ok { + return "", errorf(http.StatusUnauthorized, "missing bearer token") + } + user, err := s.auth.Authenticate(r.Context(), strings.TrimSpace(token)) + if errors.Is(err, ErrUnauthenticated) { + return "", errorf(http.StatusUnauthorized, "invalid bearer token") + } + if err != nil { + slog.Error("token review failed", "error", err) + return "", errorf(http.StatusServiceUnavailable, "authentication unavailable") + } + allowed, err := s.auth.Authorize(r.Context(), user, route.ResourceAttributes()) + if err != nil { + slog.Error("subject access review failed", "error", err) + return "", errorf(http.StatusServiceUnavailable, "authorization unavailable") + } + if !allowed { + return "", errorf(http.StatusForbidden, "forbidden") + } + return user.Username, nil +} + +// streamLines flushes after every line so a live tail reaches the browser +// as it happens, and cancels the upstream stream once no line has arrived +// for the idle timeout. +func (s *Server) streamLines(w http.ResponseWriter, rc *http.ResponseController, body io.Reader, cancel context.CancelFunc) { + var idle *time.Timer + if s.cfg.TailIdleTimeout > 0 { + idle = time.AfterFunc(s.cfg.TailIdleTimeout, cancel) + defer idle.Stop() + } + rd := bufio.NewReader(body) + for { + line, err := rd.ReadBytes('\n') + if len(line) > 0 { + if idle != nil { + idle.Reset(s.cfg.TailIdleTimeout) + } + if _, werr := w.Write(line); werr != nil { + return + } + if ferr := rc.Flush(); ferr != nil { + return + } + } + if err != nil { + return + } + } +} diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go new file mode 100644 index 000000000..ed50f72a8 --- /dev/null +++ b/internal/observabilityproxy/server_test.go @@ -0,0 +1,364 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "bufio" + "io" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "sync" + "testing" + "time" + + authorizationv1 "k8s.io/api/authorization/v1" +) + +type upstreamRequest struct { + path string + form url.Values + headers http.Header +} + +type harness struct { + t *testing.T + kube *fakeKube + proxy *httptest.Server + server *Server + mu sync.Mutex + got []upstreamRequest + stores []string + upstream *httptest.Server +} + +type mapStores map[string]string + +func (m mapStores) Resolve(ns string) (string, error) { return m[ns], nil } + +// newHarness wires the proxy to a fake apiserver that knows tokens "good" +// (alice) and "bob-token" (bob), both allowed list logs in tenant-a only, +// and to an upstream served by handler. +func newHarness(t *testing.T, cfg Config, handler http.HandlerFunc) *harness { + h := &harness{t: t} + h.kube = newFakeKube(map[string]string{"good": "alice", "bob-token": "bob"}, func(_ string, a *authorizationv1.ResourceAttributes) bool { + return a.Namespace == "tenant-a" && a.Resource == "logs" + }) + h.upstream = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if err := r.ParseForm(); err != nil { + t.Errorf("upstream ParseForm: %v", err) + } + h.mu.Lock() + h.got = append(h.got, upstreamRequest{path: r.URL.Path, form: r.PostForm, headers: r.Header.Clone()}) + h.mu.Unlock() + if handler != nil { + handler(w, r) + return + } + w.Header().Set("Content-Type", "application/stream+json") + _, _ = io.WriteString(w, `{"_msg":"hello"}`+"\n") + })) + t.Cleanup(h.upstream.Close) + cfg.UpstreamURL = func(store string) string { + h.mu.Lock() + h.stores = append(h.stores, store) + h.mu.Unlock() + return h.upstream.URL + } + if cfg.Logs == (LogsLimits{}) { + cfg.Logs = testLimits() + } + h.server = NewServer(cfg, NewAuthorizer(h.kube.client, 10*time.Second), mapStores{"tenant-a": "tenant-a-store"}, nil) + h.server.now = func() time.Time { return testNow } + h.proxy = httptest.NewServer(h.server) + t.Cleanup(h.proxy.Close) + return h +} + +const logsRoot = "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/select/logsql/" + +func (h *harness) do(method, path, token string, body url.Values, headers map[string]string) *http.Response { + h.t.Helper() + var rd io.Reader + if body != nil { + rd = strings.NewReader(body.Encode()) + } + req, err := http.NewRequest(method, h.proxy.URL+path, rd) + if err != nil { + h.t.Fatal(err) + } + if body != nil { + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + } + if token != "" { + req.Header.Set("Authorization", "Bearer "+token) + } + for k, v := range headers { + req.Header.Set(k, v) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + h.t.Fatal(err) + } + return resp +} + +func (h *harness) status(method, path, token string) int { + h.t.Helper() + resp := h.do(method, path, token, nil, nil) + _, _ = io.Copy(io.Discard, resp.Body) + _ = resp.Body.Close() + return resp.StatusCode +} + +func (h *harness) requests() []upstreamRequest { + h.mu.Lock() + defer h.mu.Unlock() + return append([]upstreamRequest(nil), h.got...) +} + +func TestServerStatusCodes(t *testing.T) { + h := newHarness(t, Config{}, nil) + tests := []struct { + name, method, path, token string + want int + }{ + {"ok", http.MethodGet, logsRoot + "query?query=error", "good", http.StatusOK}, + {"missing token", http.MethodGet, logsRoot + "query", "", http.StatusUnauthorized}, + {"invalid token", http.MethodGet, logsRoot + "query", "bad", http.StatusUnauthorized}, + {"other namespace", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-b/logs/select/logsql/query", "good", http.StatusForbidden}, + {"invalid namespace", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/Tenant/logs/select/logsql/query", "good", http.StatusBadRequest}, + {"unknown root", http.MethodGet, "/api/v1/namespaces/tenant-a/pods", "good", http.StatusNotFound}, + {"namespace metrics not served yet", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/api/v1/query", "good", http.StatusNotFound}, + {"application log not served yet", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/log/select/logsql/query", "good", http.StatusNotFound}, + {"application metrics not served yet", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/metrics/api/v1/query", "good", http.StatusNotFound}, + {"tenant_ids not proxied", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/select/tenant_ids", "good", http.StatusNotFound}, + {"unknown logsql endpoint", http.MethodGet, logsRoot + "stats_query", "good", http.StatusNotFound}, + {"logs root itself", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/", "good", http.StatusNotFound}, + {"ignore_global_time_filter", http.MethodGet, logsRoot + "query?query=" + url.QueryEscape("options(ignore_global_time_filter=true) *"), "good", http.StatusBadRequest}, + {"inverted range", http.MethodGet, logsRoot + "query?start=2026-11-04T11:00:00Z&end=2026-11-04T10:00:00Z", "good", http.StatusBadRequest}, + {"delete", http.MethodDelete, logsRoot + "query", "good", http.StatusMethodNotAllowed}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := h.status(tt.method, tt.path, tt.token); got != tt.want { + t.Fatalf("status = %d, want %d", got, tt.want) + } + }) + } + if reqs := h.requests(); len(reqs) != 1 { + t.Fatalf("upstream saw %d requests, want only the allowed one", len(reqs)) + } +} + +func TestServerRebuildsUpstreamRequest(t *testing.T) { + h := newHarness(t, Config{}, nil) + hostile := url.Values{ + "query": {"error"}, + "extra_filters": {`{kubernetes_namespace_name="tenant-b"}`}, + "extra_stream_filters": {`{kubernetes_namespace_name=~".*"}`}, + "AccountID": {"9"}, + } + headers := map[string]string{"AccountID": "9", "ProjectID": "9", "X-Forwarded-For": "1.2.3.4", "Cookie": "s=1"} + + resp := h.do(http.MethodGet, logsRoot+"query?"+hostile.Encode(), "good", nil, headers) + _ = resp.Body.Close() + resp = h.do(http.MethodPost, logsRoot+"query?extra_filters="+url.QueryEscape(`{x="y"}`), "good", hostile, headers) + body, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusOK || string(body) != `{"_msg":"hello"}`+"\n" { + t.Fatalf("POST answered %d %q", resp.StatusCode, body) + } + if ct := resp.Header.Get("Content-Type"); ct != "application/stream+json" { + t.Fatalf("Content-Type = %q, want the upstream's", ct) + } + + reqs := h.requests() + if len(reqs) != 2 { + t.Fatalf("upstream saw %d requests", len(reqs)) + } + if reqs[0].form.Encode() != reqs[1].form.Encode() { + t.Fatalf("GET and POST built different upstream requests:\n%v\n%v", reqs[0].form, reqs[1].form) + } + if n := h.kube.sars.Load(); n != 1 { + t.Fatalf("SubjectAccessReviews = %d, want 1: POST must reuse the GET decision", n) + } + for _, r := range reqs { + if r.path != "/select/logsql/query" { + t.Errorf("upstream path = %s", r.path) + } + if got := r.form["extra_stream_filters"]; len(got) != 1 || got[0] != wantScope { + t.Errorf("extra_stream_filters = %q", got) + } + if r.form.Has("extra_filters") || r.form.Has("AccountID") { + t.Errorf("client scope parameter forwarded: %v", r.form) + } + if r.headers.Get("AccountID") != "0" || r.headers.Get("ProjectID") != "0" { + t.Errorf("tenant headers = %q/%q, want 0/0", r.headers.Get("AccountID"), r.headers.Get("ProjectID")) + } + for _, k := range []string{"Authorization", "X-Forwarded-For", "Cookie"} { + if r.headers.Get(k) != "" { + t.Errorf("client header %s forwarded", k) + } + } + } + for _, s := range h.stores { + if s != "tenant-a-store" { + t.Errorf("store = %s, want the resolved one", s) + } + } +} + +func TestServerUpstreamErrors(t *testing.T) { + h := newHarness(t, Config{}, func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, "cannot parse query", http.StatusBadRequest) + }) + if got := h.status(http.MethodGet, logsRoot+"query?query=|", "good"); got != http.StatusBadRequest { + t.Fatalf("upstream 400 answered as %d", got) + } + h.upstream.Close() + if got := h.status(http.MethodGet, logsRoot+"query", "good"); got != http.StatusBadGateway { + t.Fatalf("unreachable upstream answered %d, want 502", got) + } +} + +func TestServerLimits(t *testing.T) { + release := make(chan struct{}) + var once sync.Once + t.Cleanup(func() { once.Do(func() { close(release) }) }) + h := newHarness(t, Config{Limiter: LimiterConfig{MaxConcurrentPerUser: 1}}, func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + w.(http.Flusher).Flush() + <-release + }) + first := h.do(http.MethodGet, logsRoot+"query", "good", nil, nil) + defer func() { _ = first.Body.Close() }() + if got := h.status(http.MethodGet, logsRoot+"query", "good"); got != http.StatusUnprocessableEntity { + t.Fatalf("concurrent request answered %d, want 422", got) + } + once.Do(func() { close(release) }) + + h2 := newHarness(t, Config{Limiter: LimiterConfig{RatePerUser: 0.001, RateBurst: 1}}, nil) + if got := h2.status(http.MethodGet, logsRoot+"query", "good"); got != http.StatusOK { + t.Fatalf("first request answered %d", got) + } + if got := h2.status(http.MethodGet, logsRoot+"query", "good"); got != http.StatusUnprocessableEntity { + t.Fatalf("rate-limited request answered %d, want 422", got) + } +} + +// tailUpstream sends one line per value received on lines and holds the +// stream open until done is closed. +func tailUpstream(lines <-chan string, done <-chan struct{}) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/stream+json") + w.WriteHeader(http.StatusOK) + w.(http.Flusher).Flush() + for { + select { + case l := <-lines: + _, _ = io.WriteString(w, l+"\n") + w.(http.Flusher).Flush() + case <-done: + return + case <-r.Context().Done(): + return + } + } + } +} + +func TestServerTailFlushesEveryLine(t *testing.T) { + lines := make(chan string) + done := make(chan struct{}) + t.Cleanup(func() { close(done) }) + h := newHarness(t, Config{TailIdleTimeout: time.Minute, RequestTimeout: 50 * time.Millisecond}, tailUpstream(lines, done)) + + resp := h.do(http.MethodGet, logsRoot+"tail?query=*&since=2026-11-04T08:00:00Z", "good", nil, nil) + defer func() { _ = resp.Body.Close() }() + rd := bufio.NewReader(resp.Body) + for i, l := range []string{`{"_msg":"one"}`, `{"_msg":"two"}`} { + select { + case lines <- l: + case <-time.After(2 * time.Second): + t.Fatalf("upstream tail closed before line %d", i) + } + got := make(chan string, 1) + go func() { + s, _ := rd.ReadString('\n') + got <- s + }() + select { + case s := <-got: + if s != l+"\n" { + t.Fatalf("line %d = %q", i, s) + } + case <-time.After(2 * time.Second): + t.Fatalf("line %d not flushed to the client", i) + } + // Past the request timeout: a tail must not be cut by it. + time.Sleep(60 * time.Millisecond) + } + if r := h.requests()[0]; r.path != "/select/logsql/tail" || r.form.Get("start_offset") != "900000ms" { + t.Fatalf("tail upstream = %s %v, want start_offset capped at 15m", r.path, r.form) + } +} + +func TestServerTailCapsAndIdle(t *testing.T) { + done := make(chan struct{}) + t.Cleanup(func() { close(done) }) + h := newHarness(t, Config{ + Limiter: LimiterConfig{MaxTailsPerUser: 1, MaxTails: 2}, + TailIdleTimeout: 500 * time.Millisecond, + }, tailUpstream(make(chan string), done)) + + a := h.do(http.MethodGet, logsRoot+"tail", "good", nil, nil) + defer func() { _ = a.Body.Close() }() + if a.StatusCode != http.StatusOK { + t.Fatalf("first tail answered %d", a.StatusCode) + } + if got := h.status(http.MethodGet, logsRoot+"tail", "good"); got != http.StatusUnprocessableEntity { + t.Fatalf("second tail of one user answered %d, want 422", got) + } + b := h.do(http.MethodGet, logsRoot+"tail", "bob-token", nil, nil) + defer func() { _ = b.Body.Close() }() + if b.StatusCode != http.StatusOK { + t.Fatalf("bob's tail answered %d", b.StatusCode) + } + if got := h.server.limits.openTails(); got != 2 { + t.Fatalf("open tails = %d, want 2", got) + } + + closed := make(chan error, 1) + go func() { + _, err := io.Copy(io.Discard, a.Body) + closed <- err + }() + select { + case <-closed: + case <-time.After(3 * time.Second): + t.Fatal("idle tail not closed") + } + deadline := time.Now().Add(2 * time.Second) + for h.server.limits.openTails() > 1 { + if time.Now().After(deadline) { + t.Fatal("closed tail still holds its slot") + } + time.Sleep(10 * time.Millisecond) + } +} From 8b0ac342b5b6a75e51d9a289de8442c5d484b1ca Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:32:47 +0700 Subject: [PATCH 208/518] feat(observability-proxy): add the observability-proxy command The deploy unit builds an image and chart around this binary, so its flags follow the agreed contract: listen and probe addresses, cluster domain, root tenant and auth cache TTL, plus the logs limits. Defaults are chosen so a stock install is safe without tuning; readiness waits for the namespace cache because store resolution reads it on every request. On shutdown open tails are cut after a grace period so their clients reconnect with since to another replica. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 199 +++++++++++++++++++++++++++ cmd/observability-proxy/main_test.go | 62 +++++++++ 2 files changed, 261 insertions(+) create mode 100644 cmd/observability-proxy/main.go create mode 100644 cmd/observability-proxy/main_test.go diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go new file mode 100644 index 000000000..cce06e230 --- /dev/null +++ b/cmd/observability-proxy/main.go @@ -0,0 +1,199 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Command observability-proxy serves tenant logs (and, in later waves, +// metrics) from the VictoriaLogs and VictoriaMetrics stores behind a +// Kubernetes TokenReview and SubjectAccessReview, with the tenant's +// namespace scope forced into every upstream request. +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "io" + "log/slog" + "net/http" + "os" + "os/signal" + "sync/atomic" + "syscall" + "time" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/collectors" + "github.com/prometheus/client_golang/prometheus/promhttp" + "k8s.io/client-go/informers" + "k8s.io/client-go/kubernetes" + "k8s.io/client-go/tools/cache" + "k8s.io/client-go/tools/clientcmd" + + "github.com/cozystack/cozystack/internal/observabilityproxy" +) + +type options struct { + listenAddress string + probeAddress string + clusterDomain string + rootTenant string + kubeconfig string + authCacheTTL time.Duration + requestTimeout time.Duration + tailIdleTimeout time.Duration + shutdownTimeout time.Duration + logs observabilityproxy.LogsLimits + limiter observabilityproxy.LimiterConfig +} + +func parseFlags(args []string) (options, error) { + var o options + fs := flag.NewFlagSet("observability-proxy", flag.ContinueOnError) + fs.SetOutput(io.Discard) + fs.StringVar(&o.listenAddress, "listen-address", ":8080", "address of the API listener") + fs.StringVar(&o.probeAddress, "probe-address", ":8081", "address serving /healthz, /readyz and /metrics") + fs.StringVar(&o.clusterDomain, "cluster-domain", "cozy.local", "cluster DNS domain used to reach the stores") + fs.StringVar(&o.rootTenant, "root-tenant", "tenant-root", "tenant namespace hosting the default store") + fs.StringVar(&o.kubeconfig, "kubeconfig", "", "kubeconfig path; in-cluster config when empty") + fs.DurationVar(&o.authCacheTTL, "auth-cache-ttl", 10*time.Second, "how long TokenReview and SubjectAccessReview answers are cached") + fs.DurationVar(&o.requestTimeout, "request-timeout", time.Minute, "upstream deadline for every endpoint but tail") + fs.DurationVar(&o.tailIdleTimeout, "tail-idle-timeout", 5*time.Minute, "close a tail after this long without a line") + fs.DurationVar(&o.shutdownTimeout, "shutdown-timeout", 15*time.Second, "grace period for in-flight requests on shutdown") + fs.DurationVar(&o.logs.DefaultRange, "logs-default-range", time.Hour, "window used when a logs request has no start") + // One week covers an incident investigation and keeps a single query + // to a fraction of the default one-month log retention. + fs.DurationVar(&o.logs.MaxRange, "logs-max-range", 7*24*time.Hour, "longest [start, end] a logs request may span; wider windows are clamped") + // The same cap on points as the metrics proxy. + fs.IntVar(&o.logs.MaxBuckets, "logs-max-buckets", 11000, "most buckets hits and stats_query_range may return over a window") + fs.IntVar(&o.logs.DefaultLimit, "logs-default-limit", 1000, "entries returned when a request sets no limit") + fs.IntVar(&o.logs.MaxLimit, "logs-max-limit", 10000, "largest limit a request may set") + fs.DurationVar(&o.logs.MaxTailResume, "logs-max-tail-resume", 15*time.Minute, "how far back a tail may resume with since or start_offset") + fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables") + fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained requests per second per user; 0 disables") + fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user") + fs.IntVar(&o.limiter.MaxTailsPerUser, "max-tails-per-user", 5, "open tails per user; 0 disables") + fs.IntVar(&o.limiter.MaxTails, "max-tails", 200, "open tails in total; 0 disables") + if err := fs.Parse(args); err != nil { + return o, err + } + switch { + case o.rootTenant == "": + return o, errors.New("--root-tenant must not be empty") + case o.clusterDomain == "": + return o, errors.New("--cluster-domain must not be empty") + case o.logs.DefaultRange <= 0 || o.logs.MaxRange < o.logs.DefaultRange: + return o, errors.New("--logs-default-range must be positive and not above --logs-max-range") + case o.logs.MaxBuckets <= 0: + return o, errors.New("--logs-max-buckets must be positive") + case o.logs.DefaultLimit <= 0 || o.logs.MaxLimit < o.logs.DefaultLimit: + return o, errors.New("--logs-default-limit must be positive and not above --logs-max-limit") + case o.logs.MaxTailResume < 0: + return o, errors.New("--logs-max-tail-resume must not be negative") + } + return o, nil +} + +func main() { + slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stderr, nil))) + o, err := parseFlags(os.Args[1:]) + if err != nil { + fmt.Fprintf(os.Stderr, "observability-proxy: %v\n", err) + os.Exit(2) + } + if err := run(o); err != nil { + slog.Error("exiting", "error", err) + os.Exit(1) + } +} + +func run(o options) error { + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + restConfig, err := clientcmd.BuildConfigFromFlags("", o.kubeconfig) + if err != nil { + return fmt.Errorf("loading kubeconfig: %w", err) + } + client, err := kubernetes.NewForConfig(restConfig) + if err != nil { + return fmt.Errorf("building client: %w", err) + } + + factory := informers.NewSharedInformerFactory(client, 0) + namespaces := factory.Core().V1().Namespaces() + lister := namespaces.Lister() + factory.Start(ctx.Done()) + var ready atomic.Bool + go func() { + if cache.WaitForCacheSync(ctx.Done(), namespaces.Informer().HasSynced) { + ready.Store(true) + } + }() + + reg := prometheus.NewRegistry() + reg.MustRegister(collectors.NewGoCollector(), collectors.NewProcessCollector(collectors.ProcessCollectorOpts{})) + proxy := observabilityproxy.NewServer(observabilityproxy.Config{ + ClusterDomain: o.clusterDomain, + Logs: o.logs, + Limiter: o.limiter, + RequestTimeout: o.requestTimeout, + TailIdleTimeout: o.tailIdleTimeout, + }, observabilityproxy.NewAuthorizer(client, o.authCacheTTL), observabilityproxy.NewStoreResolver(lister, o.rootTenant), reg) + + probes := http.NewServeMux() + probes.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { _, _ = io.WriteString(w, "ok") }) + probes.HandleFunc("/readyz", func(w http.ResponseWriter, _ *http.Request) { + if !ready.Load() { + http.Error(w, "namespace cache not synced", http.StatusServiceUnavailable) + return + } + _, _ = io.WriteString(w, "ok") + }) + probes.Handle("/metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{})) + + // No WriteTimeout on the API listener: it would cut live tails, which + // the handler bounds with its own idle timeout instead. + servers := []*http.Server{ + {Addr: o.listenAddress, Handler: proxy, ReadHeaderTimeout: 10 * time.Second, IdleTimeout: 2 * time.Minute}, + {Addr: o.probeAddress, Handler: probes, ReadHeaderTimeout: 10 * time.Second}, + } + errs := make(chan error, len(servers)) + for _, srv := range servers { + go func() { + slog.Info("listening", "address", srv.Addr) + if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) { + errs <- err + } + }() + } + + var runErr error + select { + case <-ctx.Done(): + case runErr = <-errs: + } + stop() + shutdownCtx, cancel := context.WithTimeout(context.Background(), o.shutdownTimeout) + defer cancel() + for _, srv := range servers { + if err := srv.Shutdown(shutdownCtx); err != nil { + // Open tails never go idle on their own; cut them so clients + // reconnect with since to another replica. + _ = srv.Close() + } + } + return runErr +} diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go new file mode 100644 index 000000000..52c147cf0 --- /dev/null +++ b/cmd/observability-proxy/main_test.go @@ -0,0 +1,62 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "testing" + "time" +) + +func TestParseFlagsDefaults(t *testing.T) { + o, err := parseFlags(nil) + if err != nil { + t.Fatal(err) + } + if o.listenAddress != ":8080" || o.probeAddress != ":8081" || o.clusterDomain != "cozy.local" || o.rootTenant != "tenant-root" { + t.Errorf("contract defaults changed: %+v", o) + } + if o.authCacheTTL != 10*time.Second || o.logs.MaxTailResume != 15*time.Minute || o.logs.MaxRange != 7*24*time.Hour { + t.Errorf("limit defaults changed: %+v", o) + } +} + +func TestParseFlagsOverrides(t *testing.T) { + o, err := parseFlags([]string{"--cluster-domain=example.org", "--logs-max-range=24h", "--max-tails=3"}) + if err != nil { + t.Fatal(err) + } + if o.clusterDomain != "example.org" || o.logs.MaxRange != 24*time.Hour || o.limiter.MaxTails != 3 { + t.Errorf("overrides not applied: %+v", o) + } +} + +func TestParseFlagsInvalid(t *testing.T) { + for _, args := range [][]string{ + {"--root-tenant="}, + {"--cluster-domain="}, + {"--logs-default-range=48h", "--logs-max-range=24h"}, + {"--logs-max-buckets=0"}, + {"--logs-default-limit=0"}, + {"--logs-default-limit=100", "--logs-max-limit=10"}, + {"--logs-max-tail-resume=-1m"}, + {"--unknown"}, + } { + if _, err := parseFlags(args); err == nil { + t.Errorf("parseFlags(%v) accepted", args) + } + } +} From cbca8ac3c9ca67a2954bff17aeaa021460a29ea8 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:38:56 +0700 Subject: [PATCH 209/518] fix(observability-proxy): refuse every LogsQL options block A substring match on ignore_global_time_filter does not see the option once it is written as a quoted name with an escape, or split by a comment: the LogsQL lexer unquotes option names and joins words across a comment, so the option still applies and lifts the time window the proxy enforces. None of the query options is needed by a tenant, so the options keyword followed by a parenthesis is refused outright, skipping whitespace and comments exactly as the lexer does. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 45 +++++++++++++++++++ .../observabilityproxy/logsparams_test.go | 3 ++ 2 files changed, 48 insertions(+) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index cca53e62d..578714d6f 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -24,6 +24,8 @@ import ( "strconv" "strings" "time" + "unicode" + "unicode/utf8" ) // LogsLimits bound what one logs request may make vlselect do. @@ -66,6 +68,13 @@ func BuildLogsParams(endpoint string, in url.Values, namespace string, now time. if strings.Contains(strings.ToLower(query), "ignore_global_time_filter") { return nil, errorf(http.StatusBadRequest, "ignore_global_time_filter is not allowed") } + // The substring check alone is bypassable: VictoriaLogs unquotes a + // quoted option name and joins words split by a comment. No option is + // needed by a tenant (the others tune concurrency or the time filter), + // so any options(...) block is refused. + if hasOptionsBlock(query) { + return nil, errorf(http.StatusBadRequest, "query options are not allowed") + } out := url.Values{} out.Set("query", query) out.Set("extra_stream_filters", "{kubernetes_namespace_name="+strconv.Quote(namespace)+"}") @@ -301,6 +310,42 @@ func parseTime(s string, now time.Time) (time.Time, error) { return time.Time{}, fmt.Errorf("unsupported time %q", s) } +// hasOptionsBlock reports whether the options keyword is followed by "(", +// skipping whitespace and "#" comments the way the LogsQL lexer does. A +// quoted "options" is never the keyword, but matching it too only refuses +// a search for that literal phrase. +func hasOptionsBlock(query string) bool { + lower := strings.ToLower(query) + for i := strings.Index(lower, "options"); i >= 0; { + rest := lower[i+len("options"):] + for rest != "" { + r, size := utf8.DecodeRuneInString(rest) + switch { + case unicode.IsSpace(r): + rest = rest[size:] + continue + case r == '#': + if n := strings.IndexByte(rest, '\n'); n >= 0 { + rest = rest[n+1:] + } else { + rest = "" + } + continue + } + break + } + if strings.HasPrefix(rest, "(") { + return true + } + next := strings.Index(lower[i+1:], "options") + if next < 0 { + return false + } + i += next + 1 + } + return false +} + func isDigits(s string) bool { if s == "" { return false diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index 1ebc03fd9..e42a73012 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -120,6 +120,9 @@ func TestBuildLogsParamsRejects(t *testing.T) { {"ignore_global_time_filter", "query", url.Values{"query": {`* options(ignore_global_time_filter=true)`}}, http.StatusBadRequest}, {"ignore_global_time_filter in subquery", "hits", url.Values{"query": {`* | union (options(ignore_global_time_filter=true) error)`}}, http.StatusBadRequest}, {"ignore_global_time_filter on tail", "tail", url.Values{"query": {`options(IGNORE_GLOBAL_TIME_FILTER=true) *`}}, http.StatusBadRequest}, + {"options with a quoted escaped name", "query", url.Values{"query": {`options("ignore_global_time_filte\x72"=true) *`}}, http.StatusBadRequest}, + {"options with a comment inside the name", "query", url.Values{"query": {"options(ignore_global#c\n_time_filter=true) *"}}, http.StatusBadRequest}, + {"options after a comment and unicode space", "tail", url.Values{"query": {"OPTIONS #c\n\u00a0(concurrency=1000) *"}}, http.StatusBadRequest}, {"inverted range", "query", url.Values{"start": {"2026-11-04T11:00:00Z"}, "end": {"2026-11-04T10:00:00Z"}}, http.StatusBadRequest}, {"inverted range on tail", "tail", url.Values{"start": {"2026-11-04T11:00:00Z"}, "end": {"2026-11-04T10:00:00Z"}}, http.StatusBadRequest}, {"bad start", "query", url.Values{"start": {"yesterday"}}, http.StatusBadRequest}, From 8fe3c222d35f721a44d528151b060f6f832c942a Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:39:32 +0700 Subject: [PATCH 210/518] fix(observability-proxy): reject non-canonical request paths The console nginx selects its location on the normalized path but forwards the original request URI. A path such as namespaces/tenant-b/../tenant-a/logs/ reaches the proxy as written, so any interpretation the proxy makes of dot segments, empty segments or percent-encoding could make the namespace in the SubjectAccessReview differ from the one a normalizer downstream would see. Such paths are answered 400 instead of being interpreted. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/route.go | 12 +++++++----- internal/observabilityproxy/route_test.go | 14 +++++++++++--- internal/observabilityproxy/server_test.go | 6 +++++- 3 files changed, 23 insertions(+), 9 deletions(-) diff --git a/internal/observabilityproxy/route.go b/internal/observabilityproxy/route.go index 7c692c0ae..56ee9e496 100644 --- a/internal/observabilityproxy/route.go +++ b/internal/observabilityproxy/route.go @@ -76,13 +76,15 @@ func errorf(code int, format string, args ...any) *HTTPError { return &HTTPError{Code: code, Message: fmt.Sprintf(format, args...)} } -// ParseRoute takes the escaped request path. Percent-encoding is refused -// outright: a decoded "%2F" would shift the segments the authorization is -// derived from. +// ParseRoute takes the escaped request path. The console nginx matches its +// location on the normalized path but forwards the original one, so a path +// that a normalizer would rewrite (dot segments, empty segments, any +// percent-encoding such as %2F or %2e) is refused instead of interpreted: +// the namespace the SubjectAccessReview names must be the one queried. func ParseRoute(escapedPath string) (Route, *HTTPError) { notFound := errorf(http.StatusNotFound, "not found") if strings.Contains(escapedPath, "%") { - return Route{}, notFound + return Route{}, errorf(http.StatusBadRequest, "percent-encoded paths are not accepted") } segs := strings.Split(strings.TrimPrefix(escapedPath, "/"), "/") if len(segs) > 0 && segs[len(segs)-1] == "" { @@ -90,7 +92,7 @@ func ParseRoute(escapedPath string) (Route, *HTTPError) { } for _, s := range segs { if s == "" || s == "." || s == ".." { - return Route{}, notFound + return Route{}, errorf(http.StatusBadRequest, "non-canonical path") } } if len(segs) < 6 || segs[0] != "apis" || segs[2] != "v1alpha1" || segs[3] != "namespaces" { diff --git a/internal/observabilityproxy/route_test.go b/internal/observabilityproxy/route_test.go index 73d8e04cd..f34311e0f 100644 --- a/internal/observabilityproxy/route_test.go +++ b/internal/observabilityproxy/route_test.go @@ -61,9 +61,17 @@ func TestParseRoute(t *testing.T) { {name: "unknown app subresource", path: "/apis/apps.cozystack.io/v1alpha1/namespaces/a/postgreses/db/exec/x", wantCode: http.StatusNotFound}, {name: "core api", path: "/api/v1/namespaces/a/pods", wantCode: http.StatusNotFound}, {name: "truncated", path: "/apis/cozystack.io/v1alpha1/namespaces/a", wantCode: http.StatusNotFound}, - {name: "empty segment", path: "/apis/cozystack.io/v1alpha1/namespaces/a/logs/select//logsql/query", wantCode: http.StatusNotFound}, - {name: "dot dot segment", path: "/apis/cozystack.io/v1alpha1/namespaces/a/logs/../metrics/x", wantCode: http.StatusNotFound}, - {name: "percent encoded", path: "/apis/cozystack.io/v1alpha1/namespaces/a%2Fb/logs/x", wantCode: http.StatusNotFound}, + {name: "empty segment", path: "/apis/cozystack.io/v1alpha1/namespaces/a/logs/select//logsql/query", wantCode: http.StatusBadRequest}, + {name: "leading double slash", path: "//apis/cozystack.io/v1alpha1/namespaces/a/logs/x", wantCode: http.StatusBadRequest}, + {name: "dot dot segment", path: "/apis/cozystack.io/v1alpha1/namespaces/b/../a/logs/x", wantCode: http.StatusBadRequest}, + {name: "dot dot under root", path: "/apis/cozystack.io/v1alpha1/namespaces/a/logs/../metrics/x", wantCode: http.StatusBadRequest}, + {name: "dot dot in unknown prefix", path: "/foo/../apis/cozystack.io/v1alpha1/namespaces/a/logs/x", wantCode: http.StatusBadRequest}, + {name: "dot segment", path: "/apis/cozystack.io/v1alpha1/namespaces/a/./logs/x", wantCode: http.StatusBadRequest}, + {name: "encoded slash", path: "/apis/cozystack.io/v1alpha1/namespaces/a%2Fb/logs/x", wantCode: http.StatusBadRequest}, + {name: "encoded slash lowercase", path: "/apis/cozystack.io/v1alpha1/namespaces/a%2fb/logs/x", wantCode: http.StatusBadRequest}, + {name: "encoded dot dot", path: "/apis/cozystack.io/v1alpha1/namespaces/b/%2e%2e/a/logs/x", wantCode: http.StatusBadRequest}, + {name: "encoded dot dot uppercase", path: "/apis/cozystack.io/v1alpha1/namespaces/b/%2E%2E/a/logs/x", wantCode: http.StatusBadRequest}, + {name: "backslash", path: `/apis/cozystack.io/v1alpha1/namespaces/b\..\a/logs/x`, wantCode: http.StatusBadRequest}, {name: "uppercase namespace", path: "/apis/cozystack.io/v1alpha1/namespaces/Tenant/logs/x", wantCode: http.StatusBadRequest}, {name: "namespace with dot", path: "/apis/cozystack.io/v1alpha1/namespaces/a.b/logs/x", wantCode: http.StatusBadRequest}, {name: "namespace with quote", path: `/apis/cozystack.io/v1alpha1/namespaces/a"b/logs/x`, wantCode: http.StatusBadRequest}, diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index ed50f72a8..b9b93936e 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -152,6 +152,10 @@ func TestServerStatusCodes(t *testing.T) { {"logs root itself", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/", "good", http.StatusNotFound}, {"ignore_global_time_filter", http.MethodGet, logsRoot + "query?query=" + url.QueryEscape("options(ignore_global_time_filter=true) *"), "good", http.StatusBadRequest}, {"inverted range", http.MethodGet, logsRoot + "query?start=2026-11-04T11:00:00Z&end=2026-11-04T10:00:00Z", "good", http.StatusBadRequest}, + {"dot dot reaching another namespace", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-b/../tenant-a/logs/select/logsql/query", "good", http.StatusBadRequest}, + {"encoded dot dot", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-b/%2e%2e/tenant-a/logs/select/logsql/query", "good", http.StatusBadRequest}, + {"encoded slash", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a%2Flogs/logs/select/logsql/query", "good", http.StatusBadRequest}, + {"double slash", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs//select/logsql/query", "good", http.StatusBadRequest}, {"delete", http.MethodDelete, logsRoot + "query", "good", http.StatusMethodNotAllowed}, } for _, tt := range tests { @@ -174,7 +178,7 @@ func TestServerRebuildsUpstreamRequest(t *testing.T) { "extra_stream_filters": {`{kubernetes_namespace_name=~".*"}`}, "AccountID": {"9"}, } - headers := map[string]string{"AccountID": "9", "ProjectID": "9", "X-Forwarded-For": "1.2.3.4", "Cookie": "s=1"} + headers := map[string]string{"AccountID": "9", "ProjectID": "9", "X-Forwarded-For": "1.2.3.4", "Cookie": "kc-access=secret; other=1"} resp := h.do(http.MethodGet, logsRoot+"query?"+hostile.Encode(), "good", nil, headers) _ = resp.Body.Close() From 5fc2cba2d56ed5439119a39a95f6976506678f80 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:39:52 +0700 Subject: [PATCH 211/518] fix(observability-proxy): refuse a monitoring label that is not a namespace The label value is spliced into the vlselect host name. A value with a dot would address a host in some other namespace instead of a tenant's store, so it is treated as a resolution error rather than followed. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/store.go | 23 +++++++++++++++++------ internal/observabilityproxy/store_test.go | 4 ++++ 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/internal/observabilityproxy/store.go b/internal/observabilityproxy/store.go index ba0234ec3..0e4041a55 100644 --- a/internal/observabilityproxy/store.go +++ b/internal/observabilityproxy/store.go @@ -17,7 +17,11 @@ limitations under the License. package observabilityproxy import ( + "fmt" + "strings" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/util/validation" corev1listers "k8s.io/client-go/listers/core/v1" ) @@ -37,9 +41,10 @@ func NewStoreResolver(namespaces corev1listers.NamespaceLister, rootTenant strin return &StoreResolver{namespaces: namespaces, root: rootTenant} } -// Resolve answers the root tenant for a namespace that no longer exists: -// its label is gone with it, and the collector's default output is the -// root store, so that is where any lines written without a route are. +// Resolve answers the root tenant for a namespace that no longer exists, +// because its label went with it. Lines it wrote while routed to another +// tenant's store are not found there; reading them needs routing history +// (an open question of the design). func (r *StoreResolver) Resolve(namespace string) (string, error) { ns, err := r.namespaces.Get(namespace) if apierrors.IsNotFound(err) { @@ -48,8 +53,14 @@ func (r *StoreResolver) Resolve(namespace string) (string, error) { if err != nil { return "", err } - if target := ns.Labels[monitoringLabel]; target != "" { - return target, nil + target := ns.Labels[monitoringLabel] + if target == "" { + return r.root, nil } - return r.root, nil + // The value becomes a DNS name; a dot in it would address a host in + // some other namespace. + if errs := validation.IsDNS1123Label(target); len(errs) > 0 { + return "", fmt.Errorf("namespace %s: invalid %s label %q: %s", namespace, monitoringLabel, target, strings.Join(errs, "; ")) + } + return target, nil } diff --git a/internal/observabilityproxy/store_test.go b/internal/observabilityproxy/store_test.go index a87bdf0f6..a4a032f57 100644 --- a/internal/observabilityproxy/store_test.go +++ b/internal/observabilityproxy/store_test.go @@ -46,7 +46,11 @@ func TestStoreResolver(t *testing.T) { } { add(ns, map[string]string{monitoringLabel: target}) } + add("tenant-forged", map[string]string{monitoringLabel: "pod.tenant-b"}) lister := corev1listers.NewNamespaceLister(indexer) + if got, err := NewStoreResolver(lister, "tenant-root").Resolve("tenant-forged"); err == nil { + t.Fatalf("label naming a host outside a namespace resolved to %q", got) + } tests := []struct { root, ns, want string From 159f90691ed6de9dbce1519fc2eac36c05e9fb2d Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:40:21 +0700 Subject: [PATCH 212/518] fix(observability-proxy): tell a slow store apart and bound tail memory A request that hits the proxy's own deadline was answered like an unreachable store, so a heavy query looked like an outage; it now answers 504, and a client that went away is no longer logged as a store failure. A tail line is read through a fixed buffer, so a single huge log line cannot make one stream hold unbounded memory. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 25 +++++++++++++++----- internal/observabilityproxy/server_test.go | 27 ++++++++++++++++++++++ 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index a0e2757c5..4ea5cebee 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -31,6 +31,10 @@ import ( "github.com/prometheus/client_golang/prometheus" ) +// tailChunkBytes bounds the memory one tail holds: a longer line is passed +// on in pieces and flushed once its end arrives. +const tailChunkBytes = 64 << 10 + // maxFormBytes bounds a POST body; LogsQL queries are a few KiB at most. const maxFormBytes = 1 << 20 @@ -194,7 +198,13 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT req.Header.Set("AccountID", "0") req.Header.Set("ProjectID", "0") resp, err := s.client.Do(req) - if err != nil { + switch { + case err == nil: + case r.Context().Err() != nil: + return errorf(http.StatusBadGateway, "client went away") + case errors.Is(err, context.DeadlineExceeded): + return errorf(http.StatusGatewayTimeout, "log store did not answer within %s", s.cfg.RequestTimeout) + default: slog.Warn("log store unreachable", "store", store, "error", err) return errorf(http.StatusBadGateway, "log store unreachable") } @@ -249,21 +259,24 @@ func (s *Server) streamLines(w http.ResponseWriter, rc *http.ResponseController, idle = time.AfterFunc(s.cfg.TailIdleTimeout, cancel) defer idle.Stop() } - rd := bufio.NewReader(body) + rd := bufio.NewReaderSize(body, tailChunkBytes) for { - line, err := rd.ReadBytes('\n') - if len(line) > 0 { + chunk, err := rd.ReadSlice('\n') + if len(chunk) > 0 { if idle != nil { idle.Reset(s.cfg.TailIdleTimeout) } - if _, werr := w.Write(line); werr != nil { + if _, werr := w.Write(chunk); werr != nil { return } + if errors.Is(err, bufio.ErrBufferFull) { + continue + } if ferr := rc.Flush(); ferr != nil { return } } - if err != nil { + if err != nil && !errors.Is(err, bufio.ErrBufferFull) { return } } diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index b9b93936e..62d809f28 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -366,3 +366,30 @@ func TestServerTailCapsAndIdle(t *testing.T) { time.Sleep(10 * time.Millisecond) } } + +func TestServerUpstreamDeadline(t *testing.T) { + release := make(chan struct{}) + t.Cleanup(func() { close(release) }) + h := newHarness(t, Config{RequestTimeout: 50 * time.Millisecond}, func(_ http.ResponseWriter, r *http.Request) { + select { + case <-release: + case <-r.Context().Done(): + } + }) + if got := h.status(http.MethodGet, logsRoot+"query", "good"); got != http.StatusGatewayTimeout { + t.Fatalf("slow upstream answered %d, want 504", got) + } +} + +func TestServerTailLongLine(t *testing.T) { + long := `{"_msg":"` + strings.Repeat("x", 3*tailChunkBytes) + `"}` + h := newHarness(t, Config{}, func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, long+"\n"+`{"_msg":"short"}`+"\n") + }) + resp := h.do(http.MethodGet, logsRoot+"tail", "good", nil, nil) + body, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if string(body) != long+"\n"+`{"_msg":"short"}`+"\n" { + t.Fatalf("long line not streamed intact: got %d bytes", len(body)) + } +} From 1c345da367a9999b56cd18fb80f85a8f6b3a8e7c Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:40:26 +0700 Subject: [PATCH 213/518] fix(observability-proxy): reject negative timeouts at startup A negative duration silently disables the bound it configures, so a typo in the chart would turn off the request deadline or the tail idle close without any sign. Refuse it when the flags are parsed. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 2 ++ cmd/observability-proxy/main_test.go | 3 +++ 2 files changed, 5 insertions(+) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index cce06e230..aa29d365f 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -102,6 +102,8 @@ func parseFlags(args []string) (options, error) { return o, errors.New("--logs-default-limit must be positive and not above --logs-max-limit") case o.logs.MaxTailResume < 0: return o, errors.New("--logs-max-tail-resume must not be negative") + case o.requestTimeout < 0 || o.tailIdleTimeout < 0 || o.authCacheTTL < 0 || o.shutdownTimeout < 0: + return o, errors.New("timeouts and the auth cache TTL must not be negative") } return o, nil } diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index 52c147cf0..8fac1aa6d 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -53,6 +53,9 @@ func TestParseFlagsInvalid(t *testing.T) { {"--logs-default-limit=0"}, {"--logs-default-limit=100", "--logs-max-limit=10"}, {"--logs-max-tail-resume=-1m"}, + {"--request-timeout=-1s"}, + {"--tail-idle-timeout=-1s"}, + {"--auth-cache-ttl=-1s"}, {"--unknown"}, } { if _, err := parseFlags(args); err == nil { From d1c49b38d4c6bc3e240380386de0988be5e31266 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:45:57 +0700 Subject: [PATCH 214/518] fix(observability-proxy): keep step from widening the logs window vlselect widens [start, end] outward to whole steps before scanning, so a step of a hundred years turned the clamped week into the whole retention. The step is now capped at the window and at a third of the maximum range, and the window is shrunk from its start until the widened span still fits the maximum range. Numeric times are accepted only as unambiguous unix seconds, because vlselect reads the unit from the digit count and a millisecond stamp or a bare year would otherwise land in the wrong place. The standalone ignore_global_time_filter substring check is dropped: the options block check already covers it, and the substring only refused plain searches for that word. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 38 ++++++++----- .../observabilityproxy/logsparams_test.go | 55 +++++++++++++++++-- 2 files changed, 75 insertions(+), 18 deletions(-) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index 578714d6f..653f1c194 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -63,15 +63,11 @@ func BuildLogsParams(endpoint string, in url.Values, namespace string, now time. if query == "" { query = "*" } - // The option lifts the time window this function enforces, and on tail - // it rescans the whole retention every tick. - if strings.Contains(strings.ToLower(query), "ignore_global_time_filter") { - return nil, errorf(http.StatusBadRequest, "ignore_global_time_filter is not allowed") - } - // The substring check alone is bypassable: VictoriaLogs unquotes a - // quoted option name and joins words split by a comment. No option is - // needed by a tenant (the others tune concurrency or the time filter), - // so any options(...) block is refused. + // ignore_global_time_filter lifts the time window this function + // enforces, and on tail rescans the whole retention every tick. A + // substring match for it is bypassable (VictoriaLogs unquotes a quoted + // option name and joins words split by a comment), and no option is + // needed by a tenant, so any options(...) block is refused. if hasOptionsBlock(query) { return nil, errorf(http.StatusBadRequest, "query options are not allowed") } @@ -94,7 +90,7 @@ func BuildLogsParams(endpoint string, in url.Values, namespace string, now time. case "query": herr = setLimit(out, in, "limit", lim.DefaultLimit, lim.MaxLimit) case "hits": - if herr = setStep(out, in, end.Sub(start), lim); herr != nil { + if herr = setStep(out, in, start, end, lim); herr != nil { return nil, herr } fields := in["field"] @@ -110,7 +106,7 @@ func BuildLogsParams(endpoint string, in url.Values, namespace string, now time. herr = setLimit(out, in, "fields_limit", 0, lim.MaxLimit) } case "stats_query_range": - herr = setStep(out, in, end.Sub(start), lim) + herr = setStep(out, in, start, end, lim) case "field_values", "stream_field_values": if in.Get("field") == "" { return nil, errorf(http.StatusBadRequest, "field is required") @@ -167,8 +163,13 @@ func window(in url.Values, now time.Time, lim LogsLimits) (time.Time, time.Time, } // setStep raises step so that hits and stats_query_range, which return one -// bucket per step, return at most MaxBuckets buckets over the window. -func setStep(out, in url.Values, span time.Duration, lim LogsLimits) *HTTPError { +// bucket per step, return at most MaxBuckets buckets, and caps it so the +// window cannot grow through it: vlselect widens [start, end] outward to +// step boundaries (alignStartEndToStep), by up to one step on each side, +// so a 100y step would otherwise scan the whole retention. The window is +// shrunk from the start until the widened span fits MaxRange. +func setStep(out, in url.Values, start, end time.Time, lim LogsLimits) *HTTPError { + span := end.Sub(start) minStep := max((span+time.Duration(lim.MaxBuckets)-1)/time.Duration(lim.MaxBuckets), time.Second) step := minStep if v := in.Get("step"); v != "" { @@ -176,9 +177,13 @@ func setStep(out, in url.Values, span time.Duration, lim LogsLimits) *HTTPError if err != nil || d <= 0 { return errorf(http.StatusBadRequest, "invalid step %q", v) } - step = max(d, minStep) + step = max(min(d, span, lim.MaxRange/3), minStep) + } + if span+2*step > lim.MaxRange { + start = end.Add(-max(lim.MaxRange-2*step, 0)) } out.Set("step", formatDuration(step)) + out.Set("start", start.Format(time.RFC3339Nano)) return nil } @@ -290,6 +295,11 @@ func parseTime(s string, now time.Time) (time.Time, error) { return now, nil } if sec, frac, ok := strings.Cut(s, "."); isDigits(sec) && (!ok || isDigits(frac)) { + // vlselect reads the unit from the digit count (four digits are a + // year, 13 are milliseconds); only unambiguous seconds are taken. + if len(sec) < 5 || len(sec) > 10 { + return time.Time{}, fmt.Errorf("numeric time %q is not unix seconds", s) + } secs, err := strconv.ParseInt(sec, 10, 64) if err != nil { return time.Time{}, err diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index e42a73012..8e6fb107d 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -126,6 +126,8 @@ func TestBuildLogsParamsRejects(t *testing.T) { {"inverted range", "query", url.Values{"start": {"2026-11-04T11:00:00Z"}, "end": {"2026-11-04T10:00:00Z"}}, http.StatusBadRequest}, {"inverted range on tail", "tail", url.Values{"start": {"2026-11-04T11:00:00Z"}, "end": {"2026-11-04T10:00:00Z"}}, http.StatusBadRequest}, {"bad start", "query", url.Values{"start": {"yesterday"}}, http.StatusBadRequest}, + {"millisecond timestamp", "query", url.Values{"start": {"1793786400000"}}, http.StatusBadRequest}, + {"year-like number", "query", url.Values{"start": {"2026"}}, http.StatusBadRequest}, {"bad limit", "query", url.Values{"limit": {"-1"}}, http.StatusBadRequest}, {"zero limit", "query", url.Values{"limit": {"0"}}, http.StatusBadRequest}, {"bad step", "hits", url.Values{"step": {"0s"}}, http.StatusBadRequest}, @@ -175,6 +177,7 @@ func TestBuildLogsParamsWindow(t *testing.T) { {"start only", url.Values{"start": {"2026-11-04T09:00:00Z"}}, ts("2026-11-04T09:00:00Z"), ts("2026-11-04T12:00:00Z")}, {"end only", url.Values{"end": {"2026-11-03T00:00:00Z"}}, ts("2026-11-02T23:00:00Z"), ts("2026-11-03T00:00:00Z")}, {"unix seconds", url.Values{"start": {"1793786400"}, "end": {"1793788200.5"}}, "2026-11-04T10:00:00Z", "2026-11-04T10:30:00.5Z"}, + {"rfc3339 with offset", url.Values{"start": {"2026-11-04T13:00:00+03:00"}}, ts("2026-11-04T10:00:00Z"), ts("2026-11-04T12:00:00Z")}, {"relative", url.Values{"start": {"3h"}, "end": {"now"}}, ts("2026-11-04T09:00:00Z"), ts("2026-11-04T12:00:00Z")}, {"relative days", url.Values{"start": {"1d"}}, ts("2026-11-03T12:00:00Z"), ts("2026-11-04T12:00:00Z")}, {"over-wide clamped", url.Values{"start": {"2026-01-01T00:00:00Z"}}, ts("2026-10-28T12:00:00Z"), ts("2026-11-04T12:00:00Z")}, @@ -182,8 +185,12 @@ func TestBuildLogsParamsWindow(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - for _, endpoint := range []string{"query", "hits", "stats_query_range", "facets"} { - out := build(t, endpoint, tt.in) + for _, endpoint := range []string{"query", "field_values", "facets"} { + in := url.Values{"field": {"f"}} + for k, v := range tt.in { + in[k] = v + } + out := build(t, endpoint, in) if out.Get("start") != tt.wantStart || out.Get("end") != tt.wantEnd { t.Errorf("%s: window = [%s, %s], want [%s, %s]", endpoint, out.Get("start"), out.Get("end"), tt.wantStart, tt.wantEnd) } @@ -209,8 +216,8 @@ func TestBuildLogsParamsStepAndLimit(t *testing.T) { if got := build(t, "stats_query_range", with(day, "step", "1h")).Get("step"); got != "3600000ms" { t.Errorf("1h step = %s", got) } - if got := build(t, "stats_query_range", with(day, "step", "1w")).Get("step"); got != "604800000ms" { - t.Errorf("1w step = %s", got) + if got := build(t, "stats_query_range", with(day, "step", "1w")).Get("step"); got != "86400000ms" { + t.Errorf("1w step on a day window = %s, want capped at the window", got) } if got := build(t, "hits", day).Get("step"); got != "86400ms" { t.Errorf("omitted step = %s, want the bucket-bounded minimum", got) @@ -273,3 +280,43 @@ func TestBuildLogsParamsTail(t *testing.T) { t.Errorf("tail start = %q, want the client value ANDed into each tick", out.Get("start")) } } + +// alignedWindow repeats vlselect's alignStartEndToStep (offset 0): with a +// step it widens [start, end] to step boundaries, end always by at least +// one more step. +func alignedWindow(t *testing.T, out url.Values) time.Duration { + t.Helper() + parse := func(k string) int64 { + v, err := time.Parse(time.RFC3339Nano, out.Get(k)) + if err != nil { + t.Fatalf("%s: %v", k, err) + } + return v.UnixNano() + } + step, err := parseDuration(out.Get("step")) + if err != nil { + t.Fatal(err) + } + start, end, st := parse("start"), parse("end"), int64(step) + start -= start % st + end += st - end%st + return time.Duration(end - start) +} + +func TestBuildLogsParamsStepCannotWidenWindow(t *testing.T) { + lim := testLimits() + for _, endpoint := range []string{"hits", "stats_query_range"} { + for _, in := range []url.Values{ + {"step": {"100y"}}, + {"step": {"100y"}, "start": {"2026-01-01T00:00:00Z"}}, + {"step": {"3d"}, "start": {"7d"}}, + {"start": {"7d"}}, + {"step": {"1s"}, "start": {"7d"}}, + } { + out := build(t, endpoint, in) + if got := alignedWindow(t, out); got > lim.MaxRange { + t.Errorf("%s %v: vlselect would scan %s, more than %s", endpoint, in, got, lim.MaxRange) + } + } + } +} From 8eef2fa5d2a226ec965dcc636bd6dcbec640cbbd Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:46:09 +0700 Subject: [PATCH 215/518] fix(observability-proxy): reject negative limits at startup Zero is the documented way to disable a limit; a negative value disabled it too, silently. Refuse it so a typo in the chart cannot turn off a per-user or total cap unnoticed. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 3 +++ cmd/observability-proxy/main_test.go | 5 +++++ 2 files changed, 8 insertions(+) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index aa29d365f..3cba0bf5f 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -104,6 +104,9 @@ func parseFlags(args []string) (options, error) { return o, errors.New("--logs-max-tail-resume must not be negative") case o.requestTimeout < 0 || o.tailIdleTimeout < 0 || o.authCacheTTL < 0 || o.shutdownTimeout < 0: return o, errors.New("timeouts and the auth cache TTL must not be negative") + case o.limiter.RatePerUser < 0 || o.limiter.RateBurst < 0 || o.limiter.MaxConcurrentPerUser < 0 || + o.limiter.MaxTailsPerUser < 0 || o.limiter.MaxTails < 0: + return o, errors.New("limits must not be negative; 0 disables a limit") } return o, nil } diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index 8fac1aa6d..866ff95d2 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -56,6 +56,11 @@ func TestParseFlagsInvalid(t *testing.T) { {"--request-timeout=-1s"}, {"--tail-idle-timeout=-1s"}, {"--auth-cache-ttl=-1s"}, + {"--rate-limit-per-user=-1"}, + {"--rate-limit-burst=-1"}, + {"--max-concurrent-per-user=-1"}, + {"--max-tails-per-user=-1"}, + {"--max-tails=-1"}, {"--unknown"}, } { if _, err := parseFlags(args); err == nil { From 4e8071aa2b6c8099b3c7e2960a187c15edf6af2c Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:50:19 +0700 Subject: [PATCH 216/518] fix(observability-proxy): accept bare seconds only as plain digits Bare seconds were parsed with ParseFloat, which also takes signs, exponents, Inf and NaN. Such a start fell through to the relative duration form and silently moved the window, so 1.7937864e9 meant 1970 rather than an error, and NaN converted differently per CPU architecture. Only digits with an optional fraction are taken now; everything else is a 400. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 9 ++++++--- internal/observabilityproxy/logsparams_test.go | 8 ++++++++ 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index 653f1c194..c9fad47db 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -264,9 +264,12 @@ var durationUnits = map[string]time.Duration{ // parseDuration accepts the LogsQL duration forms (1h30m, 1d, 2w) and bare // seconds; Go's parser knows neither d nor w. func parseDuration(s string) (time.Duration, error) { - if f, err := strconv.ParseFloat(s, 64); err == nil { - if f < 0 { - return 0, fmt.Errorf("negative duration %q", s) + // Bare seconds only as digits with an optional fraction: ParseFloat + // would also take signs, exponents, Inf and NaN. + if whole, frac, ok := strings.Cut(s, "."); isDigits(whole) && (!ok || isDigits(frac)) { + f, err := strconv.ParseFloat(s, 64) + if err != nil { + return 0, fmt.Errorf("invalid duration %q", s) } return time.Duration(f * float64(time.Second)), nil } diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index 8e6fb107d..abfbc3483 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -128,6 +128,14 @@ func TestBuildLogsParamsRejects(t *testing.T) { {"bad start", "query", url.Values{"start": {"yesterday"}}, http.StatusBadRequest}, {"millisecond timestamp", "query", url.Values{"start": {"1793786400000"}}, http.StatusBadRequest}, {"year-like number", "query", url.Values{"start": {"2026"}}, http.StatusBadRequest}, + {"exponent time", "query", url.Values{"start": {"1.7937864e9"}}, http.StatusBadRequest}, + {"signed time", "query", url.Values{"start": {"+1793786400"}}, http.StatusBadRequest}, + {"exponent duration", "query", url.Values{"start": {"3e2"}}, http.StatusBadRequest}, + {"trailing dot", "query", url.Values{"start": {"300."}}, http.StatusBadRequest}, + {"infinite time", "query", url.Values{"end": {"Inf"}}, http.StatusBadRequest}, + {"NaN time", "query", url.Values{"start": {"NaN"}}, http.StatusBadRequest}, + {"NaN step", "hits", url.Values{"step": {"NaN"}}, http.StatusBadRequest}, + {"NaN start_offset", "tail", url.Values{"start_offset": {"NaN"}}, http.StatusBadRequest}, {"bad limit", "query", url.Values{"limit": {"-1"}}, http.StatusBadRequest}, {"zero limit", "query", url.Values{"limit": {"0"}}, http.StatusBadRequest}, {"bad step", "hits", url.Values{"step": {"0s"}}, http.StatusBadRequest}, From ca7f283b6aeefa2ad25da371af86228960b93c35 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:50:19 +0700 Subject: [PATCH 217/518] fix(observability-proxy): require a range and bucket count the step cap fits The step is capped at a third of the maximum range so that vlselect's alignment to whole steps cannot widen the window past that range. With fewer than three buckets, or a range under three seconds, the minimum step exceeds that cap and the bound no longer holds, so such settings are refused at startup. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 8 ++++++-- cmd/observability-proxy/main_test.go | 4 +++- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index 3cba0bf5f..84cec030c 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -96,8 +96,12 @@ func parseFlags(args []string) (options, error) { return o, errors.New("--cluster-domain must not be empty") case o.logs.DefaultRange <= 0 || o.logs.MaxRange < o.logs.DefaultRange: return o, errors.New("--logs-default-range must be positive and not above --logs-max-range") - case o.logs.MaxBuckets <= 0: - return o, errors.New("--logs-max-buckets must be positive") + // Below three buckets, or a range under three seconds, the step cap + // that keeps vlselect's step alignment inside the range cannot hold. + case o.logs.MaxBuckets < 3: + return o, errors.New("--logs-max-buckets must be at least 3") + case o.logs.MaxRange < 3*time.Second: + return o, errors.New("--logs-max-range must be at least 3s") case o.logs.DefaultLimit <= 0 || o.logs.MaxLimit < o.logs.DefaultLimit: return o, errors.New("--logs-default-limit must be positive and not above --logs-max-limit") case o.logs.MaxTailResume < 0: diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index 866ff95d2..924c06f28 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -49,7 +49,9 @@ func TestParseFlagsInvalid(t *testing.T) { {"--root-tenant="}, {"--cluster-domain="}, {"--logs-default-range=48h", "--logs-max-range=24h"}, - {"--logs-max-buckets=0"}, + {"--logs-max-buckets=2"}, + {"--logs-max-range=2s", "--logs-default-range=1s"}, + {"--shutdown-timeout=-1s"}, {"--logs-default-limit=0"}, {"--logs-default-limit=100", "--logs-max-limit=10"}, {"--logs-max-tail-resume=-1m"}, From 7ec648d863064c4754851289d9064828eec952a5 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:16:35 +0700 Subject: [PATCH 218/518] feat(observability-proxy): add the chart and image build Tenants read their logs, and later their metrics, through an authorizing proxy that sits in front of the VictoriaLogs and VictoriaMetrics read paths, so the proxy needs a deployable package before the console or tenant Grafana can reach it. The proxy reviews every token and access decision against the API server, so its own role is limited to creating TokenReviews and SubjectAccessReviews and reading ApplicationDefinitions and namespaces. The namespace logs root is authorized as list on cozystack.io logs, which no API server serves; the rule is aggregated into the tenant view role so that a namespace viewer passes that review, as upstream grants pods/log at view. The image ref lands without a digest until the first build stamps it, and the package joins the root build list so every build path refreshes it. Assisted-by: LLM Signed-off-by: ZverGuy --- Makefile | 1 + .../system/observability-proxy/Chart.yaml | 3 + packages/system/observability-proxy/Makefile | 20 ++++ .../images/observability-proxy/Dockerfile | 25 ++++ .../templates/deployment.yaml | 68 +++++++++++ .../observability-proxy/templates/rbac.yaml | 33 ++++++ .../observability-proxy/templates/sa.yaml | 4 + .../templates/service.yaml | 15 +++ .../templates/tenant-clusterroles.yaml | 14 +++ .../tests/deployment_test.yaml | 109 ++++++++++++++++++ .../observability-proxy/tests/rbac_test.yaml | 70 +++++++++++ .../tests/service_test.yaml | 30 +++++ .../system/observability-proxy/values.yaml | 16 +++ 13 files changed, 408 insertions(+) create mode 100644 packages/system/observability-proxy/Chart.yaml create mode 100644 packages/system/observability-proxy/Makefile create mode 100644 packages/system/observability-proxy/images/observability-proxy/Dockerfile create mode 100644 packages/system/observability-proxy/templates/deployment.yaml create mode 100644 packages/system/observability-proxy/templates/rbac.yaml create mode 100644 packages/system/observability-proxy/templates/sa.yaml create mode 100644 packages/system/observability-proxy/templates/service.yaml create mode 100644 packages/system/observability-proxy/templates/tenant-clusterroles.yaml create mode 100644 packages/system/observability-proxy/tests/deployment_test.yaml create mode 100644 packages/system/observability-proxy/tests/rbac_test.yaml create mode 100644 packages/system/observability-proxy/tests/service_test.yaml create mode 100644 packages/system/observability-proxy/values.yaml diff --git a/Makefile b/Makefile index 6da0e1037..6581bf304 100644 --- a/Makefile +++ b/Makefile @@ -17,6 +17,7 @@ build: build-deps make -C packages/system/cozystack-api image make -C packages/system/cozystack-controller image make -C packages/system/backup-controller image + make -C packages/system/observability-proxy image make -C packages/system/backupstrategy-controller image make -C packages/system/lineage-controller-webhook image make -C packages/system/migration-controller image diff --git a/packages/system/observability-proxy/Chart.yaml b/packages/system/observability-proxy/Chart.yaml new file mode 100644 index 000000000..282c2234a --- /dev/null +++ b/packages/system/observability-proxy/Chart.yaml @@ -0,0 +1,3 @@ +apiVersion: v2 +name: cozy-observability-proxy +version: 0.0.0 # Placeholder, the actual version will be automatically set during the build process diff --git a/packages/system/observability-proxy/Makefile b/packages/system/observability-proxy/Makefile new file mode 100644 index 000000000..ee66c0656 --- /dev/null +++ b/packages/system/observability-proxy/Makefile @@ -0,0 +1,20 @@ +NAME=observability-proxy +NAMESPACE=cozy-observability-proxy + +include ../../../hack/common-envs.mk +include ../../../hack/package.mk + +test: + helm unittest . + +image: image-observability-proxy + +image-observability-proxy: + docker buildx build -f images/observability-proxy/Dockerfile ../../.. \ + $(call image-tags,observability-proxy,v$(COZYSTACK_VERSION)) \ + $(call cache-args,observability-proxy) \ + --metadata-file images/observability-proxy.json \ + $(BUILDX_ARGS) + IMAGE="$(REGISTRY)/observability-proxy:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/observability-proxy.json -o json -r)" \ + yq -i '.observabilityProxy.image = strenv(IMAGE)' values.yaml + rm -f images/observability-proxy.json diff --git a/packages/system/observability-proxy/images/observability-proxy/Dockerfile b/packages/system/observability-proxy/images/observability-proxy/Dockerfile new file mode 100644 index 000000000..4615e74ec --- /dev/null +++ b/packages/system/observability-proxy/images/observability-proxy/Dockerfile @@ -0,0 +1,25 @@ +FROM --platform=$BUILDPLATFORM golang:1.27-alpine@sha256:8a5910f31396cd4d89662f56c68b3ae31d374308270a1c3bd96672ee5ed43414 AS builder +ARG GOPROXY + +ARG TARGETOS +ARG TARGETARCH + +WORKDIR /workspace + +COPY go.mod go.sum ./ +RUN for attempt in 1 2 3; do GOOS=$TARGETOS GOARCH=$TARGETARCH go mod download && break; [ "$attempt" = 3 ] && exit 1; sleep $((attempt * 10)); done + +COPY api api/ +COPY pkg pkg/ +COPY cmd cmd/ +COPY internal internal/ + +RUN GOOS=$TARGETOS GOARCH=$TARGETARCH CGO_ENABLED=0 go build -ldflags="-extldflags=-static" -o /observability-proxy ./cmd/observability-proxy + +FROM scratch + +COPY --from=builder /observability-proxy /observability-proxy +COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt + +USER 65532:65532 +ENTRYPOINT ["/observability-proxy"] diff --git a/packages/system/observability-proxy/templates/deployment.yaml b/packages/system/observability-proxy/templates/deployment.yaml new file mode 100644 index 000000000..581c15fde --- /dev/null +++ b/packages/system/observability-proxy/templates/deployment.yaml @@ -0,0 +1,68 @@ +{{- $clusterDomain := index (.Values._cluster | default dict) "cluster-domain" | default "cozy.local" }} +apiVersion: apps/v1 +kind: Deployment +metadata: + name: observability-proxy + labels: + app.kubernetes.io/name: observability-proxy +spec: + replicas: {{ .Values.observabilityProxy.replicas }} + selector: + matchLabels: + app.kubernetes.io/name: observability-proxy + template: + metadata: + labels: + app.kubernetes.io/name: observability-proxy + spec: + serviceAccountName: observability-proxy + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app.kubernetes.io/name: observability-proxy + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + containers: + - name: observability-proxy + image: {{ .Values.observabilityProxy.image | quote }} + args: + - --listen-address=:8080 + - --probe-address=:8081 + - --cluster-domain={{ $clusterDomain }} + - --root-tenant={{ .Values.observabilityProxy.rootTenant }} + - --auth-cache-ttl={{ .Values.observabilityProxy.authCacheTTL }} + ports: + - name: http + containerPort: 8080 + protocol: TCP + - name: probes + containerPort: 8081 + protocol: TCP + readinessProbe: + httpGet: + path: /readyz + port: probes + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: + path: /healthz + port: probes + initialDelaySeconds: 15 + periodSeconds: 20 + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + {{- with .Values.observabilityProxy.resources }} + resources: {{- toYaml . | nindent 10 }} + {{- end }} diff --git a/packages/system/observability-proxy/templates/rbac.yaml b/packages/system/observability-proxy/templates/rbac.yaml new file mode 100644 index 000000000..f26b71fe3 --- /dev/null +++ b/packages/system/observability-proxy/templates/rbac.yaml @@ -0,0 +1,33 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: cozy:observability-proxy +rules: +# Every request is authenticated and authorized against the API server. +- apiGroups: ["authentication.k8s.io"] + resources: ["tokenreviews"] + verbs: ["create"] +- apiGroups: ["authorization.k8s.io"] + resources: ["subjectaccessreviews"] + verbs: ["create"] +# Maps an application plural in the path to its Kind. +- apiGroups: ["cozystack.io"] + resources: ["applicationdefinitions"] + verbs: ["get", "list", "watch"] +# The namespace.cozystack.io/monitoring label names the logs store. +- apiGroups: [""] + resources: ["namespaces"] + verbs: ["get", "list", "watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: cozy:observability-proxy +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cozy:observability-proxy +subjects: +- kind: ServiceAccount + name: observability-proxy + namespace: {{ .Release.Namespace }} diff --git a/packages/system/observability-proxy/templates/sa.yaml b/packages/system/observability-proxy/templates/sa.yaml new file mode 100644 index 000000000..5a4eb2182 --- /dev/null +++ b/packages/system/observability-proxy/templates/sa.yaml @@ -0,0 +1,4 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: observability-proxy diff --git a/packages/system/observability-proxy/templates/service.yaml b/packages/system/observability-proxy/templates/service.yaml new file mode 100644 index 000000000..6a398532a --- /dev/null +++ b/packages/system/observability-proxy/templates/service.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Service +metadata: + name: observability-proxy + labels: + app.kubernetes.io/name: observability-proxy +spec: + type: ClusterIP + ports: + - name: http + port: 80 + targetPort: http + protocol: TCP + selector: + app.kubernetes.io/name: observability-proxy diff --git a/packages/system/observability-proxy/templates/tenant-clusterroles.yaml b/packages/system/observability-proxy/templates/tenant-clusterroles.yaml new file mode 100644 index 000000000..23460352f --- /dev/null +++ b/packages/system/observability-proxy/templates/tenant-clusterroles.yaml @@ -0,0 +1,14 @@ +# Aggregated into cozy-tenant-view (and consequently use, admin, super-admin). +# No API server serves cozystack.io logs: the rule exists only for the +# SubjectAccessReview the observability proxy asks before it answers a read +# under /apis/cozystack.io/v1alpha1/namespaces//logs/. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: cozy:observability:view + labels: + rbac.cozystack.io/aggregate-to-tenant-view: "true" +rules: +- apiGroups: ["cozystack.io"] + resources: ["logs"] + verbs: ["list"] diff --git a/packages/system/observability-proxy/tests/deployment_test.yaml b/packages/system/observability-proxy/tests/deployment_test.yaml new file mode 100644 index 000000000..c8415be01 --- /dev/null +++ b/packages/system/observability-proxy/tests/deployment_test.yaml @@ -0,0 +1,109 @@ +suite: observability-proxy Deployment +# The flags, ports and pod label below are the contract the proxy binary, the +# Service, the console nginx and the tenant egress policy are written against. +# A rename on either side breaks routing without failing a render. +templates: + - templates/deployment.yaml + +release: + name: observability-proxy + namespace: cozy-observability-proxy + +tests: + - it: labels the pods with the name the tenant egress policy selects on + asserts: + - isKind: + of: Deployment + - equal: + path: spec.selector.matchLabels + value: + app.kubernetes.io/name: observability-proxy + - equal: + path: spec.template.metadata.labels["app.kubernetes.io/name"] + value: observability-proxy + - equal: + path: spec.template.spec.serviceAccountName + value: observability-proxy + + - it: passes the listen, probe, root tenant and auth cache flags + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --listen-address=:8080 + - contains: + path: spec.template.spec.containers[0].args + content: --probe-address=:8081 + - contains: + path: spec.template.spec.containers[0].args + content: --root-tenant=tenant-root + - contains: + path: spec.template.spec.containers[0].args + content: --auth-cache-ttl=10s + + - it: falls back to cozy.local when the platform passes no cluster domain + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --cluster-domain=cozy.local + + - it: passes the platform cluster domain + set: + _cluster: + cluster-domain: cluster.local + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --cluster-domain=cluster.local + - notContains: + path: spec.template.spec.containers[0].args + content: --cluster-domain=cozy.local + + - it: exposes the API and probe ports + asserts: + - equal: + path: spec.template.spec.containers[0].ports + value: + - name: http + containerPort: 8080 + protocol: TCP + - name: probes + containerPort: 8081 + protocol: TCP + + - it: probes /healthz and /readyz on the probe port + asserts: + - equal: + path: spec.template.spec.containers[0].livenessProbe.httpGet + value: + path: /healthz + port: probes + - equal: + path: spec.template.spec.containers[0].readinessProbe.httpGet + value: + path: /readyz + port: probes + + - it: runs from the image in values + set: + observabilityProxy.image: registry.example/observability-proxy:dev@sha256:0000 + asserts: + - equal: + path: spec.template.spec.containers[0].image + value: registry.example/observability-proxy:dev@sha256:0000 + + - it: satisfies the restricted pod security profile + asserts: + - equal: + path: spec.template.spec.securityContext + value: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + - equal: + path: spec.template.spec.containers[0].securityContext + value: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL diff --git a/packages/system/observability-proxy/tests/rbac_test.yaml b/packages/system/observability-proxy/tests/rbac_test.yaml new file mode 100644 index 000000000..d29fe87a3 --- /dev/null +++ b/packages/system/observability-proxy/tests/rbac_test.yaml @@ -0,0 +1,70 @@ +suite: observability-proxy RBAC +# The proxy authenticates and authorizes every request through the API server, +# so a dropped verb here turns every request into an error at runtime rather +# than a render failure. The tenant role is what lets a namespace viewer pass +# the SubjectAccessReview the proxy asks for the namespace logs root. +templates: + - templates/rbac.yaml + - templates/tenant-clusterroles.yaml + +release: + name: observability-proxy + namespace: cozy-observability-proxy + +tests: + - it: lets the proxy review tokens and access, and read ApplicationDefinitions and namespaces + template: templates/rbac.yaml + documentSelector: + path: kind + value: ClusterRole + asserts: + - equal: + path: rules + value: + - apiGroups: ["authentication.k8s.io"] + resources: ["tokenreviews"] + verbs: ["create"] + - apiGroups: ["authorization.k8s.io"] + resources: ["subjectaccessreviews"] + verbs: ["create"] + - apiGroups: ["cozystack.io"] + resources: ["applicationdefinitions"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["namespaces"] + verbs: ["get", "list", "watch"] + + - it: binds the proxy role to the proxy ServiceAccount in the release namespace + template: templates/rbac.yaml + documentSelector: + path: kind + value: ClusterRoleBinding + asserts: + - equal: + path: roleRef + value: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cozy:observability-proxy + - equal: + path: subjects + value: + - kind: ServiceAccount + name: observability-proxy + namespace: cozy-observability-proxy + + - it: aggregates list on cozystack.io logs into the tenant view role + template: templates/tenant-clusterroles.yaml + asserts: + - isKind: + of: ClusterRole + - equal: + path: metadata.labels + value: + rbac.cozystack.io/aggregate-to-tenant-view: "true" + - equal: + path: rules + value: + - apiGroups: ["cozystack.io"] + resources: ["logs"] + verbs: ["list"] diff --git a/packages/system/observability-proxy/tests/service_test.yaml b/packages/system/observability-proxy/tests/service_test.yaml new file mode 100644 index 000000000..0fb8b0e00 --- /dev/null +++ b/packages/system/observability-proxy/tests/service_test.yaml @@ -0,0 +1,30 @@ +suite: observability-proxy Service +# The console nginx proxies to observability-proxy..svc on port 80 and the +# tenant egress policy allows the pods' 8080, so the name, the port and the +# selector are pinned here. +templates: + - templates/service.yaml + +release: + name: observability-proxy + namespace: cozy-observability-proxy + +tests: + - it: serves port 80 named http on the pods' http port + asserts: + - isKind: + of: Service + - equal: + path: metadata.name + value: observability-proxy + - equal: + path: spec.ports + value: + - name: http + port: 80 + targetPort: http + protocol: TCP + - equal: + path: spec.selector + value: + app.kubernetes.io/name: observability-proxy diff --git a/packages/system/observability-proxy/values.yaml b/packages/system/observability-proxy/values.yaml new file mode 100644 index 000000000..919e450a9 --- /dev/null +++ b/packages/system/observability-proxy/values.yaml @@ -0,0 +1,16 @@ +# Injected by the platform from the cozystack-values Secret. The empty +# default keeps the chart renderable on its own, outside a cluster. +_cluster: {} + +observabilityProxy: + image: "ghcr.io/cozystack/cozystack/observability-proxy:v0.0.0" + replicas: 2 + rootTenant: tenant-root + authCacheTTL: 10s + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi From bbdf05c6c75b47aacac0cdb7d06a97daa19881e1 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:20:58 +0700 Subject: [PATCH 219/518] feat(platform): deploy the observability-proxy The proxy only answers for stores that the monitoring stack runs, so it follows monitoring-agents: on by default in the full variants and opt-in through enabledPackages on the slim ones, where monitoring is opt-in too. It is a stateless reader with no CRDs, so networking is its only dependency. Assisted-by: LLM Signed-off-by: ZverGuy --- .../platform/sources/observability-proxy.yaml | 21 ++++++ .../platform/templates/bundles/system.yaml | 1 + .../selective-install-postgres-values.yaml | 1 + .../observability_proxy_wiring_test.yaml | 69 +++++++++++++++++++ 4 files changed, 92 insertions(+) create mode 100644 packages/core/platform/sources/observability-proxy.yaml create mode 100644 packages/core/platform/tests/observability_proxy_wiring_test.yaml diff --git a/packages/core/platform/sources/observability-proxy.yaml b/packages/core/platform/sources/observability-proxy.yaml new file mode 100644 index 000000000..d9e9ec14b --- /dev/null +++ b/packages/core/platform/sources/observability-proxy.yaml @@ -0,0 +1,21 @@ +--- +apiVersion: cozystack.io/v1alpha1 +kind: PackageSource +metadata: + name: cozystack.observability-proxy +spec: + sourceRef: + kind: OCIRepository + name: cozystack-packages + namespace: cozy-system + path: / + variants: + - name: default + dependsOn: + - cozystack.networking + components: + - name: observability-proxy + path: system/observability-proxy + install: + namespace: cozy-observability-proxy + releaseName: observability-proxy diff --git a/packages/core/platform/templates/bundles/system.yaml b/packages/core/platform/templates/bundles/system.yaml index b175714a2..906c8d95f 100644 --- a/packages/core/platform/templates/bundles/system.yaml +++ b/packages/core/platform/templates/bundles/system.yaml @@ -344,6 +344,7 @@ variants and opt-in on the slim ones. */}} {{include "cozystack.platform.package.full.default" (list "cozystack.metrics-server" $) }} {{- $monitoringAgentsComponents := dict "monitoring-agents" (dict "values" (dict "global" (dict "target" "tenant-root"))) -}} {{include "cozystack.platform.package.full" (list "cozystack.monitoring-agents" "default" $ $monitoringAgentsComponents) }} +{{include "cozystack.platform.package.full.default" (list "cozystack.observability-proxy" $) }} {{include "cozystack.platform.package.full.default" (list "cozystack.goldpinger" $) }} {{include "cozystack.platform.package.default" (list "cozystack.prometheus-operator-crds" $) }} {{include "cozystack.platform.package.full.default" (list "cozystack.grafana-operator" $) }} diff --git a/packages/core/platform/tests/fixtures/selective-install-postgres-values.yaml b/packages/core/platform/tests/fixtures/selective-install-postgres-values.yaml index 221aeeb27..6758b1b42 100644 --- a/packages/core/platform/tests/fixtures/selective-install-postgres-values.yaml +++ b/packages/core/platform/tests/fixtures/selective-install-postgres-values.yaml @@ -42,6 +42,7 @@ bundles: - cozystack.monitoring - cozystack.nats-application - cozystack.nfs-driver + - cozystack.observability-proxy - cozystack.openbao-application - cozystack.opensearch-application - cozystack.opensearch-operator diff --git a/packages/core/platform/tests/observability_proxy_wiring_test.yaml b/packages/core/platform/tests/observability_proxy_wiring_test.yaml new file mode 100644 index 000000000..9e6ec583d --- /dev/null +++ b/packages/core/platform/tests/observability_proxy_wiring_test.yaml @@ -0,0 +1,69 @@ +suite: observability-proxy package wiring +# The proxy reads the tenant log and metrics stores, which the full variants +# run by default and the slim ones only on request, so it follows +# monitoring-agents: on by default in the full variants, opt-in on the slim. +templates: + - templates/sources.yaml + - templates/bundles/system.yaml +release: + name: cozystack + namespace: cozy-system +tests: + - it: installs into cozy-observability-proxy once networking is up + template: templates/sources.yaml + documentSelector: + path: metadata.name + value: cozystack.observability-proxy + asserts: + - equal: + path: kind + value: PackageSource + - equal: + path: spec.variants[0].dependsOn + value: + - cozystack.networking + - equal: + path: spec.variants[0].components + value: + - name: observability-proxy + path: system/observability-proxy + install: + namespace: cozy-observability-proxy + releaseName: observability-proxy + + - it: emits the package on a full variant + template: templates/bundles/system.yaml + values: + - ../values-isp-full.yaml + asserts: + - containsDocument: + apiVersion: cozystack.io/v1alpha1 + kind: Package + name: cozystack.observability-proxy + any: true + + - it: leaves the package out of a slim variant by default + template: templates/bundles/system.yaml + values: + - ../values-isp-slim.yaml + asserts: + - containsDocument: + apiVersion: cozystack.io/v1alpha1 + kind: Package + name: cozystack.observability-proxy + any: true + not: true + + - it: emits the package on a slim variant that lists it in enabledPackages + template: templates/bundles/system.yaml + values: + - ../values-isp-slim.yaml + set: + bundles.enabledPackages: + - cozystack.observability-proxy + asserts: + - containsDocument: + apiVersion: cozystack.io/v1alpha1 + kind: Package + name: cozystack.observability-proxy + any: true From f7719cf6e2b769d76c190d9406ed0afba13dcb25 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:21:22 +0700 Subject: [PATCH 220/518] feat(tenant): allow egress to the observability proxy A tenant's Grafana reads the tenant's logs and metrics through the observability proxy, and tenant egress is default-deny with one allow-to-* policy per platform service. The rule is narrowed to the proxy pods and their port, since the proxy's namespace holds nothing else a tenant should reach. Assisted-by: LLM Signed-off-by: ZverGuy --- .../apps/tenant/templates/networkpolicy.yaml | 17 ++++++ ...etworkpolicy_observability_proxy_test.yaml | 61 +++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml diff --git a/packages/apps/tenant/templates/networkpolicy.yaml b/packages/apps/tenant/templates/networkpolicy.yaml index 5285b495a..76959b89f 100644 --- a/packages/apps/tenant/templates/networkpolicy.yaml +++ b/packages/apps/tenant/templates/networkpolicy.yaml @@ -187,6 +187,23 @@ spec: --- apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy +metadata: + name: allow-to-observability-proxy + namespace: {{ include "tenant.name" . }} +spec: + endpointSelector: {} + egress: + - toEndpoints: + - matchLabels: + "k8s:io.kubernetes.pod.namespace": cozy-observability-proxy + app.kubernetes.io/name: observability-proxy + toPorts: + - ports: + - port: "8080" + protocol: TCP +--- +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy metadata: # Allow the tenant's ingress-nginx to reach both the linstor-gui # gatekeeper pods (for /oauth2/* and app traffic) AND the transient diff --git a/packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml b/packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml new file mode 100644 index 000000000..d6276ec34 --- /dev/null +++ b/packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml @@ -0,0 +1,61 @@ +suite: tenant egress to the observability proxy +# A tenant's Grafana reads the tenant's logs and metrics through the +# observability proxy in cozy-observability-proxy. Tenant egress is +# default-deny plus explicit allow-to-* policies, so without this one those +# reads time out. The console reaches the proxy from cozy-dashboard and needs +# no tenant rule. +templates: + - templates/networkpolicy.yaml +tests: + - it: renders in the root tenant + release: + name: tenant-root + namespace: tenant-root + documentSelector: + path: metadata.name + value: allow-to-observability-proxy + asserts: + - isKind: + of: CiliumNetworkPolicy + - equal: + path: metadata.namespace + value: tenant-root + + - it: renders in a nested tenant with every module off + release: + name: tenant-bar + namespace: tenant-foo + set: + monitoring: false + ingress: false + etcd: false + documentSelector: + path: metadata.name + value: allow-to-observability-proxy + asserts: + - equal: + path: metadata.namespace + value: tenant-foo-bar + + - it: lets every pod of the tenant reach only the proxy pods on their container port + release: + name: tenant-root + namespace: tenant-root + documentSelector: + path: metadata.name + value: allow-to-observability-proxy + asserts: + - equal: + path: spec.endpointSelector + value: {} + - equal: + path: spec.egress + value: + - toEndpoints: + - matchLabels: + "k8s:io.kubernetes.pod.namespace": cozy-observability-proxy + app.kubernetes.io/name: observability-proxy + toPorts: + - ports: + - port: "8080" + protocol: TCP From 74a7a444234cb9e7f799e627d356caacf3436e62 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:24:43 +0700 Subject: [PATCH 221/518] feat(dashboard): route the observability proxy roots from the console The logs and metrics roots under /apis/cozystack.io and /apis/apps.cozystack.io are served by the observability proxy, not by the API server, so the console nginx sends them there ahead of its /apis prefix location, unbuffered and with the same long timeouts, so that a live tail reaches the browser as it is written. nginx resolves a literal upstream once at startup and refuses to start when the name does not resolve, which would take the console down on every cluster that does not run the proxy. The upstream is therefore a variable resolved per request, and such a cluster answers 502 on those paths only. That resolver does not apply the pod's search domains, so the config becomes a template the image's entrypoint renders with the pod's nameservers and the platform cluster domain, which the chart now passes to the console. Assisted-by: LLM Signed-off-by: ZverGuy --- hack/console-nginx-header-buffers.bats | 2 +- hack/console-nginx-observability-proxy.bats | 146 ++++++++++++++++++ .../dashboard/images/console/Containerfile | 26 +++- .../system/dashboard/templates/console.yaml | 5 + .../tests/console_cluster_domain_test.yaml | 42 +++++ 5 files changed, 219 insertions(+), 2 deletions(-) create mode 100644 hack/console-nginx-observability-proxy.bats create mode 100644 packages/system/dashboard/tests/console_cluster_domain_test.yaml diff --git a/hack/console-nginx-header-buffers.bats b/hack/console-nginx-header-buffers.bats index f8ce4d949..e7c143e91 100644 --- a/hack/console-nginx-header-buffers.bats +++ b/hack/console-nginx-header-buffers.bats @@ -37,7 +37,7 @@ nginx_size_kib() { cd "$REPO_ROOT" conf=$(awk ' - /^COPY <<.EOF. \/etc\/nginx\/conf\.d\/default\.conf$/ { inblock = 1; next } + /^COPY <<.EOF. \/etc\/nginx\/templates\/default\.conf\.template$/ { inblock = 1; next } inblock && /^EOF$/ { inblock = 0; next } inblock { print } ' "$CONSOLE_CONTAINERFILE") diff --git a/hack/console-nginx-observability-proxy.bats b/hack/console-nginx-observability-proxy.bats new file mode 100644 index 000000000..0efb0ffcd --- /dev/null +++ b/hack/console-nginx-observability-proxy.bats @@ -0,0 +1,146 @@ +#!/usr/bin/env bats +# The console nginx sends the logs and metrics roots of the observability +# proxy to that proxy instead of the API server. The routing lives in the +# nginx config baked into the console image, which no chart template and no +# helm-unittest case can see, so it is pinned here. +# +# Three properties are checked. The location regex must take the four path +# shapes the proxy serves and leave every ordinary Kubernetes API path to the +# /apis and /api prefix locations; nginx runs the regex locations whenever the +# longest prefix match is a plain prefix, so the prefix locations must not +# carry ^~, which would stop that. The location must stream: tail and watch +# style responses stall behind proxy buffering and die at the default 60s +# read timeout. And the upstream must be a variable resolved per request, +# because nginx resolves a literal upstream host once at startup and refuses +# to start when it does not resolve, which is the case on every cluster that +# does not run the proxy. +# +# The regex is evaluated with perl, whose syntax nginx's PCRE accepts for the +# constructs used here. +# Run with: bats hack/console-nginx-observability-proxy.bats + +load test_helper + +REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)" +CONSOLE_CONTAINERFILE="packages/system/dashboard/images/console/Containerfile" + +console_nginx_template() { + awk ' + /^COPY <<.EOF. \/etc\/nginx\/templates\/default\.conf\.template$/ { inblock = 1; next } + inblock && /^EOF$/ { inblock = 0; next } + inblock { print } + ' "$REPO_ROOT/$CONSOLE_CONTAINERFILE" +} + +# Prints the regex of every `location ~ {` line, one per line. +regex_locations() { + printf '%s\n' "$1" | awk '$1 == "location" && $2 == "~" && $4 == "{" { print $3 }' +} + +# Prints the body of the location block whose regex is $2. +location_body() { + # Through ENVIRON rather than -v, which would expand the backslashes. + printf '%s\n' "$1" | LOCATION_RE="$2" awk ' + $1 == "location" && $2 == "~" && $3 == ENVIRON["LOCATION_RE"] { inblock = 1; next } + inblock && $1 == "}" { exit } + inblock { print } + ' +} + +matches() { + perl -e 'exit(($ARGV[1] =~ /$ARGV[0]/) ? 0 : 1)' "$1" "$2" +} + +@test "console nginx routes exactly the observability proxy roots to the proxy" { + conf=$(console_nginx_template) + [ -n "$conf" ] || { echo "no /etc/nginx/templates/default.conf.template heredoc in $CONSOLE_CONTAINERFILE" >&2; exit 1; } + + regexes=$(regex_locations "$conf") + [ "$(printf '%s\n' "$regexes" | grep -c .)" -eq 1 ] \ + || { echo "expected exactly one regex location, got: $regexes" >&2; exit 1; } + re=$regexes + + for path in \ + /apis/cozystack.io/v1alpha1/namespaces/tenant-foo/logs/select/logsql/query \ + /apis/cozystack.io/v1alpha1/namespaces/tenant-foo/metrics/api/v1/query_range \ + /apis/apps.cozystack.io/v1alpha1/namespaces/tenant-foo/postgreses/db/log/select/logsql/tail \ + /apis/apps.cozystack.io/v1alpha1/namespaces/tenant-foo/vminstances/vm1/metrics/api/v1/query + do + matches "$re" "$path" || { echo "$path does not reach the proxy" >&2; exit 1; } + done + + for path in \ + /apis/cozystack.io/v1alpha1/namespaces/tenant-foo/logs \ + /apis/cozystack.io/v1alpha1/namespaces/tenant-foo/workloadmonitors/db \ + /apis/cozystack.io/v1alpha1/applicationdefinitions/postgres \ + /apis/cozystack.io/v1alpha1/watch/namespaces/tenant-foo/logs/ \ + /apis/apps.cozystack.io/v1alpha1/namespaces/tenant-foo/postgreses \ + /apis/apps.cozystack.io/v1alpha1/namespaces/tenant-foo/postgreses/log \ + /apis/apps.cozystack.io/v1alpha1/namespaces/tenant-foo/postgreses/log/ \ + /apis/apps.cozystack.io/v1alpha1/namespaces/tenant-foo/postgreses/db/status \ + /apis/apps.cozystack.io/v1alpha1/namespaces/tenant-foo/postgreses/db/log \ + /apis/apps.cozystack.io/v1alpha1/watch/namespaces/tenant-foo/postgreses/db/log/ \ + /apis/metrics.k8s.io/v1beta1/namespaces/tenant-foo/pods/ \ + /api/v1/namespaces/tenant-foo/pods/db-0/log \ + /k8s/apis/cozystack.io/v1alpha1/namespaces/tenant-foo/logs/select/logsql/query + do + if matches "$re" "$path"; then + echo "FAIL: $path would reach the proxy instead of the API server" >&2 + false + fi + done + + # A ^~ on the /apis or /api prefix would stop nginx from trying the regex. + if printf '%s\n' "$conf" | grep -Eq '^[[:space:]]*location[[:space:]]+\^~'; then + echo "FAIL: a ^~ prefix location disables the regex location" >&2 + false + fi +} + +@test "the observability proxy location streams, forwards the token and resolves per request" { + conf=$(console_nginx_template) + re=$(regex_locations "$conf") + body=$(location_body "$conf" "$re") + [ -n "$body" ] || { echo "no body found for location ~ $re" >&2; exit 1; } + + for directive in \ + 'proxy_http_version 1.1;' \ + 'proxy_buffering off;' \ + 'proxy_read_timeout 86400s;' \ + 'proxy_send_timeout 86400s;' + do + printf '%s\n' "$body" | grep -Fqx " $directive" \ + || { echo "the proxy location lacks '$directive'" >&2; exit 1; } + done + + # nginx forwards client headers unless told otherwise; the proxy needs the + # Authorization header the gatekeeper set. + if printf '%s\n' "$body" | grep -Eq 'proxy_set_header[[:space:]]+Authorization|proxy_pass_request_headers[[:space:]]+off'; then + echo "FAIL: the proxy location drops or rewrites the Authorization header" >&2 + false + fi + + printf '%s\n' "$body" | grep -Eq '^[[:space:]]*resolver[[:space:]]+\$\{NGINX_LOCAL_RESOLVERS\}' \ + || { echo "the proxy location sets no resolver from the pod's resolv.conf" >&2; exit 1; } + printf '%s\n' "$body" | grep -Eq '^[[:space:]]*set[[:space:]]+\$observability_proxy[[:space:]]+observability-proxy\.cozy-observability-proxy\.svc\.\$\{CLUSTER_DOMAIN\}:80;' \ + || { echo "the proxy location does not name the Service by its fully qualified name" >&2; exit 1; } + printf '%s\n' "$body" | grep -Eq '^[[:space:]]*proxy_pass[[:space:]]+http://\$observability_proxy;' \ + || { echo "the proxy location does not proxy_pass through the variable" >&2; exit 1; } +} + +@test "the console image defines every variable its nginx template substitutes" { + cd "$REPO_ROOT" + conf=$(console_nginx_template) + + # The image's entrypoint substitutes only variables present in the + # environment; one that is not would reach nginx as ${NAME} and fail it. + grep -Eq '^ENV NGINX_ENTRYPOINT_LOCAL_RESOLVERS=1$' "$CONSOLE_CONTAINERFILE" \ + || { echo "the image does not ask its entrypoint for NGINX_LOCAL_RESOLVERS" >&2; exit 1; } + grep -Eq '^ENV CLUSTER_DOMAIN=cozy\.local$' "$CONSOLE_CONTAINERFILE" \ + || { echo "the image has no CLUSTER_DOMAIN default" >&2; exit 1; } + + used=$(printf '%s\n' "$conf" | grep -oE '\$\{[A-Za-z_][A-Za-z0-9_]*\}' | sort -u) + expected=$(printf '%s\n' '${CLUSTER_DOMAIN}' '${NGINX_LOCAL_RESOLVERS}') + [ "$used" = "$expected" ] \ + || { echo "template substitutes: $used; the image defines: $expected" >&2; exit 1; } +} diff --git a/packages/system/dashboard/images/console/Containerfile b/packages/system/dashboard/images/console/Containerfile index 84d634f46..dba07f8d0 100644 --- a/packages/system/dashboard/images/console/Containerfile +++ b/packages/system/dashboard/images/console/Containerfile @@ -32,7 +32,14 @@ FROM nginxinc/nginx-unprivileged:1.29-alpine@sha256:0c79d56aee561a1d81c63f00eee5 COPY --from=builder /src/apps/console/dist /usr/share/nginx/html -COPY <<'EOF' /etc/nginx/conf.d/default.conf +# The config is a template the image's entrypoint renders into conf.d at +# startup, substituting only variables set in the environment: CLUSTER_DOMAIN, +# which the chart overrides with the platform cluster domain, and +# NGINX_LOCAL_RESOLVERS, which the entrypoint reads from the pod's resolv.conf. +ENV CLUSTER_DOMAIN=cozy.local +ENV NGINX_ENTRYPOINT_LOCAL_RESOLVERS=1 + +COPY <<'EOF' /etc/nginx/templates/default.conf.template map $http_upgrade $connection_upgrade { default upgrade; '' close; @@ -54,6 +61,23 @@ server { root /usr/share/nginx/html; index index.html; + # Logs and metrics roots of a namespace or of one application, served by + # the observability proxy rather than the API server. A regex location wins + # over the /apis prefix below. The upstream is a variable so nginx resolves + # it per request: on a cluster without the proxy these paths answer 502 + # instead of nginx refusing to start. The resolver does not apply search + # domains, hence the fully qualified name. + location ~ ^/apis/(?:cozystack\.io/v1alpha1/namespaces/[^/]+/(?:logs|metrics)|apps\.cozystack\.io/v1alpha1/namespaces/[^/]+/[^/]+/[^/]+/(?:log|metrics))/ { + resolver ${NGINX_LOCAL_RESOLVERS}; + set $observability_proxy observability-proxy.cozy-observability-proxy.svc.${CLUSTER_DOMAIN}:80; + proxy_http_version 1.1; + proxy_buffering off; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_read_timeout 86400s; + proxy_send_timeout 86400s; + proxy_pass http://$observability_proxy; + } + # Kubernetes API proxy for both core and aggregated groups. # Chunked-encoding watches require proxy_buffering off and a long timeout. location /apis { diff --git a/packages/system/dashboard/templates/console.yaml b/packages/system/dashboard/templates/console.yaml index c4e641578..c05f3ee4b 100644 --- a/packages/system/dashboard/templates/console.yaml +++ b/packages/system/dashboard/templates/console.yaml @@ -40,6 +40,11 @@ spec: containers: - image: {{ .Values.console.image | quote }} imagePullPolicy: IfNotPresent + {{- with index .Values._cluster "cluster-domain" }} + env: + - name: CLUSTER_DOMAIN + value: {{ . | quote }} + {{- end }} livenessProbe: failureThreshold: 3 httpGet: diff --git a/packages/system/dashboard/tests/console_cluster_domain_test.yaml b/packages/system/dashboard/tests/console_cluster_domain_test.yaml new file mode 100644 index 000000000..c5e605638 --- /dev/null +++ b/packages/system/dashboard/tests/console_cluster_domain_test.yaml @@ -0,0 +1,42 @@ +suite: console cluster domain +# The console nginx names the observability proxy by its fully qualified +# Service name, because the resolver it uses for a per-request upstream does +# not apply the pod's search domains. The image defaults to cozy.local; the +# chart passes the platform cluster domain so a cluster on another domain +# still reaches the proxy. +templates: + - templates/console.yaml + - templates/configmap.yaml + +release: + name: dashboard + namespace: cozy-dashboard + +tests: + - it: passes the platform cluster domain to the console + template: templates/console.yaml + set: + _cluster: + cluster-domain: cluster.local + asserts: + - contains: + path: spec.template.spec.containers[0].env + content: + name: CLUSTER_DOMAIN + value: cluster.local + + - it: leaves the image default when the platform passes no cluster domain + template: templates/console.yaml + asserts: + - notExists: + path: spec.template.spec.containers[0].env + + # The entrypoint renders the nginx template into /etc/nginx/conf.d at + # startup. On a read-only root it logs the failure and starts nginx on the + # stock default.conf, which serves no /apis, /api or proxy route at all. + - it: keeps the root filesystem writable for the rendered nginx config + template: templates/console.yaml + asserts: + - equal: + path: spec.template.spec.containers[0].securityContext.readOnlyRootFilesystem + value: false From d452b30e42a1fac8fc61a3ce15d3a55a4d384f5f Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:31:09 +0700 Subject: [PATCH 222/518] test(dashboard): name the templated nginx config in the buffer check The console nginx config now ships as an entrypoint template, and a failure message naming the old conf.d path sends the reader to a file the Containerfile no longer writes. Assisted-by: LLM Signed-off-by: ZverGuy --- hack/console-nginx-header-buffers.bats | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hack/console-nginx-header-buffers.bats b/hack/console-nginx-header-buffers.bats index e7c143e91..21932fc25 100644 --- a/hack/console-nginx-header-buffers.bats +++ b/hack/console-nginx-header-buffers.bats @@ -41,7 +41,7 @@ nginx_size_kib() { inblock && /^EOF$/ { inblock = 0; next } inblock { print } ' "$CONSOLE_CONTAINERFILE") - [ -n "$conf" ] || { echo "no /etc/nginx/conf.d/default.conf heredoc found in $CONSOLE_CONTAINERFILE" >&2; exit 1; } + [ -n "$conf" ] || { echo "no /etc/nginx/templates/default.conf.template heredoc found in $CONSOLE_CONTAINERFILE" >&2; exit 1; } # An extraction that silently grabbed the wrong slice would let every # assertion below pass on an empty-ish config, so anchor on a line the From a56ff5050f913859d2a98cf812fdbfd014b2ff30 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:58:42 +0700 Subject: [PATCH 223/518] test(observability-proxy): accept the smallest satisfiable step cap Startup refuses --logs-max-buckets below 3 and --logs-max-range below 3s, but only the refusals were tested, so moving either bound up by one went unnoticed. Pin the smallest accepted pair. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main_test.go | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index 924c06f28..0a96db4bc 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -42,6 +42,10 @@ func TestParseFlagsOverrides(t *testing.T) { if o.clusterDomain != "example.org" || o.logs.MaxRange != 24*time.Hour || o.limiter.MaxTails != 3 { t.Errorf("overrides not applied: %+v", o) } + // The smallest bucket count and range the step cap can satisfy. + if _, err := parseFlags([]string{"--logs-max-buckets=3", "--logs-max-range=3s", "--logs-default-range=1s"}); err != nil { + t.Errorf("smallest satisfiable step cap refused: %v", err) + } } func TestParseFlagsInvalid(t *testing.T) { From 28b391ff176943ccd54fdfdbaef3f7f0beb9ccc6 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:58:42 +0700 Subject: [PATCH 224/518] fix(observability-proxy): round the step up to whole milliseconds The step is sent to vlselect in milliseconds and was truncated, so a window that the bucket cap does not divide evenly got one bucket more than --logs-max-buckets allows. Round it up before the window is shrunk to fit the step, so both caps still hold. Bare-seconds steps and start offsets are now tested on the accepting side too: only the refused forms were, and disabling the digit branch passed the suite. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 4 ++++ .../observabilityproxy/logsparams_test.go | 24 +++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index c9fad47db..7be252998 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -179,6 +179,10 @@ func setStep(out, in url.Values, start, end time.Time, lim LogsLimits) *HTTPErro } step = max(min(d, span, lim.MaxRange/3), minStep) } + // vlselect gets whole milliseconds; rounding down would add a bucket. + if r := step % time.Millisecond; r != 0 { + step += time.Millisecond - r + } if span+2*step > lim.MaxRange { start = end.Add(-max(lim.MaxRange-2*step, 0)) } diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index abfbc3483..c99639725 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -227,6 +227,12 @@ func TestBuildLogsParamsStepAndLimit(t *testing.T) { if got := build(t, "stats_query_range", with(day, "step", "1w")).Get("step"); got != "86400000ms" { t.Errorf("1w step on a day window = %s, want capped at the window", got) } + if got := build(t, "hits", with(day, "step", "3600")).Get("step"); got != "3600000ms" { + t.Errorf("bare-seconds step = %s, want 3600000ms", got) + } + if got := build(t, "hits", with(day, "step", "5400.5")).Get("step"); got != "5400500ms" { + t.Errorf("fractional bare-seconds step = %s, want 5400500ms", got) + } if got := build(t, "hits", day).Get("step"); got != "86400ms" { t.Errorf("omitted step = %s, want the bucket-bounded minimum", got) } @@ -271,6 +277,7 @@ func TestBuildLogsParamsTail(t *testing.T) { {"since in the future", url.Values{"since": {"2026-11-04T13:00:00Z"}}, "0ms"}, {"start_offset within cap", url.Values{"start_offset": {"2m"}}, "120000ms"}, {"start_offset over cap", url.Values{"start_offset": {"30d"}}, "900000ms"}, + {"start_offset in bare seconds", url.Values{"start_offset": {"120"}}, "120000ms"}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -328,3 +335,20 @@ func TestBuildLogsParamsStepCannotWidenWindow(t *testing.T) { } } } + +func TestBuildLogsParamsStepKeepsBucketCap(t *testing.T) { + lim := testLimits() + // Spans that MaxBuckets does not divide into whole milliseconds. + for _, start := range []string{"2026-11-03T12:00:00.0007Z", "2026-11-04T11:59:58.3333Z", "6d23h59m59s"} { + out := build(t, "hits", url.Values{"start": {start}}) + step, err := parseDuration(out.Get("step")) + if err != nil { + t.Fatal(err) + } + s, _ := time.Parse(time.RFC3339Nano, out.Get("start")) + e, _ := time.Parse(time.RFC3339Nano, out.Get("end")) + if buckets := (e.Sub(s) + step - 1) / step; int(buckets) > lim.MaxBuckets { + t.Errorf("start %s: step %s gives %d buckets, more than %d", start, out.Get("step"), buckets, lim.MaxBuckets) + } + } +} From ebb5a9cec4f64e492d686e2679077fc98b757657 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:07:49 +0700 Subject: [PATCH 225/518] test(observability-proxy): cover the namespace logs scope end to end Unit tests mock the API server and the store, so nothing shows that the console's nginx, the proxy's TokenReview and SubjectAccessReview and a real VictoriaLogs agree. The suite drives a viewer of one tenant through the console against a seeded store: it reads its own namespace, is refused a sibling's, cannot widen the scope with client filters, an AccountID header or dot segments, and keeps a tail open past a minute. The root store is off in e2e, so the suite runs its own VictoriaLogs under the root store's Service name rather than waiting on Fluent Bit. Assisted-by: LLM Signed-off-by: ZverGuy --- hack/e2e-chainsaw/README.md | 4 +- .../observability-proxy/chainsaw-test.yaml | 251 ++++++++++++++++++ hack/e2e-chainsaw/observability-proxy/lib.sh | 57 ++++ .../observability-proxy/log-store.yaml | 71 +++++ .../observability-proxy/viewer.yaml | 22 ++ 5 files changed, 403 insertions(+), 2 deletions(-) create mode 100644 hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml create mode 100644 hack/e2e-chainsaw/observability-proxy/lib.sh create mode 100644 hack/e2e-chainsaw/observability-proxy/log-store.yaml create mode 100644 hack/e2e-chainsaw/observability-proxy/viewer.yaml diff --git a/hack/e2e-chainsaw/README.md b/hack/e2e-chainsaw/README.md index 98400bc55..ef5a50db4 100644 --- a/hack/e2e-chainsaw/README.md +++ b/hack/e2e-chainsaw/README.md @@ -26,7 +26,7 @@ A suite is a directory directly under `hack/e2e-chainsaw/` in or below which Cha A parked `chainsaw-test.yaml.disabled` is none of these: the directory is then not a suite, is selected by nothing and cannot be allowlisted. -Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `kafka-metadata`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `monitoring`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`, `computeplane`. `kubernetes-proxmox` is parked as `chainsaw-test.yaml.disabled` and runs only in `.github/workflows/e2e-proxmox.yaml`, which needs a Proxmox hypervisor. +Suites: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `valkey`, `qdrant`, `clickhouse`, `kafka`, `kafka-metadata`, `rabbitmq`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `opensearch`, `monitoring`, `kubernetes-latest`, `kubernetes-previous`, `securitygroup`, `seaweedfs`, `computeplane`, `observability-proxy`. `kubernetes-proxmox` is parked as `chainsaw-test.yaml.disabled` and runs only in `.github/workflows/e2e-proxmox.yaml`, which needs a Proxmox hypervisor. ## What Chainsaw buys over the BATS suite @@ -63,7 +63,7 @@ Declarative suites assert on `status.conditions` and concrete fields. Inherently ## Running locally -Requires a cluster with Cozystack installed and a `tenant-test` namespace (the environment `hack/e2e-install-cozystack.bats` produces). `bucket` and `seaweedfs` additionally need `mc`, `nc`, and `jq` on the host; `kuberture`/`openbao`/`foundationdb`/`securitygroup`/`computeplane`/`kubernetes-latest`/`kubernetes-previous`/`vminstance` need `jq`. +Requires a cluster with Cozystack installed and a `tenant-test` namespace (the environment `hack/e2e-install-cozystack.bats` produces). `bucket` and `seaweedfs` additionally need `mc`, `nc`, and `jq` on the host; `kuberture`/`openbao`/`foundationdb`/`securitygroup`/`computeplane`/`kubernetes-latest`/`kubernetes-previous`/`vminstance` need `jq`; `observability-proxy` needs `curl` and `nc`. ```bash # install chainsaw diff --git a/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml b/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml new file mode 100644 index 000000000..837d72103 --- /dev/null +++ b/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml @@ -0,0 +1,251 @@ +# Namespace logs through the observability proxy, as the console reaches +# them: every request goes through the console nginx (a port-forward to its +# Service), which routes the logs root to the proxy, which authenticates the +# bearer token, asks a SubjectAccessReview and forwards a rebuilt query to +# the tenant's log store. +# +# Two sibling tenants, A (logsa) and B (logsb), both under tenant-test, so +# both resolve to the root log store. A viewer of A reads A's logs and is +# refused B's; parameters and headers a client adds cannot widen the scope +# (the proxy forces the namespace filter and tenant 0:0); a live tail stays +# open and keeps delivering lines past 60s. +# +# Not covered here: the OIDC gatekeeper (oauth2-proxy) in front of nginx, +# because driving a Keycloak login needs a browser; and Fluent Bit writing +# real pod logs, because the root log store is off in e2e and this suite +# seeds its own (log-store.yaml). +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: observability-proxy-logs +spec: + catch: + - events: {} + - podLogs: + namespace: cozy-observability-proxy + selector: app.kubernetes.io/name=observability-proxy + tail: 100 + - podLogs: + namespace: tenant-root + selector: app.kubernetes.io/name=e2e-observability-log-store + tail: 100 + - podLogs: + namespace: cozy-dashboard + selector: app.kubernetes.io/name=console + tail: 100 + steps: + - name: proxy-available + try: + - assert: + resource: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: observability-proxy + namespace: cozy-observability-proxy + status: + (conditions[?type == 'Available']): + - status: "True" + + - name: tenants + try: + - apply: + resource: + apiVersion: apps.cozystack.io/v1alpha1 + kind: Tenant + metadata: + name: logsa + spec: {} + - apply: + resource: + apiVersion: apps.cozystack.io/v1alpha1 + kind: Tenant + metadata: + name: logsb + spec: {} + # 16m: tenant-rd's 15m install timeout plus the 60s start margin; see + # "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md. Waited so + # that cleanup does not uninstall a release whose install still runs. + - assert: + timeout: 16m + resource: + apiVersion: helm.toolkit.fluxcd.io/v2 + kind: HelmRelease + metadata: + name: tenant-logsa + status: + (conditions[?type == 'Ready']): + - status: "True" + - assert: + timeout: 16m + resource: + apiVersion: helm.toolkit.fluxcd.io/v2 + kind: HelmRelease + metadata: + name: tenant-logsb + status: + (conditions[?type == 'Ready']): + - status: "True" + # The label names the store the proxy reads; the seeded store below + # stands in for the root one, so both namespaces must resolve to it. + - assert: + resource: + apiVersion: v1 + kind: Namespace + metadata: + name: tenant-test-logsa + labels: + namespace.cozystack.io/monitoring: tenant-root + - assert: + resource: + apiVersion: v1 + kind: Namespace + metadata: + name: tenant-test-logsb + labels: + namespace.cozystack.io/monitoring: tenant-root + + - name: viewer-and-store + try: + # Applying over a real root store would rewrite its Service; refuse to. + - error: + resource: + apiVersion: v1 + kind: Service + metadata: + name: vlselect-generic + namespace: tenant-root + - apply: + file: viewer.yaml + - apply: + file: log-store.yaml + - assert: + timeout: 3m + resource: + apiVersion: apps/v1 + kind: Deployment + metadata: + name: e2e-observability-log-store + namespace: tenant-root + status: + (conditions[?type == 'Available']): + - status: "True" + + - name: seed-logs + try: + - script: + # 30s forward + 4 x 15s ingests + the 30s visibility poll, each + # capped on its own, under a 3m op. + timeout: 3m + content: | + set -eu + . ./lib.sh + trap 'for p in ${pf_pids:-}; do kill "$p" 2>/dev/null || true; done; rm -f "$body"' EXIT + forward tenant-root vlselect-generic 19471:9471 + ingest tenant-test-logsa e2e-marker-a + ingest tenant-test-logsb e2e-marker-b + # Same namespace, another VictoriaLogs tenant: reachable only if a + # client AccountID header got through. + ingest tenant-test-logsa e2e-marker-account7 7 + timeout 30 sh -ec "until curl --silent --fail '$STORE/select/logsql/query?query=e2e-marker-a' | grep -q e2e-marker-a; do sleep 1; done" + + - name: namespace-scope + try: + - script: + # 2 x 30s forwards + one 30s token request + 11 calls capped at 30s. + timeout: 8m + content: | + set -eu + . ./lib.sh + trap 'for p in ${pf_pids:-}; do kill "$p" 2>/dev/null || true; done; rm -f "$body"' EXIT + forward cozy-dashboard cozy-dashboard-console 18080:8080 + token=$(k -n tenant-test-logsa create token logs-viewer --duration=10m) + auth="Authorization: Bearer $token" + + code=$(call "$LOGS_A/query?query=*") + expect 401 "$code" "no token" + + code=$(call "$LOGS_A/query?query=*" --header "$auth") + expect 200 "$code" "viewer reads A" + grep -q e2e-marker-a "$body" || { echo "FAIL: A's own line is missing" >&2; cat "$body" >&2; exit 1; } + absent e2e-marker-b "viewer reads A" + absent e2e-marker-account7 "viewer reads A" + + code=$(call "$LOGS_B/query?query=*" --header "$auth") + expect 403 "$code" "viewer reads B" + + code=$(call "$LOGS_A/query" --header "$auth" --get \ + --data-urlencode 'query=*' \ + --data-urlencode 'extra_stream_filters={kubernetes_namespace_name="tenant-test-logsb"}' \ + --data-urlencode 'extra_filters=kubernetes_namespace_name:tenant-test-logsb') + expect 200 "$code" "client extra filters in the query string" + absent e2e-marker-b "client extra filters in the query string" + + code=$(call "$LOGS_A/query" --header "$auth" \ + --data-urlencode 'query=*' \ + --data-urlencode 'extra_stream_filters={kubernetes_namespace_name="tenant-test-logsb"}') + expect 200 "$code" "client extra filters in a POST form" + absent e2e-marker-b "client extra filters in a POST form" + + code=$(call "$LOGS_A/query?query=*" --header "$auth" --header "AccountID: 7" --header "ProjectID: 0") + expect 200 "$code" "client AccountID header" + absent e2e-marker-account7 "client AccountID header" + + code=$(call "$LOGS_A/query" --header "$auth" --get \ + --data-urlencode 'query={kubernetes_namespace_name="tenant-test-logsb"}') + expect 200 "$code" "B's stream named in the query" + absent e2e-marker-b "B's stream named in the query" + + code=$(call "$LOGS_A/query" --header "$auth" --get \ + --data-urlencode 'query=options(ignore_global_time_filter=true) *') + expect 400 "$code" "query options" + + # nginx routes on the normalised path, which is B's logs root, and + # forwards the raw one; the proxy must refuse it, not read B. + code=$(call "$LOGS_A/../../../../tenant-test-logsb/logs/select/logsql/query?query=*" --path-as-is --header "$auth") + expect 400 "$code" "dot segments toward B" + + code=$(call "$LOGS_A/delete?query=*" --header "$auth") + expect 404 "$code" "an endpoint outside the allowlist" + + - name: tail-outlives-a-minute + try: + - script: + # 2 x 30s forwards + 30s token + 3 x 15s ingests + the 70s hold + + # the 30s delivery poll; the tail itself is cut at 150s. + timeout: 6m + content: | + set -eu + . ./lib.sh + tail_out=$(mktemp) + trap 'kill "${tail_pid:-}" 2>/dev/null || true; for p in ${pf_pids:-}; do kill "$p" 2>/dev/null || true; done; rm -f "$body" "$tail_out"' EXIT + forward tenant-root vlselect-generic 19471:9471 + forward cozy-dashboard cozy-dashboard-console 18080:8080 + token=$(k -n tenant-test-logsa create token logs-viewer --duration=10m) + + timeout 150 curl --silent --no-buffer --header "Authorization: Bearer $token" \ + "$CONSOLE$LOGS_A/tail?query=*" > "$tail_out" & + tail_pid=$! + ingest tenant-test-logsa e2e-tail-early + ingest tenant-test-logsb e2e-tail-b + # TODO(e2e-replace-fixed-timeouts): the hold is the property under + # test, not a wait for a condition: a tail must survive a minute + # without the client sending anything. No event can replace it. + sleep 70 + if ! kill -0 "$tail_pid" 2>/dev/null; then + echo "FAIL: the tail closed before 70s" >&2 + cat "$tail_out" >&2 + exit 1 + fi + ingest tenant-test-logsa e2e-tail-late + # vlselect re-reads every second, five seconds behind now. + timeout 30 sh -ec "until grep -q e2e-tail-late '$tail_out'; do sleep 1; done" || { + echo "FAIL: a line written at 70s did not reach the open tail" >&2 + cat "$tail_out" >&2 + exit 1 + } + grep -q e2e-tail-early "$tail_out" || { echo "FAIL: the early line is missing from the tail" >&2; exit 1; } + if grep -q e2e-tail-b "$tail_out"; then + echo "FAIL: the tail of A delivered B's line" >&2 + exit 1 + fi diff --git a/hack/e2e-chainsaw/observability-proxy/lib.sh b/hack/e2e-chainsaw/observability-proxy/lib.sh new file mode 100644 index 000000000..ff41b3e5e --- /dev/null +++ b/hack/e2e-chainsaw/observability-proxy/lib.sh @@ -0,0 +1,57 @@ +# Shared by the script steps of this suite, whose cwd is this directory. +# Every call carries its own cap so a hang fails at a known budget rather +# than at the op ceiling (docs/agents/e2e-testing.md, section 8). + +k() { timeout 30 kubectl "$@"; } + +STORE=http://localhost:19471 +CONSOLE=http://localhost:18080 +# shellcheck disable=SC2034 # read by the steps that source this file +LOGS_A=/apis/cozystack.io/v1alpha1/namespaces/tenant-test-logsa/logs/select/logsql +# shellcheck disable=SC2034 +LOGS_B=/apis/cozystack.io/v1alpha1/namespaces/tenant-test-logsb/logs/select/logsql + +# forward :. The caller's EXIT trap +# kills every pid in pf_pids; the forward gets 30s to start listening. +forward() { + kubectl port-forward -n "$1" "service/$2" "$3" >/dev/null 2>&1 & + pf_pids="${pf_pids:-} $!" + timeout 30 sh -ec "until nc -z localhost ${3%%:*}; do sleep 1; done" +} + +# ingest [account]: one entry shaped as Fluent Bit +# writes it, with kubernetes_namespace_name as a stream field. +ingest() { + printf '{"kubernetes_namespace_name":"%s","log":"%s"}\n' "$1" "$2" | + timeout 15 curl --silent --show-error --fail --data-binary @- \ + --header "AccountID: ${3:-0}" --header "ProjectID: 0" \ + "$STORE/insert/jsonline?_stream_fields=kubernetes_namespace_name&_msg_field=log" +} + +# call [curl args...]: a request through the console nginx, the way +# the browser sends it. Prints the status; the body lands in $body. +body=$(mktemp) +call() { + path=$1 + shift + timeout 30 curl --silent --show-error --output "$body" --write-out '%{http_code}' "$@" "$CONSOLE$path" +} + +# expect +expect() { + if [ "$1" != "$2" ]; then + echo "FAIL: $3: HTTP $2, want $1" >&2 + cat "$body" >&2 + exit 1 + fi + echo "ok: $3: HTTP $2" +} + +# absent : the last body must not carry the marker. +absent() { + if grep -q "$1" "$body"; then + echo "FAIL: $2: the response carries $1, which is outside the scope" >&2 + cat "$body" >&2 + exit 1 + fi +} diff --git a/hack/e2e-chainsaw/observability-proxy/log-store.yaml b/hack/e2e-chainsaw/observability-proxy/log-store.yaml new file mode 100644 index 000000000..491dc667c --- /dev/null +++ b/hack/e2e-chainsaw/observability-proxy/log-store.yaml @@ -0,0 +1,71 @@ +# A single-node VictoriaLogs under the name the proxy dials for a namespace +# whose logs belong to the root tenant: vlselect-generic.tenant-root, port +# 9471. The e2e install leaves root monitoring off, so the real one does not +# exist; the suite seeds this one itself instead of waiting for Fluent Bit, +# which keeps the test about the proxy's scope rather than log collection. +# The version is the one victoria-metrics-operator v0.68.4 deploys. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: e2e-observability-log-store + namespace: tenant-root +spec: + replicas: 1 + selector: + matchLabels: + app.kubernetes.io/name: e2e-observability-log-store + template: + metadata: + labels: + app.kubernetes.io/name: e2e-observability-log-store + spec: + securityContext: + runAsNonRoot: true + runAsUser: 65534 + seccompProfile: + type: RuntimeDefault + containers: + - name: victoria-logs + image: docker.io/victoriametrics/victoria-logs:v1.47.0 + args: + - -httpListenAddr=:9471 + - -storageDataPath=/storage + - -retentionPeriod=1d + ports: + - name: http + containerPort: 9471 + readinessProbe: + httpGet: + path: /health + port: http + resources: + requests: + cpu: 50m + memory: 128Mi + limits: + cpu: 500m + memory: 256Mi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + volumeMounts: + - name: storage + mountPath: /storage + volumes: + - name: storage + emptyDir: {} +--- +apiVersion: v1 +kind: Service +metadata: + name: vlselect-generic + namespace: tenant-root +spec: + selector: + app.kubernetes.io/name: e2e-observability-log-store + ports: + - name: http + port: 9471 + targetPort: http diff --git a/hack/e2e-chainsaw/observability-proxy/viewer.yaml b/hack/e2e-chainsaw/observability-proxy/viewer.yaml new file mode 100644 index 000000000..0563ad4a0 --- /dev/null +++ b/hack/e2e-chainsaw/observability-proxy/viewer.yaml @@ -0,0 +1,22 @@ +# The view level of tenant A and nothing else: a ServiceAccount of tenant A +# bound to cozy:tenant:view in A's namespace only. Its token is a real bearer +# token, so the proxy's TokenReview and SubjectAccessReview run unmocked. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: logs-viewer + namespace: tenant-test-logsa +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: logs-viewer + namespace: tenant-test-logsa +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cozy:tenant:view +subjects: +- kind: ServiceAccount + name: logs-viewer + namespace: tenant-test-logsa From dd96a415a9b034ea7361aac29576f825d26c7001 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:16:47 +0700 Subject: [PATCH 226/518] test(observability-proxy): make every bucket-cap case need the rounding Two of the three windows divided into whole milliseconds, so they passed with the old truncating step and guarded nothing. Each window now leaves a sub-millisecond step remainder. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index c99639725..ddb95cdc5 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -339,7 +339,7 @@ func TestBuildLogsParamsStepCannotWidenWindow(t *testing.T) { func TestBuildLogsParamsStepKeepsBucketCap(t *testing.T) { lim := testLimits() // Spans that MaxBuckets does not divide into whole milliseconds. - for _, start := range []string{"2026-11-03T12:00:00.0007Z", "2026-11-04T11:59:58.3333Z", "6d23h59m59s"} { + for _, start := range []string{"2026-11-03T12:00:00.0007Z", "2026-11-04T10:00:00.0013Z", "2026-11-04T11:00:00.0005Z"} { out := build(t, "hits", url.Values{"start": {start}}) step, err := parseDuration(out.Get("step")) if err != nil { From 8784863632d55fc98f6bc2627ae0aa7b89c120c8 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:16:47 +0700 Subject: [PATCH 227/518] test(observability-proxy): wait for every seeded line before checking scope The suite checked that B's line and the account-7 line are absent from A's answers but only waited for A's line to become searchable, so an absence could come from a line the store had not indexed yet. The step budgets in the comments now count the calls the steps make. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability-proxy/chainsaw-test.yaml | 20 +++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml b/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml index 837d72103..e0656d4f8 100644 --- a/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml @@ -134,9 +134,9 @@ spec: - name: seed-logs try: - script: - # 30s forward + 4 x 15s ingests + the 30s visibility poll, each - # capped on its own, under a 3m op. - timeout: 3m + # 30s forward + 3 x 15s ingests + 3 x 30s visibility polls, each + # capped on its own, under a 4m op. + timeout: 4m content: | set -eu . ./lib.sh @@ -147,12 +147,16 @@ spec: # Same namespace, another VictoriaLogs tenant: reachable only if a # client AccountID header got through. ingest tenant-test-logsa e2e-marker-account7 7 - timeout 30 sh -ec "until curl --silent --fail '$STORE/select/logsql/query?query=e2e-marker-a' | grep -q e2e-marker-a; do sleep 1; done" + # All three must be searchable, or an absence checked later + # proves nothing. + for m in e2e-marker-a:0 e2e-marker-b:0 e2e-marker-account7:7; do + timeout 30 sh -ec "until curl --silent --fail --header 'AccountID: ${m#*:}' '$STORE/select/logsql/query?query=${m%:*}' | grep -q ${m%:*}; do sleep 1; done" + done - name: namespace-scope try: - script: - # 2 x 30s forwards + one 30s token request + 11 calls capped at 30s. + # 30s forward + one 30s token request + 10 calls capped at 30s. timeout: 8m content: | set -eu @@ -228,9 +232,9 @@ spec: tail_pid=$! ingest tenant-test-logsa e2e-tail-early ingest tenant-test-logsb e2e-tail-b - # TODO(e2e-replace-fixed-timeouts): the hold is the property under - # test, not a wait for a condition: a tail must survive a minute - # without the client sending anything. No event can replace it. + # TODO(e2e-replace-fixed-timeouts): this hold stands in for a + # duration assertion the harness lacks; the property under test is + # that a tail survives a minute with the client sending nothing. sleep 70 if ! kill -0 "$tail_pid" 2>/dev/null; then echo "FAIL: the tail closed before 70s" >&2 From 163b079845aa6d04c5cd988e11d263f4a956fcb1 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:22:39 +0700 Subject: [PATCH 228/518] fix(observability-proxy): answer 429 to rate and concurrency limits Clients such as Grafana and browser retry logic back off on 429 Too Many Requests, not on 422, which they treat as a final rejection of the query. The per-user rate limit, the per-user concurrency limit and the open-tail caps are about the caller's pace, not the query, so they now answer 429. The rate limiter knows when its bucket refills, so that answer carries Retry-After; the concurrency caps cannot know when a slot frees and omit it. Range, step, bucket and limit violations keep 422. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/limits.go | 10 ++++++---- internal/observabilityproxy/limits_test.go | 22 ++++++++++++++-------- internal/observabilityproxy/server.go | 8 +++++--- internal/observabilityproxy/server_test.go | 15 +++++++++------ 4 files changed, 34 insertions(+), 21 deletions(-) diff --git a/internal/observabilityproxy/limits.go b/internal/observabilityproxy/limits.go index 588a4f29e..f0392f25b 100644 --- a/internal/observabilityproxy/limits.go +++ b/internal/observabilityproxy/limits.go @@ -58,9 +58,11 @@ func newLimiter(cfg LimiterConfig) *limiter { } } -func (l *limiter) allowRate(user string) bool { +// allowRate takes a token from the user's bucket; when it is empty, it +// returns how long until the next token. +func (l *limiter) allowRate(user string) (time.Duration, bool) { if l.cfg.RatePerUser <= 0 { - return true + return 0, true } l.mu.Lock() defer l.mu.Unlock() @@ -77,10 +79,10 @@ func (l *limiter) allowRate(user string) bool { b.tokens = min(burst, b.tokens+now.Sub(b.last).Seconds()*l.cfg.RatePerUser) b.last = now if b.tokens < 1 { - return false + return time.Duration((1 - b.tokens) / l.cfg.RatePerUser * float64(time.Second)), false } b.tokens-- - return true + return 0, true } // sweepBuckets drops buckets that have refilled completely; recreating one diff --git a/internal/observabilityproxy/limits_test.go b/internal/observabilityproxy/limits_test.go index 167598971..5c3af772d 100644 --- a/internal/observabilityproxy/limits_test.go +++ b/internal/observabilityproxy/limits_test.go @@ -76,29 +76,35 @@ func TestLimiterRate(t *testing.T) { now := time.Unix(1000, 0) l := newLimiter(LimiterConfig{RatePerUser: 2, RateBurst: 2}) l.now = func() time.Time { return now } - if !l.allowRate("alice") || !l.allowRate("alice") { + if !allowed(l, "alice") || !allowed(l, "alice") { t.Fatal("burst refused") } - if l.allowRate("alice") { - t.Fatal("request over the burst admitted") + if wait, ok := l.allowRate("alice"); ok || wait != 500*time.Millisecond { + t.Fatalf("request over the burst: admitted %v, wait %s, want refused with 500ms", ok, wait) } - if !l.allowRate("bob") { + if !allowed(l, "bob") { t.Fatal("bob limited by alice's rate") } now = now.Add(500 * time.Millisecond) - if !l.allowRate("alice") { + if !allowed(l, "alice") { t.Fatal("token not refilled after 1/rate") } - if l.allowRate("alice") { - t.Fatal("refilled more than elapsed*rate") + now = now.Add(100 * time.Millisecond) + if wait, ok := l.allowRate("alice"); ok || wait != 400*time.Millisecond { + t.Fatalf("partly refilled bucket: admitted %v, wait %s, want refused with 400ms", ok, wait) } } func TestLimiterRateDisabled(t *testing.T) { l := newLimiter(LimiterConfig{}) for range 100 { - if !l.allowRate("alice") { + if !allowed(l, "alice") { t.Fatal("rate limit applied while disabled") } } } + +func allowed(l *limiter, user string) bool { + _, ok := l.allowRate(user) + return ok +} diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 4ea5cebee..fcdb46141 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -23,6 +23,7 @@ import ( "fmt" "io" "log/slog" + "math" "net/http" "strconv" "strings" @@ -150,8 +151,9 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT w.Header().Set("Allow", "GET, POST") return errorf(http.StatusMethodNotAllowed, "method not allowed") } - if !s.limits.allowRate(user) { - return errorf(http.StatusUnprocessableEntity, "request rate limit exceeded") + if wait, ok := s.limits.allowRate(user); !ok { + w.Header().Set("Retry-After", strconv.FormatInt(int64(math.Ceil(wait.Seconds())), 10)) + return errorf(http.StatusTooManyRequests, "request rate limit exceeded") } endpoint, ok := strings.CutPrefix(route.Endpoint, "select/logsql/") if !ok { @@ -173,7 +175,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT } release, ok := acquire(user) if !ok { - return errorf(http.StatusUnprocessableEntity, "too many concurrent requests") + return errorf(http.StatusTooManyRequests, "too many concurrent requests") } defer release() diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index 62d809f28..53c2aaa2d 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -252,8 +252,8 @@ func TestServerLimits(t *testing.T) { }) first := h.do(http.MethodGet, logsRoot+"query", "good", nil, nil) defer func() { _ = first.Body.Close() }() - if got := h.status(http.MethodGet, logsRoot+"query", "good"); got != http.StatusUnprocessableEntity { - t.Fatalf("concurrent request answered %d, want 422", got) + if got := h.status(http.MethodGet, logsRoot+"query", "good"); got != http.StatusTooManyRequests { + t.Fatalf("concurrent request answered %d, want 429", got) } once.Do(func() { close(release) }) @@ -261,8 +261,11 @@ func TestServerLimits(t *testing.T) { if got := h2.status(http.MethodGet, logsRoot+"query", "good"); got != http.StatusOK { t.Fatalf("first request answered %d", got) } - if got := h2.status(http.MethodGet, logsRoot+"query", "good"); got != http.StatusUnprocessableEntity { - t.Fatalf("rate-limited request answered %d, want 422", got) + limited := h2.do(http.MethodGet, logsRoot+"query", "good", nil, nil) + _ = limited.Body.Close() + if limited.StatusCode != http.StatusTooManyRequests || limited.Header.Get("Retry-After") != "1000" { + t.Fatalf("rate-limited request answered %d with Retry-After %q, want 429 with 1000", + limited.StatusCode, limited.Header.Get("Retry-After")) } } @@ -336,8 +339,8 @@ func TestServerTailCapsAndIdle(t *testing.T) { if a.StatusCode != http.StatusOK { t.Fatalf("first tail answered %d", a.StatusCode) } - if got := h.status(http.MethodGet, logsRoot+"tail", "good"); got != http.StatusUnprocessableEntity { - t.Fatalf("second tail of one user answered %d, want 422", got) + if got := h.status(http.MethodGet, logsRoot+"tail", "good"); got != http.StatusTooManyRequests { + t.Fatalf("second tail of one user answered %d, want 429", got) } b := h.do(http.MethodGet, logsRoot+"tail", "bob-token", nil, nil) defer func() { _ = b.Body.Close() }() From 28b66bbc638366994a7654b912b9021e42c7a781 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 9 Nov 2026 10:08:04 +0100 Subject: [PATCH 229/518] test(computeplane): forward an explicit zero and the int32 maximum of maxUnavailable An explicit 0 is what overrides the pool chart's default of 1 for GPU pools, so it must reach the pool unchanged rather than be dropped as an empty value. The int32 maximum is the largest value the schema accepts and must be forwarded as is. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../computeplane/tests/cluster_test.yaml | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/packages/extra/computeplane/tests/cluster_test.yaml b/packages/extra/computeplane/tests/cluster_test.yaml index d56e21d30..6dc4b8835 100644 --- a/packages/extra/computeplane/tests/cluster_test.yaml +++ b/packages/extra/computeplane/tests/cluster_test.yaml @@ -301,6 +301,44 @@ tests: path: spec.values.maxUnavailable value: 1 + # An explicit 0 must reach the pool chart: there it overrides the 1 a GPU + # pool gets by default, so dropping it as empty would change the strategy. + - it: forwards an explicit maxUnavailable of 0 to the pool chart + documentSelector: + path: metadata.name + value: kubernetes-nodes-computeplane-cluster-md0 + set: + nodeGroups: + md0: + minReplicas: 1 + maxReplicas: 3 + instanceType: u1.large + diskSize: 32Gi + roles: ["ingress-nginx"] + maxUnavailable: 0 + asserts: + - equal: + path: spec.values.maxUnavailable + value: 0 + + - it: forwards the int32 maximum to the pool chart + documentSelector: + path: metadata.name + value: kubernetes-nodes-computeplane-cluster-md0 + set: + nodeGroups: + md0: + minReplicas: 1 + maxReplicas: 3 + instanceType: u1.large + diskSize: 32Gi + roles: ["ingress-nginx"] + maxUnavailable: 2147483647 + asserts: + - equal: + path: spec.values.maxUnavailable + value: 2147483647 + - it: refuses a maxUnavailable above the int32 range CAPI decodes set: nodeGroups: From e623530c479ad92afd9c8a707aa12cd645f0ae93 Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Mon, 9 Nov 2026 10:29:20 +0100 Subject: [PATCH 230/518] fix(kubernetes): detach volumes when the infra PVC is missing Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Assisted-by: LLM --- .../images/kubevirt-csi-driver/controller.go | 3 +- .../kubevirt-csi-driver/controller_test.go | 77 +++++++++++++++++-- 2 files changed, 71 insertions(+), 9 deletions(-) diff --git a/packages/apps/kubernetes/images/kubevirt-csi-driver/controller.go b/packages/apps/kubernetes/images/kubevirt-csi-driver/controller.go index 9029c9ff3..b61121758 100644 --- a/packages/apps/kubernetes/images/kubevirt-csi-driver/controller.go +++ b/packages/apps/kubernetes/images/kubevirt-csi-driver/controller.go @@ -424,7 +424,8 @@ func (w *WrappedControllerService) ControllerUnpublishVolume(ctx context.Context pvc, err := w.infraClient.CoreV1().PersistentVolumeClaims(w.infraNamespace).Get(ctx, dvName, metav1.GetOptions{}) if err != nil { if errors.IsNotFound(err) { - return &csi.ControllerUnpublishVolumeResponse{}, nil + // A failed PVC creation can still leave a hotplug entry in the VM. + return w.ControllerService.ControllerUnpublishVolume(ctx, req) } return nil, err } diff --git a/packages/apps/kubernetes/images/kubevirt-csi-driver/controller_test.go b/packages/apps/kubernetes/images/kubevirt-csi-driver/controller_test.go index 46e2e1dce..73b8ffc7d 100644 --- a/packages/apps/kubernetes/images/kubevirt-csi-driver/controller_test.go +++ b/packages/apps/kubernetes/images/kubevirt-csi-driver/controller_test.go @@ -310,13 +310,14 @@ const ( testVolumeName = "pvc-0" ) -// teardownClient models the CAPI/CAPK teardown race: the VM is gone while its -// VMI still reports the hot-plugged volume. The VMI drops the volume once -// RemoveVolumeFromVMI is called, or when the garbage collector deletes a -// VMI whose controlling VM is gone. +// teardownClient keeps hotplug state until removal or garbage collection, +// including when the VMI outlives its VM during CAPI/CAPK teardown. type teardownClient struct { kubevirtclient.Client + vm *kubevirtv1.VirtualMachine vmiOwners []metav1.OwnerReference + detached bool + vmRemovals int vmiRemovals int lookups int } @@ -328,6 +329,9 @@ func (c *teardownClient) vmiGarbageCollected() bool { func (c *teardownClient) GetWorkloadManagingVirtualMachine(context.Context, string, string) (*kubevirtv1.VirtualMachine, error) { c.lookups++ + if c.vm != nil { + return c.vm, nil + } return nil, k8serrors.NewNotFound(schema.GroupResource{Group: "kubevirt.io", Resource: "virtualmachines"}, testVMName) } @@ -336,14 +340,27 @@ func (c *teardownClient) GetVirtualMachine(context.Context, string, string) (*ku vmi := &kubevirtv1.VirtualMachineInstance{ ObjectMeta: metav1.ObjectMeta{Name: testVMName, Namespace: testInfraNamespace, OwnerReferences: c.vmiOwners}, } - if c.vmiRemovals == 0 { - vmi.Status.VolumeStatus = []kubevirtv1.VolumeStatus{{Name: testVolumeName, HotplugVolume: &kubevirtv1.HotplugVolumeStatus{}}} + if !c.detached && c.vmRemovals == 0 && c.vmiRemovals == 0 { + vmi.Status.VolumeStatus = []kubevirtv1.VolumeStatus{{Name: testVolumeName, Phase: kubevirtv1.VolumePending, Reason: "FailedPvcNotFound", HotplugVolume: &kubevirtv1.HotplugVolumeStatus{}}} } return vmi, nil } +func (c *teardownClient) EnsureVolumeRemovedVM(context.Context, string, string, string) (bool, error) { + return c.vm == nil || len(c.vm.Spec.Template.Spec.Volumes) == 0, nil +} + func (c *teardownClient) EnsureVolumeRemovedVMI(context.Context, string, string, string) (bool, error) { - return c.vmiRemovals > 0, nil + return c.detached || c.vmRemovals > 0 || c.vmiRemovals > 0, nil +} + +func (c *teardownClient) RemoveVolumeFromVM(_ context.Context, namespace, name string, opts *kubevirtv1.RemoveVolumeOptions) error { + if namespace != testInfraNamespace || name != testVMName || opts.Name != testVolumeName { + return errors.New("unexpected hotplug removal target") + } + c.vmRemovals++ + c.vm.Spec.Template.Spec.Volumes = nil + return nil } func (c *teardownClient) RemoveVolumeFromVMI(context.Context, string, string, *kubevirtv1.RemoveVolumeOptions) error { @@ -352,7 +369,7 @@ func (c *teardownClient) RemoveVolumeFromVMI(context.Context, string, string, *k } func (c *teardownClient) EnsureVolumeRemoved(context.Context, string, string, string, time.Duration) error { - if c.vmiRemovals > 0 || c.vmiGarbageCollected() { + if c.detached || c.vmRemovals > 0 || c.vmiRemovals > 0 || c.vmiGarbageCollected() { return nil } return errors.New("timed out waiting for the VMI to drop the volume") @@ -410,6 +427,50 @@ func TestControllerUnpublishVolumeHonoursVMIHotplugFallback(t *testing.T) { } } +func TestControllerUnpublishVolumeMissingPVC(t *testing.T) { + tests := []struct { + name string + vmGone bool + detached bool + wantVMCalls int + wantVMICalls int + }{ + {name: "pending hotplug volume", wantVMCalls: 1}, + {name: "already detached", detached: true}, + {name: "VM gone, VMI still has volume", vmGone: true, wantVMICalls: 1}, + {name: "VM gone, no attached volume", vmGone: true, detached: true}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + client := &teardownClient{detached: tt.detached} + if !tt.vmGone { + client.vm = &kubevirtv1.VirtualMachine{Spec: kubevirtv1.VirtualMachineSpec{Template: &kubevirtv1.VirtualMachineInstanceTemplateSpec{}}} + if !tt.detached { + client.vm.Spec.Template.Spec.Volumes = []kubevirtv1.Volume{{ + Name: testVolumeName, + VolumeSource: kubevirtv1.VolumeSource{DataVolume: &kubevirtv1.DataVolumeSource{Name: testVolumeName, Hotpluggable: true}}, + }} + } + } + w := &WrappedControllerService{ + ControllerService: service.NewControllerService(client, testInfraNamespace, nil, util.StorageClassEnforcement{AllowAll: true}, true), + infraClient: fake.NewClientset(), + infraNamespace: testInfraNamespace, + } + req := &csi.ControllerUnpublishVolumeRequest{VolumeId: testVolumeName, NodeId: testInfraNamespace + "/" + testVMName} + if _, err := w.ControllerUnpublishVolume(context.Background(), req); err != nil { + t.Fatalf("ControllerUnpublishVolume() error = %v", err) + } + if client.vmRemovals != tt.wantVMCalls || client.vmiRemovals != tt.wantVMICalls { + t.Errorf("VM/VMI removevolume calls = %d/%d, want %d/%d", client.vmRemovals, client.vmiRemovals, tt.wantVMCalls, tt.wantVMICalls) + } + if client.vm != nil && len(client.vm.Spec.Template.Spec.Volumes) != 0 { + t.Error("hotplug volume remains in VM spec") + } + }) + } +} + func nfsCNP(owners ...string) *unstructured.Unstructured { refs := make([]interface{}, 0, len(owners)) for _, owner := range owners { From 5f6679e4cff70b6e37535920427004259af8adac Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 9 Nov 2026 10:52:28 +0100 Subject: [PATCH 231/518] fix(ingress-nginx): raise client keepalive_time above the 1h default ingress-nginx has no ConfigMap key for the client-side keepalive_time, so nginx applies its 1h default and closes every HTTP/2 connection that reaches that age with a GOAWAY. Clients that multiplex a long operation over a single connection see it fail: a kopia repository maintenance run by Velero against SeaweedFS lasted just over an hour and died while listing blobs, with "http2: server sent GOAWAY and closed the connection; ErrCode=NO_ERROR", because the listing is not retried. Set keepalive_time to 12h through http-snippet. Idle connections are still closed by keep-alive after 75s, so this only lengthens the life of connections that are actually in use. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../tests/client_keepalive_time_test.yaml | 26 +++++++++++++++++++ packages/system/ingress-nginx/values.yaml | 7 +++++ 2 files changed, 33 insertions(+) create mode 100644 packages/system/ingress-nginx/tests/client_keepalive_time_test.yaml diff --git a/packages/system/ingress-nginx/tests/client_keepalive_time_test.yaml b/packages/system/ingress-nginx/tests/client_keepalive_time_test.yaml new file mode 100644 index 000000000..bb492e57c --- /dev/null +++ b/packages/system/ingress-nginx/tests/client_keepalive_time_test.yaml @@ -0,0 +1,26 @@ +suite: ingress-nginx client keepalive_time +release: + name: ingress-nginx + namespace: cozy-ingress-nginx +templates: + - charts/ingress-nginx/templates/controller-configmap.yaml +tests: + - it: raises the client keepalive_time above the nginx 1h default + asserts: + - equal: + path: data.http-snippet + value: keepalive_time 12h; + - it: keeps the snippet when a tenant ingress sets its own config keys + # The tenant ingress HelmRelease passes controller.config for whitelist, + # Cloudflare and PROXY protocol; Helm merges maps, so those keys must not + # drop the default snippet. + set: + ingress-nginx.controller.config.use-proxy-protocol: "true" + ingress-nginx.controller.config.whitelist-source-range: 10.0.0.0/8 + asserts: + - equal: + path: data.http-snippet + value: keepalive_time 12h; + - equal: + path: data.use-proxy-protocol + value: "true" diff --git a/packages/system/ingress-nginx/values.yaml b/packages/system/ingress-nginx/values.yaml index fca145843..1cb7b58da 100644 --- a/packages/system/ingress-nginx/values.yaml +++ b/packages/system/ingress-nginx/values.yaml @@ -47,6 +47,13 @@ ingress-nginx: upstream-keepalive-timeout: "60" upstream-keepalive-connections: "320" ssl-session-tickets: "true" + # Client-side keepalive_time has no ConfigMap key, so nginx keeps its + # 1h default and sends GOAWAY to any HTTP/2 connection older than that. + # A long S3 operation (kopia maintenance behind Velero) multiplexes its + # requests over one connection, and the in-flight object listing fails + # instead of being retried. Idle connections still close after + # keep-alive (75s). + http-snippet: "keepalive_time 12h;" resources: limits: cpu: "1" From f003851338f8e5824425fdc6234539b7f475c698 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 9 Nov 2026 11:18:27 +0100 Subject: [PATCH 232/518] fix(postgres): refuse system databases and never drop them postgres, template0 and template1 were accepted in databases and handled like an application database. The init job made _admin their owner, installed the auto-grant event trigger in them and, once the entry left values, ran DROP DATABASE on them. That DROP can only fail (template databases cannot be dropped, and the job is connected to postgres), so every later upgrade failed in the init job and the release stayed in UpgradeFailed even after the entry was put back. template1 also passed the takeover on to every database created from it afterwards, where the trigger, naming a role about to be dropped, makes CREATE SCHEMA fail. A release a previous version let take one of them over has to be repaired without dropping it, and without touching what this chart did not create: a role or owner that merely carries an expected name may belong to a tenant or an administrator, and a role comment can be set by any CREATEROLE holder, so only the cozystack:managed mark is trusted. What the marked roles own goes to cozystack:orphaned rather than to the superuser, since a tenant's SECURITY DEFINER function would otherwise run with superuser rights. A failed lookup of the databases cloned from template1 has to stop the run before template1 stops being selected, or those clones would never be repaired. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- api/apps/v1alpha1/postgresql/types.go | 2 +- hack/e2e-chainsaw/postgres/chainsaw-test.yaml | 145 ++++++++++ .../e2e-chainsaw/postgres/postgres-sysdb.yaml | 18 ++ hack/postgres-release-system-databases.bats | 261 ++++++++++++++++++ packages/apps/postgres/README.md | 14 +- .../apps/postgres/templates/init-script.yaml | 66 ++++- .../postgres/tests/system_databases_test.yaml | 135 +++++++++ packages/apps/postgres/values.schema.json | 2 +- packages/apps/postgres/values.yaml | 2 +- .../system/postgres-rd/cozyrds/postgres.yaml | 2 +- 10 files changed, 635 insertions(+), 12 deletions(-) create mode 100644 hack/e2e-chainsaw/postgres/postgres-sysdb.yaml create mode 100644 hack/postgres-release-system-databases.bats create mode 100644 packages/apps/postgres/tests/system_databases_test.yaml diff --git a/api/apps/v1alpha1/postgresql/types.go b/api/apps/v1alpha1/postgresql/types.go index f0415f398..4c0d80ab8 100644 --- a/api/apps/v1alpha1/postgresql/types.go +++ b/api/apps/v1alpha1/postgresql/types.go @@ -58,7 +58,7 @@ type ConfigSpec struct { // Users configuration map. Passwords are always auto-generated and stored in the `-credentials` Secret; they cannot be set from values — read a user's password from that Secret. A `password` left over in values from before the field was removed is ignored by the render and draws an admission warning, but it is not inert on an upgraded release: the chart preserves whatever password is already in the Secret, which on the first upgrade is the value that was set before removal, so that value stays the live credential until it is rotated (dedicated rotation is tracked in cozystack/community#72). The live password lives only in the Secret. // +kubebuilder:default:={} Users map[string]User `json:"users,omitempty"` - // Databases configuration map. + // Databases configuration map. The PostgreSQL system databases `postgres`, `template0` and `template1` are refused. // +kubebuilder:default:={} Databases map[string]Database `json:"databases,omitempty"` // Backup configuration. diff --git a/hack/e2e-chainsaw/postgres/chainsaw-test.yaml b/hack/e2e-chainsaw/postgres/chainsaw-test.yaml index 170c0ce36..bc0bc1c98 100644 --- a/hack/e2e-chainsaw/postgres/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/postgres/chainsaw-test.yaml @@ -347,3 +347,148 @@ spec: set -eu cd ../../.. NAMESPACE="$NAMESPACE" examples/backups/postgres/cleanup.sh +--- +# A system database that a version before the refusal took over is handed back +# by the next init-job run, and only what this chart marked changes hands. +# +# The helm-unittest and bats suites check the rendered SQL against a fake psql; +# only a real PostgreSQL can say that the predicates select what they claim. The +# state an older version left is rebuilt by hand as the superuser, since the +# chart now refuses to produce it: +# * postgres is commented as managed and owned by a postgres_admin role that +# carries the cozystack:managed mark, as the chart left it: it must come back +# to the superuser, and the role must be dropped; +# * template1 is commented as managed, but its template1_admin carries only the +# role comment, which a CREATEROLE holder can forge, and not the mark: the +# comment must be cleared and everything that role owns or was granted must +# stay as it is, in template1 and in a database cloned from it. +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: postgres-3-system-databases +spec: + catch: + - events: {} + - describe: + apiVersion: helm.toolkit.fluxcd.io/v2 + kind: HelmRelease + name: postgres-sysdb + - podLogs: + selector: job-name=postgres-sysdb-init-job + tail: 200 + steps: + - name: create-postgres + try: + - apply: + file: postgres-sysdb.yaml + # 11m: above the 600s install timeout cozystack-api gives this release; + # see "Sizing an HR-Ready budget" in docs/agents/e2e-testing.md. + - assert: + timeout: 11m + resource: + apiVersion: helm.toolkit.fluxcd.io/v2 + kind: HelmRelease + metadata: + name: postgres-sysdb + status: + (conditions[?type == 'Ready']): + - status: "True" + - assert: + timeout: 2m + resource: + apiVersion: batch/v1 + kind: Job + metadata: + name: postgres-sysdb-init-job + status: + (conditions[?type == 'Complete']): + - status: "True" + + - name: rebuild-taken-over-system-databases + try: + - script: + timeout: 3m + content: | + set -eu + pod=$(kubectl -n "$NAMESPACE" get pod \ + -l cnpg.io/cluster=postgres-sysdb,cnpg.io/instanceRole=primary -o name | head -n1) + [ -n "$pod" ] || { echo "no primary pod for postgres-sysdb" >&2; exit 1; } + sql() { kubectl -n "$NAMESPACE" exec -i "$pod" -c postgres -- psql -v ON_ERROR_STOP=1 -U postgres -d "$1"; } + sql postgres <<'SQL' + CREATE ROLE postgres_admin NOLOGIN; + COMMENT ON ROLE postgres_admin IS 'role managed by helm'; + GRANT "cozystack:managed" TO postgres_admin WITH INHERIT FALSE, SET FALSE; + COMMENT ON DATABASE postgres IS 'database managed by helm'; + ALTER DATABASE postgres OWNER TO postgres_admin; + CREATE ROLE template1_admin NOLOGIN; + COMMENT ON ROLE template1_admin IS 'role managed by helm'; + CREATE ROLE template1_readonly NOLOGIN; + COMMENT ON DATABASE template1 IS 'database managed by helm'; + ALTER DATABASE template1 OWNER TO template1_admin; + SQL + sql template1 <<'SQL' + ALTER SCHEMA public OWNER TO template1_admin; + CREATE TABLE public.kept (id int); + ALTER TABLE public.kept OWNER TO template1_admin; + GRANT SELECT ON public.kept TO template1_readonly; + SQL + sql postgres <<'SQL' + CREATE DATABASE sysdb_clone TEMPLATE template1; + SQL + # The next run must be a new Job: remove the completed one so the + # assert below cannot match it. + kubectl -n "$NAMESPACE" delete job postgres-sysdb-init-job --wait=true + # Declaring a user is enough for an upgrade, and runs the init-job. + kubectl -n "$NAMESPACE" patch postgres.apps.cozystack.io sysdb --type merge \ + -p '{"spec":{"users":{"sysdbuser":{},"sysdbprobe":{}}}}' + check: + ($error == null): true + - assert: + timeout: 11m + resource: + apiVersion: batch/v1 + kind: Job + metadata: + name: postgres-sysdb-init-job + status: + (conditions[?type == 'Complete']): + - status: "True" + + - name: verify-release + try: + - script: + timeout: 2m + content: | + set -eu + pod=$(kubectl -n "$NAMESPACE" get pod \ + -l cnpg.io/cluster=postgres-sysdb,cnpg.io/instanceRole=primary -o name | head -n1) + q() { kubectl -n "$NAMESPACE" exec "$pod" -c postgres -- psql -v ON_ERROR_STOP=1 -U postgres -tA -d "$1" -c "$2"; } + fail=0 + expect() { + got=$(q "$1" "$3") + if [ "$got" != "$4" ]; then + echo "FAIL: $2: got '$got', want '$4'" >&2 + fail=1 + fi + } + comment="SELECT coalesce(shobj_description(oid, 'pg_database'), '') FROM pg_database WHERE datname = " + owner="SELECT pg_get_userbyid(datdba) FROM pg_database WHERE datname = " + expect postgres "postgres comment cleared" "$comment'postgres'" '' + expect postgres "postgres back to the superuser" "$owner'postgres'" postgres + expect postgres "marked postgres_admin dropped" "SELECT count(*) FROM pg_roles WHERE rolname = 'postgres_admin'" 0 + expect postgres "template1 comment cleared" "$comment'template1'" '' + expect postgres "unmarked template1_admin still exists" "SELECT count(*) FROM pg_roles WHERE rolname = 'template1_admin'" 1 + expect postgres "template1 still owned by the unmarked role" "$owner'template1'" template1_admin + for db in template1 sysdb_clone; do + expect "$db" "$db public still owned by the unmarked role" \ + "SELECT pg_get_userbyid(nspowner) FROM pg_namespace WHERE nspname = 'public'" template1_admin + expect "$db" "$db kept table still owned by the unmarked role" \ + "SELECT tableowner FROM pg_tables WHERE schemaname = 'public' AND tablename = 'kept'" template1_admin + expect "$db" "$db grant to the unmarked readonly role kept" \ + "SELECT has_table_privilege('template1_readonly', 'public.kept', 'SELECT')" t + done + exit "$fail" + check: + ($error == null): true +# The Postgres CR was applied by chainsaw; deleting it removes the whole +# cluster, the databases and roles built above included. diff --git a/hack/e2e-chainsaw/postgres/postgres-sysdb.yaml b/hack/e2e-chainsaw/postgres/postgres-sysdb.yaml new file mode 100644 index 000000000..975dc6a51 --- /dev/null +++ b/hack/e2e-chainsaw/postgres/postgres-sysdb.yaml @@ -0,0 +1,18 @@ +apiVersion: apps.cozystack.io/v1alpha1 +kind: Postgres +metadata: + name: sysdb +spec: + external: false + size: 1Gi + replicas: 1 + storageClass: "" + quorum: + minSyncReplicas: 0 + maxSyncReplicas: 0 + users: + sysdbuser: {} + databases: + appdb: {} + resources: {} + resourcesPreset: "nano" diff --git a/hack/postgres-release-system-databases.bats b/hack/postgres-release-system-databases.bats new file mode 100644 index 000000000..688f20df5 --- /dev/null +++ b/hack/postgres-release-system-databases.bats @@ -0,0 +1,261 @@ +#!/usr/bin/env bats +# Behavioural tests for the "release system databases" pass of the Postgres +# init script: +# +# packages/apps/postgres/templates/init-script.yaml +# +# The pass is cut out of the rendered script and RUN against a fake psql that +# records which database each session connects to. The helm-unittest suite in +# packages/apps/postgres/tests/system_databases_test.yaml matches the SQL text, +# and stayed green when the template0 guard was inverted: the rendered pass then +# connected to template0, which accepts no connection, and skipped postgres and +# template1, which are the two it exists to hand back. +# +# What is pinned here: +# * no session ever connects to template0; +# * postgres and template1 are each connected to and released; +# * every taken-over system database, template0 included, has its comment +# cleared; +# * releasing template1 visits every other connectable database, for the +# trigger it passed on to them, and a failed lookup of those databases +# stops the pass before the comment that selects template1 is cleared; +# * the session that releases a system database hands its public schema +# and the database itself back only from a marked _admin or +# _readonly owner (or cozystack:orphaned), before the reassignment, +# an inherited public only from a marked template1_admin, and touches +# only the roles that carry the +# "cozystack:managed" mark, not a role that merely has one of the names. +# +# Run from the repo root with make bats-unit-tests. Direct status checks +# retain compatibility with cozytest, which has no Bats run/setup helpers. +# Inline cleanup preserves Bats' TAP handler on failed assertions. + +load test_helper + +PG_CHART=packages/apps/postgres + +# render_pass +# Extract the release pass, from its banner to the next one, out of the rendered +# init script. The non-empty check must be an early `return 1`: hack/cozytest.sh +# rewrites every line matching ^}$ into `return 0` plus `}`, helpers included, +# so a check left in last-command position has its status discarded. +render_pass() { + helm template pg-test "$PG_CHART" --namespace tenant-test --set-json '_cluster={"scheduling":{}}' 2>/dev/null \ + | yq eval 'select(.metadata.name == "pg-test-init-script") | .stringData["init.sh"]' - \ + | awk '/^echo "== release system databases"/ { f = 1 } /^echo "== delete databases"/ { f = 0 } f' > "$1" + grep -q 'SYSTEM_DBS=' "$1" || return 1 +} + +# write_fake_psql +# Emit a fake psql. A `-c` query answers from $SYSTEM_DBS for the system +# database lookup and from $OTHER_DBS for the connectable-database lookup, +# which exits with $CLONE_LOOKUP_FAILS instead when that is set. A +# session fed on stdin logs "connect " ("connect -" without --dbname) +# followed by its first line and every -v db= value, to $PLOG, and its whole +# input under a "-- session db=" header to $PLOG.sql. +# No line below is a bare column-0 `}`, so cozytest.sh's parser leaves it alone. +write_fake_psql() { + cat > "$1/psql" <<'PEOF' +#!/bin/sh +dbname=- query= vdb= +while [ $# -gt 0 ]; do + case "$1" in + --dbname) dbname=$2; shift ;; + -c) query=$2; shift ;; + -v) case "$2" in db=*) vdb=${2#db=} ;; esac; shift ;; + esac + shift +done +if [ -n "$query" ]; then + case "$query" in + *"d.datname IN ('postgres', 'template0', 'template1')"*) printf '%s\n' $SYSTEM_DBS ;; + *datallowconn*) + if [ -n "${CLONE_LOOKUP_FAILS:-}" ]; then + echo "fake psql: connectable-database lookup failed" >&2 + exit "$CLONE_LOOKUP_FAILS" + fi + printf '%s\n' $OTHER_DBS ;; + *) echo "fake psql: unexpected query: $query" >&2; exit 64 ;; + esac + exit 0 +fi +sql=$(cat) +echo "connect $dbname db=$vdb $(printf '%s\n' "$sql" | head -n 1)" >> "$PLOG" +printf -- '-- session %s db=%s\n%s\n' "$dbname" "$vdb" "$sql" >> "$PLOG.sql" +PEOF + chmod +x "$1/psql" +} + +# run_pass +# Run the extracted pass with the fake psql first on PATH; status in $rc. +run_pass() { + export PLOG="$1/calls" + : > "$PLOG" + rc=0 + PATH="$1:$PATH" sh -e "$1/pass.sh" > "$1/out" 2> "$1/err" || rc=$? +} + +@test "a render that produces no release pass fails instead of passing as a no-op" { + tmp=$(mktemp -d) + if PG_CHART=packages/apps/no-such-chart render_pass "$tmp/pass.sh"; then + echo "FAIL: render_pass reported success for a chart that does not exist" + false + fi + rm -rf "$tmp" +} + +@test "release pass connects to postgres and template1, never to template0" { + tmp=$(mktemp -d) + write_fake_psql "$tmp" + render_pass "$tmp/pass.sh" + export SYSTEM_DBS="postgres template0 template1" OTHER_DBS="postgres app" + run_pass "$tmp" + + if [ "$rc" -ne 0 ]; then + echo "FAIL: the pass exited $rc" + cat "$tmp/out" "$tmp/err" + false + fi + if grep -q '^connect template0 ' "$tmp/calls"; then + echo "FAIL: the pass connected to template0, which accepts no connection" + cat "$tmp/calls" + false + fi + for db in postgres template1; do + if ! grep -q "^connect $db db=$db DROP EVENT TRIGGER IF EXISTS trigger_auto_grant;" "$tmp/calls"; then + echo "FAIL: $db was not connected to and released" + cat "$tmp/calls" + false + fi + done + for db in postgres template0 template1; do + if ! grep -q "^connect - db=$db COMMENT ON DATABASE :\"db\" IS NULL;" "$tmp/calls"; then + echo "FAIL: the comment on $db was not cleared" + cat "$tmp/calls" + false + fi + done + rm -rf "$tmp" +} + +@test "release session hands public back and touches only marked roles" { + tmp=$(mktemp -d) + write_fake_psql "$tmp" + render_pass "$tmp/pass.sh" + export SYSTEM_DBS="postgres" OTHER_DBS="postgres" + run_pass "$tmp" + + if [ "$rc" -ne 0 ]; then + echo "FAIL: the pass exited $rc" + cat "$tmp/out" "$tmp/err" + false + fi + awk '/^-- session / { f = ($0 == "-- session postgres db=postgres") } f' "$tmp/calls.sql" > "$tmp/release.sql" + # Each statement must end on its whole predicate: a clause appended after the + # mark (an OR, say) would select an unmarked role again. + reassign_where="WHERE EXISTS (SELECT FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid JOIN pg_roles u ON u.oid = m.member WHERE g.rolname = 'cozystack:managed' AND u.rolname = r)\\gexec" + owner_marked="(o.rolname = 'cozystack:orphaned' OR o.rolname IN (:'db' || '_admin', :'db' || '_readonly') AND EXISTS (SELECT FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid WHERE g.rolname = 'cozystack:managed' AND m.member = o.oid))\\gexec" + public_where="WHERE n.nspname = 'public' AND $owner_marked" + database_where="WHERE d.datname = :'db' AND $owner_marked" + public_line=$(grep -n 'ALTER SCHEMA public OWNER TO' "$tmp/release.sql" | head -1) + database_line=$(grep -n 'ALTER DATABASE' "$tmp/release.sql" | head -1) + reassign_line=$(grep -n 'REASSIGN OWNED BY' "$tmp/release.sql" | head -1) + if [ "${public_line##*"$public_where"}" != "" ] || [ -z "$public_line" ]; then + echo "FAIL: the release session does not hand public back only from a marked _admin or _readonly owner" + cat "$tmp/release.sql" + false + fi + if [ "${database_line##*"$database_where"}" != "" ] || [ -z "$database_line" ]; then + echo "FAIL: the release session does not take the database back only from a marked owner" + cat "$tmp/release.sql" + false + fi + if [ "${reassign_line##*"$reassign_where"}" != "" ] || [ -z "$reassign_line" ]; then + echo "FAIL: the release session does not reassign only roles carrying the cozystack:managed mark" + cat "$tmp/release.sql" + false + fi + # public and the database must be handed back before REASSIGN OWNED would + # move them to cozystack:orphaned. + if [ "${public_line%%:*}" -ge "${reassign_line%%:*}" ] || [ "${database_line%%:*}" -ge "${reassign_line%%:*}" ]; then + echo "FAIL: public or the database is handed back after the reassignment has already moved it" + cat "$tmp/release.sql" + false + fi + rm -rf "$tmp" +} + +@test "release pass of template1 visits every other connectable database" { + tmp=$(mktemp -d) + write_fake_psql "$tmp" + render_pass "$tmp/pass.sh" + export SYSTEM_DBS="template1" OTHER_DBS="postgres app" + run_pass "$tmp" + + if [ "$rc" -ne 0 ]; then + echo "FAIL: the pass exited $rc" + cat "$tmp/out" "$tmp/err" + false + fi + for db in postgres app; do + if ! grep -q "^connect $db db= SELECT 'DROP EVENT TRIGGER IF EXISTS trigger_auto_grant'" "$tmp/calls"; then + echo "FAIL: $db was not checked for the trigger inherited from template1" + cat "$tmp/calls" + false + fi + done + # An inherited public changes hands only from a template1_admin the chart + # marked; an owner that merely has the name keeps it. + inherited_where="WHERE n.nspname = 'public' AND o.rolname = 'template1_admin' AND EXISTS (SELECT FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid WHERE g.rolname = 'cozystack:managed' AND m.member = o.oid)\\gexec" + inherited_line=$(awk '/^-- session / { f = ($0 == "-- session app db=") } f' "$tmp/calls.sql" | grep 'ALTER SCHEMA public OWNER TO' | head -1) + if [ -z "$inherited_line" ] || [ "${inherited_line##*"$inherited_where"}" != "" ]; then + echo "FAIL: an inherited public is handed back without checking that a marked template1_admin owns it" + cat "$tmp/calls.sql" + false + fi + rm -rf "$tmp" +} + +@test "release pass of template1 stops before clearing its comment when the clone lookup fails" { + # Clearing the comment is what stops later runs from selecting template1, so + # a lookup failure that let the pass go on would skip the clone repair for + # good. + tmp=$(mktemp -d) + write_fake_psql "$tmp" + render_pass "$tmp/pass.sh" + export SYSTEM_DBS="template1" OTHER_DBS="postgres app" CLONE_LOOKUP_FAILS=73 + run_pass "$tmp" + unset CLONE_LOOKUP_FAILS + + if [ "$rc" -ne 73 ]; then + echo "FAIL: the pass exited $rc instead of stopping on the failed lookup (73)" + cat "$tmp/out" "$tmp/err" "$tmp/calls" + false + fi + if grep -q 'COMMENT ON DATABASE' "$tmp/calls"; then + echo "FAIL: the comment on template1 was cleared although its clones were never visited" + cat "$tmp/calls" + false + fi + rm -rf "$tmp" +} + +@test "release pass of postgres alone leaves the other databases alone" { + tmp=$(mktemp -d) + write_fake_psql "$tmp" + render_pass "$tmp/pass.sh" + export SYSTEM_DBS="postgres" OTHER_DBS="postgres app" + run_pass "$tmp" + + if [ "$rc" -ne 0 ]; then + echo "FAIL: the pass exited $rc" + cat "$tmp/out" "$tmp/err" + false + fi + if grep -q '^connect app ' "$tmp/calls"; then + echo "FAIL: the pass visited app although template1 was not taken over" + cat "$tmp/calls" + false + fi + rm -rf "$tmp" +} diff --git a/packages/apps/postgres/README.md b/packages/apps/postgres/README.md index 50c300aff..c508476f4 100644 --- a/packages/apps/postgres/README.md +++ b/packages/apps/postgres/README.md @@ -279,13 +279,13 @@ For `sslmode=verify-full` to work, the CA bundle retrieved above must be saved t ### Databases configuration -| Name | Description | Type | Value | -| -------------------------------- | ---------------------------------------- | ------------------- | ----- | -| `databases` | Databases configuration map. | `map[string]object` | `{}` | -| `databases[name].roles` | Roles assigned to users. | `object` | `{}` | -| `databases[name].roles.admin` | List of users with admin privileges. | `[]string` | `[]` | -| `databases[name].roles.readonly` | List of users with read-only privileges. | `[]string` | `[]` | -| `databases[name].extensions` | List of enabled PostgreSQL extensions. | `[]string` | `[]` | +| Name | Description | Type | Value | +| -------------------------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------- | ----- | +| `databases` | Databases configuration map. The PostgreSQL system databases `postgres`, `template0` and `template1` are refused. | `map[string]object` | `{}` | +| `databases[name].roles` | Roles assigned to users. | `object` | `{}` | +| `databases[name].roles.admin` | List of users with admin privileges. | `[]string` | `[]` | +| `databases[name].roles.readonly` | List of users with read-only privileges. | `[]string` | `[]` | +| `databases[name].extensions` | List of enabled PostgreSQL extensions. | `[]string` | `[]` | ### Backup parameters diff --git a/packages/apps/postgres/templates/init-script.yaml b/packages/apps/postgres/templates/init-script.yaml index 395267ff0..6306a5fae 100644 --- a/packages/apps/postgres/templates/init-script.yaml +++ b/packages/apps/postgres/templates/init-script.yaml @@ -53,6 +53,9 @@ PostgreSQL identifiers too, capped at 63. */}} {{- range $database, $d := .Values.databases }} + {{- if has $database (list "postgres" "template0" "template1") }} + {{- fail (printf "databases.%s is refused: %q is a PostgreSQL system database. Managing it would hand it, its public schema and the auto-grant event trigger to the \"%s_admin\" role (template1 then passes all of that on to every database created afterwards), and removing the entry later could not drop it. Remove the entry and declare an application database instead; the next upgrade releases a system database a previous version took over, without dropping it" $database $database $database) }} + {{- end }} {{- if or (not (regexMatch "^[-._a-zA-Z0-9]+$" $database)) (gt (len $database) 54) }} {{- fail (printf "invalid database name %q: a database name may contain only letters, digits and the characters -._ and must be at most 54 characters (it is interpolated into superuser SQL and the init-script shell, and it seeds the \"_admin\" and \"_readonly\" PostgreSQL roles, which CREATE ROLE silently truncates past 63 bytes; the next init-job run then cannot drop the truncated role and fails). Changing only its key in values makes the init-job drop the database; to keep the data, a platform administrator first renames it as the CNPG superuser (ALTER DATABASE, and ALTER ROLE for its _admin and _readonly roles), then the key is changed" $database) }} {{- end }} @@ -320,6 +323,67 @@ stringData: EOT done + echo "== release system databases" + {{- /* + Versions that did not refuse postgres, template0 and template1 in + databases took them over like any other entry. DROP DATABASE on them can + only fail (template databases cannot be dropped, and this session is + connected to postgres), which wedged every later upgrade, so they are + handed back instead: the auto-grant trigger goes, since it names a role + about to be dropped and would make every CREATE SCHEMA fail; what the + _admin and _readonly roles own there goes to cozystack:orphaned, + as in the orphaned-roles pass and for the same reason (a tenant's + SECURITY DEFINER function must not end up owned by the superuser), and is + made SECURITY INVOKER (only a role marked "cozystack:managed" is touched: + a role that merely carries one of those names was not created by this + chart and keeps what it owns); the database itself returns to the superuser, + public to pg_database_owner (to the superuser before PostgreSQL 15), each + only when one of those marked roles owns it and before the reassignment + would move it (or when cozystack:orphaned does, after a run that stopped + in between); an owner the chart did not mark keeps what it owns, and + the comment is cleared. The orphaned-roles pass below then drops the + roles; it does not visit template1, hence the per-database pass here. + template0 accepts no connection, and the Job that commented it never got + further, so only the comment is cleared. A database created from + template1 while it was taken over inherited its trigger, naming + template1_admin, and its public schema owned by that role; the trigger + goes, public returns to its default owner if a marked template1_admin + still owns it, and a declared database gets + its trigger back from the grant pass further down. + */}} + SYSTEM_DBS=$(psql -v ON_ERROR_STOP=1 -t -A -c "SELECT datname FROM pg_database d JOIN pg_shdescription s ON d.oid = s.objoid WHERE s.description = 'database managed by helm' AND d.datname IN ('postgres', 'template0', 'template1')") + echo "system databases to release: $SYSTEM_DBS" + for db in $SYSTEM_DBS; do + if [ "$db" != template0 ]; then + psql -v ON_ERROR_STOP=1 --echo-all --dbname "$db" -v db="$db" <<\EOT + DROP EVENT TRIGGER IF EXISTS trigger_auto_grant; + DROP FUNCTION IF EXISTS auto_grant_schema_privileges(); + SELECT format('ALTER SCHEMA public OWNER TO %s', CASE WHEN current_setting('server_version_num')::int >= 150000 THEN 'pg_database_owner' ELSE 'CURRENT_USER' END) FROM pg_namespace n JOIN pg_roles o ON o.oid = n.nspowner WHERE n.nspname = 'public' AND (o.rolname = 'cozystack:orphaned' OR o.rolname IN (:'db' || '_admin', :'db' || '_readonly') AND EXISTS (SELECT FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid WHERE g.rolname = 'cozystack:managed' AND m.member = o.oid))\gexec + SELECT format('ALTER DATABASE %I OWNER TO CURRENT_USER', d.datname) FROM pg_database d JOIN pg_roles o ON o.oid = d.datdba WHERE d.datname = :'db' AND (o.rolname = 'cozystack:orphaned' OR o.rolname IN (:'db' || '_admin', :'db' || '_readonly') AND EXISTS (SELECT FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid WHERE g.rolname = 'cozystack:managed' AND m.member = o.oid))\gexec + SELECT format('REASSIGN OWNED BY %I TO "cozystack:orphaned"', r), format('DROP OWNED BY %I', r) FROM unnest(ARRAY[:'db' || '_admin', :'db' || '_readonly']) AS r WHERE EXISTS (SELECT FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid JOIN pg_roles u ON u.oid = m.member WHERE g.rolname = 'cozystack:managed' AND u.rolname = r)\gexec + SELECT format('ALTER %s %s SECURITY INVOKER', CASE p.prokind WHEN 'p' THEN 'PROCEDURE' ELSE 'FUNCTION' END, p.oid::regprocedure) FROM pg_proc p WHERE p.proowner = '"cozystack:orphaned"'::regrole AND p.prosecdef\gexec + SELECT format('ALTER VIEW %s SET (security_invoker = true)', c.oid::regclass) FROM pg_class c WHERE c.relowner = '"cozystack:orphaned"'::regrole AND c.relkind = 'v' AND current_setting('server_version_num')::int >= 150000 AND NOT coalesce('security_invoker=true' = ANY (c.reloptions), false)\gexec + EOT + fi + if [ "$db" = template1 ]; then + {{- /* + An assignment of its own, so that set -e stops the pass when the lookup + fails: inside the for list its status is discarded, the comment below is + cleared without a clone visited, and no later run selects template1 again. + */}} + CLONE_DBS=$(psql -v ON_ERROR_STOP=1 -t -A -c "SELECT datname FROM pg_database WHERE datallowconn AND NOT datistemplate") + for other in $CLONE_DBS; do + psql -v ON_ERROR_STOP=1 --echo-queries --dbname "$other" <<\EOT + SELECT 'DROP EVENT TRIGGER IF EXISTS trigger_auto_grant', 'DROP FUNCTION auto_grant_schema_privileges()' WHERE position('''template1_admin''' IN coalesce((SELECT prosrc FROM pg_proc WHERE oid = to_regprocedure('auto_grant_schema_privileges()')), '')) > 0\gexec + SELECT format('ALTER SCHEMA public OWNER TO %s', CASE WHEN current_setting('server_version_num')::int >= 150000 THEN 'pg_database_owner' ELSE 'CURRENT_USER' END) FROM pg_namespace n JOIN pg_roles o ON o.oid = n.nspowner WHERE n.nspname = 'public' AND o.rolname = 'template1_admin' AND EXISTS (SELECT FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid WHERE g.rolname = 'cozystack:managed' AND m.member = o.oid)\gexec + EOT + done + fi + psql -v ON_ERROR_STOP=1 --echo-all -v db="$db" <<\EOT + COMMENT ON DATABASE :"db" IS NULL; + EOT + done + echo "== delete databases" {{- /* A restore replaces spec.databases with the Backup's snapshot, but recovery @@ -338,7 +402,7 @@ stringData: ADOPTED_MARK=$(psql -v ON_ERROR_STOP=1 -t -A --dbname postgres -c "SELECT current_setting('cozystack.adopted_restore', true)") MANAGED_DBS=$(psql -v ON_ERROR_STOP=1 -t -A -c "SELECT datname FROM pg_database d JOIN pg_shdescription s ON d.oid = s.objoid WHERE s.description = 'database managed by helm'") DEFINED_DBS="{{ join " " (keys .Values.databases | sortAlpha) }}" - DELETE_DBS=$(for db in $MANAGED_DBS; do case " $DEFINED_DBS " in *" $db "*) :;; *) echo $db;; esac; done) + DELETE_DBS=$(for db in $MANAGED_DBS; do case " postgres template0 template1 $DEFINED_DBS " in *" $db "*) :;; *) echo $db;; esac; done) if [ -n "$RESTORE_MARK" ] && [ "$ADOPTED_MARK" != "$RESTORE_MARK" ]; then echo "databases the restore brought back that values do not declare, released from management: $DELETE_DBS" diff --git a/packages/apps/postgres/tests/system_databases_test.yaml b/packages/apps/postgres/tests/system_databases_test.yaml new file mode 100644 index 000000000..e4ff8400f --- /dev/null +++ b/packages/apps/postgres/tests/system_databases_test.yaml @@ -0,0 +1,135 @@ +suite: init-script never takes over or drops a system database + +# Each of postgres, template0 and template1 used to be accepted in databases and +# handled like an application database: owned by _admin, given the +# auto-grant event trigger, and on removal sent to DROP DATABASE, which fails on +# all three (template databases cannot be dropped, and the Job is connected to +# postgres) and wedged every later upgrade. template1 also passed the takeover +# on to every database created afterwards. + +templates: + - templates/init-script.yaml + +release: + name: pg-test + namespace: tenant-test + +set: + _cluster: + scheduling: {} + +tests: + - it: refuses postgres in databases + set: + databases: + postgres: {} + asserts: + - failedTemplate: + errorPattern: "databases.postgres is refused: \"postgres\" is a PostgreSQL system database" + + - it: refuses template0 in databases + set: + databases: + template0: {} + asserts: + - failedTemplate: + errorPattern: "databases.template0 is refused: \"template0\" is a PostgreSQL system database" + + - it: refuses template1 in databases + set: + databases: + template1: {} + asserts: + - failedTemplate: + errorPattern: "databases.template1 is refused: \"template1\" is a PostgreSQL system database" + + - it: still accepts an application database whose name contains a system name + set: + databases: + postgres_app: {} + mytemplate1: {} + documentSelector: + path: metadata.name + value: pg-test-init-script + asserts: + - matchRegex: + path: stringData["init.sh"] + pattern: 'CREATE DATABASE "postgres_app"' + + # A release that a previous version let manage a system database still carries + # its comment. The delete pass must never select it, whatever values say. + - it: never lists a system database for DROP DATABASE + set: + databases: + appdb: {} + documentSelector: + path: metadata.name + value: pg-test-init-script + asserts: + - matchRegex: + path: stringData["init.sh"] + pattern: 'DELETE_DBS=\$\(for db in \$MANAGED_DBS; do case " postgres template0 template1 \$DEFINED_DBS " in \*" \$db "\*\) :;; \*\) echo \$db;; esac; done\)' + + - it: hands a taken-over system database back before the delete pass + documentSelector: + path: metadata.name + value: pg-test-init-script + asserts: + - matchRegex: + path: stringData["init.sh"] + pattern: "SYSTEM_DBS=\\$\\(psql -v ON_ERROR_STOP=1 -t -A -c \"SELECT datname FROM pg_database d JOIN pg_shdescription s ON d.oid = s.objoid WHERE s.description = 'database managed by helm' AND d.datname IN \\('postgres', 'template0', 'template1'\\)\"\\)" + - matchRegex: + path: stringData["init.sh"] + pattern: '(?s)== release system databases.*DROP EVENT TRIGGER IF EXISTS trigger_auto_grant;\s+DROP FUNCTION IF EXISTS auto_grant_schema_privileges\(\);.*ALTER SCHEMA public OWNER TO %s.*ALTER DATABASE %I OWNER TO CURRENT_USER.*REASSIGN OWNED BY %I TO "cozystack:orphaned"''.*DROP OWNED BY %I.*COMMENT ON DATABASE :"db" IS NULL;.*== delete databases' + + # A tenant holding _admin may have created a SECURITY DEFINER function + # there; handed to the superuser it would run with superuser rights for + # anyone PUBLIC can execute it. It goes to cozystack:orphaned and is made + # SECURITY INVOKER, as in the orphaned-roles pass. + - it: never hands a released system database's objects to the superuser + documentSelector: + path: metadata.name + value: pg-test-init-script + asserts: + - notMatchRegex: + path: stringData["init.sh"] + pattern: 'REASSIGN OWNED BY %I TO CURRENT_USER' + - matchRegex: + path: stringData["init.sh"] + pattern: '(?s)== release system databases.*ALTER %s %s SECURITY INVOKER.*ALTER VIEW %s SET \(security_invoker = true\).*== delete databases' + + # A database created from template1 while it was taken over inherited the + # trigger naming template1_admin, which the orphaned-roles pass then drops; + # CREATE SCHEMA would fail there from that point on. + - it: drops the template1 trigger inherited by other databases + documentSelector: + path: metadata.name + value: pg-test-init-script + asserts: + - matchRegex: + path: stringData["init.sh"] + pattern: "(?s)== release system databases.*if \\[ \"\\$db\" = template1 \\]; then.*SELECT 'DROP EVENT TRIGGER IF EXISTS trigger_auto_grant', 'DROP FUNCTION auto_grant_schema_privileges\\(\\)' WHERE position\\('''template1_admin''' IN .*== delete databases" + + # That clone's public schema also came from template1, owned by + # template1_admin; left to the orphaned-roles pass it would end up owned by + # cozystack:orphaned and the database owner would lose CREATE on it. + - it: gives an inherited public schema back to its default owner + documentSelector: + path: metadata.name + value: pg-test-init-script + asserts: + - matchRegex: + path: stringData["init.sh"] + pattern: "(?s)if \\[ \"\\$db\" = template1 \\]; then.*SELECT format\\('ALTER SCHEMA public OWNER TO %s', .*\\) FROM pg_namespace n JOIN pg_roles o ON o.oid = n.nspowner WHERE n.nspname = 'public' AND o.rolname = 'template1_admin' AND EXISTS \\(SELECT FROM pg_auth_members m JOIN pg_roles g ON g.oid = m.roleid WHERE g.rolname = 'cozystack:managed' AND m.member = o.oid\\).*== delete databases" + + # template0 accepts no connection: only postgres and template1 are connected + # to and released, template0 just has its comment cleared. The behaviour is + # run against a fake psql in hack/postgres-release-system-databases.bats. + - it: connects to every released system database but template0 + documentSelector: + path: metadata.name + value: pg-test-init-script + asserts: + - matchRegex: + path: stringData["init.sh"] + pattern: "(?s)for db in \\$SYSTEM_DBS; do\\s+if \\[ \"\\$db\" != template0 \\]; then\\s+psql -v ON_ERROR_STOP=1 --echo-all --dbname \"\\$db\" -v db=\"\\$db\"" diff --git a/packages/apps/postgres/values.schema.json b/packages/apps/postgres/values.schema.json index 91747ba24..4d544ff6b 100644 --- a/packages/apps/postgres/values.schema.json +++ b/packages/apps/postgres/values.schema.json @@ -313,7 +313,7 @@ } }, "databases": { - "description": "Databases configuration map.", + "description": "Databases configuration map. The PostgreSQL system databases `postgres`, `template0` and `template1` are refused.", "type": "object", "default": {}, "additionalProperties": { diff --git a/packages/apps/postgres/values.yaml b/packages/apps/postgres/values.yaml index 83b658a64..fb4a67b35 100644 --- a/packages/apps/postgres/values.yaml +++ b/packages/apps/postgres/values.yaml @@ -287,7 +287,7 @@ users: {} ## @field {DatabaseRoles} [roles] - Roles assigned to users. ## @field {[]string} [extensions] - List of enabled PostgreSQL extensions. -## @param {map[string]Database} databases - Databases configuration map. +## @param {map[string]Database} databases - Databases configuration map. The PostgreSQL system databases `postgres`, `template0` and `template1` are refused. databases: {} ## Example: ## databases: diff --git a/packages/system/postgres-rd/cozyrds/postgres.yaml b/packages/system/postgres-rd/cozyrds/postgres.yaml index 39adc0f58..0cd63edd2 100644 --- a/packages/system/postgres-rd/cozyrds/postgres.yaml +++ b/packages/system/postgres-rd/cozyrds/postgres.yaml @@ -30,7 +30,7 @@ spec: singular: postgres plural: postgreses openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Postgres replicas. Under active autoscaling this is the resting/seed count, not the live count — KEDA drives the live count via `/scale`. Disabling autoscaling does NOT automatically bring the live count back to `replicas`: the client-side merge writes `instances` only when the rendered value changes, so a cluster KEDA grew past `replicas` stays there with no autoscaler left to reduce it — lower `replicas` to the count you want and it rebases down (shedding the surplus replicas + PVCs). Lowering it below the live count, or disabling/dry-running while it sits below the live count, sheds in one step past the scaleDown pacing; stage it to the live count first (suspend the HelmRelease and `kubectl scale` if needed) to avoid a transient. Must stay greater than `quorum.maxSyncReplicas`.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each PostgreSQL replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster.","type":"boolean","default":false},"version":{"description":"PostgreSQL major version to deploy","type":"string","default":"v18","enum":["v18","v17","v16","v15","v14","v13"]},"flavor":{"description":"PostgreSQL image flavor, fixed when the cluster is created. `postgresql` uses the CloudNativePG PostgreSQL image. `postgis` uses the CloudNativePG PostGIS image, which adds postgis, postgis_raster, postgis_topology, postgis_sfcgal, postgis_tiger_geocoder and address_standardizer; enable them per database in `databases..extensions`. Changing the flavor of an existing cluster is refused, and a backup restores only into a cluster of the flavor it was taken from: to change flavor, create a new cluster and move the data with a logical dump (pg_dump / pg_restore).","type":"string","default":"postgresql","enum":["postgresql","postgis"]},"tls":{"description":"TLS configuration for server connections.","type":"object","default":{},"properties":{"enabled":{"description":"Tri-state switch controlling whether the chart injects the external hostnames into the operator-managed CNPG cert via spec.certificates.serverAltDNSNames. When omitted, the chart injects them if `external: true` and skips them otherwise. Set explicitly to `true` to inject regardless of `external` (no-op when `external: false` since there is no external hostname to add). Set to `false` to skip injection. Note that CNPG keeps its built-in TLS on the wire regardless of this flag — this toggle only controls the chart-side SAN injection; to disable PostgreSQL TLS entirely set `postgresql.parameters.ssl = \"off\"` at the CNPG layer.","type":"boolean"},"required":{"description":"Tri-state switch controlling whether the server refuses sessions that are not encrypted. When omitted, it is on if `external: true` and off otherwise. When on, the chart renders `spec.postgresql.pg_hba` with `hostssl all all all scram-sha-256` (`md5` on PostgreSQL 13, matching CloudNativePG's default rule) followed by `host all all all reject`, so a client that connects with `sslmode=disable` is refused instead of getting a plaintext session; in-cluster clients are held to the same rule, since the pod network cannot be told apart from an external client. Set to `false` to keep accepting plaintext sessions on an external instance, or to `true` to require TLS on an in-cluster one. Rendering fails if `postgresql.parameters.ssl` is off while this is in effect.","type":"boolean"}}},"postgresql":{"description":"PostgreSQL server configuration.","type":"object","default":{},"properties":{"parameters":{"description":"PostgreSQL server parameters. Values may be strings or integers; integers are coerced to strings by the template (e.g. both `max_connections: 100` and `max_connections: \"100\"` are accepted). BLOCKED (enable arbitrary code execution): archive_command, restore_command, ssl_passphrase_command, archive_cleanup_command, recovery_end_command, dynamic_library_path, local_preload_libraries, session_preload_libraries, shared_preload_libraries. Do NOT override CloudNativePG-managed parameters: archive_mode, primary_conninfo, wal_level, max_replication_slots.","type":"object","default":{"max_connections":"100"},"additionalProperties":{"anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}},"sharedPreloadLibraries":{"description":"Libraries loaded at server start, restricted to an allowlist of modules already shipped in the PostgreSQL image. CloudNativePG also preloads `pg_stat_statements`, `auto_explain` and `pgaudit` by itself whenever `parameters` sets a key under their prefix (e.g. `pg_stat_statements.track`), so removing one of them from this list does not unload it while such a key is set; `pg_prewarm` is preloaded only through this list. Changing the list restarts PostgreSQL on every instance, the primary included, so writes pause briefly.","type":"array","items":{"type":"string","enum":["pg_stat_statements","auto_explain","pgaudit","pg_prewarm"]}}}},"quorum":{"description":"Quorum configuration for synchronous replication.","type":"object","default":{},"required":["maxSyncReplicas","minSyncReplicas"],"properties":{"maxSyncReplicas":{"description":"Maximum number of synchronous replicas allowed (must be less than total replicas). Under autoscaling it also sets the scale-down floor (the effective minimum is `max(minReplicas, maxSyncReplicas+1, 2)`); the chart fails to render if that floor exceeds `autoscaling.maxReplicas`.","type":"integer","default":0},"minSyncReplicas":{"description":"Minimum number of synchronous replicas required for commit.","type":"integer","default":0}}},"autoscaling":{"description":"Read-replica autoscaling configuration.","type":"object","default":{},"required":["dryRun","enabled","maxReplicas","maxReplicationLagSeconds","metric","minReplicas","serverAddress","target","transition"],"properties":{"dryRun":{"description":"Recommendation mode: keep the static count and render the ScaledObject with both scaling directions paused, so KEDA keeps the HPA and keeps serving its trigger metric but actuates neither way. Note both scaling policies are disabled, so the HPA's desiredReplicas is frozen to the current count - read the recommendation from the matching read-load dashboard panel for your metric (connections or CPU): desired = 1 + ceil(read load / target), clamped to [effectiveMin..maxReplicas] where effectiveMin = max(minReplicas, maxSyncReplicas+1, 2), not from the desired-replicas line. Like disabling, flipping a LIVE autoscaled cluster into dryRun re-renders `instances: replicas` (the raw static value, which can be below the autoscaling floor); if `replicas` is below the live count, stage it to the live count first (see the enablement note) or the cluster sheds down to `replicas`. No shed occurs when the rendered value is unchanged (replicas already equals the active seed) — client-side merge is then a no-op and the live count is preserved. Permanent, unlike `transition`.","type":"boolean","default":false},"enabled":{"description":"Enable horizontal autoscaling of read replicas. Requires the platform `keda` package to be enabled by an administrator first; the chart fails to render (rather than break the release) if KEDA is not installed. Prerequisite: the queried vmselect must hold BOTH the CNPG metrics (cnpg_backends_total / container CPU) AND kube_pod_labels (kube-state-metrics). This holds for a tenant on the shared root monitoring stack (`monitoring: false`, both land in tenant-root). A tenant running its OWN Monitoring app (`monitoring: true`) has the CNPG metrics but not kube_pod_labels (KSM is root-only), so it must set `serverAddress` to a vmselect carrying both series or autoscaling never actuates.","type":"boolean","default":false},"maxReplicas":{"description":"Maximum total instances. Must be >= minReplicas and >= the synchronous-quorum floor (maxSyncReplicas+1); the chart fails to render (rather than silently run an unbounded instance count) if the effective floor exceeds it - raise maxReplicas. HAZARD: lowering maxReplicas below the live count makes the HPA cut straight to the new maximum in one step (the current>max branch bypasses the scale-down pacing), shedding the surplus replicas + PVCs at once; to lower it deliberately, suspend the HelmRelease and pin the live count with `kubectl scale` first.","type":"integer","default":6},"maxReplicationLagSeconds":{"description":"Freeze scaling while replication lag exceeds this (seconds) and the primary is writing. 0 disables the brake. Default 0: the freeze branch has only been validated on a live cluster in its non-braking (pass-through) form, so the lag brake is opt-in until the braking path is exercised under load; set a positive value (e.g. 30) to enable it.","type":"integer","default":0},"metric":{"description":"Read-load signal to scale on.","type":"string","default":"ReadConnections","enum":["ReadConnections","ReadCPUUtilization"]},"minReplicas":{"description":"Minimum total instances; raised to the synchronous-quorum floor (`maxSyncReplicas + 1`) and to 2 when either is higher.","type":"integer","default":2},"serverAddress":{"description":"Override the Prometheus/vmselect URL the ScaledObject queries. Empty (default) derives `http://vmselect-shortterm..svc:8481/select/0/prometheus`. Set it when the derived vmselect does not hold both the CNPG metrics and kube_pod_labels (e.g. `monitoring: true` tenants, or a non-default `metricsStorages` name). SSRF constraint: the host must be an in-cluster Service DNS name (`..svc[.]`) whose namespace is within the tenant's own subtree (its namespace, its parents, or `tenant-root`) — the cluster-privileged KEDA operator fetches this URL every poll, so it is refused a metadata endpoint, an external host, or a sibling/system namespace. Residual surface (accepted trade-off): `tenant-root` is always allowed because the derived default points there, so a tenant can still aim the operator at any Service in `tenant-root` on any port/path (a blind in-cluster reachability probe — the operator reads no response body back to the tenant); the egress NetworkPolicy does not narrow that.","type":"string","default":""},"target":{"description":"Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold.","type":"integer","default":150},"transition":{"description":"Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject in both directions, so KEDA stands up with its HPA present but not actuating. Flip to false (phase 2) to hand the runtime count to KEDA. Because the both-direction pause keeps the HPA (not the HPA-deleting full pause), KEDA is genuinely warm at phase 2 — no HPA recreate — and because the active seed is max(replicas, effectiveMin), phase 2 re-renders the same instance count phase 1 did (when `replicas` is staged at/above the floor), so the flip is a merge no-op with no dip. The HPA's scaleUp stabilization window still smooths the first actuation.","type":"boolean","default":false}}},"users":{"description":"Users configuration map. Passwords are always auto-generated and stored in the `-credentials` Secret; they cannot be set from values — read a user's password from that Secret. A `password` left over in values from before the field was removed is ignored by the render and draws an admission warning, but it is not inert on an upgraded release: the chart preserves whatever password is already in the Secret, which on the first upgrade is the value that was set before removal, so that value stays the live credential until it is rotated (dedicated rotation is tracked in cozystack/community#72). The live password lives only in the Secret.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"bypassRls":{"description":"Whether the user can bypass row-level security policies (BYPASSRLS). A role can only create other roles carrying BYPASSRLS if it holds the attribute itself, so this is required alongside `createRole` to provision a privileged service role.","type":"boolean"},"createRole":{"description":"Whether the user can create and manage roles (CREATEROLE). Requires PostgreSQL 16 or newer. Setting it back to `false` withdraws the attribute only: the roles the user created, and its ADMIN and SET memberships in them, stay. Once any user has held it, the chart no longer adopts an existing role it did not create, so a user or database declared later under the name of such a role fails the init job until a platform administrator drops that role.","type":"boolean"},"replication":{"description":"Whether the user has replication privileges.","type":"boolean"}}}},"databases":{"description":"Databases configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"extensions":{"description":"List of enabled PostgreSQL extensions.","type":"array","items":{"type":"string"}},"roles":{"description":"Roles assigned to users.","type":"object","properties":{"admin":{"description":"List of users with admin privileges.","type":"array","items":{"type":"string"}},"readonly":{"description":"List of users with read-only privileges.","type":"array","items":{"type":"string"}}}}}}},"backup":{"description":"Backup configuration.","type":"object","default":{},"required":["enabled"],"properties":{"destinationPath":{"description":"DEPRECATED. Per-tenant S3 configuration is superseded by the platform-managed `cozy-default` BackupClass and the `cozy-backups` system bucket. Leave empty for new installations; the BackupClass driver picks up the system-managed coordinates. Kept for in-place upgrade compatibility.","type":"string","default":"s3://bucket/path/to/folder/"},"enabled":{"description":"Enable regular backups.","type":"boolean","default":false},"endpointCA":{"description":"DEPRECATED. Pre-existing Secret with the CA bundle the barman-cloud plugin should trust when reaching a self-signed S3 endpoint. Used for both backup and bootstrap recovery in the legacy chart-managed flow.","type":"object","default":{},"properties":{"key":{"description":"Key within the Secret containing the CA bundle. Defaults to `ca.crt`.","type":"string","default":""},"name":{"description":"Name of the Secret in the application namespace. Empty means no endpointCA is emitted (the plugin uses the system trust store).","type":"string","default":""}}},"endpointURL":{"description":"DEPRECATED. See `destinationPath`.","type":"string","default":"http://minio-gateway-service:9000"},"retentionPolicy":{"description":"Retention policy (e.g. \"30d\").","type":"string","default":"30d"},"s3AccessKey":{"description":"DEPRECATED. Tenants no longer supply S3 keys; the system Bucket Secret is projected into the tenant namespace by the backup controller. Ignored when `s3CredentialsSecret.name` is set or `useSystemBucket` is true. The chart skips materialising `-s3-creds` whenever this field is empty so a default install does not leak placeholder credentials into the tenant namespace.","type":"string","default":""},"s3CredentialsSecret":{"description":"DEPRECATED. Pre-existing Secret with S3 credentials. Use the platform-managed `cozy-default` BackupClass instead. When set, the chart references this Secret directly (legacy chart-managed flow). The CNPG backup driver writes this field on restore so credentials never land in the CR `.spec`.","type":"object","default":{},"properties":{"accessKeyIDKey":{"description":"Key in the Secret holding the access key ID. Defaults to `AWS_ACCESS_KEY_ID`.","type":"string","default":""},"name":{"description":"Name of the Secret in the application namespace. Empty means the chart materialises `-s3-creds` from `s3AccessKey`/`s3SecretKey`.","type":"string","default":""},"secretAccessKeyKey":{"description":"Key in the Secret holding the secret access key. Defaults to `AWS_SECRET_ACCESS_KEY`.","type":"string","default":""}}},"s3SecretKey":{"description":"DEPRECATED. See `s3AccessKey`.","type":"string","default":""},"schedule":{"description":"Legacy. Cron schedule (CNPG 6-field format) for the chart-emitted ScheduledBackup. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time.","type":"string","default":""},"useSystemBucket":{"description":"Opt-in: when true, the chart-emitted `-s3-creds` Secret is skipped AND `spec.plugins` (plus the barman-cloud ObjectStore) is left UNSET in the chart-rendered Cluster — the cozy-default BackupClass driver SSA-applies an ObjectStore (carrying destinationPath/endpointURL/credentials) and patches `spec.plugins` on the live Cluster when the first BackupJob runs. Consequence: plugin WAL archiving is NOT active until that first BackupJob fires; WAL accumulates on the PVC in the meantime, so fire an ad-hoc BackupJob immediately after enabling the flag on existing releases. Use together with the platform `cozy-default` BackupClass — tenants do not need to fill `s3AccessKey`/`s3SecretKey` or `destinationPath`/`endpointURL`. The destination path automatically scopes to `s3://cozy-backups///`.","type":"boolean","default":false}}},"bootstrap":{"description":"Bootstrap configuration.","type":"object","default":{},"required":["enabled","oldName"],"properties":{"backupID":{"description":"Barman ID of the base backup to recover from, passed as `recoveryTarget.backupID`. Empty lets the barman-cloud plugin pick one: the newest backup when `recoveryTime` is empty, otherwise the newest one ending at or before it, on any timeline. The CNPG backup driver sets this on restore.","type":"string","default":""},"enabled":{"description":"Whether to restore from a backup.","type":"boolean","default":false},"newServerName":{"description":"WAL-archive server name (S3 path prefix) the RESTORED cluster writes to. Must differ from `serverName` (the recovery source) so a restored cluster archives to a fresh, empty prefix and barman-cloud-check-wal-archive passes while it recovers from the source's prefix. The CNPG backup driver sets this on restore; empty means the cluster archives under its own name.","type":"string","default":""},"oldName":{"description":"Previous cluster name before deletion.","type":"string","default":""},"recoveryTime":{"description":"Timestamp (RFC3339) for point-in-time recovery; empty means latest.","type":"string","default":""},"serverName":{"description":"Server name (S3 path prefix) used by the original cluster when writing backups; passed to the barman-cloud plugin via `externalClusters[].plugin.parameters.serverName`. Defaults to `bootstrap.oldName`. Set this only when the original cluster wrote backups under an explicit server name that differed from its Kubernetes resource name.","type":"string","default":""}}}}} + {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Postgres replicas. Under active autoscaling this is the resting/seed count, not the live count — KEDA drives the live count via `/scale`. Disabling autoscaling does NOT automatically bring the live count back to `replicas`: the client-side merge writes `instances` only when the rendered value changes, so a cluster KEDA grew past `replicas` stays there with no autoscaler left to reduce it — lower `replicas` to the count you want and it rebases down (shedding the surplus replicas + PVCs). Lowering it below the live count, or disabling/dry-running while it sits below the live count, sheds in one step past the scaleDown pacing; stage it to the live count first (suspend the HelmRelease and `kubectl scale` if needed) to avoid a transient. Must stay greater than `quorum.maxSyncReplicas`.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each PostgreSQL replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster.","type":"boolean","default":false},"version":{"description":"PostgreSQL major version to deploy","type":"string","default":"v18","enum":["v18","v17","v16","v15","v14","v13"]},"flavor":{"description":"PostgreSQL image flavor, fixed when the cluster is created. `postgresql` uses the CloudNativePG PostgreSQL image. `postgis` uses the CloudNativePG PostGIS image, which adds postgis, postgis_raster, postgis_topology, postgis_sfcgal, postgis_tiger_geocoder and address_standardizer; enable them per database in `databases..extensions`. Changing the flavor of an existing cluster is refused, and a backup restores only into a cluster of the flavor it was taken from: to change flavor, create a new cluster and move the data with a logical dump (pg_dump / pg_restore).","type":"string","default":"postgresql","enum":["postgresql","postgis"]},"tls":{"description":"TLS configuration for server connections.","type":"object","default":{},"properties":{"enabled":{"description":"Tri-state switch controlling whether the chart injects the external hostnames into the operator-managed CNPG cert via spec.certificates.serverAltDNSNames. When omitted, the chart injects them if `external: true` and skips them otherwise. Set explicitly to `true` to inject regardless of `external` (no-op when `external: false` since there is no external hostname to add). Set to `false` to skip injection. Note that CNPG keeps its built-in TLS on the wire regardless of this flag — this toggle only controls the chart-side SAN injection; to disable PostgreSQL TLS entirely set `postgresql.parameters.ssl = \"off\"` at the CNPG layer.","type":"boolean"},"required":{"description":"Tri-state switch controlling whether the server refuses sessions that are not encrypted. When omitted, it is on if `external: true` and off otherwise. When on, the chart renders `spec.postgresql.pg_hba` with `hostssl all all all scram-sha-256` (`md5` on PostgreSQL 13, matching CloudNativePG's default rule) followed by `host all all all reject`, so a client that connects with `sslmode=disable` is refused instead of getting a plaintext session; in-cluster clients are held to the same rule, since the pod network cannot be told apart from an external client. Set to `false` to keep accepting plaintext sessions on an external instance, or to `true` to require TLS on an in-cluster one. Rendering fails if `postgresql.parameters.ssl` is off while this is in effect.","type":"boolean"}}},"postgresql":{"description":"PostgreSQL server configuration.","type":"object","default":{},"properties":{"parameters":{"description":"PostgreSQL server parameters. Values may be strings or integers; integers are coerced to strings by the template (e.g. both `max_connections: 100` and `max_connections: \"100\"` are accepted). BLOCKED (enable arbitrary code execution): archive_command, restore_command, ssl_passphrase_command, archive_cleanup_command, recovery_end_command, dynamic_library_path, local_preload_libraries, session_preload_libraries, shared_preload_libraries. Do NOT override CloudNativePG-managed parameters: archive_mode, primary_conninfo, wal_level, max_replication_slots.","type":"object","default":{"max_connections":"100"},"additionalProperties":{"anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}},"sharedPreloadLibraries":{"description":"Libraries loaded at server start, restricted to an allowlist of modules already shipped in the PostgreSQL image. CloudNativePG also preloads `pg_stat_statements`, `auto_explain` and `pgaudit` by itself whenever `parameters` sets a key under their prefix (e.g. `pg_stat_statements.track`), so removing one of them from this list does not unload it while such a key is set; `pg_prewarm` is preloaded only through this list. Changing the list restarts PostgreSQL on every instance, the primary included, so writes pause briefly.","type":"array","items":{"type":"string","enum":["pg_stat_statements","auto_explain","pgaudit","pg_prewarm"]}}}},"quorum":{"description":"Quorum configuration for synchronous replication.","type":"object","default":{},"required":["maxSyncReplicas","minSyncReplicas"],"properties":{"maxSyncReplicas":{"description":"Maximum number of synchronous replicas allowed (must be less than total replicas). Under autoscaling it also sets the scale-down floor (the effective minimum is `max(minReplicas, maxSyncReplicas+1, 2)`); the chart fails to render if that floor exceeds `autoscaling.maxReplicas`.","type":"integer","default":0},"minSyncReplicas":{"description":"Minimum number of synchronous replicas required for commit.","type":"integer","default":0}}},"autoscaling":{"description":"Read-replica autoscaling configuration.","type":"object","default":{},"required":["dryRun","enabled","maxReplicas","maxReplicationLagSeconds","metric","minReplicas","serverAddress","target","transition"],"properties":{"dryRun":{"description":"Recommendation mode: keep the static count and render the ScaledObject with both scaling directions paused, so KEDA keeps the HPA and keeps serving its trigger metric but actuates neither way. Note both scaling policies are disabled, so the HPA's desiredReplicas is frozen to the current count - read the recommendation from the matching read-load dashboard panel for your metric (connections or CPU): desired = 1 + ceil(read load / target), clamped to [effectiveMin..maxReplicas] where effectiveMin = max(minReplicas, maxSyncReplicas+1, 2), not from the desired-replicas line. Like disabling, flipping a LIVE autoscaled cluster into dryRun re-renders `instances: replicas` (the raw static value, which can be below the autoscaling floor); if `replicas` is below the live count, stage it to the live count first (see the enablement note) or the cluster sheds down to `replicas`. No shed occurs when the rendered value is unchanged (replicas already equals the active seed) — client-side merge is then a no-op and the live count is preserved. Permanent, unlike `transition`.","type":"boolean","default":false},"enabled":{"description":"Enable horizontal autoscaling of read replicas. Requires the platform `keda` package to be enabled by an administrator first; the chart fails to render (rather than break the release) if KEDA is not installed. Prerequisite: the queried vmselect must hold BOTH the CNPG metrics (cnpg_backends_total / container CPU) AND kube_pod_labels (kube-state-metrics). This holds for a tenant on the shared root monitoring stack (`monitoring: false`, both land in tenant-root). A tenant running its OWN Monitoring app (`monitoring: true`) has the CNPG metrics but not kube_pod_labels (KSM is root-only), so it must set `serverAddress` to a vmselect carrying both series or autoscaling never actuates.","type":"boolean","default":false},"maxReplicas":{"description":"Maximum total instances. Must be >= minReplicas and >= the synchronous-quorum floor (maxSyncReplicas+1); the chart fails to render (rather than silently run an unbounded instance count) if the effective floor exceeds it - raise maxReplicas. HAZARD: lowering maxReplicas below the live count makes the HPA cut straight to the new maximum in one step (the current>max branch bypasses the scale-down pacing), shedding the surplus replicas + PVCs at once; to lower it deliberately, suspend the HelmRelease and pin the live count with `kubectl scale` first.","type":"integer","default":6},"maxReplicationLagSeconds":{"description":"Freeze scaling while replication lag exceeds this (seconds) and the primary is writing. 0 disables the brake. Default 0: the freeze branch has only been validated on a live cluster in its non-braking (pass-through) form, so the lag brake is opt-in until the braking path is exercised under load; set a positive value (e.g. 30) to enable it.","type":"integer","default":0},"metric":{"description":"Read-load signal to scale on.","type":"string","default":"ReadConnections","enum":["ReadConnections","ReadCPUUtilization"]},"minReplicas":{"description":"Minimum total instances; raised to the synchronous-quorum floor (`maxSyncReplicas + 1`) and to 2 when either is higher.","type":"integer","default":2},"serverAddress":{"description":"Override the Prometheus/vmselect URL the ScaledObject queries. Empty (default) derives `http://vmselect-shortterm..svc:8481/select/0/prometheus`. Set it when the derived vmselect does not hold both the CNPG metrics and kube_pod_labels (e.g. `monitoring: true` tenants, or a non-default `metricsStorages` name). SSRF constraint: the host must be an in-cluster Service DNS name (`..svc[.]`) whose namespace is within the tenant's own subtree (its namespace, its parents, or `tenant-root`) — the cluster-privileged KEDA operator fetches this URL every poll, so it is refused a metadata endpoint, an external host, or a sibling/system namespace. Residual surface (accepted trade-off): `tenant-root` is always allowed because the derived default points there, so a tenant can still aim the operator at any Service in `tenant-root` on any port/path (a blind in-cluster reachability probe — the operator reads no response body back to the tenant); the egress NetworkPolicy does not narrow that.","type":"string","default":""},"target":{"description":"Target read load per read-serving replica (unit depends on `metric`: active connections, or CPU millicores). Dimensionless integer - a resource.Quantity here would accept `150m`, which PromQL reads as 150 minutes and KEDA cannot use as a threshold.","type":"integer","default":150},"transition":{"description":"Migration phase 1: render `.spec.instances` from `replicas` (which the operator stages to the current live count) AND pause the ScaledObject in both directions, so KEDA stands up with its HPA present but not actuating. Flip to false (phase 2) to hand the runtime count to KEDA. Because the both-direction pause keeps the HPA (not the HPA-deleting full pause), KEDA is genuinely warm at phase 2 — no HPA recreate — and because the active seed is max(replicas, effectiveMin), phase 2 re-renders the same instance count phase 1 did (when `replicas` is staged at/above the floor), so the flip is a merge no-op with no dip. The HPA's scaleUp stabilization window still smooths the first actuation.","type":"boolean","default":false}}},"users":{"description":"Users configuration map. Passwords are always auto-generated and stored in the `-credentials` Secret; they cannot be set from values — read a user's password from that Secret. A `password` left over in values from before the field was removed is ignored by the render and draws an admission warning, but it is not inert on an upgraded release: the chart preserves whatever password is already in the Secret, which on the first upgrade is the value that was set before removal, so that value stays the live credential until it is rotated (dedicated rotation is tracked in cozystack/community#72). The live password lives only in the Secret.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"bypassRls":{"description":"Whether the user can bypass row-level security policies (BYPASSRLS). A role can only create other roles carrying BYPASSRLS if it holds the attribute itself, so this is required alongside `createRole` to provision a privileged service role.","type":"boolean"},"createRole":{"description":"Whether the user can create and manage roles (CREATEROLE). Requires PostgreSQL 16 or newer. Setting it back to `false` withdraws the attribute only: the roles the user created, and its ADMIN and SET memberships in them, stay. Once any user has held it, the chart no longer adopts an existing role it did not create, so a user or database declared later under the name of such a role fails the init job until a platform administrator drops that role.","type":"boolean"},"replication":{"description":"Whether the user has replication privileges.","type":"boolean"}}}},"databases":{"description":"Databases configuration map. The PostgreSQL system databases `postgres`, `template0` and `template1` are refused.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"extensions":{"description":"List of enabled PostgreSQL extensions.","type":"array","items":{"type":"string"}},"roles":{"description":"Roles assigned to users.","type":"object","properties":{"admin":{"description":"List of users with admin privileges.","type":"array","items":{"type":"string"}},"readonly":{"description":"List of users with read-only privileges.","type":"array","items":{"type":"string"}}}}}}},"backup":{"description":"Backup configuration.","type":"object","default":{},"required":["enabled"],"properties":{"destinationPath":{"description":"DEPRECATED. Per-tenant S3 configuration is superseded by the platform-managed `cozy-default` BackupClass and the `cozy-backups` system bucket. Leave empty for new installations; the BackupClass driver picks up the system-managed coordinates. Kept for in-place upgrade compatibility.","type":"string","default":"s3://bucket/path/to/folder/"},"enabled":{"description":"Enable regular backups.","type":"boolean","default":false},"endpointCA":{"description":"DEPRECATED. Pre-existing Secret with the CA bundle the barman-cloud plugin should trust when reaching a self-signed S3 endpoint. Used for both backup and bootstrap recovery in the legacy chart-managed flow.","type":"object","default":{},"properties":{"key":{"description":"Key within the Secret containing the CA bundle. Defaults to `ca.crt`.","type":"string","default":""},"name":{"description":"Name of the Secret in the application namespace. Empty means no endpointCA is emitted (the plugin uses the system trust store).","type":"string","default":""}}},"endpointURL":{"description":"DEPRECATED. See `destinationPath`.","type":"string","default":"http://minio-gateway-service:9000"},"retentionPolicy":{"description":"Retention policy (e.g. \"30d\").","type":"string","default":"30d"},"s3AccessKey":{"description":"DEPRECATED. Tenants no longer supply S3 keys; the system Bucket Secret is projected into the tenant namespace by the backup controller. Ignored when `s3CredentialsSecret.name` is set or `useSystemBucket` is true. The chart skips materialising `-s3-creds` whenever this field is empty so a default install does not leak placeholder credentials into the tenant namespace.","type":"string","default":""},"s3CredentialsSecret":{"description":"DEPRECATED. Pre-existing Secret with S3 credentials. Use the platform-managed `cozy-default` BackupClass instead. When set, the chart references this Secret directly (legacy chart-managed flow). The CNPG backup driver writes this field on restore so credentials never land in the CR `.spec`.","type":"object","default":{},"properties":{"accessKeyIDKey":{"description":"Key in the Secret holding the access key ID. Defaults to `AWS_ACCESS_KEY_ID`.","type":"string","default":""},"name":{"description":"Name of the Secret in the application namespace. Empty means the chart materialises `-s3-creds` from `s3AccessKey`/`s3SecretKey`.","type":"string","default":""},"secretAccessKeyKey":{"description":"Key in the Secret holding the secret access key. Defaults to `AWS_SECRET_ACCESS_KEY`.","type":"string","default":""}}},"s3SecretKey":{"description":"DEPRECATED. See `s3AccessKey`.","type":"string","default":""},"schedule":{"description":"Legacy. Cron schedule (CNPG 6-field format) for the chart-emitted ScheduledBackup. Empty means no chart-managed schedule, which is the recommended setup when a `BackupClass` from `backups.cozystack.io` already drives backup orchestration. In the legacy chart-managed flow `spec.plugins` plus the barman-cloud ObjectStore is rendered when `backup.enabled=true` AND `useSystemBucket=false` AND `destinationPath` is non-empty AND inline-or-external creds are supplied; in the platform `useSystemBucket=true` flow the chart skips emitting `spec.plugins` and the CNPG driver SSA-applies the ObjectStore and patches `spec.plugins` onto the live Cluster at first BackupJob time.","type":"string","default":""},"useSystemBucket":{"description":"Opt-in: when true, the chart-emitted `-s3-creds` Secret is skipped AND `spec.plugins` (plus the barman-cloud ObjectStore) is left UNSET in the chart-rendered Cluster — the cozy-default BackupClass driver SSA-applies an ObjectStore (carrying destinationPath/endpointURL/credentials) and patches `spec.plugins` on the live Cluster when the first BackupJob runs. Consequence: plugin WAL archiving is NOT active until that first BackupJob fires; WAL accumulates on the PVC in the meantime, so fire an ad-hoc BackupJob immediately after enabling the flag on existing releases. Use together with the platform `cozy-default` BackupClass — tenants do not need to fill `s3AccessKey`/`s3SecretKey` or `destinationPath`/`endpointURL`. The destination path automatically scopes to `s3://cozy-backups///`.","type":"boolean","default":false}}},"bootstrap":{"description":"Bootstrap configuration.","type":"object","default":{},"required":["enabled","oldName"],"properties":{"backupID":{"description":"Barman ID of the base backup to recover from, passed as `recoveryTarget.backupID`. Empty lets the barman-cloud plugin pick one: the newest backup when `recoveryTime` is empty, otherwise the newest one ending at or before it, on any timeline. The CNPG backup driver sets this on restore.","type":"string","default":""},"enabled":{"description":"Whether to restore from a backup.","type":"boolean","default":false},"newServerName":{"description":"WAL-archive server name (S3 path prefix) the RESTORED cluster writes to. Must differ from `serverName` (the recovery source) so a restored cluster archives to a fresh, empty prefix and barman-cloud-check-wal-archive passes while it recovers from the source's prefix. The CNPG backup driver sets this on restore; empty means the cluster archives under its own name.","type":"string","default":""},"oldName":{"description":"Previous cluster name before deletion.","type":"string","default":""},"recoveryTime":{"description":"Timestamp (RFC3339) for point-in-time recovery; empty means latest.","type":"string","default":""},"serverName":{"description":"Server name (S3 path prefix) used by the original cluster when writing backups; passed to the barman-cloud plugin via `externalClusters[].plugin.parameters.serverName`. Defaults to `bootstrap.oldName`. Set this only when the original cluster wrote backups under an explicit server name that differed from its Kubernetes resource name.","type":"string","default":""}}}}} release: prefix: postgres- labels: From 4f35af6694116c6b5ac96482e7f110e0d28208db Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 9 Nov 2026 11:18:49 +0100 Subject: [PATCH 233/518] fix(monitoring-agents): let the release uninstall when the VictoriaMetrics operator is gone first Disabling the Monitoring Agents addon of a tenant cluster removes this release, the VictoriaMetrics operator and the VPA CRDs together, and Flux deletes them in parallel without honouring dependsOn. When the operator goes first, nothing clears apps.victoriametrics.com/finalizer on the VMAgent and the other VictoriaMetrics objects this chart created, so they stay Terminating and the release sits in UninstallFailed for good. Clearing those finalizers has to happen inside the uninstall and must never be what blocks it, and a live operator still has to get the first chance to finalize its objects. Only objects of this release may be touched, and release names are unique only within a namespace, so the release name alone does not identify them. kubectl loads in-cluster credentials only while the client config is otherwise default, which a --request-timeout breaks. A VPA recorded in the release manifest while its CRD is not served cannot be mapped at uninstall either. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- hack/monitoring-agents-vm-cleanup-hook.bats | 312 ++++++++++++++++++ .../templates/vm-cleanup-hook.yaml | 167 ++++++++++ .../monitoring-agents/templates/vpa.yaml | 6 + .../tests/uninstall_test.yaml | 141 ++++++++ packages/system/monitoring-agents/values.yaml | 12 + 5 files changed, 638 insertions(+) create mode 100644 hack/monitoring-agents-vm-cleanup-hook.bats create mode 100644 packages/system/monitoring-agents/templates/vm-cleanup-hook.yaml create mode 100644 packages/system/monitoring-agents/tests/uninstall_test.yaml diff --git a/hack/monitoring-agents-vm-cleanup-hook.bats b/hack/monitoring-agents-vm-cleanup-hook.bats new file mode 100644 index 000000000..290a91314 --- /dev/null +++ b/hack/monitoring-agents-vm-cleanup-hook.bats @@ -0,0 +1,312 @@ +#!/usr/bin/env bats +# Behavioural tests for the monitoring-agents pre-delete hook: +# +# packages/system/monitoring-agents/templates/vm-cleanup-hook.yaml +# +# The script is extracted from the rendered chart and RUN against a fake +# kubectl that keeps the VictoriaMetrics objects in a state file, with fake +# `date` and `sleep` so the 60s grace period passes in an instant. The +# helm-unittest suite in packages/system/monitoring-agents/tests/ +# uninstall_test.yaml matches the script text instead, and stayed green when the +# ownership condition was removed, when the finalizer patch was made a no-op and +# when the polling condition was inverted. Each of those turns this file red. +# +# What is pinned here: +# * only objects whose Helm annotations name this release, by name and +# namespace both, are deleted or patched: an object of another release, of +# a same-named release in another namespace, or one Helm does not own, is +# left alone; +# * with the operator gone, the hook waits before touching finalizers, then +# clears them, and nothing of this release is left behind; +# * with the operator alive, it finalizes the objects itself and the hook +# never patches a finalizer; +# * a kind whose CRD is gone, or a release with nothing left, exits 0, since a +# failing pre-delete hook would itself block the uninstall. +# +# Run from the repo root with make bats-unit-tests. Direct status checks +# retain compatibility with cozytest, which has no Bats run/setup helpers. +# Inline cleanup preserves Bats' TAP handler on failed assertions. + +load test_helper + +MA_CHART=packages/system/monitoring-agents +MA_RELEASE=monitoring-agents +MA_NAMESPACE=cozy-monitoring + +# render_hook +# Extract the named Job's shell script from the rendered chart into . The +# non-empty check must be an early `return 1`: hack/cozytest.sh rewrites every +# line matching ^}$ into `return 0` plus `}`, helpers included, so a check left +# in last-command position has its status discarded. +render_hook() { + helm template "$MA_RELEASE" "$MA_CHART" --namespace "$MA_NAMESPACE" \ + | yq eval "select(.kind == \"Job\" and .metadata.name == \"$1\") | .spec.template.spec.containers[0].command[2]" - > "$2" + [ -s "$2" ] || return 1 +} + +# write_fakes +# Emit fake `kubectl`, `date` and `sleep` into . +# +# kubectl reads and writes $STATE, one object per line as +# kind|release-name|release-namespace|namespace|name, where the two release +# fields are the Helm annotations (empty when Helm does not own the object). +# `get` prints this kind's objects in the hook's jsonpath format, or fails for a +# kind listed in $MISSING_KINDS as kubectl does when the CRD is gone. `delete` +# removes the object only when $OPERATOR_LIVE is 1, standing for an operator +# that finalizes it; otherwise the object stays, Terminating. `patch` removes it +# only when the payload actually clears the finalizers. Every call goes to $KLOG. +# +# date advances 1s per call from $CLOCK, so the 60s wait ends after sixty +# of polls, and records each time it returns in $KLOG; sleep records itself, +# and with $SLEEP_ADVANCES=1 also moves $CLOCK by its argument. The log thus +# shows how much fake time passed before each patch. +# No line below is a bare column-0 `}`, so cozytest.sh's parser leaves it alone. +write_fakes() { + cat > "$1/kubectl" <<'KEOF' +#!/bin/sh +echo "kubectl $*" >> "$KLOG" +drop() { + awk -F'|' -v k="$1" -v ns="$2" -v n="$3" '!($1 == k && $4 == ns && $5 == n)' "$STATE" > "$STATE.new" + mv "$STATE.new" "$STATE"; } +case "$1" in + get) + kind=${2%%.*} + case " ${MISSING_KINDS:-} " in + *" $kind "*) + echo "error: the server doesn't have a resource type \"$kind\"" >&2 + exit 1 ;; + esac + awk -F'|' -v k="$kind" '$1 == k { print $2 "|" $3 "|" $4 "|" $5 }' "$STATE" + exit 0 ;; + -n) + ns=$2 verb=$3 kind=${4%%.*} name=$5 + case "$verb" in + delete) + [ "${OPERATOR_LIVE:-0}" = 1 ] && drop "$kind" "$ns" "$name" + echo "$kind \"$name\" deleted" + exit 0 ;; + patch) + case "$*" in + *'"finalizers":null'*) drop "$kind" "$ns" "$name" ;; + esac + echo "$kind \"$name\" patched" + exit 0 ;; + esac ;; +esac +echo "fake kubectl: unexpected call: $*" >&2 +exit 64 +KEOF + cat > "$1/date" <<'KEOF' +#!/bin/sh +c=$(cat "$CLOCK" 2>/dev/null || echo 0) +echo $((c + 1)) > "$CLOCK" +echo "date $((1000000 + c))" >> "$KLOG" +echo $((1000000 + c)) +KEOF + cat > "$1/sleep" <<'KEOF' +#!/bin/sh +echo "sleep $*" >> "$KLOG" +if [ "${SLEEP_ADVANCES:-0}" = 1 ]; then + c=$(cat "$CLOCK" 2>/dev/null || echo 0) + echo $((c + $1)) > "$CLOCK" +fi +KEOF + chmod +x "$1/kubectl" "$1/date" "$1/sleep" +} + +# write_state +# Three objects of this release in two namespaces; a VMRule of another release; +# a VMRule whose release name matches but whose release namespace does not, so +# only the pair of annotations tells it apart; and a VMServiceScrape that +# carries no Helm annotations. +write_state() { + cat > "$1" < +# Run the rendered hook with the fakes first on PATH; its status lands in $rc. +run_hook() { + export KLOG="$1/calls" STATE="$1/state" CLOCK="$1/clock" + : > "$KLOG" + rc=0 + PATH="$1:$PATH" sh "$1/hook.sh" > "$1/out" 2> "$1/err" || rc=$? +} + +@test "a render that produces no script fails instead of passing as a no-op" { + # Guards the guard: every test below runs whatever render_hook produced, so an + # empty render would satisfy the exit-0 assertions without executing a line of + # the hook. + tmp=$(mktemp -d) + if render_hook no-such-job "$tmp/hook.sh"; then + echo "FAIL: render_hook reported success for a Job name the chart never emits" + false + fi + rm -rf "$tmp" +} + +@test "vm cleanup hook clears the finalizers of this release's objects only, after the grace period" { + tmp=$(mktemp -d) + write_fakes "$tmp" + write_state "$tmp/state" + render_hook vm-objects-cleanup "$tmp/hook.sh" + export OPERATOR_LIVE=0 MISSING_KINDS="" + run_hook "$tmp" + + if [ "$rc" -ne 0 ]; then + echo "FAIL: hook exited $rc" + cat "$tmp/out" "$tmp/err" + false + fi + for obj in "vmagents.operator.victoriametrics.com vmagent" \ + "vmrules.operator.victoriametrics.com alerts-node" \ + "vmservicescrapes.operator.victoriametrics.com operator"; do + if ! grep -q -- " delete $obj --wait=false" "$tmp/calls"; then + echo "FAIL: $obj was not deleted" + cat "$tmp/calls" + false + fi + done + # The objects of this release must be gone, which the fake only does for a + # patch that really clears the finalizers. + if grep -q "^[a-z]*|$MA_RELEASE|$MA_NAMESPACE|" "$tmp/state"; then + echo "FAIL: objects of this release are still there, Terminating" + cat "$tmp/state" "$tmp/calls" + false + fi + # Another release's object, one of a same-named release in another + # namespace, and an object Helm does not own are never touched. + if grep -E -q -- ' (delete|patch) [a-z.]+ (foreign-rule|foreign-ns-rule|unmanaged-scrape)( |$)' "$tmp/calls"; then + echo "FAIL: the hook touched an object this release does not own" + cat "$tmp/calls" + false + fi + if [ "$(grep -c '|' "$tmp/state")" -ne 3 ]; then + echo "FAIL: the three objects this release does not own should remain" + cat "$tmp/state" + false + fi + # The operator gets its chance first: the hook polls and sleeps before the + # first patch, and does not stop waiting before the 60s are up. + first_patch=$(grep -n ' patch ' "$tmp/calls" | head -1 | cut -d: -f1) + if [ -z "$first_patch" ]; then + echo "FAIL: no finalizer was cleared" + cat "$tmp/calls" + false + fi + # The first date call sets the deadline; the last one before the first patch + # is the time the hook gave up waiting. + start=$(grep '^date ' "$tmp/calls" | head -1 | cut -d' ' -f2) + waited_until=$(head -n "$first_patch" "$tmp/calls" | grep '^date ' | tail -1 | cut -d' ' -f2) + if [ -z "$start" ] || [ $((waited_until - start)) -lt 60 ]; then + echo "FAIL: finalizers cleared after $((waited_until - start))s; the 60s grace period was cut short" + cat "$tmp/calls" + false + fi + rm -rf "$tmp" +} + +@test "vm cleanup hook clears the finalizers before its Job's deadline" { + # The grace-period test above keeps sleep free so a shortened wait shows; here + # sleep costs its argument, so a poll interval long enough to overshoot the + # Job's activeDeadlineSeconds fails instead of passing. + tmp=$(mktemp -d) + write_fakes "$tmp" + write_state "$tmp/state" + render_hook vm-objects-cleanup "$tmp/hook.sh" + job_deadline=$(helm template "$MA_RELEASE" "$MA_CHART" --namespace "$MA_NAMESPACE" \ + | yq eval 'select(.kind == "Job" and .metadata.name == "vm-objects-cleanup") | .spec.activeDeadlineSeconds' -) + export OPERATOR_LIVE=0 MISSING_KINDS="" SLEEP_ADVANCES=1 + run_hook "$tmp" + unset SLEEP_ADVANCES + + if [ "$rc" -ne 0 ]; then + echo "FAIL: hook exited $rc" + cat "$tmp/out" "$tmp/err" + false + fi + first_patch=$(grep -n ' patch ' "$tmp/calls" | head -1 | cut -d: -f1) + start=$(grep '^date ' "$tmp/calls" | head -1 | cut -d' ' -f2) + waited_until=$(head -n "${first_patch:-0}" "$tmp/calls" | grep '^date ' | tail -1 | cut -d' ' -f2) + if [ -z "$first_patch" ] || [ -z "$job_deadline" ] || [ $((waited_until - start)) -ge "$job_deadline" ]; then + echo "FAIL: first finalizer patch at $((waited_until - start))s, Job deadline ${job_deadline}s" + cat "$tmp/calls" + false + fi + rm -rf "$tmp" +} + +@test "vm cleanup hook leaves the finalizers to a live operator" { + tmp=$(mktemp -d) + write_fakes "$tmp" + write_state "$tmp/state" + render_hook vm-objects-cleanup "$tmp/hook.sh" + export OPERATOR_LIVE=1 MISSING_KINDS="" + run_hook "$tmp" + + if [ "$rc" -ne 0 ]; then + echo "FAIL: hook exited $rc" + cat "$tmp/out" "$tmp/err" + false + fi + if ! grep -q ' delete vmagents.operator.victoriametrics.com vmagent ' "$tmp/calls"; then + echo "FAIL: the VMAgent was not deleted" + cat "$tmp/calls" + false + fi + if grep -q ' patch ' "$tmp/calls"; then + echo "FAIL: the hook cleared finalizers the live operator was handling" + cat "$tmp/calls" + false + fi + rm -rf "$tmp" +} + +@test "vm cleanup hook exits 0 and cleans the other kinds when a CRD is gone" { + tmp=$(mktemp -d) + write_fakes "$tmp" + write_state "$tmp/state" + render_hook vm-objects-cleanup "$tmp/hook.sh" + export OPERATOR_LIVE=0 MISSING_KINDS="vmrules vmpodscrapes vmnodescrapes" + run_hook "$tmp" + + if [ "$rc" -ne 0 ]; then + echo "FAIL: hook exited $rc with a CRD gone, which would block the uninstall" + cat "$tmp/out" "$tmp/err" + false + fi + if grep -q "^vmagents|$MA_RELEASE|" "$tmp/state"; then + echo "FAIL: the VMAgent was left Terminating because another kind's CRD was gone" + cat "$tmp/state" "$tmp/calls" + false + fi + rm -rf "$tmp" +} + +@test "vm cleanup hook exits 0 without deleting anything when the release owns nothing" { + tmp=$(mktemp -d) + write_fakes "$tmp" + printf 'vmrules|other|%s|%s|foreign-rule\nvmrules|%s|other-namespace|%s|foreign-ns-rule\n' \ + "$MA_NAMESPACE" "$MA_NAMESPACE" "$MA_RELEASE" "$MA_NAMESPACE" > "$tmp/state" + render_hook vm-objects-cleanup "$tmp/hook.sh" + export OPERATOR_LIVE=0 MISSING_KINDS="" + run_hook "$tmp" + + if [ "$rc" -ne 0 ]; then + echo "FAIL: hook exited $rc with nothing to clean" + cat "$tmp/out" "$tmp/err" + false + fi + if grep -E -q ' (delete|patch) ' "$tmp/calls"; then + echo "FAIL: the hook deleted or patched with nothing of this release left" + cat "$tmp/calls" + false + fi + rm -rf "$tmp" +} diff --git a/packages/system/monitoring-agents/templates/vm-cleanup-hook.yaml b/packages/system/monitoring-agents/templates/vm-cleanup-hook.yaml new file mode 100644 index 000000000..efdbd1158 --- /dev/null +++ b/packages/system/monitoring-agents/templates/vm-cleanup-hook.yaml @@ -0,0 +1,167 @@ +{{- /* + The VictoriaMetrics operator puts its finalizer on the VMAgent and is the only + thing that ever removes it. The operator ships as a separate release that is + gated on the same addon flag, so disabling the addon removes both at once and + Flux does not order deletions: when the operator goes first, the VMAgent stays + Terminating and the uninstall never finishes. The other kinds are handled the + same way in case an operator version finalizes them too; for those the patch + is a no-op. + + So delete them before Helm does, give a live operator a bounded chance to + finalize them itself, and only then clear the finalizers by hand. The VMAgent's + Deployment, Service and Secrets carry owner references and are garbage + collected once it is gone. + + Cluster-scoped RBAC because the objects span two namespaces, and a Role in the + operator's namespace could not be created if that namespace were already + gone. The hook always exits 0: a cleanup that cannot complete must not be what + keeps the release from uninstalling. +*/ -}} +{{- $name := "vm-objects-cleanup" }} +{{- $img := printf "%s:%s" .Values.kubectlImage.repository .Values.kubectlImage.tag }} +{{- with .Values.kubectlImage.digest }}{{- $img = printf "%s@%s" $img . }}{{- end }} +{{- $kinds := list "vmagents" "vmrules" "vmservicescrapes" "vmpodscrapes" "vmnodescrapes" }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ $name }} + namespace: {{ .Release.Namespace }} + annotations: + helm.sh/hook: pre-delete + helm.sh/hook-weight: "-10" + helm.sh/hook-delete-policy: hook-succeeded,hook-failed,before-hook-creation +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ .Release.Namespace }}-{{ $name }} + annotations: + helm.sh/hook: pre-delete + helm.sh/hook-weight: "-10" + helm.sh/hook-delete-policy: hook-succeeded,hook-failed,before-hook-creation +rules: + - apiGroups: ["operator.victoriametrics.com"] + resources: + {{- toYaml $kinds | nindent 6 }} + verbs: ["get", "list", "delete", "patch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ .Release.Namespace }}-{{ $name }} + annotations: + helm.sh/hook: pre-delete + helm.sh/hook-weight: "-10" + helm.sh/hook-delete-policy: hook-succeeded,hook-failed,before-hook-creation +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ .Release.Namespace }}-{{ $name }} +subjects: + - kind: ServiceAccount + name: {{ $name }} + namespace: {{ .Release.Namespace }} +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ $name }} + namespace: {{ .Release.Namespace }} + annotations: + helm.sh/hook: pre-delete + helm.sh/hook-weight: "0" + helm.sh/hook-delete-policy: hook-succeeded,before-hook-creation +spec: + # Bounds every kubectl call as well: --request-timeout would make kubectl + # skip the in-cluster credentials (hack/check-hook-request-timeout.bats). + # Well inside the HelmRelease's 10m uninstall budget, and above the 60s + # wait plus one call per object on either side of it. + activeDeadlineSeconds: 180 + backoffLimit: 0 + ttlSecondsAfterFinished: 600 + template: + spec: + serviceAccountName: {{ $name }} + restartPolicy: Never + securityContext: + runAsNonRoot: true + # The image user is the non-numeric `nonroot`, which the kubelet cannot + # check against runAsNonRoot. + runAsUser: 65532 + seccompProfile: + type: RuntimeDefault + volumes: + - name: home + emptyDir: {} + containers: + - name: cleanup + image: {{ $img | quote }} + imagePullPolicy: {{ .Values.kubectlImage.pullPolicy }} + resources: + requests: + cpu: 10m + memory: 32Mi + limits: + cpu: 100m + memory: 64Mi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: ["ALL"] + # kubectl writes its discovery cache under $HOME. + env: + - name: HOME + value: /tmp + volumeMounts: + - name: home + mountPath: /tmp + command: + - /bin/sh + - -c + - | + rel={{ .Release.Name | quote }} + relns={{ .Release.Namespace | quote }} + kinds={{ join " " $kinds | quote }} + + # Selected by Helm's ownership annotations rather than by name, so + # the alert rules added under alerts/ are covered without a list + # to keep in step. A kind whose CRD is gone just prints an error. + owned() { + for k in $kinds; do + kubectl get "$k.operator.victoriametrics.com" -A \ + -o jsonpath='{range .items[*]}{.metadata.annotations.meta\.helm\.sh/release-name}{"|"}{.metadata.annotations.meta\.helm\.sh/release-namespace}{"|"}{.metadata.namespace}{"|"}{.metadata.name}{"\n"}{end}' | + while IFS='|' read -r r rns ns n; do + if [ "$r" = "$rel" ] && [ "$rns" = "$relns" ]; then + echo "$k.operator.victoriametrics.com $ns $n" + fi + done + done + } + + objs=$(owned) + if [ -z "$objs" ]; then + echo "no VictoriaMetrics objects left for $relns/$rel" + exit 0 + fi + + echo "$objs" | while read -r k ns n; do + kubectl -n "$ns" delete "$k" "$n" --wait=false + done + + deadline=$(( $(date +%s) + 60 )) + while [ "$(date +%s)" -lt "$deadline" ]; do + sleep 2 + objs=$(owned) + if [ -z "$objs" ]; then + echo "the operator finalized every object" + exit 0 + fi + done + + echo "finalizers still held after 60s; clearing them by hand" + echo "$objs" | while read -r k ns n; do + kubectl -n "$ns" patch "$k" "$n" --type=merge \ + -p '{"metadata":{"finalizers":null}}' + done + exit 0 diff --git a/packages/system/monitoring-agents/templates/vpa.yaml b/packages/system/monitoring-agents/templates/vpa.yaml index 9a31fab94..1ac949f30 100644 --- a/packages/system/monitoring-agents/templates/vpa.yaml +++ b/packages/system/monitoring-agents/templates/vpa.yaml @@ -6,6 +6,11 @@ Initial sets requests at creation without evicting, removing the trigger. Do not switch back to Auto without fixing the underlying Cilium IP-reuse race. */ -}} +{{- /* + Rendered only when the CRD is served: a release manifest that records a VPA + the cluster cannot map is one Helm can no longer uninstall. +*/ -}} +{{- if .Capabilities.APIVersions.Has "autoscaling.k8s.io/v1/VerticalPodAutoscaler" }} apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: @@ -34,3 +39,4 @@ spec: maxAllowed: cpu: 4000m memory: 6G +{{- end }} diff --git a/packages/system/monitoring-agents/tests/uninstall_test.yaml b/packages/system/monitoring-agents/tests/uninstall_test.yaml new file mode 100644 index 000000000..df8a3eaa6 --- /dev/null +++ b/packages/system/monitoring-agents/tests/uninstall_test.yaml @@ -0,0 +1,141 @@ +suite: the release uninstalls whatever order its dependencies go in + +# Disabling the addon removes this release, the VictoriaMetrics operator and the +# VPA CRDs together, and Flux deletes them in parallel. When the operator goes +# first, nothing clears its finalizer on the objects this chart created and the +# uninstall hangs; when the VPA CRD goes first, a release manifest recording a +# VPA cannot be built and the uninstall fails. +release: + name: cozy-monitoring-agents + namespace: cozy-monitoring +templates: + - templates/vm-cleanup-hook.yaml + - templates/vpa.yaml +tests: + - it: runs before Helm deletes anything, bounded and with one attempt + template: templates/vm-cleanup-hook.yaml + documentSelector: + path: kind + value: Job + asserts: + - equal: + path: metadata.annotations["helm.sh/hook"] + value: pre-delete + - equal: + path: metadata.namespace + value: cozy-monitoring + - equal: + path: spec.activeDeadlineSeconds + value: 180 + - equal: + path: spec.backoffLimit + value: 0 + - matchRegex: + path: spec.template.spec.containers[0].image + pattern: ^docker\.io/clastix/kubectl:v[0-9.]+@sha256:[0-9a-f]{64}$ + + - it: its RBAC exists only for the hook and is removed with it + template: templates/vm-cleanup-hook.yaml + documentSelector: + path: kind + value: ClusterRole + asserts: + - equal: + path: metadata.annotations["helm.sh/hook"] + value: pre-delete + - equal: + path: metadata.annotations["helm.sh/hook-weight"] + value: "-10" + - matchRegex: + path: metadata.annotations["helm.sh/hook-delete-policy"] + pattern: hook-succeeded + + - it: may delete and clear finalizers on every VictoriaMetrics kind the chart renders + template: templates/vm-cleanup-hook.yaml + documentSelector: + path: kind + value: ClusterRole + asserts: + - equal: + path: rules[0].apiGroups + value: ["operator.victoriametrics.com"] + - contains: + path: rules[0].resources + content: vmagents + - contains: + path: rules[0].resources + content: vmrules + - contains: + path: rules[0].resources + content: vmservicescrapes + - contains: + path: rules[0].resources + content: vmpodscrapes + - contains: + path: rules[0].resources + content: vmnodescrapes + - equal: + path: rules[0].verbs + value: ["get", "list", "delete", "patch"] + - lengthEqual: + path: rules + count: 1 + + - it: grants that role to the hook's own service account + template: templates/vm-cleanup-hook.yaml + documentSelector: + path: kind + value: ClusterRoleBinding + asserts: + - equal: + path: roleRef.name + value: cozy-monitoring-vm-objects-cleanup + - equal: + path: subjects[0] + value: + kind: ServiceAccount + name: vm-objects-cleanup + namespace: cozy-monitoring + + - it: touches only objects this release owns, and clears their finalizers + template: templates/vm-cleanup-hook.yaml + documentSelector: + path: kind + value: Job + asserts: + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: rel="cozy-monitoring-agents" + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: relns="cozy-monitoring" + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: meta\\\.helm\\\.sh/release-name + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: 'delete "\$k" "\$n" --wait=false' + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: '"finalizers":null' + # A pre-delete hook that fails aborts the uninstall it exists to unblock. + - matchRegex: + path: spec.template.spec.containers[0].command[2] + pattern: exit 0\s*$ + + - it: the VPA is rendered when its CRD is served + template: templates/vpa.yaml + capabilities: + apiVersions: + - autoscaling.k8s.io/v1/VerticalPodAutoscaler + asserts: + - hasDocuments: + count: 1 + - isKind: + of: VerticalPodAutoscaler + + - it: the VPA stays out of the release manifest when its CRD is not served + template: templates/vpa.yaml + asserts: + - hasDocuments: + count: 0 diff --git a/packages/system/monitoring-agents/values.yaml b/packages/system/monitoring-agents/values.yaml index 78a6ec9fd..bd1b501c8 100644 --- a/packages/system/monitoring-agents/values.yaml +++ b/packages/system/monitoring-agents/values.yaml @@ -552,3 +552,15 @@ prometheus-node-exporter: memory: 32Mi limits: memory: 256Mi + +# Image for the pre-delete hook that releases the VictoriaMetrics objects (see +# templates/vm-cleanup-hook.yaml). Only needs a kubectl binary. Digest-pinned so +# an upstream retag does not change what runs across the fleet; refresh with +# `docker manifest inspect docker.io/clastix/kubectl:v1.32`. Same pin as the +# etcd-operator hook. +# renovate: datasource=docker depName=docker.io/clastix/kubectl +kubectlImage: + repository: docker.io/clastix/kubectl + tag: v1.32 + digest: sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c + pullPolicy: IfNotPresent From 89aba6e104c707bf8d566740b8634823ce3e796b Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 9 Nov 2026 11:35:53 +0100 Subject: [PATCH 234/518] feat(api): add the reservation evaluator for tenant quotas Tenant quotas are declared in instance-type units but enforced against ResourceQuota.status.used, which counts virt-launcher pods with their additive virtualization overhead and drifts without recomputation. The accepted tenant-quota-reservation design makes the declared reservation the accounting authority instead; this is its first phase, the evaluator, with no consumer yet. A kind describes its consumption as CEL over its values, so the evaluator carries no per-kind knowledge and an out-of-tree kind participates without a change here. Instance types resolve to scalars keyed by class, read from the type's spec rather than its name, because a pinned core, a hugepage and an overcommitted page are not the same capacity as a shared vCPU or ordinary memory. KubeVirt pins the emulator thread's parity core per type, from the annotations the type hands to its VMs, so the classifier reads it there too. Anything that does not resolve is an error, never a smaller figure. Stored values carry only the keys a tenant wrote, so the evaluator defaults them first. The defaulting code moves out of the application registry into its own package so both read the same defaults. Defaulting replaces a value of the wrong type with the schema's default, which would turn a malformed spec into a smaller reservation, so values are type-checked against the schema before it runs. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- go.mod | 2 +- pkg/apis/apps/defaulting/defaulting.go | 201 +++++++ pkg/registry/apps/application/rest.go | 23 +- .../apps/application/rest_defaulting.go | 136 +---- pkg/reservation/aggregate.go | 76 +++ pkg/reservation/environment.go | 84 +++ pkg/reservation/evaluator.go | 453 +++++++++++++++ pkg/reservation/evaluator_test.go | 525 ++++++++++++++++++ pkg/reservation/instancetype.go | 148 +++++ pkg/reservation/instancetype_test.go | 259 +++++++++ pkg/reservation/types.go | 113 ++++ 11 files changed, 1864 insertions(+), 156 deletions(-) create mode 100644 pkg/apis/apps/defaulting/defaulting.go create mode 100644 pkg/reservation/aggregate.go create mode 100644 pkg/reservation/environment.go create mode 100644 pkg/reservation/evaluator.go create mode 100644 pkg/reservation/evaluator_test.go create mode 100644 pkg/reservation/instancetype.go create mode 100644 pkg/reservation/instancetype_test.go create mode 100644 pkg/reservation/types.go diff --git a/go.mod b/go.mod index 34352e583..cb49f9f24 100644 --- a/go.mod +++ b/go.mod @@ -16,6 +16,7 @@ require ( github.com/go-logr/logr v1.4.4 github.com/go-logr/zapr v1.3.0 github.com/go-task/slim-sprig/v3 v3.0.0 + github.com/google/cel-go v0.29.2 github.com/minio/minio-go/v7 v7.0.98 github.com/onsi/ginkgo/v2 v2.33.0 github.com/onsi/gomega v1.43.1 @@ -83,7 +84,6 @@ require ( github.com/go-openapi/testify/enable/yaml/v2 v2.8.0 // indirect github.com/go-openapi/testify/v2 v2.8.0 // indirect github.com/golang/protobuf v1.5.4 // indirect - github.com/google/cel-go v0.29.2 // indirect github.com/google/gnostic-models v0.7.1 // indirect github.com/google/go-cmp v0.7.0 // indirect github.com/google/pprof v0.0.0-20260906184651-6331bc6350fe // indirect diff --git a/pkg/apis/apps/defaulting/defaulting.go b/pkg/apis/apps/defaulting/defaulting.go new file mode 100644 index 000000000..bbec2fa88 --- /dev/null +++ b/pkg/apis/apps/defaulting/defaulting.go @@ -0,0 +1,201 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package defaulting applies an ApplicationDefinition's structural-schema +// defaults to application values. It is shared by the aggregated apiserver, +// which defaults on the read path, and by anything that has to reason about +// stored values, which carry only the keys a tenant actually wrote. +package defaulting + +import ( + "encoding/json" + "fmt" + "strings" + + internalapiext "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions" + apiextv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + structuralschema "k8s.io/apiextensions-apiserver/pkg/apiserver/schema" + "k8s.io/apimachinery/pkg/runtime" +) + +// BuildSchema parses an OpenAPI-v3 JSON schema string and returns the +// structural-schema form used for defaulting. Returns (nil, nil) when raw +// is empty after trimming. Returns (nil, error) for malformed JSON, +// v1→internal conversion failures, or structural-schema construction +// errors — callers can decide whether to log-and-continue or fail hard. +func BuildSchema(raw string) (*structuralschema.Structural, error) { + raw = strings.TrimSpace(raw) + if raw == "" { + return nil, nil + } + var v1js apiextv1.JSONSchemaProps + if err := json.Unmarshal([]byte(raw), &v1js); err != nil { + return nil, fmt.Errorf("unmarshal v1 OpenAPI schema: %w", err) + } + scheme := runtime.NewScheme() + _ = internalapiext.AddToScheme(scheme) + _ = apiextv1.AddToScheme(scheme) + var ijs internalapiext.JSONSchemaProps + if err := scheme.Convert(&v1js, &ijs, nil); err != nil { + return nil, fmt.Errorf("convert v1->internal JSONSchemaProps: %w", err) + } + s, err := structuralschema.NewStructural(&ijs) + if err != nil { + return nil, fmt.Errorf("build structural schema: %w", err) + } + return s, nil +} + +// Apply defaults root in place against s the way the Kubernetes apiserver +// defaults a custom resource. A nil schema leaves root untouched. +func Apply(root *map[string]any, s *structuralschema.Structural) error { + if s == nil { + return nil + } + v := any(*root) + nv, err := applyDefaults(v, s, true) + if err != nil { + return err + } + obj, ok := nv.(map[string]any) + if !ok && nv != nil { + return fmt.Errorf("internal error: applyDefaults returned non-map type %T for object root", nv) + } + if obj == nil { + obj = map[string]any{} + } + *root = obj + return nil +} + +func applyDefaults(v any, s *structuralschema.Structural, top bool) (any, error) { + if s == nil { + return v, nil + } + + effType := s.Type + if effType == "" { + switch { + case len(s.Properties) > 0 || (s.AdditionalProperties != nil && s.AdditionalProperties.Structural != nil): + effType = "object" + case s.Items != nil: + effType = "array" + default: + // scalar + } + } + + switch effType { + case "object": + mv, isMap := v.(map[string]any) + if !isMap || v == nil { + if s.Default.Object != nil && !top { + if dm, ok := s.Default.Object.(map[string]any); ok { + mv = cloneMap(dm) + } + } + if mv == nil { + mv = map[string]any{} + } + } + + for name, ps := range s.Properties { + if _, ok := mv[name]; !ok { + if ps.Default.Object != nil { + mv[name] = clone(ps.Default.Object) + } + } + if cur, ok := mv[name]; ok { + cv, err := applyDefaults(cur, &ps, false) + if err != nil { + return nil, err + } + mv[name] = cv + } + } + + if s.AdditionalProperties != nil && s.AdditionalProperties.Structural != nil { + ap := s.AdditionalProperties.Structural + for k, cur := range mv { + if _, isKnown := s.Properties[k]; isKnown { + continue + } + cv, err := applyDefaults(cur, ap, false) + if err != nil { + return nil, err + } + mv[k] = cv + } + } + return mv, nil + + case "array": + sl, isSlice := v.([]any) + if !isSlice || v == nil { + if s.Default.Object != nil { + if ds, ok := s.Default.Object.([]any); ok { + sl = cloneSlice(ds) + } + } + if sl == nil { + sl = []any{} + } + } + if s.Items != nil { + for i := range sl { + cv, err := applyDefaults(sl[i], s.Items, false) + if err != nil { + return nil, err + } + sl[i] = cv + } + } + return sl, nil + + default: + if v == nil && s.Default.Object != nil { + return clone(s.Default.Object), nil + } + return v, nil + } +} + +func clone(x any) any { + switch t := x.(type) { + case map[string]any: + return cloneMap(t) + case []any: + return cloneSlice(t) + default: + return t + } +} + +func cloneMap(m map[string]any) map[string]any { + out := make(map[string]any, len(m)) + for k, v := range m { + out[k] = clone(v) + } + return out +} + +func cloneSlice(s []any) []any { + out := make([]any, len(s)) + for i := range s { + out[i] = clone(s[i]) + } + return out +} diff --git a/pkg/registry/apps/application/rest.go b/pkg/registry/apps/application/rest.go index 5f618b4e3..aa0e24a62 100644 --- a/pkg/registry/apps/application/rest.go +++ b/pkg/registry/apps/application/rest.go @@ -50,6 +50,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/client" fluxshard "github.com/cozystack/cozystack/internal/fluxshardoperator" + "github.com/cozystack/cozystack/pkg/apis/apps/defaulting" "github.com/cozystack/cozystack/pkg/apis/apps/presets" appsv1alpha1 "github.com/cozystack/cozystack/pkg/apis/apps/v1alpha1" "github.com/cozystack/cozystack/pkg/apis/apps/validation" @@ -57,7 +58,6 @@ import ( "github.com/cozystack/cozystack/pkg/registry" fieldfilter "github.com/cozystack/cozystack/pkg/registry/fields" "github.com/cozystack/cozystack/pkg/registry/sorting" - internalapiext "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions" apiextv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" structuralschema "k8s.io/apiextensions-apiserver/pkg/apiserver/schema" @@ -110,26 +110,7 @@ type REST struct { // errors — callers can decide whether to log-and-continue (NewREST does) // or fail hard (tests do). func buildSpecSchema(raw string) (*structuralschema.Structural, error) { - raw = strings.TrimSpace(raw) - if raw == "" { - return nil, nil - } - var v1js apiextv1.JSONSchemaProps - if err := json.Unmarshal([]byte(raw), &v1js); err != nil { - return nil, fmt.Errorf("unmarshal v1 OpenAPI schema: %w", err) - } - scheme := runtime.NewScheme() - _ = internalapiext.AddToScheme(scheme) - _ = apiextv1.AddToScheme(scheme) - var ijs internalapiext.JSONSchemaProps - if err := scheme.Convert(&v1js, &ijs, nil); err != nil { - return nil, fmt.Errorf("convert v1->internal JSONSchemaProps: %w", err) - } - s, err := structuralschema.NewStructural(&ijs) - if err != nil { - return nil, fmt.Errorf("build structural schema: %w", err) - } - return s, nil + return defaulting.BuildSchema(raw) } // NewREST creates a new REST storage for Application with specific configuration diff --git a/pkg/registry/apps/application/rest_defaulting.go b/pkg/registry/apps/application/rest_defaulting.go index 36518af69..4fe7409dd 100644 --- a/pkg/registry/apps/application/rest_defaulting.go +++ b/pkg/registry/apps/application/rest_defaulting.go @@ -18,8 +18,8 @@ package application import ( "encoding/json" - "fmt" + "github.com/cozystack/cozystack/pkg/apis/apps/defaulting" appsv1alpha1 "github.com/cozystack/cozystack/pkg/apis/apps/v1alpha1" apiextv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" structuralschema "k8s.io/apiextensions-apiserver/pkg/apiserver/schema" @@ -51,137 +51,5 @@ func (r *REST) applySpecDefaults(app *appsv1alpha1.Application) error { } func defaultLikeKubernetes(root *map[string]any, s *structuralschema.Structural) error { - v := any(*root) - nv, err := applyDefaults(v, s, true) - if err != nil { - return err - } - obj, ok := nv.(map[string]any) - if !ok && nv != nil { - return fmt.Errorf("internal error: applyDefaults returned non-map type %T for object root", nv) - } - if obj == nil { - obj = map[string]any{} - } - *root = obj - return nil -} - -func applyDefaults(v any, s *structuralschema.Structural, top bool) (any, error) { - if s == nil { - return v, nil - } - - effType := s.Type - if effType == "" { - switch { - case len(s.Properties) > 0 || (s.AdditionalProperties != nil && s.AdditionalProperties.Structural != nil): - effType = "object" - case s.Items != nil: - effType = "array" - default: - // scalar - } - } - - switch effType { - case "object": - mv, isMap := v.(map[string]any) - if !isMap || v == nil { - if s.Default.Object != nil && !top { - if dm, ok := s.Default.Object.(map[string]any); ok { - mv = cloneMap(dm) - } - } - if mv == nil { - mv = map[string]any{} - } - } - - for name, ps := range s.Properties { - if _, ok := mv[name]; !ok { - if ps.Default.Object != nil { - mv[name] = clone(ps.Default.Object) - } - } - if cur, ok := mv[name]; ok { - cv, err := applyDefaults(cur, &ps, false) - if err != nil { - return nil, err - } - mv[name] = cv - } - } - - if s.AdditionalProperties != nil && s.AdditionalProperties.Structural != nil { - ap := s.AdditionalProperties.Structural - for k, cur := range mv { - if _, isKnown := s.Properties[k]; isKnown { - continue - } - cv, err := applyDefaults(cur, ap, false) - if err != nil { - return nil, err - } - mv[k] = cv - } - } - return mv, nil - - case "array": - sl, isSlice := v.([]any) - if !isSlice || v == nil { - if s.Default.Object != nil { - if ds, ok := s.Default.Object.([]any); ok { - sl = cloneSlice(ds) - } - } - if sl == nil { - sl = []any{} - } - } - if s.Items != nil { - for i := range sl { - cv, err := applyDefaults(sl[i], s.Items, false) - if err != nil { - return nil, err - } - sl[i] = cv - } - } - return sl, nil - - default: - if v == nil && s.Default.Object != nil { - return clone(s.Default.Object), nil - } - return v, nil - } -} - -func clone(x any) any { - switch t := x.(type) { - case map[string]any: - return cloneMap(t) - case []any: - return cloneSlice(t) - default: - return t - } -} - -func cloneMap(m map[string]any) map[string]any { - out := make(map[string]any, len(m)) - for k, v := range m { - out[k] = clone(v) - } - return out -} - -func cloneSlice(s []any) []any { - out := make([]any, len(s)) - for i := range s { - out[i] = clone(s[i]) - } - return out + return defaulting.Apply(root, s) } diff --git a/pkg/reservation/aggregate.go b/pkg/reservation/aggregate.go new file mode 100644 index 000000000..635c41284 --- /dev/null +++ b/pkg/reservation/aggregate.go @@ -0,0 +1,76 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package reservation + +import "fmt" + +// Application is one stored application as the aggregator sees it: the +// labelled HelmRelease the aggregated apiserver persisted, reduced to what +// identifies its definition and its undefaulted values. +type Application struct { + Namespace string + Name string + Kind string + Group string + Values []byte +} + +// EvaluatorLookup returns the evaluator of a kind, found by its kind and +// group so that an out-of-tree kind resolves through the same path as an +// in-tree one. +type EvaluatorLookup func(kind, group string) (*Evaluator, error) + +// Failure is an application whose reservation could not be computed. +type Failure struct { + Application Application + Err error +} + +func (f Failure) Error() string { + return fmt.Sprintf("%s %s/%s: %v", f.Application.Kind, f.Application.Namespace, f.Application.Name, f.Err) +} + +// Aggregate sums the reservations of apps per namespace. +// +// An application that cannot be evaluated is returned as a Failure and is +// left out of the sums. The sums are therefore a lower bound whenever a +// failure is returned, and the caller must not treat them as complete: a +// stored application that stops evaluating has to keep its last known +// reservation rather than hand its share of the pool to the next writer. +func Aggregate(apps []Application, lookup EvaluatorLookup, env Environment) (map[string]ResourceList, []Failure) { + perNamespace := map[string]ResourceList{} + var failures []Failure + for _, app := range apps { + ev, err := lookup(app.Kind, app.Group) + if err != nil { + failures = append(failures, Failure{Application: app, Err: err}) + continue + } + rl, err := ev.EvaluateJSON(app.Values, env) + if err != nil { + failures = append(failures, Failure{Application: app, Err: err}) + continue + } + sum, ok := perNamespace[app.Namespace] + if !ok { + sum = ResourceList{} + perNamespace[app.Namespace] = sum + } + sum.Add(rl) + } + return perNamespace, failures +} diff --git a/pkg/reservation/environment.go b/pkg/reservation/environment.go new file mode 100644 index 000000000..f6727c867 --- /dev/null +++ b/pkg/reservation/environment.go @@ -0,0 +1,84 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package reservation + +import ( + "errors" + "fmt" + + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" +) + +// ErrNotFound is wrapped by an Environment when a named instance type or +// preset does not exist. The evaluator fails closed on it rather than +// reserving nothing. +var ErrNotFound = errors.New("not found") + +// Environment resolves the cluster objects a declaration may reference by +// name. Implementations must be safe for concurrent use and return a list +// the caller may keep: the evaluator does not copy it. +type Environment interface { + Instancetype(name string) (ResourceList, error) + Preset(name string) (ResourceList, error) +} + +// StaticEnvironment is an Environment over fixed tables. +type StaticEnvironment struct { + Instancetypes map[string]ResourceList + Presets map[string]ResourceList +} + +var _ Environment = (*StaticEnvironment)(nil) + +// NewStaticEnvironment classifies every VirtualMachineClusterInstancetype in +// instancetypes and pairs the result with presets. One unclassifiable type +// fails the whole environment, so that a broken catalog is noticed when it is +// loaded rather than when a tenant first orders that type. +func NewStaticEnvironment(instancetypes []map[string]any, presets map[string]ResourceList) (*StaticEnvironment, error) { + env := &StaticEnvironment{ + Instancetypes: make(map[string]ResourceList, len(instancetypes)), + Presets: presets, + } + for _, obj := range instancetypes { + rl, err := ClassifyInstancetype(obj) + if err != nil { + return nil, err + } + name, _, _ := unstructured.NestedString(obj, "metadata", "name") + if name == "" { + return nil, fmt.Errorf("instance type without metadata.name") + } + env.Instancetypes[name] = rl + } + return env, nil +} + +func (e *StaticEnvironment) Instancetype(name string) (ResourceList, error) { + rl, ok := e.Instancetypes[name] + if !ok { + return nil, fmt.Errorf("instance type %q: %w", name, ErrNotFound) + } + return rl, nil +} + +func (e *StaticEnvironment) Preset(name string) (ResourceList, error) { + rl, ok := e.Presets[name] + if !ok { + return nil, fmt.Errorf("resource preset %q: %w", name, ErrNotFound) + } + return rl, nil +} diff --git a/pkg/reservation/evaluator.go b/pkg/reservation/evaluator.go new file mode 100644 index 000000000..6927b0bf2 --- /dev/null +++ b/pkg/reservation/evaluator.go @@ -0,0 +1,453 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package reservation + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "maps" + "math" + "regexp" + "slices" + + "github.com/google/cel-go/cel" + "github.com/google/cel-go/common/functions" + "github.com/google/cel-go/common/types" + "github.com/google/cel-go/common/types/ref" + structuralschema "k8s.io/apiextensions-apiserver/pkg/apiserver/schema" + "k8s.io/apimachinery/pkg/api/resource" + apiservercel "k8s.io/apiserver/pkg/cel" + "k8s.io/apiserver/pkg/cel/library" + + "github.com/cozystack/cozystack/pkg/apis/apps/defaulting" +) + +// ErrNoDeclaration is returned for a kind whose definition declares neither +// its consumption nor an exemption. Such a kind is refused rather than +// treated as free: for an out-of-tree catalog an omitted declaration would +// otherwise be a quota-escalation vector no in-tree test can reach. +var ErrNoDeclaration = errors.New("the application definition declares neither its consumption nor an exemption") + +const ( + valuesVar = "values" + instancetypeFunc = "instanceType" + presetFunc = "preset" + + // costLimit bounds one expression. Declarations are platform-authored, + // but they run on the admission path, so a runaway comprehension must + // fail rather than stall a write. + costLimit = 1_000_000 +) + +var bindingName = regexp.MustCompile(`^[a-zA-Z_][a-zA-Z0-9_]*$`) + +// Evaluator computes the reservation of one kind's applications. It is built +// once per declaration and is safe for concurrent use. +type Evaluator struct { + exempt bool + schema *structuralschema.Structural + env *cel.Env + lets []compiled + resources []compiled +} + +type compiled struct { + name string + ast *cel.Ast +} + +// Compile checks decl and prepares it for evaluation. schema is the kind's +// structural OpenAPI schema; stored values are defaulted against it before +// any expression reads them, because the aggregated apiserver persists a +// tenant's spec without its defaults. A nil decl yields ErrNoDeclaration. +func Compile(decl *Declaration, schema *structuralschema.Structural) (*Evaluator, error) { + if decl == nil { + return nil, ErrNoDeclaration + } + if decl.Exempt { + if len(decl.Let) > 0 || len(decl.Resources) > 0 { + return nil, fmt.Errorf("an exempt declaration must not carry let or resources") + } + return &Evaluator{exempt: true}, nil + } + if len(decl.Resources) == 0 { + return nil, ErrNoDeclaration + } + + opts := []cel.EnvOption{ + cel.Variable(valuesVar, cel.DynType), + library.Quantity(), + cel.Function(instancetypeFunc, + cel.Overload("instanceType_string", []*cel.Type{cel.StringType}, cel.MapType(cel.StringType, cel.DynType), + cel.LateFunctionBinding())), + cel.Function(presetFunc, + cel.Overload("preset_string", []*cel.Type{cel.StringType}, cel.MapType(cel.StringType, cel.DynType), + cel.LateFunctionBinding())), + cel.Function("quantity", + cel.Overload("quantity_int", []*cel.Type{cel.IntType}, apiservercel.QuantityType, + cel.UnaryBinding(func(v ref.Val) ref.Val { + return apiservercel.Quantity{Quantity: resource.NewQuantity(int64(v.(types.Int)), resource.DecimalSI)} + }))), + // The standard _*_ is a singleton dispatching on the left operand's + // traits, so it cannot take a quantity overload; scaling is a method. + cel.Function("mul", + cel.MemberOverload("quantity_mul_int", []*cel.Type{apiservercel.QuantityType, cel.IntType}, apiservercel.QuantityType, + cel.BinaryBinding(mulQuantity))), + } + seen := map[string]bool{valuesVar: true} + for _, b := range decl.Let { + if !bindingName.MatchString(b.Name) { + return nil, fmt.Errorf("let %q: not a valid identifier", b.Name) + } + if seen[b.Name] { + return nil, fmt.Errorf("let %q: name already bound", b.Name) + } + seen[b.Name] = true + opts = append(opts, cel.Variable(b.Name, cel.DynType)) + } + env, err := cel.NewEnv(opts...) + if err != nil { + return nil, fmt.Errorf("build CEL environment: %w", err) + } + + ev := &Evaluator{schema: schema, env: env} + for _, b := range decl.Let { + ast, iss := env.Compile(b.Expression) + if iss.Err() != nil { + return nil, fmt.Errorf("let %q: %w", b.Name, iss.Err()) + } + ev.lets = append(ev.lets, compiled{name: b.Name, ast: ast}) + } + for _, key := range slices.Sorted(maps.Keys(decl.Resources)) { + if key == "" { + return nil, fmt.Errorf("resources: empty quota key") + } + ast, iss := env.Compile(decl.Resources[key]) + if iss.Err() != nil { + return nil, fmt.Errorf("resources[%q]: %w", key, iss.Err()) + } + ev.resources = append(ev.resources, compiled{name: key, ast: ast}) + } + return ev, nil +} + +// Exempt reports whether the kind declared that it consumes nothing. +func (e *Evaluator) Exempt() bool { return e.exempt } + +// EvaluateJSON evaluates the reservation of an application from its values +// as stored, a JSON object. +func (e *Evaluator) EvaluateJSON(raw []byte, env Environment) (ResourceList, error) { + values := map[string]any{} + if len(bytes.TrimSpace(raw)) > 0 { + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.UseNumber() + var v any + if err := dec.Decode(&v); err != nil { + return nil, fmt.Errorf("decode values: %w", err) + } + m, ok := v.(map[string]any) + if !ok && v != nil { + return nil, fmt.Errorf("decode values: expected an object, got %T", v) + } + if m != nil { + values = m + } + } + return e.Evaluate(values, env) +} + +// Evaluate returns the reservation of an application with the given values. +// The values are defaulted first and are not modified. Anything that cannot +// be resolved — an unknown instance type, a value of the wrong type, a key an +// expression reads without guarding it with has() — is an error, never a +// smaller figure. Keys that evaluate to zero are omitted. +func (e *Evaluator) Evaluate(values map[string]any, env Environment) (ResourceList, error) { + if e.exempt { + return ResourceList{}, nil + } + if env == nil { + return nil, fmt.Errorf("no environment to resolve instance types and presets against") + } + + defaulted := normalizeNumbers(values).(map[string]any) + // Defaulting replaces a value of the wrong type with the schema's + // default, which would turn a malformed spec into a smaller one. + if err := checkTypes(defaulted, e.schema, ""); err != nil { + return nil, err + } + if err := defaulting.Apply(&defaulted, e.schema); err != nil { + return nil, fmt.Errorf("default values: %w", err) + } + + // The environment differs per call, so instanceType and preset are + // declared with a late binding and bound here, per program; cel.Functions + // is deprecated for declaring functions but is still how cel-go supplies + // a late binding. + bindings := cel.Functions( + &functions.Overload{Operator: "instanceType_string", Unary: lookup(env.Instancetype, instancetypeZeroKeys)}, + &functions.Overload{Operator: "preset_string", Unary: lookup(env.Preset, nil)}, + ) + activation := map[string]any{valuesVar: defaulted} + + for _, c := range e.lets { + out, err := e.run(c, bindings, activation) + if err != nil { + return nil, fmt.Errorf("let %q: %w", c.name, err) + } + activation[c.name] = out + } + + result := ResourceList{} + for _, c := range e.resources { + out, err := e.run(c, bindings, activation) + if err != nil { + return nil, fmt.Errorf("resources[%q]: %w", c.name, err) + } + q, err := toQuantity(out) + if err != nil { + return nil, fmt.Errorf("resources[%q]: %w", c.name, err) + } + if q.Sign() < 0 { + return nil, fmt.Errorf("resources[%q]: negative reservation %s", c.name, q.String()) + } + if q.IsZero() { + continue + } + result[c.name] = q + } + return result, nil +} + +func (e *Evaluator) run(c compiled, bindings cel.ProgramOption, activation map[string]any) (ref.Val, error) { + prg, err := e.env.Program(c.ast, bindings, cel.CostLimit(costLimit)) + if err != nil { + return nil, err + } + out, _, err := prg.Eval(activation) + if err != nil { + return nil, err + } + return out, nil +} + +// instancetypeZeroKeys are present, at zero, in every instanceType() result, +// so that a declaration can read any class of a type without guarding it. +var instancetypeZeroKeys = []string{ + KeyCPU, KeyDedicatedCPU, KeyMemory, KeyOvercommittedMemory, + KeyHugepagesPrefix + "2Mi", KeyHugepagesPrefix + "1Gi", +} + +func lookup(resolve func(string) (ResourceList, error), zeroKeys []string) functions.UnaryOp { + return func(arg ref.Val) ref.Val { + name, ok := arg.(types.String) + if !ok { + return types.MaybeNoSuchOverloadErr(arg) + } + rl, err := resolve(string(name)) + if err != nil { + return types.WrapErr(err) + } + out := make(map[string]any, len(rl)+len(zeroKeys)) + for _, k := range zeroKeys { + out[k] = apiservercel.Quantity{Quantity: resource.NewQuantity(0, resource.DecimalSI)} + } + for k, v := range rl { + q := v.DeepCopy() + out[k] = apiservercel.Quantity{Quantity: &q} + } + return types.DefaultTypeAdapter.NativeToValue(out) + } +} + +func mulQuantity(q, n ref.Val) ref.Val { + qq, ok := q.(apiservercel.Quantity) + if !ok { + return types.MaybeNoSuchOverloadErr(q) + } + nn, ok := n.(types.Int) + if !ok { + return types.MaybeNoSuchOverloadErr(n) + } + out := qq.Quantity.DeepCopy() + if !out.Mul(int64(nn)) { + return types.NewErr("quantity %s * %d overflows", qq.Quantity.String(), int64(nn)) + } + return apiservercel.Quantity{Quantity: &out} +} + +func toQuantity(v ref.Val) (resource.Quantity, error) { + switch t := v.(type) { + case apiservercel.Quantity: + return t.Quantity.DeepCopy(), nil + case types.Int: + return *resource.NewQuantity(int64(t), resource.DecimalSI), nil + case types.Uint: + if uint64(t) > math.MaxInt64 { + return resource.Quantity{}, fmt.Errorf("%d overflows a quantity", uint64(t)) + } + return *resource.NewQuantity(int64(t), resource.DecimalSI), nil + case types.Double: + f := float64(t) + if math.IsNaN(f) || math.IsInf(f, 0) { + return resource.Quantity{}, fmt.Errorf("not a finite number: %v", f) + } + // Rounding up would carry a negative figure smaller than a milli to + // zero, past the sign check. + if f < 0 { + return resource.Quantity{}, fmt.Errorf("negative reservation %v", f) + } + // float64(math.MaxInt64) rounds up to 2^63, so the bound is + // exclusive; past it the conversion would saturate into a smaller + // figure. + milli := math.Ceil(f * 1000) + if milli >= math.MaxInt64 { + return resource.Quantity{}, fmt.Errorf("%v overflows a quantity", f) + } + return *resource.NewMilliQuantity(int64(milli), resource.DecimalSI), nil + case types.String: + q, err := resource.ParseQuantity(string(t)) + if err != nil { + return resource.Quantity{}, fmt.Errorf("parse %q as a quantity: %w", string(t), err) + } + return q, nil + default: + return resource.Quantity{}, fmt.Errorf("expression returned %s, expected an int, a double, a string or a quantity", v.Type().TypeName()) + } +} + +// checkTypes refuses a value whose JSON type the schema does not allow. It +// runs on normalized values: an integer is an int64, any other number a +// float64. A null is left to defaulting, which treats it as absent. +func checkTypes(v any, s *structuralschema.Structural, path string) error { + if s == nil || v == nil { + return nil + } + if s.XIntOrString { + switch v.(type) { + case int64, string: + return nil + } + return typeError(path, "an integer or a string", v) + } + // An untyped schema is defaulted as the shape its properties or items + // imply, so it is checked as that shape too. + effType := s.Type + if effType == "" { + switch { + case len(s.Properties) > 0 || (s.AdditionalProperties != nil && s.AdditionalProperties.Structural != nil): + effType = "object" + case s.Items != nil: + effType = "array" + } + } + switch effType { + case "object": + m, ok := v.(map[string]any) + if !ok { + return typeError(path, "an object", v) + } + for k, vv := range m { + ps, known := s.Properties[k] + switch { + case known: + if err := checkTypes(vv, &ps, path+"."+k); err != nil { + return err + } + case s.AdditionalProperties != nil && s.AdditionalProperties.Structural != nil: + if err := checkTypes(vv, s.AdditionalProperties.Structural, path+"."+k); err != nil { + return err + } + } + } + case "array": + sl, ok := v.([]any) + if !ok { + return typeError(path, "an array", v) + } + for i, vv := range sl { + if err := checkTypes(vv, s.Items, fmt.Sprintf("%s[%d]", path, i)); err != nil { + return err + } + } + case "string": + if _, ok := v.(string); !ok { + return typeError(path, "a string", v) + } + case "integer": + if _, ok := v.(int64); !ok { + return typeError(path, "an integer", v) + } + case "number": + switch v.(type) { + case int64, float64: + default: + return typeError(path, "a number", v) + } + case "boolean": + if _, ok := v.(bool); !ok { + return typeError(path, "a boolean", v) + } + } + return nil +} + +func typeError(path, want string, got any) error { + if path == "" { + path = "." + } + return fmt.Errorf("values%s: expected %s, got %T", path, want, got) +} + +// normalizeNumbers returns a copy of v in which every JSON number is an int64 +// when it is integral and a float64 otherwise. Without it every number in a +// stored spec reaches CEL as a double, and replicas * quantity(...) would not +// type-check at runtime. +func normalizeNumbers(v any) any { + switch t := v.(type) { + case map[string]any: + out := make(map[string]any, len(t)) + for k, vv := range t { + out[k] = normalizeNumbers(vv) + } + return out + case []any: + out := make([]any, len(t)) + for i, vv := range t { + out[i] = normalizeNumbers(vv) + } + return out + case json.Number: + if i, err := t.Int64(); err == nil { + return i + } + // 2.0 and 2e0 are integers to the schema validator; spelled that + // way they fail Int64 and are classified as a float64 would be. + f, _ := t.Float64() + return normalizeNumbers(f) + case float64: + if t == math.Trunc(t) && math.Abs(t) < 1<<53 { + return int64(t) + } + return t + case int: + return int64(t) + default: + return v + } +} diff --git a/pkg/reservation/evaluator_test.go b/pkg/reservation/evaluator_test.go new file mode 100644 index 000000000..7aa7ef9a7 --- /dev/null +++ b/pkg/reservation/evaluator_test.go @@ -0,0 +1,525 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package reservation + +import ( + "errors" + "os" + "sync" + "testing" + + structuralschema "k8s.io/apiextensions-apiserver/pkg/apiserver/schema" + "k8s.io/apimachinery/pkg/api/resource" + "sigs.k8s.io/yaml" + + "github.com/cozystack/cozystack/pkg/apis/apps/defaulting" +) + +// The declarations below are test fixtures in the shape a kind would carry +// on its ApplicationDefinition. They are evaluated against the kinds' real +// shipped schemas so that defaulting is exercised with the defaults tenants +// actually get. + +var postgresDecl = &Declaration{ + Let: []Binding{{ + Name: "count", + // The autoscaler writes the Cluster's scale subresource, never the + // application, so the ceiling is what has to be reserved. + Expression: `values.autoscaling.enabled && values.autoscaling.maxReplicas > values.replicas + ? values.autoscaling.maxReplicas : values.replicas`, + }}, + Resources: map[string]string{ + // cozy-lib fills a preset per key, so each key is overridden on its own. + "cpu": `(has(values.resources.cpu) ? quantity(values.resources.cpu) : preset(values.resourcesPreset).cpu).mul(count)`, + "memory": `(has(values.resources.memory) ? quantity(values.resources.memory) : preset(values.resourcesPreset).memory).mul(count)`, + "storage": `quantity(values.size).mul(count)`, + }, +} + +var vmInstanceDecl = &Declaration{ + Let: []Binding{ + // vm-instance treats resources as all-or-nothing, and cpu is per socket. + {Name: "sized", Expression: `has(values.resources.cpu) && has(values.resources.sockets) && has(values.resources.memory)`}, + {Name: "it", Expression: `instanceType(values.instanceType)`}, + }, + Resources: map[string]string{ + "cpu": `sized ? int(values.resources.cpu) * int(values.resources.sockets) : it.cpu`, + "dedicated-cpu": `sized ? 0 : it['dedicated-cpu']`, + "memory": `sized ? quantity(values.resources.memory) : it.memory`, + "hugepages-2Mi": `sized ? 0 : it['hugepages-2Mi']`, + "hugepages-1Gi": `sized ? 0 : it['hugepages-1Gi']`, + "overcommitted-memory": `sized ? 0 : it['overcommitted-memory']`, + "services.loadbalancers": `values.external ? 1 : 0`, + }, +} + +var kubernetesNodesDecl = &Declaration{ + Let: []Binding{{Name: "it", Expression: `instanceType(values.instanceType)`}}, + Resources: map[string]string{ + "cpu": `it.cpu.mul(values.maxReplicas)`, + "memory": `it.memory.mul(values.maxReplicas)`, + }, +} + +func shippedSchema(t *testing.T, kind string) *structuralschema.Structural { + t.Helper() + raw, err := os.ReadFile("../../packages/system/" + kind + "-rd/cozyrds/" + kind + ".yaml") + if err != nil { + t.Fatal(err) + } + var def struct { + Spec struct { + Application struct { + OpenAPISchema string `json:"openAPISchema"` + } `json:"application"` + } `json:"spec"` + } + if err := yaml.Unmarshal(raw, &def); err != nil { + t.Fatal(err) + } + s, err := defaulting.BuildSchema(def.Spec.Application.OpenAPISchema) + if err != nil || s == nil { + t.Fatalf("%s schema: %v", kind, err) + } + return s +} + +func testEnvironment(t *testing.T) Environment { + t.Helper() + env, err := NewStaticEnvironment(shippedInstancetypes(t), map[string]ResourceList{ + "t1.nano": {"cpu": resource.MustParse("250m"), "memory": resource.MustParse("128Mi"), "ephemeral-storage": resource.MustParse("2Gi")}, + "t1.micro": {"cpu": resource.MustParse("500m"), "memory": resource.MustParse("256Mi"), "ephemeral-storage": resource.MustParse("2Gi")}, + }) + if err != nil { + t.Fatal(err) + } + return env +} + +func mustCompile(t *testing.T, decl *Declaration, kind string) *Evaluator { + t.Helper() + ev, err := Compile(decl, shippedSchema(t, kind)) + if err != nil { + t.Fatalf("compile %s: %v", kind, err) + } + return ev +} + +func rl(kv ...string) ResourceList { + out := ResourceList{} + for i := 0; i < len(kv); i += 2 { + out[kv[i]] = resource.MustParse(kv[i+1]) + } + return out +} + +func TestEvaluateKinds(t *testing.T) { + env := testEnvironment(t) + postgres := mustCompile(t, postgresDecl, "postgres") + vm := mustCompile(t, vmInstanceDecl, "vm-instance") + + cases := []struct { + name string + ev *Evaluator + values string + want ResourceList + }{ + { + // Two replicas of t1.micro with 10Gi each, all from schema defaults. + name: "postgres defaults", ev: postgres, values: `{}`, + want: rl("cpu", "1", "memory", "512Mi", "storage", "20Gi"), + }, + { + // The per-key fill: cpu from resources, memory from the preset. + name: "postgres partial resources on a preset", ev: postgres, + values: `{"replicas": 1, "resourcesPreset": "t1.nano", "resources": {"cpu": "1"}}`, + want: rl("cpu", "1", "memory", "128Mi", "storage", "10Gi"), + }, + { + name: "postgres integer cpu", ev: postgres, + values: `{"replicas": 1, "resources": {"cpu": 2, "memory": "1Gi"}}`, + want: rl("cpu", "2", "memory", "1Gi", "storage", "10Gi"), + }, + { + name: "postgres autoscaling reserves the ceiling", ev: postgres, + values: `{"autoscaling": {"enabled": true}}`, + want: rl("cpu", "3", "memory", "1536Mi", "storage", "60Gi"), + }, + { + name: "postgres autoscaling off reserves replicas", ev: postgres, + values: `{"autoscaling": {"enabled": false, "maxReplicas": 6}}`, + want: rl("cpu", "1", "memory", "512Mi", "storage", "20Gi"), + }, + { + name: "vm-instance default instance type", ev: vm, values: `{}`, + want: rl("cpu", "1", "memory", "4Gi"), + }, + { + name: "vm-instance dedicated type pins the emulator core", ev: vm, + values: `{"instanceType": "cx1.2xlarge"}`, + want: rl("dedicated-cpu", "9", "hugepages-2Mi", "16Gi"), + }, + { + name: "vm-instance overcommitted type", ev: vm, + values: `{"instanceType": "o1.large"}`, + want: rl("cpu", "2", "overcommitted-memory", "8Gi"), + }, + { + // cpu is cores per socket. + name: "vm-instance complete resources block", ev: vm, + values: `{"instanceType": "u1.small", "resources": {"cpu": 4, "sockets": 2, "memory": "8Gi"}}`, + want: rl("cpu", "8", "memory", "8Gi"), + }, + { + // Without sockets the chart sizes the VM from its instance type. + name: "vm-instance cpu without sockets", ev: vm, + values: `{"instanceType": "u1.small", "resources": {"cpu": 4, "memory": "8Gi"}}`, + want: rl("cpu", "1", "memory", "2Gi"), + }, + { + name: "vm-instance external", ev: vm, + values: `{"instanceType": "u1.small", "external": true}`, + want: rl("cpu", "1", "memory", "2Gi", "services.loadbalancers", "1"), + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, err := tc.ev.EvaluateJSON([]byte(tc.values), env) + if err != nil { + t.Fatal(err) + } + assertList(t, tc.name, got, tc.want) + }) + } +} + +// A release a platform chart renders, such as ComputePlane's default md0 +// pool, carries only the keys the chart wrote. It has to reserve exactly what +// the same pool with its defaults written out reserves. +func TestEvaluateDefaultsStoredValues(t *testing.T) { + env := testEnvironment(t) + ev := mustCompile(t, kubernetesNodesDecl, "kubernetes-nodes") + + sparse, err := ev.EvaluateJSON([]byte(`{"roles": ["ingress-nginx"], "minReplicas": 0}`), env) + if err != nil { + t.Fatal(err) + } + explicit, err := ev.EvaluateJSON([]byte(`{"roles": ["ingress-nginx"], "minReplicas": 0, "maxReplicas": 10, "instanceType": "u1.medium"}`), env) + if err != nil { + t.Fatal(err) + } + assertList(t, "sparse vs explicit", sparse, explicit) + assertList(t, "sparse", sparse, rl("cpu", "10", "memory", "40Gi")) +} + +func TestEvaluateFailsClosed(t *testing.T) { + env := testEnvironment(t) + postgres := mustCompile(t, postgresDecl, "postgres") + vm := mustCompile(t, vmInstanceDecl, "vm-instance") + + if _, err := vm.EvaluateJSON([]byte(`{"instanceType": "u9.imaginary"}`), env); err == nil { + t.Error("unknown instance type: expected an error") + } else if !errors.Is(err, ErrNotFound) { + t.Errorf("unknown instance type: expected ErrNotFound, got %v", err) + } + if _, err := postgres.EvaluateJSON([]byte(`{"resourcesPreset": "t9.huge"}`), env); !errors.Is(err, ErrNotFound) { + t.Errorf("unknown preset: expected ErrNotFound, got %v", err) + } + if _, err := postgres.EvaluateJSON([]byte(`{"replicas": "two"}`), env); err == nil { + t.Error("wrong value type: expected an error") + } + if _, err := postgres.EvaluateJSON([]byte(`{"size": "lots"}`), env); err == nil { + t.Error("unparseable quantity: expected an error") + } + if _, err := postgres.EvaluateJSON([]byte(`[1, 2]`), env); err == nil { + t.Error("non-object values: expected an error") + } + if _, err := postgres.EvaluateJSON([]byte(`{}`), nil); err == nil { + t.Error("no environment: expected an error") + } + + unguarded, err := Compile(&Declaration{Resources: map[string]string{"cpu": `quantity(values.notInTheSchema)`}}, shippedSchema(t, "postgres")) + if err != nil { + t.Fatal(err) + } + if _, err := unguarded.EvaluateJSON([]byte(`{}`), env); err == nil { + t.Error("unguarded absent key: expected an error, not a zero") + } + + negative, err := Compile(&Declaration{Resources: map[string]string{"cpu": `-1`}}, nil) + if err != nil { + t.Fatal(err) + } + if _, err := negative.EvaluateJSON(nil, env); err == nil { + t.Error("negative reservation: expected an error") + } + + // A negative double too small to survive rounding to millis is still + // negative. + tiny, err := Compile(&Declaration{Resources: map[string]string{"cpu": `-0.0001`}}, nil) + if err != nil { + t.Fatal(err) + } + if _, err := tiny.EvaluateJSON(nil, env); err == nil { + t.Error("small negative reservation: expected an error, not a zero") + } + + // A double past the range of an int64 in millis must not be clamped into + // a smaller figure. + for _, expr := range []string{`1e100`, `1e308`, `9.3e15`} { + huge, err := Compile(&Declaration{Resources: map[string]string{"cpu": expr}}, nil) + if err != nil { + t.Fatal(err) + } + if got, err := huge.EvaluateJSON(nil, env); err == nil { + t.Errorf("%s: expected an error, got %v", expr, got) + } + } + finite, err := Compile(&Declaration{Resources: map[string]string{"cpu": `1e3`, "memory": `0.25`}}, nil) + if err != nil { + t.Fatal(err) + } + got, err := finite.EvaluateJSON(nil, env) + if err != nil { + t.Fatal(err) + } + assertList(t, "finite doubles", got, rl("cpu", "1k", "memory", "250m")) +} + +func TestEvaluateRejectsValuesOfTheWrongType(t *testing.T) { + env := testEnvironment(t) + // Defaulting replaces a value whose type does not match the schema with + // the schema's own default, so the type is checked before it runs: + // otherwise this guard would read an erased resources object and + // reserve its fallback. + ev, err := Compile(&Declaration{Resources: map[string]string{ + "cpu": `has(values.resources.cpu) ? values.resources.cpu : 1`, + }}, shippedSchema(t, "postgres")) + if err != nil { + t.Fatal(err) + } + for _, raw := range []string{ + `{"resources": "invalid"}`, + `{"resources": [1]}`, + `{"resources": true}`, + `{"resources": {"cpu": true}}`, + `{"replicas": 2.5}`, + `{"replicas": "2"}`, + `{"external": "yes"}`, + `{"postgresql": {"sharedPreloadLibraries": "pg_prewarm"}}`, + `{"users": {"app": "admin"}}`, + } { + if got, err := ev.EvaluateJSON([]byte(raw), env); err == nil { + t.Errorf("%s: expected an error, got %v", raw, got) + } + } + for raw, want := range map[string]ResourceList{ + `{}`: rl("cpu", "1"), + `{"resources": {}}`: rl("cpu", "1"), + `{"resources": {"cpu": 2}}`: rl("cpu", "2"), + `{"resources": {"cpu": "500m"}}`: rl("cpu", "500m"), + `{"resources": null, "replicas": 3}`: rl("cpu", "1"), + `{"users": {"app": {"replication": true}}}`: rl("cpu", "1"), + } { + got, err := ev.EvaluateJSON([]byte(raw), env) + if err != nil { + t.Errorf("%s: %v", raw, err) + continue + } + assertList(t, raw, got, want) + } +} + +func TestEvaluateAcceptsIntegralSpellings(t *testing.T) { + env := testEnvironment(t) + // The Kubernetes schema validator accepts an integral number however it + // is spelled, so the evaluator must too. + ev, err := Compile(&Declaration{Resources: map[string]string{"cpu": `values.replicas`}}, shippedSchema(t, "postgres")) + if err != nil { + t.Fatal(err) + } + for _, raw := range []string{`{"replicas": 2}`, `{"replicas": 2.0}`, `{"replicas": 2e0}`, `{"replicas": 0.2e1}`} { + got, err := ev.EvaluateJSON([]byte(raw), env) + if err != nil { + t.Errorf("%s: %v", raw, err) + continue + } + assertList(t, raw, got, rl("cpu", "2")) + } + for _, raw := range []string{`{"replicas": 2.5}`, `{"replicas": 25e-1}`} { + if got, err := ev.EvaluateJSON([]byte(raw), env); err == nil { + t.Errorf("%s: expected an error, got %v", raw, got) + } + } +} + +func TestEvaluateChecksUntypedSchemasByShape(t *testing.T) { + env := testEnvironment(t) + // BuildSchema does not require a structural schema to be typed, and + // defaulting treats an untyped node with properties as an object and one + // with items as an array, so the type check must as well. + s, err := defaulting.BuildSchema(`{"type":"object","properties":{ + "resources":{"properties":{"cpu":{"type":"integer"}}}, + "disks":{"items":{"type":"string"}}, + "labels":{"additionalProperties":{"type":"string"}}}}`) + if err != nil { + t.Fatal(err) + } + ev, err := Compile(&Declaration{Resources: map[string]string{ + "cpu": `has(values.resources) && has(values.resources.cpu) ? values.resources.cpu : 1`, + }}, s) + if err != nil { + t.Fatal(err) + } + for _, raw := range []string{`{"resources": "invalid"}`, `{"resources": {"cpu": "2"}}`, `{"disks": "a"}`, `{"disks": [1]}`, `{"labels": "a"}`, `{"labels": {"a": 1}}`} { + if got, err := ev.EvaluateJSON([]byte(raw), env); err == nil { + t.Errorf("%s: expected an error, got %v", raw, got) + } + } + for raw, want := range map[string]ResourceList{ + `{"resources": {"cpu": 2}}`: rl("cpu", "2"), + `{"disks": ["a"]}`: rl("cpu", "1"), + `{"labels": {"a": "b"}}`: rl("cpu", "1"), + } { + got, err := ev.EvaluateJSON([]byte(raw), env) + if err != nil { + t.Errorf("%s: %v", raw, err) + continue + } + assertList(t, raw, got, want) + } +} + +func TestCompile(t *testing.T) { + if _, err := Compile(nil, nil); !errors.Is(err, ErrNoDeclaration) { + t.Errorf("nil declaration: got %v, want ErrNoDeclaration", err) + } + if _, err := Compile(&Declaration{}, nil); !errors.Is(err, ErrNoDeclaration) { + t.Errorf("empty declaration: got %v, want ErrNoDeclaration", err) + } + + exempt, err := Compile(&Declaration{Exempt: true}, nil) + if err != nil { + t.Fatal(err) + } + got, err := exempt.EvaluateJSON([]byte(`{"anything": 1}`), nil) + if err != nil || len(got) != 0 || !exempt.Exempt() { + t.Errorf("exempt: got %v, %v", got, err) + } + + invalid := map[string]*Declaration{ + "exempt with resources": {Exempt: true, Resources: map[string]string{"cpu": "1"}}, + "syntax error": {Resources: map[string]string{"cpu": "values.("}}, + "undeclared variable": {Resources: map[string]string{"cpu": "count"}}, + "empty key": {Resources: map[string]string{"": "1"}}, + "bad let name": {Let: []Binding{{Name: "a-b", Expression: "1"}}, Resources: map[string]string{"cpu": "1"}}, + "let shadows values": {Let: []Binding{{Name: "values", Expression: "1"}}, Resources: map[string]string{"cpu": "1"}}, + "duplicate let": {Let: []Binding{{Name: "a", Expression: "1"}, {Name: "a", Expression: "2"}}, Resources: map[string]string{"cpu": "a"}}, + } + for name, decl := range invalid { + if _, err := Compile(decl, nil); err == nil { + t.Errorf("%s: expected a compile error", name) + } + } +} + +func TestEvaluateIsConcurrencySafe(t *testing.T) { + env := testEnvironment(t) + vm := mustCompile(t, vmInstanceDecl, "vm-instance") + catalog := shippedInstancetypes(t) + var wg sync.WaitGroup + for _, it := range []string{"u1.small", "cx1.2xlarge", "o1.large", "u1.medium"} { + want, err := ClassifyInstancetype(findInstancetype(t, catalog, it)) + if err != nil { + t.Fatal(err) + } + for range 8 { + wg.Add(1) + go func() { + defer wg.Done() + got, err := vm.EvaluateJSON([]byte(`{"instanceType": "`+it+`"}`), env) + if err != nil { + t.Error(err) + return + } + assertList(t, it, got, want) + }() + } + } + wg.Wait() +} + +func findInstancetype(t *testing.T, catalog []map[string]any, name string) map[string]any { + t.Helper() + for _, obj := range catalog { + if obj["metadata"].(map[string]any)["name"] == name { + return obj + } + } + t.Fatalf("%s is not shipped", name) + return nil +} + +func TestAggregate(t *testing.T) { + env := testEnvironment(t) + evaluators := map[string]*Evaluator{ + "VMInstance": mustCompile(t, vmInstanceDecl, "vm-instance"), + "Postgres": mustCompile(t, postgresDecl, "postgres"), + } + lookup := func(kind, group string) (*Evaluator, error) { + if group != "apps.cozystack.io" { + return nil, errors.New("unknown group") + } + if ev, ok := evaluators[kind]; ok { + return ev, nil + } + return nil, ErrNoDeclaration + } + apps := []Application{ + {Namespace: "tenant-a", Name: "vm1", Kind: "VMInstance", Group: "apps.cozystack.io", Values: []byte(`{"instanceType": "u1.small"}`)}, + {Namespace: "tenant-a", Name: "vm2", Kind: "VMInstance", Group: "apps.cozystack.io", Values: []byte(`{"instanceType": "u1.small", "external": true}`)}, + {Namespace: "tenant-a", Name: "db", Kind: "Postgres", Group: "apps.cozystack.io", Values: []byte(`{}`)}, + {Namespace: "tenant-b", Name: "vm3", Kind: "VMInstance", Group: "apps.cozystack.io", Values: []byte(`{"instanceType": "u1.small"}`)}, + {Namespace: "tenant-b", Name: "broken", Kind: "VMInstance", Group: "apps.cozystack.io", Values: []byte(`{"instanceType": "gone"}`)}, + {Namespace: "tenant-b", Name: "undeclared", Kind: "Redis", Group: "apps.cozystack.io", Values: []byte(`{}`)}, + } + sums, failures := Aggregate(apps, lookup, env) + assertList(t, "tenant-a", sums["tenant-a"], rl("cpu", "3", "memory", "4608Mi", "storage", "20Gi", "services.loadbalancers", "1")) + assertList(t, "tenant-b", sums["tenant-b"], rl("cpu", "1", "memory", "2Gi")) + if len(failures) != 2 { + t.Fatalf("expected 2 failures, got %v", failures) + } + if failures[0].Application.Name != "broken" || !errors.Is(failures[0].Err, ErrNotFound) { + t.Errorf("first failure: %v", failures[0]) + } + if failures[1].Application.Name != "undeclared" || !errors.Is(failures[1].Err, ErrNoDeclaration) { + t.Errorf("second failure: %v", failures[1]) + } +} + +func TestResourceListArithmetic(t *testing.T) { + a := rl("cpu", "2", "memory", "4Gi") + b := a.DeepCopy() + b.Add(rl("cpu", "1", "storage", "10Gi")) + assertList(t, "copy unaffected", a, rl("cpu", "2", "memory", "4Gi")) + assertList(t, "add", b, rl("cpu", "3", "memory", "4Gi", "storage", "10Gi")) + b.Sub(rl("cpu", "4", "services.loadbalancers", "1")) + assertList(t, "sub", b, rl("cpu", "-1", "memory", "4Gi", "storage", "10Gi", "services.loadbalancers", "-1")) + if keys := b.Keys(); len(keys) != 4 || keys[0] != "cpu" || keys[3] != "storage" { + t.Errorf("keys not sorted: %v", keys) + } +} diff --git a/pkg/reservation/instancetype.go b/pkg/reservation/instancetype.go new file mode 100644 index 000000000..b4373164f --- /dev/null +++ b/pkg/reservation/instancetype.go @@ -0,0 +1,148 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package reservation + +import ( + "fmt" + + "k8s.io/apimachinery/pkg/api/resource" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" +) + +// EmulatorThreadCompleteToEvenParityAnnotation, set in an instance type's +// spec.annotations, makes KubeVirt give the isolated emulator thread one or +// two cores, whichever makes the pinned total even. KubeVirt copies those +// annotations onto the VMI and tests the key for presence, so its value is +// irrelevant. +const EmulatorThreadCompleteToEvenParityAnnotation = "alpha.kubevirt.io/EmulatorThreadCompleteToEvenParity" + +// ClassifyInstancetype resolves a VirtualMachineClusterInstancetype into the +// class-keyed reservation of one VM of that type. +// +// The class is read from the type's spec, never from its name, so a type an +// operator adds classifies itself. Cores include the emulator and +// supplemental I/O threads a dedicated type pins, because those are fixed +// properties of the type a tenant can see before ordering; the launcher's +// memory overhead is computed by virt-controller at runtime and is platform +// margin, not part of the reservation. +func ClassifyInstancetype(obj map[string]any) (ResourceList, error) { + name, _, _ := unstructured.NestedString(obj, "metadata", "name") + + guestCPU, err := nestedInt(obj, "spec", "cpu", "guest") + if err != nil { + return nil, fmt.Errorf("instance type %q: spec.cpu.guest: %w", name, err) + } + if guestCPU <= 0 { + return nil, fmt.Errorf("instance type %q: spec.cpu.guest must be positive, got %d", name, guestCPU) + } + guestMemoryRaw, found, err := unstructured.NestedFieldNoCopy(obj, "spec", "memory", "guest") + if err != nil || !found { + return nil, fmt.Errorf("instance type %q: spec.memory.guest is required", name) + } + guestMemory, err := parseQuantity(guestMemoryRaw) + if err != nil { + return nil, fmt.Errorf("instance type %q: spec.memory.guest: %w", name, err) + } + + out := ResourceList{} + + dedicated, _, _ := unstructured.NestedBool(obj, "spec", "cpu", "dedicatedCPUPlacement") + if dedicated { + cores := guestCPU + if isolate, _, _ := unstructured.NestedBool(obj, "spec", "cpu", "isolateEmulatorThread"); isolate { + cores++ + annotations, _, _ := unstructured.NestedMap(obj, "spec", "annotations") + if _, parity := annotations[EmulatorThreadCompleteToEvenParityAnnotation]; parity && cores%2 == 1 { + cores++ + } + } + if policy, _, _ := unstructured.NestedString(obj, "spec", "ioThreadsPolicy"); policy == "supplementalPool" { + extra, err := nestedInt(obj, "spec", "ioThreads", "supplementalPoolThreadCount") + if err != nil { + return nil, fmt.Errorf("instance type %q: ioThreadsPolicy supplementalPool: spec.ioThreads.supplementalPoolThreadCount: %w", name, err) + } + cores += extra + } + out[KeyDedicatedCPU] = *resource.NewQuantity(cores, resource.DecimalSI) + } else { + out[KeyCPU] = *resource.NewQuantity(guestCPU, resource.DecimalSI) + } + + pageSize, hasHugepages, _ := unstructured.NestedString(obj, "spec", "memory", "hugepages", "pageSize") + overcommit, hasOvercommit, _ := unstructured.NestedFieldNoCopy(obj, "spec", "memory", "overcommitPercent") + overcommitted := hasOvercommit && overcommit != nil && fmt.Sprint(overcommit) != "0" + switch { + case hasHugepages && overcommitted: + return nil, fmt.Errorf("instance type %q: hugepages and overcommitPercent are mutually exclusive", name) + case hasHugepages: + if pageSize == "" { + return nil, fmt.Errorf("instance type %q: spec.memory.hugepages.pageSize is empty", name) + } + out[KeyHugepagesPrefix+pageSize] = guestMemory + case overcommitted: + // The launcher requests less than the guest here, but the tenant + // declared the guest figure, and the same page cannot be granted as + // both overcommitted and ordinary memory. + out[KeyOvercommittedMemory] = guestMemory + default: + out[KeyMemory] = guestMemory + } + return out, nil +} + +func nestedInt(obj map[string]any, fields ...string) (int64, error) { + v, found, err := unstructured.NestedFieldNoCopy(obj, fields...) + if err != nil { + return 0, err + } + if !found || v == nil { + return 0, fmt.Errorf("is required") + } + switch n := v.(type) { + case int64: + return n, nil + case int: + return int64(n), nil + case int32: + return int64(n), nil + case float64: + if n != float64(int64(n)) { + return 0, fmt.Errorf("must be an integer, got %v", n) + } + return int64(n), nil + default: + return 0, fmt.Errorf("must be an integer, got %T", v) + } +} + +func parseQuantity(v any) (resource.Quantity, error) { + switch q := v.(type) { + case string: + return resource.ParseQuantity(q) + case int64: + return *resource.NewQuantity(q, resource.BinarySI), nil + case int: + return *resource.NewQuantity(int64(q), resource.BinarySI), nil + case float64: + if q != float64(int64(q)) { + return resource.Quantity{}, fmt.Errorf("fractional byte count %v", q) + } + return *resource.NewQuantity(int64(q), resource.BinarySI), nil + default: + return resource.Quantity{}, fmt.Errorf("must be a quantity, got %T", v) + } +} diff --git a/pkg/reservation/instancetype_test.go b/pkg/reservation/instancetype_test.go new file mode 100644 index 000000000..0d072b548 --- /dev/null +++ b/pkg/reservation/instancetype_test.go @@ -0,0 +1,259 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package reservation + +import ( + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "k8s.io/apimachinery/pkg/api/resource" + utilyaml "k8s.io/apimachinery/pkg/util/yaml" +) + +// shippedInstancetypes decodes every VirtualMachineClusterInstancetype the +// platform ships, so that the classification is pinned against the real +// catalog rather than against hand-written fixtures. +func shippedInstancetypes(t *testing.T) []map[string]any { + t.Helper() + files, err := filepath.Glob("../../packages/system/kubevirt-instancetypes/templates/*.yaml") + if err != nil || len(files) == 0 { + t.Fatalf("no instance type templates found: %v", err) + } + var out []map[string]any + for _, f := range files { + fh, err := os.Open(f) + if err != nil { + t.Fatal(err) + } + dec := utilyaml.NewYAMLOrJSONDecoder(fh, 4096) + for { + var obj map[string]any + if err := dec.Decode(&obj); err != nil { + if errors.Is(err, io.EOF) { + break + } + t.Fatalf("%s: %v", f, err) + } + if obj["kind"] == "VirtualMachineClusterInstancetype" { + out = append(out, obj) + } + } + fh.Close() + } + return out +} + +func TestClassifyShippedInstancetypes(t *testing.T) { + // The expected class per series is written from what each series is, + // not derived from the spec, so a name-based rule and a spec-based rule + // disagreeing shows up here. m1 is shared CPU with hugepages, n1 a + // dedicated series with hugepages, o1 overcommits memory. + type class struct{ cpu, memory string } + want := map[string]class{ + "u1": {KeyCPU, KeyMemory}, + "gn1": {KeyCPU, KeyMemory}, + "m1": {KeyCPU, "hugepages"}, + "o1": {KeyCPU, KeyOvercommittedMemory}, + "d1": {KeyDedicatedCPU, KeyMemory}, + "cx1": {KeyDedicatedCPU, "hugepages"}, + "n1": {KeyDedicatedCPU, "hugepages"}, + "rt1": {KeyDedicatedCPU, "hugepages"}, + } + + types := shippedInstancetypes(t) + if len(types) < 60 { + t.Fatalf("expected the full shipped catalog, decoded only %d instance types", len(types)) + } + seen := map[string]bool{} + for _, obj := range types { + name := obj["metadata"].(map[string]any)["name"].(string) + series := strings.SplitN(name, ".", 2)[0] + seen[series] = true + w, ok := want[series] + if !ok { + t.Errorf("%s: series %q has no expected class; add it to this table", name, series) + continue + } + rl, err := ClassifyInstancetype(obj) + if err != nil { + t.Errorf("%s: %v", name, err) + continue + } + if len(rl) != 2 { + t.Errorf("%s: expected one CPU class and one memory class, got %v", name, rl) + } + if _, ok := rl[w.cpu]; !ok { + t.Errorf("%s: expected CPU class %q, got %v", name, w.cpu, rl) + } + if w.memory == "hugepages" { + found := false + for k := range rl { + found = found || strings.HasPrefix(k, KeyHugepagesPrefix) + } + if !found { + t.Errorf("%s: expected a hugepages class, got %v", name, rl) + } + } else if _, ok := rl[w.memory]; !ok { + t.Errorf("%s: expected memory class %q, got %v", name, w.memory, rl) + } + } + for series := range want { + if !seen[series] { + t.Errorf("series %q is no longer shipped; drop it from this table", series) + } + } +} + +func TestClassifyShippedFigures(t *testing.T) { + byName := map[string]map[string]any{} + for _, obj := range shippedInstancetypes(t) { + byName[obj["metadata"].(map[string]any)["name"].(string)] = obj + } + cases := []struct { + name string + want ResourceList + }{ + // The KubeVirt u1.small is 2Gi; the cozy-lib preset of the same name + // is 4Gi. The two catalogs share names, not figures. + {"u1.small", ResourceList{KeyCPU: resource.MustParse("1"), KeyMemory: resource.MustParse("2Gi")}}, + // Eight guest vCPUs on nine pinned cores: the isolated emulator + // thread takes a core nobody else can be given. + {"cx1.2xlarge", ResourceList{KeyDedicatedCPU: resource.MustParse("9"), KeyHugepagesPrefix + "2Mi": resource.MustParse("16Gi")}}, + {"d1.large", ResourceList{KeyDedicatedCPU: resource.MustParse("3"), KeyMemory: resource.MustParse("8Gi")}}, + {"m1.large", ResourceList{KeyCPU: resource.MustParse("2"), KeyHugepagesPrefix + "2Mi": resource.MustParse("16Gi")}}, + } + for _, tc := range cases { + obj, ok := byName[tc.name] + if !ok { + t.Fatalf("%s is not shipped", tc.name) + } + got, err := ClassifyInstancetype(obj) + if err != nil { + t.Fatalf("%s: %v", tc.name, err) + } + assertList(t, tc.name, got, tc.want) + } +} + +func TestClassifyEmulatorThreadParity(t *testing.T) { + // KubeVirt copies an instance type's spec.annotations onto the VMI and + // pins the extra parity core when the VMI carries the annotation, so the + // figure is a property of each type, not of the catalog. + dedicated := func(guest int64, annotations map[string]any) map[string]any { + spec := map[string]any{ + "cpu": map[string]any{"guest": guest, "dedicatedCPUPlacement": true, "isolateEmulatorThread": true}, + "memory": map[string]any{"guest": "1Gi"}, + } + if annotations != nil { + spec["annotations"] = annotations + } + return map[string]any{"metadata": map[string]any{"name": "test"}, "spec": spec} + } + parity := map[string]any{EmulatorThreadCompleteToEvenParityAnnotation: ""} + cases := []struct { + guest int64 + annotations map[string]any + want int64 + }{ + {guest: 8, want: 9}, + {guest: 8, annotations: parity, want: 10}, + {guest: 8, annotations: map[string]any{EmulatorThreadCompleteToEvenParityAnnotation: "false"}, want: 10}, + {guest: 8, annotations: map[string]any{"example.com/unrelated": ""}, want: 9}, + {guest: 3, want: 4}, + {guest: 3, annotations: parity, want: 4}, + } + for _, tc := range cases { + got, err := ClassifyInstancetype(dedicated(tc.guest, tc.annotations)) + if err != nil { + t.Fatal(err) + } + cores := got[KeyDedicatedCPU] + if cores.Value() != tc.want { + t.Errorf("guest %d annotations %v: got %s dedicated cores, want %d", tc.guest, tc.annotations, cores.String(), tc.want) + } + } + + // Only the type's own annotations reach the VMI; metadata.annotations + // describe the object and are not copied. + onMetadata := dedicated(8, nil) + onMetadata["metadata"].(map[string]any)["annotations"] = parity + got, err := ClassifyInstancetype(onMetadata) + if err != nil { + t.Fatal(err) + } + if cores := got[KeyDedicatedCPU]; cores.Value() != 9 { + t.Errorf("annotation on metadata: got %s dedicated cores, want 9", cores.String()) + } +} + +func TestClassifySupplementalIOThreads(t *testing.T) { + obj := map[string]any{ + "metadata": map[string]any{"name": "test"}, + "spec": map[string]any{ + "cpu": map[string]any{"guest": int64(4), "dedicatedCPUPlacement": true}, + "ioThreadsPolicy": "supplementalPool", + "ioThreads": map[string]any{"supplementalPoolThreadCount": int64(2)}, + "memory": map[string]any{"guest": "1Gi"}, + }, + } + got, err := ClassifyInstancetype(obj) + if err != nil { + t.Fatal(err) + } + if cores := got[KeyDedicatedCPU]; cores.Value() != 6 { + t.Errorf("got %s dedicated cores, want 6", cores.String()) + } +} + +func TestClassifyRejectsIncompleteTypes(t *testing.T) { + cases := map[string]map[string]any{ + "no guest cpu": {"spec": map[string]any{"memory": map[string]any{"guest": "1Gi"}}}, + "no guest memory": {"spec": map[string]any{"cpu": map[string]any{"guest": int64(1)}}}, + "hugepages and overcommit": {"spec": map[string]any{ + "cpu": map[string]any{"guest": int64(1)}, + "memory": map[string]any{"guest": "1Gi", "overcommitPercent": int64(50), "hugepages": map[string]any{"pageSize": "2Mi"}}, + }}, + "supplementalPool without a count": {"spec": map[string]any{ + "cpu": map[string]any{"guest": int64(1), "dedicatedCPUPlacement": true}, + "ioThreadsPolicy": "supplementalPool", + "memory": map[string]any{"guest": "1Gi"}, + }}, + } + for name, obj := range cases { + if got, err := ClassifyInstancetype(obj); err == nil { + t.Errorf("%s: expected an error, got %v", name, got) + } + } +} + +func assertList(t *testing.T, label string, got, want ResourceList) { + t.Helper() + if len(got) != len(want) { + t.Errorf("%s: got %v, want %v", label, got, want) + return + } + for k, w := range want { + g, ok := got[k] + if !ok || g.Cmp(w) != 0 { + t.Errorf("%s: %s = %s, want %s (full: %v)", label, k, g.String(), w.String(), got) + } + } +} diff --git a/pkg/reservation/types.go b/pkg/reservation/types.go new file mode 100644 index 000000000..1a01a0763 --- /dev/null +++ b/pkg/reservation/types.go @@ -0,0 +1,113 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +// Package reservation computes what an application consumes from a tenant's +// quota budget, from its declared values, before anything runs. +// +// A reservation is denominated in the shorthand keys a Tenant's +// resourceQuotas uses (cpu, memory, storage, services.loadbalancers, ...), +// never in rendered pod units: no allocation ratio is applied and no +// virtualization overhead is added. The evaluator holds no per-kind +// knowledge; each kind describes its consumption in a Declaration carried by +// its ApplicationDefinition, so a kind shipped from an out-of-tree catalog +// participates without a change here. +package reservation + +import ( + "sort" + + "k8s.io/apimachinery/pkg/api/resource" +) + +// Quota keys for KubeVirt instance-type classes. A shared vCPU and a pinned +// core are not the same capacity, nor are ordinary, hugepage-backed and +// overcommitted memory, so each class counts against its own key. +const ( + KeyCPU = "cpu" + KeyDedicatedCPU = "dedicated-cpu" + KeyMemory = "memory" + KeyOvercommittedMemory = "overcommitted-memory" + // KeyHugepagesPrefix is followed by the page size, e.g. "hugepages-2Mi", + // matching the corev1 resource name for the same pages. + KeyHugepagesPrefix = "hugepages-" +) + +// Declaration is how a kind's reservation is computed from its values. +// +// Exactly one of Exempt or Resources is meaningful: a kind that consumes +// nothing says so with Exempt rather than with an empty Resources map, so +// that a zero reservation is always a reviewed statement and never an +// omission. +type Declaration struct { + // Exempt declares that the kind consumes nothing from a tenant's budget. + Exempt bool `json:"exempt,omitempty"` + + // Let binds intermediate CEL expressions, evaluated in order, each one + // visible to the later bindings and to every Resources expression. + Let []Binding `json:"let,omitempty"` + + // Resources maps a quota key to the CEL expression computing the + // amount of it the application reserves. An expression may return an + // int, a double, a quantity string or a quantity. + Resources map[string]string `json:"resources,omitempty"` +} + +// Binding names one intermediate CEL expression of a Declaration. +type Binding struct { + Name string `json:"name"` + Expression string `json:"expression"` +} + +// ResourceList is a reservation, keyed by quota key. +type ResourceList map[string]resource.Quantity + +// Add adds other into l in place. +func (l ResourceList) Add(other ResourceList) { + for k, v := range other { + cur := l[k] + cur.Add(v) + l[k] = cur + } +} + +// Sub subtracts other from l in place. A key absent from l goes negative, +// which is what the delta of an Update needs. +func (l ResourceList) Sub(other ResourceList) { + for k, v := range other { + cur := l[k] + cur.Sub(v) + l[k] = cur + } +} + +// DeepCopy returns an independent copy of l. +func (l ResourceList) DeepCopy() ResourceList { + out := make(ResourceList, len(l)) + for k, v := range l { + out[k] = v.DeepCopy() + } + return out +} + +// Keys returns the keys of l in a stable order, for deterministic messages. +func (l ResourceList) Keys() []string { + keys := make([]string, 0, len(l)) + for k := range l { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} From 0a310cc507535acc10d64906dfe07eddbf8cb3d6 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 9 Nov 2026 11:36:14 +0100 Subject: [PATCH 235/518] fix(api): default into typed nil maps and slices A nil map or slice held in an interface is not == nil, so the defaulting walk took it for a present value: writing the first default into a nil map panicked, and a nil slice kept no default. The registry always passes a decoded map, but Apply is now a shared entry point and nothing guarantees that for its other callers. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- pkg/apis/apps/defaulting/defaulting.go | 4 +- pkg/apis/apps/defaulting/defaulting_test.go | 54 +++++++++++++++++++++ 2 files changed, 56 insertions(+), 2 deletions(-) create mode 100644 pkg/apis/apps/defaulting/defaulting_test.go diff --git a/pkg/apis/apps/defaulting/defaulting.go b/pkg/apis/apps/defaulting/defaulting.go index bbec2fa88..f361a7bfd 100644 --- a/pkg/apis/apps/defaulting/defaulting.go +++ b/pkg/apis/apps/defaulting/defaulting.go @@ -101,7 +101,7 @@ func applyDefaults(v any, s *structuralschema.Structural, top bool) (any, error) switch effType { case "object": mv, isMap := v.(map[string]any) - if !isMap || v == nil { + if !isMap || mv == nil { if s.Default.Object != nil && !top { if dm, ok := s.Default.Object.(map[string]any); ok { mv = cloneMap(dm) @@ -144,7 +144,7 @@ func applyDefaults(v any, s *structuralschema.Structural, top bool) (any, error) case "array": sl, isSlice := v.([]any) - if !isSlice || v == nil { + if !isSlice || sl == nil { if s.Default.Object != nil { if ds, ok := s.Default.Object.([]any); ok { sl = cloneSlice(ds) diff --git a/pkg/apis/apps/defaulting/defaulting_test.go b/pkg/apis/apps/defaulting/defaulting_test.go new file mode 100644 index 000000000..0ce580e56 --- /dev/null +++ b/pkg/apis/apps/defaulting/defaulting_test.go @@ -0,0 +1,54 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package defaulting + +import "testing" + +// A nil map[string]any in an interface is not == nil, so it has to be +// recognised as empty explicitly, at the root and below it, or writing the +// first default into it panics. +func TestApplyDefaultsTypedNilMaps(t *testing.T) { + s, err := BuildSchema(`{"type":"object","properties":{ + "replicas":{"type":"integer","default":2}, + "roles":{"type":"array","default":["worker"],"items":{"type":"string"}}, + "resources":{"type":"object","default":{},"properties":{"preset":{"type":"string","default":"t1.micro"}}}}}`) + if err != nil { + t.Fatal(err) + } + + var root map[string]any + if err := Apply(&root, s); err != nil { + t.Fatal(err) + } + if root["replicas"] != int64(2) { + t.Errorf("root: replicas = %#v, want 2", root["replicas"]) + } + + var nested map[string]any + var roles []any + root = map[string]any{"resources": nested, "roles": roles} + if err := Apply(&root, s); err != nil { + t.Fatal(err) + } + resources, _ := root["resources"].(map[string]any) + if resources["preset"] != "t1.micro" { + t.Errorf("nested: resources = %#v, want the preset default", root["resources"]) + } + if got, _ := root["roles"].([]any); len(got) != 1 || got[0] != "worker" { + t.Errorf("nested: roles = %#v, want the array default", root["roles"]) + } +} From 0dc5a5473194bc8a0c112d313c9c2dc9eb5e52a5 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 13:49:42 +0300 Subject: [PATCH 236/518] test(clickhouse): remove the round-trip's TLS objects a killed run left Chainsaw deletes an applied object only when its own apply created it; one left behind by a run killed past its cleanup is patched and kept. The Certificate would then outlive every later run and keep re-issuing its Secret after each step cleanup, and a probe Pod whose sleep had ended never turns Ready, failing the next run's assert. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- ...clickhouse-system-bucket-shared-state.bats | 37 ++++++++++++++++++- .../clickhouse/chainsaw-test.yaml | 6 +++ 2 files changed, 42 insertions(+), 1 deletion(-) diff --git a/hack/clickhouse-system-bucket-shared-state.bats b/hack/clickhouse-system-bucket-shared-state.bats index 10a869af6..e912be42f 100644 --- a/hack/clickhouse-system-bucket-shared-state.bats +++ b/hack/clickhouse-system-bucket-shared-state.bats @@ -221,8 +221,43 @@ STUBEOF echo "the cleanup of step create-certificate-and-probe is not the one delete of Secret e2e-system-bucket-s3-tls: $deletes" >&2 false } - if yq "$test | (.try[], .finally[]?) | .script.content // \"\"" "$suite" | grep -qE 'delete +(secret +e2e-system-bucket-s3-tls|certificate +e2e-system-bucket-s3)( |$)'; then + # The preconditions run before the apply, when Chainsaw holds no Certificate. + if yq "$test | select(.name != \"preconditions\") | (.try[], .finally[]?) | .script.content // \"\"" "$suite" | grep -qE 'delete +(secret +e2e-system-bucket-s3-tls|certificate +e2e-system-bucket-s3)( |$)'; then echo "a step script deletes the Certificate or its Secret while Chainsaw still holds the Certificate" >&2 false fi } + +# Chainsaw registers an applied object for deletion only when its apply created +# it; an object left by a run killed past its cleanup is patched and kept. The +# Certificate would then outlive every later run, re-issuing its Secret after +# each step cleanup, and a probe Pod whose `sleep` had ended never turns Ready +# again, so the preconditions remove all three before the apply. +@test "the preconditions remove the TLS objects and probe Pod a killed run left" { + suite=hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml + test='select(.metadata.name == "clickhouse-3-system-bucket-roundtrip") | .spec.steps[]' + calls="$STUB/calls" + cat >"$STUB/kubectl" <<'STUBEOF' +#!/bin/bash +echo "$*" >>"$STUB/calls" +case "$*" in + *" get secret "*) exit 1 ;; +esac +STUBEOF + : >"$calls" + script=$(yq "$test | select(.name == \"preconditions\") | .try[].script.content" "$suite") + PATH="$STUB:$PATH" NAMESPACE=tenant-root sh -c "$script" + line_of() { grep -nE "^-n tenant-root delete $1 --ignore-not-found --wait=true( |$)" "$calls" | head -n1 | cut -d: -f1; } + cert=$(line_of 'certificate e2e-system-bucket-s3') + secret=$(line_of 'secret e2e-system-bucket-s3-tls') + pod=$(line_of 'pod e2e-system-bucket-probe') + [ -n "$cert" ] && [ -n "$secret" ] && [ -n "$pod" ] || { + echo "the preconditions do not delete, and wait for, the Certificate, its Secret and the probe Pod; kubectl calls:" >&2 + cat "$calls" >&2 + false + } + [ "$cert" -lt "$secret" ] || { + echo "the preconditions delete Secret e2e-system-bucket-s3-tls before its Certificate, which cert-manager then issues again" >&2 + false + } +} diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index b71728720..70fe6693a 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -458,6 +458,12 @@ spec: kubectl -n "$NAMESPACE" delete backupjob sysbucket-backup --ignore-not-found --wait=true --timeout=30s kubectl -n "$NAMESPACE" delete backup sysbucket-backup --ignore-not-found --wait=true --timeout=60s kubectl -n "$NAMESPACE" delete secret cozy-backups-creds --ignore-not-found --wait=true --timeout=30s + # Chainsaw reclaims only what its apply created, so objects a killed + # run left would be patched and kept. The Certificate goes before its + # Secret, which cert-manager issues again while the Certificate exists. + kubectl -n "$NAMESPACE" delete certificate e2e-system-bucket-s3 --ignore-not-found --wait=true --timeout=30s + kubectl -n "$NAMESPACE" delete secret e2e-system-bucket-s3-tls --ignore-not-found --wait=true --timeout=30s + kubectl -n "$NAMESPACE" delete pod e2e-system-bucket-probe --ignore-not-found --wait=true --timeout=60s check: ($error == null): true From 0972690f0d31426f892e7d7e04ad48cf33d5e9c3 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 13:52:38 +0300 Subject: [PATCH 237/518] fix(backups): contest a recorded MariaDB dump before deleting it The size probe records the dump it attributes to a Backup, and it lets only runs that had already completed contest the attribution. A run can upload its dump and prune an older run's through maxRetention before its own operator Backup completes, so the probe of the pruned run can record the newer run's dump, and deleting that Backup then deleted a dump that another Backup still restores from. Deletion now contests the recorded dump against every run on the same bucket and prefix as they stand at deletion time, the unfinished ones included, and leaves it behind with a Warning when any could have written it, as the unrecorded path already does. A run that never completes therefore keeps every later dump its lifetime covers, which leaks storage rather than losing a backup. The probe is unchanged here. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadbstrategy_cleanup.go | 53 ++++- .../mariadbstrategy_cleanup_test.go | 215 ++++++++++++++++++ 2 files changed, 264 insertions(+), 4 deletions(-) diff --git a/internal/backupcontroller/mariadbstrategy_cleanup.go b/internal/backupcontroller/mariadbstrategy_cleanup.go index 0a2fc1e34..94a7d10f4 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup.go @@ -5,6 +5,7 @@ import ( "context" "errors" "fmt" + "slices" "strings" "time" @@ -37,7 +38,8 @@ var mariadbDumpCleanupTimeout = 10 * time.Second // cleanupMariaDBBackup deletes the dump a MariaDB Backup was taken as, then // the operator Backup that wrote it. Every run of an application writes // under one prefix, so the only key deleted is one proven to be this -// Backup's: the dump it records, or, for a Backup taken before dumps were +// Backup's: the dump it records, unless contestedMariaDBDump finds another +// run that could have written it, or, for a Backup taken before dumps were // recorded, the one findMariaDBDump attributes to its operator Backup. When // no dump is proven the Backup is released with a Warning and nothing is // deleted, the operator Backup included, since it is what any later @@ -112,11 +114,20 @@ func (r *BackupReconciler) cleanupMariaDBBackup(ctx context.Context, backup *bac s3ctx, cancel := context.WithTimeout(ctx, mariadbDumpCleanupTimeout) defer cancel() store := r.objectStore() - if name == "" { - lister, err := store.lister(conn) + lister, err := store.lister(conn) + if err != nil { + return r.retryMariaDBCleanup(ctx, backup, location, fmt.Errorf("build S3 client: %w", err)) + } + if name != "" { + other, err := contestedMariaDBDump(s3ctx, lister, r.Client, op, name) if err != nil { - return r.retryMariaDBCleanup(ctx, backup, location, fmt.Errorf("build S3 client: %w", err)) + return r.retryMariaDBCleanup(ctx, backup, location+name, err) } + if other != "" { + return r.releaseMariaDBCleanup(ctx, backup, location+name, + fmt.Sprintf("the recorded dump also falls in the lifetime of k8s.mariadb.com/Backup %s, which may have written it; the operator Backup is kept", other)), nil + } + } else { dump, err := findMariaDBDumpWith(s3ctx, lister, r.Client, op, backup.Spec.TakenAt.Time, true) if errors.Is(err, errMariaDBDumpNotFound) || errors.Is(err, errMariaDBDumpAmbiguous) { return r.releaseMariaDBCleanup(ctx, backup, location, @@ -151,6 +162,40 @@ func (r *BackupReconciler) cleanupMariaDBBackup(ctx context.Context, backup *bac return ctrl.Result{}, nil } +// contestedMariaDBDump returns the name of another operator Backup on op's +// bucket and prefix whose run, finished or not, could have written the dump +// name, or "" when none could. The size probe that recorded name let only +// runs finished by then contest it, and a run can upload its dump and prune +// op's through maxRetention before it completes. A dump no longer in the +// bucket is contested by nobody: deleting it again removes nothing. +func contestedMariaDBDump(ctx context.Context, lister s3ObjectLister, reader client.Reader, op *mariadbtypes.Backup, name string) (string, error) { + ts, ok := parseMariaDBLogicalBackupFile(name) + if !ok { + return "", fmt.Errorf("%q is not the name of a mariadb-operator dump", name) + } + s3 := op.Spec.Storage.S3 + key := mariadbOperatorS3Prefix(s3.Prefix) + name + objects, err := lister.ListObjects(ctx, s3.Bucket, key, false) + if err != nil { + return "", fmt.Errorf("list s3://%s/%s: %w", s3.Bucket, key, err) + } + i := slices.IndexFunc(objects, func(obj s3ObjectInfo) bool { return obj.Key == key }) + if i < 0 { + return "", nil + } + others, err := neighbourMariaDBRuns(ctx, reader, op, true) + if err != nil { + return "", fmt.Errorf("list operator Backups: %w", err) + } + d := mariadbDump{Key: key, Name: name, Timestamp: ts, LastModified: objects[i].LastModified} + for _, other := range others { + if other.placesOnEitherClock(d) { + return other.name, nil + } + } + return "", nil +} + // activeRestoreOf returns the name of a RestoreJob that restores from // backup and has not finished, or "". func (r *BackupReconciler) activeRestoreOf(ctx context.Context, backup *backupsv1alpha1.Backup) (string, error) { diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index 9b41f4701..ddca34523 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -22,6 +22,7 @@ import ( strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" + "github.com/cozystack/cozystack/internal/backupcontroller/mariadbapp" "github.com/cozystack/cozystack/internal/backupcontroller/mariadbtypes" ) @@ -437,6 +438,220 @@ func TestCleanupMariaDBBackup_LegacyBackupDeletesItsDumpDespiteALaterUnfinishedR } } +// The size probe that records a dump lets only finished runs contest it, so +// a run still unfinished then, which uploaded its dump and pruned this +// run's through maxRetention, can have its dump recorded here. Deletion +// contests the recorded dump again, with every run as it stands now. +func TestCleanupMariaDBBackup_RecordedDumpARunUnfinishedAtTheProbeCouldOwnIsReleased(t *testing.T) { + started := cleanupCreated.Add(time.Minute) + theirs := dumpKey(cleanupPrefix, started.Add(30*time.Second), "gzip") + cases := map[string][]metav1.Condition{ + "running": {{ + Type: mariadbtypes.ConditionTypeComplete, Status: metav1.ConditionFalse, + Reason: mariadbtypes.ConditionReasonJobRunning, Message: "Running", LastTransitionTime: metav1.NewTime(started), + }}, + "not yet reconciled": nil, + "completed since": {mariadbBackupSucceeded(metav1.NewTime(cleanupCompleted.Add(5 * time.Minute)))}, + } + for name, conditions := range cases { + t.Run(name, func(t *testing.T) { + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: theirs, LastModified: started.Add(time.Minute)}}} + other := cleanupOperatorBackup(t) + other.Name = "op-other" + other.CreationTimestamp = metav1.NewTime(started) + other.Status.Conditions = conditions + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), other, s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(strings.TrimPrefix(theirs, cleanupPrefix))) + assertReleasedUntouched(t, r, rec, bucket, res, err) + }) + } +} + +// Either clock placing the recorded dump in another run's lifetime is +// enough: the backup pod's clock can be skewed while the store's is not. +func TestCleanupMariaDBBackup_RecordedDumpAnotherRunPlacesOnOneClockIsReleased(t *testing.T) { + stamped := cleanupCreated.Add(20 * time.Second) + uploaded := cleanupCreated.Add(2 * time.Minute) + theirs := dumpKey(cleanupPrefix, stamped, "gzip") + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: theirs, LastModified: uploaded}}} + other := cleanupOperatorBackup(t) + other.Name = "op-other" + other.CreationTimestamp = metav1.NewTime(uploaded.Add(30 * time.Second)) + other.Status.Conditions = nil + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), other, s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(strings.TrimPrefix(theirs, cleanupPrefix))) + assertReleasedUntouched(t, r, rec, bucket, res, err) +} + +// The listing is by prefix, so a longer key sharing it is not the dump and +// contests nothing. +func TestCleanupMariaDBBackup_RecordedDumpIgnoresAKeyItIsOnlyAPrefixOf(t *testing.T) { + key := cleanupPrefix + cleanupDumpName + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: key + ".tmp", LastModified: cleanupCompleted.Add(2 * time.Hour)}}} + running := cleanupOperatorBackup(t) + running.Name = "op-running" + running.CreationTimestamp = metav1.NewTime(cleanupCompleted.Add(time.Hour)) + running.Status.Conditions = nil + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), running, s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup released", res, err) + } + if want := []string{"bkt/" + key}; fmt.Sprint(bucket.removed) != fmt.Sprint(want) { + t.Errorf("removed %v, want %v", bucket.removed, want) + } +} + +// A name the dump time cannot be read from leaves one clock out of the +// contest, so it is an error rather than a contest on half the evidence. +func TestContestedMariaDBDump_RejectsANameWithoutADumpTime(t *testing.T) { + op := cleanupOperatorBackup(t) + name := "backup.not-a-time.sql" + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: cleanupPrefix + name, LastModified: cleanupCreated}}} + c := newMariaDBStrategyTestClient(t, op) + if _, err := contestedMariaDBDump(context.Background(), bucket, c, op, name); err == nil { + t.Fatal("err = nil, want the unreadable dump time reported") + } +} + +// A retry after the dump was deleted but the operator Backup was not finds +// nothing to contest, whichever run's lifetime the dump's name falls in. +func TestCleanupMariaDBBackup_RecordedDumpAlreadyGoneIsNotContested(t *testing.T) { + bucket := &fakeS3Bucket{} + running := cleanupOperatorBackup(t) + running.Name = "op-running" + running.CreationTimestamp = metav1.NewTime(cleanupCreated) + running.Status.Conditions = nil + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), running, s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup released", res, err) + } + if operatorBackupExists(t, r.Client) { + t.Error("operator Backup kept, want it deleted") + } + if events := drainEvents(rec); hasEvent(events, "Warning ArtifactNotDeleted") { + t.Errorf("events = %v, want no ArtifactNotDeleted", events) + } +} + +// A run started after the recorded dump was uploaded cannot have written it. +func TestCleanupMariaDBBackup_DeletesItsRecordedDumpDespiteALaterUnfinishedRun(t *testing.T) { + key := cleanupPrefix + cleanupDumpName + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: key, LastModified: cleanupCreated.Add(2 * time.Minute)}}} + running := cleanupOperatorBackup(t) + running.Name = "op-running" + running.CreationTimestamp = metav1.NewTime(cleanupCompleted.Add(time.Hour)) + running.Status.Conditions = nil + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), running, s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup released", res, err) + } + if want := []string{"bkt/" + key}; fmt.Sprint(bucket.removed) != fmt.Sprint(want) { + t.Errorf("removed %v, want %v", bucket.removed, want) + } + if operatorBackupExists(t, r.Client) { + t.Error("operator Backup kept, want it deleted once its dump is") + } +} + +func TestCleanupMariaDBBackup_RecordedDumpListingFailureRetries(t *testing.T) { + bucket := &fakeS3Bucket{listErr: errors.New("connection refused")} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || !requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup held and retried", res, err) + } + if len(bucket.removed) != 0 || !operatorBackupExists(t, r.Client) { + t.Errorf("removed=%v operatorBackupExists=%v, want nothing deleted", bucket.removed, operatorBackupExists(t, r.Client)) + } + if events := drainEvents(rec); !hasEvent(events, "Warning ArtifactCleanupFailed") { + t.Errorf("events = %v, want a Warning ArtifactCleanupFailed", events) + } +} + +// Without the list of runs nothing is known about who else could have +// written the dump, so it is kept and the delete retried. +func TestCleanupMariaDBBackup_RecordedDumpWithUnreadableRunsRetries(t *testing.T) { + key := cleanupPrefix + cleanupDumpName + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: key, LastModified: cleanupCreated.Add(2 * time.Minute)}}} + c := interceptor.NewClient(newMariaDBStrategyTestClient(t, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()).(client.WithWatch), interceptor.Funcs{ + List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*mariadbtypes.BackupList); ok { + return errors.New("apiserver unavailable") + } + return c.List(ctx, list, opts...) + }, + }) + r, rec := cleanupReconcilerWithClient(c, bucket) + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || !requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup held and retried", res, err) + } + if len(bucket.removed) != 0 || !operatorBackupExists(t, r.Client) { + t.Errorf("removed=%v operatorBackupExists=%v, want nothing deleted", bucket.removed, operatorBackupExists(t, r.Client)) + } + if events := drainEvents(rec); !hasEvent(events, "Warning ArtifactCleanupFailed") { + t.Errorf("events = %v, want a Warning ArtifactCleanupFailed", events) + } +} + +// The whole sequence: run B uploads its dump and prunes this run's before +// B completes, this run's Backup is created and probed, then deleted. B's +// dump must survive whatever the probe recorded. +func TestMariaDBBackup_ProbeThenCleanupKeepsADumpAnUnfinishedRunUploaded(t *testing.T) { + apps := mariadbapp.GroupName + started := cleanupCreated.Add(time.Minute) + theirs := s3ObjectInfo{Key: dumpKey(cleanupPrefix, started.Add(30*time.Second), "gzip"), Size: 42, LastModified: started.Add(time.Minute)} + rendered := newRenderedMariaDBTemplate() + rendered.Storage.S3.Prefix = "tenant/db" + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "mariadb-strategy-default"}, + Parameters: map[string]string{}, + } + job := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bk"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{Kind: "MariaDB", Name: "db", APIGroup: &apps}, + }, + } + other := func() *mariadbtypes.Backup { + b := cleanupOperatorBackup(t) + b.Name = "op-other" + b.CreationTimestamp = metav1.NewTime(started) + b.Status.Conditions = nil + return b + } + + prober := reconcilerWithLister(t, &fakeS3Lister{objects: []s3ObjectInfo{theirs}}, job, other(), s3CredsSecret("tenant", "creds")) + created, err := prober.createMariaDBBackupArtifact(context.Background(), job, resolved, cleanupOperatorBackup(t), rendered) + if err != nil { + t.Fatalf("createMariaDBBackupArtifact: %v", err) + } + if got, want := created.Spec.DriverMetadata[mariadbDumpFileKey], strings.TrimPrefix(theirs.Key, cleanupPrefix); got != want { + t.Fatalf("probe recorded dump %q, want %q: this test covers the recorded path", got, want) + } + + backup := created.DeepCopy() + backup.ResourceVersion = "" + now := metav1.Now() + backup.DeletionTimestamp = &now + backup.Finalizers = []string{backupFinalizer} + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{theirs}} + r, rec := cleanupReconciler(t, bucket, backup, cleanupOperatorBackup(t), other(), s3CredsSecret("tenant", "creds"), tenantNamespace()) + + res, err := r.cleanupMariaDBBackup(context.Background(), backup) + assertReleasedUntouched(t, r, rec, bucket, res, err) +} + func TestCleanupMariaDBBackup_LegacyListingFailureRetries(t *testing.T) { bucket := &fakeS3Bucket{listErr: errors.New("connection refused")} r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) From 72a94ee426017bafb48e4a19aaac187e4c351cf2 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 14:26:02 +0300 Subject: [PATCH 238/518] test(clickhouse): give the round-trip's preconditions room for their deletes The step's ceiling SIGKILLs the shell and reports only "signal: killed", while a kubectl --timeout that fires first names the object that would not go. The deletes it runs now wait up to 270s in all, past the 3m ceiling, so a stuck delete late in the list would be reported by the ceiling instead of by kubectl. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index 70fe6693a..e67b2674a 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -438,8 +438,11 @@ spec: steps: - name: preconditions try: + # The deletes below wait up to 270s in all (30+30+60+30+30+30+60), so a + # stuck one is named by its kubectl --timeout before the 5m ceiling kills + # the shell with no object named. A new delete needs its own --timeout. - script: - timeout: 3m + timeout: 5m content: | set -eu if kubectl -n "$NAMESPACE" get secret seaweedfs-s3-ingress-tls -o name >/dev/null 2>&1; then From a1e28cccddce7a76a57da0b95b0fe80cf5b276ff Mon Sep 17 00:00:00 2001 From: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Date: Mon, 9 Nov 2026 12:51:14 +0100 Subject: [PATCH 239/518] test(kubernetes): cover missing VMI during volume unpublish Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Assisted-by: LLM --- .../kubevirt-csi-driver/controller_test.go | 49 +++++++++++++------ 1 file changed, 34 insertions(+), 15 deletions(-) diff --git a/packages/apps/kubernetes/images/kubevirt-csi-driver/controller_test.go b/packages/apps/kubernetes/images/kubevirt-csi-driver/controller_test.go index 73b8ffc7d..c6029f17b 100644 --- a/packages/apps/kubernetes/images/kubevirt-csi-driver/controller_test.go +++ b/packages/apps/kubernetes/images/kubevirt-csi-driver/controller_test.go @@ -314,12 +314,15 @@ const ( // including when the VMI outlives its VM during CAPI/CAPK teardown. type teardownClient struct { kubevirtclient.Client - vm *kubevirtv1.VirtualMachine - vmiOwners []metav1.OwnerReference - detached bool - vmRemovals int - vmiRemovals int - lookups int + vm *kubevirtv1.VirtualMachine + vmiOwners []metav1.OwnerReference + detached bool + vmiGone bool + vmiGoneAfterCheck bool + vmiLookups int + vmRemovals int + vmiRemovals int + lookups int } func (c *teardownClient) vmiGarbageCollected() bool { @@ -337,6 +340,10 @@ func (c *teardownClient) GetWorkloadManagingVirtualMachine(context.Context, stri func (c *teardownClient) GetVirtualMachine(context.Context, string, string) (*kubevirtv1.VirtualMachineInstance, error) { c.lookups++ + c.vmiLookups++ + if c.vmiGone { + return nil, k8serrors.NewNotFound(schema.GroupResource{Group: "kubevirt.io", Resource: "virtualmachineinstances"}, testVMName) + } vmi := &kubevirtv1.VirtualMachineInstance{ ObjectMeta: metav1.ObjectMeta{Name: testVMName, Namespace: testInfraNamespace, OwnerReferences: c.vmiOwners}, } @@ -351,7 +358,11 @@ func (c *teardownClient) EnsureVolumeRemovedVM(context.Context, string, string, } func (c *teardownClient) EnsureVolumeRemovedVMI(context.Context, string, string, string) (bool, error) { - return c.detached || c.vmRemovals > 0 || c.vmiRemovals > 0, nil + removed := c.vmiGone || c.detached || c.vmRemovals > 0 || c.vmiRemovals > 0 + if c.vmiGoneAfterCheck { + c.vmiGone = true + } + return removed, nil } func (c *teardownClient) RemoveVolumeFromVM(_ context.Context, namespace, name string, opts *kubevirtv1.RemoveVolumeOptions) error { @@ -369,7 +380,7 @@ func (c *teardownClient) RemoveVolumeFromVMI(context.Context, string, string, *k } func (c *teardownClient) EnsureVolumeRemoved(context.Context, string, string, string, time.Duration) error { - if c.detached || c.vmRemovals > 0 || c.vmiRemovals > 0 || c.vmiGarbageCollected() { + if c.vmiGone || c.detached || c.vmRemovals > 0 || c.vmiRemovals > 0 || c.vmiGarbageCollected() { return nil } return errors.New("timed out waiting for the VMI to drop the volume") @@ -429,20 +440,25 @@ func TestControllerUnpublishVolumeHonoursVMIHotplugFallback(t *testing.T) { func TestControllerUnpublishVolumeMissingPVC(t *testing.T) { tests := []struct { - name string - vmGone bool - detached bool - wantVMCalls int - wantVMICalls int + name string + vmGone bool + detached bool + vmiGone bool + vmiGoneAfterCheck bool + wantVMILookups int + wantVMCalls int + wantVMICalls int }{ {name: "pending hotplug volume", wantVMCalls: 1}, {name: "already detached", detached: true}, - {name: "VM gone, VMI still has volume", vmGone: true, wantVMICalls: 1}, + {name: "VM gone, VMI still has volume", vmGone: true, wantVMICalls: 1, wantVMILookups: 1}, {name: "VM gone, no attached volume", vmGone: true, detached: true}, + {name: "VM and VMI gone", vmGone: true, vmiGone: true}, + {name: "VMI disappears after attachment check", vmGone: true, vmiGoneAfterCheck: true, wantVMILookups: 1}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - client := &teardownClient{detached: tt.detached} + client := &teardownClient{detached: tt.detached, vmiGone: tt.vmiGone, vmiGoneAfterCheck: tt.vmiGoneAfterCheck} if !tt.vmGone { client.vm = &kubevirtv1.VirtualMachine{Spec: kubevirtv1.VirtualMachineSpec{Template: &kubevirtv1.VirtualMachineInstanceTemplateSpec{}}} if !tt.detached { @@ -464,6 +480,9 @@ func TestControllerUnpublishVolumeMissingPVC(t *testing.T) { if client.vmRemovals != tt.wantVMCalls || client.vmiRemovals != tt.wantVMICalls { t.Errorf("VM/VMI removevolume calls = %d/%d, want %d/%d", client.vmRemovals, client.vmiRemovals, tt.wantVMCalls, tt.wantVMICalls) } + if client.vmiLookups != tt.wantVMILookups { + t.Errorf("GetVirtualMachine calls = %d, want %d", client.vmiLookups, tt.wantVMILookups) + } if client.vm != nil && len(client.vm.Spec.Template.Spec.Volumes) != 0 { t.Error("hotplug volume remains in VM spec") } From f2ee3ec8ad742649c6359a4d50e94496aa967fce Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 9 Nov 2026 13:24:36 +0100 Subject: [PATCH 240/518] test(computeplane): pin the maxUnavailable bounds and zero through the API A zero dropped anywhere between the ComputePlane API and the pool chart would let a GPU pool fall back to the chart default of 1, and the render tests never cross the aggregated API. A negative value let through by the ComputePlane schema would only be refused by the pool chart, after the parent release had already been upgraded. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- hack/e2e-chainsaw/computeplane/chainsaw-test.yaml | 14 +++++++++----- .../extra/computeplane/tests/cluster_test.yaml | 14 ++++++++++++++ 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/hack/e2e-chainsaw/computeplane/chainsaw-test.yaml b/hack/e2e-chainsaw/computeplane/chainsaw-test.yaml index 2a02dc06e..a81ff89ef 100644 --- a/hack/e2e-chainsaw/computeplane/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/computeplane/chainsaw-test.yaml @@ -102,7 +102,9 @@ spec: # # nameOverride is outside the module's per-pool allowlist. The ComputePlane # schema accepts it, so it reaches the module and the pool check below can - # see whether the module dropped it. + # see whether the module dropped it. maxUnavailable is set to 0 because a + # zero lost anywhere on the way would let a GPU pool fall back to the + # chart's default of 1; only this path crosses the aggregated API. - script: timeout: 3m content: | @@ -118,6 +120,7 @@ spec: diskSize: 20Gi instanceType: u1.medium logSerialConsole: true + maxUnavailable: 0 minReplicas: 2 maxReplicas: 2 podCpuLimit: 2 @@ -175,8 +178,9 @@ spec: enabled: true exposeMethod: Proxied - # The pool takes the patched shape, and the nameOverride the patch sent - # does not reach it, nor does any other key outside the allowlist. + # The pool takes the patched shape, the explicit maxUnavailable of 0 + # included, and the nameOverride the patch sent does not reach it, nor does + # any other key outside the allowlist. - script: timeout: 1m content: | @@ -186,8 +190,8 @@ spec: sc=$(cozy_e2e_storage_class) values=$(timeout -k 5 30 kubectl -n tenant-cplane get helmrelease kubernetes-nodes-computeplane-cluster-md0 -o json | jq -c '.spec.values') echo "pool values: $values" - echo "$values" | jq -e --arg sc "$sc" '.storageClass == $sc and .minReplicas == 2 and .cluster == "computeplane-cluster" and .clusterReleaseName == "computeplane-cluster"' - extra=$(echo "$values" | jq -r 'keys - ["cluster","clusterReleaseName","minReplicas","maxReplicas","instanceType","diskSize","storageClass","roles","resources","gpus","kubelet","logSerialConsole","maxUnhealthy","nodeStartupTimeout","readyUnknownTimeout","readyFalseTimeout","podCpuLimit","podCpuRequest","maxNodeProvisionTime"] | .[]') + echo "$values" | jq -e --arg sc "$sc" '.storageClass == $sc and .minReplicas == 2 and .maxUnavailable == 0 and .cluster == "computeplane-cluster" and .clusterReleaseName == "computeplane-cluster"' + extra=$(echo "$values" | jq -r 'keys - ["cluster","clusterReleaseName","minReplicas","maxReplicas","instanceType","diskSize","storageClass","roles","resources","gpus","kubelet","logSerialConsole","maxUnhealthy","nodeStartupTimeout","readyUnknownTimeout","readyFalseTimeout","podCpuLimit","podCpuRequest","maxNodeProvisionTime","maxUnavailable"] | .[]') if [ -n "$extra" ]; then echo "pool HelmRelease carries keys outside the module allowlist: $extra" >&2 exit 1 diff --git a/packages/extra/computeplane/tests/cluster_test.yaml b/packages/extra/computeplane/tests/cluster_test.yaml index 6dc4b8835..ad9af03fd 100644 --- a/packages/extra/computeplane/tests/cluster_test.yaml +++ b/packages/extra/computeplane/tests/cluster_test.yaml @@ -353,6 +353,20 @@ tests: - failedTemplate: errorPattern: "maxUnavailable" + - it: refuses a negative maxUnavailable + set: + nodeGroups: + md0: + minReplicas: 1 + maxReplicas: 3 + instanceType: u1.large + diskSize: 32Gi + roles: ["ingress-nginx"] + maxUnavailable: -1 + asserts: + - failedTemplate: + errorPattern: "maxUnavailable" + # Helm caps a HelmRelease name at 53 chars and each pool renders a 38-char fixed # prefix, so a pool key over 15 chars would create a release the apiserver # accepts but Flux never reconciles. The render must refuse instead (the same From d530ca8cf534eb4387934eccae6a94402ddfe01b Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 17:51:39 +0300 Subject: [PATCH 241/518] fix(backups): keep a ClickHouse archive an earlier attempt finished A backup Pod that loses sight of its upload, by dying or by losing six polls in a row, is retried while the sidecar finishes that upload. The retry found it in the action log as a success, took the sidecar for idle and asked for create_remote with resume again. clickhouse-backup then rebuilds the local backup and uploads it once more over the archive already in the bucket, which is left broken if that second upload dies halfway. The archive name belongs to one BackupJob, so a success row of exactly this upload can only be an earlier attempt's, and the retry now takes it as its result. An error or cancel row still leads to a resumed upload, and a restore still runs again, since a restore is not named per RestoreJob. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/altinity-backup-strategy.bats | 44 ++++++++++++++++--- .../templates/strategy-altinity-default.yaml | 9 +++- 2 files changed, 47 insertions(+), 6 deletions(-) diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats index 2597338a2..46cf62d9a 100644 --- a/hack/altinity-backup-strategy.bats +++ b/hack/altinity-backup-strategy.bats @@ -483,15 +483,49 @@ run_script() { grep -qx 'create_remote --resume clickhouse-myapp-20260304T050607Z' "$tmp/state/accepted" } -# The action log is never trimmed, so a finished row of the same archive may -# be an earlier run's and is not taken for this attempt's result. -@test "backup starts create_remote again when the log holds a finished one of the same archive" { - for status in success error cancel; do +# The archive is named per BackupJob, so a success row of exactly this +# upload is an earlier attempt of the same BackupJob that finished while it +# could not see it. Uploading again with resume would re-upload over the +# finished archive and break it if that second upload dies halfway. +@test "a retried backup takes the upload of its archive an earlier attempt finished as its result" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo '{"command":"create_remote --resume clickhouse-myapp-20260304T050607Z","status":"success","start":"2026-01-01 00:01:30"}' > "$tmp/state/prior" + render_script "$tmp/s.sh" backup myapp clickhouse-myapp-20260304T050607Z + run_script "$tmp" + if grep -q '^POST ' "$tmp/state/requests"; then + echo "FAIL: uploaded again over a finished archive: $(cat "$tmp/state/requests")" + false + fi +} + +@test "a retried backup resumes an upload of its archive that ended in error or was cancelled" { + for status in error cancel; do tmp=$(mktemp -d) make_stubs "$tmp/bin" "$tmp/state" echo "{\"command\":\"create_remote --resume clickhouse-myapp-20260304T050607Z\",\"status\":\"$status\",\"start\":\"2026-01-01 00:01:30\"}" > "$tmp/state/prior" render_script "$tmp/s.sh" backup myapp clickhouse-myapp-20260304T050607Z run_script "$tmp" - if [ "$(accepted "$tmp")" != 1 ]; then echo "FAIL: a $status row of an earlier upload must not stand in for this one"; false; fi + if [ "$(accepted "$tmp")" != 1 ]; then echo "FAIL: a $status row must lead to a resumed upload"; false; fi done } + +@test "a retried backup resumes when its latest upload failed after an earlier one succeeded" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + printf '%s\n' \ + '{"command":"create_remote --resume clickhouse-myapp-20260304T050607Z","status":"success","start":"2026-01-01 00:01:00"}' \ + '{"command":"create_remote --resume clickhouse-myapp-20260304T050607Z","status":"error","start":"2026-01-01 00:01:30"}' > "$tmp/state/prior" + render_script "$tmp/s.sh" backup myapp clickhouse-myapp-20260304T050607Z + run_script "$tmp" + [ "$(accepted "$tmp")" = 1 ] +} + +@test "backup does not take another archive's finished upload as its own" { + tmp=$(mktemp -d) + make_stubs "$tmp/bin" "$tmp/state" + echo '{"command":"create_remote --resume clickhouse-myapp-20260304T040000Z","status":"success","start":"2026-01-01 00:01:30"}' > "$tmp/state/prior" + render_script "$tmp/s.sh" backup myapp clickhouse-myapp-20260304T050607Z + run_script "$tmp" + [ "$(accepted "$tmp")" = 1 ] +} diff --git a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml index 18174d61a..331a89be8 100644 --- a/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml +++ b/packages/system/backupstrategy-controller/templates/strategy-altinity-default.yaml @@ -104,11 +104,18 @@ spec: # so a local or partial remote backup an earlier attempt left is # resumed rather than refused as existing, whatever # use_resumable_state the sidecar runs with. The sidecar logs - # the request as "create_remote --resume ". + # the request as "create_remote --resume ". The name is + # this BackupJob's alone, so an upload of it that last ended in + # success is an earlier attempt's finished result: resuming + # over it would upload the archive again and leave it broken if + # that upload died halfway. cmd="create_remote --resume ${target}" wait_turn "${cmd}" if [ "${turn}" = "join" ]; then echo "${cmd} is already running; waiting for it" + elif printf '%s\n' "${actions}" | jq -es --arg cmd "${cmd}" \ + '[.[] | select(.command == $cmd)] | sort_by(.start) | last | .status == "success"' >/dev/null; then + echo "${cmd} already succeeded in an earlier attempt" else echo "${cmd}" curl -fsS ${auth} -X POST "${api}/backup/create_remote?name=${target}&resume=true" >/dev/null From 1291e616c700def517907d072289e9970b8e694c Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 16:39:02 +0300 Subject: [PATCH 242/518] fix(clickhouse): give every backup run an archive of its own An archive was named from the application and the BackupJob's start time, which has one-second resolution. Two runs of one application started in the same second, as two Plans on the same schedule are, wrote the same archive. The strategy takes a finished upload of its archive as an earlier attempt of its own run, so the later run could succeed on the other run's upload and record an archive missing any table created since. The BackupJob's UID now ends the name. It goes after the timestamp so names keep sorting chronologically, and the restore of a Backup that records no archive, which accepts only a bare timestamp after the release prefix, never picks one of these. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/altinity-backup-strategy.bats | 1 + .../altinitystrategy_controller.go | 14 +++--- .../altinitystrategy_controller_test.go | 44 +++++++++++++++---- 3 files changed, 46 insertions(+), 13 deletions(-) diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats index 46cf62d9a..d26b22ee4 100644 --- a/hack/altinity-backup-strategy.bats +++ b/hack/altinity-backup-strategy.bats @@ -393,6 +393,7 @@ run_script() { remote_rows clickhouse-myapp-20260304T040000Z clickhouse-myapp-20260304T050607Z \ "clickhouse-myapp-20260304T050700Z|broken (can't stat metadata.json)" \ clickhouse-myapp-20260304T050710Z-copy \ + clickhouse-myapp-20260304T050720Z-0f3c9a1e-5b7d-4c2a-9e8f-1a2b3c4d5e6f \ clickhouse-myapp-20260304T060000Z > "$tmp/state/remote" out=$(run_script "$tmp" 2>&1) grep -qxF 'POST http://chi-clickhouse-myapp-clickhouse-0-0.tenant-x.svc:7171/backup/restore_remote/clickhouse-myapp-20260304T050607Z' "$tmp/state/requests" diff --git a/internal/backupcontroller/altinitystrategy_controller.go b/internal/backupcontroller/altinitystrategy_controller.go index a0868ec38..1aec3a585 100644 --- a/internal/backupcontroller/altinitystrategy_controller.go +++ b/internal/backupcontroller/altinitystrategy_controller.go @@ -117,12 +117,16 @@ func backupParameters(b *backupsv1alpha1.Backup) map[string]string { } // altinityArchiveName mints the archive name for one BackupJob from its -// persisted StartedAt, so every reconcile and every Pod retry of the Job -// writes the same archive. It keeps the clickhouse-- -// shape, because a Backup that records no archive is restored by the -// timestamp in that shape. +// persisted StartedAt and its UID, so every reconcile and every Pod retry of +// the Job writes the same archive and no other run writes it. StartedAt has +// one-second resolution, and two Plans on one schedule start their runs in +// the same second; the strategy takes a finished upload of its archive as an +// earlier attempt of its own run, so a shared name would let one run report +// the other's archive. The UID goes after the timestamp so names still sort +// chronologically, and the restore of a Backup that records no archive, which +// accepts only clickhouse--, never picks one of these. func altinityArchiveName(j *backupsv1alpha1.BackupJob) string { - return fmt.Sprintf("clickhouse-%s-%s", j.Spec.ApplicationRef.Name, j.Status.StartedAt.UTC().Format(altinityArchiveTimeLayout)) + return fmt.Sprintf("clickhouse-%s-%s-%s", j.Spec.ApplicationRef.Name, j.Status.StartedAt.UTC().Format(altinityArchiveTimeLayout), j.UID) } // altinityJobCarriesArchive reports whether the Job's Pod was rendered with diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index e5f1e1695..18cead060 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -14,6 +14,7 @@ import ( "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" dynamicfake "k8s.io/client-go/dynamic/fake" "k8s.io/client-go/kubernetes/scheme" "k8s.io/client-go/tools/record" @@ -730,14 +731,41 @@ func newAltinityArchiveStrategy(name string) *strategyv1alpha1.Altinity { func TestAltinityArchiveName_FromStartedAtInUTC(t *testing.T) { startedAt := metav1.NewTime(time.Date(2026, 3, 4, 8, 6, 7, 890000000, time.FixedZone("UTC+3", 3*3600))) j := &backupsv1alpha1.BackupJob{ - Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: newAltinityClickHouseRef("ch-test")}, - Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt}, + ObjectMeta: metav1.ObjectMeta{UID: "0f3c9a1e-5b7d-4c2a-9e8f-1a2b3c4d5e6f"}, + Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: newAltinityClickHouseRef("ch-test")}, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt}, } - if got, want := altinityArchiveName(j), "clickhouse-ch-test-20260304T050607Z"; got != want { + if got, want := altinityArchiveName(j), "clickhouse-ch-test-20260304T050607Z-0f3c9a1e-5b7d-4c2a-9e8f-1a2b3c4d5e6f"; got != want { t.Errorf("altinityArchiveName = %q, want %q", got, want) } } +// TestAltinityArchiveName_DistinctForRunsStartedInTheSameSecond pins that two +// BackupJobs of one application started within the same second, as two Plans +// on the same schedule are, write different archives. With a shared name, the +// strategy takes the other run's finished upload for an earlier attempt of its +// own and records an archive missing what changed since that upload. +func TestAltinityArchiveName_DistinctForRunsStartedInTheSameSecond(t *testing.T) { + startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC)) + run := func(name, uid string) *backupsv1alpha1.BackupJob { + return &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "tenant-test", UID: types.UID(uid)}, + Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: newAltinityClickHouseRef("ch-test")}, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt}, + } + } + daily := altinityArchiveName(run("daily-1", "11111111-1111-1111-1111-111111111111")) + weekly := altinityArchiveName(run("weekly-1", "22222222-2222-2222-2222-222222222222")) + if daily == weekly { + t.Fatalf("two runs started in the same second share archive %q", daily) + } + for _, name := range []string{daily, weekly} { + if !strings.HasPrefix(name, "clickhouse-ch-test-20260304T050607Z-") { + t.Errorf("archive %q does not start with the application and start time, so names stop sorting chronologically", name) + } + } +} + // TestReconcileAltinity_RecordsTheArchiveTheJobWasGiven drives one BackupJob // through Job creation and completion in separate reconciles. The archive // name has to come out of the persisted StartedAt both times: the Pod is @@ -748,7 +776,7 @@ func TestReconcileAltinity_RecordsTheArchiveTheJobWasGiven(t *testing.T) { strategy := newAltinityArchiveStrategy("clickhouse-strategy") startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC)) backupJob := &backupsv1alpha1.BackupJob{ - ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test", UID: "test-bj-uid"}, Spec: backupsv1alpha1.BackupJobSpec{ ApplicationRef: newAltinityClickHouseRef("ch-test"), BackupClassName: "clickhouse-backup", @@ -765,7 +793,7 @@ func TestReconcileAltinity_RecordsTheArchiveTheJobWasGiven(t *testing.T) { } r, _, _ := newAltinityTestEnv(t, app, clientfake.NewClientBuilder().WithObjects(backupJob, strategy)) ctx := context.Background() - const want = "clickhouse-ch-test-20260304T050607Z" + const want = "clickhouse-ch-test-20260304T050607Z-test-bj-uid" if _, err := r.reconcileAltinity(ctx, backupJob, resolved); err != nil { t.Fatalf("reconcileAltinity() creating the Job: %v", err) @@ -869,13 +897,13 @@ func TestReconcileAltinity_RecordsNoArchiveTheJobWasNotGiven(t *testing.T) { // A Backup recording this run's own archive is the one an earlier reconcile // created before its status write was lost, and is adopted. func TestReconcileAltinity_BackupNameAlreadyTaken(t *testing.T) { - const thisRun = "clickhouse-ch-test-20260305T050607Z" + const thisRun = "clickhouse-ch-test-20260305T050607Z-test-bj-uid" cases := []struct { name string existing string // archive the existing Backup records; empty for none wantSucceeded bool }{ - {name: "earlier run's Backup records another archive", existing: "clickhouse-ch-test-20260304T050607Z"}, + {name: "earlier run's Backup records another archive", existing: "clickhouse-ch-test-20260304T050607Z-earlier-bj-uid"}, {name: "earlier run's Backup records no archive", existing: ""}, {name: "this run's Backup from a reconcile whose status write was lost", existing: thisRun, wantSucceeded: true}, } @@ -885,7 +913,7 @@ func TestReconcileAltinity_BackupNameAlreadyTaken(t *testing.T) { strategy := newAltinityArchiveStrategy("clickhouse-strategy") startedAt := metav1.NewTime(time.Date(2026, 3, 5, 5, 6, 7, 0, time.UTC)) backupJob := &backupsv1alpha1.BackupJob{ - ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test", UID: "test-bj-uid"}, Spec: backupsv1alpha1.BackupJobSpec{ ApplicationRef: newAltinityClickHouseRef("ch-test"), BackupClassName: "clickhouse-backup", From 0f336e99c51999117712d6a67e074dcbaaec6bad Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 16:39:33 +0300 Subject: [PATCH 243/518] test(backups): drop the comment restating what accepted() returns Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/altinity-backup-strategy.bats | 1 - 1 file changed, 1 deletion(-) diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats index d26b22ee4..97ec869cc 100644 --- a/hack/altinity-backup-strategy.bats +++ b/hack/altinity-backup-strategy.bats @@ -175,7 +175,6 @@ remote_rows() { done } -# The number of operations the sidecar accepted. accepted() { if [ -e "$1/state/accepted" ]; then wc -l < "$1/state/accepted" | tr -d ' '; else echo 0; fi } From fc15ed7171322c1e67bd008e2a59a3102740f107 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 16:42:13 +0300 Subject: [PATCH 244/518] fix(clickhouse): keep the archive name of a backup running across the upgrade A backup Job created before archive names carried the BackupJob's UID writes its archive under the name without it, and the Job is not re-rendered when the controller is upgraded. Matching only the new name left that Backup with no archive recorded, so its restore fell back to guessing by timestamp. The name without the UID is matched second, because it is a prefix of the one with it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../altinitystrategy_controller.go | 16 ++++- .../altinitystrategy_controller_test.go | 59 +++++++++++++++++++ 2 files changed, 73 insertions(+), 2 deletions(-) diff --git a/internal/backupcontroller/altinitystrategy_controller.go b/internal/backupcontroller/altinitystrategy_controller.go index 1aec3a585..cd03b1d46 100644 --- a/internal/backupcontroller/altinitystrategy_controller.go +++ b/internal/backupcontroller/altinitystrategy_controller.go @@ -126,7 +126,14 @@ func backupParameters(b *backupsv1alpha1.Backup) map[string]string { // chronologically, and the restore of a Backup that records no archive, which // accepts only clickhouse--, never picks one of these. func altinityArchiveName(j *backupsv1alpha1.BackupJob) string { - return fmt.Sprintf("clickhouse-%s-%s-%s", j.Spec.ApplicationRef.Name, j.Status.StartedAt.UTC().Format(altinityArchiveTimeLayout), j.UID) + return altinityArchiveNameBeforeUIDs(j) + "-" + string(j.UID) +} + +// altinityArchiveNameBeforeUIDs is the name a controller that did not append +// the UID gave the same BackupJob, which a Job it created before an upgrade +// still writes. +func altinityArchiveNameBeforeUIDs(j *backupsv1alpha1.BackupJob) string { + return fmt.Sprintf("clickhouse-%s-%s", j.Spec.ApplicationRef.Name, j.Status.StartedAt.UTC().Format(altinityArchiveTimeLayout)) } // altinityJobCarriesArchive reports whether the Job's Pod was rendered with @@ -286,8 +293,13 @@ func (r *BackupJobReconciler) reconcileAltinity(ctx context.Context, j *backupsv if j.Status.BackupRef != nil { return ctrl.Result{}, nil } + // The name without the UID is a prefix of the one with it, so it is + // tried only after the full name is not found. if !altinityJobCarriesArchive(batchJob, archive) { - archive = "" + archive = altinityArchiveNameBeforeUIDs(j) + if !altinityJobCarriesArchive(batchJob, archive) { + archive = "" + } } artifact, err := r.createAltinityBackupArtifact(ctx, j, resolved, archive) if err != nil { diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index 18cead060..08c6e5466 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -829,6 +829,65 @@ func TestReconcileAltinity_RecordsTheArchiveTheJobWasGiven(t *testing.T) { } } +// TestReconcileAltinity_RecordsTheArchiveOfAJobNamedBeforeUIDs covers a backup +// Job created by a controller that named archives without the BackupJob's UID +// and still running when the controller is upgraded: the Job is not +// re-rendered, so its archive carries the old name, and the Backup has to +// record that name rather than none. +func TestReconcileAltinity_RecordsTheArchiveOfAJobNamedBeforeUIDs(t *testing.T) { + app := newAltinityClickHouseApp("ch-test", "tenant-test") + strategy := newAltinityArchiveStrategy("clickhouse-strategy") + startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC)) + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test", UID: "test-bj-uid"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newAltinityClickHouseRef("ch-test"), + BackupClassName: "clickhouse-backup", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt, Phase: backupsv1alpha1.BackupJobPhaseRunning}, + } + resolved := &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{ + APIGroup: new(strategyv1alpha1.GroupVersion.Group), + Kind: strategyv1alpha1.AltinityStrategyKind, + Name: "clickhouse-strategy", + }, + } + r, _, _ := newAltinityTestEnv(t, app, clientfake.NewClientBuilder().WithObjects(backupJob, strategy)) + ctx := context.Background() + const legacy = "clickhouse-ch-test-20260304T050607Z" + + if _, err := r.reconcileAltinity(ctx, backupJob, resolved); err != nil { + t.Fatalf("reconcileAltinity() creating the Job: %v", err) + } + job := &batchv1.Job{} + if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}, job); err != nil { + t.Fatalf("get backup Job: %v", err) + } + job.Spec.Template.Spec.Containers[0].Args[1] = "--archive=" + legacy + if err := r.Update(ctx, job); err != nil { + t.Fatalf("give the backup Job the name without the UID: %v", err) + } + job.Status.Conditions = []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}} + if err := r.Status().Update(ctx, job); err != nil { + t.Fatalf("mark backup Job complete: %v", err) + } + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), backupJob); err != nil { + t.Fatalf("refresh BackupJob: %v", err) + } + if _, err := r.reconcileAltinity(ctx, backupJob, resolved); err != nil { + t.Fatalf("reconcileAltinity() on the completed Job: %v", err) + } + + created := &backupsv1alpha1.Backup{} + if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: backupJob.Name}, created); err != nil { + t.Fatalf("get Backup: %v", err) + } + if got := created.Spec.DriverMetadata[altinityArchiveMetaKey]; got != legacy { + t.Errorf("Backup records archive %q, want %q", got, legacy) + } +} + // TestReconcileAltinity_RecordsNoArchiveTheJobWasNotGiven covers a Job whose // Pod never carried the driver's archive name: one created before the // upgrade that introduced the names, or rendered from a strategy that names From 35c47f20c15ff85f20764f107e1c01532c39ea6e Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 9 Nov 2026 14:52:13 +0000 Subject: [PATCH 245/518] fix(release): fail check --cut while a workflow original waits alone check --cut returned 0 whenever nothing was missing and nothing was only on the batch, so a workflow original waiting for a maintainer passed the cut once the pick beside it was labelled backport/skip or reached the line by hand. The waiting original was not even printed: the case that held it swallowed the warning the plain check gives. check now sorts every original into one disposition and exits once, so a wait fails the cut on its own. A backport/skip original the open batch still carries is named with the run --defer that takes it out. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 4 +- cmd/backport-batch/check.go | 127 ++++++++++++++------ cmd/backport-batch/check_test.go | 173 +++++++++++++++++++++++++++ cmd/backport-batch/cut_test.go | 64 ++++++++++ cmd/backport-batch/main.go | 9 +- cmd/backport-batch/workflows_test.go | 8 +- docs/release/backports.md | 2 +- 7 files changed, 342 insertions(+), 45 deletions(-) create mode 100644 cmd/backport-batch/cut_test.go diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index 33ec9ee89..60eac648d 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -42,7 +42,9 @@ The pull request body goes to stdout, built from the line's own `.github/PULL_RE Exits 1 and prints the URL of every pull request the rules take that is neither on the line nor on the open batch pull request, 0 when there is none, and 2 when it cannot tell. A pull request that the open batch lists as deferred, a conflict or one taken out with `--defer`, prints as a warning and does not fail the check: deferring is a maintainer's call. So does one it lists as waiting for a maintainer because it changes workflows. -`--cut` makes it the gate before a patch release or an rc is cut from a line. A cut tags the line, so a pull request that is only on the open batch pull request fails it too, with one line each: `#N is only on open batch PR #M, merge it before the cut`. One that waits there for a maintainer fails it as well, since nobody decided to leave it out: bring it over, or take it out with `run --defer` and a reason. Everything else is as without it. The scheduled run checks without `--cut`, since its batch is open by design. +`--cut` makes it the gate before a patch release or an rc is cut from a line. A cut tags the line, so a pull request that is only on the open batch pull request fails it too, with one line each: `#N is only on open batch PR #M, merge it before the cut`. One that waits there for a maintainer fails it as well, since nobody decided to leave it out, also when it is all the batch has left: bring it over, or take it out with `run --defer` and a reason. Everything else is as without it, and a cut that passes says `no change the policy takes is missing before the cut`. The scheduled run checks without `--cut`, since its batch is open by design. + +An original with `backport/skip` that the open batch picked before the label came prints a warning in either mode, naming the batch and `run --defer`, which takes it out; it fails neither. ### fold diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index 00e3d5f16..a565c3c45 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -18,6 +18,7 @@ package main import ( "fmt" + "slices" "strconv" ) @@ -46,59 +47,111 @@ func (a *app) check(args []string) int { if err != nil { return a.fail(err) } - deferred := map[int]deferredEntry{} - if b.pr != nil { - if deferred, err = parseDeferred(b.pr.Body, l); err != nil { - return a.fail(fmt.Errorf("batch pull request #%d: %w", b.pr.Number, err)) + disp, err := b.dispositions(*cut) + if err != nil { + return a.fail(err) + } + var missing, failing []string + for _, v := range b.verdicts { + d, ok := disp[v] + switch { + case !ok: + case d.kind == outcomeMissing: + missing = append(missing, d.text) + case d.kind == outcomeUnmerged || d.kind == outcomeWaiting: + failing = append(failing, d.text) + case d.kind == outcomeWarning: + a.warn("%s", d.text) + case d.kind == outcomeNote: + fmt.Fprintf(a.stderr, "note: %s\n", d.text) } } - var missing, unmerged, waiting []*verdict - for _, v := range b.qualifying() { - e, isDeferred := deferred[v.o.number] - switch { - case v.onBatch != "": - if *cut { - unmerged = append(unmerged, v) - } - case isDeferred && e.Workflows && *cut: - waiting = append(waiting, v) - case isDeferred && e.Workflows: - a.warn("waiting on #%d for a maintainer, since it changes %s: %s %s", b.pr.Number, workflowsDir, v.o.url(), terminalSafe(v.o.title)) - case isDeferred && e.Reason != "": - a.warn("deferred on #%d to the next batch: %s %s (%s)", b.pr.Number, v.o.url(), terminalSafe(v.o.title), terminalSafe(e.Reason)) - case isDeferred: - a.warn("deferred conflict on #%d: %s %s", b.pr.Number, v.o.url(), terminalSafe(v.o.title)) - default: - missing = append(missing, v) + if len(missing)+len(failing) == 0 { + if *cut { + fmt.Fprintf(a.stdout, "%s: no change the policy takes is missing before the cut\n", l.branch()) + } else { + fmt.Fprintf(a.stdout, "%s: every change the policy takes is on the line or on the batch\n", l.branch()) } + return 0 } where := "and no batch pull request is open" if b.pr != nil { where = "nor on batch pull request #" + strconv.Itoa(b.pr.Number) } - switch { - case len(missing) == 0 && len(unmerged) == 0 && len(waiting) == 0 && *cut: - fmt.Fprintf(a.stdout, "%s: every change the policy takes is on the line\n", l.branch()) - return 0 - case len(missing) == 0 && len(unmerged) == 0: - fmt.Fprintf(a.stdout, "%s: every change the policy takes is on the line or on the batch\n", l.branch()) - return 0 - } if len(missing) > 0 { fmt.Fprintf(a.stdout, "%s: %d changes the policy takes are not on the line, %s:\n", l.branch(), len(missing), where) - for _, v := range missing { - fmt.Fprintf(a.stdout, " %s %s\n", v.o.url(), terminalSafe(v.o.title)) + for _, text := range missing { + fmt.Fprintf(a.stdout, " %s\n", text) } } - for _, v := range unmerged { - fmt.Fprintf(a.stdout, "%s: #%d is only on open batch PR #%d, merge it before the cut: %s %s\n", l.branch(), v.o.number, b.pr.Number, v.o.url(), terminalSafe(v.o.title)) - } - for _, v := range waiting { - fmt.Fprintf(a.stdout, "%s: #%d changes %s and waits on batch PR #%d for a maintainer, bring it over or defer it before the cut: %s %s\n", l.branch(), v.o.number, workflowsDir, b.pr.Number, v.o.url(), terminalSafe(v.o.title)) + for _, text := range failing { + fmt.Fprintf(a.stdout, "%s: %s\n", l.branch(), text) } return 1 } +// outcome is what check makes of one original. +type outcome int + +const ( + // outcomeMissing fails check and the cut: nothing accounts for an + // original the line lacks. + outcomeMissing outcome = iota + 1 + // outcomeUnmerged and outcomeWaiting fail the cut only: the cut would not + // carry what is only on the open batch, nor what waits there. + outcomeUnmerged + outcomeWaiting + outcomeWarning + outcomeNote +) + +type disposition struct { + kind outcome + text string +} + +// dispositions says what check reports about each original that needs a line +// of its own; the others pass in silence. The text is what follows the line's +// name, or the original alone for one that is missing. +func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { + deferred := map[int]deferredEntry{} + var stale []*verdict + if b.pr != nil { + var err error + if deferred, err = parseDeferred(b.pr.Body, b.line); err != nil { + return nil, fmt.Errorf("batch pull request #%d: %w", b.pr.Number, err) + } + if stale, err = keptOut(b, b.picks); err != nil { + return nil, err + } + } + out := map[*verdict]disposition{} + for _, v := range b.verdicts { + named := v.o.url() + " " + terminalSafe(v.o.title) + e, isDeferred := deferred[v.o.number] + switch { + case slices.Contains(stale, v): + out[v] = disposition{outcomeWarning, fmt.Sprintf("#%d is still on batch #%d; take it out with backport-batch run --defer: %s", v.o.number, b.pr.Number, named)} + case v.rule != 0: + case v.onBatch != "": + if cut { + out[v] = disposition{outcomeUnmerged, fmt.Sprintf("#%d is only on open batch PR #%d, merge it before the cut: %s", v.o.number, b.pr.Number, named)} + } + case isDeferred && e.Workflows && cut: + out[v] = disposition{outcomeWaiting, fmt.Sprintf("#%d changes %s and waits on batch PR #%d for a maintainer, bring it over or defer it before the cut: %s", v.o.number, workflowsDir, b.pr.Number, named)} + case isDeferred && e.Workflows: + out[v] = disposition{outcomeWarning, fmt.Sprintf("waiting on #%d for a maintainer, since it changes %s: %s", b.pr.Number, workflowsDir, named)} + case isDeferred && e.Reason != "": + out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred on #%d to the next batch: %s (%s)", b.pr.Number, named, terminalSafe(e.Reason))} + case isDeferred: + out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred conflict on #%d: %s", b.pr.Number, named)} + default: + out[v] = disposition{outcomeMissing, named} + } + } + return out, nil +} + // terminalSafe keeps a title from carrying escape sequences into a terminal. func terminalSafe(s string) string { return capped(oneLine(clean(s, false)), titleLimit) } diff --git a/cmd/backport-batch/check_test.go b/cmd/backport-batch/check_test.go index 36cf253e8..5f4bf4f86 100644 --- a/cmd/backport-batch/check_test.go +++ b/cmd/backport-batch/check_test.go @@ -186,6 +186,179 @@ func TestCheck(t *testing.T) { } } +// Every category an original can be in, through the commands that put it +// there, with what check and check --cut make of it: the exit code, and the +// warning or note on stderr that leaves it 0. +func TestCheckDispositions(t *testing.T) { + plain := func(t *testing.T) *fixture { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + return f + } + // conflicting has #5 conflict on the line, beside #6, which picks. + conflicting := func(t *testing.T) *fixture { + f := newFixture(t) + f.lines() + f.onLine("release-1.1", "fix(app): a line-only five", map[string]string{"app/five.txt": "line\n"}) + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + return f + } + must := func(t *testing.T, f *fixture, gh *fakeGH, args ...string) { + t.Helper() + if code, _, stderr := f.exec(gh, args...); code != 0 { + t.Fatalf("%q = %d\n%s", args, code, stderr) + } + f.git("switch", "-q", "main") + if len(gh.bodies) > 0 { + reopen(f, gh) + } + } + sixOnLine := func(f *fixture) { pickOnLine(f, 6, map[string]string{"app/six.txt": "6\n"}) } + cases := []struct { + name string + arrange func(t *testing.T) (*fixture, *fakeGH) + line string + check int + cut int + checkErr string // on stderr from check + cutOut string // on stdout from check --cut + cutErr string // on stderr from check --cut + notCutOut string + }{ + { + name: "missing", + arrange: func(t *testing.T) (*fixture, *fakeGH) { return plain(t), &fakeGH{} }, + check: 1, cut: 1, cutOut: "/pull/5 ", + }, + { + name: "carried", + arrange: func(t *testing.T) (*fixture, *fakeGH) { + f := plain(t) + pickOnLine(f, 5, map[string]string{"app/five.txt": "5\n"}) + sixOnLine(f) + return f, &fakeGH{} + }, + check: 0, cut: 0, notCutOut: "/pull/5", + }, + { + name: "picked, on the open batch", + arrange: func(t *testing.T) (*fixture, *fakeGH) { + f, gh := plain(t), &fakeGH{} + openBatch(t, f, gh) + return f, gh + }, + check: 0, cut: 1, cutOut: "#5 is only on open batch PR #90", + }, + { + name: "deferred conflict, open record", + arrange: func(t *testing.T) (*fixture, *fakeGH) { + f, gh := conflicting(t), &fakeGH{} + openBatch(t, f, gh) + sixOnLine(f) + return f, gh + }, + check: 0, cut: 0, checkErr: "warning: deferred conflict on #90: https://github.com/cozystack/cozystack/pull/5 ", cutErr: "warning: deferred conflict on #90", + }, + { + name: "deferred with a reason, open record", + arrange: func(t *testing.T) (*fixture, *fakeGH) { + f, gh := plain(t), &fakeGH{} + openBatch(t, f, gh) + must(t, f, gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "fails the app e2e", "--push") + sixOnLine(f) + return f, gh + }, + check: 0, cut: 0, checkErr: "warning: deferred on #90 to the next batch: https://github.com/cozystack/cozystack/pull/5 ", cutErr: "(fails the app e2e)", + }, + { + name: "workflow wait, open record, beside a pick", + arrange: func(t *testing.T) (*fixture, *fakeGH) { + f, gh := workflowWorld(t), &fakeGH{} + must(t, f, gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push") + return f, gh + }, + check: 0, cut: 1, checkErr: "warning: waiting on #90 for a maintainer", cutOut: "#5 changes .github/workflows/ and waits on batch PR #90", + }, + { + name: "workflow wait, open record, alone", + arrange: func(t *testing.T) (*fixture, *fakeGH) { + f, gh := workflowWorld(t), &fakeGH{} + must(t, f, gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push") + pickOnLine(f, 6, map[string]string{"app/six.txt": "6\n", ".github/renovate.json": "{}\n"}) + return f, gh + }, + check: 0, cut: 1, checkErr: "warning: waiting on #90 for a maintainer", cutOut: "#5 changes .github/workflows/ and waits on batch PR #90", + }, + { + name: "backport/skip, not picked", + arrange: func(t *testing.T) (*fixture, *fakeGH) { + f := plain(t) + sixOnLine(f) + return f, &fakeGH{labels: map[string][]int{labelSkip: {5}}} + }, + check: 0, cut: 0, notCutOut: "/pull/5", + }, + { + name: "backport/skip after the pick, still on the open batch", + arrange: func(t *testing.T) (*fixture, *fakeGH) { + f, gh := plain(t), &fakeGH{} + openBatch(t, f, gh) + sixOnLine(f) + gh.labels = map[string][]int{labelSkip: {5}} + return f, gh + }, + check: 0, cut: 0, checkErr: "warning: #5 is still on batch #90; take it out with backport-batch run --defer", cutErr: "still on batch #90", + }, + { + name: "rule 6", + arrange: func(t *testing.T) (*fixture, *fakeGH) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + return f, &fakeGH{} + }, + line: "release-1.0", + check: 0, cut: 0, notCutOut: "/pull/5", + }, + { + name: "rules 2 to 4", + arrange: func(t *testing.T) (*fixture, *fakeGH) { + f := newFixture(t) + f.lines() + f.pr(5, "feat(app): five", "", map[string]string{"app/five.txt": "5\n"}) + return f, &fakeGH{} + }, + check: 0, cut: 0, notCutOut: "/pull/5", + }, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f, gh := tc.arrange(t) + line := tc.line + if line == "" { + line = "release-1.1" + } + code, stdout, stderr := f.exec(gh, "check", "--line", line) + if code != tc.check || !strings.Contains(stderr, tc.checkErr) { + t.Errorf("check = %d, want %d with %q\nstdout:\n%s\nstderr:\n%s", code, tc.check, tc.checkErr, stdout, stderr) + } + code, stdout, stderr = f.exec(gh, "check", "--line", line, "--cut") + if code != tc.cut || !strings.Contains(stdout, tc.cutOut) || !strings.Contains(stderr, tc.cutErr) { + t.Errorf("check --cut = %d, want %d with %q, %q\nstdout:\n%s\nstderr:\n%s", code, tc.cut, tc.cutOut, tc.cutErr, stdout, stderr) + } + if tc.notCutOut != "" && strings.Contains(stdout, tc.notCutOut) { + t.Errorf("check --cut lists %q:\n%s", tc.notCutOut, stdout) + } + if tc.cut == 0 && !strings.Contains(stdout, "no change the policy takes is missing before the cut") { + t.Errorf("check --cut passes without saying so:\n%s", stdout) + } + }) + } +} + type errString string func (e errString) Error() string { return string(e) } diff --git a/cmd/backport-batch/cut_test.go b/cmd/backport-batch/cut_test.go new file mode 100644 index 000000000..5d3e33e14 --- /dev/null +++ b/cmd/backport-batch/cut_test.go @@ -0,0 +1,64 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "strings" + "testing" +) + +// pickOnLine lands original n on release-1.1 as a maintainer's pick of its +// merge commit, with the -x line git appends, so the line carries it. +func pickOnLine(f *fixture, n int, files map[string]string) { + f.t.Helper() + mc, _, _ := strings.Cut(f.git("log", "--first-parent", "--format=%H", "--grep", fmt.Sprintf("(#%d)$", n), "main"), "\n") + f.onLine("release-1.1", fmt.Sprintf("fix: original %d\n\n(cherry picked from commit %s)", n, mc), files) +} + +// A workflow original that waits on the open batch fails the cut, also when +// it is all the batch has left: the pick beside it was labelled +// backport/skip after the pick, or reached the line by hand. +func TestCutFailsWhenOnlyAWorkflowWaitIsLeft(t *testing.T) { + for _, tc := range []struct { + name string + leave func(f *fixture, gh *fakeGH) + }{ + {"skip after the pick", func(_ *fixture, gh *fakeGH) { gh.labels = map[string][]int{labelSkip: {6}} }}, + {"picked onto the line by hand", func(f *fixture, _ *fakeGH) { + pickOnLine(f, 6, map[string]string{"app/six.txt": "6\n", ".github/renovate.json": "{}\n"}) + }}, + } { + t.Run(tc.name, func(t *testing.T) { + f := workflowWorld(t) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + tc.leave(f, gh) + code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut") + if code != 1 || !strings.Contains(stdout, "#5 changes .github/workflows/ and waits on batch PR #90 for a maintainer") { + t.Errorf("check --cut = %d, want 1 naming #5:\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } + if strings.Contains(stdout, "on the line or on the batch") { + t.Errorf("check --cut passes the line while #5 waits:\n%s", stdout) + } + }) + } +} diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index f93282daf..cb27b3e54 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -203,6 +203,8 @@ type batch struct { verdicts []*verdict pr *pullRequest prHead string + // picks is what the open batch pull request carries beyond the line. + picks *history // start is when the line started under the policy, zero for its fork point. start time.Time // textUntil is the batch-only day: what a commit's text says counts as @@ -300,13 +302,12 @@ func (a *app) prepare(l line) (*batch, error) { if err != nil { return nil, err } - var prPicks *history if pr != nil { b.pr = pr if b.prHead, err = batchHead(pr); err != nil { return nil, err } - if prPicks, err = readPicks(b.prHead, b.lineHead, b.textUntil); err != nil { + if b.picks, err = readPicks(b.prHead, b.lineHead, b.textUntil); err != nil { return nil, err } } @@ -315,8 +316,8 @@ func (a *app) prepare(l line) (*batch, error) { if err != nil { return nil, err } - if v.rule == 0 && prPicks != nil { - v.onBatch, _ = prPicks.found(v.o, v.commits) + if v.rule == 0 && b.picks != nil { + v.onBatch, _ = b.picks.found(v.o, v.commits) } b.verdicts = append(b.verdicts, v) } diff --git a/cmd/backport-batch/workflows_test.go b/cmd/backport-batch/workflows_test.go index 69f402974..35f2d9f14 100644 --- a/cmd/backport-batch/workflows_test.go +++ b/cmd/backport-batch/workflows_test.go @@ -173,8 +173,12 @@ func TestDeferKeepsWorkflowChangesWaiting(t *testing.T) { } f.git("switch", "-q", "main") reopen(f, gh) - if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); strings.Contains(stdout, "#5 changes") { - t.Errorf("check --cut = %d, %q; a deferred #5 still stops the cut as waiting", code, stdout) + // #6 and #7 are still only on the batch, which fails the cut; #5 does not. + if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || strings.Contains(stdout, "#5 changes") || !strings.Contains(stdout, "#6 is only on open batch PR #90") { + t.Errorf("check --cut = %d, %q; want 1 for #6 and #7 alone, a deferred #5 no longer waiting", code, stdout) + } + if code, _, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "deferred on #90 to the next batch: https://github.com/cozystack/cozystack/pull/5 ") { + t.Errorf("check = %d, want 0 with a warning about #5:\n%s", code, stderr) } } diff --git a/docs/release/backports.md b/docs/release/backports.md index c613cc980..1c50c7096 100644 --- a/docs/release/backports.md +++ b/docs/release/backports.md @@ -85,7 +85,7 @@ A run by hand trusts the open batch pull request the scheduled run opened as `co A deferred conflict waits for the next batch. To land it sooner, pick the original's commits onto the batch head with `git cherry-pick -x`, resolving each conflict, and join them to the batch head with a merge commit that names the original; an original squash-merged into `main` is one pick of its commit there, with no merge. Check that merge or that pick with `./backport-batch fold --line --original --commit `, and push it onto the batch branch, as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes. CI runs `backport-batch verify` on every push to a batch pull request, which holds every original on it to the same shape. When CI on the batch pull request fails, the original it comes from is taken out with `./backport-batch run --line --defer --reason --push` and goes to the next batch, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the rest of the batch merges without it. -Before a patch release or an rc is cut from a line, `./backport-batch check --line --cut` has to exit 0. A cut tags the line, so the check fails while a pull request the rules take is not on the line, including one that is only on the open batch pull request: merge the batch pull request first. It prints the URLs, and prints what the batch deferred as warnings, since deferring is a maintainer's call. Without `--cut`, as the scheduled run uses it, a change on the open batch pull request counts as carried. +Before a patch release or an rc is cut from a line, `./backport-batch check --line --cut` has to exit 0. A cut tags the line, so the check fails while a pull request the rules take is not on the line, including one that is only on the open batch pull request: merge the batch pull request first. It prints the URLs, and prints what the batch deferred as warnings, since deferring is a maintainer's call; so is a pull request labelled `backport/skip` that the open batch still carries. An original that waits on the batch for a maintainer fails the cut, also when it is all the batch has left. A cut that passes says so. Without `--cut`, as the scheduled run uses it, a change on the open batch pull request counts as carried. ### Changes to workflows From 015fc51c0d3687e5d29f9f59b214562f971bf8a5 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 9 Nov 2026 14:53:29 +0000 Subject: [PATCH 246/518] fix(release): name policy rules instead of internal numbers in output fold, run --defer and the run's own stderr printed "rule 101" for an original merged before its line started under the policy, and "rule 100" for backport/skip. Those numbers exist only inside the tool; the policy has six rules, so a reader looking for rule 101 finds nothing. Output now names such a rule in words, and the policy's own rules by their number. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/assemble.go | 2 +- cmd/backport-batch/body.go | 14 +++++++++++++- cmd/backport-batch/check.go | 2 +- cmd/backport-batch/check_test.go | 27 +++++++++++++++++++++++++++ cmd/backport-batch/defer.go | 2 +- 5 files changed, 43 insertions(+), 4 deletions(-) diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index ef26aace8..48c95428b 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -279,7 +279,7 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r } for _, rule := range left { for _, v := range r.skipped[rule] { - fmt.Fprintf(a.stderr, "skipped by rule %d: %s %s (%s)\n", rule, v.o.url(), terminalSafe(v.o.title), terminalSafe(v.reason)) + fmt.Fprintf(a.stderr, "skipped by %s: %s %s (%s)\n", ruleName(rule), v.o.url(), terminalSafe(v.o.title), terminalSafe(v.reason)) } } for _, p := range r.waiting { diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index a2d3a4c09..2f813233e 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -297,6 +297,18 @@ type report struct { shapes map[*verdict]string } +// ruleName is how output names what kept an original out: the policy's +// number for its own rules, and words for what the policy does not number. +func ruleName(rule int) string { + switch rule { + case ruleSkipLabel: + return labelSkip + case ruleBeforeStart: + return "rule 1, before the line's start" + } + return "rule " + strconv.Itoa(rule) +} + func (r *report) ruleTitle(rule int) string { switch rule { case ruleBeforeStart: @@ -417,7 +429,7 @@ func (r *report) outcome(v *verdict) string { case slices.ContainsFunc(r.waiting, func(p pick) bool { return p.v == v }): return "waiting for a maintainer: it changes " + codeSpan(workflowsDir) case v.rule != 0: - return "skipped by rule " + strconv.Itoa(v.rule) + ", " + titleSpan(v.reason) + return "skipped by " + ruleName(v.rule) + ", " + titleSpan(v.reason) } return "not picked" } diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index a565c3c45..1546a0197 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -209,7 +209,7 @@ func refuseFold(b *batch, n int, commit string) (string, error) { case v.rule == ruleSkipLabel: return fmt.Sprintf("#%d carries %s", n, labelSkip), nil case v.rule != 0: - return fmt.Sprintf("#%d is not one the batch takes: rule %d, %s", n, v.rule, v.reason), nil + return fmt.Sprintf("#%d is not one the batch takes: %s, %s", n, ruleName(v.rule), v.reason), nil case b.pr == nil: return fmt.Sprintf("no batch pull request is open on %s", b.line.branch()), nil case v.onBatch != "": diff --git a/cmd/backport-batch/check_test.go b/cmd/backport-batch/check_test.go index 5f4bf4f86..f93149f70 100644 --- a/cmd/backport-batch/check_test.go +++ b/cmd/backport-batch/check_test.go @@ -17,6 +17,7 @@ limitations under the License. package main import ( + "regexp" "strings" "testing" "unicode" @@ -359,6 +360,32 @@ func TestCheckDispositions(t *testing.T) { } } +// Output names a rule as the policy does: the numbers the tool keeps for +// what is no rule of the policy never reach a reader. +func TestOutputNamesNoInternalRule(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.setPolicy("release-1.1: 2099-01-01\n") + gh := &fakeGH{labels: map[string][]int{labelInclude: {6}}} + head := openBatch(t, f, gh) + internal := regexp.MustCompile(`rule 1[0-9][0-9]`) + for _, args := range [][]string{ + {"fold", "--line", "release-1.1", "--original", "5", "--commit", head}, + {"run", "--line", "release-1.1", "--defer", "5", "--reason", "x"}, + } { + code, stdout, stderr := f.exec(gh, args...) + f.git("switch", "-q", "main") + if code == 0 || !strings.Contains(stdout+stderr, "#5 is not one the batch takes: rule 1, before the line's start, merged ") { + t.Errorf("%s = %d, want a refusal naming rule 1:\n%s%s", args[0], code, stdout, stderr) + } + if m := internal.FindString(stdout + stderr); m != "" { + t.Errorf("%s prints %q:\n%s%s", args[0], m, stdout, stderr) + } + } +} + type errString string func (e errString) Error() string { return string(e) } diff --git a/cmd/backport-batch/defer.go b/cmd/backport-batch/defer.go index ab9741b00..1f0679844 100644 --- a/cmd/backport-batch/defer.go +++ b/cmd/backport-batch/defer.go @@ -82,7 +82,7 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { case v == nil: return a.fail(fmt.Errorf("#%d did not merge into main after %s branched", n, b.line.branch())) case v.rule != 0 && !slices.Contains(stale, v): - return a.fail(fmt.Errorf("#%d is not one the batch takes: rule %d, %s", n, v.rule, terminalSafe(v.reason))) + return a.fail(fmt.Errorf("#%d is not one the batch takes: %s, %s", n, ruleName(v.rule), terminalSafe(v.reason))) } hold[n] = reason } From 2046649b4cbcf064df9478698b348f18f05517eb Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 17:53:43 +0300 Subject: [PATCH 247/518] fix(clickhouse): match a backup Job's archive by its whole name The archive a backup Job carries was found as a substring of its Pod spec. The name without the BackupJob's UID starts the name with it, and a strategy may append to either, so a Job that wrote a longer archive could get the shorter name recorded on its Backup, and a restore of that Backup would read another archive. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../altinitystrategy_controller.go | 11 ++++--- .../altinitystrategy_controller_test.go | 33 +++++++++++++++---- 2 files changed, 33 insertions(+), 11 deletions(-) diff --git a/internal/backupcontroller/altinitystrategy_controller.go b/internal/backupcontroller/altinitystrategy_controller.go index cd03b1d46..c29162474 100644 --- a/internal/backupcontroller/altinitystrategy_controller.go +++ b/internal/backupcontroller/altinitystrategy_controller.go @@ -2,11 +2,11 @@ package backupcontroller import ( - "bytes" "context" "encoding/json" "fmt" "maps" + "regexp" "strings" "time" @@ -140,13 +140,16 @@ func altinityArchiveNameBeforeUIDs(j *backupsv1alpha1.BackupJob) string { // the archive name. A Job created by a controller that predates archive // names, or rendered from a strategy that names its own archives, wrote an // archive under another name, and recording this one would point the Backup -// at an archive that does not exist. +// at an archive that does not exist. The name has to stand on its own: the +// name without the UID starts the name with it, and a strategy may append to +// either. func altinityJobCarriesArchive(job *batchv1.Job, archive string) bool { spec, err := json.Marshal(job.Spec.Template.Spec) if err != nil { return false } - return bytes.Contains(spec, []byte(archive)) + whole := regexp.MustCompile(`(^|[^A-Za-z0-9._-])` + regexp.QuoteMeta(archive) + `($|[^A-Za-z0-9._-])`) + return whole.Match(spec) } // renderAltinityTemplate runs the strategy's PodTemplateSpec through the @@ -293,8 +296,6 @@ func (r *BackupJobReconciler) reconcileAltinity(ctx context.Context, j *backupsv if j.Status.BackupRef != nil { return ctrl.Result{}, nil } - // The name without the UID is a prefix of the one with it, so it is - // tried only after the full name is not found. if !altinityJobCarriesArchive(batchJob, archive) { archive = altinityArchiveNameBeforeUIDs(j) if !altinityJobCarriesArchive(batchJob, archive) { diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index 08c6e5466..15a450e59 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -835,6 +835,30 @@ func TestReconcileAltinity_RecordsTheArchiveTheJobWasGiven(t *testing.T) { // re-rendered, so its archive carries the old name, and the Backup has to // record that name rather than none. func TestReconcileAltinity_RecordsTheArchiveOfAJobNamedBeforeUIDs(t *testing.T) { + const legacy = "clickhouse-ch-test-20260304T050607Z" + cases := []struct { + name string + archive string // archive the Job's Pod was rendered with + want string + }{ + {name: "the name without the UID", archive: legacy, want: legacy}, + {name: "a longer name that starts with it", archive: legacy + "-copy", want: ""}, + {name: "a longer name that ends with it", archive: "copy-" + legacy, want: ""}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := recordAltinityArchiveOfJobRenderedWith(t, tc.archive); got != tc.want { + t.Errorf("Backup records archive %q, want %q", got, tc.want) + } + }) + } +} + +// recordAltinityArchiveOfJobRenderedWith completes a backup Job whose Pod +// carries the given archive in place of the one the controller rendered, and +// returns the archive the resulting Backup records. +func recordAltinityArchiveOfJobRenderedWith(t *testing.T, archive string) string { + t.Helper() app := newAltinityClickHouseApp("ch-test", "tenant-test") strategy := newAltinityArchiveStrategy("clickhouse-strategy") startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC)) @@ -855,7 +879,6 @@ func TestReconcileAltinity_RecordsTheArchiveOfAJobNamedBeforeUIDs(t *testing.T) } r, _, _ := newAltinityTestEnv(t, app, clientfake.NewClientBuilder().WithObjects(backupJob, strategy)) ctx := context.Background() - const legacy = "clickhouse-ch-test-20260304T050607Z" if _, err := r.reconcileAltinity(ctx, backupJob, resolved); err != nil { t.Fatalf("reconcileAltinity() creating the Job: %v", err) @@ -864,9 +887,9 @@ func TestReconcileAltinity_RecordsTheArchiveOfAJobNamedBeforeUIDs(t *testing.T) if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: jobNameForBackupJob(backupJob)}, job); err != nil { t.Fatalf("get backup Job: %v", err) } - job.Spec.Template.Spec.Containers[0].Args[1] = "--archive=" + legacy + job.Spec.Template.Spec.Containers[0].Args[1] = "--archive=" + archive if err := r.Update(ctx, job); err != nil { - t.Fatalf("give the backup Job the name without the UID: %v", err) + t.Fatalf("render the backup Job with archive %q: %v", archive, err) } job.Status.Conditions = []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}} if err := r.Status().Update(ctx, job); err != nil { @@ -883,9 +906,7 @@ func TestReconcileAltinity_RecordsTheArchiveOfAJobNamedBeforeUIDs(t *testing.T) if err := r.Get(ctx, client.ObjectKey{Namespace: "tenant-test", Name: backupJob.Name}, created); err != nil { t.Fatalf("get Backup: %v", err) } - if got := created.Spec.DriverMetadata[altinityArchiveMetaKey]; got != legacy { - t.Errorf("Backup records archive %q, want %q", got, legacy) - } + return created.Spec.DriverMetadata[altinityArchiveMetaKey] } // TestReconcileAltinity_RecordsNoArchiveTheJobWasNotGiven covers a Job whose From 3fc444874efb2a8d7d3be1f81fdfc5e7150fcc57 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 9 Nov 2026 14:54:47 +0000 Subject: [PATCH 248/518] fix(release): print the batch pull request's URL when --push publishes it run --push opened or updated the batch pull request and said nothing about which one, so the scheduled job's log and a maintainer's terminal had no link to follow. gh pr create prints the URL; the run now passes it on, on stderr, so stdout stays the body alone, which the scheduled script saves as it is. An update prints the open pull request's URL. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 2 +- cmd/backport-batch/assemble.go | 21 +++++++++++++++++++-- cmd/backport-batch/assemble_test.go | 16 ++++++++++++++-- cmd/backport-batch/fixture_test.go | 4 ++-- cmd/backport-batch/github.go | 18 ++++++++++++++---- cmd/backport-batch/overrides_test.go | 10 +++++++--- 6 files changed, 57 insertions(+), 14 deletions(-) diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index 60eac648d..665ced89a 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -36,7 +36,7 @@ The pull request body goes to stdout, built from the line's own `.github/PULL_RE `--defer N[,N] --reason TEXT` takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off `origin/` without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got `backport/skip` after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the `backport-batch` block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. -`--push` pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with `gh pr create --draft`, or updates the open batch pull request with `gh pr edit`. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the `run --onto --push` that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no `backport-batch` block either: someone wrote that body by hand, and the batch is theirs to merge or close. After `--defer` alone it replaces the branch, with `--force-with-lease=refs/heads/:`, where head is what `git ls-remote` reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. +`--push` pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with `gh pr create --draft`, or updates the open batch pull request with `gh pr edit`, and prints its URL on stderr. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the `run --onto --push` that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no `backport-batch` block either: someone wrote that body by hand, and the batch is theirs to merge or close. After `--defer` alone it replaces the branch, with `--force-with-lease=refs/heads/:`, where head is what `git ls-remote` reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. ### check diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index 48c95428b..d3fa66ac0 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -517,9 +517,26 @@ func (a *app) publish(b *batch, branch, head, title, body string, rebuilt bool) return err } if b.pr == nil { - return a.gh.createPull(b.line.branch(), branch, title, f.Name()) + url, err := a.gh.createPull(b.line.branch(), branch, title, f.Name()) + if err != nil { + return err + } + fmt.Fprintf(a.stderr, "opened draft %s\n", url) + return nil } - return a.gh.editPull(b.pr.Number, title, f.Name()) + if err := a.gh.editPull(b.pr.Number, title, f.Name()); err != nil { + return err + } + fmt.Fprintf(a.stderr, "updated %s\n", pullURL(b.pr)) + return nil +} + +// pullURL is where a pull request is on GitHub. +func pullURL(pr *pullRequest) string { + if pullURLRE.MatchString(pr.HTMLURL) { + return pr.HTMLURL + } + return fmt.Sprintf("%spull/%d", ourRepoURL, pr.Number) } // pushArgs pushes head to branch: as a fast-forward, which the remote checks, diff --git a/cmd/backport-batch/assemble_test.go b/cmd/backport-batch/assemble_test.go index 513ae6ff2..31dfd7dda 100644 --- a/cmd/backport-batch/assemble_test.go +++ b/cmd/backport-batch/assemble_test.go @@ -258,9 +258,17 @@ func TestRunPush(t *testing.T) { f.lines() f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) gh := &fakeGH{} - if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 0 { + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push") + if code != 0 { t.Fatalf("run --push = %d\n%s", code, stderr) } + if !strings.Contains(stderr, "opened draft https://github.com/cozystack/cozystack/pull/90\n") { + t.Errorf("the run does not print the pull request it opened:\n%s", stderr) + } + l, _ := parseLine("release-1.1") + if _, err := parseDeferred(stdout, l); err != nil || !strings.HasPrefix(stdout, "## What this PR does") || strings.Contains(stdout, "https://github.com/cozystack/cozystack/pull/90") { + t.Errorf("stdout is no longer the body alone (%v):\n%s", err, stdout) + } head := f.git("rev-parse", "HEAD") if got := f.remoteHead(batchBranch1); got != head { t.Errorf("remote %s = %q, want %s", batchBranch1, got, head) @@ -272,9 +280,13 @@ func TestRunPush(t *testing.T) { f.git("switch", "-q", "main") f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, gh.bodies[0])} - if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--onto", head, "--push"); code != 0 { + code, _, stderr = f.exec(gh, "run", "--line", "release-1.1", "--onto", head, "--push") + if code != 0 { t.Fatalf("run --onto --push = %d\n%s", code, stderr) } + if !strings.Contains(stderr, "updated https://github.com/cozystack/cozystack/pull/90\n") { + t.Errorf("the run does not print the pull request it updated:\n%s", stderr) + } if got := f.subjects("origin/release-1.1", f.remoteHead(batchBranch1)); !slices.Equal(got, []string{"fix(app): five (#5)", "fix(app): six (#6)"}) { t.Errorf("the pushed batch carries %q, want #5 then #6", got) } diff --git a/cmd/backport-batch/fixture_test.go b/cmd/backport-batch/fixture_test.go index cb14f8e7e..2ac7b4639 100644 --- a/cmd/backport-batch/fixture_test.go +++ b/cmd/backport-batch/fixture_test.go @@ -309,9 +309,9 @@ func (g *fakeGH) openPulls(base string) ([]pullRequest, error) { return out, g.err } -func (g *fakeGH) createPull(base, head, title, bodyFile string) error { +func (g *fakeGH) createPull(base, head, title, bodyFile string) (string, error) { g.created = append(g.created, base+" "+head+" "+title) - return g.readBody(bodyFile) + return fmt.Sprintf("https://github.com/%s/pull/90", upstreamRepo), g.readBody(bodyFile) } func (g *fakeGH) editPull(number int, title, bodyFile string) error { diff --git a/cmd/backport-batch/github.go b/cmd/backport-batch/github.go index b77ac0945..4b6a92507 100644 --- a/cmd/backport-batch/github.go +++ b/cmd/backport-batch/github.go @@ -21,6 +21,7 @@ import ( "fmt" "regexp" "strconv" + "strings" ) // openPullsCap is the page size of the open pull request listing. The REST @@ -43,6 +44,7 @@ var ( batchBranchRE = regexp.MustCompile(`^backport-batch/(release-\d+\.\d+)-(\d{8})$`) loginRE = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?$`) botLoginRE = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?\[bot\]$`) + pullURLRE = regexp.MustCompile(`^https://[^/\s]+/[^/\s]+/[^/\s]+/pull/[1-9][0-9]*$`) ) // batchBotEnv names the GitHub App bot the scheduled batch opens its pull @@ -107,7 +109,8 @@ type github interface { permission(login string) (string, error) // labelled lists the pull requests merged into main that carry label. labelled(label string) ([]int, error) - createPull(base, head, title, bodyFile string) error + // createPull opens the draft and returns its URL. + createPull(base, head, title, bodyFile string) (string, error) editPull(number int, title, bodyFile string) error comments(number int) ([]issueComment, error) createComment(number int, bodyFile string) error @@ -175,9 +178,16 @@ func (ghCLI) labelled(label string) ([]int, error) { return numbers, nil } -func (ghCLI) createPull(base, head, title, bodyFile string) error { - _, err := capture("gh", "pr", "create", "--draft", "--base", base, "--head", head, "--title", title, "--body-file", bodyFile) - return err +func (ghCLI) createPull(base, head, title, bodyFile string) (string, error) { + out, err := capture("gh", "pr", "create", "--draft", "--base", base, "--head", head, "--title", title, "--body-file", bodyFile) + if err != nil { + return "", err + } + url := strings.TrimSpace(out) + if !pullURLRE.MatchString(url) { + return "", fmt.Errorf("gh pr create printed %q, not the URL of a pull request", url) + } + return url, nil } func (ghCLI) editPull(number int, title, bodyFile string) error { diff --git a/cmd/backport-batch/overrides_test.go b/cmd/backport-batch/overrides_test.go index f1b1a5bcb..38eca5e0b 100644 --- a/cmd/backport-batch/overrides_test.go +++ b/cmd/backport-batch/overrides_test.go @@ -230,13 +230,17 @@ func TestGHCLI(t *testing.T) { // A batch pull request carries no label: the release workflows act on a // pull request cozystack-ci[bot] opened only when it has the release label. - write("") - if err := (ghCLI{}).createPull("release-1.7", batchBranch1, "fix(backport): batch", "/tmp/body.md"); err != nil { - t.Fatal(err) + write("https://github.com/cozystack/cozystack/pull/5300\n") + if url, err := (ghCLI{}).createPull("release-1.7", batchBranch1, "fix(backport): batch", "/tmp/body.md"); err != nil || url != "https://github.com/cozystack/cozystack/pull/5300" { + t.Errorf("createPull = %q, %v; want the URL gh printed", url, err) } if want := "pr create --draft --base release-1.7 --head " + batchBranch1 + " --title fix(backport): batch --body-file /tmp/body.md"; called() != want { t.Errorf("gh %q, want gh %q", called(), want) } + write("Warning: 1 uncommitted change\n") + if _, err := (ghCLI{}).createPull("release-1.7", batchBranch1, "fix(backport): batch", "/tmp/body.md"); err == nil { + t.Error("createPull took output that is no pull request URL") + } write("") if err := (ghCLI{}).editPull(90, "fix(backport): batch", "/tmp/body.md"); err != nil { t.Fatal(err) From 5945d2ddd11ab566f300759e78c907ba144b4c70 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 9 Nov 2026 15:03:43 +0000 Subject: [PATCH 249/518] feat(release): read a line's record from its newest batch pull request in any state The deferrals a batch recorded were read only from the open batch pull request. Once it merged, a conflict it had deferred showed as missing and failed check and the cut, until some later run picked something else and opened a new batch; a run that picked nothing opened none. run, check, fold and notify now find the line's record in the newest batch pull request whatever its state: one opened from a batch branch of this repository by an author the batch trusts. A record that is in doubt stops the command instead of letting it fall back to an older one: a block by an author no longer trusted, a bot's batch whose block was removed, a record that does not read whole, a closed record newer than the open batch, or no record within the pages read. The record is still written only to the open batch, and a merged or closed batch is never written again, so a run that picks nothing with no batch open keeps nothing and says so. The reader takes the version 2 record the next change writes; the writer stays at version 1. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 12 +- cmd/backport-batch/assemble.go | 10 +- cmd/backport-batch/batchbot_test.go | 9 +- cmd/backport-batch/body.go | 51 +--- cmd/backport-batch/check.go | 29 +- cmd/backport-batch/check_test.go | 12 +- cmd/backport-batch/cut_test.go | 54 ++++ cmd/backport-batch/defer.go | 6 +- cmd/backport-batch/fixture_test.go | 84 +++++- cmd/backport-batch/github.go | 53 +++- cmd/backport-batch/history_test.go | 3 + cmd/backport-batch/main.go | 6 + cmd/backport-batch/notify.go | 15 +- cmd/backport-batch/notify_test.go | 31 ++- cmd/backport-batch/overrides_test.go | 7 + cmd/backport-batch/record.go | 397 +++++++++++++++++++++++++++ cmd/backport-batch/record_test.go | 244 ++++++++++++++++ docs/release/backports.md | 4 + 18 files changed, 947 insertions(+), 80 deletions(-) create mode 100644 cmd/backport-batch/record.go create mode 100644 cmd/backport-batch/record_test.go diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index 665ced89a..945d6fac1 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -40,7 +40,7 @@ The pull request body goes to stdout, built from the line's own `.github/PULL_RE ### check -Exits 1 and prints the URL of every pull request the rules take that is neither on the line nor on the open batch pull request, 0 when there is none, and 2 when it cannot tell. A pull request that the open batch lists as deferred, a conflict or one taken out with `--defer`, prints as a warning and does not fail the check: deferring is a maintainer's call. So does one it lists as waiting for a maintainer because it changes workflows. +Exits 1 and prints the URL of every pull request the rules take that is neither on the line nor on the open batch pull request, 0 when there is none, and 2 when it cannot tell. A pull request that the [line's record](#the-lines-record) lists as deferred, a conflict or one taken out with `--defer`, prints as a warning and does not fail the check: deferring is a maintainer's call. That holds once the batch that recorded it merged or closed too. So does one the open batch lists as waiting for a maintainer because it changes workflows; once that batch is gone, nothing lists it any more, and it is missing. `--cut` makes it the gate before a patch release or an rc is cut from a line. A cut tags the line, so a pull request that is only on the open batch pull request fails it too, with one line each: `#N is only on open batch PR #M, merge it before the cut`. One that waits there for a maintainer fails it as well, since nobody decided to leave it out, also when it is all the batch has left: bring it over, or take it out with `run --defer` and a reason. Everything else is as without it, and a cut that passes says `no change the policy takes is missing before the cut`. The scheduled run checks without `--cut`, since its batch is open by design. @@ -56,7 +56,7 @@ Holds every unit of a batch to the shape fold checks for one, the clean picks as ### notify -Reads the open batch pull request of the line, trusted as described below, and prints one line for each thing that waits on it: a batch open for three days or more, the shortest gap between two scheduled runs, so one the previous run already saw; the conflicts it deferred; and the originals waiting for a maintainer because they change workflows. Nothing open prints nothing. +Reads the [line's record](#the-lines-record) first, as the other commands do, and exits 2 when it cannot read it or when it says the open batch is being closed, with a batch open or not. Then it reads the open batch pull request of the line, trusted as described below, and prints one line for each thing that waits on it: a batch open for three days or more, the shortest gap between two scheduled runs, so one the previous run already saw; the conflicts it deferred; and the originals waiting for a maintainer because they change workflows. Nothing open prints nothing. With `--push` it writes the same into one comment on the batch pull request: the conflicts, each to be resolved as the [conflict task](../../docs/release/backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows. @@ -96,6 +96,14 @@ Only the line's own commits count, the ones after the point where it branched fr Titles and bodies are read from the merge commits on `main`, which hold them as a maintainer merged them; the live pull request stays editable by its author. +## The line's record + +The line's record is the `backport-batch` block of its newest batch pull request: the newest pull request against the line from a `backport-batch/-` branch of this repository, opened by an author the batch trusts as above, whether it is open, merged or closed. `run`, `check`, `fold` and `notify` find it in GitHub's list of the line's pull requests, newest first, reading at most ten pages of 100. A batch pull request a person opened without the block is passed over. A block whose author the batch does not trust, a bot's batch without its block, a block that cannot be read whole, a closed record newer than the open batch, or ten pages with no record stop the command with exit 2, naming the pull request, rather than fall back to an older record. + +The record is written only while its pull request is open: a merged or closed batch pull request is never edited or commented on again. What it recorded keeps counting, so a conflict or a deferral a merged batch recorded still prints as a warning. A workflow original it listed as waiting is missing, since no open batch lists it. A run that picks nothing while no batch is open has nowhere to write: it prints the body, says that its classification is not retained, and exits 0. + +The record is only as trustworthy as the body it sits in. Anyone with write access can edit any body, the bot's included, and neither the bot's authorship nor anything in the block proves who edited it last. A hand edit that leaves a readable record is believed; one that breaks it stops every command with exit 2, naming the pull request. When the newest batch pull request is deleted and no later one exists, nothing shows it was there: the older record governs, and what was recorded since is lost. + ## What CI verifies The tool does not build or test anything. CI on the batch pull request runs the line's own build and tests against what was picked, and `verify` against its shape, and a maintainer reviews and merges it. A deferred conflict waits for the next batch unless someone resolves it, checks the resolution with `fold` and pushes it, and then CI runs again. diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index d3fa66ac0..ad6de0a1b 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -147,10 +147,8 @@ func (a *app) run(args []string) int { // Continuing the open batch keeps out what was deferred from it by hand; // a new batch tries those again. held := map[int]deferredEntry{} - if carried != nil && b.pr != nil && b.pr.Head.Ref == branch { - if held, err = parseDeferred(b.pr.Body, l); err != nil { - return a.fail(fmt.Errorf("batch pull request #%d: %w", b.pr.Number, err)) - } + if carried != nil && b.src.open && b.pr.Head.Ref == branch { + held = b.src.rec.deferred() } if err := a.warnReset(branch, start); err != nil { return a.fail(err) @@ -292,8 +290,10 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r } if head == b.lineHead && b.pr == nil { // Picking nothing is not a failure of the run; check still reports - // whatever the line lacks. + // whatever the line lacks. Merged and closed batches are never + // written, so there is nowhere to keep what the run found. fmt.Fprintf(a.stderr, "%s carries nothing beyond %s; nothing to push\n", branch, b.line.branch()) + fmt.Fprintf(a.stderr, "no batch pull request is open on %s; this run's classification is not retained\n", b.line.branch()) return 0 } if err := a.publish(b, branch, head, title, body, rebuilt); err != nil { diff --git a/cmd/backport-batch/batchbot_test.go b/cmd/backport-batch/batchbot_test.go index 987a134c2..ff1cb342a 100644 --- a/cmd/backport-batch/batchbot_test.go +++ b/cmd/backport-batch/batchbot_test.go @@ -24,7 +24,8 @@ import ( // A maintainer's run by hand continues the batch pull request the scheduled // run opened as its bot, with nothing set: it edits that pull request rather -// than opening a second one for the line. +// than opening a second one for the line. A record another bot wrote is no +// record the batch trusts, and the run stops rather than reach past it. func TestHandRunContinuesTheBotsBatch(t *testing.T) { for _, tc := range []struct { name, author string @@ -44,12 +45,12 @@ func TestHandRunContinuesTheBotsBatch(t *testing.T) { f.now = f.now.AddDate(0, 0, 3) code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push") - if code != 0 { + if tc.continued && code != 0 || !tc.continued && (code != 2 || !strings.Contains(stderr, "batch pull request #90 (open) carries a ```backport-batch block but was opened by \"renovate[bot]\"")) { t.Fatalf("run --continue = %d\n%s", code, stderr) } continued := slices.Equal(gh.edited, []string{"#90 fix(backport): batch for release-1.1, 2026-10-26"}) && len(gh.created) == 1 - if continued != tc.continued { - t.Errorf("edited %q, created %q; want the batch of %s continued=%v\n%s", gh.edited, gh.created, tc.author, tc.continued, stderr) + if continued != tc.continued || len(gh.created) != 1 { + t.Errorf("edited %q, created %q; want the batch of %s continued=%v and no second one\n%s", gh.edited, gh.created, tc.author, tc.continued, stderr) } if !tc.continued && !strings.Contains(stderr, "ignoring #90") { t.Errorf("the run does not say it ignores #90:\n%s", stderr) diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index 2f813233e..bfb908d2d 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -220,54 +220,13 @@ func cleanFiles(files []string) []string { return out } -// parseDeferred reads what a batch body defers: exactly one block, for this -// line, naming each original once. +// parseDeferred reads what a batch body's record defers. func parseDeferred(body string, l line) (map[int]deferredEntry, error) { - var blocks []string - lines := strings.Split(strings.ReplaceAll(body, "\r\n", "\n"), "\n") - for i := 0; i < len(lines); i++ { - if strings.TrimSpace(lines[i]) != deferredFence { - continue - } - if i+2 >= len(lines) || strings.TrimSpace(lines[i+2]) != "```" { - return nil, fmt.Errorf("the %s block is not one line of JSON", deferredFence) - } - blocks = append(blocks, lines[i+1]) - i += 2 + rec, err := parseRecord(body, l) + if err != nil { + return nil, err } - switch len(blocks) { - case 0: - return map[int]deferredEntry{}, nil - case 1: - default: - return nil, fmt.Errorf("the body holds %d %s blocks", len(blocks), deferredFence) - } - dec := json.NewDecoder(strings.NewReader(blocks[0])) - dec.DisallowUnknownFields() - var rec deferredRecord - if err := dec.Decode(&rec); err != nil { - return nil, fmt.Errorf("reading the %s block: %w", deferredFence, err) - } - if rec.Line != l.branch() { - return nil, fmt.Errorf("the %s block is for %q, not %s", deferredFence, rec.Line, l.branch()) - } - out := map[int]deferredEntry{} - for _, d := range rec.Deferred { - if d.Original <= 0 { - return nil, fmt.Errorf("the %s block names original %d", deferredFence, d.Original) - } - if d.Commit != "" && checkSHA(d.Commit) != nil { - return nil, fmt.Errorf("the %s block names commit %q for #%d", deferredFence, d.Commit, d.Original) - } - if d.Workflows && (d.Commit != "" || d.Reason != "") { - return nil, fmt.Errorf("the %s block has #%d wait for a maintainer and defers it as well", deferredFence, d.Original) - } - if _, dup := out[d.Original]; dup { - return nil, fmt.Errorf("the %s block names #%d twice", deferredFence, d.Original) - } - out[d.Original] = d - } - return out, nil + return rec.deferred(), nil } // prTitle is built from the line and the date alone, never from carried text. diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index 1546a0197..9398a37e6 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -114,17 +114,25 @@ type disposition struct { // of its own; the others pass in silence. The text is what follows the line's // name, or the original alone for one that is missing. func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { - deferred := map[int]deferredEntry{} + if c := b.src.closingOpen(); c != nil { + return nil, fmt.Errorf("batch pull request #%d is being closed; backport-batch run --line %s --push finishes it", c.PR, b.line.branch()) + } var stale []*verdict if b.pr != nil { var err error - if deferred, err = parseDeferred(b.pr.Body, b.line); err != nil { - return nil, fmt.Errorf("batch pull request #%d: %w", b.pr.Number, err) - } if stale, err = keptOut(b, b.picks); err != nil { return nil, err } } + // A record a closed batch left still says what a maintainer decided, but + // nothing waits on a batch that is gone. + deferred, on := b.src.rec.deferred(), "" + if c := b.src.carrier; c != nil { + on = fmt.Sprintf("#%d", c.Number) + if !b.src.open { + on += " (" + prState(c) + ")" + } + } out := map[*verdict]disposition{} for _, v := range b.verdicts { named := v.o.url() + " " + terminalSafe(v.o.title) @@ -137,14 +145,16 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { if cut { out[v] = disposition{outcomeUnmerged, fmt.Sprintf("#%d is only on open batch PR #%d, merge it before the cut: %s", v.o.number, b.pr.Number, named)} } + case isDeferred && e.Workflows && !b.src.open: + out[v] = disposition{outcomeMissing, named} case isDeferred && e.Workflows && cut: - out[v] = disposition{outcomeWaiting, fmt.Sprintf("#%d changes %s and waits on batch PR #%d for a maintainer, bring it over or defer it before the cut: %s", v.o.number, workflowsDir, b.pr.Number, named)} + out[v] = disposition{outcomeWaiting, fmt.Sprintf("#%d changes %s and waits on batch PR %s for a maintainer, bring it over or defer it before the cut: %s", v.o.number, workflowsDir, on, named)} case isDeferred && e.Workflows: - out[v] = disposition{outcomeWarning, fmt.Sprintf("waiting on #%d for a maintainer, since it changes %s: %s", b.pr.Number, workflowsDir, named)} + out[v] = disposition{outcomeWarning, fmt.Sprintf("waiting on %s for a maintainer, since it changes %s: %s", on, workflowsDir, named)} case isDeferred && e.Reason != "": - out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred on #%d to the next batch: %s (%s)", b.pr.Number, named, terminalSafe(e.Reason))} + out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred on %s to the next batch: %s (%s)", on, named, terminalSafe(e.Reason))} case isDeferred: - out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred conflict on #%d: %s", b.pr.Number, named)} + out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred conflict on %s: %s", on, named)} default: out[v] = disposition{outcomeMissing, named} } @@ -183,6 +193,9 @@ func (a *app) fold(args []string) int { if err != nil { return a.fail(err) } + if c := b.src.closingOpen(); c != nil { + return a.fail(fmt.Errorf("batch pull request #%d is being closed; backport-batch run --line %s --push finishes it", c.PR, l.branch())) + } why, err := refuseFold(b, *number, *commit) if err != nil { return a.fail(err) diff --git a/cmd/backport-batch/check_test.go b/cmd/backport-batch/check_test.go index f93149f70..b011b18b8 100644 --- a/cmd/backport-batch/check_test.go +++ b/cmd/backport-batch/check_test.go @@ -141,12 +141,22 @@ func TestCheck(t *testing.T) { { name: "a batch opened by a member with read access claims nothing", arrange: func(s *state) { - batch(s, "body\n\n"+deferred6+"\n", s.mc5) + batch(s, "body\n\n", s.mc5) s.gh.perms = map[string]string{"maintainer": "read"} }, code: 1, listed: []string{"/pull/5 ", "/pull/6 "}, }, + { + // An older record is never taken in place of a newer one nobody + // can vouch for. + name: "a record by a member with read access stops it", + arrange: func(s *state) { + batch(s, "body\n\n"+deferred6+"\n", s.mc5) + s.gh.perms = map[string]string{"maintainer": "read"} + }, + code: 2, + }, { name: "a deferred list for another line", arrange: func(s *state) { batch(s, strings.Replace(deferred6, "release-1.1", "release-1.0", 1), s.mc5) }, diff --git a/cmd/backport-batch/cut_test.go b/cmd/backport-batch/cut_test.go index 5d3e33e14..02466b304 100644 --- a/cmd/backport-batch/cut_test.go +++ b/cmd/backport-batch/cut_test.go @@ -62,3 +62,57 @@ func TestCutFailsWhenOnlyAWorkflowWaitIsLeft(t *testing.T) { }) } } + +// A conflict the batch deferred is still deferred once that batch merged: +// the next run picks nothing, opens nothing and writes nothing to the merged +// pull request, and check reads the merged batch's record. +func TestCheckAfterTheBatchMerged(t *testing.T) { + f := newFixture(t) + f.lines() + f.onLine("release-1.1", "fix(app): a line-only five", map[string]string{"app/five.txt": "line\n"}) + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + mergeBatch(f, gh) + written := len(gh.created) + len(gh.edited) + len(gh.issueComments) + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push"); code != 0 { + t.Fatalf("run --continue --push = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + if got := len(gh.created) + len(gh.edited) + len(gh.issueComments); got != written || f.remoteHead(batchBranch1) != "" { + t.Errorf("the run picked nothing and still wrote: created %q, edited %q", gh.created, gh.edited) + } + for _, cut := range []bool{false, true} { + args := []string{"check", "--line", "release-1.1"} + if cut { + args = append(args, "--cut") + } + code, stdout, stderr := f.exec(gh, args...) + if code != 0 || !strings.Contains(stderr, "warning: deferred conflict on #90 (merged): https://github.com/cozystack/cozystack/pull/5 ") { + t.Errorf("%q = %d, want 0 with #5 a deferred conflict:\nstdout:\n%s\nstderr:\n%s", args, code, stdout, stderr) + } + } +} + +// A workflow original waits only while a batch is open to list it: once the +// batch that listed it merged, it is missing. +func TestWorkflowWaitAfterTheBatchMergedIsMissing(t *testing.T) { + f := workflowWorld(t) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + mergeBatch(f, gh) + for _, cut := range []bool{false, true} { + args := []string{"check", "--line", "release-1.1"} + if cut { + args = append(args, "--cut") + } + if code, stdout, stderr := f.exec(gh, args...); code != 1 || !strings.Contains(stdout, "/pull/5 ") { + t.Errorf("%q = %d, want 1 with #5 missing:\nstdout:\n%s\nstderr:\n%s", args, code, stdout, stderr) + } + } +} diff --git a/cmd/backport-batch/defer.go b/cmd/backport-batch/defer.go index 1f0679844..5f2b29c6a 100644 --- a/cmd/backport-batch/defer.go +++ b/cmd/backport-batch/defer.go @@ -59,9 +59,9 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { if err != nil { return a.fail(err) } - recorded, err := parseDeferred(b.pr.Body, b.line) - if err != nil { - return a.fail(fmt.Errorf("batch pull request #%d: %w", b.pr.Number, err)) + recorded := map[int]deferredEntry{} + if b.src.open { + recorded = b.src.rec.deferred() } hold := map[int]string{} for n, e := range recorded { diff --git a/cmd/backport-batch/fixture_test.go b/cmd/backport-batch/fixture_test.go index 2ac7b4639..23e157f9f 100644 --- a/cmd/backport-batch/fixture_test.go +++ b/cmd/backport-batch/fixture_test.go @@ -251,6 +251,46 @@ type fakeGH struct { issueComments []issueComment // bot is what BACKPORT_BATCH_BOT would say. bot string + // listed counts the pages of recentPulls read. + listed int + // closedWrites names every write the tool attempted on a pull request + // that is no longer open; exec fails the test on one. + closedWrites []string +} + +// isOpen says whether the fake has pull request n open, or does not know it. +func (g *fakeGH) isOpen(n int) bool { + for _, pr := range g.prs { + if pr.Number == n { + return pr.State == "" || pr.State == "open" + } + } + return true +} + +// write refuses a write to a pull request that is no longer open, which the +// tool must never attempt. +func (g *fakeGH) write(n int, what string) error { + if g.isOpen(n) { + return nil + } + g.closedWrites = append(g.closedWrites, fmt.Sprintf("%s on #%d", what, n)) + return fmt.Errorf("#%d is closed", n) +} + +// recentPulls pages every pull request against base, newest first, as the +// REST listing sorted by creation does; a higher number is a newer one. +func (g *fakeGH) recentPulls(base string, page int) ([]pullRequest, error) { + g.listed++ + var all []pullRequest + for _, pr := range g.prs { + if pr.Base.Ref == base { + all = append(all, pr) + } + } + slices.SortFunc(all, func(a, b pullRequest) int { return b.Number - a.Number }) + from := min(len(all), (page-1)*recentPullsPage) + return slices.Clone(all[from:min(len(all), from+recentPullsPage)]), g.err } func (g *fakeGH) batchBot() string { return g.bot } @@ -260,6 +300,9 @@ func (g *fakeGH) comments(int) ([]issueComment, error) { } func (g *fakeGH) createComment(number int, bodyFile string) error { + if err := g.write(number, "a comment"); err != nil { + return err + } data, err := os.ReadFile(bodyFile) var c issueComment c.ID, c.Body = int64(1000+len(g.issueComments)), string(data) @@ -302,7 +345,7 @@ func (g *fakeGH) permission(login string) (string, error) { func (g *fakeGH) openPulls(base string) ([]pullRequest, error) { var out []pullRequest for _, pr := range g.prs { - if pr.Base.Ref == base { + if pr.Base.Ref == base && (pr.State == "" || pr.State == "open") { out = append(out, pr) } } @@ -315,6 +358,9 @@ func (g *fakeGH) createPull(base, head, title, bodyFile string) (string, error) } func (g *fakeGH) editPull(number int, title, bodyFile string) error { + if err := g.write(number, "an edit"); err != nil { + return err + } g.edited = append(g.edited, fmt.Sprintf("#%d %s", number, title)) return g.readBody(bodyFile) } @@ -338,10 +384,40 @@ func (f *fixture) batchPR(n int, base, branch, body string) pullRequest { FullName string `json:"full_name"` }{upstreamRepo} pr.Base.Ref, pr.Base.Repo.FullName = base, upstreamRepo + pr.State = "open" return pr } -// exec runs the tool in the fixture with gh faked, on the fixture's date. +// botBatchPR is batchPR opened by the bot the scheduled batch runs as. +func (f *fixture) botBatchPR(gh *fakeGH, n int, base, branch, body string) pullRequest { + f.t.Helper() + gh.bot = defaultBatchBot + pr := f.batchPR(n, base, branch, body) + pr.User.Login, pr.User.Type, pr.AuthorAssociation = defaultBatchBot, "Bot", "CONTRIBUTOR" + return pr +} + +// mergeBatch merges the open batch into release-1.1 with a merge commit, as +// GitHub does, and deletes its branch, as the repository's +// delete_branch_on_merge does. The pull request stays, merged, with its body +// and comments. +func mergeBatch(f *fixture, gh *fakeGH) { + f.t.Helper() + f.git("fetch", "-q", "origin") + f.git("switch", "-q", "--detach", "origin/release-1.1") + f.git("merge", "-q", "--no-ff", "-m", "Merge pull request #90 from cozystack/"+batchBranch1, "origin/"+batchBranch1) + f.git("push", "-q", "origin", "HEAD:refs/heads/release-1.1", ":refs/heads/"+batchBranch1) + f.git("switch", "-q", "main") + for i := range gh.prs { + if gh.prs[i].Number == 90 { + at := "2026-10-26T12:00:00Z" + gh.prs[i].State, gh.prs[i].MergedAt = "closed", &at + } + } +} + +// exec runs the tool in the fixture with gh faked, on the fixture's date, and +// fails the test when the tool wrote to a pull request that is not open. func (f *fixture) exec(gh *fakeGH, args ...string) (int, string, string) { f.t.Helper() if gh == nil { @@ -349,7 +425,11 @@ func (f *fixture) exec(gh *fakeGH, args ...string) (int, string, string) { } var stdout, stderr bytes.Buffer a := &app{gh: gh, stdout: &stdout, stderr: &stderr, now: func() time.Time { return f.now }} + written := len(gh.closedWrites) code := a.main(args) + if len(gh.closedWrites) > written { + f.t.Errorf("%v wrote to a pull request that is not open: %q", args, gh.closedWrites[written:]) + } return code, stdout.String(), stderr.String() } diff --git a/cmd/backport-batch/github.go b/cmd/backport-batch/github.go index 4b6a92507..3a7fa39d4 100644 --- a/cmd/backport-batch/github.go +++ b/cmd/backport-batch/github.go @@ -28,6 +28,14 @@ import ( // listing stops at it in silence, so a full page is an error, not an answer. const openPullsCap = 100 +// recentPullsPage is the page size of the listing of every pull request +// against a line, newest first, and recentPullsPages how many pages of it a +// search for the line's record reads. +const ( + recentPullsPage = 100 + recentPullsPages = 10 +) + // searchResultCap is the most results GitHub search returns for one query, // however far it is paginated. gh pr list answers a --label filter through // search, so a listing that long has to be taken as cut short. @@ -68,10 +76,12 @@ func batchBotLogin(lookup func(string) (string, bool)) string { // pullRequest is the REST view of a pull request: unlike gh pr list, it // carries the author's association with the repository. type pullRequest struct { - Number int `json:"number"` - HTMLURL string `json:"html_url"` - Body string `json:"body"` - AuthorAssociation string `json:"author_association"` + Number int `json:"number"` + HTMLURL string `json:"html_url"` + Body string `json:"body"` + State string `json:"state"` + MergedAt *string `json:"merged_at"` + AuthorAssociation string `json:"author_association"` User struct { Login string `json:"login"` Type string `json:"type"` @@ -104,6 +114,9 @@ type issueComment struct { // github is the part of GitHub the tool talks to: gh outside tests. type github interface { openPulls(base string) ([]pullRequest, error) + // recentPulls is one page of every pull request against base, open, + // merged or closed, newest first. + recentPulls(base string, page int) ([]pullRequest, error) // permission is login's permission on the repository as GitHub's // legacy field gives it: admin, write, read or none. permission(login string) (string, error) @@ -139,6 +152,38 @@ func (ghCLI) openPulls(base string) ([]pullRequest, error) { return prs, nil } +func (ghCLI) recentPulls(base string, page int) ([]pullRequest, error) { + out, err := capture("gh", "api", "-X", "GET", "repos/{owner}/{repo}/pulls", + "-f", "state=all", "-f", "base="+base, "-f", "sort=created", "-f", "direction=desc", + "-f", "per_page="+strconv.Itoa(recentPullsPage), "-f", "page="+strconv.Itoa(page)) + if err != nil { + return nil, err + } + var prs []pullRequest + if err := json.Unmarshal([]byte(out), &prs); err != nil { + return nil, fmt.Errorf("reading the pull requests against %s: %w", base, err) + } + return prs, nil +} + +// permCache asks GitHub for a login's permission once per command: the open +// batch's author is asked when it is chosen and again when its record is. +type permCache struct { + github + perms map[string]string +} + +func (c *permCache) permission(login string) (string, error) { + if p, ok := c.perms[login]; ok { + return p, nil + } + p, err := c.github.permission(login) + if err == nil { + c.perms[login] = p + } + return p, err +} + func (ghCLI) permission(login string) (string, error) { if !loginRE.MatchString(login) { return "", fmt.Errorf("%q is not a GitHub login", login) diff --git a/cmd/backport-batch/history_test.go b/cmd/backport-batch/history_test.go index f753c258c..299016c61 100644 --- a/cmd/backport-batch/history_test.go +++ b/cmd/backport-batch/history_test.go @@ -202,6 +202,9 @@ func TestCarriedByTheBatch(t *testing.T) { {name: "the batch's bot", edit: func(pr *pullRequest, gh *fakeGH) { gh.bot = "cozystack-ci[bot]" pr.User.Login, pr.User.Type, pr.AuthorAssociation = "cozystack-ci[bot]", "Bot", "CONTRIBUTOR" + // The bot writes the record into every body it opens. + l, _ := parseLine("release-1.1") + pr.Body = deferredBlock(l, nil, nil, nil) }, carried: true, notAsked: true}, {name: "the batch's bot when no bot is configured", edit: func(pr *pullRequest, _ *fakeGH) { pr.User.Login, pr.User.Type = "cozystack-ci[bot]", "Bot" diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index cb27b3e54..b921656f7 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -102,6 +102,7 @@ func main() { func (a *app) main(args []string) int { a.stdout, a.stderr = textWriter{a.stdout}, textWriter{a.stderr} + a.gh = &permCache{github: a.gh, perms: map[string]string{}} if len(args) == 0 { fmt.Fprint(a.stderr, usageText) return 2 @@ -205,6 +206,8 @@ type batch struct { prHead string // picks is what the open batch pull request carries beyond the line. picks *history + // src is the line's record and where it was found. + src recordSource // start is when the line started under the policy, zero for its fork point. start time.Time // textUntil is the batch-only day: what a commit's text says counts as @@ -311,6 +314,9 @@ func (a *app) prepare(l line) (*batch, error) { return nil, err } } + if b.src, err = findLineRecord(a.gh, l, pr); err != nil { + return nil, err + } for _, o := range merges { v, err := judge(o, labels[o.number], b.start, lineHist, b.newer, newerHist) if err != nil { diff --git a/cmd/backport-batch/notify.go b/cmd/backport-batch/notify.go index 87b3a123f..80c5aa1da 100644 --- a/cmd/backport-batch/notify.go +++ b/cmd/backport-batch/notify.go @@ -70,6 +70,15 @@ func (a *app) notify(args []string) int { if err != nil { return a.fail(err) } + // With no batch open there is nothing to report on and nowhere to write, + // but a record nobody can read is said all the same. + src, err := findLineRecord(a.gh, l, pr) + if err != nil { + return a.fail(err) + } + if c := src.closingOpen(); c != nil { + return a.fail(fmt.Errorf("batch pull request #%d is being closed; backport-batch run --line %s --push finishes it", c.PR, l.branch())) + } if pr == nil { return 0 } @@ -77,9 +86,9 @@ func (a *app) notify(args []string) int { if err != nil { return a.fail(err) } - deferred, err := parseDeferred(pr.Body, l) - if err != nil { - return a.fail(fmt.Errorf("batch pull request #%d: %w", pr.Number, err)) + deferred := map[int]deferredEntry{} + if src.open { + deferred = src.rec.deferred() } var conflicts, waiting []int for n, e := range deferred { diff --git a/cmd/backport-batch/notify_test.go b/cmd/backport-batch/notify_test.go index 606953be5..6b91a4159 100644 --- a/cmd/backport-batch/notify_test.go +++ b/cmd/backport-batch/notify_test.go @@ -148,7 +148,8 @@ func TestNotifyFindsTheBotsComment(t *testing.T) { ours.User.Login, ours.User.Type = bot, "Bot" other := issueComment{ID: 8, Body: notifyMarker + "\nplanted"} other.User.Login, other.User.Type = "renovate[bot]", "Bot" - pr := notifyPR("20261019", "") + // The bot writes the record into every body it opens. + pr := notifyPR("20261019", deferredBlock(notifyLine, nil, nil, nil)) pr.User.Login, pr.User.Type = bot, "Bot" gh := &fakeGH{prs: []pullRequest{pr}, issueComments: []issueComment{ours, other}, bot: bot} if code, stdout, stderr := notifyRun(gh, "--push"); code != 0 || !strings.Contains(stdout, "has been open since") { @@ -172,7 +173,7 @@ func TestNotifyFailsOnAnUnreadablePermission(t *testing.T) { } const bot = "cozystack-ci[bot]" - pr := notifyPR("20261019", "") + pr := notifyPR("20261019", deferredBlock(notifyLine, nil, nil, nil)) pr.User.Login, pr.User.Type = bot, "Bot" ours := issueComment{ID: 7, Body: notifyMarker + "\nearlier"} ours.User.Login, ours.User.Type = "maintainer", "User" @@ -203,6 +204,32 @@ func TestNotifyRefuses(t *testing.T) { } } +// notify reads the line's record the way the other commands do, and stays +// silent with no batch open, but a record it cannot read, or one that says +// its open batch is being closed, stops it. +func TestNotifyValidatesTheRecord(t *testing.T) { + sha := strings.Repeat("a", 40) + closingRecord := strings.Replace(v2(""), `"closing":null`, `"closing":{"pr":91,"branch":"backport-batch/release-1.1-20261002","head":"`+sha+`"}`, 1) + block := func(json string) string { return "text\n\n" + deferredFence + "\n" + json + "\n```\n" } + for _, tc := range []struct { + name string + pr pullRequest + code int + quiet bool + }{ + {"a clean closed record", recordPR(91, "closed", "maintainer", block(v2(""))), 0, true}, + {"a malformed record", recordPR(91, "closed", "maintainer", block(`{"line":`)), 2, false}, + {"a closed record that was closing", recordPR(91, "closed", "maintainer", block(closingRecord)), 0, true}, + {"an open record that is closing", recordPR(91, "open", "maintainer", block(closingRecord)), 2, false}, + } { + gh := &fakeGH{prs: []pullRequest{tc.pr}} + code, stdout, stderr := notifyRun(gh, "--push") + if code != tc.code || tc.quiet && (stdout != "" || stderr != "") || len(gh.issueComments) != 0 { + t.Errorf("%s: notify = %d, %q, %q, %d comments; want %d and nothing written", tc.name, code, stdout, stderr, len(gh.issueComments), tc.code) + } + } +} + // The maintainer's example: with 1.6 and 1.7 released, release-1.8 joins them // when it branches, and 1.8.0 drops 1.6. func TestSupportedLines(t *testing.T) { diff --git a/cmd/backport-batch/overrides_test.go b/cmd/backport-batch/overrides_test.go index 38eca5e0b..740713359 100644 --- a/cmd/backport-batch/overrides_test.go +++ b/cmd/backport-batch/overrides_test.go @@ -196,6 +196,13 @@ func TestGHCLI(t *testing.T) { if want := "api -X GET repos/{owner}/{repo}/pulls -f state=open -f base=release-1.6 -f per_page=100"; called() != want { t.Errorf("gh %q, want gh %q", called(), want) } + write("[]") + if _, err := (ghCLI{}).recentPulls("release-1.6", 2); err != nil { + t.Fatal(err) + } + if want := "api -X GET repos/{owner}/{repo}/pulls -f state=all -f base=release-1.6 -f sort=created -f direction=desc -f per_page=100 -f page=2"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } var page []string for i := range openPullsCap { page = append(page, fmt.Sprintf(`{"number":%d}`, i+1)) diff --git a/cmd/backport-batch/record.go b/cmd/backport-batch/record.go new file mode 100644 index 000000000..648b000ba --- /dev/null +++ b/cmd/backport-batch/record.go @@ -0,0 +1,397 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "regexp" + "slices" + "strings" + "time" +) + +// A line's record is the backport-batch block of its newest batch pull +// request, read whatever that pull request's state and written only while it +// is open: a merged or closed pull request is never written again, so what it +// recorded stays what the line's later commands act on until a newer batch +// carries a record of its own. + +// lineRecord is the one line of JSON in the block. Version 1, the block +// before it had a version, holds only line and deferred. +type lineRecord struct { + Version int `json:"version"` + Line string `json:"line"` + Built *builtFrom `json:"built,omitempty"` + Report string `json:"report,omitempty"` + Closing *closing `json:"closing"` + Deferred []deferredEntry `json:"deferred"` + Dropped []droppedEntry `json:"dropped"` + Stale []int `json:"stale"` +} + +// builtFrom is what a classification was made of; batch is the head it +// describes. +type builtFrom struct { + Main string `json:"main"` + Line string `json:"line"` + Newer string `json:"newer"` + Policy string `json:"policy"` + Batch string `json:"batch"` + Tool string `json:"tool"` + Run string `json:"run"` + At string `json:"at"` + Options runOptions `json:"options"` +} + +type runOptions struct { + NoWorkflowFiles bool `json:"noWorkflowFiles"` + Mode string `json:"mode"` +} + +// The operations a record is written by: a run that assembles the batch, and +// the ones that only change what the record says about it. +const ( + modeNew = "new" + modeContinue = "continue" + modeDefer = "defer" + modeDrop = "drop" + modeUndrop = "undrop" + modeClose = "close" +) + +var modes = []string{modeNew, modeContinue, modeDefer, modeDrop, modeUndrop, modeClose} + +// closing is written into the record of an open batch that the run is about +// to close: the head it froze, which the close and the branch's deletion are +// leased on. +type closing struct { + PR int `json:"pr"` + Branch string `json:"branch"` + Head string `json:"head"` +} + +// droppedEntry is a maintainer's decision that an original stays off the +// line, bound to its merge commit on main. +type droppedEntry struct { + Original int `json:"original"` + Merge string `json:"merge"` + Reason string `json:"reason"` + On int `json:"on"` + Date string `json:"date"` +} + +var ( + toolRE = regexp.MustCompile(`^[0-9a-f]{40}(\+dirty)?$`) + reportIDRE = regexp.MustCompile(`^[0-9a-f]{64}$`) +) + +// fencedLines lists the one line of each block that fence opens. +func fencedLines(body, fence string) ([]string, error) { + var blocks []string + lines := strings.Split(strings.ReplaceAll(body, "\r\n", "\n"), "\n") + for i := 0; i < len(lines); i++ { + if strings.TrimSpace(lines[i]) != fence { + continue + } + if i+2 >= len(lines) || strings.TrimSpace(lines[i+2]) != "```" { + return nil, fmt.Errorf("the %s block is not one line of JSON", fence) + } + blocks = append(blocks, lines[i+1]) + i += 2 + } + return blocks, nil +} + +// decodeStrict reads exactly one JSON value into v, refusing a key v does not +// have and anything after the value. +func decodeStrict(data string, v any) error { + dec := json.NewDecoder(strings.NewReader(data)) + dec.DisallowUnknownFields() + if err := dec.Decode(v); err != nil { + return err + } + var rest json.RawMessage + if dec.More() || dec.Decode(&rest) != io.EOF { + return errors.New("something follows the JSON value") + } + return nil +} + +// parseRecord reads the line's record from a batch body: nil when the body +// has no block, an error when it has more than one or one it cannot vouch +// for whole. +func parseRecord(body string, l line) (*lineRecord, error) { + blocks, err := fencedLines(body, deferredFence) + switch { + case err != nil: + return nil, err + case len(blocks) == 0: + return nil, nil + case len(blocks) > 1: + return nil, fmt.Errorf("the body holds %d %s blocks", len(blocks), deferredFence) + } + var head struct { + Version *int `json:"version"` + } + if err := json.Unmarshal([]byte(blocks[0]), &head); err != nil { + return nil, fmt.Errorf("reading the %s block: %w", deferredFence, err) + } + version := 1 + if head.Version != nil { + version = *head.Version + } + var rec lineRecord + switch version { + case 1: + var v1 struct { + Version int `json:"version"` + Line string `json:"line"` + Deferred []deferredEntry `json:"deferred"` + } + err = decodeStrict(blocks[0], &v1) + rec = lineRecord{Version: 1, Line: v1.Line, Deferred: v1.Deferred} + case 2: + err = decodeStrict(blocks[0], &rec) + default: + return nil, fmt.Errorf("the %s block is version %d, which this backport-batch does not read; it reads 1 and 2", deferredFence, version) + } + if err != nil { + return nil, fmt.Errorf("reading the %s block: %w", deferredFence, err) + } + if err := rec.validate(l); err != nil { + return nil, fmt.Errorf("the %s block %w", deferredFence, err) + } + return &rec, nil +} + +// validate holds the record to what a run writes. Its errors read after +// "the block". +func (rec *lineRecord) validate(l line) error { + if rec.Line != l.branch() { + return fmt.Errorf("is for %q, not %s", rec.Line, l.branch()) + } + seen := map[int]bool{} + once := func(n int) error { + if n <= 0 { + return fmt.Errorf("names original %d", n) + } + if seen[n] { + return fmt.Errorf("names #%d twice", n) + } + seen[n] = true + return nil + } + for _, d := range rec.Deferred { + if err := once(d.Original); err != nil { + return err + } + if d.Commit != "" && checkSHA(d.Commit) != nil { + return fmt.Errorf("names commit %q for #%d", d.Commit, d.Original) + } + // A conflict names the commit that failed and its files, a deferral + // its reason, and a wait its workflow files. + kinds := 0 + for _, is := range []bool{d.Commit != "" || len(d.Files) > 0 && !d.Workflows && d.Reason == "", d.Reason != "", d.Workflows} { + if is { + kinds++ + } + } + switch { + case d.Workflows && (d.Commit != "" || d.Reason != ""): + return fmt.Errorf("has #%d wait for a maintainer and defers it as well", d.Original) + case kinds != 1: + return fmt.Errorf("does not say whether #%d is a conflict, deferred for a reason or waiting", d.Original) + case d.Reason != "" && strings.TrimSpace(d.Reason) == "": + return fmt.Errorf("defers #%d for an empty reason", d.Original) + case d.More < 0: + return fmt.Errorf("counts %d more files for #%d", d.More, d.Original) + } + } + for _, d := range rec.Dropped { + if err := once(d.Original); err != nil { + return err + } + if _, err := time.Parse(time.DateOnly, d.Date); err != nil || checkSHA(d.Merge) != nil || strings.TrimSpace(d.Reason) == "" || d.On <= 0 { + return fmt.Errorf("drops #%d with a merge, reason, pull request or date it cannot have", d.Original) + } + } + staleSeen := map[int]bool{} + for _, n := range rec.Stale { + if n <= 0 || staleSeen[n] { + return fmt.Errorf("lists %d as still on the batch more than once or not at all", n) + } + staleSeen[n] = true + } + if rec.Version < 2 { + return nil + } + if b := rec.Built; b == nil { + return errors.New("does not say what it was built from") + } else if err := b.validate(); err != nil { + return err + } + if !reportIDRE.MatchString(rec.Report) { + return fmt.Errorf("names report %q", rec.Report) + } + if c := rec.Closing; c != nil { + m := batchBranchRE.FindStringSubmatch(c.Branch) + if c.PR <= 0 || m == nil || m[1] != l.branch() || checkSHA(c.Head) != nil { + return fmt.Errorf("closes #%d on %q at %q, which is no batch of %s", c.PR, c.Branch, c.Head, l.branch()) + } + } + return nil +} + +func (b *builtFrom) validate() error { + for _, sha := range []string{b.Main, b.Line, b.Policy, b.Batch} { + if checkSHA(sha) != nil { + return fmt.Errorf("says it was built from %q", sha) + } + } + if b.Newer != "" && checkSHA(b.Newer) != nil { + return fmt.Errorf("says it was built from %q", b.Newer) + } + if b.Tool != "" && !toolRE.MatchString(b.Tool) { + return fmt.Errorf("names tool %q", b.Tool) + } + if b.Run != "" && !runURLRE.MatchString(b.Run) { + return fmt.Errorf("names run %q", b.Run) + } + if _, err := time.Parse(time.RFC3339, b.At); err != nil { + return fmt.Errorf("was built at %q", b.At) + } + if !slices.Contains(modes, b.Options.Mode) { + return fmt.Errorf("was written by %q, which is no operation of backport-batch", b.Options.Mode) + } + return nil +} + +var runURLRE = regexp.MustCompile(`^https://[^/\s]+/[^/\s]+/[^/\s]+/actions/runs/[0-9]+(/attempts/[0-9]+)?$`) + +// deferred maps the originals the record defers to their entries. +func (rec *lineRecord) deferred() map[int]deferredEntry { + out := map[int]deferredEntry{} + if rec != nil { + for _, d := range rec.Deferred { + out[d.Original] = d + } + } + return out +} + +// recordSource is where a line's record was found. +type recordSource struct { + // carrier is the pull request whose block is the record, nil when the + // line has none. + carrier *pullRequest + // open says the carrier is the open batch, so the record may be written. + open bool + rec *lineRecord + // code is the open batch pull request, if any. One whose body someone + // wrote by hand is the batch the line builds on but carries no record. + code *pullRequest +} + +// closingOpen is the close the record says is under way on a batch that is +// still open, or nil. +func (s recordSource) closingOpen() *closing { + if !s.open || s.rec == nil { + return nil + } + return s.rec.Closing +} + +func prState(pr *pullRequest) string { + switch { + case pr.MergedAt != nil: + return "merged" + case pr.State == "": + return "open" + } + return pr.State +} + +func hasRecord(body string) bool { + return slices.ContainsFunc(strings.Split(body, "\n"), func(ln string) bool { return strings.TrimSpace(ln) == deferredFence }) +} + +// findLineRecord finds the line's record: the block of the newest pull +// request opened from a batch branch of this repository for the line by an +// author the batch trusts, open, merged or closed. open is the open batch +// pull request batchPull chose, or nil. Anything that leaves in doubt which +// record is the newest is an error, never a fall back to an older one: a +// newer block by an author the batch does not trust, a block the bot wrote +// and someone removed, a record it cannot read, a closed record newer than +// the open batch, or more pull requests than it reads. +func findLineRecord(gh github, l line, open *pullRequest) (recordSource, error) { + src := recordSource{code: open} + for page := 1; src.carrier == nil; page++ { + if page > recentPullsPages { + return src, fmt.Errorf("none of the %d newest pull requests against %s is a batch pull request with a record, so the line's record cannot be told apart from none", recentPullsPages*recentPullsPage, l.branch()) + } + prs, err := gh.recentPulls(l.branch(), page) + if err != nil { + return src, err + } + for i := range prs { + pr := &prs[i] + m := batchBranchRE.FindStringSubmatch(pr.Head.Ref) + if m == nil || m[1] != l.branch() || pr.Base.Ref != l.branch() || pr.Head.Repo == nil || pr.Head.Repo.FullName != pr.Base.Repo.FullName { + continue + } + if open != nil && pr.Number == open.Number { + pr = open + } + if !hasRecord(pr.Body) { + if pr.User.Type == "Bot" && gh.batchBot() != "" && pr.User.Login == gh.batchBot() { + return src, fmt.Errorf("batch pull request #%d (%s) was opened by %s but carries no %s block; someone removed it, so the line's record is unknown", pr.Number, prState(pr), pr.User.Login, deferredFence) + } + continue + } + why, err := trusted(gh, *pr) + if err != nil { + return src, fmt.Errorf("batch pull request #%d: %w", pr.Number, err) + } + if why != "" { + return src, fmt.Errorf("batch pull request #%d (%s) carries a %s block but was %s; an older record is not taken in its place", pr.Number, prState(pr), deferredFence, why) + } + rec, err := parseRecord(pr.Body, l) + if err != nil { + return src, fmt.Errorf("batch pull request #%d (%s): %w", pr.Number, prState(pr), err) + } + src.carrier, src.rec = pr, rec + break + } + if len(prs) < recentPullsPage { + break + } + } + switch c := src.carrier; { + case open != nil && hasRecord(open.Body) && (c == nil || c.Number < open.Number): + return src, fmt.Errorf("GitHub does not list the open batch pull request #%d among the pull requests against %s; run again", open.Number, l.branch()) + case open != nil && hasRecord(open.Body) && c.Number > open.Number: + return src, fmt.Errorf("batch pull request #%d (%s) carries a record newer than the open batch pull request #%d; close one of them by hand", c.Number, prState(c), open.Number) + case open != nil && c == open: + src.open = true + case c != nil && prState(c) == "open": + return src, fmt.Errorf("batch pull request #%d carries the line's record and is open, but is not the batch this run builds on", c.Number) + } + return src, nil +} diff --git a/cmd/backport-batch/record_test.go b/cmd/backport-batch/record_test.go new file mode 100644 index 000000000..b512a2cda --- /dev/null +++ b/cmd/backport-batch/record_test.go @@ -0,0 +1,244 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "strings" + "testing" +) + +// The records the live batches carried before records had a version, as +// GitHub returned them on 2026-11-09, read as version 1. +func TestRecordReadsVersion1(t *testing.T) { + for line, block := range map[string]string{ + "release-1.7": `{"line":"release-1.7","deferred":[{"original":4756,"commit":"647407545d630291cfe2a20880cce45af22edf0c","files":["packages/system/backupstrategy-controller/values.yaml"]},{"original":4867,"commit":"813c6180861efe1896c26d62a7afcbc1ff590ebc","files":[".github/renovate.json"]},{"original":5082,"commit":"e93db8848027450c2e23d21348409efa641cb6f1","files":["pkg/registry/core/tap/rest.go"]},{"original":4888,"files":[".github/workflows/cut-prerelease.yaml"],"workflows":true},{"original":4971,"files":[".github/workflows/e2e-proxmox.yaml"],"workflows":true}]}`, + "release-1.6": `{"line":"release-1.6","deferred":[{"original":5071,"commit":"3fa534f3cf6cb5774a183f0202f7f5c13e9ae211","files":["internal/operator/tapmaterializer_reconciler.go","internal/operator/tapmaterializer_reconciler_test.go"]},{"original":5086,"commit":"5265ee662c591db409c8ef1ed8bb457cf29f5c22","files":["packages/apps/tenant/templates/gateway.yaml"]},{"original":5087,"commit":"8d8179f20a2c10de03face1472435d5083cde38d","files":["internal/controller/applicationdefinition_helmreconciler.go","internal/controller/applicationdefinition_helmreconciler_test.go"]},{"original":5142,"files":[".github/workflows/pull-requests-release.yaml",".github/workflows/update-releasenotes.yaml"],"workflows":true}]}`, + } { + l, _ := parseLine(line) + rec, err := parseRecord("## What this PR does\n\ntext\n\n"+deferredFence+"\n"+block+"\n```\n\n### Release note\n", l) + if err != nil { + t.Fatalf("%s: parseRecord = %v", line, err) + } + waits, conflicts := 0, 0 + for _, e := range rec.Deferred { + switch { + case e.Workflows: + waits++ + case e.Commit != "": + conflicts++ + } + } + if rec.Version != 1 || rec.Line != line || rec.Closing != nil || len(rec.Dropped) != 0 || waits+conflicts != len(rec.Deferred) || waits == 0 || conflicts == 0 { + t.Errorf("%s: record = %+v; want version 1 with its conflicts and waits", line, rec) + } + } +} + +// v2 is a version 2 record with each value it may hold. +func v2(extra string) string { + sha := strings.Repeat("a", 40) + return `{"version":2,"line":"release-1.1","built":{"main":"` + sha + `","line":"` + sha + `","newer":"","policy":"` + sha + `","batch":"` + sha + `","tool":"` + sha + `+dirty","run":"","at":"2026-10-26T06:00:00Z","options":{"noWorkflowFiles":true,"mode":"continue"}},` + + `"report":"` + strings.Repeat("b", 64) + `","closing":null,"deferred":[{"original":5,"commit":"` + sha + `","files":["app/five.txt"]},{"original":6,"files":[],"reason":"fails the e2e"},{"original":7,"files":[".github/workflows/x.yaml"],"workflows":true}],` + + `"dropped":[{"original":8,"merge":"` + sha + `","reason":"needs #4","on":90,"date":"2026-10-26"}],"stale":[9]` + extra + `}` +} + +// A record is read whole or not at all: anything it cannot vouch for stops +// the command rather than being half believed. +func TestRecordRefuses(t *testing.T) { + l, _ := parseLine("release-1.1") + block := func(json string) string { return "text\n\n" + deferredFence + "\n" + json + "\n```\n" } + if rec, err := parseRecord(block(v2("")), l); err != nil || rec.Version != 2 || len(rec.Dropped) != 1 || rec.Built.Options.Mode != "continue" { + t.Fatalf("parseRecord of a whole v2 record = %+v, %v", rec, err) + } + sha := strings.Repeat("a", 40) + for name, json := range map[string]string{ + "version 4": strings.Replace(v2(""), `"version":2`, `"version":4`, 1), + "trailing data": v2("") + `{}`, + "trailing text": v2("") + ` x`, + "an original listed twice": strings.Replace(v2(""), `"original":8`, `"original":5`, 1), + "a needs entry with a commit": strings.Replace(v2(""), `{"original":6,"files":[],"reason":"fails the e2e"}`, `{"original":6,"commit":"`+sha+`","files":[],"needs":[4]}`, 1), + "an empty reason": strings.Replace(v2(""), `"reason":"fails the e2e"`, `"reason":""`, 1), + "an empty drop reason": strings.Replace(v2(""), `"reason":"needs #4"`, `"reason":" "`, 1), + "a bad tool": strings.Replace(v2(""), `+dirty"`, `-dirty"`, 1), + "an unknown mode": strings.Replace(v2(""), `"mode":"continue"`, `"mode":"rebase"`, 1), + "a reason with a commit": strings.Replace(v2(""), `"reason":"fails the e2e"`, `"reason":"x","commit":"`+sha+`"`, 1), + "a short merge": strings.Replace(v2(""), `"merge":"`+sha+`"`, `"merge":"abc"`, 1), + "a bad date": strings.Replace(v2(""), `"date":"2026-10-26"`, `"date":"26.10.2026"`, 1), + "a bad time": strings.Replace(v2(""), `"at":"2026-10-26T06:00:00Z"`, `"at":"yesterday"`, 1), + "a short report": strings.Replace(v2(""), strings.Repeat("b", 64), "bbbb", 1), + "a closing on another line": strings.Replace(v2(""), `"closing":null`, `"closing":{"pr":90,"branch":"backport-batch/release-1.0-20261026","head":"`+sha+`"}`, 1), + "a closing without a head": strings.Replace(v2(""), `"closing":null`, `"closing":{"pr":90,"branch":"`+batchBranch1+`","head":""}`, 1), + "a negative stale": strings.Replace(v2(""), `"stale":[9]`, `"stale":[-9]`, 1), + "an unknown key": v2(`,"parked":[]`), + "an entry of no kind": strings.Replace(v2(""), `{"original":6,"files":[],"reason":"fails the e2e"}`, `{"original":6,"files":[]}`, 1), + "another line": strings.Replace(v2(""), `"line":"release-1.1"`, `"line":"release-1.0"`, 1), + } { + if _, err := parseRecord(block(json), l); err == nil { + t.Errorf("%s: parseRecord accepted it", name) + } + } + closing := strings.Replace(v2(""), `"closing":null`, `"closing":{"pr":90,"branch":"`+batchBranch1+`","head":"`+sha+`"}`, 1) + if rec, err := parseRecord(block(closing), l); err != nil || rec.Closing == nil || rec.Closing.PR != 90 { + t.Errorf("a record that is closing = %+v, %v", rec, err) + } +} + +// recordPR is a batch pull request on release-1.1 carrying body, numbered n, +// in state, opened by login. +func recordPR(n int, state, login, body string) pullRequest { + pr := notifyPR("20261026", body) + pr.Number, pr.State = n, state + pr.Head.Ref = fmt.Sprintf("backport-batch/release-1.1-202610%02d", n%30+1) + pr.User.Login = login + if strings.HasSuffix(login, "[bot]") { + pr.User.Type = "Bot" + } + return pr +} + +func recordBody(conflict int) string { + return "text\n\n" + deferredFence + "\n" + fmt.Sprintf(`{"line":"release-1.1","deferred":[{"original":%d,"commit":"%s","files":["a"]}]}`, conflict, strings.Repeat("a", 40)) + "\n```\n" +} + +// The record is the block of the newest batch pull request that carries one, +// whatever its state, among those a trusted author opened from a batch branch +// of this repository; anything that leaves the newest record in doubt stops +// the command rather than falling back to an older one. +func TestFindLineRecord(t *testing.T) { + const bot = defaultBatchBot + other := func(n int) pullRequest { + pr := recordPR(n, "closed", "maintainer", "") + pr.Head.Ref = fmt.Sprintf("fix-%d", n) + return pr + } + fork := recordPR(92, "closed", "maintainer", recordBody(92)) + fork.Head.Repo = &struct { + FullName string `json:"full_name"` + }{"someone/cozystack"} + cases := []struct { + name string + prs []pullRequest + open int // the open batch the caller found, or 0 + bot string + carrier int // 0 for none + isOpen bool + refusal string + pages int + }{ + {name: "the open batch first", prs: []pullRequest{recordPR(92, "open", "maintainer", recordBody(92)), recordPR(91, "closed", "maintainer", recordBody(91))}, open: 92, carrier: 92, isOpen: true}, + {name: "the newest closed record", prs: []pullRequest{recordPR(91, "closed", bot, recordBody(91)), recordPR(90, "closed", bot, recordBody(90))}, bot: bot, carrier: 91}, + {name: "a person's batch without a block is passed over", prs: []pullRequest{recordPR(92, "closed", "maintainer", "picked by hand"), recordPR(91, "closed", "maintainer", recordBody(91))}, carrier: 91}, + {name: "the bot's batch without a block", prs: []pullRequest{recordPR(92, "closed", bot, "edited away"), recordPR(91, "closed", bot, recordBody(91))}, bot: bot, refusal: "carries no ```backport-batch block"}, + {name: "a fork is no batch", prs: []pullRequest{fork, recordPR(91, "closed", "maintainer", recordBody(91))}, carrier: 91}, + {name: "a newer record by someone untrusted", prs: []pullRequest{recordPR(92, "closed", "someone", recordBody(92)), recordPR(91, "closed", "maintainer", recordBody(91))}, refusal: "#92"}, + {name: "a closed record newer than the open batch", prs: []pullRequest{recordPR(92, "closed", "maintainer", recordBody(92)), recordPR(91, "open", "maintainer", recordBody(91))}, open: 91, refusal: "newer than the open batch pull request #91"}, + {name: "a record on the second page", prs: append(func() (out []pullRequest) { + for n := 1000; n < 1000+recentPullsPage; n++ { + out = append(out, other(n)) + } + return out + }(), recordPR(91, "closed", "maintainer", recordBody(91))), carrier: 91, pages: 2}, + {name: "ten full pages without a record", prs: func() (out []pullRequest) { + for n := 1000; n < 1000+10*recentPullsPage+1; n++ { + out = append(out, other(n)) + } + return out + }(), refusal: "pull requests", pages: 10}, + {name: "the bot's record when no bot is trusted", prs: []pullRequest{recordPR(91, "closed", bot, recordBody(91))}, refusal: "#91"}, + {name: "a hand-written open batch beside a closed record", prs: []pullRequest{recordPR(92, "open", "maintainer", "picked by hand"), recordPR(91, "closed", "maintainer", recordBody(91))}, open: 92, carrier: 91}, + {name: "no batch ever", prs: []pullRequest{other(5)}}, + {name: "an unreadable record", prs: []pullRequest{recordPR(91, "closed", "maintainer", "text\n\n"+deferredFence+"\n{\n```\n")}, refusal: "#91 (closed)"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + l, _ := parseLine("release-1.1") + gh := &fakeGH{prs: tc.prs, bot: tc.bot} + var open *pullRequest + for i := range tc.prs { + if tc.prs[i].Number == tc.open { + open = &tc.prs[i] + } + } + src, err := findLineRecord(gh, l, open) + switch { + case tc.refusal != "": + if err == nil || !strings.Contains(err.Error(), tc.refusal) { + t.Errorf("findLineRecord = %+v, %v; want a refusal with %q", src, err, tc.refusal) + } + case err != nil: + t.Errorf("findLineRecord: %v", err) + case tc.carrier == 0 && (src.carrier != nil || src.rec != nil): + t.Errorf("found #%d, want no record", src.carrier.Number) + case tc.carrier != 0 && (src.carrier == nil || src.carrier.Number != tc.carrier || src.rec == nil || src.open != tc.isOpen || src.rec.Deferred[0].Original != tc.carrier): + t.Errorf("found %+v, want #%d open=%v", src, tc.carrier, tc.isOpen) + } + if tc.pages != 0 && gh.listed != tc.pages { + t.Errorf("read %d pages, want %d", gh.listed, tc.pages) + } + if tc.open != 0 && src.code != open { + t.Errorf("the batch to build on is %+v, want #%d", src.code, tc.open) + } + }) + } +} + +// A run that picks nothing while no batch is open writes nothing, since +// merged and closed batches are never written: it says its classification is +// not kept, and check still reads the merged batch's record. +func TestZeroPickRunLeavesNoRecord(t *testing.T) { + f := newFixture(t) + f.lines() + f.onLine("release-1.1", "fix(app): a line-only five", map[string]string{"app/five.txt": "line\n", "app/seven.txt": "line\n"}) + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + mergeBatch(f, gh) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + edits, comments := len(gh.edited), len(gh.issueComments) + + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push") + if code != 0 || !strings.Contains(stderr, "no batch pull request is open on release-1.1; this run's classification is not retained") { + t.Errorf("run = %d, want 0 saying nothing is retained:\n%s", code, stderr) + } + if len(gh.created) != 1 || len(gh.edited) != edits || len(gh.issueComments) != comments || f.remoteHead(batchBranch1) != "" { + t.Errorf("created %q, edited %q, %d comments, branch at %q; want nothing written", gh.created, gh.edited, len(gh.issueComments), f.remoteHead(batchBranch1)) + } + f.git("switch", "-q", "main") + code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1") + if code != 1 || !strings.Contains(stdout, "/pull/7 ") || strings.Contains(stdout, "/pull/5 ") || !strings.Contains(stderr, "deferred conflict on #90 (merged)") { + t.Errorf("check = %d, want 1 for #7 alone and #5 still a deferred conflict:\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } +} + +// The open batch's author is asked for once although both the open batch and +// its record are read. +func TestRecordAsksForAPermissionOnce(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + gh.asked = nil + if code, _, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 { + t.Fatalf("check = %d\n%s", code, stderr) + } + if len(gh.asked) != 1 { + t.Errorf("asked GitHub for %q, want the author once", gh.asked) + } +} diff --git a/docs/release/backports.md b/docs/release/backports.md index 1c50c7096..f63b6e089 100644 --- a/docs/release/backports.md +++ b/docs/release/backports.md @@ -87,6 +87,10 @@ A deferred conflict waits for the next batch. To land it sooner, pick the origin Before a patch release or an rc is cut from a line, `./backport-batch check --line --cut` has to exit 0. A cut tags the line, so the check fails while a pull request the rules take is not on the line, including one that is only on the open batch pull request: merge the batch pull request first. It prints the URLs, and prints what the batch deferred as warnings, since deferring is a maintainer's call; so is a pull request labelled `backport/skip` that the open batch still carries. An original that waits on the batch for a maintainer fails the cut, also when it is all the batch has left. A cut that passes says so. Without `--cut`, as the scheduled run uses it, a change on the open batch pull request counts as carried. +### The line's record + +What the batch decided about a line, the conflicts it deferred, the originals taken out with `--defer` and those waiting for a maintainer, is kept in a `backport-batch` block in the body of the line's newest batch pull request, its record. `backport-batch` reads the record whether that pull request is open, merged or closed, and writes it only while the pull request is open: a merged or closed batch pull request is never edited or commented on again. So a conflict the batch deferred is still a warning in `check` after the batch merged, until a later batch tries it again. A run that picks nothing while no batch is open writes nothing, and says so. A record `backport-batch` cannot read, or one written by someone it does not trust, stops it rather than let it fall back to an older one; how it decides is in [its README](../../cmd/backport-batch/README.md#the-lines-record). + ### Changes to workflows The scheduled run's token may not push a commit that changes `.github/workflows/`, so the scheduled run leaves such an original out (`run --no-workflow-files`). The batch pull request lists it under "Waiting for a maintainer", with the workflow files it changes, and so does the batch's comment. It is no conflict. The policy still takes it: `check` warns about it while it waits, and `check --cut` fails on it. When nothing else is picked there is no batch pull request to list it on, and `check` reports it as missing. From 74fbb34aca35beabe6c42a7c02f89fd3bc7a40e8 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 18:47:08 +0300 Subject: [PATCH 250/518] docs(backups): say who deletes a MariaDB Backup The comment justified deleting a dump in a tenant's own bucket by the tenant asking for it, but tenant roles grant no delete on Backups: only an administrator or the namespace teardown deletes one, and the reader of this branch should know whose request the delete answers. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/backup_controller.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/internal/backupcontroller/backup_controller.go b/internal/backupcontroller/backup_controller.go index f817e696b..9d755059e 100644 --- a/internal/backupcontroller/backup_controller.go +++ b/internal/backupcontroller/backup_controller.go @@ -137,8 +137,8 @@ func (r *BackupReconciler) cleanupOnDelete(ctx context.Context, backup *backupsv // backup run of the same prefix and a Plan never deletes a Backup, so // a dump on the platform bucket is deleted here or never. A dump in // a tenant's own bucket is deleted too: the Backup proves it owns - // it, and the tenant deleting the Backup is asking for exactly that - // (see cleanupMariaDBBackup). + // it, and tenants cannot delete Backups, so the delete comes from an + // administrator or from namespace teardown (see cleanupMariaDBBackup). return r.cleanupMariaDBBackup(ctx, backup) case strategyv1alpha1.MongoDBStrategyKind: // Split by flow (see cleanupMongoDBBackup). A legacy backup writes to the From 4be80657beaf164067a0e08675f951452bd8b3dd Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 00:41:10 +0700 Subject: [PATCH 251/518] test(dashboard): prove /observability/logs opens the application page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The list page shows the same "Loading…" text as the detail page, so the previous check could not tell which route matched. A missing instance makes only the detail page answer with its own name, and the GET it issues names the plural and the application. Assisted-by: LLM Signed-off-by: ZverGuy --- .../src/routes/ConsolePage.routing.test.tsx | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx index 8e71ff98a..c3b014f4d 100644 --- a/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/routes/ConsolePage.routing.test.tsx @@ -98,14 +98,26 @@ describe("ConsolePage routing", () => { // No ApplicationDefinition plural or application name can be "-", so only // routes under it are safe from shadowing an application's own pages. it("leaves /observability/logs to an application named logs of kind observability", async () => { + const client = makeClient() + vi.mocked(client.list).mockImplementation(async (_g, _v, plural) => ({ + apiVersion: "v1", + kind: `${plural}List`, + metadata: {}, + items: + plural === "tenantnamespaces" + ? [{ apiVersion: "core.cozystack.io/v1alpha1", kind: "TenantNamespace", metadata: { name: "tenant-root" } }] + : [], + }) as K8sList) + const get = vi.spyOn(client, "get").mockRejectedValue(new Error("not found")) renderWithK8sProvider( , - { client: makeClient(), initialRoute: "/observability/logs" }, + { client, initialRoute: "/observability/logs" }, ) - expect(await screen.findByText("Loading…")).toBeInTheDocument() + expect(await screen.findByText(/^Application/)).toHaveTextContent("Application logs not found.") + expect(get).toHaveBeenCalledWith(expect.anything(), expect.anything(), "observability", "logs", expect.anything()) expect(screen.queryByText(/^Every application in/)).toBeNull() }) }) From 889712d2447a7be0ae3180faa25c0467e8672601 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:23:07 +0700 Subject: [PATCH 252/518] fix(observability-proxy): grant the proxy only the API calls it makes Nothing the proxy does reads ApplicationDefinitions, so the rule for them is dropped. Namespaces are read through an informer, which only lists and watches, so get is dropped as well. Assisted-by: LLM Signed-off-by: ZverGuy --- packages/system/observability-proxy/templates/rbac.yaml | 9 +++------ packages/system/observability-proxy/tests/rbac_test.yaml | 7 ++----- 2 files changed, 5 insertions(+), 11 deletions(-) diff --git a/packages/system/observability-proxy/templates/rbac.yaml b/packages/system/observability-proxy/templates/rbac.yaml index f26b71fe3..3c5392903 100644 --- a/packages/system/observability-proxy/templates/rbac.yaml +++ b/packages/system/observability-proxy/templates/rbac.yaml @@ -10,14 +10,11 @@ rules: - apiGroups: ["authorization.k8s.io"] resources: ["subjectaccessreviews"] verbs: ["create"] -# Maps an application plural in the path to its Kind. -- apiGroups: ["cozystack.io"] - resources: ["applicationdefinitions"] - verbs: ["get", "list", "watch"] -# The namespace.cozystack.io/monitoring label names the logs store. +# The namespace.cozystack.io/monitoring label names the logs store; it is +# read from an informer cache, which only lists and watches. - apiGroups: [""] resources: ["namespaces"] - verbs: ["get", "list", "watch"] + verbs: ["list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding diff --git a/packages/system/observability-proxy/tests/rbac_test.yaml b/packages/system/observability-proxy/tests/rbac_test.yaml index d29fe87a3..af339b5d0 100644 --- a/packages/system/observability-proxy/tests/rbac_test.yaml +++ b/packages/system/observability-proxy/tests/rbac_test.yaml @@ -12,7 +12,7 @@ release: namespace: cozy-observability-proxy tests: - - it: lets the proxy review tokens and access, and read ApplicationDefinitions and namespaces + - it: lets the proxy review tokens and access, and list and watch namespaces template: templates/rbac.yaml documentSelector: path: kind @@ -27,12 +27,9 @@ tests: - apiGroups: ["authorization.k8s.io"] resources: ["subjectaccessreviews"] verbs: ["create"] - - apiGroups: ["cozystack.io"] - resources: ["applicationdefinitions"] - verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["namespaces"] - verbs: ["get", "list", "watch"] + verbs: ["list", "watch"] - it: binds the proxy role to the proxy ServiceAccount in the release namespace template: templates/rbac.yaml From 97a536b89adcf7ba42ef7c35bef0ca80a2638fa2 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:24:04 +0700 Subject: [PATCH 253/518] fix(observability-proxy): refuse durations past the int64 range parseDuration summed duration parts without an overflow check and converted out-of-range floats to Duration, which Go leaves implementation-defined. A start_offset such as 99999999999h99999999999h1s wrapped to 999ms and was accepted, and a huge step saturated on some architectures and wrapped on others. Such values now answer 400 like any other invalid duration. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 24 +++++++++++++++++-- .../observabilityproxy/logsparams_test.go | 18 ++++++++++++++ 2 files changed, 40 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index 7be252998..520f89801 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -18,6 +18,7 @@ package observabilityproxy import ( "fmt" + "math" "net/http" "net/url" "regexp" @@ -275,7 +276,11 @@ func parseDuration(s string) (time.Duration, error) { if err != nil { return 0, fmt.Errorf("invalid duration %q", s) } - return time.Duration(f * float64(time.Second)), nil + d, ok := scaleDuration(f, time.Second) + if !ok { + return 0, fmt.Errorf("duration %q out of range", s) + } + return d, nil } if s == "" { return 0, fmt.Errorf("empty duration") @@ -287,12 +292,27 @@ func parseDuration(s string) (time.Duration, error) { return 0, fmt.Errorf("invalid duration %q", s) } f, _ := strconv.ParseFloat(m[1], 64) - total += time.Duration(f * float64(durationUnits[m[2]])) + d, ok := scaleDuration(f, durationUnits[m[2]]) + if !ok || total > math.MaxInt64-d { + return 0, fmt.Errorf("duration %q out of range", s) + } + total += d rest = rest[len(m[0]):] } return total, nil } +// scaleDuration refuses what does not fit a Duration: Go leaves an +// out-of-range float to int conversion implementation-defined. +func scaleDuration(f float64, unit time.Duration) (time.Duration, bool) { + v := f * float64(unit) + // float64(math.MaxInt64) rounds up to 2^63, which is already too large. + if v >= float64(math.MaxInt64) { + return 0, false + } + return time.Duration(v), true +} + // parseTime accepts "now", unix seconds with an optional fraction, RFC3339, // and a duration meaning that long before now (VictoriaLogs' "start=3h"). // Anything else is refused rather than guessed, since a misread timestamp diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index ddb95cdc5..46c62ffb9 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -143,6 +143,11 @@ func TestBuildLogsParamsRejects(t *testing.T) { {"too many hits fields", "hits", url.Values{"field": strings.Fields("a b c d e f g h i j k")}, http.StatusBadRequest}, {"since and start_offset", "tail", url.Values{"since": {"2026-11-04T11:59:00Z"}, "start_offset": {"1m"}}, http.StatusBadRequest}, {"bad since", "tail", url.Values{"since": {"soon"}}, http.StatusBadRequest}, + {"start_offset summing past int64", "tail", url.Values{"start_offset": {"99999999999h99999999999h1s"}}, http.StatusBadRequest}, + {"step summing past int64", "hits", url.Values{"step": {"2562047h2562047h"}}, http.StatusBadRequest}, + {"step token past int64", "hits", url.Values{"step": {"99999999999h"}}, http.StatusBadRequest}, + {"bare-seconds step past int64", "hits", url.Values{"step": {"99999999999999999999"}}, http.StatusBadRequest}, + {"relative start summing past int64", "query", url.Values{"start": {"106751d1d"}}, http.StatusBadRequest}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -352,3 +357,16 @@ func TestBuildLogsParamsStepKeepsBucketCap(t *testing.T) { } } } + +func TestParseDurationBounds(t *testing.T) { + for _, in := range []string{"2562047h", "106751d", "9223372036s", "292y"} { + if d, err := parseDuration(in); err != nil || d <= 0 { + t.Errorf("parseDuration(%q) = %s, %v; want a positive duration", in, d, err) + } + } + for _, in := range []string{"2562048h", "106751d1d", "9223372037s", "9223372037", "293y", "1y292y"} { + if d, err := parseDuration(in); err == nil { + t.Errorf("parseDuration(%q) = %s; want an error past the int64 range", in, d) + } + } +} From f19363ec6250feef6c06508910bc79613f026b6f Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:31:06 +0700 Subject: [PATCH 254/518] fix(observability-proxy): keep Retry-After positive at tiny rates The wait until the next token was converted from float to Duration without a range check, so a rate limit below about 1e-10 requests per second produced a negative Retry-After on amd64. The wait now saturates at the largest Duration. The earlier 429 change said range, step, bucket and limit violations keep 422; on the logs root they are clamped or answer 400, and nothing answers 422. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/limits.go | 7 ++++++- internal/observabilityproxy/limits_test.go | 8 ++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/limits.go b/internal/observabilityproxy/limits.go index f0392f25b..eaebe3104 100644 --- a/internal/observabilityproxy/limits.go +++ b/internal/observabilityproxy/limits.go @@ -17,6 +17,7 @@ limitations under the License. package observabilityproxy import ( + "math" "sync" "time" ) @@ -79,7 +80,11 @@ func (l *limiter) allowRate(user string) (time.Duration, bool) { b.tokens = min(burst, b.tokens+now.Sub(b.last).Seconds()*l.cfg.RatePerUser) b.last = now if b.tokens < 1 { - return time.Duration((1 - b.tokens) / l.cfg.RatePerUser * float64(time.Second)), false + wait, ok := scaleDuration((1-b.tokens)/l.cfg.RatePerUser, time.Second) + if !ok { + wait = math.MaxInt64 + } + return wait, false } b.tokens-- return 0, true diff --git a/internal/observabilityproxy/limits_test.go b/internal/observabilityproxy/limits_test.go index 5c3af772d..f7ff45d3a 100644 --- a/internal/observabilityproxy/limits_test.go +++ b/internal/observabilityproxy/limits_test.go @@ -108,3 +108,11 @@ func allowed(l *limiter, user string) bool { _, ok := l.allowRate(user) return ok } + +func TestLimiterRateWaitAtTinyRate(t *testing.T) { + l := newLimiter(LimiterConfig{RatePerUser: 1e-12, RateBurst: 1}) + allowed(l, "alice") + if wait, ok := l.allowRate("alice"); ok || wait <= 0 { + t.Fatalf("empty bucket at a tiny rate: admitted %v, wait %s, want refused with a positive wait", ok, wait) + } +} From e6e3466b7d98c037ac5fdc7dca204f96d55312c8 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 19:35:38 +0700 Subject: [PATCH 255/518] fix(observability-proxy): open live tails without waiting for a line Go's transport asks for gzip on its own when a request sets no Accept-Encoding. vlselect answers a gzip-negotiated tail through a gzip writer that holds even the response headers until the first line, so the proxy could not forward its 200 and the console kept showing a connecting state for quiet applications; buffered lines were also delayed. The tail request now sends Accept-Encoding: identity, which also turns off the transport's transparent gzip. Other endpoints keep it: they are bounded responses read to the end, compression saves bandwidth between the proxy and the store, and the transport decompresses before the body is copied to the client. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 5 +++ internal/observabilityproxy/server_test.go | 49 ++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index fcdb46141..043fd93d5 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -199,6 +199,11 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT req.Header.Set("Content-Type", "application/x-www-form-urlencoded") req.Header.Set("AccountID", "0") req.Header.Set("ProjectID", "0") + if tail { + // vlselect holds even the headers of a gzip-encoded tail until the + // first line; an explicit value stops the transport from asking for gzip. + req.Header.Set("Accept-Encoding", "identity") + } resp, err := s.client.Do(req) switch { case err == nil: diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index 53c2aaa2d..dc782d60d 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -326,6 +326,55 @@ func TestServerTailFlushesEveryLine(t *testing.T) { } } +// TestServerTailHeadersBeforeFirstLine mimics vlselect, which holds even the +// response headers of a gzip-negotiated tail until the first line arrives. +func TestServerTailHeadersBeforeFirstLine(t *testing.T) { + done := make(chan struct{}) + h := newHarness(t, Config{TailIdleTimeout: time.Minute}, func(w http.ResponseWriter, r *http.Request) { + if strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") { + select { + case <-done: + case <-r.Context().Done(): + } + return + } + tailUpstream(nil, done)(w, r) + }) + // Registered after the harness so it runs before the servers close and + // releases a held upstream when the assertion below fails. + t.Cleanup(func() { close(done) }) + + req, err := http.NewRequest(http.MethodGet, h.proxy.URL+logsRoot+"tail?query=*", nil) + if err != nil { + t.Fatal(err) + } + req.Header.Set("Authorization", "Bearer good") + got := make(chan *http.Response, 1) + go func() { + resp, err := http.DefaultClient.Do(req) + if err != nil { + close(got) + return + } + got <- resp + }() + select { + case resp, ok := <-got: + if !ok { + t.Fatal("tail request failed") + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200", resp.StatusCode) + } + case <-time.After(2 * time.Second): + t.Fatal("tail response headers did not reach the client before the first line") + } + if ae := h.requests()[0].headers.Get("Accept-Encoding"); ae != "identity" { + t.Fatalf("upstream tail Accept-Encoding = %q, want identity", ae) + } +} + func TestServerTailCapsAndIdle(t *testing.T) { done := make(chan struct{}) t.Cleanup(func() { close(done) }) From 4101dac9b2a212b95cef56fef278cae9ee4a174c Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 19:38:22 +0700 Subject: [PATCH 256/518] test(observability-proxy): pin gzip on non-tail log requests Only tails opt out of the transport's gzip negotiation; bounded requests keep it on purpose. Assert that side too, so a change that applies identity to every request fails a test instead of silently dropping compression between the proxy and the store. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server_test.go | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index dc782d60d..fe3805961 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -215,6 +215,9 @@ func TestServerRebuildsUpstreamRequest(t *testing.T) { if r.headers.Get("AccountID") != "0" || r.headers.Get("ProjectID") != "0" { t.Errorf("tenant headers = %q/%q, want 0/0", r.headers.Get("AccountID"), r.headers.Get("ProjectID")) } + if ae := r.headers.Get("Accept-Encoding"); !strings.Contains(ae, "gzip") { + t.Errorf("upstream query Accept-Encoding = %q, want gzip outside tails", ae) + } for _, k := range []string{"Authorization", "X-Forwarded-For", "Cookie"} { if r.headers.Get(k) != "" { t.Errorf("client header %s forwarded", k) @@ -370,7 +373,11 @@ func TestServerTailHeadersBeforeFirstLine(t *testing.T) { case <-time.After(2 * time.Second): t.Fatal("tail response headers did not reach the client before the first line") } - if ae := h.requests()[0].headers.Get("Accept-Encoding"); ae != "identity" { + reqs := h.requests() + if len(reqs) != 1 { + t.Fatalf("upstream saw %d requests, want 1", len(reqs)) + } + if ae := reqs[0].headers.Get("Accept-Encoding"); ae != "identity" { t.Fatalf("upstream tail Accept-Encoding = %q, want identity", ae) } } From f81aaf4d256c3d8ef2554b8a2a029dc36ea500de Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:18:23 +0700 Subject: [PATCH 257/518] fix(observability-proxy): admit a request before reading its body The form was parsed before the concurrency slot was taken, and the listener sets no body-read deadline, so a client that never finished a POST body held a handler outside every limit for as long as it liked. Tenant workloads reach the Service directly, so nginx buffering does not cover this path. The slot is now taken first and the body is read under the request timeout; a body that does not arrive in time answers 408 and frees the slot. The deadline is lifted once the body is read, because the server's background read would take a later expiry for a client that went away and cancel a live tail. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 40 +++-- .../server_deadline_test.go | 137 ++++++++++++++++++ 2 files changed, 168 insertions(+), 9 deletions(-) create mode 100644 internal/observabilityproxy/server_deadline_test.go diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 043fd93d5..72d380e21 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -25,6 +25,7 @@ import ( "log/slog" "math" "net/http" + "os" "strconv" "strings" "time" @@ -159,15 +160,6 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT if !ok { return errorf(http.StatusNotFound, "not found") } - r.Body = http.MaxBytesReader(w, r.Body, maxFormBytes) - if err := r.ParseForm(); err != nil { - return errorf(http.StatusBadRequest, "invalid form: %v", err) - } - params, herr := BuildLogsParams(endpoint, r.Form, route.Namespace, s.now(), s.cfg.Logs) - if herr != nil { - return herr - } - tail := endpoint == "tail" acquire := s.limits.acquire if tail { @@ -178,6 +170,13 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT return errorf(http.StatusTooManyRequests, "too many concurrent requests") } defer release() + if herr := s.readForm(w, r); herr != nil { + return herr + } + params, herr := BuildLogsParams(endpoint, r.Form, route.Namespace, s.now(), s.cfg.Logs) + if herr != nil { + return herr + } store, err := s.stores.Resolve(route.Namespace) if err != nil { @@ -233,6 +232,29 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT return nil } +// readForm reads the body under the request timeout, so a client that +// sends it slowly holds its slot only that long. Once the body is read the +// deadline is lifted: the server's background read would take a later +// expiry for a client that went away and cancel a tail. After a failed +// read it stays, so the server's drain of the unread body is bounded too. +func (s *Server) readForm(w http.ResponseWriter, r *http.Request) *HTTPError { + r.Body = http.MaxBytesReader(w, r.Body, maxFormBytes) + rc := http.NewResponseController(w) + if s.cfg.RequestTimeout > 0 { + _ = rc.SetReadDeadline(time.Now().Add(s.cfg.RequestTimeout)) + } + err := r.ParseForm() + switch { + case err == nil: + _ = rc.SetReadDeadline(time.Time{}) + return nil + case errors.Is(err, os.ErrDeadlineExceeded): + return errorf(http.StatusRequestTimeout, "request body not received within %s", s.cfg.RequestTimeout) + default: + return errorf(http.StatusBadRequest, "invalid form: %v", err) + } +} + func (s *Server) authorize(r *http.Request, route Route) (string, *HTTPError) { token, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ") if !ok { diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go new file mode 100644 index 000000000..6931d8c01 --- /dev/null +++ b/internal/observabilityproxy/server_deadline_test.go @@ -0,0 +1,137 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "bufio" + "net" + "net/http" + "sync" + "testing" + "time" +) + +// pipeListener hands the server one end of a net.Pipe per dial. A pipe has +// no buffer, so a client that stops reading blocks the server's next write +// at once, as a full TCP window eventually does. +type pipeListener struct { + conns chan net.Conn + closed chan struct{} + once sync.Once +} + +type pipeAddr struct{} + +func (pipeAddr) Network() string { return "pipe" } +func (pipeAddr) String() string { return "pipe" } + +func servePipe(t *testing.T, handler http.Handler) *pipeListener { + t.Helper() + l := &pipeListener{conns: make(chan net.Conn), closed: make(chan struct{})} + srv := &http.Server{Handler: handler, ReadHeaderTimeout: 10 * time.Second} + go func() { _ = srv.Serve(l) }() + t.Cleanup(func() { _ = srv.Close() }) + return l +} + +func (l *pipeListener) Accept() (net.Conn, error) { + select { + case c := <-l.conns: + return c, nil + case <-l.closed: + return nil, net.ErrClosed + } +} + +func (l *pipeListener) Close() error { + l.once.Do(func() { close(l.closed) }) + return nil +} + +func (l *pipeListener) Addr() net.Addr { return pipeAddr{} } + +// send opens a connection and writes raw to it without waiting for the +// server to read all of it. +func (l *pipeListener) send(t *testing.T, raw string) net.Conn { + t.Helper() + client, server := net.Pipe() + t.Cleanup(func() { _ = client.Close() }) + select { + case l.conns <- server: + case <-time.After(2 * time.Second): + t.Fatal("pipe server is not accepting") + } + go func() { _, _ = client.Write([]byte(raw)) }() + return client +} + +func eventually(t *testing.T, within time.Duration, cond func() bool, msg string) { + t.Helper() + deadline := time.Now().Add(within) + for !cond() { + if time.Now().After(deadline) { + t.Fatal(msg) + } + time.Sleep(10 * time.Millisecond) + } +} + +func readResponse(t *testing.T, conn net.Conn) *http.Response { + t.Helper() + got := make(chan *http.Response, 1) + go func() { + resp, err := http.ReadResponse(bufio.NewReader(conn), nil) + if err != nil { + close(got) + return + } + got <- resp + }() + select { + case resp, ok := <-got: + if !ok { + t.Fatal("reading the response failed") + } + return resp + case <-time.After(3 * time.Second): + t.Fatal("no response") + } + return nil +} + +// TestServerSlowBodyHoldsSlotUntilDeadline sends a POST whose body never +// completes: it must count against the user's concurrency, and the read +// must give up at the request timeout instead of holding the handler. +func TestServerSlowBodyHoldsSlotUntilDeadline(t *testing.T) { + h := newHarness(t, Config{ + RequestTimeout: 200 * time.Millisecond, + Limiter: LimiterConfig{MaxConcurrentPerUser: 1}, + }, nil) + pipe := servePipe(t, h.server) + conn := pipe.send(t, "POST "+logsRoot+"query HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n"+ + "Content-Type: application/x-www-form-urlencoded\r\nContent-Length: 100\r\n\r\nquery=") + + eventually(t, time.Second, func() bool { + return h.status("GET", logsRoot+"query", "good") == http.StatusTooManyRequests + }, "a request with an unfinished body does not hold a concurrency slot") + if resp := readResponse(t, conn); resp.StatusCode != http.StatusRequestTimeout { + t.Fatalf("unfinished body answered %d, want 408", resp.StatusCode) + } + eventually(t, 2*time.Second, func() bool { + return h.status("GET", logsRoot+"query", "good") == http.StatusOK + }, "the slot of an unfinished body was not released") +} From ddc37f9c4f07275a1fd7def6ac4abeb22110ae15 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:20:02 +0700 Subject: [PATCH 258/518] fix(observability-proxy): put a deadline on writes to the client A client that stopped reading kept its slot for good: the first flush ran before the tail idle timer started, and cancelling the upstream request does not interrupt a write that is blocked on the client. Every response now carries a write deadline. A tail renews it by the idle timeout before each line, so a tail that keeps receiving stays open while a stalled reader is cut and frees its slot; any other response must be written by the request deadline. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 24 +++++-- .../server_deadline_test.go | 62 +++++++++++++++++++ 2 files changed, 80 insertions(+), 6 deletions(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 72d380e21..f05350ddc 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -219,14 +219,25 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT if ct := resp.Header.Get("Content-Type"); ct != "" { w.Header().Set("Content-Type", ct) } - w.WriteHeader(resp.StatusCode) + // Cancelling the upstream request does not interrupt a write to a + // client that stopped reading; only a write deadline does. A tail + // renews it before every line, so only a stalled client is cut; the + // server clears it once the response is finished. rc := http.NewResponseController(w) - _ = rc.Flush() - if tail { - s.streamLines(w, rc, resp.Body, cancel) + arm := func() {} + if tail && s.cfg.TailIdleTimeout > 0 { + arm = func() { _ = rc.SetWriteDeadline(time.Now().Add(s.cfg.TailIdleTimeout)) } + } else if deadline, ok := ctx.Deadline(); ok { + _ = rc.SetWriteDeadline(deadline) + } + arm() + w.WriteHeader(resp.StatusCode) + if rc.Flush() != nil { return nil } - if _, err := io.Copy(w, resp.Body); err != nil && !errors.Is(err, context.Canceled) { + if tail { + s.streamLines(w, rc, resp.Body, cancel, arm) + } else if _, err := io.Copy(w, resp.Body); err != nil && !errors.Is(err, context.Canceled) { slog.Debug("copying upstream response", "error", err) } return nil @@ -282,7 +293,7 @@ func (s *Server) authorize(r *http.Request, route Route) (string, *HTTPError) { // streamLines flushes after every line so a live tail reaches the browser // as it happens, and cancels the upstream stream once no line has arrived // for the idle timeout. -func (s *Server) streamLines(w http.ResponseWriter, rc *http.ResponseController, body io.Reader, cancel context.CancelFunc) { +func (s *Server) streamLines(w http.ResponseWriter, rc *http.ResponseController, body io.Reader, cancel context.CancelFunc, arm func()) { var idle *time.Timer if s.cfg.TailIdleTimeout > 0 { idle = time.AfterFunc(s.cfg.TailIdleTimeout, cancel) @@ -295,6 +306,7 @@ func (s *Server) streamLines(w http.ResponseWriter, rc *http.ResponseController, if idle != nil { idle.Reset(s.cfg.TailIdleTimeout) } + arm() if _, werr := w.Write(chunk); werr != nil { return } diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index 6931d8c01..c2535f3ae 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -135,3 +135,65 @@ func TestServerSlowBodyHoldsSlotUntilDeadline(t *testing.T) { return h.status("GET", logsRoot+"query", "good") == http.StatusOK }, "the slot of an unfinished body was not released") } + +// TestServerStalledClientReleasesTail holds a tail whose client stops +// reading, once before the headers and once after the first line: the +// slot must come back within a few idle timeouts in both cases. +func TestServerStalledClientReleasesTail(t *testing.T) { + for _, readHeaders := range []bool{false, true} { + lines := make(chan string, 1) + done := make(chan struct{}) + h := newHarness(t, Config{ + TailIdleTimeout: 200 * time.Millisecond, + Limiter: LimiterConfig{MaxTails: 1}, + }, tailUpstream(lines, done)) + t.Cleanup(func() { close(done) }) + pipe := servePipe(t, h.server) + conn := pipe.send(t, "GET "+logsRoot+"tail?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n\r\n") + if readHeaders { + if resp := readResponse(t, conn); resp.StatusCode != http.StatusOK { + t.Fatalf("tail answered %d", resp.StatusCode) + } + lines <- `{"_msg":"unread"}` + } + eventually(t, time.Second, func() bool { return h.server.limits.openTails() == 1 }, "the tail never opened") + eventually(t, 2*time.Second, func() bool { return h.server.limits.openTails() == 0 }, + "a client that stopped reading keeps its tail slot") + } +} + +// TestServerStalledClientReleasesQuery does the same for a non-tail +// response, which must be written by the request deadline. +func TestServerStalledClientReleasesQuery(t *testing.T) { + h := newHarness(t, Config{ + RequestTimeout: 200 * time.Millisecond, + Limiter: LimiterConfig{MaxConcurrentPerUser: 1}, + }, nil) + pipe := servePipe(t, h.server) + pipe.send(t, "GET "+logsRoot+"query?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n\r\n") + eventually(t, time.Second, func() bool { + return h.status("GET", logsRoot+"query", "good") == http.StatusTooManyRequests + }, "the stalled request never held its slot") + eventually(t, 2*time.Second, func() bool { + return h.status("GET", logsRoot+"query", "good") == http.StatusOK + }, "a client that stopped reading keeps its concurrency slot") +} + +// TestServerActiveTailOutlivesWriteDeadline keeps a tail busy for several +// idle timeouts: the write deadline must be renewed, not fixed. +func TestServerActiveTailOutlivesWriteDeadline(t *testing.T) { + lines := make(chan string) + done := make(chan struct{}) + t.Cleanup(func() { close(done) }) + h := newHarness(t, Config{TailIdleTimeout: 150 * time.Millisecond}, tailUpstream(lines, done)) + resp := h.do(http.MethodGet, logsRoot+"tail?query=*", "good", nil, nil) + defer func() { _ = resp.Body.Close() }() + rd := bufio.NewReader(resp.Body) + for i := range 8 { + lines <- `{"_msg":"line"}` + if _, err := rd.ReadString('\n'); err != nil { + t.Fatalf("line %d: %v", i, err) + } + time.Sleep(75 * time.Millisecond) + } +} From 8b8b81013cc9fba21eca235a959192a674c1e0a9 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:23:09 +0700 Subject: [PATCH 259/518] fix(observability-proxy): bound the wait for tail response headers The tail idle timer started only once the store had sent its headers, and a tail request carries no deadline, so a store that accepted the request and stayed silent held the tail slot until the client left. The timer now starts before the request, and the same timer bounds the header wait and the gap between lines. A store that never answers gets 504, like a non-tail request past its deadline. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 41 +++++++++++-------- .../server_deadline_test.go | 31 ++++++++++++++ 2 files changed, 56 insertions(+), 16 deletions(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index f05350ddc..fc61eb5ac 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -37,6 +37,10 @@ import ( // on in pieces and flushed once its end arrives. const tailChunkBytes = 64 << 10 +// errTailIdle cancels a tail that received nothing for the idle timeout; +// it wraps DeadlineExceeded so a store that never answered reads as 504. +var errTailIdle = fmt.Errorf("tail idle: %w", context.DeadlineExceeded) + // maxFormBytes bounds a POST body; LogsQL queries are a few KiB at most. const maxFormBytes = 1 << 20 @@ -184,11 +188,21 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT return errorf(http.StatusServiceUnavailable, "cannot resolve the log store") } - ctx, cancel := context.WithCancel(r.Context()) - defer cancel() - if !tail && s.cfg.RequestTimeout > 0 { - ctx, cancel = context.WithTimeout(ctx, s.cfg.RequestTimeout) - defer cancel() + ctx, cancel := context.WithCancelCause(r.Context()) + defer cancel(nil) + limit := s.cfg.RequestTimeout + var idle *time.Timer + if tail { + // Started before the request, so a store that never sends the + // headers is bounded as well as one that stops sending lines. + if limit = s.cfg.TailIdleTimeout; limit > 0 { + idle = time.AfterFunc(limit, func() { cancel(errTailIdle) }) + defer idle.Stop() + } + } else if limit > 0 { + var stop context.CancelFunc + ctx, stop = context.WithTimeout(ctx, limit) + defer stop() } req, err := http.NewRequestWithContext(ctx, http.MethodPost, s.cfg.UpstreamURL(store)+"/select/logsql/"+endpoint, strings.NewReader(params.Encode())) @@ -208,8 +222,8 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT case err == nil: case r.Context().Err() != nil: return errorf(http.StatusBadGateway, "client went away") - case errors.Is(err, context.DeadlineExceeded): - return errorf(http.StatusGatewayTimeout, "log store did not answer within %s", s.cfg.RequestTimeout) + case errors.Is(context.Cause(ctx), context.DeadlineExceeded): + return errorf(http.StatusGatewayTimeout, "log store did not answer within %s", limit) default: slog.Warn("log store unreachable", "store", store, "error", err) return errorf(http.StatusBadGateway, "log store unreachable") @@ -236,7 +250,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT return nil } if tail { - s.streamLines(w, rc, resp.Body, cancel, arm) + s.streamLines(w, rc, resp.Body, idle, arm) } else if _, err := io.Copy(w, resp.Body); err != nil && !errors.Is(err, context.Canceled) { slog.Debug("copying upstream response", "error", err) } @@ -291,14 +305,9 @@ func (s *Server) authorize(r *http.Request, route Route) (string, *HTTPError) { } // streamLines flushes after every line so a live tail reaches the browser -// as it happens, and cancels the upstream stream once no line has arrived -// for the idle timeout. -func (s *Server) streamLines(w http.ResponseWriter, rc *http.ResponseController, body io.Reader, cancel context.CancelFunc, arm func()) { - var idle *time.Timer - if s.cfg.TailIdleTimeout > 0 { - idle = time.AfterFunc(s.cfg.TailIdleTimeout, cancel) - defer idle.Stop() - } +// as it happens; each line pushes back idle, which cancels the upstream +// stream once no line has arrived for the idle timeout. +func (s *Server) streamLines(w http.ResponseWriter, rc *http.ResponseController, body io.Reader, idle *time.Timer, arm func()) { rd := bufio.NewReaderSize(body, tailChunkBytes) for { chunk, err := rd.ReadSlice('\n') diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index c2535f3ae..fc039c22f 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -197,3 +197,34 @@ func TestServerActiveTailOutlivesWriteDeadline(t *testing.T) { time.Sleep(75 * time.Millisecond) } } + +// TestServerTailUpstreamWithoutHeaders gives up on a store that accepts a +// tail but never answers it, instead of holding the slot until the client +// leaves. +func TestServerTailUpstreamWithoutHeaders(t *testing.T) { + release := make(chan struct{}) + h := newHarness(t, Config{ + TailIdleTimeout: 200 * time.Millisecond, + Limiter: LimiterConfig{MaxTails: 1}, + }, func(_ http.ResponseWriter, r *http.Request) { + select { + case <-release: + case <-r.Context().Done(): + } + }) + // After the harness, so it runs before the servers wait for handlers. + t.Cleanup(func() { close(release) }) + got := make(chan int, 1) + go func() { got <- h.status(http.MethodGet, logsRoot+"tail?query=*", "good") }() + select { + case code := <-got: + if code != http.StatusGatewayTimeout { + t.Fatalf("silent store answered %d, want 504", code) + } + case <-time.After(2 * time.Second): + t.Fatal("a tail waits for upstream headers without bound") + } + if n := h.server.limits.openTails(); n != 0 { + t.Fatalf("open tails = %d after the tail ended", n) + } +} From 0dc8a49c1979fef398ed61f7cf90bb98870467a3 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:23:46 +0700 Subject: [PATCH 260/518] fix(observability-proxy): cut the connection when a log answer breaks off Once the status line has gone out, a read error from the store left the handler returning normally, so the chunked body ended cleanly and the client took a partial answer for a complete one. The handler now aborts the response in that case. The request is counted before the abort, because the abort skips the rest of the handler. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 13 +++++++++++-- .../observabilityproxy/server_deadline_test.go | 16 ++++++++++++++++ 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index fc61eb5ac..2e391442c 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -41,6 +41,11 @@ const tailChunkBytes = 64 << 10 // it wraps DeadlineExceeded so a store that never answered reads as 504. var errTailIdle = fmt.Errorf("tail idle: %w", context.DeadlineExceeded) +// errAborted ends a response whose status line already went out by +// cutting the connection, so a client sees a broken body instead of a +// complete one with lines missing. +var errAborted = &HTTPError{Code: http.StatusBadGateway, Message: "response aborted"} + // maxFormBytes bounds a POST body; LogsQL queries are a few KiB at most. const maxFormBytes = 1 << 20 @@ -129,10 +134,13 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { if herr == nil { herr = s.serve(rec, r, route) } - if herr != nil { + if herr != nil && herr != errAborted { writeError(rec, herr) } s.requests.WithLabelValues(route.Kind.String(), strconv.Itoa(rec.code)).Inc() + if herr == errAborted { + panic(http.ErrAbortHandler) + } } func writeError(w http.ResponseWriter, e *HTTPError) { @@ -251,8 +259,9 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT } if tail { s.streamLines(w, rc, resp.Body, idle, arm) - } else if _, err := io.Copy(w, resp.Body); err != nil && !errors.Is(err, context.Canceled) { + } else if _, err := io.Copy(w, resp.Body); err != nil { slog.Debug("copying upstream response", "error", err) + return errAborted } return nil } diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index fc039c22f..afd69ed30 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -18,6 +18,7 @@ package observabilityproxy import ( "bufio" + "io" "net" "net/http" "sync" @@ -228,3 +229,18 @@ func TestServerTailUpstreamWithoutHeaders(t *testing.T) { t.Fatalf("open tails = %d after the tail ended", n) } } + +// TestServerAbortsTruncatedResponse cuts the client connection when the +// store's body breaks off after the headers went out, so a partial answer +// does not end as a clean 200. +func TestServerAbortsTruncatedResponse(t *testing.T) { + h := newHarness(t, Config{}, func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Length", "100") + _, _ = io.WriteString(w, `{"_msg":"a"}`+"\n") + }) + resp := h.do(http.MethodGet, logsRoot+"query?query=*", "good", nil, nil) + defer func() { _ = resp.Body.Close() }() + if _, err := io.ReadAll(resp.Body); err == nil { + t.Fatal("a truncated store answer reached the client as a complete response") + } +} From bb073eec1b9e801515801bc1e5bef3d928502426 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:24:30 +0700 Subject: [PATCH 261/518] fix(observability-proxy): do not cache reviews that failed to decide A TokenReview whose authenticator failed comes back unauthenticated with status.error set, and a SubjectAccessReview whose authorizer failed comes back with an evaluation error and no decision. Both were cached as refusals, so a valid caller stayed locked out for the cache TTL after the outage ended. Such answers are now refused for the current request only. A review that failed to decide still answers 401 or 403 rather than 503: RBAC sets an evaluation error for any binding to a missing role, which a retryable status would turn into a permanent 503 for that user. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/auth.go | 26 ++++++++----- internal/observabilityproxy/auth_test.go | 49 +++++++++++++++++++++++- 2 files changed, 65 insertions(+), 10 deletions(-) diff --git a/internal/observabilityproxy/auth.go b/internal/observabilityproxy/auth.go index 90756bdad..0baac558a 100644 --- a/internal/observabilityproxy/auth.go +++ b/internal/observabilityproxy/auth.go @@ -37,8 +37,9 @@ var ErrUnauthenticated = errors.New("unauthenticated") // Authorizer asks kube-apiserver who a token belongs to and whether that // identity may read a scope. Answers, including negative ones, are cached -// for the TTL; transport errors are not, so an apiserver blip is retried on -// the next request instead of locking a user out for the TTL. +// for the TTL; transport errors and reviews that failed to decide are not, +// so an outage is retried on the next request instead of locking a user +// out for the TTL. type Authorizer struct { client kubernetes.Interface now func() time.Time @@ -72,14 +73,16 @@ func (a *Authorizer) Authenticate(ctx context.Context, token string) (*authentic if err != nil { return nil, fmt.Errorf("token review: %w", err) } - var u *authenticationv1.UserInfo - if tr.Status.Authenticated && tr.Status.User.Username != "" { - u = tr.Status.User.DeepCopy() - } - a.tokens.set(key, u, a.now()) - if u == nil { + if !tr.Status.Authenticated || tr.Status.User.Username == "" { + // status.error also marks an authenticator that failed to decide; + // a cached refusal would outlast its recovery. + if tr.Status.Error == "" { + a.tokens.set(key, nil, a.now()) + } return nil, ErrUnauthenticated } + u := tr.Status.User.DeepCopy() + a.tokens.set(key, u, a.now()) return u, nil } @@ -109,7 +112,12 @@ func (a *Authorizer) Authorize(ctx context.Context, user *authenticationv1.UserI return false, fmt.Errorf("subject access review: %w", err) } allowed := sar.Status.Allowed && !sar.Status.Denied - a.decisions.set(key, allowed, a.now()) + // An evaluation error without a decision denies this request only: it + // may be a webhook authorizer that is down, and RBAC reports one for + // any binding to a missing role, so it cannot be told from a refusal. + if allowed || sar.Status.Denied || sar.Status.EvaluationError == "" { + a.decisions.set(key, allowed, a.now()) + } return allowed, nil } diff --git a/internal/observabilityproxy/auth_test.go b/internal/observabilityproxy/auth_test.go index b57b80cf1..fe940ce04 100644 --- a/internal/observabilityproxy/auth_test.go +++ b/internal/observabilityproxy/auth_test.go @@ -38,6 +38,10 @@ type fakeKube struct { sars atomic.Int32 lastSAR atomic.Pointer[authorizationv1.SubjectAccessReviewSpec] reviewFn func() error + // reviewError and evaluationError, when set, are what the + // authenticator or the authorizer report as failing to decide. + reviewError atomic.Pointer[string] + evaluationError atomic.Pointer[string] } func newFakeKube(users map[string]string, allow func(user string, attrs *authorizationv1.ResourceAttributes) bool) *fakeKube { @@ -50,7 +54,9 @@ func newFakeKube(users map[string]string, allow func(user string, attrs *authori } } tr := action.(k8stesting.CreateAction).GetObject().(*authenticationv1.TokenReview).DeepCopy() - if name, ok := users[tr.Spec.Token]; ok { + if e := f.reviewError.Load(); e != nil { + tr.Status = authenticationv1.TokenReviewStatus{Error: *e} + } else if name, ok := users[tr.Spec.Token]; ok { tr.Status = authenticationv1.TokenReviewStatus{Authenticated: true, User: authenticationv1.UserInfo{Username: name, Groups: []string{"tenant-a-view"}}} } return true, tr, nil @@ -60,6 +66,10 @@ func newFakeKube(users map[string]string, allow func(user string, attrs *authori sar := action.(k8stesting.CreateAction).GetObject().(*authorizationv1.SubjectAccessReview).DeepCopy() spec := sar.Spec f.lastSAR.Store(&spec) + if e := f.evaluationError.Load(); e != nil { + sar.Status.EvaluationError = *e + return true, sar, nil + } sar.Status.Allowed = allow(sar.Spec.User, sar.Spec.ResourceAttributes) return true, sar, nil }) @@ -171,3 +181,40 @@ func TestAuthorize(t *testing.T) { t.Fatalf("a different group set must not hit another identity's cache entry, SARs = %d", n) } } + +// TestAuthenticateFailedReviewNotCached: a TokenReview the authenticator +// could not decide carries status.error; caching it would refuse a valid +// token for the TTL after the authenticator recovers. +func TestAuthenticateFailedReviewNotCached(t *testing.T) { + kube := newFakeKube(map[string]string{"good": "alice"}, allowNamespace("tenant-a")) + outage := "webhook authenticator unreachable" + kube.reviewError.Store(&outage) + a := NewAuthorizer(kube.client, 10*time.Second) + ctx := context.Background() + if _, err := a.Authenticate(ctx, "good"); !errors.Is(err, ErrUnauthenticated) { + t.Fatalf("err = %v, want ErrUnauthenticated", err) + } + kube.reviewError.Store(nil) + if u, err := a.Authenticate(ctx, "good"); err != nil || u.Username != "alice" { + t.Fatalf("after recovery: %+v, %v", u, err) + } +} + +// TestAuthorizeEvaluationErrorNotCached: a SubjectAccessReview with an +// evaluation error and no decision is a denial for this request only. +func TestAuthorizeEvaluationErrorNotCached(t *testing.T) { + kube := newFakeKube(map[string]string{"good": "alice"}, allowNamespace("tenant-a")) + failure := "webhook authorizer unreachable" + kube.evaluationError.Store(&failure) + a := NewAuthorizer(kube.client, 10*time.Second) + ctx := context.Background() + user := &authenticationv1.UserInfo{Username: "alice"} + attrs := Route{Kind: NamespaceLogs, Namespace: "tenant-a"}.ResourceAttributes() + if ok, err := a.Authorize(ctx, user, attrs); err != nil || ok { + t.Fatalf("undecided review = %v, %v; want a denial", ok, err) + } + kube.evaluationError.Store(nil) + if ok, err := a.Authorize(ctx, user, attrs); err != nil || !ok { + t.Fatalf("after recovery: %v, %v", ok, err) + } +} From d3e6fd6aa83f18624ce462f3886663f87e5f16eb Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:25:07 +0700 Subject: [PATCH 262/518] fix(observability-proxy): keep the rate-limit buckets under their bound When the bucket map was full and the sweep freed nothing, the new user's bucket was added anyway, so the map grew past its bound and every further newcomer paid a full scan under the limiter lock. A newcomer is now refused with 429 until the first bucket refills, and the Retry-After tells it when. Resetting the map instead would hand every throttled user a fresh burst. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/limits.go | 21 +++++++++++++---- internal/observabilityproxy/limits_test.go | 27 ++++++++++++++++++++++ 2 files changed, 44 insertions(+), 4 deletions(-) diff --git a/internal/observabilityproxy/limits.go b/internal/observabilityproxy/limits.go index eaebe3104..06cc41d41 100644 --- a/internal/observabilityproxy/limits.go +++ b/internal/observabilityproxy/limits.go @@ -72,7 +72,12 @@ func (l *limiter) allowRate(user string) (time.Duration, bool) { b, ok := l.buckets[user] if !ok { if len(l.buckets) >= ttlCacheMaxEntries { - l.sweepBuckets(now, burst) + // Every bucket still refilling means that many users within one + // refill period; a newcomer waits for the first to drop out + // rather than grow the map past its bound. + if soonest := l.sweepBuckets(now, burst); len(l.buckets) >= ttlCacheMaxEntries { + return soonest, false + } } b = &bucket{tokens: burst, last: now} l.buckets[user] = b @@ -91,13 +96,21 @@ func (l *limiter) allowRate(user string) (time.Duration, bool) { } // sweepBuckets drops buckets that have refilled completely; recreating one -// later starts it full, which is the same state. -func (l *limiter) sweepBuckets(now time.Time, burst float64) { +// later starts it full, which is the same state. It returns how long until +// the first remaining bucket is full. +func (l *limiter) sweepBuckets(now time.Time, burst float64) time.Duration { + soonest := time.Duration(math.MaxInt64) for user, b := range l.buckets { - if b.tokens+now.Sub(b.last).Seconds()*l.cfg.RatePerUser >= burst { + missing := burst - (b.tokens + now.Sub(b.last).Seconds()*l.cfg.RatePerUser) + if missing <= 0 { delete(l.buckets, user) + continue + } + if wait, ok := scaleDuration(missing/l.cfg.RatePerUser, time.Second); ok { + soonest = min(soonest, max(wait, time.Nanosecond)) } } + return soonest } func (l *limiter) acquire(user string) (func(), bool) { diff --git a/internal/observabilityproxy/limits_test.go b/internal/observabilityproxy/limits_test.go index f7ff45d3a..706ed71c7 100644 --- a/internal/observabilityproxy/limits_test.go +++ b/internal/observabilityproxy/limits_test.go @@ -17,6 +17,7 @@ limitations under the License. package observabilityproxy import ( + "fmt" "testing" "time" ) @@ -116,3 +117,29 @@ func TestLimiterRateWaitAtTinyRate(t *testing.T) { t.Fatalf("empty bucket at a tiny rate: admitted %v, wait %s, want refused with a positive wait", ok, wait) } } + +// TestLimiterRateBucketsBounded fills the bucket map with users that are +// all still refilling: a new user must be refused rather than grow it, and +// admitted once a bucket can be dropped. +func TestLimiterRateBucketsBounded(t *testing.T) { + now := time.Unix(1000, 0) + l := newLimiter(LimiterConfig{RatePerUser: 2, RateBurst: 2}) + l.now = func() time.Time { return now } + for i := range ttlCacheMaxEntries { + if !allowed(l, fmt.Sprintf("user-%d", i)) { + t.Fatalf("user-%d refused", i) + } + now = now.Add(time.Microsecond) + } + wait, ok := l.allowRate("newcomer") + if ok || wait <= 0 { + t.Fatalf("newcomer on a full map: admitted %v, wait %s; want refused with a positive wait", ok, wait) + } + if n := len(l.buckets); n > ttlCacheMaxEntries { + t.Fatalf("bucket map grew to %d, above %d", n, ttlCacheMaxEntries) + } + now = now.Add(wait) + if !allowed(l, "newcomer") { + t.Fatalf("newcomer still refused after the advertised wait %s", wait) + } +} From 2e3edcf30dd76256df71d6f3f794f7a139ba069b Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:25:31 +0700 Subject: [PATCH 263/518] fix(observability-proxy): refuse a non-finite per-user rate NaN fails the negative check, as every comparison does, and +Inf refills a bucket instantly, so either value switched the per-user rate limit off while the flag looked set. Both now stop the proxy at startup. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 5 +++++ cmd/observability-proxy/main_test.go | 2 ++ 2 files changed, 7 insertions(+) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index 84cec030c..b05e03807 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -27,6 +27,7 @@ import ( "fmt" "io" "log/slog" + "math" "net/http" "os" "os/signal" @@ -108,6 +109,10 @@ func parseFlags(args []string) (options, error) { return o, errors.New("--logs-max-tail-resume must not be negative") case o.requestTimeout < 0 || o.tailIdleTimeout < 0 || o.authCacheTTL < 0 || o.shutdownTimeout < 0: return o, errors.New("timeouts and the auth cache TTL must not be negative") + // NaN passes every comparison and +Inf never empties a bucket; either + // would switch the rate limit off without saying so. + case math.IsNaN(o.limiter.RatePerUser) || math.IsInf(o.limiter.RatePerUser, 0): + return o, errors.New("--rate-limit-per-user must be a finite number") case o.limiter.RatePerUser < 0 || o.limiter.RateBurst < 0 || o.limiter.MaxConcurrentPerUser < 0 || o.limiter.MaxTailsPerUser < 0 || o.limiter.MaxTails < 0: return o, errors.New("limits must not be negative; 0 disables a limit") diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index 0a96db4bc..3f77f8157 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -63,6 +63,8 @@ func TestParseFlagsInvalid(t *testing.T) { {"--tail-idle-timeout=-1s"}, {"--auth-cache-ttl=-1s"}, {"--rate-limit-per-user=-1"}, + {"--rate-limit-per-user=NaN"}, + {"--rate-limit-per-user=+Inf"}, {"--rate-limit-burst=-1"}, {"--max-concurrent-per-user=-1"}, {"--max-tails-per-user=-1"}, From 2d8de087248bb84ac139fca6d4048c387e8d85ef Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:25:49 +0700 Subject: [PATCH 264/518] fix(observability-proxy): refuse positional arguments The flag parser stops at the first argument that is not a flag, so a stray word in the container args made every flag after it fall back to its default without an error, the root tenant included. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 5 +++++ cmd/observability-proxy/main_test.go | 1 + 2 files changed, 6 insertions(+) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index b05e03807..c228a0a12 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -90,6 +90,11 @@ func parseFlags(args []string) (options, error) { if err := fs.Parse(args); err != nil { return o, err } + // Parsing stops at the first non-flag, so every flag after it would + // be dropped silently. + if fs.NArg() > 0 { + return o, fmt.Errorf("unexpected argument %q", fs.Arg(0)) + } switch { case o.rootTenant == "": return o, errors.New("--root-tenant must not be empty") diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index 3f77f8157..2d4fcece6 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -70,6 +70,7 @@ func TestParseFlagsInvalid(t *testing.T) { {"--max-tails-per-user=-1"}, {"--max-tails=-1"}, {"--unknown"}, + {"unexpected", "--root-tenant=tenant-x"}, } { if _, err := parseFlags(args); err == nil { t.Errorf("parseFlags(%v) accepted", args) From cd04c3e6c40428950324fd43fdc8950e57671283 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:27:06 +0700 Subject: [PATCH 265/518] fix(observability-proxy): count the aligned buckets in the logs step vlselect widens the window to step boundaries and returns a zero-filled bucket for each one, so a start off a boundary yields one bucket more than the span divided by the step. Sizing the step from the unaligned span let hits and stats_query_range return MaxBuckets+1 buckets. The minimum step now spans MaxBuckets-1 steps, and the smallest accepted --logs-max-buckets rises to 4 so that this step still fits the third-of-range cap that keeps alignment inside the window. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 9 ++++--- cmd/observability-proxy/main_test.go | 4 +-- internal/observabilityproxy/logsparams.go | 6 ++++- .../observabilityproxy/logsparams_test.go | 25 ++++++++++++------- 4 files changed, 28 insertions(+), 16 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index c228a0a12..bd2c2e744 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -102,10 +102,11 @@ func parseFlags(args []string) (options, error) { return o, errors.New("--cluster-domain must not be empty") case o.logs.DefaultRange <= 0 || o.logs.MaxRange < o.logs.DefaultRange: return o, errors.New("--logs-default-range must be positive and not above --logs-max-range") - // Below three buckets, or a range under three seconds, the step cap - // that keeps vlselect's step alignment inside the range cannot hold. - case o.logs.MaxBuckets < 3: - return o, errors.New("--logs-max-buckets must be at least 3") + // Below four buckets (three steps once one is held back for + // alignment), or a range under three seconds, the step cap that keeps + // vlselect's step alignment inside the range cannot hold. + case o.logs.MaxBuckets < 4: + return o, errors.New("--logs-max-buckets must be at least 4") case o.logs.MaxRange < 3*time.Second: return o, errors.New("--logs-max-range must be at least 3s") case o.logs.DefaultLimit <= 0 || o.logs.MaxLimit < o.logs.DefaultLimit: diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index 2d4fcece6..ea678461c 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -43,7 +43,7 @@ func TestParseFlagsOverrides(t *testing.T) { t.Errorf("overrides not applied: %+v", o) } // The smallest bucket count and range the step cap can satisfy. - if _, err := parseFlags([]string{"--logs-max-buckets=3", "--logs-max-range=3s", "--logs-default-range=1s"}); err != nil { + if _, err := parseFlags([]string{"--logs-max-buckets=4", "--logs-max-range=3s", "--logs-default-range=1s"}); err != nil { t.Errorf("smallest satisfiable step cap refused: %v", err) } } @@ -53,7 +53,7 @@ func TestParseFlagsInvalid(t *testing.T) { {"--root-tenant="}, {"--cluster-domain="}, {"--logs-default-range=48h", "--logs-max-range=24h"}, - {"--logs-max-buckets=2"}, + {"--logs-max-buckets=3"}, {"--logs-max-range=2s", "--logs-default-range=1s"}, {"--shutdown-timeout=-1s"}, {"--logs-default-limit=0"}, diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index 520f89801..e98915288 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -171,7 +171,11 @@ func window(in url.Values, now time.Time, lim LogsLimits) (time.Time, time.Time, // shrunk from the start until the widened span fits MaxRange. func setStep(out, in url.Values, start, end time.Time, lim LogsLimits) *HTTPError { span := end.Sub(start) - minStep := max((span+time.Duration(lim.MaxBuckets)-1)/time.Duration(lim.MaxBuckets), time.Second) + // vlselect returns a zero-filled bucket for every step boundary from + // the aligned start through the aligned end, one more than span/step + // whenever start is off a boundary, so one bucket is held back. + spans := time.Duration(lim.MaxBuckets - 1) + minStep := max((span+spans-1)/spans, time.Second) step := minStep if v := in.Get("step"); v != "" { d, err := parseDuration(v) diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index 46c62ffb9..6fe370f08 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -222,9 +222,10 @@ func TestBuildLogsParamsStepAndLimit(t *testing.T) { c.Set(k, v) return c } - // 24h over at most 1000 buckets is 86.4s. - if got := build(t, "hits", with(day, "step", "1s")).Get("step"); got != "86400ms" { - t.Errorf("tiny step = %s, want clamped to 86400ms", got) + // 24h over 999 steps, 1000 buckets once aligned, is 86486.5ms, rounded + // up to whole milliseconds. + if got := build(t, "hits", with(day, "step", "1s")).Get("step"); got != "86487ms" { + t.Errorf("tiny step = %s, want clamped to 86487ms", got) } if got := build(t, "stats_query_range", with(day, "step", "1h")).Get("step"); got != "3600000ms" { t.Errorf("1h step = %s", got) @@ -238,7 +239,7 @@ func TestBuildLogsParamsStepAndLimit(t *testing.T) { if got := build(t, "hits", with(day, "step", "5400.5")).Get("step"); got != "5400500ms" { t.Errorf("fractional bare-seconds step = %s, want 5400500ms", got) } - if got := build(t, "hits", day).Get("step"); got != "86400ms" { + if got := build(t, "hits", day).Get("step"); got != "86487ms" { t.Errorf("omitted step = %s, want the bucket-bounded minimum", got) } if got := build(t, "hits", url.Values{"start": {"2026-11-04T11:59:59Z"}}).Get("step"); got != "1000ms" { @@ -341,18 +342,24 @@ func TestBuildLogsParamsStepCannotWidenWindow(t *testing.T) { } } +// TestBuildLogsParamsStepKeepsBucketCap counts buckets the way vlselect +// returns them: one per step boundary from the aligned start through the +// aligned end, zero-filled, which is one more than the unaligned span +// holds whenever start is not on a boundary. func TestBuildLogsParamsStepKeepsBucketCap(t *testing.T) { lim := testLimits() - // Spans that MaxBuckets does not divide into whole milliseconds. - for _, start := range []string{"2026-11-03T12:00:00.0007Z", "2026-11-04T10:00:00.0013Z", "2026-11-04T11:00:00.0005Z"} { + // Spans that MaxBuckets does not divide into whole milliseconds, and + // unaligned starts of spans it does divide. + for _, start := range []string{ + "2026-11-03T12:00:00.0007Z", "2026-11-04T10:00:00.0013Z", "2026-11-04T11:00:00.0005Z", + "2026-11-01T12:00:00.5Z", "2026-11-04T11:00:00.25Z", + } { out := build(t, "hits", url.Values{"start": {start}}) step, err := parseDuration(out.Get("step")) if err != nil { t.Fatal(err) } - s, _ := time.Parse(time.RFC3339Nano, out.Get("start")) - e, _ := time.Parse(time.RFC3339Nano, out.Get("end")) - if buckets := (e.Sub(s) + step - 1) / step; int(buckets) > lim.MaxBuckets { + if buckets := alignedWindow(t, out) / step; int(buckets) > lim.MaxBuckets { t.Errorf("start %s: step %s gives %d buckets, more than %d", start, out.Get("step"), buckets, lim.MaxBuckets) } } From b1108e836a98819d487ea29398a7d78b0013ff84 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:27:38 +0700 Subject: [PATCH 266/518] fix(observability-proxy): refuse to resolve a store before the cache syncs An absent namespace resolves to the root store, and before the informer's first list every namespace is absent, so a request served in that window read a labeled namespace's logs from the wrong store. Readiness keeps the Service from routing there, but a client that addresses the pod directly is not stopped by it. The resolver now fails while the cache is unsynced, which answers 503 like any other resolution error. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 2 +- internal/observabilityproxy/store.go | 14 ++++++++++++-- internal/observabilityproxy/store_test.go | 16 ++++++++++++++-- 3 files changed, 27 insertions(+), 5 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index bd2c2e744..b90bb4ded 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -171,7 +171,7 @@ func run(o options) error { Limiter: o.limiter, RequestTimeout: o.requestTimeout, TailIdleTimeout: o.tailIdleTimeout, - }, observabilityproxy.NewAuthorizer(client, o.authCacheTTL), observabilityproxy.NewStoreResolver(lister, o.rootTenant), reg) + }, observabilityproxy.NewAuthorizer(client, o.authCacheTTL), observabilityproxy.NewStoreResolver(lister, namespaces.Informer().HasSynced, o.rootTenant), reg) probes := http.NewServeMux() probes.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { _, _ = io.WriteString(w, "ok") }) diff --git a/internal/observabilityproxy/store.go b/internal/observabilityproxy/store.go index 0e4041a55..b8881e3ec 100644 --- a/internal/observabilityproxy/store.go +++ b/internal/observabilityproxy/store.go @@ -17,6 +17,7 @@ limitations under the License. package observabilityproxy import ( + "errors" "fmt" "strings" @@ -34,11 +35,14 @@ const monitoringLabel = "namespace.cozystack.io/monitoring" // its logs, by the same rule the collector writes with. type StoreResolver struct { namespaces corev1listers.NamespaceLister + synced func() bool root string } -func NewStoreResolver(namespaces corev1listers.NamespaceLister, rootTenant string) *StoreResolver { - return &StoreResolver{namespaces: namespaces, root: rootTenant} +// NewStoreResolver reads namespaces from a cache that synced reports as +// filled. +func NewStoreResolver(namespaces corev1listers.NamespaceLister, synced func() bool, rootTenant string) *StoreResolver { + return &StoreResolver{namespaces: namespaces, synced: synced, root: rootTenant} } // Resolve answers the root tenant for a namespace that no longer exists, @@ -46,6 +50,12 @@ func NewStoreResolver(namespaces corev1listers.NamespaceLister, rootTenant strin // tenant's store are not found there; reading them needs routing history // (an open question of the design). func (r *StoreResolver) Resolve(namespace string) (string, error) { + // Before the first list every namespace looks gone, which would send + // a labeled one to the root store; readiness alone does not stop a + // client that addresses the pod directly. + if !r.synced() { + return "", errors.New("namespace cache not synced") + } ns, err := r.namespaces.Get(namespace) if apierrors.IsNotFound(err) { return r.root, nil diff --git a/internal/observabilityproxy/store_test.go b/internal/observabilityproxy/store_test.go index a4a032f57..d21476b98 100644 --- a/internal/observabilityproxy/store_test.go +++ b/internal/observabilityproxy/store_test.go @@ -48,7 +48,8 @@ func TestStoreResolver(t *testing.T) { } add("tenant-forged", map[string]string{monitoringLabel: "pod.tenant-b"}) lister := corev1listers.NewNamespaceLister(indexer) - if got, err := NewStoreResolver(lister, "tenant-root").Resolve("tenant-forged"); err == nil { + synced := func() bool { return true } + if got, err := NewStoreResolver(lister, synced, "tenant-root").Resolve("tenant-forged"); err == nil { t.Fatalf("label naming a host outside a namespace resolved to %q", got) } @@ -70,7 +71,7 @@ func TestStoreResolver(t *testing.T) { {"tenant-a", "cozy-system", "tenant-a"}, } for _, tt := range tests { - r := NewStoreResolver(lister, tt.root) + r := NewStoreResolver(lister, synced, tt.root) got, err := r.Resolve(tt.ns) if err != nil { t.Fatalf("root=%s Resolve(%s) error: %v", tt.root, tt.ns, err) @@ -80,3 +81,14 @@ func TestStoreResolver(t *testing.T) { } } } + +// TestStoreResolverUnsynced: before the first list, every namespace looks +// absent, and an absent one resolves to the root store; a labeled +// namespace would read the wrong store. +func TestStoreResolverUnsynced(t *testing.T) { + indexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) + r := NewStoreResolver(corev1listers.NewNamespaceLister(indexer), func() bool { return false }, "tenant-root") + if got, err := r.Resolve("tenant-a"); err == nil { + t.Fatalf("unsynced cache resolved tenant-a to %q", got) + } +} From f775948d2cdb0add0f14371c3cb4e7fcde6f7bfd Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:27:45 +0700 Subject: [PATCH 267/518] test(observability-proxy): replay the early line into the e2e tail The background tail goes through the console route and the proxy's reviews before it subscribes, so the early ingest can land first, and a tail without start_offset returns only lines written after it starts. The tail now replays the last minute, which still leaves the hold and the late-line check meaningful. Assisted-by: LLM Signed-off-by: ZverGuy --- hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml b/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml index e0656d4f8..576f6fa6d 100644 --- a/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/observability-proxy/chainsaw-test.yaml @@ -227,8 +227,10 @@ spec: forward cozy-dashboard cozy-dashboard-console 18080:8080 token=$(k -n tenant-test-logsa create token logs-viewer --duration=10m) + # The tail is not subscribed yet when the next ingest runs, so it + # replays the last minute to include the early line. timeout 150 curl --silent --no-buffer --header "Authorization: Bearer $token" \ - "$CONSOLE$LOGS_A/tail?query=*" > "$tail_out" & + "$CONSOLE$LOGS_A/tail?query=*&start_offset=1m" > "$tail_out" & tail_pid=$! ingest tenant-test-logsa e2e-tail-early ingest tenant-test-logsb e2e-tail-b From b703a54cb4920e221073851d831d8df5ff9532d9 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:47:39 +0700 Subject: [PATCH 268/518] fix(tenant): drop the tenant egress rule to the observability proxy The rule was added for a tenant Grafana reading through the proxy, but the tenant Grafana datasources still point straight at vlselect and vmselect, and the console reaches the proxy from cozy-dashboard. The rule therefore served no caller and only let every pod of every tenant reach the proxy directly, outside the console route, where each request with a fresh token costs a TokenReview. It can come back together with a tenant client that needs it. Assisted-by: LLM Signed-off-by: ZverGuy --- .../apps/tenant/templates/networkpolicy.yaml | 17 ------ ...etworkpolicy_observability_proxy_test.yaml | 61 ------------------- 2 files changed, 78 deletions(-) delete mode 100644 packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml diff --git a/packages/apps/tenant/templates/networkpolicy.yaml b/packages/apps/tenant/templates/networkpolicy.yaml index 76959b89f..5285b495a 100644 --- a/packages/apps/tenant/templates/networkpolicy.yaml +++ b/packages/apps/tenant/templates/networkpolicy.yaml @@ -187,23 +187,6 @@ spec: --- apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy -metadata: - name: allow-to-observability-proxy - namespace: {{ include "tenant.name" . }} -spec: - endpointSelector: {} - egress: - - toEndpoints: - - matchLabels: - "k8s:io.kubernetes.pod.namespace": cozy-observability-proxy - app.kubernetes.io/name: observability-proxy - toPorts: - - ports: - - port: "8080" - protocol: TCP ---- -apiVersion: cilium.io/v2 -kind: CiliumNetworkPolicy metadata: # Allow the tenant's ingress-nginx to reach both the linstor-gui # gatekeeper pods (for /oauth2/* and app traffic) AND the transient diff --git a/packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml b/packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml deleted file mode 100644 index d6276ec34..000000000 --- a/packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml +++ /dev/null @@ -1,61 +0,0 @@ -suite: tenant egress to the observability proxy -# A tenant's Grafana reads the tenant's logs and metrics through the -# observability proxy in cozy-observability-proxy. Tenant egress is -# default-deny plus explicit allow-to-* policies, so without this one those -# reads time out. The console reaches the proxy from cozy-dashboard and needs -# no tenant rule. -templates: - - templates/networkpolicy.yaml -tests: - - it: renders in the root tenant - release: - name: tenant-root - namespace: tenant-root - documentSelector: - path: metadata.name - value: allow-to-observability-proxy - asserts: - - isKind: - of: CiliumNetworkPolicy - - equal: - path: metadata.namespace - value: tenant-root - - - it: renders in a nested tenant with every module off - release: - name: tenant-bar - namespace: tenant-foo - set: - monitoring: false - ingress: false - etcd: false - documentSelector: - path: metadata.name - value: allow-to-observability-proxy - asserts: - - equal: - path: metadata.namespace - value: tenant-foo-bar - - - it: lets every pod of the tenant reach only the proxy pods on their container port - release: - name: tenant-root - namespace: tenant-root - documentSelector: - path: metadata.name - value: allow-to-observability-proxy - asserts: - - equal: - path: spec.endpointSelector - value: {} - - equal: - path: spec.egress - value: - - toEndpoints: - - matchLabels: - "k8s:io.kubernetes.pod.namespace": cozy-observability-proxy - app.kubernetes.io/name: observability-proxy - toPorts: - - ports: - - port: "8080" - protocol: TCP From 9d9f5fcb6cd9c4c74dfdff8bea2d95f4cd150b1c Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:47:39 +0700 Subject: [PATCH 269/518] docs(observability-proxy): name the source of the logs bucket default The default of 11000 buckets was justified by a metrics proxy that does not exist on this branch. The number is Prometheus's cap on points per series in a range query, which the metrics root reuses. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index b90bb4ded..f8ccd38be 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -77,7 +77,7 @@ func parseFlags(args []string) (options, error) { // One week covers an incident investigation and keeps a single query // to a fraction of the default one-month log retention. fs.DurationVar(&o.logs.MaxRange, "logs-max-range", 7*24*time.Hour, "longest [start, end] a logs request may span; wider windows are clamped") - // The same cap on points as the metrics proxy. + // Prometheus's cap on points per series in a range query. fs.IntVar(&o.logs.MaxBuckets, "logs-max-buckets", 11000, "most buckets hits and stats_query_range may return over a window") fs.IntVar(&o.logs.DefaultLimit, "logs-default-limit", 1000, "entries returned when a request sets no limit") fs.IntVar(&o.logs.MaxLimit, "logs-max-limit", 10000, "largest limit a request may set") From c15441034d13906b5a758ca0c2b4d12811513cc5 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:47:49 +0700 Subject: [PATCH 270/518] chore(observability-proxy): ignore the root-level command binary go build ./cmd/observability-proxy drops the binary at the repository root like every other command, and unlike them it was not ignored, so git add . would commit it. Assisted-by: LLM Signed-off-by: ZverGuy --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index db787248d..b2cc2be1e 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,7 @@ _repos /flux-shard-operator /kubeovn-plunger /lineage-controller-webhook +/observability-proxy /securitygroup-controller # User-specific stuff From 2b6a1f45481836f37f2478510a7a2fcea5a6b621 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:54:56 +0700 Subject: [PATCH 271/518] test(observability-proxy): drop the tenant egress policy from test notes The chart tests named the tenant egress policy as one of the parties the pod label and port are pinned for. That policy is gone, so the notes now name only the Service and the console nginx, which still depend on them. Assisted-by: LLM Signed-off-by: ZverGuy --- .../system/observability-proxy/tests/deployment_test.yaml | 6 +++--- packages/system/observability-proxy/tests/service_test.yaml | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/packages/system/observability-proxy/tests/deployment_test.yaml b/packages/system/observability-proxy/tests/deployment_test.yaml index c8415be01..ddf9d34b8 100644 --- a/packages/system/observability-proxy/tests/deployment_test.yaml +++ b/packages/system/observability-proxy/tests/deployment_test.yaml @@ -1,7 +1,7 @@ suite: observability-proxy Deployment # The flags, ports and pod label below are the contract the proxy binary, the -# Service, the console nginx and the tenant egress policy are written against. -# A rename on either side breaks routing without failing a render. +# Service and the console nginx are written against. A rename on either side +# breaks routing without failing a render. templates: - templates/deployment.yaml @@ -10,7 +10,7 @@ release: namespace: cozy-observability-proxy tests: - - it: labels the pods with the name the tenant egress policy selects on + - it: labels the pods with the name the Service selects on asserts: - isKind: of: Deployment diff --git a/packages/system/observability-proxy/tests/service_test.yaml b/packages/system/observability-proxy/tests/service_test.yaml index 0fb8b0e00..123cc01ae 100644 --- a/packages/system/observability-proxy/tests/service_test.yaml +++ b/packages/system/observability-proxy/tests/service_test.yaml @@ -1,7 +1,7 @@ suite: observability-proxy Service -# The console nginx proxies to observability-proxy..svc on port 80 and the -# tenant egress policy allows the pods' 8080, so the name, the port and the -# selector are pinned here. +# The console nginx proxies to observability-proxy..svc on port 80, which +# must reach the pods' 8080, so the name, the port and the selector are pinned +# here. templates: - templates/service.yaml From 3e907e6f9a829667e744f8ea812d3f2175b37462 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:54:56 +0700 Subject: [PATCH 272/518] test(observability-proxy): declare the shell of the e2e helper library The library is sourced by sh scripts and has no shebang, so shellcheck could not tell which shell to check it as and refused the file. Assisted-by: LLM Signed-off-by: ZverGuy --- hack/e2e-chainsaw/observability-proxy/lib.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/hack/e2e-chainsaw/observability-proxy/lib.sh b/hack/e2e-chainsaw/observability-proxy/lib.sh index ff41b3e5e..a9a1f28a5 100644 --- a/hack/e2e-chainsaw/observability-proxy/lib.sh +++ b/hack/e2e-chainsaw/observability-proxy/lib.sh @@ -1,3 +1,4 @@ +# shellcheck shell=sh # Shared by the script steps of this suite, whose cwd is this directory. # Every call carries its own cap so a hang fails at a known budget rather # than at the op ceiling (docs/agents/e2e-testing.md, section 8). From 7dc6ac3687fcb9fe5fba35686878158d18bd3232 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:20:00 +0700 Subject: [PATCH 273/518] fix(observability-proxy): read a leftover request body under the deadline ParseForm reads no body on a GET or on a POST that is not a form. When such a request declared a body and never sent it, net/http drained it after the handler flushed the headers, with no read deadline, so the request held its slot for as long as the client kept the connection. The handler now reads the rest of the body itself, under the same deadline and size cap as the form. Draining was chosen over refusing a declared body so that clients which send one stay working. A body over the cap now answers 413 instead of a generic 400. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 14 ++++- .../server_deadline_test.go | 53 +++++++++++++++++++ 2 files changed, 65 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 2e391442c..0f9e533b7 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -266,8 +266,12 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT return nil } -// readForm reads the body under the request timeout, so a client that -// sends it slowly holds its slot only that long. Once the body is read the +// readForm reads the whole body under the request timeout, so a client +// that sends it slowly holds its slot only that long. ParseForm leaves the +// body of a GET or of a POST that is not a form unread, and net/http +// drains a leftover body with no deadline once the headers go out, so the +// rest is read here as well; draining rather than refusing a declared +// body keeps any client that sends one working. Once the body is read the // deadline is lifted: the server's background read would take a later // expiry for a client that went away and cancel a tail. After a failed // read it stays, so the server's drain of the unread body is bounded too. @@ -278,7 +282,13 @@ func (s *Server) readForm(w http.ResponseWriter, r *http.Request) *HTTPError { _ = rc.SetReadDeadline(time.Now().Add(s.cfg.RequestTimeout)) } err := r.ParseForm() + if err == nil { + _, err = io.Copy(io.Discard, r.Body) + } + var tooLarge *http.MaxBytesError switch { + case errors.As(err, &tooLarge): + return errorf(http.StatusRequestEntityTooLarge, "request body larger than %d bytes", tooLarge.Limit) case err == nil: _ = rc.SetReadDeadline(time.Time{}) return nil diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index afd69ed30..ac62dd782 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -21,6 +21,7 @@ import ( "io" "net" "net/http" + "strings" "sync" "testing" "time" @@ -244,3 +245,55 @@ func TestServerAbortsTruncatedResponse(t *testing.T) { t.Fatal("a truncated store answer reached the client as a complete response") } } + +// TestServerUndeliveredBodyReleasesSlot declares a body that never comes +// on requests whose form parsing does not read it: a GET, a POST that is +// not a form, and a GET tail. net/http would drain that body with no +// deadline once the headers go out, so the slot must still come back. +func TestServerUndeliveredBodyReleasesSlot(t *testing.T) { + for name, raw := range map[string]string{ + "get query": "GET " + logsRoot + "query?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\nContent-Length: 10\r\n\r\n", + "post text query": "POST " + logsRoot + "query?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\nContent-Type: text/plain\r\nContent-Length: 10\r\n\r\n", + "get tail": "GET " + logsRoot + "tail?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\nContent-Length: 10\r\n\r\n", + } { + t.Run(name, func(t *testing.T) { + lines := make(chan string, 1) + done := make(chan struct{}) + h := newHarness(t, Config{ + RequestTimeout: 200 * time.Millisecond, + TailIdleTimeout: 200 * time.Millisecond, + Limiter: LimiterConfig{MaxConcurrentPerUser: 1, MaxTails: 1}, + }, tailUpstream(lines, done)) + t.Cleanup(func() { close(done) }) + pipe := servePipe(t, h.server) + conn := pipe.send(t, raw) + held := func() bool { + h.server.limits.mu.Lock() + defer h.server.limits.mu.Unlock() + return h.server.limits.active["alice"]+h.server.limits.total > 0 + } + eventually(t, time.Second, held, "the request never took a slot") + if resp := readResponse(t, conn); resp.StatusCode != http.StatusRequestTimeout { + t.Fatalf("undelivered body answered %d, want 408", resp.StatusCode) + } + eventually(t, time.Second, func() bool { return !held() }, "an undelivered body keeps its slot") + }) + } +} + +func TestServerOversizedBody(t *testing.T) { + h := newHarness(t, Config{}, nil) + for _, ct := range []string{"application/x-www-form-urlencoded", "text/plain"} { + req, _ := http.NewRequest(http.MethodPost, h.proxy.URL+logsRoot+"query?query=*", strings.NewReader("query="+strings.Repeat("x", maxFormBytes))) + req.Header.Set("Authorization", "Bearer good") + req.Header.Set("Content-Type", ct) + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatal(err) + } + _ = resp.Body.Close() + if resp.StatusCode != http.StatusRequestEntityTooLarge { + t.Errorf("%s body over the cap answered %d, want 413", ct, resp.StatusCode) + } + } +} From f2944c572d33b55a790ddc187c8fff76b64308ed Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:21:58 +0700 Subject: [PATCH 274/518] fix(dashboard): answer not-installed when the observability proxy is absent The proxy can be left out of an installation, and then the console nginx answered the observability roots with its own 502. The console read that as an unreachable store: three errors, a Retry button and a live tail reconnecting forever, instead of a notice that the feature is not available on this cluster. nginx now turns the 502 and 504 it generates itself, when it cannot resolve or reach the proxy, into a 404 Kubernetes Status, which the console already treats as not installed. Upstream errors are not intercepted, so the proxy's own 502 and 504 for a store still reach the console unchanged. The bats test runs the extracted locations in a real nginx when one is installed. Assisted-by: LLM Signed-off-by: ZverGuy --- hack/console-nginx-observability-proxy.bats | 90 +++++++++++++++++++ .../dashboard/images/console/Containerfile | 18 +++- 2 files changed, 105 insertions(+), 3 deletions(-) diff --git a/hack/console-nginx-observability-proxy.bats b/hack/console-nginx-observability-proxy.bats index 0efb0ffcd..ef9c297b3 100644 --- a/hack/console-nginx-observability-proxy.bats +++ b/hack/console-nginx-observability-proxy.bats @@ -144,3 +144,93 @@ matches() { [ "$used" = "$expected" ] \ || { echo "template substitutes: $used; the image defines: $expected" >&2; exit 1; } } + +# Prints the named location $2 of the template $1, braces included. +named_location() { + printf '%s\n' "$1" | LOCATION_NAME="$2" awk ' + $1 == "location" && $2 == ENVIRON["LOCATION_NAME"] { inblock = 1 } + inblock { print } + inblock && $1 == "}" { exit } + ' +} + +free_port() { + python3 -c 'import socket; s = socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1])' +} + +# Starts nginx with the observability locations of the console template, the +# upstream pointed at 127.0.0.1:$1, and a stand-in proxy on 127.0.0.1:$2 +# that answers 502 itself. Prints the console port. +start_console_nginx() { + upstream_port=$1 standin_port=$2 + conf=$(console_nginx_template) + re=$(regex_locations "$conf") + console_port=$(free_port) + { + echo "pid $BATS_TEST_TMPDIR/nginx.pid;" + echo "error_log $BATS_TEST_TMPDIR/error.log;" + echo "events {}" + echo "http {" + echo " access_log off;" + echo " client_body_temp_path $BATS_TEST_TMPDIR; proxy_temp_path $BATS_TEST_TMPDIR;" + echo " fastcgi_temp_path $BATS_TEST_TMPDIR; uwsgi_temp_path $BATS_TEST_TMPDIR; scgi_temp_path $BATS_TEST_TMPDIR;" + echo " server {" + echo " listen 127.0.0.1:$console_port;" + echo " location ~ $re {" + # The test upstream is an address, so the resolver is never asked. + location_body "$conf" "$re" \ + | sed -e 's/\${NGINX_LOCAL_RESOLVERS}/127.0.0.1/' \ + -e "s/^\([[:space:]]*set[[:space:]]*\$observability_proxy[[:space:]]\).*/\1127.0.0.1:$upstream_port;/" + echo " }" + named_location "$conf" "@observability_proxy_unavailable" + echo " }" + echo " server {" + echo " listen 127.0.0.1:$standin_port;" + echo " location / { return 502 'log store unreachable'; }" + echo " }" + echo "}" + } > "$BATS_TEST_TMPDIR/nginx.conf" + nginx -p "$BATS_TEST_TMPDIR" -c "$BATS_TEST_TMPDIR/nginx.conf" \ + || { cat "$BATS_TEST_TMPDIR/error.log" >&2; return 1; } + echo "$console_port" +} + +teardown() { + if [ -f "${BATS_TEST_TMPDIR:-}/nginx.pid" ]; then + kill "$(cat "$BATS_TEST_TMPDIR/nginx.pid")" 2>/dev/null || true + fi +} + +@test "a cluster without the proxy answers the console's not-installed Status, the proxy's own 502 passes through" { + command -v nginx >/dev/null 2>&1 || skip "nginx is not installed" + command -v curl >/dev/null 2>&1 || skip "curl is not installed" + path=/apis/cozystack.io/v1alpha1/namespaces/tenant-foo/logs/select/logsql/query + + # Nothing listens on the upstream port: nginx itself fails the request. + standin=$(free_port) + port=$(start_console_nginx "$(free_port)" "$standin") + out=$(curl --silent --show-error --include --max-time 10 "http://127.0.0.1:$port$path") + printf '%s\n' "$out" | head -n 1 | grep -q ' 404' \ + || { echo "FAIL: an unreachable proxy did not answer 404: $out" >&2; false; } + printf '%s\n' "$out" | grep -iq '^content-type: application/json' \ + || { echo "FAIL: the not-installed answer is not JSON: $out" >&2; false; } + body=$(printf '%s\n' "$out" | tail -n 1) + printf '%s' "$body" | python3 -c ' +import json, sys +s = json.load(sys.stdin) +assert s["kind"] == "Status" and s["apiVersion"] == "v1" and s["status"] == "Failure", s +assert s["reason"] == "NotFound" and s["code"] == 404, s +assert s["message"] == "the observability proxy is not available on this cluster", s +' || { echo "FAIL: the not-installed body is not the expected Status: $body" >&2; false; } + kill "$(cat "$BATS_TEST_TMPDIR/nginx.pid")" + rm -f "$BATS_TEST_TMPDIR/nginx.pid" + sleep 0.2 + + # The proxy answers 502 itself: the console must see that answer. + port=$(start_console_nginx "$standin" "$standin") + out=$(curl --silent --show-error --include --max-time 10 "http://127.0.0.1:$port$path") + printf '%s\n' "$out" | head -n 1 | grep -q ' 502' \ + || { echo "FAIL: the proxy's own 502 did not pass through: $out" >&2; false; } + printf '%s\n' "$out" | tail -n 1 | grep -qx 'log store unreachable' \ + || { echo "FAIL: the proxy's own 502 body was replaced: $out" >&2; false; } +} diff --git a/packages/system/dashboard/images/console/Containerfile b/packages/system/dashboard/images/console/Containerfile index dba07f8d0..d7493c60c 100644 --- a/packages/system/dashboard/images/console/Containerfile +++ b/packages/system/dashboard/images/console/Containerfile @@ -64,9 +64,9 @@ server { # Logs and metrics roots of a namespace or of one application, served by # the observability proxy rather than the API server. A regex location wins # over the /apis prefix below. The upstream is a variable so nginx resolves - # it per request: on a cluster without the proxy these paths answer 502 - # instead of nginx refusing to start. The resolver does not apply search - # domains, hence the fully qualified name. + # it per request: on a cluster without the proxy nginx still starts, and a + # request it cannot deliver gets the not-installed answer below. The + # resolver does not apply search domains, hence the fully qualified name. location ~ ^/apis/(?:cozystack\.io/v1alpha1/namespaces/[^/]+/(?:logs|metrics)|apps\.cozystack\.io/v1alpha1/namespaces/[^/]+/[^/]+/[^/]+/(?:log|metrics))/ { resolver ${NGINX_LOCAL_RESOLVERS}; set $observability_proxy observability-proxy.cozy-observability-proxy.svc.${CLUSTER_DOMAIN}:80; @@ -75,9 +75,21 @@ server { proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 86400s; proxy_send_timeout 86400s; + # With proxy_intercept_errors off this catches only the 502 and 504 + # nginx makes itself when the proxy cannot be resolved or reached; + # the proxy's own 502 and 504 for an unreachable store pass through. + proxy_intercept_errors off; + error_page 502 504 = @observability_proxy_unavailable; proxy_pass http://$observability_proxy; } + # The console reads a Kubernetes Status body as "not installed", the same + # answer kube-apiserver gives for a root nothing serves. + location @observability_proxy_unavailable { + default_type application/json; + return 404 '{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"the observability proxy is not available on this cluster","reason":"NotFound","code":404}'; + } + # Kubernetes API proxy for both core and aggregated groups. # Chunked-encoding watches require proxy_buffering off and a long timeout. location /apis { From 95ed8134e4a540c57f5e79ba74a3b90f9bf25fea Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:22:32 +0700 Subject: [PATCH 275/518] fix(observability-proxy): refuse an empty fields_limit on hits An empty fields_limit fell back to the default of 0 and reached vlselect as fields_limit=0, instead of being refused like any other malformed limit. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 4 ++++ internal/observabilityproxy/logsparams_test.go | 2 ++ 2 files changed, 6 insertions(+) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index e98915288..67462c989 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -104,6 +104,10 @@ func BuildLogsParams(endpoint string, in url.Values, namespace string, now time. } } if in.Has("fields_limit") { + // setLimit reads an empty value as the default, which here is 0. + if in.Get("fields_limit") == "" { + return nil, errorf(http.StatusBadRequest, "invalid fields_limit \"\"") + } herr = setLimit(out, in, "fields_limit", 0, lim.MaxLimit) } case "stats_query_range": diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index 6fe370f08..62fdc26ac 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -116,6 +116,8 @@ func TestBuildLogsParamsRejects(t *testing.T) { code int }{ {"unknown endpoint", "tenant_ids", url.Values{}, http.StatusNotFound}, + // An empty value used to reach vlselect as fields_limit=0. + {"empty fields_limit", "hits", url.Values{"field": {"level"}, "fields_limit": {""}}, http.StatusBadRequest}, {"stats_query is not proxied", "stats_query", url.Values{}, http.StatusNotFound}, {"ignore_global_time_filter", "query", url.Values{"query": {`* options(ignore_global_time_filter=true)`}}, http.StatusBadRequest}, {"ignore_global_time_filter in subquery", "hits", url.Values{"query": {`* | union (options(ignore_global_time_filter=true) error)`}}, http.StatusBadRequest}, From 650a99ff272ce55afc7bc77ea1350f44491e2a0e Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:23:00 +0700 Subject: [PATCH 276/518] fix(observability-proxy): accept the bearer scheme in any case The authorization scheme is case-insensitive, and kube-apiserver accepts "bearer" as well as "Bearer", so a client that works against the API server got 401 from the proxy with the same header. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 4 ++-- internal/observabilityproxy/server_test.go | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 0f9e533b7..2e29d3146 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -300,8 +300,8 @@ func (s *Server) readForm(w http.ResponseWriter, r *http.Request) *HTTPError { } func (s *Server) authorize(r *http.Request, route Route) (string, *HTTPError) { - token, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ") - if !ok { + scheme, token, _ := strings.Cut(r.Header.Get("Authorization"), " ") + if !strings.EqualFold(scheme, "Bearer") { return "", errorf(http.StatusUnauthorized, "missing bearer token") } user, err := s.auth.Authenticate(r.Context(), strings.TrimSpace(token)) diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index fe3805961..7f397fcd3 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -452,3 +452,21 @@ func TestServerTailLongLine(t *testing.T) { t.Fatalf("long line not streamed intact: got %d bytes", len(body)) } } + +// The auth scheme is case-insensitive (RFC 9110, section 11.1), as +// kube-apiserver reads it. +func TestServerBearerSchemeCase(t *testing.T) { + h := newHarness(t, Config{}, nil) + for _, auth := range []string{"Bearer good", "bearer good", "BEARER good"} { + resp := h.do(http.MethodGet, logsRoot+"query", "", nil, map[string]string{"Authorization": auth}) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusOK { + t.Errorf("Authorization %q answered %d, want 200", auth, resp.StatusCode) + } + } + resp := h.do(http.MethodGet, logsRoot+"query", "", nil, map[string]string{"Authorization": "Basic good"}) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusUnauthorized { + t.Errorf("Basic scheme answered %d, want 401", resp.StatusCode) + } +} From 1ef10537f6ecb74b0d6c4f87b1c73ff0e3de6b21 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:23:29 +0700 Subject: [PATCH 277/518] fix(observability-proxy): send Retry-After with the concurrency 429 Only the rate-limit 429 said when to retry. A concurrency or tail 429 came without it, so a client had no hint and could retry at once in a loop. It now carries a one-second hint; when a slot frees depends on the user's other requests, which the proxy cannot predict. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 12 +++++++++++- internal/observabilityproxy/server_test.go | 22 ++++++++++++++++++++++ 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 2e29d3146..f511a6691 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -179,7 +179,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT } release, ok := acquire(user) if !ok { - return errorf(http.StatusTooManyRequests, "too many concurrent requests") + return tooManyConcurrent(w) } defer release() if herr := s.readForm(w, r); herr != nil { @@ -266,6 +266,16 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT return nil } +// concurrencyRetryAfter is a hint only: a slot frees when another request +// of the user ends, which the proxy cannot predict, and one second keeps +// a polling client from retrying in a tight loop. +const concurrencyRetryAfter = "1" + +func tooManyConcurrent(w http.ResponseWriter) *HTTPError { + w.Header().Set("Retry-After", concurrencyRetryAfter) + return errorf(http.StatusTooManyRequests, "too many concurrent requests") +} + // readForm reads the whole body under the request timeout, so a client // that sends it slowly holds its slot only that long. ParseForm leaves the // body of a GET or of a POST that is not a form unread, and net/http diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index 7f397fcd3..ed1aaedc0 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -470,3 +470,25 @@ func TestServerBearerSchemeCase(t *testing.T) { t.Errorf("Basic scheme answered %d, want 401", resp.StatusCode) } } + +// A 429 for concurrency tells the client when to retry, as the rate 429 +// does, so console polls back off instead of retrying at once. +func TestServerConcurrencyRetryAfter(t *testing.T) { + release := make(chan struct{}) + h := newHarness(t, Config{Limiter: LimiterConfig{MaxConcurrentPerUser: 1, MaxTails: 1}}, func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(http.StatusOK) + w.(http.Flusher).Flush() + <-release + }) + t.Cleanup(func() { close(release) }) + for _, endpoint := range []string{"query", "tail"} { + first := h.do(http.MethodGet, logsRoot+endpoint, "good", nil, nil) + defer func() { _ = first.Body.Close() }() + second := h.do(http.MethodGet, logsRoot+endpoint, "good", nil, nil) + _ = second.Body.Close() + if second.StatusCode != http.StatusTooManyRequests || second.Header.Get("Retry-After") == "" { + t.Errorf("%s over the limit answered %d with Retry-After %q, want 429 with a value", + endpoint, second.StatusCode, second.Header.Get("Retry-After")) + } + } +} From 38f9837f144dd503623caecc4048ee6371e5a79e Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:38:47 +0700 Subject: [PATCH 278/518] fix(observability-proxy): cap the hits groups when fields_limit is absent The fields_limit cap applied only when the client sent the parameter. vlselect returns every group when it is absent, as it does for 0, so leaving it out bypassed the cap. The capped value is now always sent. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 12 ++++++------ internal/observabilityproxy/logsparams_test.go | 7 ++++++- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index 67462c989..93e1bd9a4 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -103,13 +103,13 @@ func BuildLogsParams(endpoint string, in url.Values, namespace string, now time. out.Add("field", f) } } - if in.Has("fields_limit") { - // setLimit reads an empty value as the default, which here is 0. - if in.Get("fields_limit") == "" { - return nil, errorf(http.StatusBadRequest, "invalid fields_limit \"\"") - } - herr = setLimit(out, in, "fields_limit", 0, lim.MaxLimit) + // vlselect returns every group without fields_limit or with 0, so + // the cap is sent either way; an empty value is malformed rather + // than a request for the default. + if in.Has("fields_limit") && in.Get("fields_limit") == "" { + return nil, errorf(http.StatusBadRequest, "invalid fields_limit \"\"") } + herr = setLimit(out, in, "fields_limit", lim.MaxLimit, lim.MaxLimit) case "stats_query_range": herr = setStep(out, in, start, end, lim) case "field_values", "stream_field_values": diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index 62fdc26ac..f39cc8846 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -116,7 +116,7 @@ func TestBuildLogsParamsRejects(t *testing.T) { code int }{ {"unknown endpoint", "tenant_ids", url.Values{}, http.StatusNotFound}, - // An empty value used to reach vlselect as fields_limit=0. + // vlselect reads fields_limit=0 as no limit on the groups. {"empty fields_limit", "hits", url.Values{"field": {"level"}, "fields_limit": {""}}, http.StatusBadRequest}, {"stats_query is not proxied", "stats_query", url.Values{}, http.StatusNotFound}, {"ignore_global_time_filter", "query", url.Values{"query": {`* options(ignore_global_time_filter=true)`}}, http.StatusBadRequest}, @@ -264,6 +264,11 @@ func TestBuildLogsParamsStepAndLimit(t *testing.T) { if facets.Get("max_values_per_field") != "5000" || facets.Get("max_value_len") != "5000" { t.Errorf("facets caps not clamped: %v", facets) } + // Without fields_limit vlselect returns every group, so the cap is + // sent even when the client sends none. + if got := build(t, "hits", url.Values{"field": {"level"}}).Get("fields_limit"); got != "5000" { + t.Errorf("hits without fields_limit forwards %q, want the 5000 cap", got) + } hits := build(t, "hits", url.Values{"field": {"b", "a"}, "fields_limit": {"99999"}}) fields := hits["field"] sort.Strings(fields) From 1004f2f2d89e89fb7099c80ff994170b4e0e6223 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:38:47 +0700 Subject: [PATCH 279/518] test(observability-proxy): pin the refused forms of the bearer header Parsing the scheme with a case-insensitive split changed which header forms reach the token review; a bare scheme, an empty token and a scheme run into the token must still answer 401. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server_test.go | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index ed1aaedc0..a5a9ee2d8 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -464,10 +464,12 @@ func TestServerBearerSchemeCase(t *testing.T) { t.Errorf("Authorization %q answered %d, want 200", auth, resp.StatusCode) } } - resp := h.do(http.MethodGet, logsRoot+"query", "", nil, map[string]string{"Authorization": "Basic good"}) - _ = resp.Body.Close() - if resp.StatusCode != http.StatusUnauthorized { - t.Errorf("Basic scheme answered %d, want 401", resp.StatusCode) + for _, auth := range []string{"Basic good", "Bearer", "Bearer ", "Bearergood"} { + resp := h.do(http.MethodGet, logsRoot+"query", "", nil, map[string]string{"Authorization": auth}) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusUnauthorized { + t.Errorf("Authorization %q answered %d, want 401", auth, resp.StatusCode) + } } } From 5db02ab47e623156b59137ca545864aa99679924 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:44:29 +0700 Subject: [PATCH 280/518] test(observability-proxy): pin the concurrency Retry-After value The test only checked that a Retry-After was present, while the rate limit test pins its value, so a change to the promised hint passed. It now expects the literal value. The note on the empty fields_limit case now gives the reason it is refused. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams_test.go | 2 +- internal/observabilityproxy/server_test.go | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index f39cc8846..009b1721d 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -116,7 +116,7 @@ func TestBuildLogsParamsRejects(t *testing.T) { code int }{ {"unknown endpoint", "tenant_ids", url.Values{}, http.StatusNotFound}, - // vlselect reads fields_limit=0 as no limit on the groups. + // Malformed rather than a request for the default. {"empty fields_limit", "hits", url.Values{"field": {"level"}, "fields_limit": {""}}, http.StatusBadRequest}, {"stats_query is not proxied", "stats_query", url.Values{}, http.StatusNotFound}, {"ignore_global_time_filter", "query", url.Values{"query": {`* options(ignore_global_time_filter=true)`}}, http.StatusBadRequest}, diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index a5a9ee2d8..df3e23bc4 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -488,8 +488,8 @@ func TestServerConcurrencyRetryAfter(t *testing.T) { defer func() { _ = first.Body.Close() }() second := h.do(http.MethodGet, logsRoot+endpoint, "good", nil, nil) _ = second.Body.Close() - if second.StatusCode != http.StatusTooManyRequests || second.Header.Get("Retry-After") == "" { - t.Errorf("%s over the limit answered %d with Retry-After %q, want 429 with a value", + if second.StatusCode != http.StatusTooManyRequests || second.Header.Get("Retry-After") != "1" { + t.Errorf("%s over the limit answered %d with Retry-After %q, want 429 with 1", endpoint, second.StatusCode, second.Header.Get("Retry-After")) } } From 429bf5ae7696a5d39b22d37090c97900471a3078 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:10:16 +0700 Subject: [PATCH 281/518] fix(observability-proxy): answer early refusals without waiting on the body A request refused before its body was read (401, 403, 405, or a rate or concurrency 429) had no read deadline yet, and net/http drains up to 256 KiB of an unread body before it sends the headers. A client that declared a body and never sent it got no answer, and held the handler, for as long as it kept the connection; for a second form POST arriving while the concurrency slot was taken, that was more than five request timeouts. The read deadline is now set when the request arrives and lifted once the body is read, so tails are not cut. A refusal sent with the body still unread closes the connection instead of draining it, so it is answered at once. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 17 +++++++ .../server_deadline_test.go | 51 +++++++++++++++++++ 2 files changed, 68 insertions(+) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index f511a6691..511499933 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -110,6 +110,8 @@ func NewServer(cfg Config, auth *Authorizer, stores StoreLookup, reg prometheus. type statusRecorder struct { http.ResponseWriter code int + // bodyRead is set once readForm has read the whole body. + bodyRead bool } func (r *statusRecorder) WriteHeader(code int) { @@ -130,11 +132,23 @@ func (r *statusRecorder) Unwrap() http.ResponseWriter { return r.ResponseWriter func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { rec := &statusRecorder{ResponseWriter: w} + // The listener has no ReadTimeout, which would cut tails, so the body + // is bounded from here; readForm lifts the deadline once it has read + // the body. A refusal sent before that leaves the deadline in place. + if s.cfg.RequestTimeout > 0 { + _ = http.NewResponseController(w).SetReadDeadline(time.Now().Add(s.cfg.RequestTimeout)) + } route, herr := ParseRoute(r.URL.EscapedPath()) if herr == nil { herr = s.serve(rec, r, route) } if herr != nil && herr != errAborted { + // net/http drains up to 256 KiB of an unread body before it sends + // the headers; closing the connection instead answers a refusal at + // once rather than after a body that may never come. + if r.ContentLength != 0 && !rec.bodyRead { + rec.Header().Set("Connection", "close") + } writeError(rec, herr) } s.requests.WithLabelValues(route.Kind.String(), strconv.Itoa(rec.code)).Inc() @@ -301,6 +315,9 @@ func (s *Server) readForm(w http.ResponseWriter, r *http.Request) *HTTPError { return errorf(http.StatusRequestEntityTooLarge, "request body larger than %d bytes", tooLarge.Limit) case err == nil: _ = rc.SetReadDeadline(time.Time{}) + if rec, ok := w.(*statusRecorder); ok { + rec.bodyRead = true + } return nil case errors.Is(err, os.ErrDeadlineExceeded): return errorf(http.StatusRequestTimeout, "request body not received within %s", s.cfg.RequestTimeout) diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index ac62dd782..4b5c98887 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -18,6 +18,7 @@ package observabilityproxy import ( "bufio" + "fmt" "io" "net" "net/http" @@ -297,3 +298,53 @@ func TestServerOversizedBody(t *testing.T) { } } } + +// TestServerEarlyRefusalWithUnreadBody refuses requests before the body is +// read: a bad token, a full concurrency slot, and a full tail slot. +// net/http drains up to 256 KiB of an unread body before it sends the +// headers, so the answer must not wait on a body that never comes, even +// under a long request timeout. +func TestServerEarlyRefusalWithUnreadBody(t *testing.T) { + for _, timeout := range []time.Duration{200 * time.Millisecond, time.Minute} { + for _, tc := range []struct { + name, raw string + hold string + want int + }{ + {"bad token", "POST " + logsRoot + "query HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer bad\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: 1000\r\n\r\nquery=", "", http.StatusUnauthorized}, + {"concurrency slot taken", "POST " + logsRoot + "query HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\nContent-Type: application/x-www-form-urlencoded\r\nContent-Length: 1000\r\n\r\nquery=", "query", http.StatusTooManyRequests}, + {"tail slot taken", "GET " + logsRoot + "tail?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\nContent-Length: 1000\r\n\r\n", "tail", http.StatusTooManyRequests}, + } { + // A held query ends at the request timeout by design, so the + // slot is only reliably taken under the long one. + if tc.hold == "query" && timeout < time.Second { + continue + } + t.Run(fmt.Sprintf("%s/%s", timeout, tc.name), func(t *testing.T) { + lines := make(chan string, 1) + done := make(chan struct{}) + h := newHarness(t, Config{ + RequestTimeout: timeout, + TailIdleTimeout: time.Minute, + Limiter: LimiterConfig{MaxConcurrentPerUser: 1, MaxTails: 1}, + }, tailUpstream(lines, done)) + t.Cleanup(func() { close(done) }) + pipe := servePipe(t, h.server) + if tc.hold != "" { + pipe.send(t, "GET "+logsRoot+tc.hold+"?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n\r\n") + eventually(t, time.Second, func() bool { + h.server.limits.mu.Lock() + defer h.server.limits.mu.Unlock() + return h.server.limits.active["alice"]+h.server.limits.total == 1 + }, "the first request never took its slot") + } + start := time.Now() + conn := pipe.send(t, tc.raw) + resp := readResponse(t, conn) + if resp.StatusCode != tc.want || time.Since(start) > time.Second { + t.Fatalf("answered %d after %s, want %d within a second", resp.StatusCode, time.Since(start), tc.want) + } + }) + } + } +} From 15eec1e5b198a92ef614bf0d14a555d01730fc6d Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:11:50 +0700 Subject: [PATCH 282/518] fix(observability-proxy): refuse logs times vlselect cannot hold The proxy sends vlselect RFC3339 times, which it never reads as a guessed-unit number, but it accepted any RFC3339 year and unix seconds up to 2286. vlselect keeps times as int64 nanoseconds through time.UnixNano, which is undefined outside 1678..2262, so such a window landed anywhere. An absolute time outside 1970..2262 now answers 400; nothing was logged before 1970. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 17 +++++++++++++++-- internal/observabilityproxy/logsparams_test.go | 6 ++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index 93e1bd9a4..c3e8170c5 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -325,6 +325,19 @@ func scaleDuration(f float64, unit time.Duration) (time.Duration, bool) { // and a duration meaning that long before now (VictoriaLogs' "start=3h"). // Anything else is refused rather than guessed, since a misread timestamp // silently moves the window. +// maxLogsTime is the last instant int64 nanoseconds hold. +var maxLogsTime = time.Unix(0, math.MaxInt64) + +// absoluteLogsTime refuses a time vlselect cannot hold: it keeps times as +// int64 nanoseconds (time.UnixNano), which is undefined outside 1678..2262 +// and would move the window anywhere; nothing was logged before 1970. +func absoluteLogsTime(s string, t time.Time) (time.Time, error) { + if t.Before(time.Unix(0, 0)) || !t.Before(maxLogsTime) { + return time.Time{}, fmt.Errorf("time %q is outside 1970..2262", s) + } + return t, nil +} + func parseTime(s string, now time.Time) (time.Time, error) { if s == "now" { return now, nil @@ -344,10 +357,10 @@ func parseTime(s string, now time.Time) (time.Time, error) { frac = (frac + "000000000")[:9] nanos, _ = strconv.ParseInt(frac, 10, 64) } - return time.Unix(secs, nanos), nil + return absoluteLogsTime(s, time.Unix(secs, nanos)) } if t, err := time.Parse(time.RFC3339Nano, s); err == nil { - return t, nil + return absoluteLogsTime(s, t) } if d, err := parseDuration(s); err == nil { return now.Add(-d), nil diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index 009b1721d..c5cc9d4c5 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -135,6 +135,12 @@ func TestBuildLogsParamsRejects(t *testing.T) { {"exponent duration", "query", url.Values{"start": {"3e2"}}, http.StatusBadRequest}, {"trailing dot", "query", url.Values{"start": {"300."}}, http.StatusBadRequest}, {"infinite time", "query", url.Values{"end": {"Inf"}}, http.StatusBadRequest}, + // vlselect keeps times as int64 nanoseconds (time.UnixNano), which + // overflows outside 1678..2262; before 1970 is no log time either. + {"time before 1970", "query", url.Values{"start": {"0001-01-01T00:00:00Z"}, "end": {"0001-01-02T00:00:00Z"}}, http.StatusBadRequest}, + {"time past int64 nanoseconds", "query", url.Values{"start": {"9999-01-01T00:00:00Z"}, "end": {"9999-01-02T00:00:00Z"}}, http.StatusBadRequest}, + {"tail since before 1970", "tail", url.Values{"since": {"1969-12-31T23:59:59Z"}}, http.StatusBadRequest}, + {"unix seconds past int64 nanoseconds", "query", url.Values{"start": {"9999999990"}, "end": {"9999999999"}}, http.StatusBadRequest}, {"NaN time", "query", url.Values{"start": {"NaN"}}, http.StatusBadRequest}, {"NaN step", "hits", url.Values{"step": {"NaN"}}, http.StatusBadRequest}, {"NaN start_offset", "tail", url.Values{"start_offset": {"NaN"}}, http.StatusBadRequest}, From 109980ffa72e6e4b3c48a56d618f377b7538f913 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:12:29 +0700 Subject: [PATCH 283/518] fix(observability-proxy): refuse every empty limit the same way An empty fields_limit answered 400, but an empty limit or facets cap fell back to the default, so one malformed parameter got two answers depending on its name. Every cap now refuses an empty value with 400: a cap the client sent but left blank is a client bug, and reading it as the default would hide it. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 12 +++++++----- internal/observabilityproxy/logsparams_test.go | 2 ++ 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index c3e8170c5..2952209c4 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -104,11 +104,7 @@ func BuildLogsParams(endpoint string, in url.Values, namespace string, now time. } } // vlselect returns every group without fields_limit or with 0, so - // the cap is sent either way; an empty value is malformed rather - // than a request for the default. - if in.Has("fields_limit") && in.Get("fields_limit") == "" { - return nil, errorf(http.StatusBadRequest, "invalid fields_limit \"\"") - } + // the cap is sent either way. herr = setLimit(out, in, "fields_limit", lim.MaxLimit, lim.MaxLimit) case "stats_query_range": herr = setStep(out, in, start, end, lim) @@ -200,8 +196,14 @@ func setStep(out, in url.Values, start, end time.Time, lim LogsLimits) *HTTPErro return nil } +// setLimit refuses an empty value rather than read it as the default: a +// cap the client sent but left blank is malformed, the same for every cap, +// where reading it as the default would hide a client bug. func setLimit(out, in url.Values, key string, def, maxv int) *HTTPError { n := def + if in.Has(key) && in.Get(key) == "" { + return errorf(http.StatusBadRequest, "invalid %s \"\"", key) + } if v := in.Get(key); v != "" { parsed, err := strconv.Atoi(v) if err != nil || parsed <= 0 { diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index c5cc9d4c5..3884428f0 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -118,6 +118,8 @@ func TestBuildLogsParamsRejects(t *testing.T) { {"unknown endpoint", "tenant_ids", url.Values{}, http.StatusNotFound}, // Malformed rather than a request for the default. {"empty fields_limit", "hits", url.Values{"field": {"level"}, "fields_limit": {""}}, http.StatusBadRequest}, + {"empty limit", "query", url.Values{"limit": {""}}, http.StatusBadRequest}, + {"empty facets cap", "facets", url.Values{"max_values_per_field": {""}}, http.StatusBadRequest}, {"stats_query is not proxied", "stats_query", url.Values{}, http.StatusNotFound}, {"ignore_global_time_filter", "query", url.Values{"query": {`* options(ignore_global_time_filter=true)`}}, http.StatusBadRequest}, {"ignore_global_time_filter in subquery", "hits", url.Values{"query": {`* | union (options(ignore_global_time_filter=true) error)`}}, http.StatusBadRequest}, From a2eb175139a8d42d0cc72b9bcee0bdc61c9caa9a Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:12:50 +0700 Subject: [PATCH 284/518] test(dashboard): start the pass-through nginx on fresh ports The second nginx of the not-installed test reused the stand-in port of the first one right after killing it, behind a fixed sleep, so a slow exit could leave the port taken. It now waits for the first nginx to exit and takes a new port. The file header also names the fourth property it now checks. Assisted-by: LLM Signed-off-by: ZverGuy --- hack/console-nginx-observability-proxy.bats | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/hack/console-nginx-observability-proxy.bats b/hack/console-nginx-observability-proxy.bats index ef9c297b3..de2348dd7 100644 --- a/hack/console-nginx-observability-proxy.bats +++ b/hack/console-nginx-observability-proxy.bats @@ -4,7 +4,7 @@ # nginx config baked into the console image, which no chart template and no # helm-unittest case can see, so it is pinned here. # -# Three properties are checked. The location regex must take the four path +# Four properties are checked. The location regex must take the four path # shapes the proxy serves and leave every ordinary Kubernetes API path to the # /apis and /api prefix locations; nginx runs the regex locations whenever the # longest prefix match is a plain prefix, so the prefix locations must not @@ -13,7 +13,10 @@ # read timeout. And the upstream must be a variable resolved per request, # because nginx resolves a literal upstream host once at startup and refuses # to start when it does not resolve, which is the case on every cluster that -# does not run the proxy. +# does not run the proxy. Finally, when nginx cannot reach the proxy it +# answers with the Kubernetes Status the console reads as not installed, +# while the proxy's own 502 passes through; that check runs the extracted +# locations in a real nginx and is skipped where none is installed. # # The regex is evaluated with perl, whose syntax nginx's PCRE accepts for the # constructs used here. @@ -222,11 +225,15 @@ assert s["kind"] == "Status" and s["apiVersion"] == "v1" and s["status"] == "Fai assert s["reason"] == "NotFound" and s["code"] == 404, s assert s["message"] == "the observability proxy is not available on this cluster", s ' || { echo "FAIL: the not-installed body is not the expected Status: $body" >&2; false; } - kill "$(cat "$BATS_TEST_TMPDIR/nginx.pid")" + # The next nginx gets fresh ports, so the old master need not have + # released its sockets; it is still stopped first to keep one running. + pid=$(cat "$BATS_TEST_TMPDIR/nginx.pid") + kill "$pid" + for _ in $(seq 1 50); do kill -0 "$pid" 2>/dev/null || break; sleep 0.1; done rm -f "$BATS_TEST_TMPDIR/nginx.pid" - sleep 0.2 # The proxy answers 502 itself: the console must see that answer. + standin=$(free_port) port=$(start_console_nginx "$standin" "$standin") out=$(curl --silent --show-error --include --max-time 10 "http://127.0.0.1:$port$path") printf '%s\n' "$out" | head -n 1 | grep -q ' 502' \ From 3722808050c7bd70333cc7fdb10da9f4c1753056 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:16:34 +0700 Subject: [PATCH 285/518] test(observability-proxy): watch the slot instead of probing for it The slow-body and stalled-client tests checked the slot by sending requests of their own, which competed for the one slot under test: a probe that held the slot when the request under test arrived got that request refused, and the test failed. The tests now read the limiter's slot count directly. Assisted-by: LLM Signed-off-by: ZverGuy --- .../server_deadline_test.go | 44 +++++++++---------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index 4b5c98887..463576e09 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -82,6 +82,19 @@ func (l *pipeListener) send(t *testing.T, raw string) net.Conn { return client } +// slotsHeld counts the concurrency and tail slots in use. Tests watch it +// instead of probing with requests of their own, which would compete for +// the slot under test. +func slotsHeld(h *harness) int { + h.server.limits.mu.Lock() + defer h.server.limits.mu.Unlock() + n := h.server.limits.total + for _, c := range h.server.limits.active { + n += c + } + return n +} + func eventually(t *testing.T, within time.Duration, cond func() bool, msg string) { t.Helper() deadline := time.Now().Add(within) @@ -128,15 +141,13 @@ func TestServerSlowBodyHoldsSlotUntilDeadline(t *testing.T) { conn := pipe.send(t, "POST "+logsRoot+"query HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n"+ "Content-Type: application/x-www-form-urlencoded\r\nContent-Length: 100\r\n\r\nquery=") - eventually(t, time.Second, func() bool { - return h.status("GET", logsRoot+"query", "good") == http.StatusTooManyRequests - }, "a request with an unfinished body does not hold a concurrency slot") + eventually(t, time.Second, func() bool { return slotsHeld(h) == 1 }, + "a request with an unfinished body does not hold a concurrency slot") if resp := readResponse(t, conn); resp.StatusCode != http.StatusRequestTimeout { t.Fatalf("unfinished body answered %d, want 408", resp.StatusCode) } - eventually(t, 2*time.Second, func() bool { - return h.status("GET", logsRoot+"query", "good") == http.StatusOK - }, "the slot of an unfinished body was not released") + eventually(t, 2*time.Second, func() bool { return slotsHeld(h) == 0 }, + "the slot of an unfinished body was not released") } // TestServerStalledClientReleasesTail holds a tail whose client stops @@ -174,12 +185,9 @@ func TestServerStalledClientReleasesQuery(t *testing.T) { }, nil) pipe := servePipe(t, h.server) pipe.send(t, "GET "+logsRoot+"query?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n\r\n") - eventually(t, time.Second, func() bool { - return h.status("GET", logsRoot+"query", "good") == http.StatusTooManyRequests - }, "the stalled request never held its slot") - eventually(t, 2*time.Second, func() bool { - return h.status("GET", logsRoot+"query", "good") == http.StatusOK - }, "a client that stopped reading keeps its concurrency slot") + eventually(t, time.Second, func() bool { return slotsHeld(h) == 1 }, "the stalled request never held its slot") + eventually(t, 2*time.Second, func() bool { return slotsHeld(h) == 0 }, + "a client that stopped reading keeps its concurrency slot") } // TestServerActiveTailOutlivesWriteDeadline keeps a tail busy for several @@ -268,11 +276,7 @@ func TestServerUndeliveredBodyReleasesSlot(t *testing.T) { t.Cleanup(func() { close(done) }) pipe := servePipe(t, h.server) conn := pipe.send(t, raw) - held := func() bool { - h.server.limits.mu.Lock() - defer h.server.limits.mu.Unlock() - return h.server.limits.active["alice"]+h.server.limits.total > 0 - } + held := func() bool { return slotsHeld(h) > 0 } eventually(t, time.Second, held, "the request never took a slot") if resp := readResponse(t, conn); resp.StatusCode != http.StatusRequestTimeout { t.Fatalf("undelivered body answered %d, want 408", resp.StatusCode) @@ -332,11 +336,7 @@ func TestServerEarlyRefusalWithUnreadBody(t *testing.T) { pipe := servePipe(t, h.server) if tc.hold != "" { pipe.send(t, "GET "+logsRoot+tc.hold+"?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n\r\n") - eventually(t, time.Second, func() bool { - h.server.limits.mu.Lock() - defer h.server.limits.mu.Unlock() - return h.server.limits.active["alice"]+h.server.limits.total == 1 - }, "the first request never took its slot") + eventually(t, time.Second, func() bool { return slotsHeld(h) == 1 }, "the first request never took its slot") } start := time.Now() conn := pipe.send(t, tc.raw) From f8ff307f710f894b2ba81b972788f8d719ce703e Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:31:58 +0700 Subject: [PATCH 286/518] fix(observability-proxy): bound only the logs times sent to vlselect The 1970..2262 check refused times on parsing, including a start that the window clamps to the maximum range before anything is sent and a since that only becomes a capped start_offset, so an old start or a tail resume from before 1970 answered 400 where the documented answer is to clamp. It had also detached the parseTime doc comment. The check now applies to the times actually sent, after clamping, and uses the range vlselect can hold, int64 nanoseconds (1677..2262); 1970 is not a limit of vlselect. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 36 +++++++++++-------- .../observabilityproxy/logsparams_test.go | 24 ++++++++++--- 2 files changed, 41 insertions(+), 19 deletions(-) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index 2952209c4..220a06762 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -160,6 +160,11 @@ func window(in url.Values, now time.Time, lim LogsLimits) (time.Time, time.Time, if end.Sub(start) > lim.MaxRange { start = end.Add(-lim.MaxRange) } + for k, t := range map[string]time.Time{"start": start, "end": end} { + if herr := sendableTime(k, t); herr != nil { + return time.Time{}, time.Time{}, herr + } + } return start.UTC(), end.UTC(), nil } @@ -228,6 +233,9 @@ func tailParams(in, out url.Values, now time.Time, lim LogsLimits) *HTTPError { if err != nil { return errorf(http.StatusBadRequest, "invalid %s: %v", k, err) } + if herr := sendableTime(k, t); herr != nil { + return herr + } out.Set(k, t.UTC().Format(time.RFC3339Nano)) if k == "start" { start = t @@ -323,23 +331,21 @@ func scaleDuration(f float64, unit time.Duration) (time.Duration, bool) { return time.Duration(v), true } +// sendableTime refuses a time vlselect cannot hold: it reads the RFC3339 +// the proxy sends through time.UnixNano, which is undefined outside the +// int64 range (1677..2262) and would move the window anywhere. Only times +// actually sent are checked; an older start is clamped first. +func sendableTime(key string, t time.Time) *HTTPError { + if t.Before(time.Unix(0, math.MinInt64)) || t.After(time.Unix(0, math.MaxInt64)) { + return errorf(http.StatusBadRequest, "invalid %s: %s is outside 1677..2262", key, t.Format(time.RFC3339)) + } + return nil +} + // parseTime accepts "now", unix seconds with an optional fraction, RFC3339, // and a duration meaning that long before now (VictoriaLogs' "start=3h"). // Anything else is refused rather than guessed, since a misread timestamp // silently moves the window. -// maxLogsTime is the last instant int64 nanoseconds hold. -var maxLogsTime = time.Unix(0, math.MaxInt64) - -// absoluteLogsTime refuses a time vlselect cannot hold: it keeps times as -// int64 nanoseconds (time.UnixNano), which is undefined outside 1678..2262 -// and would move the window anywhere; nothing was logged before 1970. -func absoluteLogsTime(s string, t time.Time) (time.Time, error) { - if t.Before(time.Unix(0, 0)) || !t.Before(maxLogsTime) { - return time.Time{}, fmt.Errorf("time %q is outside 1970..2262", s) - } - return t, nil -} - func parseTime(s string, now time.Time) (time.Time, error) { if s == "now" { return now, nil @@ -359,10 +365,10 @@ func parseTime(s string, now time.Time) (time.Time, error) { frac = (frac + "000000000")[:9] nanos, _ = strconv.ParseInt(frac, 10, 64) } - return absoluteLogsTime(s, time.Unix(secs, nanos)) + return time.Unix(secs, nanos), nil } if t, err := time.Parse(time.RFC3339Nano, s); err == nil { - return absoluteLogsTime(s, t) + return t, nil } if d, err := parseDuration(s); err == nil { return now.Add(-d), nil diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index 3884428f0..f7d21753a 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -138,11 +138,11 @@ func TestBuildLogsParamsRejects(t *testing.T) { {"trailing dot", "query", url.Values{"start": {"300."}}, http.StatusBadRequest}, {"infinite time", "query", url.Values{"end": {"Inf"}}, http.StatusBadRequest}, // vlselect keeps times as int64 nanoseconds (time.UnixNano), which - // overflows outside 1678..2262; before 1970 is no log time either. - {"time before 1970", "query", url.Values{"start": {"0001-01-01T00:00:00Z"}, "end": {"0001-01-02T00:00:00Z"}}, http.StatusBadRequest}, - {"time past int64 nanoseconds", "query", url.Values{"start": {"9999-01-01T00:00:00Z"}, "end": {"9999-01-02T00:00:00Z"}}, http.StatusBadRequest}, - {"tail since before 1970", "tail", url.Values{"since": {"1969-12-31T23:59:59Z"}}, http.StatusBadRequest}, + // is undefined outside 1677..2262; a time sent there must fit. + {"end before int64 nanoseconds", "query", url.Values{"start": {"0001-01-01T00:00:00Z"}, "end": {"0001-01-02T00:00:00Z"}}, http.StatusBadRequest}, + {"end past int64 nanoseconds", "query", url.Values{"end": {"9999-01-02T00:00:00Z"}}, http.StatusBadRequest}, {"unix seconds past int64 nanoseconds", "query", url.Values{"start": {"9999999990"}, "end": {"9999999999"}}, http.StatusBadRequest}, + {"tail end past int64 nanoseconds", "tail", url.Values{"end": {"9999-01-02T00:00:00Z"}}, http.StatusBadRequest}, {"NaN time", "query", url.Values{"start": {"NaN"}}, http.StatusBadRequest}, {"NaN step", "hits", url.Values{"step": {"NaN"}}, http.StatusBadRequest}, {"NaN start_offset", "tail", url.Values{"start_offset": {"NaN"}}, http.StatusBadRequest}, @@ -392,3 +392,19 @@ func TestParseDurationBounds(t *testing.T) { } } } + +// TestBuildLogsParamsOldTimesClamped: only times that reach vlselect must +// fit its int64 nanoseconds. An old start is clamped to MaxRange before it +// is sent, and an old since becomes a start_offset capped at the resume +// limit, so both are served. +func TestBuildLogsParamsOldTimesClamped(t *testing.T) { + lim := testLimits() + out := build(t, "query", url.Values{"start": {"1900-01-01T00:00:00Z"}}) + if want := testNow.Add(-lim.MaxRange).Format(time.RFC3339Nano); out.Get("start") != want { + t.Errorf("start = %s, want clamped to %s", out.Get("start"), want) + } + out = build(t, "tail", url.Values{"since": {"1969-12-31T23:59:59Z"}}) + if got := out.Get("start_offset"); got != formatDuration(lim.MaxTailResume) { + t.Errorf("start_offset = %s, want the %s cap", got, lim.MaxTailResume) + } +} From f48a94de6d53d23801f4a27b016090e03ff24632 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:38:25 +0700 Subject: [PATCH 287/518] fix(observability-proxy): name the logs time out of range exactly When both bounds were out of range the answer named one of them at random, as the check walked a map, and it printed the instant without its fraction, so a time one nanosecond past the last one vlselect can hold was shown as a time inside the range. The end is now checked first and the instant is printed in full; tests pin the inclusive bound. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/logsparams.go | 11 +++++---- .../observabilityproxy/logsparams_test.go | 24 +++++++++++++++++++ 2 files changed, 30 insertions(+), 5 deletions(-) diff --git a/internal/observabilityproxy/logsparams.go b/internal/observabilityproxy/logsparams.go index 220a06762..ec6043c6f 100644 --- a/internal/observabilityproxy/logsparams.go +++ b/internal/observabilityproxy/logsparams.go @@ -160,10 +160,11 @@ func window(in url.Values, now time.Time, lim LogsLimits) (time.Time, time.Time, if end.Sub(start) > lim.MaxRange { start = end.Add(-lim.MaxRange) } - for k, t := range map[string]time.Time{"start": start, "end": end} { - if herr := sendableTime(k, t); herr != nil { - return time.Time{}, time.Time{}, herr - } + if herr := sendableTime("end", end); herr != nil { + return time.Time{}, time.Time{}, herr + } + if herr := sendableTime("start", start); herr != nil { + return time.Time{}, time.Time{}, herr } return start.UTC(), end.UTC(), nil } @@ -337,7 +338,7 @@ func scaleDuration(f float64, unit time.Duration) (time.Duration, bool) { // actually sent are checked; an older start is clamped first. func sendableTime(key string, t time.Time) *HTTPError { if t.Before(time.Unix(0, math.MinInt64)) || t.After(time.Unix(0, math.MaxInt64)) { - return errorf(http.StatusBadRequest, "invalid %s: %s is outside 1677..2262", key, t.Format(time.RFC3339)) + return errorf(http.StatusBadRequest, "invalid %s: %s is outside 1677..2262", key, t.UTC().Format(time.RFC3339Nano)) } return nil } diff --git a/internal/observabilityproxy/logsparams_test.go b/internal/observabilityproxy/logsparams_test.go index f7d21753a..82ed29190 100644 --- a/internal/observabilityproxy/logsparams_test.go +++ b/internal/observabilityproxy/logsparams_test.go @@ -17,6 +17,7 @@ limitations under the License. package observabilityproxy import ( + "math" "net/http" "net/url" "sort" @@ -408,3 +409,26 @@ func TestBuildLogsParamsOldTimesClamped(t *testing.T) { t.Errorf("start_offset = %s, want the %s cap", got, lim.MaxTailResume) } } + +// TestBuildLogsParamsSendableBoundary pins the inclusive int64-nanosecond +// bound and that the refusal names the instant exactly. +func TestBuildLogsParamsSendableBoundary(t *testing.T) { + last := time.Unix(0, math.MaxInt64).UTC() + in := url.Values{"start": {last.Add(-time.Hour).Format(time.RFC3339Nano)}, "end": {last.Format(time.RFC3339Nano)}} + if _, herr := BuildLogsParams("query", in, "tenant-a", testNow, testLimits()); herr != nil { + t.Fatalf("the last int64 nanosecond refused: %v", herr) + } + past := last.Add(time.Nanosecond).Format(time.RFC3339Nano) + in = url.Values{"start": {last.Add(-time.Hour).Format(time.RFC3339Nano)}, "end": {past}} + _, herr := BuildLogsParams("query", in, "tenant-a", testNow, testLimits()) + if herr == nil || !strings.Contains(herr.Message, "invalid end") || !strings.Contains(herr.Message, past) { + t.Fatalf("one nanosecond past answered %v, want end refused naming %s", herr, past) + } + // Both out of range: the answer names end, deterministically. + for range 20 { + _, herr := BuildLogsParams("query", url.Values{"start": {"0001-01-01T00:00:00Z"}, "end": {"0001-01-02T00:00:00Z"}}, "tenant-a", testNow, testLimits()) + if herr == nil || !strings.Contains(herr.Message, "invalid end") { + t.Fatalf("answer = %v, want end named", herr) + } + } +} From dff23f76e0d421ac0e79f25c056ba787ec081e1e Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 00:45:01 +0700 Subject: [PATCH 288/518] test(observability-proxy): spend the rate burst in a loop Calling allowed twice in one condition reads as a duplicated expression to staticcheck, which cannot see that each call takes a token. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/limits_test.go | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/limits_test.go b/internal/observabilityproxy/limits_test.go index 706ed71c7..d7f4c76f4 100644 --- a/internal/observabilityproxy/limits_test.go +++ b/internal/observabilityproxy/limits_test.go @@ -77,8 +77,10 @@ func TestLimiterRate(t *testing.T) { now := time.Unix(1000, 0) l := newLimiter(LimiterConfig{RatePerUser: 2, RateBurst: 2}) l.now = func() time.Time { return now } - if !allowed(l, "alice") || !allowed(l, "alice") { - t.Fatal("burst refused") + for i := range 2 { + if !allowed(l, "alice") { + t.Fatalf("burst request %d refused", i+1) + } } if wait, ok := l.allowRate("alice"); ok || wait != 500*time.Millisecond { t.Fatalf("request over the burst: admitted %v, wait %s, want refused with 500ms", ok, wait) From f350b7ed9c3adc50c487e9ff53b14b4ab1244327 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 00:45:01 +0700 Subject: [PATCH 289/518] test(observability-proxy): observe held slots where they cannot be missed Polling for a slot that lives only until a 200ms deadline misses it when the test goroutine is descheduled for longer, as under -race on a busy runner. The tail and stalled-query tests now read the slot when the request reaches the store, which the proxy does while holding it; the slow-body test, which never reaches the store, gets a wider deadline. Assisted-by: LLM Signed-off-by: ZverGuy --- .../server_deadline_test.go | 48 +++++++++++++++++-- 1 file changed, 43 insertions(+), 5 deletions(-) diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index 463576e09..32ab93773 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -24,6 +24,7 @@ import ( "net/http" "strings" "sync" + "sync/atomic" "testing" "time" ) @@ -129,12 +130,38 @@ func readResponse(t *testing.T, conn net.Conn) *http.Response { return nil } +// atUpstream calls probe each time a request reaches the upstream and +// sends the result on the returned channel. The proxy holds the request's +// slot from before it calls the upstream until the upstream answers, so +// the probe sees it held however late the test goroutine gets scheduled. +func atUpstream(probe func(*harness) int, next http.HandlerFunc) (http.HandlerFunc, *atomic.Pointer[harness], <-chan int) { + var h atomic.Pointer[harness] + seen := make(chan int, 1) + return func(w http.ResponseWriter, r *http.Request) { + seen <- probe(h.Load()) + next(w, r) + }, &h, seen +} + +func receive(t *testing.T, ch <-chan int, msg string) int { + t.Helper() + select { + case v := <-ch: + return v + case <-time.After(3 * time.Second): + t.Fatal(msg) + } + return 0 +} + // TestServerSlowBodyHoldsSlotUntilDeadline sends a POST whose body never // completes: it must count against the user's concurrency, and the read // must give up at the request timeout instead of holding the handler. func TestServerSlowBodyHoldsSlotUntilDeadline(t *testing.T) { + // The slot is only observable while the body is pending, so the + // timeout is long enough for the poll below to see it on a busy runner. h := newHarness(t, Config{ - RequestTimeout: 200 * time.Millisecond, + RequestTimeout: time.Second, Limiter: LimiterConfig{MaxConcurrentPerUser: 1}, }, nil) pipe := servePipe(t, h.server) @@ -157,10 +184,12 @@ func TestServerStalledClientReleasesTail(t *testing.T) { for _, readHeaders := range []bool{false, true} { lines := make(chan string, 1) done := make(chan struct{}) + upstream, hp, held := atUpstream(func(h *harness) int { return h.server.limits.openTails() }, tailUpstream(lines, done)) h := newHarness(t, Config{ TailIdleTimeout: 200 * time.Millisecond, Limiter: LimiterConfig{MaxTails: 1}, - }, tailUpstream(lines, done)) + }, upstream) + hp.Store(h) t.Cleanup(func() { close(done) }) pipe := servePipe(t, h.server) conn := pipe.send(t, "GET "+logsRoot+"tail?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n\r\n") @@ -170,7 +199,9 @@ func TestServerStalledClientReleasesTail(t *testing.T) { } lines <- `{"_msg":"unread"}` } - eventually(t, time.Second, func() bool { return h.server.limits.openTails() == 1 }, "the tail never opened") + if n := receive(t, held, "the tail never reached the store"); n != 1 { + t.Fatalf("open tails while the store answers = %d, want 1", n) + } eventually(t, 2*time.Second, func() bool { return h.server.limits.openTails() == 0 }, "a client that stopped reading keeps its tail slot") } @@ -179,13 +210,20 @@ func TestServerStalledClientReleasesTail(t *testing.T) { // TestServerStalledClientReleasesQuery does the same for a non-tail // response, which must be written by the request deadline. func TestServerStalledClientReleasesQuery(t *testing.T) { + upstream, hp, held := atUpstream(slotsHeld, func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/stream+json") + _, _ = io.WriteString(w, `{"_msg":"hello"}`+"\n") + }) h := newHarness(t, Config{ RequestTimeout: 200 * time.Millisecond, Limiter: LimiterConfig{MaxConcurrentPerUser: 1}, - }, nil) + }, upstream) + hp.Store(h) pipe := servePipe(t, h.server) pipe.send(t, "GET "+logsRoot+"query?query=* HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n\r\n") - eventually(t, time.Second, func() bool { return slotsHeld(h) == 1 }, "the stalled request never held its slot") + if n := receive(t, held, "the stalled request never reached the store"); n != 1 { + t.Fatalf("slots held while the store answers = %d, want 1", n) + } eventually(t, 2*time.Second, func() bool { return slotsHeld(h) == 0 }, "a client that stopped reading keeps its concurrency slot") } From 7eff63b028cf769fcd1237e9f7fdd3dc568588a9 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 00:51:01 +0700 Subject: [PATCH 290/518] refactor(dashboard): drop the scope-keyed placeholder from the logs view The logs view remounts under a key per scope, so its queries start with no previous result to carry over and the scope check never had anything to refuse. The key is the one guard, and the tenant-switch tests fail without it. Assisted-by: LLM Signed-off-by: ZverGuy --- .../console/src/components/observability/LogsView.tsx | 5 +---- .../console/apps/console/src/lib/observability/proxy.ts | 9 --------- 2 files changed, 1 insertion(+), 13 deletions(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index 8c05a8743..05535e12e 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -14,7 +14,6 @@ import { import { fieldValues, isScopeWide, - keepWithinScope, logsRoot, queryLogs, type ObservabilityScope, @@ -81,7 +80,7 @@ function placeholderFor({ follow, stopped, loading, failed, longestRange, retry return } -// Filter, pod and container belong to one scope; a new tenant or application starts clean. +// Filter, pod, container and the lines already read belong to one scope; a new tenant or application starts clean. export function LogsView({ scope }: { scope: ObservabilityScope }) { const root = logsRoot(scope) return @@ -107,7 +106,6 @@ function ScopedLogsView({ root }: { root: string }) { query: buildLogsQuery({ filter: "", pod: narrowTo }), ...windowFor(range), }, signal), - placeholderData: keepWithinScope(root), retry: false, }) const pods = useQuery(pickerQuery(POD_FIELD)) @@ -117,7 +115,6 @@ function ScopedLogsView({ root }: { root: string }) { queryKey: ["observability", "logs", root, query, range], queryFn: ({ signal }) => queryLogs(client, root, { query, limit: HISTORY_LIMIT, ...windowFor(range) }, signal), enabled: !follow, - placeholderData: keepWithinScope(root), retry: false, }) const tail = useLogTail(root, query, followRun, { cap: FOLLOW_CAP }) diff --git a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts index d33052f70..c28bd12d4 100644 --- a/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts +++ b/packages/system/dashboard/images/console/apps/console/src/lib/observability/proxy.ts @@ -20,15 +20,6 @@ export function logsRoot(scope: ObservabilityScope): string { return app ? `${app}/log/` : `/apis/cozystack.io/v1alpha1/namespaces/${scope.namespace}/logs/` } -// The provider's default keeps a useQuery's previous result while its key -// changes; across scopes that would show one tenant's or application's data -// under another. (useQueries starts a new observer per key, so it never does.) -// Every observability query key carries its root at index 2. -export function keepWithinScope(root: string) { - return (previous: T | undefined, previousQuery?: { queryKey: readonly unknown[] }): T | undefined => - previousQuery?.queryKey[2] === root ? previous : undefined -} - export type ProxyErrorKind = | "bad-request" | "unauthorized" From 48f459e103f4b3a3bc6a79e3932160cee659c788 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 00:54:51 +0700 Subject: [PATCH 291/518] fix(dashboard): say the logs are loading while an empty result stands in The previous result stays on screen while a new range or filter loads. When that result was empty, the query reports success, so the view said no lines matched the range it had not read yet. Assisted-by: LLM Signed-off-by: ZverGuy --- .../components/observability/LogsView.test.tsx | 17 +++++++++++++++++ .../src/components/observability/LogsView.tsx | 2 +- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index a0df67c68..47cbf1c22 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -93,6 +93,23 @@ describe("LogsView with the production query defaults", () => { expect(screen.getByText("ready to accept connections")).toBeInTheDocument() }) + + it("says it is loading, not that nothing matched, while a longer range replaces an empty one", async () => { + const { client } = clientWith((endpoint, params) => { + if (!endpoint.endsWith("query")) return defaultHandler(endpoint, params) + return Date.parse(params.get("start") ?? "") < Date.now() - 2 * 3600_000 + ? new Promise(() => {}) + : new Response("") + }) + renderWithProductionDefaults(client, "tenant-range-a") + const frame = () => screen.getByRole("region", { name: "Log lines" }) + expect(await within(frame()).findByText("No log lines for this range and filter.")).toBeInTheDocument() + + await userEvent.click(screen.getByRole("button", { name: "24h" })) + + await waitFor(() => expect(within(frame()).getByRole("status")).toHaveAttribute("aria-busy", "true")) + expect(within(frame()).queryByText("No log lines for this range and filter.")).toBeNull() + }) }) describe("LogsView", () => { diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index 05535e12e..1563d90b5 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -168,7 +168,7 @@ function ScopedLogsView({ root }: { root: string }) { // Previous results stay as placeholder data while a new query loads, so only a new key resets the scroll pin. key={followRun ?? `history\n${query}\n${range}`} entries={entries} - placeholder={placeholderFor({ follow, stopped: tail.state === "stopped", loading: history.isLoading, failed: Boolean(error), longestRange: range === LONGEST_RANGE, retry: () => void history.refetch() })} + placeholder={placeholderFor({ follow, stopped: tail.state === "stopped", loading: history.isLoading || (history.isPlaceholderData && history.isFetching), failed: Boolean(error), longestRange: range === LONGEST_RANGE, retry: () => void history.refetch() })} /> ) From 051442b3187abbe0eb73e21924ee1eebd1b1e701 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 00:58:29 +0700 Subject: [PATCH 292/518] fix(dashboard): mark previous log lines while a new query replaces them The previous result stays on screen while a new range or filter loads, with nothing saying those lines answer the old query. A status line now says so until the new result arrives. The empty case reads a placeholder as loading whether or not the request is in flight, so a paused query does not claim that nothing matched either. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/LogsView.test.tsx | 31 +++++++++++++++++++ .../src/components/observability/LogsView.tsx | 7 ++++- 2 files changed, 37 insertions(+), 1 deletion(-) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index 47cbf1c22..be8bff1a8 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -110,6 +110,22 @@ describe("LogsView with the production query defaults", () => { await waitFor(() => expect(within(frame()).getByRole("status")).toHaveAttribute("aria-busy", "true")) expect(within(frame()).queryByText("No log lines for this range and filter.")).toBeNull() }) + + it("marks the lines of the previous range as standing in while a longer range loads", async () => { + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("query") && Date.parse(params.get("start") ?? "") < Date.now() - 2 * 3600_000 + ? new Promise(() => {}) + : defaultHandler(endpoint, params), + ) + renderWithProductionDefaults(client, "tenant-range-b") + expect(await screen.findByText("ready to accept connections")).toBeInTheDocument() + expect(screen.queryByText(/previous lines/)).toBeNull() + + await userEvent.click(screen.getByRole("button", { name: "24h" })) + + expect(await screen.findByText(/previous lines/)).toHaveAttribute("aria-busy", "true") + expect(screen.getByText("ready to accept connections")).toBeInTheDocument() + }) }) describe("LogsView", () => { @@ -319,6 +335,21 @@ describe("LogsView", () => { expect(screen.queryByRole("button", { name: "Retry" })).toBeNull() }) + it("says it is loading while a retry runs", async () => { + let calls = 0 + const { client } = clientWith((endpoint, params) => { + if (!endpoint.endsWith("query")) return defaultHandler(endpoint, params) + return calls++ === 0 ? new Response("upstream down", { status: 502 }) : new Promise(() => {}) + }) + renderWithK8sProvider(, { client }) + + await userEvent.click(await screen.findByRole("button", { name: "Retry" })) + + const frame = screen.getByRole("region", { name: "Log lines" }) + await waitFor(() => expect(within(frame).getByRole("status")).toHaveAttribute("aria-busy", "true")) + expect(within(frame).queryByRole("button", { name: "Retry" })).toBeNull() + }) + it("opens a new history result at its newest line, not the old scroll position", async () => { const { client } = clientWith((endpoint, params) => endpoint.endsWith("query") diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx index 1563d90b5..a4376df47 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.tsx @@ -163,12 +163,17 @@ function ScopedLogsView({ root }: { root: string }) { {` · newest ${FOLLOW_CAP} lines kept`}

)} + {!follow && history.isPlaceholderData && entries.length > 0 && ( +

+ Showing the previous lines until the new range or filter loads… +

+ )} {error ? : null} void history.refetch() })} + placeholder={placeholderFor({ follow, stopped: tail.state === "stopped", loading: history.isLoading || history.isPlaceholderData, failed: Boolean(error), longestRange: range === LONGEST_RANGE, retry: () => void history.refetch() })} /> ) From 7105364161b9b0dea4afaae49681647c32edf1ca Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 01:02:17 +0700 Subject: [PATCH 293/518] test(dashboard): pin when the previous-lines note goes away The note must leave as soon as the new result lands and must not appear over live lines while following; neither was exercised, since the only test kept the longer range pending forever. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observability/LogsView.test.tsx | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx index be8bff1a8..faba85a4e 100644 --- a/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx +++ b/packages/system/dashboard/images/console/apps/console/src/components/observability/LogsView.test.tsx @@ -126,6 +126,45 @@ describe("LogsView with the production query defaults", () => { expect(await screen.findByText(/previous lines/)).toHaveAttribute("aria-busy", "true") expect(screen.getByText("ready to accept connections")).toBeInTheDocument() }) + + it("drops the standing-in note once the longer range answers", async () => { + let answer: (r: Response) => void = () => {} + const longer = new Promise((resolve) => { answer = resolve }) + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("query") && Date.parse(params.get("start") ?? "") < Date.now() - 2 * 3600_000 + ? longer + : defaultHandler(endpoint, params), + ) + renderWithProductionDefaults(client, "tenant-range-c") + expect(await screen.findByText("ready to accept connections")).toBeInTheDocument() + await userEvent.click(screen.getByRole("button", { name: "24h" })) + expect(await screen.findByText(/previous lines/)).toBeInTheDocument() + + answer(new Response(`${line("2026-11-03T12:00:00Z", "checkpoint complete")}\n`)) + + expect(await screen.findByText("checkpoint complete")).toBeInTheDocument() + expect(screen.queryByText(/previous lines/)).toBeNull() + expect(screen.queryByText("ready to accept connections")).toBeNull() + }) + + it("shows no standing-in note while following", async () => { + const { client } = clientWith((endpoint, params) => + endpoint.endsWith("tail") + ? new Response(`${line("2026-11-04T10:00:02Z", "live line")}\n`) + : endpoint.endsWith("query") && params.get("query")?.includes("error") + ? new Promise(() => {}) + : defaultHandler(endpoint, params), + ) + renderWithProductionDefaults(client, "tenant-follow-a") + expect(await screen.findByText("ready to accept connections")).toBeInTheDocument() + + await userEvent.type(screen.getByLabelText("LogsQL filter"), "error{Enter}") + expect(await screen.findByText(/previous lines/)).toBeInTheDocument() + await userEvent.click(screen.getByRole("button", { name: "Follow" })) + + expect(await screen.findByText("live line")).toBeInTheDocument() + expect(screen.queryByText(/previous lines/)).toBeNull() + }) }) describe("LogsView", () => { From 9cc9a3b48d96fcac4137096dff1f8b0675acff8d Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 4 Nov 2026 15:53:45 +0300 Subject: [PATCH 294/518] fix(postgres): stop backup jobs from fighting over the WAL archive destination The ObjectStore a Postgres Cluster archives its WAL to is shared by every backup of the app, yet each running BackupJob server-side applied its own strategy's destination onto it every five seconds with forced ownership. Two BackupJobs of one app with different BackupClasses flipped the destination back and forth until the healthy backup failed too, and a webhook or apiserver blip during that apply failed the BackupJob at once. A BackupJob now writes the destination only before its cnpg.io/Backup exists, and waits, with a WaitingForOtherBackup condition, while another running BackupJob of the app has a Backup in flight. The check reads the informer cache, so jobs released together, or one created while another's Backup is not yet cached, can still both proceed; an atomic per-application claim is a separate change. Write errors that cannot succeed on retry (invalid, bad request, forbidden) still fail the job at once. Any other is retried for the backup deadline counted from its first failure since the job last left the queue, recorded in a DestinationApplied condition. Once the Backup exists the Cluster is only read, and a failed read is returned for a controller retry instead of failing the running backup. The plugin rollout deadline counts from the attach rather than from the job's start, so queueing before the attach no longer shortens it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backupcontroller/backupjob_controller.go | 8 - .../backupjob_restore_lock.go | 12 +- .../cnpgstrategy_controller.go | 136 +++++- .../cnpgstrategy_controller_test.go | 438 +++++++++++++++++- 4 files changed, 563 insertions(+), 31 deletions(-) diff --git a/internal/backupcontroller/backupjob_controller.go b/internal/backupcontroller/backupjob_controller.go index b86278649..bbba19fad 100644 --- a/internal/backupcontroller/backupjob_controller.go +++ b/internal/backupcontroller/backupjob_controller.go @@ -315,14 +315,6 @@ func (r *BackupJobReconciler) handleProjectionError(ctx context.Context, j *back } // markBackupJobFailed records a terminal Failed phase on the BackupJob. -// -// Coupling note: a failure that fires before reconcileCNPG's StartedAt block -// has set backupJob.Status.StartedAt leaves StartedAt nil. The CNPG path's -// cnpgDefaultBackupDeadline check only fires once StartedAt is set, so an -// early-failure retry that reaches the StartedAt block restarts the deadline -// budget from the retry's StartedAt - intentional, since retries are -// user-driven and a fresh budget is what users expect after correcting the -// cause of the previous failure. func (r *BackupJobReconciler) markBackupJobFailed(ctx context.Context, backupJob *backupsv1alpha1.BackupJob, message string) (ctrl.Result, error) { logger := getLogger(ctx) now := metav1.Now() diff --git a/internal/backupcontroller/backupjob_restore_lock.go b/internal/backupcontroller/backupjob_restore_lock.go index 06f165285..c40d59aef 100644 --- a/internal/backupcontroller/backupjob_restore_lock.go +++ b/internal/backupcontroller/backupjob_restore_lock.go @@ -32,12 +32,12 @@ const ConditionReasonRestoreInProgress = "RestoreInProgress" // // A backup must not start against an application a restore is rewriting. It // would capture a half-restored state at best, and on the CNPG path it does -// worse: every BackupJob reconcile re-applies the Cluster's barman plugin, and -// one landing between the restore re-rendering the Cluster with its new -// serverName and the plugin being attached can put the restored cluster back -// on the source's WAL prefix, where its recovery fails with "Expected empty -// archive". Plan-created runs are the ones that hit this, since nothing else -// looks at restores before creating a BackupJob. +// worse: a BackupJob applies the Cluster's barman plugin before it creates +// its Backup, and an apply landing between the restore re-rendering the +// Cluster with its new serverName and the plugin being attached can put the +// restored cluster back on the source's WAL prefix, where its recovery fails +// with "Expected empty archive". Plan-created runs are the ones that hit +// this, since nothing else looks at restores before creating a BackupJob. func activeRestoreTargeting(ctx context.Context, c client.Client, namespace string, app corev1.TypedLocalObjectReference) (string, error) { app = NormalizeApplicationRef(app) list := &backupsv1alpha1.RestoreJobList{} diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 19f255c6c..a2a429b4f 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -176,7 +176,9 @@ const ( cnpgRecoveryMaxInspectPods = 5 // Cap on the wall-clock time a BackupJob waits for the barman-cloud - // plugin rollout before the cnpg.io/Backup exists. A Backup that exists + // plugin rollout, counted from the attach, or retries writing the WAL + // archive destination, counted from the first failure since the job last + // left the queue, before the cnpg.io/Backup exists. A Backup that exists // is not capped: CloudNativePG keeps a healthy one pending while another // backup of the same Cluster runs, or while it restarts one on another // instance, so no fixed deadline tells those apart from a stuck one. @@ -195,6 +197,12 @@ const ( // instance pods must postdate (see cnpgPluginRolloutPending). backupCondPluginRolledOut = "PluginRolledOut" + // Condition Type recorded while writing the WAL archive destination keeps + // failing. Its LastTransitionTime marks the first failure, so the retry + // budget is not spent by time the job waited for another backup of the + // app before it tried. + backupCondDestinationApplied = "DestinationApplied" + cnpgPodRoleLabel = "cnpg.io/podRole" cnpgPodRoleInstance = "instance" ) @@ -321,7 +329,23 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp rollout := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) awaitingRollout := existingBackup == nil && rollout != nil && rollout.Status == metav1.ConditionFalse - attachment, err := r.applyClusterPluginBackup(ctx, j.Namespace, clusterName, rendered, serverName, existingBackup != nil || awaitingRollout) + // The ObjectStore is the Cluster's WAL archive destination, shared by + // every backup of the app, so it is written only before this job's + // Backup exists and only while no other job's Backup is in flight. + writeDestination := existingBackup == nil + if writeDestination { + holder, err := r.otherCNPGBackupInFlight(ctx, j, clusterName) + if err != nil { + return ctrl.Result{}, err + } + if holder != "" { + apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondDestinationApplied) + return r.waitCNPGBackupJob(ctx, j, "WaitingForOtherBackup", fmt.Sprintf( + "waiting for BackupJob %s of the same application to finish its backup, which writes to the shared WAL archive destination", holder)) + } + } + + attachment, err := r.applyClusterPluginBackup(ctx, j.Namespace, clusterName, rendered, serverName, existingBackup != nil || awaitingRollout, writeDestination) if err != nil { if apierrors.IsNotFound(err) { // HelmRelease has not yet rendered the Cluster (fresh app, or @@ -340,7 +364,33 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil } - return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to attach barman-cloud plugin to Cluster: %v", err)) + if !writeDestination { + // Only the live Cluster was read for a Backup already running; + // a failed read says nothing about that Backup. + return ctrl.Result{}, fmt.Errorf("read cnpg.io/Cluster %s/%s: %w", j.Namespace, clusterName, err) + } + message := fmt.Sprintf("failed to attach barman-cloud plugin to Cluster: %v", err) + if apierrors.IsInvalid(err) || apierrors.IsBadRequest(err) || apierrors.IsForbidden(err) { + return r.markBackupJobFailed(ctx, j, message) + } + apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ + Type: backupCondDestinationApplied, + Status: metav1.ConditionFalse, + Reason: "ClusterPluginApplyRetrying", + Message: message, + }) + failingSince := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationApplied).LastTransitionTime + if cnpgBackupDeadlineExceeded(&failingSince) { + return r.markBackupJobFailed(ctx, j, message) + } + return r.waitCNPGBackupJob(ctx, j, "ClusterPluginApplyRetrying", message) + } + if writeDestination { + apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondDestinationApplied) + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready != nil && + (ready.Reason == "WaitingForOtherBackup" || ready.Reason == "ClusterPluginApplyRetrying") { + apimeta.RemoveStatusCondition(&j.Status.Conditions, "Ready") + } } if attachment.serverName != serverName { logger.Info("preserving the live Cluster's barman serverName over the strategy template's", @@ -377,7 +427,7 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp return ctrl.Result{}, err } if pending != "" { - if cnpgBackupDeadlineExceeded(j.Status.StartedAt) { + if cnpgBackupDeadlineExceeded(&rollout.LastTransitionTime) { return r.markBackupJobFailed(ctx, j, fmt.Sprintf( "barman-cloud plugin rollout did not finish within %s: %s", cnpgDefaultBackupDeadline, pending)) } @@ -486,16 +536,16 @@ func cnpgBackupFailedMessage(b *cnpgtypes.Backup, errText string) string { return message } -// cnpgBackupDeadlineExceeded reports whether enough wall-clock time has -// elapsed since the BackupJob started that we should stop waiting for the -// barman-cloud plugin rollout and fail the run. Returns false when -// StartedAt is nil so the very first reconcile (which sets StartedAt) does -// not trip the gate. -func cnpgBackupDeadlineExceeded(startedAt *metav1.Time) bool { - if startedAt == nil { +// cnpgBackupDeadlineExceeded reports whether more than the backup deadline +// has passed since the BackupJob began the step it is waiting on: the plugin +// rollout, counted from the attach, or retrying the destination write, +// counted from the first failure since the job last left the queue behind +// another backup. Returns false for a nil time. +func cnpgBackupDeadlineExceeded(since *metav1.Time) bool { + if since == nil { return false } - return time.Since(startedAt.Time) > cnpgDefaultBackupDeadline + return time.Since(since.Time) > cnpgDefaultBackupDeadline } // cnpgPurgeNeeded decides whether the controller should delete the live @@ -601,7 +651,10 @@ var cnpgClusterGVR = schema.GroupVersionResource{ // With attach unset, a Cluster that has no barman-cloud plugin yet is left // untouched and only reported missing, so the caller can record that it is // about to roll the Cluster before it does. -func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, namespace, clusterName string, t *strategyv1alpha1.CNPGTemplate, serverName string, attach bool) (cnpgPluginAttachment, error) { +// +// With write unset nothing is written: only what the live Cluster carries is +// reported. +func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, namespace, clusterName string, t *strategyv1alpha1.CNPGTemplate, serverName string, attach, write bool) (cnpgPluginAttachment, error) { // Live read: the cached Cluster can predate a restore re-render, and the // serverName, the flavor and the UID below must come from the current // object. @@ -626,6 +679,9 @@ func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, name return attachment, nil } } + if !write { + return attachment, nil + } serverName = attachment.serverName // The ObjectStore is named after the Cluster (distinct kind, same @@ -801,6 +857,60 @@ func (r *BackupJobReconciler) findCNPGBackupForJob(ctx context.Context, j *backu return &list.Items[0], nil } +// otherCNPGBackupInFlight returns the name of another BackupJob that is still +// running and whose cnpg.io/Backup of clusterName has not finished, or "" when +// there is none. A Backup left behind by a finished or deleted job does not +// hold the queue. +func (r *BackupJobReconciler) otherCNPGBackupInFlight(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string) (string, error) { + list := &cnpgtypes.BackupList{} + if err := r.List(ctx, list, client.InNamespace(j.Namespace)); err != nil { + return "", err + } + var holders []string + for i := range list.Items { + b := &list.Items[i] + owner := b.Labels[backupsv1alpha1.OwningJobNameLabel] + if b.Spec.Cluster.Name != clusterName || owner == "" || owner == j.Name { + continue + } + if b.Status.Phase == cnpgBackupPhaseComplete || b.Status.Phase == cnpgBackupPhaseFailed { + continue + } + ownerJob := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, client.ObjectKey{Namespace: j.Namespace, Name: owner}, ownerJob); err != nil { + if apierrors.IsNotFound(err) { + continue + } + return "", err + } + if ownerJob.Status.Phase == backupsv1alpha1.BackupJobPhaseSucceeded || ownerJob.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + continue + } + holders = append(holders, owner) + } + if len(holders) == 0 { + return "", nil + } + sort.Strings(holders) + return holders[0], nil +} + +// waitCNPGBackupJob requeues the BackupJob and records why it waits in its +// Ready condition, writing the status only when the conditions changed. +func (r *BackupJobReconciler) waitCNPGBackupJob(ctx context.Context, j *backupsv1alpha1.BackupJob, reason, message string) (ctrl.Result, error) { + if apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ + Type: "Ready", + Status: metav1.ConditionFalse, + Reason: reason, + Message: message, + }) { + if err := r.Status().Update(ctx, j); err != nil { + return ctrl.Result{}, err + } + } + return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil +} + // createCNPGBackupArtifact materialises a Cozystack Backup resource carrying // the metadata callers need to drive a future restore. The source app's // spec.databases / spec.users are snapshotted into Status.UnderlyingResources diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 551209fac..c94630470 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -3,6 +3,7 @@ package backupcontroller import ( "context" + "errors" "fmt" "strings" "testing" @@ -19,6 +20,7 @@ import ( "k8s.io/client-go/dynamic" dynamicfake "k8s.io/client-go/dynamic/fake" "k8s.io/client-go/kubernetes/scheme" + clienttesting "k8s.io/client-go/testing" "k8s.io/client-go/tools/record" "sigs.k8s.io/controller-runtime/pkg/client" clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake" @@ -1099,7 +1101,7 @@ func TestApplyClusterPluginBackup_NotFoundOnMissingCluster(t *testing.T) { }, } - _, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-missing", tmpl, "postgres-missing", true) + _, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-missing", tmpl, "postgres-missing", true, true) if err == nil { t.Fatalf("expected NotFound error, got nil") } @@ -1739,7 +1741,7 @@ func TestApplyClusterPluginBackup_ReadsTheLiveClusterNotTheCache(t *testing.T) { BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://bucket/x/"}, } - got, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "strategy-name", true) + got, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "strategy-name", true, true) if err != nil { t.Fatalf("unexpected error: %v", err) } @@ -1949,7 +1951,7 @@ func TestApplyClusterPluginBackup_PreservesLiveServerName(t *testing.T) { BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://bucket/x/"}, } - got, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "strategy-name", true) + got, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "strategy-name", true, true) if err != nil { t.Fatalf("unexpected error: %v", err) } @@ -2118,7 +2120,7 @@ func TestApplyClusterPluginBackup_PatchesExistingCluster(t *testing.T) { }, } - if _, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "tenant-app", true); err != nil { + if _, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "tenant-app", true, true); err != nil { t.Fatalf("unexpected error: %v", err) } @@ -2480,6 +2482,434 @@ func TestReconcileCNPG_BackupPhaseOutcome(t *testing.T) { } } +// newAttachedObjectStoreFixture is a Cluster that already runs the barman-cloud +// plugin, whose ObjectStore currently points at liveDestination; the strategy +// of the fixture's BackupJob points at s3://bucket/. +func newAttachedObjectStoreFixture(t *testing.T, startedAt time.Time, liveDestination string) *cnpgRolloutFixture { + t.Helper() + f := newCNPGRolloutFixture(t, newRolloutBackupJob(startedAt), + newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)), + newRolloutInstancePod(time.Now().Add(-time.Hour))) + store := &cnpgtypes.ObjectStore{ + ObjectMeta: metav1.ObjectMeta{Namespace: rolloutNS, Name: rolloutCluster}, + Spec: cnpgtypes.ObjectStoreSpec{Configuration: cnpgtypes.BarmanObjectStoreConfiguration{DestinationPath: liveDestination}}, + } + if err := f.c.Create(context.Background(), store); err != nil { + t.Fatalf("seed ObjectStore: %v", err) + } + return f +} + +func (f *cnpgRolloutFixture) liveDestination() string { + f.t.Helper() + store := &cnpgtypes.ObjectStore{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: rolloutCluster}, store); err != nil { + f.t.Fatalf("get ObjectStore: %v", err) + } + return store.Spec.Configuration.DestinationPath +} + +// seedCNPGBackup creates a cnpg.io/Backup of cluster owned by the BackupJob +// named owner; an empty owner leaves it unlabelled, as a Backup created by +// hand. +func (f *cnpgRolloutFixture) seedCNPGBackup(name, owner, cluster, phase string) { + f.t.Helper() + var labels map[string]string + if owner != "" { + labels = map[string]string{ + backupsv1alpha1.OwningJobNameLabel: owner, + backupsv1alpha1.OwningJobNamespaceLabel: rolloutNS, + } + } + b := &cnpgtypes.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: rolloutNS, Name: name, Labels: labels}, + Spec: cnpgtypes.BackupSpec{Cluster: cnpgtypes.ClusterReference{Name: cluster}}, + Status: cnpgtypes.BackupStatus{Phase: phase}, + } + if err := f.c.Create(context.Background(), b); err != nil { + f.t.Fatalf("seed cnpg.io/Backup %s: %v", name, err) + } +} + +func (f *cnpgRolloutFixture) setJobCondition(cond metav1.Condition) { + f.t.Helper() + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + f.t.Fatalf("get BackupJob: %v", err) + } + j.Status.Conditions = append(j.Status.Conditions, cond) + if err := f.c.Status().Update(context.Background(), j); err != nil { + f.t.Fatalf("seed BackupJob condition: %v", err) + } +} + +func (f *cnpgRolloutFixture) seedOtherBackupJob(name string, phase backupsv1alpha1.BackupJobPhase) { + f.t.Helper() + other := newRolloutBackupJob(time.Now().Add(-time.Hour)) + other.Name = name + other.Status.Phase = phase + if err := f.c.Create(context.Background(), other); err != nil { + f.t.Fatalf("seed BackupJob %s: %v", name, err) + } +} + +// The ObjectStore is the Cluster's WAL archive destination, so a BackupJob +// whose cnpg.io/Backup is already running must not keep re-applying it: a +// second BackupJob of the same app with another destination would otherwise +// flip it back and forth under both backups. +func TestReconcileCNPG_RunningBackupDoesNotReapplyObjectStore(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f.seedCNPGBackup("bj-cnpg", "bj", rolloutCluster, "running") + + j, _ := f.pass() + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("BackupJob failed: %s", j.Status.Message) + } + if got := f.liveDestination(); got != "s3://other/" { + t.Fatalf("a pass over a running backup re-applied the ObjectStore: destination %q", got) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForOtherBackup" { + t.Fatalf("a running backup waits on itself: %+v", ready) + } +} + +func TestReconcileCNPG_WaitsForAnotherRunningBackupOfTheApp(t *testing.T) { + cases := []struct { + name string + otherJob backupsv1alpha1.BackupJobPhase + seedJob bool + otherBackup string + owner string + cluster string + wantWait bool + }{ + {name: "another running job with a running Backup holds it", seedJob: true, otherJob: backupsv1alpha1.BackupJobPhaseRunning, otherBackup: "running", wantWait: true}, + {name: "another running job with a pending Backup holds it", seedJob: true, otherJob: backupsv1alpha1.BackupJobPhaseRunning, otherBackup: "pending", wantWait: true}, + {name: "a completed Backup of another job does not", seedJob: true, otherJob: backupsv1alpha1.BackupJobPhaseRunning, otherBackup: cnpgBackupPhaseComplete}, + {name: "a failed Backup of another job does not", seedJob: true, otherJob: backupsv1alpha1.BackupJobPhaseRunning, otherBackup: cnpgBackupPhaseFailed}, + {name: "a Backup of a failed job does not", seedJob: true, otherJob: backupsv1alpha1.BackupJobPhaseFailed, otherBackup: "pending"}, + {name: "a Backup of a succeeded job does not", seedJob: true, otherJob: backupsv1alpha1.BackupJobPhaseSucceeded, otherBackup: "pending"}, + {name: "a Backup whose job is gone does not", otherBackup: "pending"}, + {name: "a Backup of another Cluster does not", seedJob: true, otherJob: backupsv1alpha1.BackupJobPhaseRunning, otherBackup: "running", cluster: "postgres-other"}, + {name: "a Backup created by hand does not", otherBackup: "running", owner: "-"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + if tc.seedJob { + f.seedOtherBackupJob("bj-other", tc.otherJob) + } + owner, cluster := "bj-other", rolloutCluster + if tc.owner == "-" { + owner = "" + } + if tc.cluster != "" { + cluster = tc.cluster + } + f.seedCNPGBackup("bj-other-cnpg", owner, cluster, tc.otherBackup) + // client-go refuses a Get with an empty name before it reaches + // the apiserver; the fake client answers NotFound instead. + statusWrites := 0 + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if key.Name == "" { + return errors.New("resource name may not be empty") + } + return cl.Get(ctx, key, obj, opts...) + }, + SubResourceUpdate: func(ctx context.Context, cl client.Client, sub string, obj client.Object, opts ...client.SubResourceUpdateOption) error { + statusWrites++ + return cl.SubResource(sub).Update(ctx, obj, opts...) + }, + }) + + j, backups := f.pass() + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("BackupJob failed: %s", j.Status.Message) + } + ours := 0 + for _, b := range backups { + if b.Labels[backupsv1alpha1.OwningJobNameLabel] == "bj" { + ours++ + } + } + ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready") + if tc.wantWait { + if got := f.liveDestination(); got != "s3://other/" { + t.Errorf("the waiting job applied its destination over a running backup: %q", got) + } + if ours != 0 { + t.Errorf("the waiting job created %d cnpg.io/Backup(s)", ours) + } + if ready == nil || ready.Status != metav1.ConditionFalse || ready.Reason != "WaitingForOtherBackup" || !strings.Contains(ready.Message, "bj-other") { + t.Errorf("Ready = %+v, want False/WaitingForOtherBackup naming bj-other", ready) + } + statusWrites = 0 + f.pass() + if statusWrites != 0 { + t.Errorf("a second pass with the same holder wrote the BackupJob status %d time(s)", statusWrites) + } + + held := &cnpgtypes.Backup{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj-other-cnpg"}, held); err != nil { + t.Fatalf("get holder Backup: %v", err) + } + held.Status.Phase = cnpgBackupPhaseComplete + if err := f.c.Update(context.Background(), held); err != nil { + t.Fatalf("complete holder Backup: %v", err) + } + j, _ = f.pass() + if got := f.liveDestination(); got != "s3://bucket/" { + t.Errorf("once the holder finished, destination = %q, want the job's own s3://bucket/", got) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForOtherBackup" { + t.Errorf("Ready still names the finished holder: %+v", ready) + } + return + } + if got := f.liveDestination(); got != "s3://bucket/" { + t.Errorf("destination = %q, want the job's own s3://bucket/", got) + } + if ours != 1 { + t.Errorf("got %d cnpg.io/Backups for the job, want 1", ours) + } + }) + } +} + +func TestReconcileCNPG_ObjectStoreApplyErrors(t *testing.T) { + expired := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)) + webhookDown := apierrors.NewInternalError(errors.New(`failed calling webhook "mcluster.cnpg.io": connect: connection refused`)) + invalid := apierrors.NewInvalid(schema.GroupKind{Group: cnpgtypes.BarmanGroupVersion.Group, Kind: "ObjectStore"}, rolloutCluster, nil) + cases := []struct { + name string + err error + startedAt time.Time + // failingSince seeds an earlier failure of the same write. + failingSince time.Time + wantPhase backupsv1alpha1.BackupJobPhase + }{ + {name: "a transient error is retried", err: webhookDown, startedAt: time.Now()}, + {name: "a transient error failing since past the deadline fails", err: webhookDown, startedAt: expired, failingSince: expired, wantPhase: backupsv1alpha1.BackupJobPhaseFailed}, + {name: "a transient error after a long wait is retried", err: webhookDown, startedAt: expired}, + {name: "an invalid object fails at once", err: invalid, startedAt: time.Now(), wantPhase: backupsv1alpha1.BackupJobPhaseFailed}, + {name: "a bad request fails at once", err: apierrors.NewBadRequest("malformed"), startedAt: time.Now(), wantPhase: backupsv1alpha1.BackupJobPhaseFailed}, + {name: "a forbidden write fails at once", err: apierrors.NewForbidden(schema.GroupResource{Resource: "objectstores"}, rolloutCluster, errors.New("denied")), startedAt: time.Now(), wantPhase: backupsv1alpha1.BackupJobPhaseFailed}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f := newAttachedObjectStoreFixture(t, tc.startedAt, "s3://other/") + if !tc.failingSince.IsZero() { + f.setJobCondition(metav1.Condition{ + Type: backupCondDestinationApplied, Status: metav1.ConditionFalse, Reason: "ClusterPluginApplyRetrying", + Message: "earlier failure", LastTransitionTime: metav1.NewTime(tc.failingSince), + }) + } + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + if _, ok := obj.(*cnpgtypes.ObjectStore); ok { + return tc.err + } + return cl.Patch(ctx, obj, patch, opts...) + }, + }) + + j, backups := f.pass() + if j.Status.Phase != tc.wantPhase { + t.Fatalf("BackupJob phase = %q (%s), want %q", j.Status.Phase, j.Status.Message, tc.wantPhase) + } + if len(backups) != 0 { + t.Errorf("a cnpg.io/Backup was created although the ObjectStore was not applied: %+v", backups) + } + if tc.wantPhase == "" { + ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready") + if ready == nil || ready.Reason != "ClusterPluginApplyRetrying" || !strings.Contains(ready.Message, "connection refused") { + t.Errorf("Ready = %+v, want ClusterPluginApplyRetrying carrying the error", ready) + } + } + }) + } +} + +// A write that recovers clears the retry state, so a later failure gets a +// fresh budget and Ready no longer reports the retry. +func TestReconcileCNPG_ObjectStoreApplyRecovers(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + failing := true + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + if _, ok := obj.(*cnpgtypes.ObjectStore); ok && failing { + return apierrors.NewInternalError(errors.New("webhook down")) + } + return cl.Patch(ctx, obj, patch, opts...) + }, + }) + j, _ := f.pass() + if cond := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationApplied); cond == nil || cond.Status != metav1.ConditionFalse { + t.Fatalf("%s = %+v after a failed write, want False", backupCondDestinationApplied, cond) + } + + failing = false + j, backups := f.pass() + if len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups after the write recovered, want 1", len(backups)) + } + if cond := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationApplied); cond != nil { + t.Errorf("%s = %+v left over after the write recovered", backupCondDestinationApplied, cond) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready != nil && ready.Reason == "ClusterPluginApplyRetrying" { + t.Errorf("Ready still reports the retry: %+v", ready) + } +} + +// An error reading the holder's BackupJob is not taken as "no holder". +func TestReconcileCNPG_HolderLookupErrorIsReturned(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f.seedCNPGBackup("bj-other-cnpg", "bj-other", rolloutCluster, "running") + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if key.Name == "bj-other" { + return apierrors.NewServiceUnavailable("cache not synced") + } + return cl.Get(ctx, key, obj, opts...) + }, + }) + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := f.r.reconcileCNPG(context.Background(), j, f.resolved); err == nil { + t.Fatalf("reconcileCNPG swallowed the holder lookup error") + } + if got := f.liveDestination(); got != "s3://other/" { + t.Errorf("destination written although the holder could not be read: %q", got) + } +} + +// The rollout deadline counts from when the plugin was attached, not from +// the job's start, which may lie behind a long wait for another backup. +func TestReconcileCNPG_PluginRolloutDeadlineCountsFromAttach(t *testing.T) { + expired := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)) + f := newCNPGRolloutFixture(t, newRolloutBackupJob(expired), + newRolloutCluster(cnpgClusterHealthyPhase), newRolloutInstancePod(time.Now().Add(-time.Hour))) + + f.pass() // latch + j, _ := f.pass() + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("a rollout that just began failed on the job's old start: %s", j.Status.Message) + } + if cond := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); cond == nil || cond.Reason != "WaitingForPluginRollout" { + t.Fatalf("Ready = %+v, want WaitingForPluginRollout", cond) + } + + rollout := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) + rollout.LastTransitionTime = metav1.NewTime(expired) + if err := f.c.Status().Update(context.Background(), j); err != nil { + t.Fatalf("age the rollout: %v", err) + } + j, _ = f.pass() + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed || !strings.Contains(j.Status.Message, "rollout did not finish") { + t.Fatalf("phase = %q (%s), want Failed on a rollout older than the deadline", j.Status.Phase, j.Status.Message) + } +} + +// Waiting for another backup resets the retry budget of the destination +// write, so the queue does not spend it. +func TestReconcileCNPG_QueueResetsDestinationRetryBudget(t *testing.T) { + expired := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)) + f := newAttachedObjectStoreFixture(t, expired, "s3://other/") + f.setJobCondition(metav1.Condition{ + Type: backupCondDestinationApplied, Status: metav1.ConditionFalse, Reason: "ClusterPluginApplyRetrying", + Message: "earlier failure", LastTransitionTime: metav1.NewTime(expired), + }) + f.seedOtherBackupJob("bj-other", backupsv1alpha1.BackupJobPhaseRunning) + f.seedCNPGBackup("bj-other-cnpg", "bj-other", rolloutCluster, "running") + + j, _ := f.pass() + if cond := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationApplied); cond != nil { + t.Fatalf("%s = %+v kept while waiting for another backup", backupCondDestinationApplied, cond) + } + + held := &cnpgtypes.Backup{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj-other-cnpg"}, held); err != nil { + t.Fatalf("get holder Backup: %v", err) + } + held.Status.Phase = cnpgBackupPhaseComplete + if err := f.c.Update(context.Background(), held); err != nil { + t.Fatalf("complete holder Backup: %v", err) + } + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + if _, ok := obj.(*cnpgtypes.ObjectStore); ok { + return apierrors.NewInternalError(errors.New("webhook down")) + } + return cl.Patch(ctx, obj, patch, opts...) + }, + }) + j, _ = f.pass() + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("the first failure after the queue failed the job: %s", j.Status.Message) + } +} + +func TestOtherCNPGBackupInFlight_IgnoresTheJobsOwnBackup(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f.seedCNPGBackup("bj-cnpg", "bj", rolloutCluster, "running") + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + holder, err := f.r.otherCNPGBackupInFlight(context.Background(), j, rolloutCluster) + if err != nil { + t.Fatalf("otherCNPGBackupInFlight: %v", err) + } + if holder != "" { + t.Fatalf("the job waits on its own Backup: holder %q", holder) + } +} + +// Once the job's Backup exists the Cluster is only read; a failed read is +// returned for a retry and neither fails the running backup nor starts the +// destination retry budget. +func TestReconcileCNPG_ClusterReadErrorDoesNotFailRunningBackup(t *testing.T) { + for name, readErr := range map[string]error{ + "internal error": apierrors.NewInternalError(errors.New("blip")), + // A write answered Forbidden fails the job; a read must not. + "forbidden": apierrors.NewForbidden(schema.GroupResource{Group: cnpgtypes.GroupName, Resource: "clusters"}, rolloutCluster, errors.New("denied")), + } { + t.Run(name, func(t *testing.T) { testClusterReadErrorKeepsBackup(t, readErr) }) + } +} + +func testClusterReadErrorKeepsBackup(t *testing.T, readErr error) { + expired := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)) + f := newAttachedObjectStoreFixture(t, expired, "s3://other/") + f.seedCNPGBackup("bj-cnpg", "bj", rolloutCluster, "running") + f.setJobCondition(metav1.Condition{ + Type: backupCondDestinationApplied, Status: metav1.ConditionFalse, Reason: "ClusterPluginApplyRetrying", + Message: "earlier failure", LastTransitionTime: metav1.NewTime(expired), + }) + dyn := f.r.Interface.(*dynamicfake.FakeDynamicClient) + dyn.PrependReactor("get", "clusters", func(clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, readErr + }) + + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := f.r.reconcileCNPG(context.Background(), j, f.resolved); err == nil { + t.Fatalf("a failed Cluster read was not returned for a retry") + } + persisted := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, persisted); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if persisted.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("a failed Cluster read failed the running backup: %s", persisted.Status.Message) + } +} + // cnpgRolloutFixture drives reconcileCNPG against one Postgres app whose // Cluster and instance pods a test swaps between passes, standing in for // CNPG rolling the instances. From 570fcec7253286d597308dc14fc8dc31ad5d0ef2 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:04:57 +0300 Subject: [PATCH 295/518] fix(postgres): hold a queued backup job while its application is restored The restore gate in Reconcile looks only at a BackupJob that has not started, and the CNPG driver stamps StartedAt on its first pass. A job that then waits behind another backup of the application, for as long as that backup runs, writes the WAL archive destination and the Cluster plugin once released without looking for a restore that began in the meantime. A write landing while the restore re-renders the Cluster can put the restored Cluster back on the source's WAL prefix, where recovery fails with "Expected empty archive". Every pass that is about to write the destination now looks for a restore of the application and holds the job the way the gate does. A job whose Backup already runs writes nothing and is not held. The hold resets the retry budget of the destination write, as the wait for another backup does, so time spent behind a restore does not fail the job on the first error after it. Signed-off-by: Alexey Artamonov Assisted-by: LLM --- .../backupjob_restore_lock.go | 11 +- .../cnpgstrategy_controller.go | 11 ++ .../cnpgstrategy_controller_test.go | 172 +++++++++++++++++- 3 files changed, 187 insertions(+), 7 deletions(-) diff --git a/internal/backupcontroller/backupjob_restore_lock.go b/internal/backupcontroller/backupjob_restore_lock.go index c40d59aef..3782527d9 100644 --- a/internal/backupcontroller/backupjob_restore_lock.go +++ b/internal/backupcontroller/backupjob_restore_lock.go @@ -130,13 +130,14 @@ func ptrString(s *string) string { return *s } -// holdForRestore keeps a BackupJob that has not started yet waiting while a -// restore writes into its application. It stays non-terminal, so a scheduled -// run still happens once the restore ends, only later than its slot. +// holdForRestore keeps a BackupJob waiting while a restore writes into its +// application: one that has not started yet, or a CNPG one about to write +// its WAL archive destination. It stays non-terminal, so a scheduled run +// still happens once the restore ends, only later than its slot. // // A failed status write is returned so the manager retries it with backoff. -// The job is held either way, since nothing past the gate ran, but a hold -// nobody can see reads as a run stuck for no reason. +// The job is held either way, since it writes nothing past the hold, but a +// hold nobody can see reads as a run stuck for no reason. func (r *BackupJobReconciler) holdForRestore(ctx context.Context, j *backupsv1alpha1.BackupJob, restore string) (ctrl.Result, error) { if j.Status.Phase == backupsv1alpha1.BackupJobPhaseEmpty { j.Status.Phase = backupsv1alpha1.BackupJobPhasePending diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index a2a429b4f..eec19819c 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -334,6 +334,17 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp // Backup exists and only while no other job's Backup is in flight. writeDestination := existingBackup == nil if writeDestination { + // The gate in Reconcile sees only a job that has not started, and + // this one may have waited in the queue for as long as a restore + // takes to begin. + restore, err := activeRestoreTargeting(ctx, r.Client, j.Namespace, j.Spec.ApplicationRef) + if err != nil { + return ctrl.Result{}, err + } + if restore != "" { + apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondDestinationApplied) + return r.holdForRestore(ctx, j, restore) + } holder, err := r.otherCNPGBackupInFlight(ctx, j, clusterName) if err != nil { return ctrl.Result{}, err diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index c94630470..f4ec68fb9 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -22,6 +22,7 @@ import ( "k8s.io/client-go/kubernetes/scheme" clienttesting "k8s.io/client-go/testing" "k8s.io/client-go/tools/record" + ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake" "sigs.k8s.io/controller-runtime/pkg/client/interceptor" @@ -2852,6 +2853,163 @@ func TestReconcileCNPG_QueueResetsDestinationRetryBudget(t *testing.T) { } } +// seedRestoreOfApp starts a RestoreJob that writes into the fixture's app. +func (f *cnpgRolloutFixture) seedRestoreOfApp() *backupsv1alpha1.RestoreJob { + f.t.Helper() + appGroup := backupsv1alpha1.DefaultApplicationAPIGroup + strategyGroup := strategyv1alpha1.GroupVersion.Group + backup := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: rolloutNS, Name: "app-backup"}, + Spec: backupsv1alpha1.BackupSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &appGroup, Kind: postgresAppKind, Name: rolloutApp}, + StrategyRef: corev1.TypedLocalObjectReference{APIGroup: &strategyGroup, Kind: strategyv1alpha1.CNPGStrategyKind, Name: "cnpg-strategy"}, + }, + } + restore := &backupsv1alpha1.RestoreJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: rolloutNS, Name: "app-restore"}, + Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}}, + } + for _, obj := range []client.Object{backup, restore} { + if err := f.c.Create(context.Background(), obj); err != nil { + f.t.Fatalf("seed %T: %v", obj, err) + } + } + return restore +} + +// A job past its start writes the WAL archive destination only while no +// restore rewrites its application. The restore gate in Reconcile looks only +// at a job that has not started, and a job released from the queue, or one +// still before its first write, has started long ago: a write landing during +// the restore can put the restored Cluster back on the source's WAL prefix. +func TestReconcileCNPG_HoldsTheDestinationWriteWhileTheAppIsRestored(t *testing.T) { + for _, queued := range []bool{true, false} { + t.Run(fmt.Sprintf("released from the queue=%t", queued), func(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + if queued { + f.seedOtherBackupJob("bj-other", backupsv1alpha1.BackupJobPhaseRunning) + f.seedCNPGBackup("bj-other-cnpg", "bj-other", rolloutCluster, "running") + if j, _ := f.pass(); apimeta.FindStatusCondition(j.Status.Conditions, "Ready").Reason != "WaitingForOtherBackup" { + t.Fatalf("Ready = %+v, want the job queued", apimeta.FindStatusCondition(j.Status.Conditions, "Ready")) + } + } + restore := f.seedRestoreOfApp() + if queued { + held := &cnpgtypes.Backup{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj-other-cnpg"}, held); err != nil { + t.Fatalf("get holder Backup: %v", err) + } + held.Status.Phase = cnpgBackupPhaseComplete + if err := f.c.Update(context.Background(), held); err != nil { + t.Fatalf("complete holder Backup: %v", err) + } + holder := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj-other"}, holder); err != nil { + t.Fatalf("get holder BackupJob: %v", err) + } + holder.Status.Phase = backupsv1alpha1.BackupJobPhaseSucceeded + if err := f.c.Status().Update(context.Background(), holder); err != nil { + t.Fatalf("finish holder BackupJob: %v", err) + } + } + + j, backups, res := f.passResult() + if got := f.liveDestination(); got != "s3://other/" { + t.Errorf("destination written during a restore of the app: %q", got) + } + for _, b := range backups { + if b.Labels[backupsv1alpha1.OwningJobNameLabel] == "bj" { + t.Errorf("a cnpg.io/Backup was created during a restore of the app: %s", b.Name) + } + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != ConditionReasonRestoreInProgress { + t.Errorf("Ready = %+v, want %s", ready, ConditionReasonRestoreInProgress) + } + if res.RequeueAfter == 0 { + t.Errorf("a held job is not requeued: %+v", res) + } + + restore.Status.Phase = backupsv1alpha1.RestoreJobPhaseSucceeded + if err := f.c.Status().Update(context.Background(), restore); err != nil { + t.Fatalf("finish the restore: %v", err) + } + f.pass() + if got := f.liveDestination(); got != "s3://bucket/" { + t.Errorf("once the restore finished, destination = %q, want the job's own s3://bucket/", got) + } + }) + } +} + +// A job whose Backup already runs writes nothing, so a restore does not hold +// it: holding would strand a backup that is already moving data. +func TestReconcileCNPG_RestoreDoesNotHoldARunningBackup(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + f.seedCNPGBackup("bj-cnpg", "bj", rolloutCluster, "running") + f.seedRestoreOfApp() + + j, _ := f.pass() + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForCNPGBackup" { + t.Errorf("Ready = %+v, want the running backup still followed", ready) + } +} + +// An error looking for a restore is not taken as "no restore". +func TestReconcileCNPG_RestoreLookupErrorIsReturned(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*backupsv1alpha1.RestoreJobList); ok { + return apierrors.NewServiceUnavailable("cache not synced") + } + return cl.List(ctx, list, opts...) + }, + }) + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := f.r.reconcileCNPG(context.Background(), j, f.resolved); err == nil { + t.Fatalf("reconcileCNPG swallowed the restore lookup error") + } + if got := f.liveDestination(); got != "s3://other/" { + t.Errorf("destination written although restores could not be listed: %q", got) + } +} + +// Waiting for a restore resets the retry budget of the destination write, +// as waiting for another backup does, so the restore does not spend it. +func TestReconcileCNPG_RestoreResetsDestinationRetryBudget(t *testing.T) { + expired := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)) + f := newAttachedObjectStoreFixture(t, expired, "s3://other/") + f.setJobCondition(metav1.Condition{ + Type: backupCondDestinationApplied, Status: metav1.ConditionFalse, Reason: "ClusterPluginApplyRetrying", + Message: "earlier failure", LastTransitionTime: metav1.NewTime(expired), + }) + restore := f.seedRestoreOfApp() + + j, _ := f.pass() + if cond := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationApplied); cond != nil { + t.Fatalf("%s = %+v kept while waiting for a restore", backupCondDestinationApplied, cond) + } + + restore.Status.Phase = backupsv1alpha1.RestoreJobPhaseSucceeded + if err := f.c.Status().Update(context.Background(), restore); err != nil { + t.Fatalf("finish the restore: %v", err) + } + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + if _, ok := obj.(*cnpgtypes.ObjectStore); ok { + return apierrors.NewInternalError(errors.New("webhook down")) + } + return cl.Patch(ctx, obj, patch, opts...) + }, + }) + if j, _ = f.pass(); j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("the first failure after the restore failed the job: %s", j.Status.Message) + } +} + func TestOtherCNPGBackupInFlight_IgnoresTheJobsOwnBackup(t *testing.T) { f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") f.seedCNPGBackup("bj-cnpg", "bj", rolloutCluster, "running") @@ -2987,6 +3145,15 @@ func newRolloutInstancePod(created time.Time) *corev1.Pod { // pass runs one reconcile and returns the persisted BackupJob and the // cnpg.io/Backups that exist afterwards. func (f *cnpgRolloutFixture) pass() (*backupsv1alpha1.BackupJob, []cnpgtypes.Backup) { + f.t.Helper() + j, backups, _ := f.passResult() + return j, backups +} + +// passResult is pass that also returns the reconcile result: the controller +// watches only BackupJobs, so a pass that waits on anything else is woken +// only by its own requeue. +func (f *cnpgRolloutFixture) passResult() (*backupsv1alpha1.BackupJob, []cnpgtypes.Backup, ctrl.Result) { f.t.Helper() ctx := context.Background() key := client.ObjectKey{Namespace: rolloutNS, Name: "bj"} @@ -2994,7 +3161,8 @@ func (f *cnpgRolloutFixture) pass() (*backupsv1alpha1.BackupJob, []cnpgtypes.Bac if err := f.c.Get(ctx, key, j); err != nil { f.t.Fatalf("get BackupJob: %v", err) } - if _, err := f.r.reconcileCNPG(ctx, j, f.resolved); err != nil { + res, err := f.r.reconcileCNPG(ctx, j, f.resolved) + if err != nil { f.t.Fatalf("reconcileCNPG: %v", err) } got := &backupsv1alpha1.BackupJob{} @@ -3005,7 +3173,7 @@ func (f *cnpgRolloutFixture) pass() (*backupsv1alpha1.BackupJob, []cnpgtypes.Bac if err := f.c.List(ctx, backups, client.InNamespace(rolloutNS)); err != nil { f.t.Fatalf("list cnpg.io/Backups: %v", err) } - return got, backups.Items + return got, backups.Items, res } // observeCNPG makes the next live read see cluster, as CNPG has written it. From 567f07d7080223b8808834092b8e313fbd79262d Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:06:31 +0300 Subject: [PATCH 296/518] fix(postgres): time the plugin rollout from the write that attached it The PluginRolledOut latch is recorded before the Cluster write that attaches the barman-cloud plugin, and its time was used both as the start of the rollout deadline and as the age every instance pod must exceed to count as rolled. Now that a failing write is retried for up to the backup deadline, the attach can come long after the latch: the deadline was then spent before the rollout began, and a pod recreated between the latch and the attach, without the plugin sidecar, passed as rolled, so a Backup was created that CloudNativePG fails for good. The latch now carries its own reason, and the write that finds the plugin missing and attaches it re-stamps the condition, once, with the time it was attempted. A plugin someone else attached after the latch, another backup job or a restore, keeps the latch time: the rollout it started can be over by the time this job writes, and its instances would otherwise never count as rolled. Signed-off-by: Alexey Artamonov Assisted-by: LLM --- .../cnpgstrategy_controller.go | 31 +++++ .../cnpgstrategy_controller_test.go | 109 ++++++++++++++++++ 2 files changed, 140 insertions(+) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index eec19819c..e1fe84f8a 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -197,6 +197,11 @@ const ( // instance pods must postdate (see cnpgPluginRolloutPending). backupCondPluginRolledOut = "PluginRolledOut" + // Reason of the PluginRolledOut latch recorded before the write that + // attaches the plugin. The write that attaches it replaces the latch with + // its own time, which retries can put long after the latch. + cnpgReasonAttachingPlugin = "AttachingPlugin" + // Condition Type recorded while writing the WAL archive destination keeps // failing. Its LastTransitionTime marks the first failure, so the retry // budget is not spent by time the job waited for another backup of the @@ -356,6 +361,7 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } } + attemptedAt := metav1.Now() attachment, err := r.applyClusterPluginBackup(ctx, j.Namespace, clusterName, rendered, serverName, existingBackup != nil || awaitingRollout, writeDestination) if err != nil { if apierrors.IsNotFound(err) { @@ -419,9 +425,34 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ Type: backupCondPluginRolledOut, Status: metav1.ConditionFalse, + Reason: cnpgReasonAttachingPlugin, + Message: message, + }) + apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ + Type: "Ready", + Status: metav1.ConditionFalse, Reason: "WaitingForPluginRollout", Message: message, }) + if err := r.Status().Update(ctx, j); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil + } + if rollout.Reason == cnpgReasonAttachingPlugin && attachment.missing { + // Instances recreated between the latch and this write run + // without the plugin, and the rollout deadline starts here. A + // plugin someone else attached since the latch keeps it: the + // rollout that attach started may already be over. + message := fmt.Sprintf("attached the barman-cloud plugin to cnpg.io/Cluster %s/%s, waiting for its instances to roll", j.Namespace, clusterName) + apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondPluginRolledOut) + apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ + Type: backupCondPluginRolledOut, + Status: metav1.ConditionFalse, + Reason: "WaitingForPluginRollout", + Message: message, + LastTransitionTime: attemptedAt, + }) apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ Type: "Ready", Status: metav1.ConditionFalse, diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index f4ec68fb9..b4b73b7e1 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2814,6 +2814,115 @@ func TestReconcileCNPG_PluginRolloutDeadlineCountsFromAttach(t *testing.T) { } } +// The rollout is measured from the write that attached the plugin, not from +// the latch recorded before it: retries of a failing write can lie between +// the two, and an instance recreated in that time runs without the plugin. +func TestReconcileCNPG_PluginRolloutCountsFromTheSuccessfulAttach(t *testing.T) { + cases := []struct { + name string + // podBetween recreates the instance after the latch and before the + // attach; otherwise it predates both. + podBetween bool + }{ + {name: "an instance recreated before the attach still has to roll", podBetween: true}, + {name: "retries before the attach do not spend the rollout deadline"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + expired := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)).Truncate(time.Second) + oldPod := newRolloutInstancePod(expired.Add(-time.Hour)) + f := newCNPGRolloutFixture(t, newRolloutBackupJob(expired.Add(-time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase), oldPod) + + j, _ := f.pass() // latch + latch := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) + if latch == nil { + t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) + } + latch.LastTransitionTime = metav1.NewTime(expired) + if err := f.c.Status().Update(context.Background(), j); err != nil { + t.Fatalf("age the latch: %v", err) + } + + failing := true + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + if _, ok := obj.(*cnpgtypes.Cluster); ok && failing { + return apierrors.NewInternalError(errors.New(`failed calling webhook "mcluster.cnpg.io"`)) + } + return cl.Patch(ctx, obj, patch, opts...) + }, + }) + if j, _ = f.pass(); j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("a failed attach failed the job: %s", j.Status.Message) + } + if tc.podBetween { + f.replacePod(oldPod, newRolloutInstancePod(expired.Add(time.Minute))) + } + + failing = false + attachAt := time.Now().Truncate(time.Second) + j, backups := f.pass() + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created in the pass that attached the plugin: %+v", backups) + } + rollout := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) + if rollout == nil || rollout.Status != metav1.ConditionFalse || rollout.LastTransitionTime.Time.Before(attachAt) { + t.Fatalf("%s = %+v, want False since the attach at %s", backupCondPluginRolledOut, rollout, attachAt) + } + + attached := &cnpgtypes.Cluster{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: rolloutCluster}, attached); err != nil { + t.Fatalf("get Cluster: %v", err) + } + f.observeCNPG(newRolloutCluster(cnpgClusterHealthyPhase, attached.Spec.Plugins...)) + j, backups = f.pass() + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("a rollout that began with the attach failed: %s", j.Status.Message) + } + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created against an instance older than the attach: %+v", backups) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || !strings.Contains(ready.Message, "predates") { + t.Errorf("Ready = %+v, want the instance that predates the plugin", ready) + } + }) + } +} + +// A plugin attached by someone else after the latch, another backup job or +// a restore, started the rollout before this job's write, which only finds +// it there: instances that rolled since the latch carry the plugin. +func TestReconcileCNPG_PluginAttachedByAnotherWriterKeepsTheLatch(t *testing.T) { + oldPod := newRolloutInstancePod(time.Now().Add(-time.Hour).Truncate(time.Second)) + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase), oldPod) + + j, _ := f.pass() // latch + latch := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) + if latch == nil { + t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) + } + latchedAt := time.Now().Add(-10 * time.Minute).Truncate(time.Second) + latch.LastTransitionTime = metav1.NewTime(latchedAt) + if err := f.c.Status().Update(context.Background(), j); err != nil { + t.Fatalf("age the latch: %v", err) + } + f.observeCNPG(newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster))) + f.replacePod(oldPod, newRolloutInstancePod(latchedAt.Add(5*time.Minute))) + + var backups []cnpgtypes.Backup + for range 3 { + if j, backups = f.pass(); len(backups) != 0 || j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + break + } + } + if len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups against instances that rolled with the plugin; Ready = %+v", + len(backups), apimeta.FindStatusCondition(j.Status.Conditions, "Ready")) + } +} + // Waiting for another backup resets the retry budget of the destination // write, so the queue does not spend it. func TestReconcileCNPG_QueueResetsDestinationRetryBudget(t *testing.T) { From 61d64a69f12e6712a5950c943a46dcc565191fd4 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:07:56 +0300 Subject: [PATCH 297/518] test(postgres): pin the requeue and the read-only pass of a running backup Three guards of the CNPG driver could be reverted with the package tests still green. The controller watches only BackupJobs, so a job waiting in the queue or retrying the destination write is woken by nothing but its own requeue, and the tests dropped the result. A job whose Backup runs must not consult the queue, or two jobs that passed it together wait on each other forever, and must patch neither the ObjectStore nor the Cluster plugin, yet only the ObjectStore content was checked. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller_test.go | 32 ++++++++++++++++--- 1 file changed, 27 insertions(+), 5 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index b4b73b7e1..e738b6c0e 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2561,6 +2561,17 @@ func (f *cnpgRolloutFixture) seedOtherBackupJob(name string, phase backupsv1alph func TestReconcileCNPG_RunningBackupDoesNotReapplyObjectStore(t *testing.T) { f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") f.seedCNPGBackup("bj-cnpg", "bj", rolloutCluster, "running") + // Two jobs can pass the queue together, since it reads the cache; once + // both have a Backup neither may wait on the other. + f.seedOtherBackupJob("bj-other", backupsv1alpha1.BackupJobPhaseRunning) + f.seedCNPGBackup("bj-other-cnpg", "bj-other", rolloutCluster, "running") + writes := 0 + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + writes++ + return cl.Patch(ctx, obj, patch, opts...) + }, + }) j, _ := f.pass() if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { @@ -2569,8 +2580,11 @@ func TestReconcileCNPG_RunningBackupDoesNotReapplyObjectStore(t *testing.T) { if got := f.liveDestination(); got != "s3://other/" { t.Fatalf("a pass over a running backup re-applied the ObjectStore: destination %q", got) } - if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForOtherBackup" { - t.Fatalf("a running backup waits on itself: %+v", ready) + if writes != 0 { + t.Errorf("a pass over a running backup patched %d object(s); the ObjectStore and the Cluster plugin are only read", writes) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForCNPGBackup" { + t.Fatalf("Ready = %+v, want the running backup followed rather than queued", ready) } } @@ -2624,7 +2638,7 @@ func TestReconcileCNPG_WaitsForAnotherRunningBackupOfTheApp(t *testing.T) { }, }) - j, backups := f.pass() + j, backups, res := f.passResult() if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { t.Fatalf("BackupJob failed: %s", j.Status.Message) } @@ -2645,8 +2659,13 @@ func TestReconcileCNPG_WaitsForAnotherRunningBackupOfTheApp(t *testing.T) { if ready == nil || ready.Status != metav1.ConditionFalse || ready.Reason != "WaitingForOtherBackup" || !strings.Contains(ready.Message, "bj-other") { t.Errorf("Ready = %+v, want False/WaitingForOtherBackup naming bj-other", ready) } + if res.RequeueAfter == 0 { + t.Errorf("a queued job is not requeued, and nothing else wakes it when the holder finishes: %+v", res) + } statusWrites = 0 - f.pass() + if _, _, res = f.passResult(); res.RequeueAfter == 0 { + t.Errorf("a queued job whose status did not change is not requeued: %+v", res) + } if statusWrites != 0 { t.Errorf("a second pass with the same holder wrote the BackupJob status %d time(s)", statusWrites) } @@ -2715,7 +2734,7 @@ func TestReconcileCNPG_ObjectStoreApplyErrors(t *testing.T) { }, }) - j, backups := f.pass() + j, backups, res := f.passResult() if j.Status.Phase != tc.wantPhase { t.Fatalf("BackupJob phase = %q (%s), want %q", j.Status.Phase, j.Status.Message, tc.wantPhase) } @@ -2727,6 +2746,9 @@ func TestReconcileCNPG_ObjectStoreApplyErrors(t *testing.T) { if ready == nil || ready.Reason != "ClusterPluginApplyRetrying" || !strings.Contains(ready.Message, "connection refused") { t.Errorf("Ready = %+v, want ClusterPluginApplyRetrying carrying the error", ready) } + if res.RequeueAfter == 0 { + t.Errorf("a retried write is not requeued: %+v", res) + } } }) } From 66268d8508271b82670db719ceb9e72240df04e7 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:08:10 +0300 Subject: [PATCH 298/518] refactor(postgres): drop a Ready reset every later path overwrites After a successful destination write each path of the CNPG driver sets or removes Ready later in the same pass, and writes the status after it, so clearing a stale WaitingForOtherBackup or ClusterPluginApplyRetrying reason there could never outlive the pass, and no test could pin it. Signed-off-by: Alexey Artamonov Assisted-by: LLM --- internal/backupcontroller/cnpgstrategy_controller.go | 4 ---- 1 file changed, 4 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index e1fe84f8a..821f82e07 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -404,10 +404,6 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } if writeDestination { apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondDestinationApplied) - if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready != nil && - (ready.Reason == "WaitingForOtherBackup" || ready.Reason == "ClusterPluginApplyRetrying") { - apimeta.RemoveStatusCondition(&j.Status.Conditions, "Ready") - } } if attachment.serverName != serverName { logger.Info("preserving the live Cluster's barman serverName over the strategy template's", From a02bff7fadba6bd00b9fb531a74d8bb2f1c4c9d9 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:21:59 +0300 Subject: [PATCH 299/518] docs(postgres): say what the CNPG backup deadlines are counted from The rollout deadline starts at the latch when another writer attached the plugin and at the write otherwise, and a wait for a restore resets the destination retry budget just as a wait for another backup does. The comments on the deadline and on the two conditions still described only the attach and the queue. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 821f82e07..1d8c6eebf 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -176,10 +176,11 @@ const ( cnpgRecoveryMaxInspectPods = 5 // Cap on the wall-clock time a BackupJob waits for the barman-cloud - // plugin rollout, counted from the attach, or retries writing the WAL - // archive destination, counted from the first failure since the job last - // left the queue, before the cnpg.io/Backup exists. A Backup that exists - // is not capped: CloudNativePG keeps a healthy one pending while another + // plugin rollout, counted from the PluginRolledOut condition, or retries + // writing the WAL archive destination, counted from the first failure + // since the last successful write or the last wait for another backup or + // a restore, before the cnpg.io/Backup exists. A Backup that exists is + // not capped: CloudNativePG keeps a healthy one pending while another // backup of the same Cluster runs, or while it restarts one on another // instance, so no fixed deadline tells those apart from a stuck one. cnpgDefaultBackupDeadline = 30 * time.Minute @@ -193,8 +194,8 @@ const ( // instance, and a cnpg.io/Backup created meanwhile fails with "requested // plugin is not available" or "instance manager was restarted during // backup" and never recovers. False holds the Backup back; its - // LastTransitionTime, written before the plugin is attached, is what the - // instance pods must postdate (see cnpgPluginRolloutPending). + // LastTransitionTime is what the instance pods must postdate (see + // cnpgReasonAttachingPlugin and cnpgPluginRolloutPending). backupCondPluginRolledOut = "PluginRolledOut" // Reason of the PluginRolledOut latch recorded before the write that @@ -205,7 +206,7 @@ const ( // Condition Type recorded while writing the WAL archive destination keeps // failing. Its LastTransitionTime marks the first failure, so the retry // budget is not spent by time the job waited for another backup of the - // app before it tried. + // app, or for a restore of it, before it tried. backupCondDestinationApplied = "DestinationApplied" cnpgPodRoleLabel = "cnpg.io/podRole" @@ -576,9 +577,8 @@ func cnpgBackupFailedMessage(b *cnpgtypes.Backup, errText string) string { // cnpgBackupDeadlineExceeded reports whether more than the backup deadline // has passed since the BackupJob began the step it is waiting on: the plugin -// rollout, counted from the attach, or retrying the destination write, -// counted from the first failure since the job last left the queue behind -// another backup. Returns false for a nil time. +// rollout or retrying the destination write (see cnpgDefaultBackupDeadline). +// Returns false for a nil time. func cnpgBackupDeadlineExceeded(since *metav1.Time) bool { if since == nil { return false From 93c3ee62c45d33c2e8b31857f81624cd08e59d99 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 22:15:46 +0300 Subject: [PATCH 300/518] test(postgres): pin which holder a queued backup job names When two jobs of the application hold the queue, the job waiting behind them names one of them in its Ready message, and the sort is what keeps that name stable however the cache orders their Backups. Without it the message can change from one poll to the next, and each change writes the BackupJob status again; nothing failed when the sort was removed. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller_test.go | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index e738b6c0e..6240500e5 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -3141,6 +3141,34 @@ func TestReconcileCNPG_RestoreResetsDestinationRetryBudget(t *testing.T) { } } +// With two jobs holding the queue the wait names the same one on every pass, +// whatever order the cache lists their Backups in, so a waiting job does not +// rewrite its status each poll. +func TestReconcileCNPG_QueueNamesOneHolderAcrossPasses(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f.seedOtherBackupJob("holder-z", backupsv1alpha1.BackupJobPhaseRunning) + f.seedOtherBackupJob("holder-a", backupsv1alpha1.BackupJobPhaseRunning) + f.seedCNPGBackup("a-cnpg", "holder-z", rolloutCluster, "running") + f.seedCNPGBackup("z-cnpg", "holder-a", rolloutCluster, "running") + statusWrites := 0 + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + SubResourceUpdate: func(ctx context.Context, cl client.Client, sub string, obj client.Object, opts ...client.SubResourceUpdateOption) error { + statusWrites++ + return cl.SubResource(sub).Update(ctx, obj, opts...) + }, + }) + + j, _ := f.pass() + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForOtherBackup" || !strings.Contains(ready.Message, "holder-a ") { + t.Fatalf("Ready = %+v, want WaitingForOtherBackup naming holder-a", ready) + } + statusWrites = 0 + f.pass() + if statusWrites != 0 { + t.Errorf("a second pass with the same two holders wrote the BackupJob status %d time(s)", statusWrites) + } +} + func TestOtherCNPGBackupInFlight_IgnoresTheJobsOwnBackup(t *testing.T) { f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") f.seedCNPGBackup("bj-cnpg", "bj", rolloutCluster, "running") From 7efe33e87de1e3683e1e4378ba0843846151922b Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 21:05:32 +0300 Subject: [PATCH 301/518] test(postgres): check the stored phase of a backup job per cnpg Backup phase The phase test read the BackupJob it passed to the reconciler, which holds the phase in memory whether or not the status write happened. A completed Backup could then leave the stored BackupJob Running, so the Plan would never see the run finish, and no test in the package noticed. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/cnpgstrategy_controller_test.go | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 6240500e5..a5d1d5b44 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2421,6 +2421,13 @@ func TestReconcileCNPG_BackupPhaseOutcome(t *testing.T) { if j.Status.Phase != tc.wantPhase { t.Fatalf("BackupJob phase = %q (%s), want %q", j.Status.Phase, j.Status.Message, tc.wantPhase) } + stored := &backupsv1alpha1.BackupJob{} + if err := c.Get(context.Background(), client.ObjectKeyFromObject(job), stored); err != nil { + t.Fatalf("get BackupJob after reconcile: %v", err) + } + if stored.Status.Phase != tc.wantPhase { + t.Fatalf("stored BackupJob phase = %q, want %q", stored.Status.Phase, tc.wantPhase) + } message := j.Status.Message if tc.wantPhase == backupsv1alpha1.BackupJobPhaseRunning { From 8061e2dbd9c4f875d2e620847aaa10a005322d4a Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 5 Nov 2026 22:13:55 +0300 Subject: [PATCH 302/518] test(postgres): describe what the backup deadline test still guards The deadline no longer applies to a failed cnpg.io/Backup, which now fails its BackupJob at once; it only bounds the wait for the barman-cloud plugin rollout, and the comment still claimed the former. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/cnpgstrategy_controller_test.go | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index a5d1d5b44..65f979afb 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -1000,10 +1000,9 @@ func TestUnmarshalCNPGBackupSnapshot_RejectsMissingOrWrongKind(t *testing.T) { } } -// TestCNPGBackupDeadlineExceeded locks in the wall-clock cap that prevents a -// perpetually-failed cnpg.io/Backup from pinning the BackupJob in Running -// forever. Without this guard the Plan-controller queue would back up -// behind a single broken backup. +// TestCNPGBackupDeadlineExceeded locks in the wall-clock cap on waiting for +// the barman-cloud plugin rollout, so a rollout that never finishes fails the +// BackupJob instead of pinning it in Running. func TestCNPGBackupDeadlineExceeded(t *testing.T) { cases := []struct { name string From 32ca678077c3437b0ebb646b28e2fcfa719afff3 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 15:58:37 +0300 Subject: [PATCH 303/518] fix(postgres): give a held backup job its plugin rollout deadline back A job that latched the barman-cloud plugin rollout and then waited for another backup of the application, or for a restore of it, had that wait charged to the rollout deadline. Released after more than the deadline, it failed on the first instance that was not ready yet with "rollout did not finish", although no rollout of its own had been running while it waited. The hold is now recorded in a DestinationWriteHeld condition, and its release time restarts the rollout deadline when it is later than the rollout's start. The instance pods are still measured against the rollout's start: one that rolled while the job was held carries the plugin and counts. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 36 +++- .../cnpgstrategy_controller_test.go | 171 ++++++++++++++++++ 2 files changed, 205 insertions(+), 2 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 1d8c6eebf..fc3fded4c 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -176,7 +176,8 @@ const ( cnpgRecoveryMaxInspectPods = 5 // Cap on the wall-clock time a BackupJob waits for the barman-cloud - // plugin rollout, counted from the PluginRolledOut condition, or retries + // plugin rollout, counted from the PluginRolledOut condition or from the + // release recorded in DestinationWriteHeld, whichever is later, or retries // writing the WAL archive destination, counted from the first failure // since the last successful write or the last wait for another backup or // a restore, before the cnpg.io/Backup exists. A Backup that exists is @@ -209,6 +210,13 @@ const ( // app, or for a restore of it, before it tried. backupCondDestinationApplied = "DestinationApplied" + // Condition Type recorded while the job is held off writing the WAL + // archive destination, by another backup of the app in flight or by a + // restore of it, and set False when released. The release time restarts + // the plugin rollout deadline, which that wait must not spend; the + // instance pods still have to postdate the rollout's own start. + backupCondDestinationWriteHeld = "DestinationWriteHeld" + cnpgPodRoleLabel = "cnpg.io/podRole" cnpgPodRoleInstance = "instance" ) @@ -349,6 +357,7 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } if restore != "" { apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondDestinationApplied) + apimeta.SetStatusCondition(&j.Status.Conditions, cnpgDestinationWriteHeld(ConditionReasonRestoreInProgress)) return r.holdForRestore(ctx, j, restore) } holder, err := r.otherCNPGBackupInFlight(ctx, j, clusterName) @@ -357,9 +366,18 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } if holder != "" { apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondDestinationApplied) + apimeta.SetStatusCondition(&j.Status.Conditions, cnpgDestinationWriteHeld("WaitingForOtherBackup")) return r.waitCNPGBackupJob(ctx, j, "WaitingForOtherBackup", fmt.Sprintf( "waiting for BackupJob %s of the same application to finish its backup, which writes to the shared WAL archive destination", holder)) } + if apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationWriteHeld) != nil { + apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ + Type: backupCondDestinationWriteHeld, + Status: metav1.ConditionFalse, + Reason: "Released", + Message: "nothing holds the WAL archive destination write any more", + }) + } } attemptedAt := metav1.Now() @@ -466,7 +484,12 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp return ctrl.Result{}, err } if pending != "" { - if cnpgBackupDeadlineExceeded(&rollout.LastTransitionTime) { + deadlineFrom := rollout.LastTransitionTime + if held := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationWriteHeld); held != nil && + held.LastTransitionTime.After(deadlineFrom.Time) { + deadlineFrom = held.LastTransitionTime + } + if cnpgBackupDeadlineExceeded(&deadlineFrom) { return r.markBackupJobFailed(ctx, j, fmt.Sprintf( "barman-cloud plugin rollout did not finish within %s: %s", cnpgDefaultBackupDeadline, pending)) } @@ -933,6 +956,15 @@ func (r *BackupJobReconciler) otherCNPGBackupInFlight(ctx context.Context, j *ba return holders[0], nil } +func cnpgDestinationWriteHeld(reason string) metav1.Condition { + return metav1.Condition{ + Type: backupCondDestinationWriteHeld, + Status: metav1.ConditionTrue, + Reason: reason, + Message: "held off writing the WAL archive destination", + } +} + // waitCNPGBackupJob requeues the BackupJob and records why it waits in its // Ready condition, writing the status only when the conditions changed. func (r *BackupJobReconciler) waitCNPGBackupJob(ctx context.Context, j *backupsv1alpha1.BackupJob, reason, message string) (ctrl.Result, error) { diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 65f979afb..1f0b88312 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -3447,6 +3447,41 @@ func TestReconcileCNPG_AttachedPluginDoesNotDelayBackup(t *testing.T) { if cond := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut); cond != nil { t.Errorf("%s condition = %+v on a Cluster the driver did not roll", backupCondPluginRolledOut, cond) } + if cond := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationWriteHeld); cond != nil { + t.Errorf("%s condition = %+v on a job nothing held", backupCondDestinationWriteHeld, cond) + } +} + +// A release older than the rollout, from a wait before the plugin was +// attached, does not shorten the rollout that began after it. +func TestReconcileCNPG_EarlierReleaseDoesNotShortenTheRollout(t *testing.T) { + expired := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)) + j := newRolloutBackupJob(expired.Add(-time.Hour)) + j.Status.Conditions = []metav1.Condition{ + { + Type: backupCondDestinationWriteHeld, + Status: metav1.ConditionFalse, + Reason: "Released", + LastTransitionTime: metav1.NewTime(expired), + }, + { + Type: backupCondPluginRolledOut, + Status: metav1.ConditionFalse, + Reason: "WaitingForPluginRollout", + LastTransitionTime: metav1.NewTime(time.Now().Add(-time.Minute)), + }, + } + f := newCNPGRolloutFixture(t, j, + newRolloutCluster("Waiting for user action", buildBarmanPlugin(rolloutCluster, rolloutCluster)), + newRolloutInstancePod(expired.Add(-time.Hour))) + + got, backups := f.pass() + if got.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("a rollout that began a minute ago failed on an older release: %s", got.Status.Message) + } + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created against an unfinished rollout: %+v", backups) + } } // TestReconcileCNPG_PluginRolloutWaitIsBounded: a rollout that never ends @@ -3476,6 +3511,142 @@ func TestReconcileCNPG_PluginRolloutWaitIsBounded(t *testing.T) { } } +// A job whose plugin rollout started, and which then waited for another +// backup of the app or for a restore of it, gets the rollout deadline back +// from the moment it was released: that wait is not a rollout of its own. +// The instance pods still count as rolled from the rollout's own start. +func TestReconcileCNPG_HeldJobRolloutDeadlineCountsFromRelease(t *testing.T) { + type hold struct { + name string + seed func(f *cnpgRolloutFixture) (release func()) + waiting string + } + holds := []hold{ + { + name: "another backup", + waiting: "WaitingForOtherBackup", + seed: func(f *cnpgRolloutFixture) func() { + f.seedOtherBackupJob("bj-other", backupsv1alpha1.BackupJobPhaseRunning) + f.seedCNPGBackup("bj-other-cnpg", "bj-other", rolloutCluster, "running") + return func() { + held := &cnpgtypes.Backup{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj-other-cnpg"}, held); err != nil { + f.t.Fatalf("get holder Backup: %v", err) + } + held.Status.Phase = cnpgBackupPhaseComplete + if err := f.c.Update(context.Background(), held); err != nil { + f.t.Fatalf("complete holder Backup: %v", err) + } + } + }, + }, + { + name: "a restore", + waiting: ConditionReasonRestoreInProgress, + seed: func(f *cnpgRolloutFixture) func() { + restore := f.seedRestoreOfApp() + return func() { + restore.Status.Phase = backupsv1alpha1.RestoreJobPhaseSucceeded + if err := f.c.Status().Update(context.Background(), restore); err != nil { + f.t.Fatalf("finish the restore: %v", err) + } + } + }, + }, + } + stampedAt := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)).Truncate(time.Second) + newHeldRollout := func(t *testing.T, h hold, cluster *cnpgtypes.Cluster, pod *corev1.Pod) *cnpgRolloutFixture { + t.Helper() + j := newRolloutBackupJob(stampedAt.Add(-time.Hour)) + j.Status.Conditions = []metav1.Condition{{ + Type: backupCondPluginRolledOut, + Status: metav1.ConditionFalse, + Reason: "WaitingForPluginRollout", + LastTransitionTime: metav1.NewTime(stampedAt), + }} + f := newCNPGRolloutFixture(t, j, cluster, pod) + release := h.seed(f) + got, backups := f.pass() + for _, b := range backups { + if b.Labels[backupsv1alpha1.OwningJobNameLabel] == "bj" { + t.Fatalf("a held job created cnpg.io/Backup %s", b.Name) + } + } + if got.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("a held job did not wait: %s", got.Status.Message) + } + if ready := apimeta.FindStatusCondition(got.Status.Conditions, "Ready"); ready == nil || ready.Reason != h.waiting { + t.Fatalf("Ready = %+v, want %s", ready, h.waiting) + } + release() + return f + } + + for _, h := range holds { + t.Run(h.name+": the rollout is still waited for after the release", func(t *testing.T) { + f := newHeldRollout(t, h, + newRolloutCluster("Waiting for user action", buildBarmanPlugin(rolloutCluster, rolloutCluster)), + newRolloutInstancePod(stampedAt.Add(-time.Hour))) + + releasedAt := time.Now().Truncate(time.Second) + j, backups := f.pass() + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("time held by %s was charged to the rollout: %s", h.name, j.Status.Message) + } + for _, b := range backups { + if b.Labels[backupsv1alpha1.OwningJobNameLabel] == "bj" { + t.Fatalf("cnpg.io/Backup %s was created against an unfinished rollout", b.Name) + } + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForPluginRollout" { + t.Fatalf("Ready = %+v, want WaitingForPluginRollout", ready) + } + released := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationWriteHeld) + if released == nil || released.Status != metav1.ConditionFalse || released.LastTransitionTime.Time.Before(releasedAt) { + t.Fatalf("%s = %+v, want False since the release at %s", backupCondDestinationWriteHeld, released, releasedAt) + } + if rollout := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut); rollout == nil || !rollout.LastTransitionTime.Time.Equal(stampedAt) { + t.Errorf("%s = %+v, want its stamp kept at %s", backupCondPluginRolledOut, rollout, stampedAt) + } + + released.LastTransitionTime = metav1.NewTime(stampedAt) + if err := f.c.Status().Update(context.Background(), j); err != nil { + t.Fatalf("age the release: %v", err) + } + j, _ = f.pass() + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed || !strings.Contains(j.Status.Message, "rollout did not finish") { + t.Fatalf("phase = %q (%s), want Failed once the rollout outlived the deadline after the release", j.Status.Phase, j.Status.Message) + } + }) + + t.Run(h.name+": an instance that rolled while the job was held counts", func(t *testing.T) { + f := newHeldRollout(t, h, + newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)), + newRolloutInstancePod(stampedAt.Add(time.Minute))) + + var j *backupsv1alpha1.BackupJob + own := 0 + for range 3 { + var backups []cnpgtypes.Backup + j, backups = f.pass() + own = 0 + for _, b := range backups { + if b.Labels[backupsv1alpha1.OwningJobNameLabel] == "bj" { + own++ + } + } + if own != 0 || j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + break + } + } + if own != 1 { + t.Fatalf("got %d cnpg.io/Backups of the job against an instance that rolled after the rollout began; Ready = %+v, phase %q (%s)", + own, apimeta.FindStatusCondition(j.Status.Conditions, "Ready"), j.Status.Phase, j.Status.Message) + } + }) + } +} + // TestReconcileCNPGRestore_MissingStrategyFailsClosedAfterDeadline covers the // restore-path blocker: a RestoreJob whose referenced CNPG Strategy CR never // appears must NOT requeue forever. requeueRestoreStrategyNotReady gates From 258c5359b606897d144248500dc2c752d2498756 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Fri, 6 Nov 2026 16:08:26 +0300 Subject: [PATCH 304/518] fix(postgres): restart the plugin rollout whenever a backup job attaches it Only the first attach after the latch restarted the rollout. In the platform flow the chart renders no spec.plugins, so an in-place restore recreates the Cluster without the plugin, and a backup job held through that restore attaches it again once released. The rollout condition then still carried its earlier start, the restored instances were newer than it and ready, and the Backup was created in the same pass, before CloudNativePG had rolled anything; such a Backup fails with "requested plugin is not available" or "instance manager was restarted". Every write of the job that finds the plugin missing now starts the rollout over. A plugin attached by another writer is still timed from the latch. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 25 ++++++----- .../cnpgstrategy_controller_test.go | 45 ++++++++++++++++++- 2 files changed, 57 insertions(+), 13 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index fc3fded4c..83ce9b41e 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -175,12 +175,13 @@ const ( // against an unreachable target reproduces the same FATAL. cnpgRecoveryMaxInspectPods = 5 - // Cap on the wall-clock time a BackupJob waits for the barman-cloud - // plugin rollout, counted from the PluginRolledOut condition or from the - // release recorded in DestinationWriteHeld, whichever is later, or retries - // writing the WAL archive destination, counted from the first failure - // since the last successful write or the last wait for another backup or - // a restore, before the cnpg.io/Backup exists. A Backup that exists is + // Cap on the wall-clock time a BackupJob spends before its + // cnpg.io/Backup exists on each of two waits. The plugin rollout is + // counted from the PluginRolledOut condition, or from the release + // recorded in DestinationWriteHeld when that is later. Retrying the WAL + // archive destination write is counted from the first failure since the + // last successful write or the last wait for another backup or a + // restore. A Backup that exists is // not capped: CloudNativePG keeps a healthy one pending while another // backup of the same Cluster runs, or while it restarts one on another // instance, so no fixed deadline tells those apart from a stuck one. @@ -200,8 +201,9 @@ const ( backupCondPluginRolledOut = "PluginRolledOut" // Reason of the PluginRolledOut latch recorded before the write that - // attaches the plugin. The write that attaches it replaces the latch with - // its own time, which retries can put long after the latch. + // attaches the plugin. Every write of this job that attaches it replaces + // the condition with its own time, which retries can put long after the + // latch. cnpgReasonAttachingPlugin = "AttachingPlugin" // Condition Type recorded while writing the WAL archive destination keeps @@ -454,9 +456,10 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil } - if rollout.Reason == cnpgReasonAttachingPlugin && attachment.missing { - // Instances recreated between the latch and this write run - // without the plugin, and the rollout deadline starts here. A + if attachment.missing { + // This write attached the plugin, so its rollout starts here: + // instances recreated since the latch, or by a restore that + // re-rendered the Cluster without the plugin, run without it. A // plugin someone else attached since the latch keeps it: the // rollout that attach started may already be over. message := fmt.Sprintf("attached the barman-cloud plugin to cnpg.io/Cluster %s/%s, waiting for its instances to roll", j.Namespace, clusterName) diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 1f0b88312..8f0035aa6 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2830,6 +2830,11 @@ func TestReconcileCNPG_PluginRolloutDeadlineCountsFromAttach(t *testing.T) { if cond := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); cond == nil || cond.Reason != "WaitingForPluginRollout" { t.Fatalf("Ready = %+v, want WaitingForPluginRollout", cond) } + attached := &cnpgtypes.Cluster{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: rolloutCluster}, attached); err != nil { + t.Fatalf("get Cluster: %v", err) + } + f.observeCNPG(newRolloutCluster(cnpgClusterHealthyPhase, attached.Spec.Plugins...)) rollout := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) rollout.LastTransitionTime = metav1.NewTime(expired) @@ -2919,8 +2924,9 @@ func TestReconcileCNPG_PluginRolloutCountsFromTheSuccessfulAttach(t *testing.T) } // A plugin attached by someone else after the latch, another backup job or -// a restore, started the rollout before this job's write, which only finds -// it there: instances that rolled since the latch carry the plugin. +// the chart re-rendering a restored Cluster outside the platform flow, +// started the rollout before this job's write, which only finds it there: +// instances that rolled since the latch carry the plugin. func TestReconcileCNPG_PluginAttachedByAnotherWriterKeepsTheLatch(t *testing.T) { oldPod := newRolloutInstancePod(time.Now().Add(-time.Hour).Truncate(time.Second)) f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), @@ -3452,6 +3458,41 @@ func TestReconcileCNPG_AttachedPluginDoesNotDelayBackup(t *testing.T) { } } +// An in-place restore of the platform flow recreates the Cluster without the +// barman-cloud plugin, so a job held through it attaches the plugin again +// once released. That attach starts a rollout of its own, which the +// instances restored after the job's first rollout began have not done. +func TestReconcileCNPG_ReattachAfterRestoreRestartsTheRollout(t *testing.T) { + stampedAt := time.Now().Add(-10 * time.Minute).Truncate(time.Second) + j := newRolloutBackupJob(stampedAt.Add(-time.Hour)) + j.Status.Conditions = []metav1.Condition{{ + Type: backupCondPluginRolledOut, + Status: metav1.ConditionFalse, + Reason: "WaitingForPluginRollout", + LastTransitionTime: metav1.NewTime(stampedAt), + }} + f := newCNPGRolloutFixture(t, j, newRolloutCluster(cnpgClusterHealthyPhase), + newRolloutInstancePod(time.Now().Add(-2*time.Minute))) + restore := f.seedRestoreOfApp() + if got, _ := f.pass(); apimeta.FindStatusCondition(got.Status.Conditions, "Ready").Reason != ConditionReasonRestoreInProgress { + t.Fatalf("Ready = %+v, want the job held by the restore", apimeta.FindStatusCondition(got.Status.Conditions, "Ready")) + } + restore.Status.Phase = backupsv1alpha1.RestoreJobPhaseSucceeded + if err := f.c.Status().Update(context.Background(), restore); err != nil { + t.Fatalf("finish the restore: %v", err) + } + + attachAt := time.Now().Truncate(time.Second) + got, backups := f.pass() + if len(backups) != 0 { + t.Fatalf("cnpg.io/Backup created in the pass that attached the plugin to the restored Cluster: %+v", backups) + } + rollout := apimeta.FindStatusCondition(got.Status.Conditions, backupCondPluginRolledOut) + if rollout == nil || rollout.Status != metav1.ConditionFalse || rollout.LastTransitionTime.Time.Before(attachAt) { + t.Fatalf("%s = %+v, want False since the attach at %s", backupCondPluginRolledOut, rollout, attachAt) + } +} + // A release older than the rollout, from a wait before the plugin was // attached, does not shorten the rollout that began after it. func TestReconcileCNPG_EarlierReleaseDoesNotShortenTheRollout(t *testing.T) { From 4eff1617e52f32780699abb6af7667eef81141f7 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 03:37:30 +0700 Subject: [PATCH 305/518] fix(observability-proxy): bound the token and access reviews Both reviews ran on the request context, which net/http cancels for a dropped client only once the body is read. A POST whose body was still unread therefore waited on a stalled kube-apiserver past every request limit. The reviews now share the request timeout and answer 503 when it passes. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 13 ++++- .../server_deadline_test.go | 58 +++++++++++++++++++ 2 files changed, 69 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 511499933..a749f5526 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -331,7 +331,16 @@ func (s *Server) authorize(r *http.Request, route Route) (string, *HTTPError) { if !strings.EqualFold(scheme, "Bearer") { return "", errorf(http.StatusUnauthorized, "missing bearer token") } - user, err := s.auth.Authenticate(r.Context(), strings.TrimSpace(token)) + // The kube client sets no timeout of its own, and net/http cancels + // r.Context on a dropped client only once the body is read, so a + // stalled review would otherwise outlive every other limit. + ctx := r.Context() + if s.cfg.RequestTimeout > 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, s.cfg.RequestTimeout) + defer cancel() + } + user, err := s.auth.Authenticate(ctx, strings.TrimSpace(token)) if errors.Is(err, ErrUnauthenticated) { return "", errorf(http.StatusUnauthorized, "invalid bearer token") } @@ -339,7 +348,7 @@ func (s *Server) authorize(r *http.Request, route Route) (string, *HTTPError) { slog.Error("token review failed", "error", err) return "", errorf(http.StatusServiceUnavailable, "authentication unavailable") } - allowed, err := s.auth.Authorize(r.Context(), user, route.ResourceAttributes()) + allowed, err := s.auth.Authorize(ctx, user, route.ResourceAttributes()) if err != nil { slog.Error("subject access review failed", "error", err) return "", errorf(http.StatusServiceUnavailable, "authorization unavailable") diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index 32ab93773..7415b46fa 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -22,11 +22,16 @@ import ( "io" "net" "net/http" + "net/http/httptest" + "net/url" "strings" "sync" "sync/atomic" "testing" "time" + + "k8s.io/client-go/kubernetes" + "k8s.io/client-go/rest" ) // pipeListener hands the server one end of a net.Pipe per dial. A pipe has @@ -386,3 +391,56 @@ func TestServerEarlyRefusalWithUnreadBody(t *testing.T) { } } } + +// stalledAPIServer answers the TokenReview for "good" unless stallTokens +// is set, and never answers a SubjectAccessReview: it holds each stalled +// call until the caller gives up or the test ends. +func stalledAPIServer(t *testing.T, stallTokens bool) kubernetes.Interface { + t.Helper() + release := make(chan struct{}) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if strings.HasSuffix(r.URL.Path, "/tokenreviews") && !stallTokens { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"apiVersion":"authentication.k8s.io/v1","kind":"TokenReview","status":{"authenticated":true,"user":{"username":"alice"}}}`) + return + } + select { + case <-r.Context().Done(): + case <-release: + } + })) + t.Cleanup(srv.Close) + // After srv, so it runs first and srv.Close does not wait on a stalled call. + t.Cleanup(func() { close(release) }) + return kubernetes.NewForConfigOrDie(&rest.Config{Host: srv.URL}) +} + +// TestServerStalledReviewAnswersInTime holds TokenReview or +// SubjectAccessReview open on a POST whose body is complete but unread: +// net/http does not watch the connection until the body is read, so only +// a bounded review call ends the request. +func TestServerStalledReviewAnswersInTime(t *testing.T) { + for _, tc := range []struct { + name string + stallTokens bool + }{{"token review", true}, {"subject access review", false}} { + t.Run(tc.name, func(t *testing.T) { + h := newHarness(t, Config{RequestTimeout: 200 * time.Millisecond}, nil) + h.server.auth = NewAuthorizer(stalledAPIServer(t, tc.stallTokens), 10*time.Second) + got := make(chan int, 1) + go func() { + resp := h.do(http.MethodPost, logsRoot+"query", "good", url.Values{"query": {"*"}}, nil) + _ = resp.Body.Close() + got <- resp.StatusCode + }() + select { + case code := <-got: + if code != http.StatusServiceUnavailable { + t.Fatalf("stalled %s answered %d, want 503", tc.name, code) + } + case <-time.After(2 * time.Second): + t.Fatalf("a stalled %s holds the request past ten request timeouts", tc.name) + } + }) + } +} From d2ef5e031f65440c29d8bac8a1e89cb1b8589984 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 03:37:39 +0700 Subject: [PATCH 306/518] docs(observability-proxy): describe the proxy as it stands The package and serve comments described a delivery plan rather than the code: the command named metrics it does not serve, and serve gave a plan as the reason to refuse unknown roots before authenticating. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 7 +++---- internal/observabilityproxy/server.go | 4 ++-- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index f8ccd38be..a2e1ef92d 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -14,10 +14,9 @@ See the License for the specific language governing permissions and limitations under the License. */ -// Command observability-proxy serves tenant logs (and, in later waves, -// metrics) from the VictoriaLogs and VictoriaMetrics stores behind a -// Kubernetes TokenReview and SubjectAccessReview, with the tenant's -// namespace scope forced into every upstream request. +// Command observability-proxy serves tenant logs from the VictoriaLogs +// stores behind a Kubernetes TokenReview and SubjectAccessReview, with the +// tenant's namespace scope forced into every upstream request. package main import ( diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index a749f5526..e980818d9 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -164,8 +164,8 @@ func writeError(w http.ResponseWriter, e *HTTPError) { http.Error(w, e.Message, e.Code) } -// serve answers an unimplemented root before authenticating, so later -// waves only add a case here. +// serve answers a root it has no handler for before authenticating, so an +// unknown path costs no review calls. func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTTPError { if route.Kind != NamespaceLogs { return errorf(http.StatusNotFound, "not found") From 59cbf4b28e7349442c177cf71f10382613cc3732 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 03:44:46 +0700 Subject: [PATCH 307/518] docs(observability-proxy): say what the request timeout bounds The flag now also bounds the body read and the token and access reviews, and 0 lifts all of those at once; an operator raising it for slow store queries needs to know that. The store comment named a design document the repository does not have. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 2 +- internal/observabilityproxy/store.go | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index a2e1ef92d..558908671 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -69,7 +69,7 @@ func parseFlags(args []string) (options, error) { fs.StringVar(&o.rootTenant, "root-tenant", "tenant-root", "tenant namespace hosting the default store") fs.StringVar(&o.kubeconfig, "kubeconfig", "", "kubeconfig path; in-cluster config when empty") fs.DurationVar(&o.authCacheTTL, "auth-cache-ttl", 10*time.Second, "how long TokenReview and SubjectAccessReview answers are cached") - fs.DurationVar(&o.requestTimeout, "request-timeout", time.Minute, "upstream deadline for every endpoint but tail") + fs.DurationVar(&o.requestTimeout, "request-timeout", time.Minute, "deadline for reading the request body, the token and access reviews, and every upstream endpoint but tail; 0 disables it") fs.DurationVar(&o.tailIdleTimeout, "tail-idle-timeout", 5*time.Minute, "close a tail after this long without a line") fs.DurationVar(&o.shutdownTimeout, "shutdown-timeout", 15*time.Second, "grace period for in-flight requests on shutdown") fs.DurationVar(&o.logs.DefaultRange, "logs-default-range", time.Hour, "window used when a logs request has no start") diff --git a/internal/observabilityproxy/store.go b/internal/observabilityproxy/store.go index b8881e3ec..f24f0b17b 100644 --- a/internal/observabilityproxy/store.go +++ b/internal/observabilityproxy/store.go @@ -48,7 +48,7 @@ func NewStoreResolver(namespaces corev1listers.NamespaceLister, synced func() bo // Resolve answers the root tenant for a namespace that no longer exists, // because its label went with it. Lines it wrote while routed to another // tenant's store are not found there; reading them needs routing history -// (an open question of the design). +// the proxy does not keep. func (r *StoreResolver) Resolve(namespace string) (string, error) { // Before the first list every namespace looks gone, which would send // a labeled one to the root store; readiness alone does not stop a From 4f3260e5d432410ca8a3397d16af189e255faab4 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 04:58:49 +0700 Subject: [PATCH 308/518] feat(observability-proxy): rebuild metrics queries from an allowlist vmselect ORs repeated extra_filters, so a client-supplied value would widen the tenant scope. The upstream request is therefore built from scratch: only query, start, end, step, time, match[] and a clamped timeout are read from the client, and the namespace selector is sent as the single extra_filters[] value. The query itself is not parsed; vmselect ANDs that filter into every selector. The label endpoints always carry a match[] (the namespace selector when the client sent none), because a vmselect started with -search.ignoreExtraFiltersAtLabelsAPI applies extra filters there only when a match[] is present. Range, points per series and step are bounded with 422 per the cost limits of the metrics proposal. Times are accepted only in the two forms the Prometheus API defines, since vmselect guesses the unit of a numeric timestamp from its digit count. The window start picks the short-term store when it lies within that store's retention. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricsparams.go | 284 ++++++++++++++++++ .../observabilityproxy/metricsparams_test.go | 246 +++++++++++++++ 2 files changed, 530 insertions(+) create mode 100644 internal/observabilityproxy/metricsparams.go create mode 100644 internal/observabilityproxy/metricsparams_test.go diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go new file mode 100644 index 000000000..0e24b6597 --- /dev/null +++ b/internal/observabilityproxy/metricsparams.go @@ -0,0 +1,284 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "fmt" + "net/http" + "net/url" + "regexp" + "strconv" + "strings" + "time" +) + +// MetricsLimits bound what one metrics request may make vmselect do. +type MetricsLimits struct { + MaxRange time.Duration + MaxPoints int + MaxTimeout time.Duration +} + +// MetricsConfig names the vmselect stores of the root tenant and the +// retention that decides between them. +type MetricsConfig struct { + RootTenant string + ShortTermStore string + LongTermStore string + ShortTermRetention time.Duration + Limits MetricsLimits + // UpstreamURL maps a store name to its Prometheus API base URL, without + // a trailing slash; nil means the in-cluster vmselect Service. + UpstreamURL func(store string) string +} + +// store picks the short-term store when everything from start on is still +// within its retention, so a live graph never waits on the downsampled one. +func (c MetricsConfig) store(start, now time.Time) string { + if now.Sub(start) <= c.ShortTermRetention { + return c.ShortTermStore + } + return c.LongTermStore +} + +// maxMetricsMatchers bounds match[]: vmselect runs one index search per +// matcher and has no cap of its own. +const maxMetricsMatchers = 20 + +// labelsDefaultWindow is vmselect's own default for the label and series +// APIs (defaultStep in app/vmselect/prometheus); the proxy sends it +// explicitly so the store choice and the range check see the same window. +const labelsDefaultWindow = 5 * time.Minute + +var labelName = regexp.MustCompile(`^[a-zA-Z_][a-zA-Z0-9_]*$`) + +type metricsRequest struct { + path string + params url.Values + // start is the earliest instant the request reads, for the store choice. + start time.Time +} + +// namespaceSelector quotes ns the way MetricsQL unquotes a label value +// (strconv.Unquote in metricsql's extractStringValue). +func namespaceSelector(ns string) string { + return "{namespace=" + strconv.Quote(ns) + "}" +} + +// BuildMetricsParams rebuilds the upstream parameters from an allowlist and +// adds the namespace scope as the only extra_filters[] value. vmselect ORs +// repeated extra_filters, so a client value would widen the scope; it ANDs +// the single one into every selector of the query and every match[], which +// is what lets the query pass through unparsed. +func BuildMetricsParams(endpoint string, in url.Values, namespace string, now time.Time, lim MetricsLimits) (metricsRequest, *HTTPError) { + req := metricsRequest{path: endpoint, params: url.Values{}} + out := req.params + out.Set("extra_filters[]", namespaceSelector(namespace)) + timeout := lim.MaxTimeout + if v := in.Get("timeout"); v != "" { + d, herr := parseMetricsDuration("timeout", v) + if herr != nil { + return req, herr + } + timeout = min(d, lim.MaxTimeout) + } + out.Set("timeout", formatSeconds(timeout)) + + switch endpoint { + case "api/v1/query": + if herr := copyQuery(out, in); herr != nil { + return req, herr + } + req.start = now + if v := in.Get("time"); v != "" { + t, herr := parseMetricsTime("time", v) + if herr != nil { + return req, herr + } + req.start = t + out.Set("time", formatTime(t)) + } + if v := in.Get("step"); v != "" { + step, herr := parseStep(v, lim) + if herr != nil { + return req, herr + } + out.Set("step", formatSeconds(step)) + } + return req, nil + + case "api/v1/query_range": + if herr := copyQuery(out, in); herr != nil { + return req, herr + } + for _, k := range []string{"start", "end", "step"} { + if in.Get(k) == "" { + return req, errorf(http.StatusBadRequest, "%s is required", k) + } + } + start, end, herr := metricsWindow(in, now, lim) + if herr != nil { + return req, herr + } + step, herr := parseStep(in.Get("step"), lim) + if herr != nil { + return req, herr + } + if points := end.Sub(start)/step + 1; points > time.Duration(lim.MaxPoints) { + return req, errorf(http.StatusUnprocessableEntity, "%d points per series exceed the limit of %d; raise step", points, lim.MaxPoints) + } + req.start = start + out.Set("start", formatTime(start)) + out.Set("end", formatTime(end)) + out.Set("step", formatSeconds(step)) + return req, nil + } + + name, isLabelValues := strings.CutPrefix(endpoint, "api/v1/label/") + if isLabelValues { + var ok bool + if name, ok = strings.CutSuffix(name, "/values"); !ok || strings.Contains(name, "/") { + return req, errorf(http.StatusNotFound, "not found") + } + if !labelName.MatchString(name) { + return req, errorf(http.StatusBadRequest, "invalid label name %q", name) + } + } else if endpoint != "api/v1/series" && endpoint != "api/v1/labels" { + return req, errorf(http.StatusNotFound, "not found") + } + start, end, herr := metricsWindow(in, now, lim) + if herr != nil { + return req, herr + } + req.start = start + out.Set("start", formatTime(start)) + out.Set("end", formatTime(end)) + + matchers := in["match[]"] + if len(matchers) > maxMetricsMatchers { + return req, errorf(http.StatusUnprocessableEntity, "at most %d match[] are allowed", maxMetricsMatchers) + } + for _, m := range matchers { + if m == "" { + return req, errorf(http.StatusBadRequest, "empty match[]") + } + out.Add("match[]", m) + } + if len(matchers) == 0 { + if endpoint == "api/v1/series" { + return req, errorf(http.StatusBadRequest, "match[] is required") + } + // vmselect started with -search.ignoreExtraFiltersAtLabelsAPI drops + // extra_filters[] on the label APIs unless a match[] is present + // (getCommonParamsInternal), so the scope is repeated here. + out.Set("match[]", namespaceSelector(namespace)) + } + return req, nil +} + +func copyQuery(out, in url.Values) *HTTPError { + q := in.Get("query") + if q == "" { + return errorf(http.StatusBadRequest, "query is required") + } + out.Set("query", q) + return nil +} + +// metricsWindow reads start and end with vmselect's label API defaults: +// end is now and start is five minutes before end. +func metricsWindow(in url.Values, now time.Time, lim MetricsLimits) (time.Time, time.Time, *HTTPError) { + end := now + if v := in.Get("end"); v != "" { + t, herr := parseMetricsTime("end", v) + if herr != nil { + return time.Time{}, time.Time{}, herr + } + end = t + } + start := end.Add(-labelsDefaultWindow) + if v := in.Get("start"); v != "" { + t, herr := parseMetricsTime("start", v) + if herr != nil { + return time.Time{}, time.Time{}, herr + } + start = t + } + if start.After(end) { + return time.Time{}, time.Time{}, errorf(http.StatusBadRequest, "start is after end") + } + if end.Sub(start) > lim.MaxRange { + return time.Time{}, time.Time{}, errorf(http.StatusUnprocessableEntity, "range %s exceeds the limit of %s", end.Sub(start), lim.MaxRange) + } + return start, end, nil +} + +// parseStep refuses a step above the range limit: vmselect aligns the +// window outward to whole steps, and an implicit rollup window is one step, +// so a huge step reads far more than the range it was sent with. +func parseStep(v string, lim MetricsLimits) (time.Duration, *HTTPError) { + step, herr := parseMetricsDuration("step", v) + if herr != nil { + return 0, herr + } + if step > lim.MaxRange { + return 0, errorf(http.StatusUnprocessableEntity, "step %s exceeds the limit of %s", step, lim.MaxRange) + } + return step, nil +} + +// parseMetricsDuration takes Prometheus float seconds or a duration such as +// 1m30s. vmselect works in milliseconds and refuses anything shorter. +func parseMetricsDuration(key, v string) (time.Duration, *HTTPError) { + d, err := parseDuration(v) + if err != nil || d < time.Millisecond { + return 0, errorf(http.StatusBadRequest, "invalid %s %q", key, v) + } + return d, nil +} + +// parseMetricsTime takes the two forms the Prometheus API defines: float +// unix seconds and RFC3339. vmselect would also guess milliseconds from the +// digit count and accept relative times; a guessed unit silently moves the +// window, so those are refused. +func parseMetricsTime(key, v string) (time.Time, *HTTPError) { + if sec, frac, ok := strings.Cut(v, "."); isDigits(sec) && (!ok || isDigits(frac)) { + if len(sec) > 10 { + return time.Time{}, errorf(http.StatusBadRequest, "invalid %s %q: not unix seconds", key, v) + } + secs, _ := strconv.ParseInt(sec, 10, 64) + var nanos int64 + if ok { + nanos, _ = strconv.ParseInt((frac + "000000000")[:9], 10, 64) + } + return time.Unix(secs, nanos).UTC(), nil + } + t, err := time.Parse(time.RFC3339Nano, v) + if err != nil || t.Before(time.Unix(0, 0)) { + return time.Time{}, errorf(http.StatusBadRequest, "invalid %s %q", key, v) + } + return t.UTC(), nil +} + +func formatTime(t time.Time) string { + ms := t.UnixMilli() + return fmt.Sprintf("%d.%03d", ms/1000, ms%1000) +} + +func formatSeconds(d time.Duration) string { + return strconv.FormatFloat(float64(d.Milliseconds())/1000, 'f', -1, 64) +} diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go new file mode 100644 index 000000000..be1d69c38 --- /dev/null +++ b/internal/observabilityproxy/metricsparams_test.go @@ -0,0 +1,246 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "net/http" + "net/url" + "strconv" + "strings" + "testing" + "time" +) + +func testMetricsLimits() MetricsLimits { + return MetricsLimits{MaxRange: 30 * 24 * time.Hour, MaxPoints: 11000, MaxTimeout: 30 * time.Second} +} + +const wantMetricsScope = `{namespace="tenant-a"}` + +func unix(t time.Time) string { return strconv.FormatInt(t.Unix(), 10) } + +func buildMetrics(t *testing.T, endpoint string, in url.Values) metricsRequest { + t.Helper() + req, err := BuildMetricsParams(endpoint, in, "tenant-a", testNow, testMetricsLimits()) + if err != nil { + t.Fatalf("BuildMetricsParams(%s, %v): %v", endpoint, in, err) + } + return req +} + +func metricsStatus(endpoint string, in url.Values) int { + _, err := BuildMetricsParams(endpoint, in, "tenant-a", testNow, testMetricsLimits()) + if err == nil { + return http.StatusOK + } + return err.Code +} + +func TestBuildMetricsParamsAllowlist(t *testing.T) { + hostile := url.Values{ + "extra_filters": {`{namespace=~".+"}`}, + "extra_filters[]": {`{namespace="tenant-b"}`, `{namespace=~".*"}`}, + "extra_label": {"namespace=tenant-b"}, + "match": {`{namespace="tenant-b"}`}, + "nocache": {"1"}, + "limit": {"1"}, + "AccountID": {"7"}, + "latency_offset": {"1h"}, + "round_digits": {"2"}, + } + cases := map[string]url.Values{ + "api/v1/query": {"query": {"up"}, "time": {unix(testNow)}, "step": {"30s"}}, + "api/v1/query_range": {"query": {"up"}, "start": {unix(testNow.Add(-time.Hour))}, "end": {unix(testNow)}, "step": {"60"}}, + "api/v1/series": {"match[]": {"up"}}, + "api/v1/labels": {"match[]": {"up"}}, + "api/v1/label/pod/values": {"match[]": {"up"}}, + } + allowed := map[string]bool{"query": true, "start": true, "end": true, "step": true, "time": true, "match[]": true, "timeout": true, "extra_filters[]": true} + for endpoint, in := range cases { + t.Run(endpoint, func(t *testing.T) { + for k, v := range hostile { + in[k] = v + } + req := buildMetrics(t, endpoint, in) + if req.path != endpoint { + t.Errorf("path = %q", req.path) + } + for k := range req.params { + if !allowed[k] { + t.Errorf("parameter %q forwarded: %v", k, req.params) + } + } + if got := req.params["extra_filters[]"]; len(got) != 1 || got[0] != wantMetricsScope { + t.Errorf("extra_filters[] = %q, want exactly [%s]", got, wantMetricsScope) + } + for _, m := range req.params["match[]"] { + if strings.Contains(m, "tenant-b") { + t.Errorf("client match without brackets forwarded as match[]: %q", m) + } + } + }) + } +} + +func TestNamespaceSelectorEscapes(t *testing.T) { + for _, ns := range []string{"tenant-a", `a"} or {namespace=~".+`, `back\slash`, "new\nline"} { + sel := namespaceSelector(ns) + lit, ok := strings.CutPrefix(sel, "{namespace=") + lit, ok2 := strings.CutSuffix(lit, "}") + if !ok || !ok2 { + t.Fatalf("selector %q has the wrong shape", sel) + } + // MetricsQL reads a double-quoted label value with strconv.Unquote + // (metricsql extractStringValue). + got, err := strconv.Unquote(lit) + if err != nil || got != ns { + t.Errorf("selector %q decodes to %q, %v; want %q", sel, got, err, ns) + } + } +} + +func TestBuildMetricsParamsLabelEndpointsAlwaysMatch(t *testing.T) { + for _, endpoint := range []string{"api/v1/labels", "api/v1/label/__name__/values"} { + req := buildMetrics(t, endpoint, url.Values{}) + if got := req.params["match[]"]; len(got) != 1 || got[0] != wantMetricsScope { + t.Errorf("%s without match[] sends %q, want the namespace selector", endpoint, got) + } + req = buildMetrics(t, endpoint, url.Values{"match[]": {"up", `{job="x"}`}}) + if got := req.params["match[]"]; len(got) != 2 || got[0] != "up" || got[1] != `{job="x"}` { + t.Errorf("%s client match[] = %q, want them kept", endpoint, got) + } + } + if got := metricsStatus("api/v1/series", url.Values{}); got != http.StatusBadRequest { + t.Errorf("series without match[] = %d, want 400", got) + } + if got := metricsStatus("api/v1/series", url.Values{"match[]": {""}}); got != http.StatusBadRequest { + t.Errorf("empty match[] = %d, want 400", got) + } + many := url.Values{} + for i := range maxMetricsMatchers + 1 { + many.Add("match[]", "up"+strconv.Itoa(i)) + } + if got := metricsStatus("api/v1/labels", many); got != http.StatusUnprocessableEntity { + t.Errorf("%d match[] = %d, want 422", maxMetricsMatchers+1, got) + } +} + +func TestBuildMetricsParamsLimits(t *testing.T) { + ago := func(d time.Duration) string { return unix(testNow.Add(-d)) } + day := 24 * time.Hour + rng := func(start, end, step string) url.Values { + return url.Values{"query": {"up"}, "start": {start}, "end": {end}, "step": {step}} + } + tests := []struct { + name string + endpoint string + in url.Values + want int + }{ + {"range at the limit", "api/v1/query_range", rng(ago(30*day), ago(0), "1h"), http.StatusOK}, + {"range above the limit", "api/v1/query_range", rng(ago(31*day), ago(0), "1h"), http.StatusUnprocessableEntity}, + {"points at the limit", "api/v1/query_range", rng(ago(10999*time.Minute), ago(0), "1m"), http.StatusOK}, + {"points above the limit", "api/v1/query_range", rng(ago(11000*time.Minute), ago(0), "1m"), http.StatusUnprocessableEntity}, + {"step above the range limit", "api/v1/query_range", rng(ago(time.Hour), ago(0), "31d"), http.StatusUnprocessableEntity}, + {"zero step", "api/v1/query_range", rng(ago(time.Hour), ago(0), "0"), http.StatusBadRequest}, + {"sub-millisecond step", "api/v1/query_range", rng(ago(time.Hour), ago(0), "0.0001"), http.StatusBadRequest}, + {"missing step", "api/v1/query_range", url.Values{"query": {"up"}, "start": {ago(time.Hour)}, "end": {ago(0)}}, http.StatusBadRequest}, + {"missing start", "api/v1/query_range", url.Values{"query": {"up"}, "end": {ago(0)}, "step": {"1m"}}, http.StatusBadRequest}, + {"missing query", "api/v1/query_range", url.Values{"start": {ago(time.Hour)}, "end": {ago(0)}, "step": {"1m"}}, http.StatusBadRequest}, + {"inverted range", "api/v1/query_range", rng(ago(0), ago(time.Hour), "1m"), http.StatusBadRequest}, + {"millisecond timestamp", "api/v1/query_range", rng(ago(time.Hour)+"000", ago(0)+"000", "1m"), http.StatusBadRequest}, + {"relative time", "api/v1/query_range", rng("-1h", "now", "1m"), http.StatusBadRequest}, + {"rfc3339", "api/v1/query_range", rng("2026-11-04T11:00:00Z", "2026-11-04T12:00:00.5Z", "15s"), http.StatusOK}, + {"instant without time", "api/v1/query", url.Values{"query": {"up"}}, http.StatusOK}, + {"instant step above the range limit", "api/v1/query", url.Values{"query": {"up"}, "step": {"31d"}}, http.StatusUnprocessableEntity}, + {"instant bad time", "api/v1/query", url.Values{"query": {"up"}, "time": {"yesterday"}}, http.StatusBadRequest}, + {"negative timeout", "api/v1/query", url.Values{"query": {"up"}, "timeout": {"-1s"}}, http.StatusBadRequest}, + {"series range above the limit", "api/v1/series", url.Values{"match[]": {"up"}, "start": {ago(31 * day)}}, http.StatusUnprocessableEntity}, + {"labels inverted range", "api/v1/labels", url.Values{"start": {ago(0)}, "end": {ago(time.Hour)}}, http.StatusBadRequest}, + {"invalid label name", "api/v1/label/a-b/values", url.Values{}, http.StatusBadRequest}, + {"unknown endpoint", "api/v1/status/tsdb", url.Values{}, http.StatusNotFound}, + {"export", "api/v1/export", url.Values{"match[]": {"up"}}, http.StatusNotFound}, + {"label without values", "api/v1/label/pod", url.Values{}, http.StatusNotFound}, + {"root", "", url.Values{}, http.StatusNotFound}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := metricsStatus(tt.endpoint, tt.in); got != tt.want { + t.Fatalf("status = %d, want %d", got, tt.want) + } + }) + } +} + +func TestBuildMetricsParamsCanonicalForms(t *testing.T) { + req := buildMetrics(t, "api/v1/query_range", url.Values{ + "query": {"up"}, "start": {"2026-11-04T11:00:00.25Z"}, "end": {unix(testNow)}, "step": {"1m30s"}, + }) + want := url.Values{ + "query": {"up"}, "start": {"1793790000.250"}, "end": {"1793793600.000"}, "step": {"90"}, + "timeout": {"30"}, "extra_filters[]": {wantMetricsScope}, + } + if req.params.Encode() != want.Encode() { + t.Errorf("params = %v\nwant %v", req.params, want) + } + + for in, want := range map[string]string{"": "30", "10s": "10", "5m": "30", "1.5": "1.5"} { + v := url.Values{"query": {"up"}} + if in != "" { + v.Set("timeout", in) + } + if got := buildMetrics(t, "api/v1/query", v).params.Get("timeout"); got != want { + t.Errorf("timeout %q forwarded as %q, want %q", in, got, want) + } + } + + req = buildMetrics(t, "api/v1/labels", url.Values{}) + if req.params.Get("end") != "1793793600.000" || req.params.Get("start") != "1793793300.000" { + t.Errorf("labels window = %s..%s, want the last 5m", req.params.Get("start"), req.params.Get("end")) + } + if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Has("time") || !q.start.Equal(testNow) { + t.Errorf("instant query without time: params %v, start %v", q.params, q.start) + } +} + +func TestMetricsStoreByRange(t *testing.T) { + cfg := MetricsConfig{ShortTermStore: "shortterm", LongTermStore: "longterm", ShortTermRetention: 72 * time.Hour} + tests := []struct { + name string + endpoint string + in url.Values + want string + }{ + {"instant now", "api/v1/query", url.Values{"query": {"up"}}, "shortterm"}, + {"instant within retention", "api/v1/query", url.Values{"query": {"up"}, "time": {unix(testNow.Add(-71 * time.Hour))}}, "shortterm"}, + {"instant past retention", "api/v1/query", url.Values{"query": {"up"}, "time": {unix(testNow.Add(-73 * time.Hour))}}, "longterm"}, + {"live graph", "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-6 * time.Hour))}, "end": {unix(testNow)}, "step": {"60"}}, "shortterm"}, + {"range fits exactly", "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-72 * time.Hour))}, "end": {unix(testNow)}, "step": {"600"}}, "shortterm"}, + {"range past retention", "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-7 * 24 * time.Hour))}, "end": {unix(testNow)}, "step": {"600"}}, "longterm"}, + {"old window ending long ago", "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-5 * 24 * time.Hour))}, "end": {unix(testNow.Add(-4 * 24 * time.Hour))}, "step": {"600"}}, "longterm"}, + {"labels default window", "api/v1/labels", url.Values{}, "shortterm"}, + {"series past retention", "api/v1/series", url.Values{"match[]": {"up"}, "start": {unix(testNow.Add(-4 * 24 * time.Hour))}}, "longterm"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + req := buildMetrics(t, tt.endpoint, tt.in) + if got := cfg.store(req.start, testNow); got != tt.want { + t.Fatalf("store = %s, want %s", got, tt.want) + } + }) + } +} From 47409db70c303fb36ad31ac3d34465c8bcbbf428 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:00:14 +0700 Subject: [PATCH 309/518] feat(observability-proxy): serve the namespace metrics root The namespace metrics root shares the logs root's authorization path: the SubjectAccessReview comes from the path alone, so a POST query, which Grafana sends by default, is authorized exactly like a GET. Both roots also share the per-user rate and concurrency limits and the way an unreachable store (502) is told apart from a slow one (504); the upstream call moves into one helper so the two stay identical. Every namespace resolves to the root tenant's vmselect for now, because the series the graphs need are only scraped there. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 103 ++++++++--- .../observabilityproxy/server_metrics_test.go | 163 ++++++++++++++++++ internal/observabilityproxy/server_test.go | 17 +- 3 files changed, 259 insertions(+), 24 deletions(-) create mode 100644 internal/observabilityproxy/server_metrics_test.go diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index e980818d9..3e24be109 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -25,6 +25,7 @@ import ( "log/slog" "math" "net/http" + "net/url" "os" "strconv" "strings" @@ -56,6 +57,7 @@ type Config struct { // means the in-cluster Service of the "generic" logs storage. UpstreamURL func(store string) string Logs LogsLimits + Metrics MetricsConfig Limiter LimiterConfig RequestTimeout time.Duration TailIdleTimeout time.Duration @@ -84,6 +86,12 @@ func NewServer(cfg Config, auth *Authorizer, stores StoreLookup, reg prometheus. return fmt.Sprintf("http://vlselect-generic.%s.svc.%s:9471", store, domain) } } + if cfg.Metrics.UpstreamURL == nil { + domain, root := cfg.ClusterDomain, cfg.Metrics.RootTenant + cfg.Metrics.UpstreamURL = func(store string) string { + return fmt.Sprintf("http://vmselect-%s.%s.svc.%s:8481/select/0/prometheus", store, root, domain) + } + } s := &Server{ cfg: cfg, auth: auth, @@ -167,7 +175,7 @@ func writeError(w http.ResponseWriter, e *HTTPError) { // serve answers a root it has no handler for before authenticating, so an // unknown path costs no review calls. func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTTPError { - if route.Kind != NamespaceLogs { + if route.Kind != NamespaceLogs && route.Kind != NamespaceMetrics { return errorf(http.StatusNotFound, "not found") } user, herr := s.authorize(r, route) @@ -182,6 +190,9 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT w.Header().Set("Retry-After", strconv.FormatInt(int64(math.Ceil(wait.Seconds())), 10)) return errorf(http.StatusTooManyRequests, "request rate limit exceeded") } + if route.Kind == NamespaceMetrics { + return s.serveMetrics(w, r, route, user) + } endpoint, ok := strings.CutPrefix(route.Endpoint, "select/logsql/") if !ok { return errorf(http.StatusNotFound, "not found") @@ -226,29 +237,17 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT ctx, stop = context.WithTimeout(ctx, limit) defer stop() } - req, err := http.NewRequestWithContext(ctx, http.MethodPost, - s.cfg.UpstreamURL(store)+"/select/logsql/"+endpoint, strings.NewReader(params.Encode())) - if err != nil { - return errorf(http.StatusInternalServerError, "building upstream request: %v", err) - } - req.Header.Set("Content-Type", "application/x-www-form-urlencoded") - req.Header.Set("AccountID", "0") - req.Header.Set("ProjectID", "0") + tenant := http.Header{} + tenant.Set("AccountID", "0") + tenant.Set("ProjectID", "0") if tail { // vlselect holds even the headers of a gzip-encoded tail until the // first line; an explicit value stops the transport from asking for gzip. - req.Header.Set("Accept-Encoding", "identity") + tenant.Set("Accept-Encoding", "identity") } - resp, err := s.client.Do(req) - switch { - case err == nil: - case r.Context().Err() != nil: - return errorf(http.StatusBadGateway, "client went away") - case errors.Is(context.Cause(ctx), context.DeadlineExceeded): - return errorf(http.StatusGatewayTimeout, "log store did not answer within %s", limit) - default: - slog.Warn("log store unreachable", "store", store, "error", err) - return errorf(http.StatusBadGateway, "log store unreachable") + resp, herr := s.post(ctx, r, s.cfg.UpstreamURL(store)+"/select/logsql/"+endpoint, params, "log store", tenant) + if herr != nil { + return herr } defer func() { _ = resp.Body.Close() }() @@ -280,6 +279,70 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT return nil } +func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Route, user string) *HTTPError { + r.Body = http.MaxBytesReader(w, r.Body, maxFormBytes) + if err := r.ParseForm(); err != nil { + return errorf(http.StatusBadRequest, "invalid form: %v", err) + } + now := s.now() + req, herr := BuildMetricsParams(route.Endpoint, r.Form, route.Namespace, now, s.cfg.Metrics.Limits) + if herr != nil { + return herr + } + release, ok := s.limits.acquire(user) + if !ok { + return errorf(http.StatusUnprocessableEntity, "too many concurrent requests") + } + defer release() + + ctx := r.Context() + if s.cfg.RequestTimeout > 0 { + var cancel context.CancelFunc + ctx, cancel = context.WithTimeout(ctx, s.cfg.RequestTimeout) + defer cancel() + } + store := s.cfg.Metrics.store(req.start, now) + resp, herr := s.post(ctx, r, s.cfg.Metrics.UpstreamURL(store)+"/"+req.path, req.params, "metrics store", nil) + if herr != nil { + return herr + } + defer func() { _ = resp.Body.Close() }() + if ct := resp.Header.Get("Content-Type"); ct != "" { + w.Header().Set("Content-Type", ct) + } + w.WriteHeader(resp.StatusCode) + if _, err := io.Copy(w, resp.Body); err != nil && !errors.Is(err, context.Canceled) { + slog.Debug("copying upstream response", "error", err) + } + return nil +} + +// post sends params as a form with only the given headers, so nothing the +// client sent reaches the store, and tells an unreachable store (502) from +// one that missed its deadline or, for a tail, its idle timeout (504). +func (s *Server) post(ctx context.Context, r *http.Request, target string, params url.Values, what string, header http.Header) (*http.Response, *HTTPError) { + req, err := http.NewRequestWithContext(ctx, http.MethodPost, target, strings.NewReader(params.Encode())) + if err != nil { + return nil, errorf(http.StatusInternalServerError, "building upstream request: %v", err) + } + for k, v := range header { + req.Header[k] = v + } + req.Header.Set("Content-Type", "application/x-www-form-urlencoded") + resp, err := s.client.Do(req) + switch { + case err == nil: + return resp, nil + case r.Context().Err() != nil: + return nil, errorf(http.StatusBadGateway, "client went away") + case errors.Is(context.Cause(ctx), context.DeadlineExceeded): + return nil, errorf(http.StatusGatewayTimeout, "%s did not answer in time", what) + default: + slog.Warn(what+" unreachable", "url", target, "error", err) + return nil, errorf(http.StatusBadGateway, "%s unreachable", what) + } +} + // concurrencyRetryAfter is a hint only: a slot frees when another request // of the user ends, which the proxy cannot predict, and one second keeps // a polling client from retrying in a tight loop. diff --git a/internal/observabilityproxy/server_metrics_test.go b/internal/observabilityproxy/server_metrics_test.go new file mode 100644 index 000000000..887790d34 --- /dev/null +++ b/internal/observabilityproxy/server_metrics_test.go @@ -0,0 +1,163 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "io" + "net/http" + "net/url" + "testing" + "time" + + authorizationv1 "k8s.io/api/authorization/v1" +) + +const metricsRoot = "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/" + +func promUpstream(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"status":"success","data":{"resultType":"vector","result":[]}}`) +} + +func TestServerMetricsRebuildsUpstreamRequest(t *testing.T) { + h := newHarness(t, Config{}, promUpstream) + hostile := url.Values{ + "query": {"sum(rate(container_cpu_usage_seconds_total[5m]))"}, + "start": {unix(testNow.Add(-time.Hour))}, + "end": {unix(testNow)}, + "step": {"60"}, + "extra_filters": {`{namespace=~".+"}`}, + "extra_filters[]": {`{namespace="tenant-b"}`, `{namespace=~".*"}`}, + "extra_label": {"namespace=tenant-b"}, + "nocache": {"1"}, + } + headers := map[string]string{"AccountID": "9", "X-Forwarded-For": "1.2.3.4", "Cookie": "kc-access=secret"} + + resp := h.do(http.MethodGet, metricsRoot+"api/v1/query_range?"+hostile.Encode(), "good", nil, headers) + _ = resp.Body.Close() + resp = h.do(http.MethodPost, metricsRoot+"api/v1/query_range?extra_filters[]="+url.QueryEscape(`{namespace="x"}`), "good", hostile, headers) + body, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusOK || resp.Header.Get("Content-Type") != "application/json" { + t.Fatalf("POST answered %d %s %q", resp.StatusCode, resp.Header.Get("Content-Type"), body) + } + + reqs := h.requests() + if len(reqs) != 2 { + t.Fatalf("upstream saw %d requests", len(reqs)) + } + if reqs[0].form.Encode() != reqs[1].form.Encode() { + t.Fatalf("GET and POST built different upstream requests:\n%v\n%v", reqs[0].form, reqs[1].form) + } + if n := h.kube.sars.Load(); n != 1 { + t.Fatalf("SubjectAccessReviews = %d, want 1: POST must reuse the GET decision", n) + } + want := authorizationv1.ResourceAttributes{Verb: "list", Group: "cozystack.io", Resource: "metrics", Namespace: "tenant-a"} + if got := h.kube.lastSAR.Load().ResourceAttributes; *got != want { + t.Fatalf("SubjectAccessReview = %+v, want %+v", *got, want) + } + for _, r := range reqs { + if r.path != "/api/v1/query_range" { + t.Errorf("upstream path = %s", r.path) + } + if got := r.form["extra_filters[]"]; len(got) != 1 || got[0] != wantMetricsScope { + t.Errorf("extra_filters[] = %q", got) + } + if r.form.Has("extra_filters") || r.form.Has("extra_label") || r.form.Has("nocache") { + t.Errorf("client parameter forwarded: %v", r.form) + } + for _, k := range []string{"Authorization", "AccountID", "X-Forwarded-For", "Cookie"} { + if r.headers.Get(k) != "" { + t.Errorf("client header %s forwarded", k) + } + } + } +} + +func TestServerMetricsStoreByRange(t *testing.T) { + h := newHarness(t, Config{}, promUpstream) + for _, path := range []string{ + "api/v1/query?query=up", + "api/v1/query_range?query=up&step=600&start=" + unix(testNow.Add(-7*24*time.Hour)) + "&end=" + unix(testNow), + "api/v1/label/pod/values", + } { + if got := h.status(http.MethodGet, metricsRoot+path, "good"); got != http.StatusOK { + t.Fatalf("%s answered %d", path, got) + } + } + h.mu.Lock() + defer h.mu.Unlock() + if want := []string{"shortterm", "longterm", "shortterm"}; len(h.stores) != 3 || h.stores[0] != want[0] || h.stores[1] != want[1] || h.stores[2] != want[2] { + t.Fatalf("stores = %v, want %v", h.stores, want) + } + if r := h.got[2]; r.path != "/api/v1/label/pod/values" || r.form.Get("match[]") != wantMetricsScope { + t.Fatalf("label values upstream = %s %v", r.path, r.form) + } +} + +func TestServerMetricsStatusCodes(t *testing.T) { + h := newHarness(t, Config{}, promUpstream) + tooLong := "api/v1/query_range?query=up&step=3600&start=" + unix(testNow.Add(-31*24*time.Hour)) + "&end=" + unix(testNow) + if got := h.status(http.MethodGet, metricsRoot+tooLong, "good"); got != http.StatusUnprocessableEntity { + t.Errorf("range above the limit answered %d, want 422", got) + } + if got := h.status(http.MethodGet, metricsRoot+"api/v1/series", "good"); got != http.StatusBadRequest { + t.Errorf("series without match[] answered %d, want 400", got) + } + if got := h.status(http.MethodPut, metricsRoot+"api/v1/query?query=up", "good"); got != http.StatusMethodNotAllowed { + t.Errorf("PUT answered %d, want 405", got) + } + if n := len(h.requests()); n != 0 { + t.Errorf("upstream saw %d requests for refused ones", n) + } + + h2 := newHarness(t, Config{}, func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, `{"status":"error","errorType":"422"}`, http.StatusUnprocessableEntity) + }) + if got := h2.status(http.MethodGet, metricsRoot+"api/v1/query?query=up", "good"); got != http.StatusUnprocessableEntity { + t.Errorf("upstream 422 answered as %d", got) + } + h2.upstream.Close() + if got := h2.status(http.MethodGet, metricsRoot+"api/v1/query?query=up", "good"); got != http.StatusBadGateway { + t.Errorf("unreachable upstream answered %d, want 502", got) + } + + release := make(chan struct{}) + t.Cleanup(func() { close(release) }) + h3 := newHarness(t, Config{RequestTimeout: 50 * time.Millisecond}, func(_ http.ResponseWriter, r *http.Request) { + select { + case <-release: + case <-r.Context().Done(): + } + }) + if got := h3.status(http.MethodGet, metricsRoot+"api/v1/query?query=up", "good"); got != http.StatusGatewayTimeout { + t.Errorf("slow upstream answered %d, want 504", got) + } + + h4 := newHarness(t, Config{Limiter: LimiterConfig{RatePerUser: 0.001, RateBurst: 1}}, promUpstream) + h4.status(http.MethodGet, metricsRoot+"api/v1/query?query=up", "good") + if got := h4.status(http.MethodGet, metricsRoot+"api/v1/query?query=up", "good"); got != http.StatusUnprocessableEntity { + t.Errorf("rate-limited request answered %d, want 422", got) + } +} + +func TestDefaultMetricsUpstreamURL(t *testing.T) { + s := NewServer(Config{ClusterDomain: "cozy.local", Metrics: MetricsConfig{RootTenant: "tenant-root"}}, nil, nil, nil) + if got, want := s.cfg.Metrics.UpstreamURL("shortterm"), "http://vmselect-shortterm.tenant-root.svc.cozy.local:8481/select/0/prometheus"; got != want { + t.Fatalf("upstream = %s, want %s", got, want) + } +} diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index df3e23bc4..2670d0e3e 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -52,12 +52,12 @@ type mapStores map[string]string func (m mapStores) Resolve(ns string) (string, error) { return m[ns], nil } // newHarness wires the proxy to a fake apiserver that knows tokens "good" -// (alice) and "bob-token" (bob), both allowed list logs in tenant-a only, -// and to an upstream served by handler. +// (alice) and "bob-token" (bob), both allowed list logs and metrics in +// tenant-a only, and to an upstream served by handler. func newHarness(t *testing.T, cfg Config, handler http.HandlerFunc) *harness { h := &harness{t: t} h.kube = newFakeKube(map[string]string{"good": "alice", "bob-token": "bob"}, func(_ string, a *authorizationv1.ResourceAttributes) bool { - return a.Namespace == "tenant-a" && a.Resource == "logs" + return a.Namespace == "tenant-a" && (a.Resource == "logs" || a.Resource == "metrics") }) h.upstream = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if err := r.ParseForm(); err != nil { @@ -80,6 +80,13 @@ func newHarness(t *testing.T, cfg Config, handler http.HandlerFunc) *harness { h.mu.Unlock() return h.upstream.URL } + cfg.Metrics.UpstreamURL = cfg.UpstreamURL + if cfg.Metrics.ShortTermStore == "" { + cfg.Metrics.ShortTermStore, cfg.Metrics.LongTermStore, cfg.Metrics.ShortTermRetention = "shortterm", "longterm", 72*time.Hour + } + if cfg.Metrics.Limits == (MetricsLimits{}) { + cfg.Metrics.Limits = testMetricsLimits() + } if cfg.Logs == (LogsLimits{}) { cfg.Logs = testLimits() } @@ -144,7 +151,9 @@ func TestServerStatusCodes(t *testing.T) { {"other namespace", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-b/logs/select/logsql/query", "good", http.StatusForbidden}, {"invalid namespace", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/Tenant/logs/select/logsql/query", "good", http.StatusBadRequest}, {"unknown root", http.MethodGet, "/api/v1/namespaces/tenant-a/pods", "good", http.StatusNotFound}, - {"namespace metrics not served yet", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/api/v1/query", "good", http.StatusNotFound}, + {"unknown metrics endpoint", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/api/v1/status/tsdb", "good", http.StatusNotFound}, + {"metrics in another namespace", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-b/metrics/api/v1/query?query=up", "good", http.StatusForbidden}, + {"metrics without token", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/api/v1/query?query=up", "", http.StatusUnauthorized}, {"application log not served yet", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/log/select/logsql/query", "good", http.StatusNotFound}, {"application metrics not served yet", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/metrics/api/v1/query", "good", http.StatusNotFound}, {"tenant_ids not proxied", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/select/tenant_ids", "good", http.StatusNotFound}, From cd3d889ee8b489e36034c64dcdd1a56798fedfac Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:00:45 +0700 Subject: [PATCH 310/518] feat(observability-proxy): add metrics store and cost limit flags The store names and the short-term retention mirror metricsStorages in the monitoring package, which an operator can rename or resize; they are flags so the chart can follow such a change without a rebuild. The cost limit defaults are the ones the metrics proposal settles on: 30d range, 11000 points per series, 30s timeout. Store names become part of a Service host name, so they are checked as DNS labels at startup. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 29 +++++++++++++++++++++++++--- cmd/observability-proxy/main_test.go | 18 +++++++++++++++++ 2 files changed, 44 insertions(+), 3 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index 558908671..6acfb1e08 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -14,9 +14,10 @@ See the License for the specific language governing permissions and limitations under the License. */ -// Command observability-proxy serves tenant logs from the VictoriaLogs -// stores behind a Kubernetes TokenReview and SubjectAccessReview, with the -// tenant's namespace scope forced into every upstream request. +// Command observability-proxy serves tenant logs and metrics from the +// VictoriaLogs and VictoriaMetrics stores behind a Kubernetes TokenReview +// and SubjectAccessReview, with the tenant's namespace scope forced into +// every upstream request. package main import ( @@ -37,6 +38,7 @@ import ( "github.com/prometheus/client_golang/prometheus" "github.com/prometheus/client_golang/prometheus/collectors" "github.com/prometheus/client_golang/prometheus/promhttp" + "k8s.io/apimachinery/pkg/util/validation" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/cache" @@ -56,6 +58,7 @@ type options struct { tailIdleTimeout time.Duration shutdownTimeout time.Duration logs observabilityproxy.LogsLimits + metrics observabilityproxy.MetricsConfig limiter observabilityproxy.LimiterConfig } @@ -81,6 +84,14 @@ func parseFlags(args []string) (options, error) { fs.IntVar(&o.logs.DefaultLimit, "logs-default-limit", 1000, "entries returned when a request sets no limit") fs.IntVar(&o.logs.MaxLimit, "logs-max-limit", 10000, "largest limit a request may set") fs.DurationVar(&o.logs.MaxTailResume, "logs-max-tail-resume", 15*time.Minute, "how far back a tail may resume with since or start_offset") + // Store names and retention follow metricsStorages in + // packages/system/monitoring/values.yaml. + fs.StringVar(&o.metrics.ShortTermStore, "metrics-shortterm-store", "shortterm", "VMCluster name of the short-term metrics store of the root tenant") + fs.StringVar(&o.metrics.LongTermStore, "metrics-longterm-store", "longterm", "VMCluster name of the long-term metrics store of the root tenant") + fs.DurationVar(&o.metrics.ShortTermRetention, "metrics-shortterm-retention", 72*time.Hour, "retention of the short-term store; older windows are read from the long-term one") + fs.DurationVar(&o.metrics.Limits.MaxRange, "metrics-max-range", 30*24*time.Hour, "longest [start, end] and step a metrics request may use") + fs.IntVar(&o.metrics.Limits.MaxPoints, "metrics-max-points", 11000, "most points per series a range query may return") + fs.DurationVar(&o.metrics.Limits.MaxTimeout, "metrics-max-timeout", 30*time.Second, "timeout forwarded to vmselect; a longer client timeout is clamped") fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables") fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained requests per second per user; 0 disables") fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user") @@ -94,6 +105,13 @@ func parseFlags(args []string) (options, error) { if fs.NArg() > 0 { return o, fmt.Errorf("unexpected argument %q", fs.Arg(0)) } + o.metrics.RootTenant = o.rootTenant + for _, store := range []string{o.metrics.ShortTermStore, o.metrics.LongTermStore} { + // The name becomes part of a Service host name. + if errs := validation.IsDNS1123Label("vmselect-" + store); store == "" || len(errs) > 0 { + return o, fmt.Errorf("invalid metrics store name %q", store) + } + } switch { case o.rootTenant == "": return o, errors.New("--root-tenant must not be empty") @@ -110,6 +128,10 @@ func parseFlags(args []string) (options, error) { return o, errors.New("--logs-max-range must be at least 3s") case o.logs.DefaultLimit <= 0 || o.logs.MaxLimit < o.logs.DefaultLimit: return o, errors.New("--logs-default-limit must be positive and not above --logs-max-limit") + case o.metrics.ShortTermRetention <= 0 || o.metrics.Limits.MaxRange <= 0 || o.metrics.Limits.MaxTimeout <= 0: + return o, errors.New("--metrics-shortterm-retention, --metrics-max-range and --metrics-max-timeout must be positive") + case o.metrics.Limits.MaxPoints < 2: + return o, errors.New("--metrics-max-points must be at least 2") case o.logs.MaxTailResume < 0: return o, errors.New("--logs-max-tail-resume must not be negative") case o.requestTimeout < 0 || o.tailIdleTimeout < 0 || o.authCacheTTL < 0 || o.shutdownTimeout < 0: @@ -167,6 +189,7 @@ func run(o options) error { proxy := observabilityproxy.NewServer(observabilityproxy.Config{ ClusterDomain: o.clusterDomain, Logs: o.logs, + Metrics: o.metrics, Limiter: o.limiter, RequestTimeout: o.requestTimeout, TailIdleTimeout: o.tailIdleTimeout, diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index ea678461c..fe7d3ea47 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -32,6 +32,18 @@ func TestParseFlagsDefaults(t *testing.T) { if o.authCacheTTL != 10*time.Second || o.logs.MaxTailResume != 15*time.Minute || o.logs.MaxRange != 7*24*time.Hour { t.Errorf("limit defaults changed: %+v", o) } + // Retention and store names of packages/system/monitoring/values.yaml; + // cost limits of the metrics proposal (Design 7). + m := o.metrics + if m.ShortTermStore != "shortterm" || m.LongTermStore != "longterm" || m.ShortTermRetention != 72*time.Hour { + t.Errorf("metrics store defaults changed: %+v", m) + } + if m.Limits.MaxRange != 30*24*time.Hour || m.Limits.MaxPoints != 11000 || m.Limits.MaxTimeout != 30*time.Second { + t.Errorf("metrics limit defaults changed: %+v", m.Limits) + } + if m.RootTenant != o.rootTenant { + t.Errorf("metrics root tenant = %q, want --root-tenant", m.RootTenant) + } } func TestParseFlagsOverrides(t *testing.T) { @@ -69,6 +81,12 @@ func TestParseFlagsInvalid(t *testing.T) { {"--max-concurrent-per-user=-1"}, {"--max-tails-per-user=-1"}, {"--max-tails=-1"}, + {"--metrics-shortterm-store="}, + {"--metrics-longterm-store=Long.Term"}, + {"--metrics-shortterm-retention=0s"}, + {"--metrics-max-range=0s"}, + {"--metrics-max-points=1"}, + {"--metrics-max-timeout=0s"}, {"--unknown"}, {"unexpected", "--root-tenant=tenant-x"}, } { From 5630de4ab13077ef9a1861d29de2add980f477e3 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:01:25 +0700 Subject: [PATCH 311/518] feat(observability-proxy): grant tenant viewers the metrics root The proxy authorizes the namespace metrics root with a SubjectAccessReview for list on cozystack.io metrics, which no role grants yet. Aggregating it into the tenant view role reaches every tenant group and ancestor subject without a tenant chart change. The Deployment passes the metrics store names and short-term retention from values, so a platform that renames or resizes its metricsStorages can keep the proxy's store choice in step. Assisted-by: LLM Signed-off-by: ZverGuy --- .../templates/deployment.yaml | 3 ++ .../templates/tenant-clusterroles.yaml | 8 ++--- .../tests/deployment_test.yaml | 29 +++++++++++++++++++ .../observability-proxy/tests/rbac_test.yaml | 6 ++-- .../system/observability-proxy/values.yaml | 7 +++++ 5 files changed, 46 insertions(+), 7 deletions(-) diff --git a/packages/system/observability-proxy/templates/deployment.yaml b/packages/system/observability-proxy/templates/deployment.yaml index 581c15fde..d49de140b 100644 --- a/packages/system/observability-proxy/templates/deployment.yaml +++ b/packages/system/observability-proxy/templates/deployment.yaml @@ -38,6 +38,9 @@ spec: - --cluster-domain={{ $clusterDomain }} - --root-tenant={{ .Values.observabilityProxy.rootTenant }} - --auth-cache-ttl={{ .Values.observabilityProxy.authCacheTTL }} + - --metrics-shortterm-store={{ .Values.observabilityProxy.metrics.shorttermStore }} + - --metrics-longterm-store={{ .Values.observabilityProxy.metrics.longtermStore }} + - --metrics-shortterm-retention={{ .Values.observabilityProxy.metrics.shorttermRetention }} ports: - name: http containerPort: 8080 diff --git a/packages/system/observability-proxy/templates/tenant-clusterroles.yaml b/packages/system/observability-proxy/templates/tenant-clusterroles.yaml index 23460352f..a2bf94f3c 100644 --- a/packages/system/observability-proxy/templates/tenant-clusterroles.yaml +++ b/packages/system/observability-proxy/templates/tenant-clusterroles.yaml @@ -1,7 +1,7 @@ # Aggregated into cozy-tenant-view (and consequently use, admin, super-admin). -# No API server serves cozystack.io logs: the rule exists only for the -# SubjectAccessReview the observability proxy asks before it answers a read -# under /apis/cozystack.io/v1alpha1/namespaces//logs/. +# No API server serves cozystack.io logs or metrics: the rule exists only for +# the SubjectAccessReview the observability proxy asks before it answers a +# read under /apis/cozystack.io/v1alpha1/namespaces//{logs,metrics}/. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: @@ -10,5 +10,5 @@ metadata: rbac.cozystack.io/aggregate-to-tenant-view: "true" rules: - apiGroups: ["cozystack.io"] - resources: ["logs"] + resources: ["logs", "metrics"] verbs: ["list"] diff --git a/packages/system/observability-proxy/tests/deployment_test.yaml b/packages/system/observability-proxy/tests/deployment_test.yaml index ddf9d34b8..a5e948920 100644 --- a/packages/system/observability-proxy/tests/deployment_test.yaml +++ b/packages/system/observability-proxy/tests/deployment_test.yaml @@ -40,6 +40,35 @@ tests: path: spec.template.spec.containers[0].args content: --auth-cache-ttl=10s + - it: passes the metrics store names and short-term retention + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --metrics-shortterm-store=shortterm + - contains: + path: spec.template.spec.containers[0].args + content: --metrics-longterm-store=longterm + - contains: + path: spec.template.spec.containers[0].args + content: --metrics-shortterm-retention=72h + + - it: follows renamed metrics stores + set: + observabilityProxy.metrics: + shorttermStore: fast + longtermStore: slow + shorttermRetention: 24h + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --metrics-shortterm-store=fast + - contains: + path: spec.template.spec.containers[0].args + content: --metrics-longterm-store=slow + - contains: + path: spec.template.spec.containers[0].args + content: --metrics-shortterm-retention=24h + - it: falls back to cozy.local when the platform passes no cluster domain asserts: - contains: diff --git a/packages/system/observability-proxy/tests/rbac_test.yaml b/packages/system/observability-proxy/tests/rbac_test.yaml index af339b5d0..ef96c65d7 100644 --- a/packages/system/observability-proxy/tests/rbac_test.yaml +++ b/packages/system/observability-proxy/tests/rbac_test.yaml @@ -2,7 +2,7 @@ suite: observability-proxy RBAC # The proxy authenticates and authorizes every request through the API server, # so a dropped verb here turns every request into an error at runtime rather # than a render failure. The tenant role is what lets a namespace viewer pass -# the SubjectAccessReview the proxy asks for the namespace logs root. +# the SubjectAccessReview the proxy asks for the namespace logs and metrics roots. templates: - templates/rbac.yaml - templates/tenant-clusterroles.yaml @@ -50,7 +50,7 @@ tests: name: observability-proxy namespace: cozy-observability-proxy - - it: aggregates list on cozystack.io logs into the tenant view role + - it: aggregates list on cozystack.io logs and metrics into the tenant view role template: templates/tenant-clusterroles.yaml asserts: - isKind: @@ -63,5 +63,5 @@ tests: path: rules value: - apiGroups: ["cozystack.io"] - resources: ["logs"] + resources: ["logs", "metrics"] verbs: ["list"] diff --git a/packages/system/observability-proxy/values.yaml b/packages/system/observability-proxy/values.yaml index 919e450a9..67334a237 100644 --- a/packages/system/observability-proxy/values.yaml +++ b/packages/system/observability-proxy/values.yaml @@ -7,6 +7,13 @@ observabilityProxy: replicas: 2 rootTenant: tenant-root authCacheTTL: 10s + # Names and retention of the root tenant's metricsStorages + # (packages/system/monitoring/values.yaml). The retention is a Go + # duration: 3d is written 72h. + metrics: + shorttermStore: shortterm + longtermStore: longterm + shorttermRetention: 72h resources: requests: cpu: 10m From 1ac7fd29bf30a15d9dedcd365649718dd8157484 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:06:54 +0700 Subject: [PATCH 312/518] fix(observability-proxy): round the metrics step up to whole milliseconds vmselect reads step and timeout in whole milliseconds and drops the remainder, so a step of 1.9ms passed the points check as 1.9ms and then ran as 1ms, close to twice the points the limit allows. The duration is now rounded up before the limits are checked, so they see the value vmselect will use. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricsparams.go | 6 ++++-- internal/observabilityproxy/metricsparams_test.go | 10 ++++++++++ 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 0e24b6597..e74d2cbb5 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -242,13 +242,15 @@ func parseStep(v string, lim MetricsLimits) (time.Duration, *HTTPError) { } // parseMetricsDuration takes Prometheus float seconds or a duration such as -// 1m30s. vmselect works in milliseconds and refuses anything shorter. +// 1m30s. vmselect works in whole milliseconds and refuses anything shorter; +// the value is rounded up to the millisecond so the limits are checked +// against what vmselect will use, not against a fraction it drops. func parseMetricsDuration(key, v string) (time.Duration, *HTTPError) { d, err := parseDuration(v) if err != nil || d < time.Millisecond { return 0, errorf(http.StatusBadRequest, "invalid %s %q", key, v) } - return d, nil + return (d + time.Millisecond - 1).Truncate(time.Millisecond), nil } // parseMetricsTime takes the two forms the Prometheus API defines: float diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index be1d69c38..ac9ff0ada 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -155,6 +155,7 @@ func TestBuildMetricsParamsLimits(t *testing.T) { {"range above the limit", "api/v1/query_range", rng(ago(31*day), ago(0), "1h"), http.StatusUnprocessableEntity}, {"points at the limit", "api/v1/query_range", rng(ago(10999*time.Minute), ago(0), "1m"), http.StatusOK}, {"points above the limit", "api/v1/query_range", rng(ago(11000*time.Minute), ago(0), "1m"), http.StatusUnprocessableEntity}, + {"points above the limit after rounding the step up", "api/v1/query_range", rng(ago(22*time.Second), ago(0), "0.0019"), http.StatusUnprocessableEntity}, {"step above the range limit", "api/v1/query_range", rng(ago(time.Hour), ago(0), "31d"), http.StatusUnprocessableEntity}, {"zero step", "api/v1/query_range", rng(ago(time.Hour), ago(0), "0"), http.StatusBadRequest}, {"sub-millisecond step", "api/v1/query_range", rng(ago(time.Hour), ago(0), "0.0001"), http.StatusBadRequest}, @@ -198,6 +199,15 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) { t.Errorf("params = %v\nwant %v", req.params, want) } + // vmselect drops a sub-millisecond remainder; the points check has to + // see the step it will use. + for in, want := range map[string]string{"60.0001": "60.001", "0.0019": "0.002"} { + req = buildMetrics(t, "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-20 * time.Second))}, "end": {unix(testNow)}, "step": {in}}) + if got := req.params.Get("step"); got != want { + t.Errorf("step %s forwarded as %q, want %s", in, got, want) + } + } + for in, want := range map[string]string{"": "30", "10s": "10", "5m": "30", "1.5": "1.5"} { v := url.Values{"query": {"up"}} if in != "" { From 81cd278f63bc69cd6ecd7ee164deffb5bae779a1 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:13:30 +0700 Subject: [PATCH 313/518] test(observability-proxy): cover metrics concurrency and flag overrides The per-user concurrency cap on the metrics root had no test, so dropping it from the handler went unnoticed. The new test holds one request in the upstream, expects the same user's second one to be refused and another user's to go through. Overrides of the metrics flags were only checked for their defaults and refusals. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main_test.go | 15 +++++++ .../observabilityproxy/server_metrics_test.go | 40 +++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index fe7d3ea47..be89e20d5 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -60,6 +60,21 @@ func TestParseFlagsOverrides(t *testing.T) { } } +func TestParseFlagsMetricsOverrides(t *testing.T) { + o, err := parseFlags([]string{ + "--root-tenant=tenant-x", "--metrics-shortterm-store=fast", "--metrics-longterm-store=slow", + "--metrics-shortterm-retention=24h", "--metrics-max-range=168h", "--metrics-max-points=500", "--metrics-max-timeout=10s", + }) + if err != nil { + t.Fatal(err) + } + m := o.metrics + if m.RootTenant != "tenant-x" || m.ShortTermStore != "fast" || m.LongTermStore != "slow" || m.ShortTermRetention != 24*time.Hour || + m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second { + t.Errorf("overrides not applied: %+v", m) + } +} + func TestParseFlagsInvalid(t *testing.T) { for _, args := range [][]string{ {"--root-tenant="}, diff --git a/internal/observabilityproxy/server_metrics_test.go b/internal/observabilityproxy/server_metrics_test.go index 887790d34..981644668 100644 --- a/internal/observabilityproxy/server_metrics_test.go +++ b/internal/observabilityproxy/server_metrics_test.go @@ -20,6 +20,7 @@ import ( "io" "net/http" "net/url" + "sync" "testing" "time" @@ -155,6 +156,45 @@ func TestServerMetricsStatusCodes(t *testing.T) { } } +func TestServerMetricsConcurrency(t *testing.T) { + entered, release := make(chan struct{}, 3), make(chan struct{}) + h := newHarness(t, Config{Limiter: LimiterConfig{MaxConcurrentPerUser: 1}}, func(w http.ResponseWriter, _ *http.Request) { + entered <- struct{}{} + <-release + promUpstream(w, nil) + }) + // Registered after the harness so it runs before the servers close. + var once sync.Once + t.Cleanup(func() { once.Do(func() { close(release) }) }) + held := make(chan int, 2) + hold := func(token string) { + go func() { held <- h.status(http.MethodGet, metricsRoot+"api/v1/query?query=up", token) }() + select { + case <-entered: + case <-time.After(2 * time.Second): + t.Fatalf("request of %s did not reach the upstream", token) + } + } + hold("good") + refused := make(chan int, 1) + go func() { refused <- h.status(http.MethodPost, metricsRoot+"api/v1/query?query=up", "good") }() + select { + case got := <-refused: + if got != http.StatusUnprocessableEntity { + t.Fatalf("concurrent request answered %d, want 422", got) + } + case <-time.After(2 * time.Second): + t.Fatal("concurrent request of the same user reached the upstream") + } + hold("bob-token") + once.Do(func() { close(release) }) + for range 2 { + if got := <-held; got != http.StatusOK { + t.Fatalf("held request answered %d", got) + } + } +} + func TestDefaultMetricsUpstreamURL(t *testing.T) { s := NewServer(Config{ClusterDomain: "cozy.local", Metrics: MetricsConfig{RootTenant: "tenant-root"}}, nil, nil, nil) if got, want := s.cfg.Metrics.UpstreamURL("shortterm"), "http://vmselect-shortterm.tenant-root.svc.cozy.local:8481/select/0/prometheus"; got != want { From 6d6f05a1c7980cf369d607732daadb29df4351d8 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:17:15 +0700 Subject: [PATCH 314/518] fix(observability-proxy): refuse metrics durations vmselect would refuse Rounding a duration up to the millisecond overflows for values within a millisecond of the largest one, which turned such a step or timeout negative and let it past the range, points and timeout limits. vmselect refuses durations above 100 years itself, so the proxy now answers 400 for them before rounding. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricsparams.go | 7 ++++++- internal/observabilityproxy/metricsparams_test.go | 2 ++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index e74d2cbb5..fe9c8aafd 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -241,13 +241,18 @@ func parseStep(v string, lim MetricsLimits) (time.Duration, *HTTPError) { return step, nil } +// maxMetricsDuration is vmselect's own bound on step and timeout +// (maxDurationMsecs in lib/httputil); it also keeps the rounding below +// from overflowing. +const maxMetricsDuration = 100 * 365 * 24 * time.Hour + // parseMetricsDuration takes Prometheus float seconds or a duration such as // 1m30s. vmselect works in whole milliseconds and refuses anything shorter; // the value is rounded up to the millisecond so the limits are checked // against what vmselect will use, not against a fraction it drops. func parseMetricsDuration(key, v string) (time.Duration, *HTTPError) { d, err := parseDuration(v) - if err != nil || d < time.Millisecond { + if err != nil || d < time.Millisecond || d > maxMetricsDuration { return 0, errorf(http.StatusBadRequest, "invalid %s %q", key, v) } return (d + time.Millisecond - 1).Truncate(time.Millisecond), nil diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index ac9ff0ada..fc403191b 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -156,6 +156,8 @@ func TestBuildMetricsParamsLimits(t *testing.T) { {"points at the limit", "api/v1/query_range", rng(ago(10999*time.Minute), ago(0), "1m"), http.StatusOK}, {"points above the limit", "api/v1/query_range", rng(ago(11000*time.Minute), ago(0), "1m"), http.StatusUnprocessableEntity}, {"points above the limit after rounding the step up", "api/v1/query_range", rng(ago(22*time.Second), ago(0), "0.0019"), http.StatusUnprocessableEntity}, + {"overflowing step", "api/v1/query_range", rng(ago(time.Hour), ago(0), "2562047h47m16.854775807s"), http.StatusBadRequest}, + {"overflowing timeout", "api/v1/query", url.Values{"query": {"up"}, "timeout": {"2562047h47m16.854775807s"}}, http.StatusBadRequest}, {"step above the range limit", "api/v1/query_range", rng(ago(time.Hour), ago(0), "31d"), http.StatusUnprocessableEntity}, {"zero step", "api/v1/query_range", rng(ago(time.Hour), ago(0), "0"), http.StatusBadRequest}, {"sub-millisecond step", "api/v1/query_range", rng(ago(time.Hour), ago(0), "0.0001"), http.StatusBadRequest}, From 19030c7a1a286f724f92b91628973bfc26aed3aa Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:25:58 +0700 Subject: [PATCH 315/518] fix(observability-proxy): answer 429 to the metrics concurrency limit The logs root answers 429 when a user has too many requests in flight, so clients back off instead of treating the refusal as a bad query. The metrics root kept its own 422 for the same limit; it now matches. The rate limit is shared and already answers 429 with Retry-After, which the metrics test now expects too. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 2 +- internal/observabilityproxy/server_metrics_test.go | 11 +++++++---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 3e24be109..4760f7421 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -291,7 +291,7 @@ func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Rout } release, ok := s.limits.acquire(user) if !ok { - return errorf(http.StatusUnprocessableEntity, "too many concurrent requests") + return errorf(http.StatusTooManyRequests, "too many concurrent requests") } defer release() diff --git a/internal/observabilityproxy/server_metrics_test.go b/internal/observabilityproxy/server_metrics_test.go index 981644668..a1c07d944 100644 --- a/internal/observabilityproxy/server_metrics_test.go +++ b/internal/observabilityproxy/server_metrics_test.go @@ -151,8 +151,11 @@ func TestServerMetricsStatusCodes(t *testing.T) { h4 := newHarness(t, Config{Limiter: LimiterConfig{RatePerUser: 0.001, RateBurst: 1}}, promUpstream) h4.status(http.MethodGet, metricsRoot+"api/v1/query?query=up", "good") - if got := h4.status(http.MethodGet, metricsRoot+"api/v1/query?query=up", "good"); got != http.StatusUnprocessableEntity { - t.Errorf("rate-limited request answered %d, want 422", got) + limited := h4.do(http.MethodGet, metricsRoot+"api/v1/query?query=up", "good", nil, nil) + _ = limited.Body.Close() + if limited.StatusCode != http.StatusTooManyRequests || limited.Header.Get("Retry-After") != "1000" { + t.Errorf("rate-limited request answered %d with Retry-After %q, want 429 with 1000", + limited.StatusCode, limited.Header.Get("Retry-After")) } } @@ -180,8 +183,8 @@ func TestServerMetricsConcurrency(t *testing.T) { go func() { refused <- h.status(http.MethodPost, metricsRoot+"api/v1/query?query=up", "good") }() select { case got := <-refused: - if got != http.StatusUnprocessableEntity { - t.Fatalf("concurrent request answered %d, want 422", got) + if got != http.StatusTooManyRequests { + t.Fatalf("concurrent request answered %d, want 429", got) } case <-time.After(2 * time.Second): t.Fatal("concurrent request of the same user reached the upstream") From 560e4d7ba5c2bb7123fd1e1e618b1995fdaf3578 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:40:06 +0700 Subject: [PATCH 316/518] fix(observability-proxy): count the query lookback in the store choice The short-term store was picked from the request window alone, so an instant increase(x[7d]) or x offset 10d went to the store that keeps 72h and silently returned a wrong number. A lexical scan of the query now bounds how far back it reads: the sum of every [...] window and subquery step, positive offsets, and the step and silence interval vmselect reads before each rollup. An @ modifier, step-multiple units or any duration the scan cannot read make the bound unknown and pick the long-term store, which receives the same writes. The scan sees quoted label values too, so a false hit only moves a query to the long-term store. The query is still forwarded unchanged. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observabilityproxy/metricslookback.go | 172 ++++++++++++++++++ internal/observabilityproxy/metricsparams.go | 15 +- .../observabilityproxy/metricsparams_test.go | 109 ++++++++++- 3 files changed, 288 insertions(+), 8 deletions(-) create mode 100644 internal/observabilityproxy/metricslookback.go diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go new file mode 100644 index 000000000..17a586bb9 --- /dev/null +++ b/internal/observabilityproxy/metricslookback.go @@ -0,0 +1,172 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "regexp" + "strconv" + "strings" + "time" +) + +// instantDefaultStep is the step vmselect uses for an instant query without +// one (defaultStep in app/vmselect/prometheus); it is also the implicit +// rollup window of a bare selector. +const instantDefaultStep = 5 * time.Minute + +// silenceInterval is how far before a rollup or subquery window vmselect +// starts reading (maxSilenceInterval in app/vmselect/promql: the value of +// -search.minStalenessInterval, 5m when unset; cozystack does not set it). +const silenceInterval = 5 * time.Minute + +// queryDurationPart is the subset of the MetricsQL duration grammar +// (scanSingleDuration in metricsql) the scan understands. Step multiples +// (5i, $__interval), negative parts and upper-case units are left out on +// purpose: anything else makes the bound unknown. +var ( + queryDurationPart = regexp.MustCompile(`(\d+(?:\.\d+)?)(ms|s|m|h|d|w|y)`) + queryDuration = regexp.MustCompile(`^(?:\d+(?:\.\d+)?(?:ms|s|m|h|d|w|y))+$`) + querySeconds = regexp.MustCompile(`^\d+(?:\.\d+)?$`) +) + +var queryDurationUnits = map[string]time.Duration{ + "ms": time.Millisecond, "s": time.Second, "m": time.Minute, "h": time.Hour, + "d": 24 * time.Hour, "w": 7 * 24 * time.Hour, "y": 365 * 24 * time.Hour, +} + +// queryStart is the earliest instant a query evaluated from start may read; +// zero when the bound is unknown. +func queryStart(query string, start time.Time, step time.Duration) time.Time { + lookback, ok := queryLookback(query, step) + if !ok { + return time.Time{} + } + return start.Add(-lookback) +} + +// queryLookback is an upper bound on how far before its evaluation window +// a query reads, from a lexical scan that never parses the query. Every +// [...] and offset counts, including ones inside quoted label values: an +// extra hit only makes the bound larger or unknown, never smaller. Windows +// and offsets are summed rather than maxed, which also covers nesting. +// vmselect starts a rollup max(window, step) plus the silence interval +// before the first point, a subquery window+step+silence before it and +// aligns both down to a step; each step term uses the largest step seen. +func queryLookback(query string, step time.Duration) (time.Duration, bool) { + if strings.Contains(query, "@") { + return 0, false + } + var sum float64 + subqueries := 0 + maxStep := float64(step) + for rest := query; ; { + open := strings.IndexByte(rest, '[') + if open < 0 { + break + } + closing := strings.IndexByte(rest[open:], ']') + if closing < 0 { + return 0, false + } + window, subStep, isSubquery := strings.Cut(rest[open+1:open+closing], ":") + rest = rest[open+closing+1:] + if isSubquery { + subqueries++ + if strings.TrimSpace(window) == "" { + window = "0s" + } + if strings.TrimSpace(subStep) != "" { + d, ok := parseQueryDuration(subStep) + if !ok { + return 0, false + } + maxStep = max(maxStep, d) + } + } + d, ok := parseQueryDuration(window) + if !ok { + return 0, false + } + sum += d + } + lower := strings.ToLower(query) + for i := 0; ; { + k := strings.Index(lower[i:], "offset") + if k < 0 { + break + } + k += i + i = k + len("offset") + if k > 0 && continuesIdent(lower[k-1], false) || i < len(lower) && continuesIdent(lower[i], true) { + continue + } + arg := strings.TrimLeft(lower[i:], " \t\n\r") + negative := strings.HasPrefix(arg, "-") + if negative { + arg = strings.TrimLeft(arg[1:], " \t\n\r") + } + end := strings.IndexFunc(arg, func(r rune) bool { + return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'z' || r == '.') + }) + if end < 0 { + end = len(arg) + } + d, ok := parseQueryDuration(arg[:end]) + if !ok { + return 0, false + } + if !negative { + sum += d + } + } + n := float64(subqueries) + sum += (2+3*n)*maxStep + (1+n)*float64(silenceInterval) + if sum > float64(maxMetricsDuration) { + return 0, false + } + return time.Duration(sum), true +} + +// continuesIdent reports whether c, next to "offset", makes it part of an +// identifier rather than the keyword. Digits and dots only continue an +// identifier after it, so before it they still count as a keyword hit. +func continuesIdent(c byte, after bool) bool { + return c >= 'a' && c <= 'z' || c == '_' || c == ':' || c >= 0x80 || + after && (c >= '0' && c <= '9' || c == '.' || c == '\\') +} + +// parseQueryDuration returns nanoseconds as a float so that a sum of huge +// durations cannot wrap; the caller compares it against the bound. +func parseQueryDuration(s string) (float64, bool) { + s = strings.TrimSpace(s) + if querySeconds.MatchString(s) { + f, err := strconv.ParseFloat(s, 64) + return f * float64(time.Second), err == nil + } + if !queryDuration.MatchString(s) { + return 0, false + } + var d float64 + for _, m := range queryDurationPart.FindAllStringSubmatch(s, -1) { + f, err := strconv.ParseFloat(m[1], 64) + if err != nil { + return 0, false + } + d += f * float64(queryDurationUnits[m[2]]) + } + return d, true +} diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index fe9c8aafd..117f1b573 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -69,7 +69,8 @@ var labelName = regexp.MustCompile(`^[a-zA-Z_][a-zA-Z0-9_]*$`) type metricsRequest struct { path string params url.Values - // start is the earliest instant the request reads, for the store choice. + // start is the earliest instant the request reads, for the store choice; + // zero when that is unknown, which picks the long-term store. start time.Time } @@ -103,22 +104,24 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti if herr := copyQuery(out, in); herr != nil { return req, herr } - req.start = now + at := now if v := in.Get("time"); v != "" { t, herr := parseMetricsTime("time", v) if herr != nil { return req, herr } - req.start = t + at = t out.Set("time", formatTime(t)) } + step := instantDefaultStep if v := in.Get("step"); v != "" { - step, herr := parseStep(v, lim) - if herr != nil { + var herr *HTTPError + if step, herr = parseStep(v, lim); herr != nil { return req, herr } out.Set("step", formatSeconds(step)) } + req.start = queryStart(out.Get("query"), at, step) return req, nil case "api/v1/query_range": @@ -141,7 +144,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti if points := end.Sub(start)/step + 1; points > time.Duration(lim.MaxPoints) { return req, errorf(http.StatusUnprocessableEntity, "%d points per series exceed the limit of %d; raise step", points, lim.MaxPoints) } - req.start = start + req.start = queryStart(out.Get("query"), start, step) out.Set("start", formatTime(start)) out.Set("end", formatTime(end)) out.Set("step", formatSeconds(step)) diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index fc403191b..54e5acd04 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -224,7 +224,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) { if req.params.Get("end") != "1793793600.000" || req.params.Get("start") != "1793793300.000" { t.Errorf("labels window = %s..%s, want the last 5m", req.params.Get("start"), req.params.Get("end")) } - if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Has("time") || !q.start.Equal(testNow) { + if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Has("time") || !q.start.Equal(testNow.Add(-15*time.Minute)) { t.Errorf("instant query without time: params %v, start %v", q.params, q.start) } } @@ -241,7 +241,8 @@ func TestMetricsStoreByRange(t *testing.T) { {"instant within retention", "api/v1/query", url.Values{"query": {"up"}, "time": {unix(testNow.Add(-71 * time.Hour))}}, "shortterm"}, {"instant past retention", "api/v1/query", url.Values{"query": {"up"}, "time": {unix(testNow.Add(-73 * time.Hour))}}, "longterm"}, {"live graph", "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-6 * time.Hour))}, "end": {unix(testNow)}, "step": {"60"}}, "shortterm"}, - {"range fits exactly", "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-72 * time.Hour))}, "end": {unix(testNow)}, "step": {"600"}}, "shortterm"}, + {"range fits with lookback", "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-71 * time.Hour))}, "end": {unix(testNow)}, "step": {"600"}}, "shortterm"}, + {"range lookback past retention", "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-72 * time.Hour))}, "end": {unix(testNow)}, "step": {"600"}}, "longterm"}, {"range past retention", "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-7 * 24 * time.Hour))}, "end": {unix(testNow)}, "step": {"600"}}, "longterm"}, {"old window ending long ago", "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-5 * 24 * time.Hour))}, "end": {unix(testNow.Add(-4 * 24 * time.Hour))}, "step": {"600"}}, "longterm"}, {"labels default window", "api/v1/labels", url.Values{}, "shortterm"}, @@ -256,3 +257,107 @@ func TestMetricsStoreByRange(t *testing.T) { }) } } + +func TestQueryLookback(t *testing.T) { + tests := []struct { + query string + step time.Duration + want time.Duration + }{ + {`up`, 5 * time.Minute, 15 * time.Minute}, + {`increase(x[7d])`, 5 * time.Minute, 7*24*time.Hour + 15*time.Minute}, + {`rate(x[300])`, time.Minute, 12 * time.Minute}, + {`rate(x[ 1h30m ])`, time.Minute, 97 * time.Minute}, + {`max_over_time(x[1h:5m])`, time.Minute, 95 * time.Minute}, + {`max_over_time(x[1h:])`, time.Minute, 75 * time.Minute}, + {`max_over_time(max_over_time(rate(x[5m])[1h:1m])[1d:1h])`, time.Minute, 33*time.Hour + 20*time.Minute}, + {`x offset 10d`, 5 * time.Minute, 10*24*time.Hour + 15*time.Minute}, + {`sum(x OFFSET 2h)`, 5 * time.Minute, 135 * time.Minute}, + {`rate(x[5m] offset 1h)`, time.Minute, 72 * time.Minute}, + {`x offset -1h`, 5 * time.Minute, 15 * time.Minute}, + {`x offset - 1h`, 5 * time.Minute, 15 * time.Minute}, + {`node_offset_seconds + offset_total`, 5 * time.Minute, 15 * time.Minute}, + {`x{a="[1h]"}`, 5 * time.Minute, 75 * time.Minute}, + {`x{a="]"}[7d]`, 5 * time.Minute, 7*24*time.Hour + 15*time.Minute}, + } + for _, tt := range tests { + t.Run(tt.query, func(t *testing.T) { + got, ok := queryLookback(tt.query, tt.step) + if !ok || got != tt.want { + t.Fatalf("queryLookback = %s, %t; want %s, true", got, ok, tt.want) + } + }) + } +} + +func TestQueryLookbackUnknown(t *testing.T) { + for _, q := range []string{ + `x @ 123`, + `x @ start()`, + `x{a="a@b"}`, + `rate(x[5i])`, + `rate(x[$__interval])`, + `x[1h-5m]`, + `x[7D]`, + `x[1.d]`, + `x[]`, + `rate(x[7d)`, + `x offset`, + `x offset w`, + `WITH (w = 7d) rate(x[w])`, + `x{a="["}`, + `x{a="[5m"}[7d]`, + `x{path="/offset/1"}`, + `x[99999999999y]`, + `x offset 99999999999y`, + } { + t.Run(q, func(t *testing.T) { + if got, ok := queryLookback(q, 5*time.Minute); ok { + t.Fatalf("queryLookback = %s, true; want unknown", got) + } + }) + } +} + +func TestMetricsStoreByLookback(t *testing.T) { + cfg := MetricsConfig{ShortTermStore: "shortterm", LongTermStore: "longterm", ShortTermRetention: 72 * time.Hour} + rangeFrom := func(q string, ago time.Duration, step string) url.Values { + return url.Values{"query": {q}, "start": {unix(testNow.Add(-ago))}, "end": {unix(testNow)}, "step": {step}} + } + tests := []struct { + name string + endpoint string + in url.Values + want string + }{ + {"instant short window", "api/v1/query", url.Values{"query": {"rate(x[5m])"}}, "shortterm"}, + {"instant 7d window", "api/v1/query", url.Values{"query": {"increase(x[7d])"}}, "longterm"}, + {"instant offset 10d", "api/v1/query", url.Values{"query": {"x offset 10d"}}, "longterm"}, + {"instant negative offset", "api/v1/query", url.Values{"query": {"x offset -1h"}}, "shortterm"}, + {"instant at timestamp", "api/v1/query", url.Values{"query": {"x @ 123"}}, "longterm"}, + {"instant at start", "api/v1/query", url.Values{"query": {"x @ start()"}}, "longterm"}, + {"instant step window", "api/v1/query", url.Values{"query": {"rate(x[5i])"}}, "longterm"}, + {"instant step past retention", "api/v1/query", url.Values{"query": {"up"}, "step": {"259200"}}, "longterm"}, + {"instant subquery past retention", "api/v1/query", url.Values{"query": {"max_over_time(rate(x[5m])[4d:1h])"}}, "longterm"}, + {"quoted at sign", "api/v1/query", url.Values{"query": {`x{a="a@b"}`}}, "longterm"}, + {"quoted open bracket hides window", "api/v1/query", url.Values{"query": {`rate(x{a="[5m"}[7d])`}}, "longterm"}, + {"quoted close bracket before window", "api/v1/query", url.Values{"query": {`rate(x{a="]"}[7d])`}}, "longterm"}, + {"quoted offset before offset", "api/v1/query", url.Values{"query": {`x{a="offset 1m"} offset 10d`}}, "longterm"}, + {"range short window", "api/v1/query_range", rangeFrom("rate(x[1h])", 6*time.Hour, "60"), "shortterm"}, + {"range 7d window", "api/v1/query_range", rangeFrom("increase(x[7d])", 6*time.Hour, "60"), "longterm"}, + {"range offset past retention", "api/v1/query_range", rangeFrom("x offset 3d", time.Hour, "60"), "longterm"}, + {"range lookback fits", "api/v1/query_range", rangeFrom("rate(x[5m])", 70*time.Hour, "600"), "shortterm"}, + {"range lookback crosses retention", "api/v1/query_range", rangeFrom("rate(x[5m])", 71*time.Hour+50*time.Minute, "600"), "longterm"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + req := buildMetrics(t, tt.endpoint, tt.in) + if got := cfg.store(req.start, testNow); got != tt.want { + t.Fatalf("store = %s, want %s", got, tt.want) + } + if q := req.params.Get("query"); q != tt.in.Get("query") { + t.Fatalf("query forwarded as %q, want it unchanged", q) + } + }) + } +} From 2fad05b01f1dc35ed7c0e9d3a2c8467f00f81d88 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:46:37 +0700 Subject: [PATCH 317/518] fix(observability-proxy): count the latency offset in instant lookback vmselect moves an instant query evaluated near the current time back by -search.latencyOffset (30s). With a tiny explicit step the lookback bound did not cover that shift, so a window ending at the retention edge could still be sent to the short-term store. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricslookback.go | 6 ++++++ internal/observabilityproxy/metricsparams.go | 2 +- internal/observabilityproxy/metricsparams_test.go | 3 ++- 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index 17a586bb9..a1de3ca32 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -33,6 +33,12 @@ const instantDefaultStep = 5 * time.Minute // -search.minStalenessInterval, 5m when unset; cozystack does not set it). const silenceInterval = 5 * time.Minute +// latencyOffset is how far vmselect moves an instant query evaluated within +// it of the current time back (-search.latencyOffset, 30s by default and +// not set by cozystack); the proxy forwards neither nocache nor +// latency_offset. +const latencyOffset = 30 * time.Second + // queryDurationPart is the subset of the MetricsQL duration grammar // (scanSingleDuration in metricsql) the scan understands. Step multiples // (5i, $__interval), negative parts and upper-case units are left out on diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 117f1b573..5aae58479 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -121,7 +121,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti } out.Set("step", formatSeconds(step)) } - req.start = queryStart(out.Get("query"), at, step) + req.start = queryStart(out.Get("query"), at.Add(-latencyOffset), step) return req, nil case "api/v1/query_range": diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 54e5acd04..4b034c661 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -224,7 +224,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) { if req.params.Get("end") != "1793793600.000" || req.params.Get("start") != "1793793300.000" { t.Errorf("labels window = %s..%s, want the last 5m", req.params.Get("start"), req.params.Get("end")) } - if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Has("time") || !q.start.Equal(testNow.Add(-15*time.Minute)) { + if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Has("time") || !q.start.Equal(testNow.Add(-15*time.Minute-30*time.Second)) { t.Errorf("instant query without time: params %v, start %v", q.params, q.start) } } @@ -338,6 +338,7 @@ func TestMetricsStoreByLookback(t *testing.T) { {"instant at start", "api/v1/query", url.Values{"query": {"x @ start()"}}, "longterm"}, {"instant step window", "api/v1/query", url.Values{"query": {"rate(x[5i])"}}, "longterm"}, {"instant step past retention", "api/v1/query", url.Values{"query": {"up"}, "step": {"259200"}}, "longterm"}, + {"instant latency offset at the edge", "api/v1/query", url.Values{"query": {"default_rollup(x[71h54m59s])"}, "step": {"0.001"}}, "longterm"}, {"instant subquery past retention", "api/v1/query", url.Values{"query": {"max_over_time(rate(x[5m])[4d:1h])"}}, "longterm"}, {"quoted at sign", "api/v1/query", url.Values{"query": {`x{a="a@b"}`}}, "longterm"}, {"quoted open bracket hides window", "api/v1/query", url.Values{"query": {`rate(x{a="[5m"}[7d])`}}, "longterm"}, From 1ead35a37ae90294ae3134c786ba41c66d0d5a11 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:52:21 +0700 Subject: [PATCH 318/518] fix(observability-proxy): bound implicit subqueries and WITH templates The lookback scan assumed every window and offset applies once and explicitly. A rollup function or offset over anything but a selector is an implicit subquery in vmselect, so every "(" now counts as one level; a WITH template can apply one window several times, so WITH makes the bound unknown; and an offset written with digit separators was read as its first digits only, so "_" now ends in an unknown bound instead. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observabilityproxy/metricslookback.go | 40 +++++++++++++++---- .../observabilityproxy/metricsparams_test.go | 27 ++++++++----- 2 files changed, 50 insertions(+), 17 deletions(-) diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index a1de3ca32..ed7e9d166 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -66,12 +66,15 @@ func queryStart(query string, start time.Time, step time.Duration) time.Time { // queryLookback is an upper bound on how far before its evaluation window // a query reads, from a lexical scan that never parses the query. Every -// [...] and offset counts, including ones inside quoted label values: an -// extra hit only makes the bound larger or unknown, never smaller. Windows -// and offsets are summed rather than maxed, which also covers nesting. +// [...], offset and "(" counts, including ones inside quoted label values: +// an extra hit only makes the bound larger or unknown, never smaller. +// Windows and offsets are summed rather than maxed, which covers nesting; +// WITH is refused because a template applies one window several times. // vmselect starts a rollup max(window, step) plus the silence interval // before the first point, a subquery window+step+silence before it and -// aligns both down to a step; each step term uses the largest step seen. +// aligns both down to a step. A rollup function or offset over anything +// but a selector is an implicit subquery (getRollupExprArg), so every "(" +// is counted as one. Each step term uses the largest step seen. func queryLookback(query string, step time.Duration) (time.Duration, bool) { if strings.Contains(query, "@") { return 0, false @@ -117,7 +120,7 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { } k += i i = k + len("offset") - if k > 0 && continuesIdent(lower[k-1], false) || i < len(lower) && continuesIdent(lower[i], true) { + if !isKeyword(lower, k, i) { continue } arg := strings.TrimLeft(lower[i:], " \t\n\r") @@ -126,7 +129,7 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { arg = strings.TrimLeft(arg[1:], " \t\n\r") } end := strings.IndexFunc(arg, func(r rune) bool { - return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'z' || r == '.') + return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'z' || r == '.' || r == '_') }) if end < 0 { end = len(arg) @@ -139,7 +142,10 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { sum += d } } - n := float64(subqueries) + if hasKeyword(lower, "with") { + return 0, false + } + n := float64(subqueries + strings.Count(query, "(")) sum += (2+3*n)*maxStep + (1+n)*float64(silenceInterval) if sum > float64(maxMetricsDuration) { return 0, false @@ -147,6 +153,26 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { return time.Duration(sum), true } +// isKeyword reports whether lower[start:end] stands on its own rather than +// inside an identifier. +func isKeyword(lower string, start, end int) bool { + return !(start > 0 && continuesIdent(lower[start-1], false) || end < len(lower) && continuesIdent(lower[end], true)) +} + +func hasKeyword(lower, word string) bool { + for i := 0; ; { + k := strings.Index(lower[i:], word) + if k < 0 { + return false + } + k += i + i = k + len(word) + if isKeyword(lower, k, i) { + return true + } + } +} + // continuesIdent reports whether c, next to "offset", makes it part of an // identifier rather than the keyword. Digits and dots only continue an // identifier after it, so before it they still count as a keyword hit. diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 4b034c661..4a8565564 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -265,18 +265,19 @@ func TestQueryLookback(t *testing.T) { want time.Duration }{ {`up`, 5 * time.Minute, 15 * time.Minute}, - {`increase(x[7d])`, 5 * time.Minute, 7*24*time.Hour + 15*time.Minute}, - {`rate(x[300])`, time.Minute, 12 * time.Minute}, - {`rate(x[ 1h30m ])`, time.Minute, 97 * time.Minute}, - {`max_over_time(x[1h:5m])`, time.Minute, 95 * time.Minute}, - {`max_over_time(x[1h:])`, time.Minute, 75 * time.Minute}, - {`max_over_time(max_over_time(rate(x[5m])[1h:1m])[1d:1h])`, time.Minute, 33*time.Hour + 20*time.Minute}, + {`increase(x[7d])`, 5 * time.Minute, 7*24*time.Hour + 35*time.Minute}, + {`rate(x[300])`, time.Minute, 20 * time.Minute}, + {`rate(x[ 1h30m ])`, time.Minute, 105 * time.Minute}, + {`max_over_time(x[1h:5m])`, time.Minute, 115 * time.Minute}, + {`max_over_time(x[1h:])`, time.Minute, 83 * time.Minute}, + {`max_over_time(max_over_time(rate(x[5m])[1h:1m])[1d:1h])`, time.Minute, 42*time.Hour + 35*time.Minute}, + {`rate(rate(x))`, time.Hour, 2*(3*time.Hour+5*time.Minute) + 2*time.Hour + 5*time.Minute}, {`x offset 10d`, 5 * time.Minute, 10*24*time.Hour + 15*time.Minute}, - {`sum(x OFFSET 2h)`, 5 * time.Minute, 135 * time.Minute}, - {`rate(x[5m] offset 1h)`, time.Minute, 72 * time.Minute}, + {`sum(x OFFSET 2h)`, 5 * time.Minute, 155 * time.Minute}, + {`rate(x[5m] offset 1h)`, time.Minute, 80 * time.Minute}, {`x offset -1h`, 5 * time.Minute, 15 * time.Minute}, {`x offset - 1h`, 5 * time.Minute, 15 * time.Minute}, - {`node_offset_seconds + offset_total`, 5 * time.Minute, 15 * time.Minute}, + {`node_offset_seconds + offset_total + with_x`, 5 * time.Minute, 15 * time.Minute}, {`x{a="[1h]"}`, 5 * time.Minute, 75 * time.Minute}, {`x{a="]"}[7d]`, 5 * time.Minute, 7*24*time.Hour + 15*time.Minute}, } @@ -310,6 +311,9 @@ func TestQueryLookbackUnknown(t *testing.T) { `x{path="/offset/1"}`, `x[99999999999y]`, `x offset 99999999999y`, + `x offset 3_000_000`, + `WITH (f(q) = max_over_time(q[2d:1h])) f(f(x))`, + `with (s(m) = m offset 2d) s(s(x))`, } { t.Run(q, func(t *testing.T) { if got, ok := queryLookback(q, 5*time.Minute); ok { @@ -338,8 +342,11 @@ func TestMetricsStoreByLookback(t *testing.T) { {"instant at start", "api/v1/query", url.Values{"query": {"x @ start()"}}, "longterm"}, {"instant step window", "api/v1/query", url.Values{"query": {"rate(x[5i])"}}, "longterm"}, {"instant step past retention", "api/v1/query", url.Values{"query": {"up"}, "step": {"259200"}}, "longterm"}, - {"instant latency offset at the edge", "api/v1/query", url.Values{"query": {"default_rollup(x[71h54m59s])"}, "step": {"0.001"}}, "longterm"}, + {"instant latency offset at the edge", "api/v1/query", url.Values{"query": {"x[71h54m59s]"}, "step": {"0.001"}}, "longterm"}, {"instant subquery past retention", "api/v1/query", url.Values{"query": {"max_over_time(rate(x[5m])[4d:1h])"}}, "longterm"}, + {"instant offset with underscores", "api/v1/query", url.Values{"query": {"x offset 3_000_000"}}, "longterm"}, + {"instant WITH template", "api/v1/query", url.Values{"query": {"WITH (f(q) = max_over_time(q[2d:1h])) f(f(x))"}}, "longterm"}, + {"instant implicit subqueries", "api/v1/query", url.Values{"query": {"rate(rate(rate(rate(rate(rate(x))))))"}, "step": {"43200"}}, "longterm"}, {"quoted at sign", "api/v1/query", url.Values{"query": {`x{a="a@b"}`}}, "longterm"}, {"quoted open bracket hides window", "api/v1/query", url.Values{"query": {`rate(x{a="[5m"}[7d])`}}, "longterm"}, {"quoted close bracket before window", "api/v1/query", url.Values{"query": {`rate(x{a="]"}[7d])`}}, "longterm"}, From 0d3a0487e470152817e15796e83d4ca84d80d4a6 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:31:11 +0700 Subject: [PATCH 319/518] fix(observability-proxy): bound the metrics request lifetime The metrics root parsed its form before taking a concurrency slot, under no read deadline, and wrote its answer under no write deadline, so a client that sent its body slowly or stopped reading held the handler outside every limit. A store answer that broke off after the status line still ended as a clean 200. The metrics handler now follows the logs root: the slot is taken before the body is read under the request timeout, the answer must be taken by the request deadline, and a failed copy cuts the connection. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 23 +++++--- .../server_deadline_test.go | 56 +++++++++++++++++++ 2 files changed, 70 insertions(+), 9 deletions(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 4760f7421..50b6ac89e 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -280,20 +280,19 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT } func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Route, user string) *HTTPError { - r.Body = http.MaxBytesReader(w, r.Body, maxFormBytes) - if err := r.ParseForm(); err != nil { - return errorf(http.StatusBadRequest, "invalid form: %v", err) + release, ok := s.limits.acquire(user) + if !ok { + return errorf(http.StatusTooManyRequests, "too many concurrent requests") + } + defer release() + if herr := s.readForm(w, r); herr != nil { + return herr } now := s.now() req, herr := BuildMetricsParams(route.Endpoint, r.Form, route.Namespace, now, s.cfg.Metrics.Limits) if herr != nil { return herr } - release, ok := s.limits.acquire(user) - if !ok { - return errorf(http.StatusTooManyRequests, "too many concurrent requests") - } - defer release() ctx := r.Context() if s.cfg.RequestTimeout > 0 { @@ -310,9 +309,15 @@ func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Rout if ct := resp.Header.Get("Content-Type"); ct != "" { w.Header().Set("Content-Type", ct) } + // As for a bounded logs answer: the client must take it by the request + // deadline, and a break-off cuts the connection. + if deadline, ok := ctx.Deadline(); ok { + _ = http.NewResponseController(w).SetWriteDeadline(deadline) + } w.WriteHeader(resp.StatusCode) - if _, err := io.Copy(w, resp.Body); err != nil && !errors.Is(err, context.Canceled) { + if _, err := io.Copy(w, resp.Body); err != nil { slog.Debug("copying upstream response", "error", err) + return errAborted } return nil } diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index 7415b46fa..03d03f218 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -444,3 +444,59 @@ func TestServerStalledReviewAnswersInTime(t *testing.T) { }) } } + +// TestServerMetricsRequestLifetime repeats the body, write and break-off +// checks on the metrics root, which has its own handler. +func TestServerMetricsRequestLifetime(t *testing.T) { + query := metricsRoot + "api/v1/query?query=up" + t.Run("slow body", func(t *testing.T) { + h := newHarness(t, Config{ + RequestTimeout: 200 * time.Millisecond, + Limiter: LimiterConfig{MaxConcurrentPerUser: 1}, + }, promUpstream) + pipe := servePipe(t, h.server) + conn := pipe.send(t, "POST "+metricsRoot+"api/v1/query HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n"+ + "Content-Type: application/x-www-form-urlencoded\r\nContent-Length: 100\r\n\r\nquery=") + eventually(t, time.Second, func() bool { + return h.status(http.MethodGet, query, "good") == http.StatusTooManyRequests + }, "a metrics request with an unfinished body does not hold a concurrency slot") + if resp := readResponse(t, conn); resp.StatusCode != http.StatusRequestTimeout { + t.Fatalf("unfinished body answered %d, want 408", resp.StatusCode) + } + }) + t.Run("stalled client", func(t *testing.T) { + // Larger than the server's write buffer, so the handler itself + // blocks on the client. + big := `{"status":"success","data":"` + strings.Repeat("x", 64<<10) + `"}` + h := newHarness(t, Config{ + RequestTimeout: 200 * time.Millisecond, + Limiter: LimiterConfig{MaxConcurrentPerUser: 1}, + }, func(w http.ResponseWriter, _ *http.Request) { _, _ = io.WriteString(w, big) }) + pipe := servePipe(t, h.server) + pipe.send(t, "GET "+query+" HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n\r\n") + eventually(t, time.Second, func() bool { + return h.status(http.MethodGet, query, "good") == http.StatusTooManyRequests + }, "the stalled metrics request never held its slot") + eventually(t, 2*time.Second, func() bool { + return h.status(http.MethodGet, query, "good") == http.StatusOK + }, "a metrics client that stopped reading keeps its concurrency slot") + }) + t.Run("truncated answer", func(t *testing.T) { + h := newHarness(t, Config{}, func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Length", "100") + _, _ = io.WriteString(w, `{"status":"success",`) + }) + // Nothing has been flushed yet when the copy fails, so the + // connection may close before the status line. + req, _ := http.NewRequest(http.MethodGet, h.proxy.URL+query, nil) + req.Header.Set("Authorization", "Bearer good") + resp, err := http.DefaultClient.Do(req) + if err != nil { + return + } + defer func() { _ = resp.Body.Close() }() + if _, err := io.ReadAll(resp.Body); err == nil { + t.Fatal("a truncated store answer reached the client as a complete response") + } + }) +} From 6f5b0b666e068345d9d1814c3bffd7fa09b040ec Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:31:54 +0700 Subject: [PATCH 320/518] fix(observability-proxy): check metrics points on millisecond bounds The points limit was checked on the parsed bounds, but vmselect gets both cut to the millisecond. A start cut back by more than the end widened the forwarded range, so a 1ms step admitted at 11000 points reached vmselect as 11001. Times are now cut to the millisecond when they are parsed, so the range and points checks and the forwarded request see one window. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricsparams.go | 7 ++++--- internal/observabilityproxy/metricsparams_test.go | 3 +++ 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 5aae58479..44761a2ea 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -264,7 +264,8 @@ func parseMetricsDuration(key, v string) (time.Duration, *HTTPError) { // parseMetricsTime takes the two forms the Prometheus API defines: float // unix seconds and RFC3339. vmselect would also guess milliseconds from the // digit count and accept relative times; a guessed unit silently moves the -// window, so those are refused. +// window, so those are refused. The result is cut to the millisecond, as +// formatTime sends it, so the limits see the window vmselect will use. func parseMetricsTime(key, v string) (time.Time, *HTTPError) { if sec, frac, ok := strings.Cut(v, "."); isDigits(sec) && (!ok || isDigits(frac)) { if len(sec) > 10 { @@ -275,13 +276,13 @@ func parseMetricsTime(key, v string) (time.Time, *HTTPError) { if ok { nanos, _ = strconv.ParseInt((frac + "000000000")[:9], 10, 64) } - return time.Unix(secs, nanos).UTC(), nil + return time.UnixMilli(time.Unix(secs, nanos).UnixMilli()).UTC(), nil } t, err := time.Parse(time.RFC3339Nano, v) if err != nil || t.Before(time.Unix(0, 0)) { return time.Time{}, errorf(http.StatusBadRequest, "invalid %s %q", key, v) } - return t.UTC(), nil + return time.UnixMilli(t.UnixMilli()).UTC(), nil } func formatTime(t time.Time) string { diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 4a8565564..729af997a 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -156,6 +156,9 @@ func TestBuildMetricsParamsLimits(t *testing.T) { {"points at the limit", "api/v1/query_range", rng(ago(10999*time.Minute), ago(0), "1m"), http.StatusOK}, {"points above the limit", "api/v1/query_range", rng(ago(11000*time.Minute), ago(0), "1m"), http.StatusUnprocessableEntity}, {"points above the limit after rounding the step up", "api/v1/query_range", rng(ago(22*time.Second), ago(0), "0.0019"), http.StatusUnprocessableEntity}, + // 10.9999s admits 11000 points, but vmselect gets both bounds cut to + // the millisecond: 11.000s, 11001 points. + {"points above the limit once the bounds are cut to milliseconds", "api/v1/query_range", rng("2026-11-04T11:59:49.0009Z", "2026-11-04T12:00:00.0008Z", "0.001"), http.StatusUnprocessableEntity}, {"overflowing step", "api/v1/query_range", rng(ago(time.Hour), ago(0), "2562047h47m16.854775807s"), http.StatusBadRequest}, {"overflowing timeout", "api/v1/query", url.Values{"query": {"up"}, "timeout": {"2562047h47m16.854775807s"}}, http.StatusBadRequest}, {"step above the range limit", "api/v1/query_range", rng(ago(time.Hour), ago(0), "31d"), http.StatusUnprocessableEntity}, From eaadba4adddd8988c84b959de0f202bfb581d935 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:44:11 +0700 Subject: [PATCH 321/518] docs(observability-proxy): state what the metrics limits bound The comment claimed the limits bound everything a request makes vmselect do, but windows and offsets inside the expression pass through unchecked: they are bounded by the store's retention and by vmselect's own series and sample limits, not by the proxy. A reader taking the comment at its word would rely on a cost cap that does not exist. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricsparams.go | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 44761a2ea..414821302 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -26,7 +26,10 @@ import ( "time" ) -// MetricsLimits bound what one metrics request may make vmselect do. +// MetricsLimits bound the window, resolution and timeout a metrics request +// asks vmselect for. Windows and offsets inside the expression are not +// limited here: they cannot read past the store's retention, and vmselect +// caps the series and samples one query may touch. type MetricsLimits struct { MaxRange time.Duration MaxPoints int From b4ddb93b6f5e376ee298a6a4371ad33d1f03b433 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:55:03 +0700 Subject: [PATCH 322/518] docs(observability-proxy): name the timeout as the bound on rollup work vmselect's sample limit counts raw samples read, not the work of a rollup that revisits each sample across overlapping windows, so for that work the forwarded timeout is the limit that applies. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricsparams.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 414821302..5b4169050 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -28,8 +28,9 @@ import ( // MetricsLimits bound the window, resolution and timeout a metrics request // asks vmselect for. Windows and offsets inside the expression are not -// limited here: they cannot read past the store's retention, and vmselect -// caps the series and samples one query may touch. +// limited here: they cannot read past the store's retention, vmselect caps +// the series and samples one query may touch, and MaxTimeout bounds the +// work of rollups over long overlapping windows. type MetricsLimits struct { MaxRange time.Duration MaxPoints int From 843d360ee08597da49d89b0d98e08b9db8848f58 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:24:53 +0700 Subject: [PATCH 323/518] fix(observability-proxy): send Retry-After with the metrics concurrency 429 The metrics root has its own admission, so the hint added to the logs root did not reach it, and a console graph polling into the limit had no back-off hint. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server.go | 2 +- internal/observabilityproxy/server_metrics_test.go | 12 ++++++++---- 2 files changed, 9 insertions(+), 5 deletions(-) diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 50b6ac89e..eb540a271 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -282,7 +282,7 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Route, user string) *HTTPError { release, ok := s.limits.acquire(user) if !ok { - return errorf(http.StatusTooManyRequests, "too many concurrent requests") + return tooManyConcurrent(w) } defer release() if herr := s.readForm(w, r); herr != nil { diff --git a/internal/observabilityproxy/server_metrics_test.go b/internal/observabilityproxy/server_metrics_test.go index a1c07d944..ea47cf782 100644 --- a/internal/observabilityproxy/server_metrics_test.go +++ b/internal/observabilityproxy/server_metrics_test.go @@ -179,12 +179,16 @@ func TestServerMetricsConcurrency(t *testing.T) { } } hold("good") - refused := make(chan int, 1) - go func() { refused <- h.status(http.MethodPost, metricsRoot+"api/v1/query?query=up", "good") }() + refused := make(chan *http.Response, 1) + go func() { + resp := h.do(http.MethodPost, metricsRoot+"api/v1/query?query=up", "good", nil, nil) + _ = resp.Body.Close() + refused <- resp + }() select { case got := <-refused: - if got != http.StatusTooManyRequests { - t.Fatalf("concurrent request answered %d, want 429", got) + if got.StatusCode != http.StatusTooManyRequests || got.Header.Get("Retry-After") == "" { + t.Fatalf("concurrent request answered %d with Retry-After %q, want 429 with a value", got.StatusCode, got.Header.Get("Retry-After")) } case <-time.After(2 * time.Second): t.Fatal("concurrent request of the same user reached the upstream") From 3a5bcf2567fd6fc00da6085ac505b447c6a4c3e2 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:26:41 +0700 Subject: [PATCH 324/518] fix(observability-proxy): send durations vmselect reads back exactly vmselect reads step and timeout as float seconds multiplied by 1000 and truncated, so a step the proxy checked as 1001ms and sent as "1.001" arrived as 1000ms. The points limit was then checked against a coarser step than vmselect used: a limit of 11000 let 11010 points through, and many steps between 1ms and 100000s were affected. Durations are now written so that the truncation lands on the milliseconds the limits were checked with: the shortest form when it reads back exactly, otherwise with half a millisecond added. Step and timeout share the formatter, and a sweep test mirrors vmselect's parsing. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricsparams.go | 12 +++- .../observabilityproxy/metricsparams_test.go | 66 +++++++++++++++++++ 2 files changed, 77 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 5b4169050..54c46f527 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -294,6 +294,16 @@ func formatTime(t time.Time) string { return fmt.Sprintf("%d.%03d", ms/1000, ms%1000) } +// formatSeconds writes a whole-millisecond duration so that vmselect reads +// back exactly those milliseconds. vmselect multiplies the float seconds +// by 1e3 and truncates (httputil.GetDuration), so "1.001" arrives as +// 1000ms; where the shortest form truncates low, half a millisecond more +// lands the truncation on the intended value without reaching the next. func formatSeconds(d time.Duration) string { - return strconv.FormatFloat(float64(d.Milliseconds())/1000, 'f', -1, 64) + ms := d.Milliseconds() + s := strconv.FormatFloat(float64(ms)/1000, 'f', -1, 64) + if secs, _ := strconv.ParseFloat(s, 64); int64(secs*1e3) != ms { + s = fmt.Sprintf("%d.%03d5", ms/1000, ms%1000) + } + return s } diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 729af997a..a4957a19a 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -17,6 +17,7 @@ limitations under the License. package observabilityproxy import ( + "math" "net/http" "net/url" "strconv" @@ -372,3 +373,68 @@ func TestMetricsStoreByLookback(t *testing.T) { }) } } + +// vmselectMillis repeats how vmselect reads a duration argument +// (httputil.GetDuration in lib/httputil/duration.go, v1.136.22 and +// v1.139.0): float seconds, multiplied by 1e3 and truncated. +func vmselectMillis(t *testing.T, v string) int64 { + t.Helper() + secs, err := strconv.ParseFloat(v, 64) + if err != nil { + t.Fatalf("vmselect cannot parse %q: %v", v, err) + } + return int64(secs * 1e3) +} + +// TestFormatSecondsVmselectExact sweeps every step up to 2000s and the +// large end of the range: vmselect must read back the milliseconds the +// limits were checked with. +func TestFormatSecondsVmselectExact(t *testing.T) { + check := func(ms int64) { + if got := vmselectMillis(t, formatSeconds(time.Duration(ms)*time.Millisecond)); got != ms { + t.Fatalf("%dms formatted as %q reaches vmselect as %dms", ms, formatSeconds(time.Duration(ms)*time.Millisecond), got) + } + } + for ms := int64(1); ms <= 2_000_000; ms++ { + check(ms) + } + for ms := maxMetricsDuration.Milliseconds() - 100_000; ms <= maxMetricsDuration.Milliseconds(); ms++ { + check(ms) + } + // A spread of the range in between, on a stride no power of ten divides. + for ms := int64(2_000_001); ms < maxMetricsDuration.Milliseconds(); ms += 3_153_599_999 { + check(ms) + } + // Whole seconds keep their plain form. + if got := formatSeconds(90 * time.Second); got != "90" { + t.Errorf("90s formatted as %q", got) + } +} + +// TestBuildMetricsParamsPointsAsVmselectCounts: a 1.001s step is 1001ms +// to the proxy; vmselect must count the same 11000 points at the limit. +func TestBuildMetricsParamsPointsAsVmselectCounts(t *testing.T) { + end := testNow + step := 1001 * time.Millisecond + rng := func(points int64) url.Values { + start := end.Add(-time.Duration(points-1) * step) + return url.Values{"query": {"up"}, "start": {start.Format(time.RFC3339Nano)}, "end": {end.Format(time.RFC3339Nano)}, "step": {"1.001"}} + } + req := buildMetrics(t, "api/v1/query_range", rng(11000)) + stepMs := vmselectMillis(t, req.params.Get("step")) + // Times, unlike durations, are rounded (timeutil.ParseTimeMsec). + timeMs := func(v string) int64 { + secs, err := strconv.ParseFloat(v, 64) + if err != nil { + t.Fatal(err) + } + return int64(math.Round(secs * 1e3)) + } + startMs, endMs := timeMs(req.params.Get("start")), timeMs(req.params.Get("end")) + if points := (endMs-startMs)/stepMs + 1; stepMs != 1001 || points != 11000 { + t.Fatalf("vmselect reads step %dms and %d points, want 1001ms and 11000", stepMs, points) + } + if got := metricsStatus("api/v1/query_range", rng(11001)); got != http.StatusUnprocessableEntity { + t.Fatalf("11001 points answered %d, want 422", got) + } +} From d2feb9b7c2f15a62fa82f84329d6a7843be8d6f2 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:39:09 +0700 Subject: [PATCH 325/518] test(observability-proxy): pin the metrics concurrency Retry-After value The metrics concurrency test only checked that a Retry-After was present, while the rate limit test pins its value, so a change to the promised hint passed. It now expects the literal value. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server_metrics_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/server_metrics_test.go b/internal/observabilityproxy/server_metrics_test.go index ea47cf782..1ef5e761e 100644 --- a/internal/observabilityproxy/server_metrics_test.go +++ b/internal/observabilityproxy/server_metrics_test.go @@ -187,8 +187,8 @@ func TestServerMetricsConcurrency(t *testing.T) { }() select { case got := <-refused: - if got.StatusCode != http.StatusTooManyRequests || got.Header.Get("Retry-After") == "" { - t.Fatalf("concurrent request answered %d with Retry-After %q, want 429 with a value", got.StatusCode, got.Header.Get("Retry-After")) + if got.StatusCode != http.StatusTooManyRequests || got.Header.Get("Retry-After") != "1" { + t.Fatalf("concurrent request answered %d with Retry-After %q, want 429 with 1", got.StatusCode, got.Header.Get("Retry-After")) } case <-time.After(2 * time.Second): t.Fatal("concurrent request of the same user reached the upstream") From ec251cbd3b1db23a26b1961c968eace8d2807db0 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:21:21 +0700 Subject: [PATCH 326/518] test(observability-proxy): watch the metrics slot instead of probing The metrics slow-body and stalled-client tests probed the slot with requests of their own, which competed for the one slot under test and made them fail at random, under one shuffle seed reliably. They now read the limiter's slot count, as the logs tests do; 300 runs of the test and 20 shuffled runs of the package pass. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observabilityproxy/server_deadline_test.go | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index 03d03f218..abe1b3703 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -457,12 +457,12 @@ func TestServerMetricsRequestLifetime(t *testing.T) { pipe := servePipe(t, h.server) conn := pipe.send(t, "POST "+metricsRoot+"api/v1/query HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n"+ "Content-Type: application/x-www-form-urlencoded\r\nContent-Length: 100\r\n\r\nquery=") - eventually(t, time.Second, func() bool { - return h.status(http.MethodGet, query, "good") == http.StatusTooManyRequests - }, "a metrics request with an unfinished body does not hold a concurrency slot") + eventually(t, time.Second, func() bool { return slotsHeld(h) == 1 }, + "a metrics request with an unfinished body does not hold a concurrency slot") if resp := readResponse(t, conn); resp.StatusCode != http.StatusRequestTimeout { t.Fatalf("unfinished body answered %d, want 408", resp.StatusCode) } + eventually(t, time.Second, func() bool { return slotsHeld(h) == 0 }, "the slot of an unfinished body was not released") }) t.Run("stalled client", func(t *testing.T) { // Larger than the server's write buffer, so the handler itself @@ -474,12 +474,9 @@ func TestServerMetricsRequestLifetime(t *testing.T) { }, func(w http.ResponseWriter, _ *http.Request) { _, _ = io.WriteString(w, big) }) pipe := servePipe(t, h.server) pipe.send(t, "GET "+query+" HTTP/1.1\r\nHost: proxy\r\nAuthorization: Bearer good\r\n\r\n") - eventually(t, time.Second, func() bool { - return h.status(http.MethodGet, query, "good") == http.StatusTooManyRequests - }, "the stalled metrics request never held its slot") - eventually(t, 2*time.Second, func() bool { - return h.status(http.MethodGet, query, "good") == http.StatusOK - }, "a metrics client that stopped reading keeps its concurrency slot") + eventually(t, time.Second, func() bool { return slotsHeld(h) == 1 }, "the stalled metrics request never held its slot") + eventually(t, 2*time.Second, func() bool { return slotsHeld(h) == 0 }, + "a metrics client that stopped reading keeps its concurrency slot") }) t.Run("truncated answer", func(t *testing.T) { h := newHarness(t, Config{}, func(w http.ResponseWriter, _ *http.Request) { From 65944fcb79171fdf542192cb243fcd685231214d Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:23:09 +0700 Subject: [PATCH 327/518] fix(observability-proxy): send metrics times vmselect cannot misread Times went to vmselect as "%d.%03d" unix seconds. vmselect reads a number without "-" as a unix time whose unit it guesses from its size, so a time in the first 24.8 days of 1970 arrived as a thousand times as many seconds: a window the proxy checked as 8.8 days and 30 points reached vmselect as 2001..2025 and 30000 points, and the label and series APIs, which have no points check, scanned the whole retention. Times past 2038 were read as microseconds. Times now go out as RFC3339 with milliseconds, which vmselect parses as a date and never as a guessed unit, and times outside 1970..2262, which vmselect cannot hold as nanoseconds, are refused. A test repeats vmselect's parsing of the exact strings the proxy sends. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricsparams.go | 24 +++- .../observabilityproxy/metricsparams_test.go | 15 +-- .../observabilityproxy/vmselect_parse_test.go | 117 ++++++++++++++++++ 3 files changed, 140 insertions(+), 16 deletions(-) create mode 100644 internal/observabilityproxy/vmselect_parse_test.go diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 54c46f527..8beb347d0 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -18,6 +18,7 @@ package observabilityproxy import ( "fmt" + "math" "net/http" "net/url" "regexp" @@ -280,18 +281,33 @@ func parseMetricsTime(key, v string) (time.Time, *HTTPError) { if ok { nanos, _ = strconv.ParseInt((frac + "000000000")[:9], 10, 64) } - return time.UnixMilli(time.Unix(secs, nanos).UnixMilli()).UTC(), nil + return metricsTimeInRange(key, v, time.Unix(secs, nanos)) } t, err := time.Parse(time.RFC3339Nano, v) - if err != nil || t.Before(time.Unix(0, 0)) { + if err != nil { return time.Time{}, errorf(http.StatusBadRequest, "invalid %s %q", key, v) } + return metricsTimeInRange(key, v, t) +} + +// maxMetricsTime is the last instant vmselect can read: it takes an +// RFC3339 time through time.UnixNano (timeutil.ParseTimeAt). +var maxMetricsTime = time.Unix(0, math.MaxInt64) + +func metricsTimeInRange(key, v string, t time.Time) (time.Time, *HTTPError) { + if t.Before(time.Unix(0, 0)) || !t.Before(maxMetricsTime) { + return time.Time{}, errorf(http.StatusBadRequest, "invalid %s %q: outside 1970..2262", key, v) + } return time.UnixMilli(t.UnixMilli()).UTC(), nil } +// formatTime writes RFC3339 with milliseconds. vmselect reads a number +// without "-" as a unix time whose unit it guesses from its size +// (timeutil.getUnixTimestampNanoseconds): "1000000.000" is taken as +// 1000000000 seconds, and anything past 2038 in milliseconds as +// microseconds. A dated form never reaches that guess. func formatTime(t time.Time) string { - ms := t.UnixMilli() - return fmt.Sprintf("%d.%03d", ms/1000, ms%1000) + return t.UTC().Format("2006-01-02T15:04:05.000Z07:00") } // formatSeconds writes a whole-millisecond duration so that vmselect reads diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index a4957a19a..a4a435567 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -17,7 +17,6 @@ limitations under the License. package observabilityproxy import ( - "math" "net/http" "net/url" "strconv" @@ -198,7 +197,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) { "query": {"up"}, "start": {"2026-11-04T11:00:00.25Z"}, "end": {unix(testNow)}, "step": {"1m30s"}, }) want := url.Values{ - "query": {"up"}, "start": {"1793790000.250"}, "end": {"1793793600.000"}, "step": {"90"}, + "query": {"up"}, "start": {"2026-11-04T11:00:00.250Z"}, "end": {"2026-11-04T12:00:00.000Z"}, "step": {"90"}, "timeout": {"30"}, "extra_filters[]": {wantMetricsScope}, } if req.params.Encode() != want.Encode() { @@ -225,7 +224,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) { } req = buildMetrics(t, "api/v1/labels", url.Values{}) - if req.params.Get("end") != "1793793600.000" || req.params.Get("start") != "1793793300.000" { + if req.params.Get("end") != "2026-11-04T12:00:00.000Z" || req.params.Get("start") != "2026-11-04T11:55:00.000Z" { t.Errorf("labels window = %s..%s, want the last 5m", req.params.Get("start"), req.params.Get("end")) } if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Has("time") || !q.start.Equal(testNow.Add(-15*time.Minute-30*time.Second)) { @@ -422,15 +421,7 @@ func TestBuildMetricsParamsPointsAsVmselectCounts(t *testing.T) { } req := buildMetrics(t, "api/v1/query_range", rng(11000)) stepMs := vmselectMillis(t, req.params.Get("step")) - // Times, unlike durations, are rounded (timeutil.ParseTimeMsec). - timeMs := func(v string) int64 { - secs, err := strconv.ParseFloat(v, 64) - if err != nil { - t.Fatal(err) - } - return int64(math.Round(secs * 1e3)) - } - startMs, endMs := timeMs(req.params.Get("start")), timeMs(req.params.Get("end")) + startMs, endMs := vmselectTimeMillis(t, req.params.Get("start")), vmselectTimeMillis(t, req.params.Get("end")) if points := (endMs-startMs)/stepMs + 1; stepMs != 1001 || points != 11000 { t.Fatalf("vmselect reads step %dms and %d points, want 1001ms and 11000", stepMs, points) } diff --git a/internal/observabilityproxy/vmselect_parse_test.go b/internal/observabilityproxy/vmselect_parse_test.go new file mode 100644 index 000000000..6d75bbf7f --- /dev/null +++ b/internal/observabilityproxy/vmselect_parse_test.go @@ -0,0 +1,117 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "math" + "net/url" + "strconv" + "strings" + "testing" + "time" +) + +// vmselectTimeMillis repeats how vmselect v1.136.0-cluster reads a time +// argument (httputil.GetTime -> timeutil.ParseTimeMsec -> ParseTimeAt in +// lib/timeutil/time.go) for the forms a client or the proxy may send: a +// value without "-" is a number whose unit vmselect guesses from its size +// (getUnixTimestampNanoseconds), anything else with a zone is RFC3339 read +// through time.UnixNano. Relative and partial dates are left out; the +// proxy refuses them. +func vmselectTimeMillis(t *testing.T, s string) int64 { + t.Helper() + if strings.Contains(s, "-") { + ts, err := time.Parse(time.RFC3339, s) + if err != nil { + t.Fatalf("vmselect cannot parse %q: %v", s, err) + } + return int64(math.Round(float64(ts.UnixNano()) / 1e6)) + } + intPart, frac, _ := strings.Cut(s, ".") + n, err := strconv.ParseInt(intPart+frac, 10, 64) + if err != nil { + t.Fatalf("vmselect cannot parse %q: %v", s, err) + } + for e := len(frac); e%3 != 0; e++ { + n *= 10 + } + var nanos int64 + switch { + case n < 1<<31: + nanos = n * 1e9 + case n < 1e3*(1<<31): + nanos = n * 1e6 + case n < 1e6*(1<<31): + nanos = n * 1e3 + default: + nanos = n + } + return int64(math.Round(float64(nanos) / 1e6)) +} + +// TestVmselectReadsProxyTimes sends times across the range the proxy +// accepts, in both client forms, and checks that vmselect reads the +// forwarded value as the same instant the proxy validated. A number such +// as "1000000.000" is seconds to the proxy but milliseconds-as-seconds +// guesswork to vmselect, so the proxy must not forward one. +func TestVmselectReadsProxyTimes(t *testing.T) { + for _, client := range []string{ + "0", "1000000", "1762400", "2147483", "2147484", "1000000.5", + "1793793600", "1793793600.123", "2147483647", "2147483648", "9223372035", + "1970-01-01T00:00:00Z", "1970-01-12T13:46:40.001Z", "2026-11-04T12:00:00.123Z", "2262-01-01T00:00:00Z", + } { + want, herr := parseMetricsTime("time", client) + if herr != nil { + t.Fatalf("%q refused: %v", client, herr) + } + req := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}, "time": {client}}) + sent := req.params.Get("time") + if got := vmselectTimeMillis(t, sent); got != want.UnixMilli() { + t.Errorf("client time %q forwarded as %q reaches vmselect as %dms, want %dms", client, sent, got, want.UnixMilli()) + } + } +} + +// TestVmselectReadsSecondsWithFraction is the window that passed the proxy as +// 8.8 days and 30 points while vmselect, reading "1000000.000" as a +// thousand times that many seconds, saw 2001..2025 and 30000 points. +func TestVmselectReadsSecondsWithFraction(t *testing.T) { + in := url.Values{"query": {"up"}, "start": {"1000000"}, "end": {"1762400"}, "step": {"25414"}} + req := buildMetrics(t, "api/v1/query_range", in) + start, end := vmselectTimeMillis(t, req.params.Get("start")), vmselectTimeMillis(t, req.params.Get("end")) + step := vmselectMillis(t, req.params.Get("step")) + if start != 1000000*1000 || end != 1762400*1000 { + t.Fatalf("vmselect reads %dms..%dms, want 1000000s..1762400s", start, end) + } + if points := (end-start)/step + 1; points != 30 { + t.Fatalf("vmselect counts %d points, want the 30 the proxy checked", points) + } + labels := buildMetrics(t, "api/v1/labels", url.Values{"start": {"1000000"}, "end": {"1000300"}}) + if got := vmselectTimeMillis(t, labels.params.Get("start")); got != 1000000*1000 { + t.Fatalf("label API start reaches vmselect as %dms, want 1000000s", got) + } +} + +func TestParseMetricsTimeRange(t *testing.T) { + // vmselect reads an RFC3339 time through time.UnixNano, which is + // undefined from 2262-04-11 on. + for _, v := range []string{"9223372037", "9999999999", "2262-04-12T00:00:00Z", "9999-01-01T00:00:00Z"} { + if _, herr := parseMetricsTime("time", v); herr == nil { + t.Errorf("%q accepted, want refused", v) + } + } +} From 89fbdb136f8fc1e7b4af07d9f81f534e4dd26f51 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:32:21 +0700 Subject: [PATCH 328/518] test(observability-proxy): cover the 1970 bound of metrics times Only the upper half of the refused 1970..2262 range had a test, so the lower check could be dropped unnoticed. vmselect clamps a time before 1970 to 0, which would answer for a window other than the one asked for. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/vmselect_parse_test.go | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/vmselect_parse_test.go b/internal/observabilityproxy/vmselect_parse_test.go index 6d75bbf7f..1fb23d18f 100644 --- a/internal/observabilityproxy/vmselect_parse_test.go +++ b/internal/observabilityproxy/vmselect_parse_test.go @@ -109,7 +109,9 @@ func TestVmselectReadsSecondsWithFraction(t *testing.T) { func TestParseMetricsTimeRange(t *testing.T) { // vmselect reads an RFC3339 time through time.UnixNano, which is // undefined from 2262-04-11 on. - for _, v := range []string{"9223372037", "9999999999", "2262-04-12T00:00:00Z", "9999-01-01T00:00:00Z"} { + // Before 1970 vmselect clamps to 0, which would quietly answer for a + // window other than the one asked for. + for _, v := range []string{"9223372037", "9999999999", "2262-04-12T00:00:00Z", "9999-01-01T00:00:00Z", "1969-12-31T23:59:59Z", "1969-12-31T23:59:59.999Z"} { if _, herr := parseMetricsTime("time", v); herr == nil { t.Errorf("%q accepted, want refused", v) } From 0c5d1cd93f6220b5497c0c6c479c112ffa3961d9 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:32:42 +0700 Subject: [PATCH 329/518] fix(observability-proxy): keep the vmselect timeout within the request's The timeout forwarded to vmselect could be set longer than the proxy's own request timeout. The proxy then cut a query vmselect was still allowed to run and answered 504 for a store that was within its time. Such a combination now stops the proxy at startup. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 4 ++++ cmd/observability-proxy/main_test.go | 3 +++ 2 files changed, 7 insertions(+) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index 6acfb1e08..a8ac36efd 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -134,6 +134,10 @@ func parseFlags(args []string) (options, error) { return o, errors.New("--metrics-max-points must be at least 2") case o.logs.MaxTailResume < 0: return o, errors.New("--logs-max-tail-resume must not be negative") + // Past the request timeout the proxy cuts a query vmselect is still + // allowed to run, and answers 504 for a store that is within its time. + case o.requestTimeout > 0 && o.metrics.Limits.MaxTimeout > o.requestTimeout: + return o, errors.New("--metrics-max-timeout must not exceed --request-timeout") case o.requestTimeout < 0 || o.tailIdleTimeout < 0 || o.authCacheTTL < 0 || o.shutdownTimeout < 0: return o, errors.New("timeouts and the auth cache TTL must not be negative") // NaN passes every comparison and +Inf never empties a bucket; either diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index be89e20d5..f27317515 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -103,6 +103,9 @@ func TestParseFlagsInvalid(t *testing.T) { {"--metrics-max-points=1"}, {"--metrics-max-timeout=0s"}, {"--unknown"}, + // The proxy would cut the request while vmselect is still within + // the timeout it was given. + {"--request-timeout=10s", "--metrics-max-timeout=30s"}, {"unexpected", "--root-tenant=tenant-x"}, } { if _, err := parseFlags(args); err == nil { From 3e2472280e776beb8060bd65d11a251aa9206474 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:38:50 +0700 Subject: [PATCH 330/518] fix(observability-proxy): keep the vmselect timeout below the request's An equal vmselect timeout still lost the race: the proxy's deadline starts before the request reaches vmselect, so the proxy cut first and answered 504 for a store within its time. The vmselect timeout must now be strictly below a set request timeout, the flag help says so, and tests pin the accepted side too, including no request timeout. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 11 ++++++----- cmd/observability-proxy/main_test.go | 14 ++++++++++++++ 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index a8ac36efd..a3687fe08 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -91,7 +91,7 @@ func parseFlags(args []string) (options, error) { fs.DurationVar(&o.metrics.ShortTermRetention, "metrics-shortterm-retention", 72*time.Hour, "retention of the short-term store; older windows are read from the long-term one") fs.DurationVar(&o.metrics.Limits.MaxRange, "metrics-max-range", 30*24*time.Hour, "longest [start, end] and step a metrics request may use") fs.IntVar(&o.metrics.Limits.MaxPoints, "metrics-max-points", 11000, "most points per series a range query may return") - fs.DurationVar(&o.metrics.Limits.MaxTimeout, "metrics-max-timeout", 30*time.Second, "timeout forwarded to vmselect; a longer client timeout is clamped") + fs.DurationVar(&o.metrics.Limits.MaxTimeout, "metrics-max-timeout", 30*time.Second, "timeout forwarded to vmselect; a longer client timeout is clamped; must be below --request-timeout when that is set") fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables") fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained requests per second per user; 0 disables") fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user") @@ -134,10 +134,11 @@ func parseFlags(args []string) (options, error) { return o, errors.New("--metrics-max-points must be at least 2") case o.logs.MaxTailResume < 0: return o, errors.New("--logs-max-tail-resume must not be negative") - // Past the request timeout the proxy cuts a query vmselect is still - // allowed to run, and answers 504 for a store that is within its time. - case o.requestTimeout > 0 && o.metrics.Limits.MaxTimeout > o.requestTimeout: - return o, errors.New("--metrics-max-timeout must not exceed --request-timeout") + // The proxy's deadline starts before the request reaches vmselect, so + // at an equal or longer timeout it cuts a query vmselect is still + // allowed to run and answers 504 for a store that is within its time. + case o.requestTimeout > 0 && o.metrics.Limits.MaxTimeout >= o.requestTimeout: + return o, errors.New("--metrics-max-timeout must be below --request-timeout") case o.requestTimeout < 0 || o.tailIdleTimeout < 0 || o.authCacheTTL < 0 || o.shutdownTimeout < 0: return o, errors.New("timeouts and the auth cache TTL must not be negative") // NaN passes every comparison and +Inf never empties a bucket; either diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index f27317515..6531550e9 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -106,6 +106,7 @@ func TestParseFlagsInvalid(t *testing.T) { // The proxy would cut the request while vmselect is still within // the timeout it was given. {"--request-timeout=10s", "--metrics-max-timeout=30s"}, + {"--request-timeout=30s", "--metrics-max-timeout=30s"}, {"unexpected", "--root-tenant=tenant-x"}, } { if _, err := parseFlags(args); err == nil { @@ -113,3 +114,16 @@ func TestParseFlagsInvalid(t *testing.T) { } } } + +// The vmselect timeout must stay strictly below a request timeout, and is +// free when the request has none. +func TestParseFlagsTimeoutPairs(t *testing.T) { + for _, args := range [][]string{ + {"--request-timeout=31s", "--metrics-max-timeout=30s"}, + {"--request-timeout=0", "--metrics-max-timeout=10m"}, + } { + if _, err := parseFlags(args); err != nil { + t.Errorf("parseFlags(%v) refused: %v", args, err) + } + } +} From 41b3237c07718e79f8bb1faf96b04d62d4e1c592 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 00:46:05 +0700 Subject: [PATCH 331/518] refactor(observability-proxy): state the keyword and duration tests positively staticcheck flags the negated disjunctions; naming the duration characters and spelling out both word boundaries reads the same way the rule is stated. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricslookback.go | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index ed7e9d166..e55d189a5 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -128,9 +128,7 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { if negative { arg = strings.TrimLeft(arg[1:], " \t\n\r") } - end := strings.IndexFunc(arg, func(r rune) bool { - return !(r >= '0' && r <= '9' || r >= 'a' && r <= 'z' || r == '.' || r == '_') - }) + end := strings.IndexFunc(arg, func(r rune) bool { return !inDuration(r) }) if end < 0 { end = len(arg) } @@ -156,7 +154,12 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { // isKeyword reports whether lower[start:end] stands on its own rather than // inside an identifier. func isKeyword(lower string, start, end int) bool { - return !(start > 0 && continuesIdent(lower[start-1], false) || end < len(lower) && continuesIdent(lower[end], true)) + return (start == 0 || !continuesIdent(lower[start-1], false)) && + (end == len(lower) || !continuesIdent(lower[end], true)) +} + +func inDuration(r rune) bool { + return r >= '0' && r <= '9' || r >= 'a' && r <= 'z' || r == '.' || r == '_' } func hasKeyword(lower, word string) bool { From 16bcaa447dc17d1060b5f47212c26a93e7666553 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 03:38:59 +0700 Subject: [PATCH 332/518] test(observability-proxy): cover stalled reviews on the metrics root The metrics handler shares the bounded review calls with the logs handler; a POST to it with an unread body must answer 503 as well rather than wait on a stalled kube-apiserver. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server_deadline_test.go | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index abe1b3703..6b367229c 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -421,15 +421,20 @@ func stalledAPIServer(t *testing.T, stallTokens bool) kubernetes.Interface { // a bounded review call ends the request. func TestServerStalledReviewAnswersInTime(t *testing.T) { for _, tc := range []struct { - name string + name, path string stallTokens bool - }{{"token review", true}, {"subject access review", false}} { + }{ + {"logs token review", logsRoot + "query", true}, + {"logs subject access review", logsRoot + "query", false}, + {"metrics token review", metricsRoot + "api/v1/query", true}, + {"metrics subject access review", metricsRoot + "api/v1/query", false}, + } { t.Run(tc.name, func(t *testing.T) { h := newHarness(t, Config{RequestTimeout: 200 * time.Millisecond}, nil) h.server.auth = NewAuthorizer(stalledAPIServer(t, tc.stallTokens), 10*time.Second) got := make(chan int, 1) go func() { - resp := h.do(http.MethodPost, logsRoot+"query", "good", url.Values{"query": {"*"}}, nil) + resp := h.do(http.MethodPost, tc.path, "good", url.Values{"query": {"up"}}, nil) _ = resp.Body.Close() got <- resp.StatusCode }() From b3288be9038656d74efa06a0e519deb426f21c9b Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 04:03:06 +0700 Subject: [PATCH 333/518] test(observability-proxy): cite only sources the repository has The flag defaults test named a design document that is not in the repository as the origin of the cost limits; the limits stand on their own as flag defaults. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main_test.go | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index 6531550e9..bf7ac396b 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -32,8 +32,7 @@ func TestParseFlagsDefaults(t *testing.T) { if o.authCacheTTL != 10*time.Second || o.logs.MaxTailResume != 15*time.Minute || o.logs.MaxRange != 7*24*time.Hour { t.Errorf("limit defaults changed: %+v", o) } - // Retention and store names of packages/system/monitoring/values.yaml; - // cost limits of the metrics proposal (Design 7). + // Retention and store names of packages/system/monitoring/values.yaml. m := o.metrics if m.ShortTermStore != "shortterm" || m.LongTermStore != "longterm" || m.ShortTermRetention != 72*time.Hour { t.Errorf("metrics store defaults changed: %+v", m) From ecb5de6051b9ece64dacf12dfa0a1aa9b29634d8 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:57:50 +0700 Subject: [PATCH 334/518] feat(observability-proxy): map application plurals to their kinds Paths and RBAC name an application by its plural, while the lineage labels that scope its logs and series carry the kind. The mapping comes from ApplicationDefinitions, kept in one place so both application roots scope by the same kind. A plural declared for two kinds is refused rather than resolved by cache order. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appkinds.go | 97 ++++++++++++++++++++ internal/observabilityproxy/appkinds_test.go | 77 ++++++++++++++++ 2 files changed, 174 insertions(+) create mode 100644 internal/observabilityproxy/appkinds.go create mode 100644 internal/observabilityproxy/appkinds_test.go diff --git a/internal/observabilityproxy/appkinds.go b/internal/observabilityproxy/appkinds.go new file mode 100644 index 000000000..5e545f390 --- /dev/null +++ b/internal/observabilityproxy/appkinds.go @@ -0,0 +1,97 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "errors" + "fmt" + "slices" + + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/tools/cache" +) + +// ApplicationDefinitionsResource is where the plural of every application +// kind is declared (api/v1alpha1/applicationdefinitions_types.go). +var ApplicationDefinitionsResource = schema.GroupVersionResource{Group: "cozystack.io", Version: "v1alpha1", Resource: "applicationdefinitions"} + +// ErrKindsNotSynced means the ApplicationDefinitions cache has not been +// filled yet, so an unknown plural cannot be told from a missing one. +var ErrKindsNotSynced = errors.New("application definitions not synced") + +const pluralIndex = "plural" + +// KindLookup maps an application plural, as it appears in paths and RBAC, +// to the kind the lineage labels carry. An unknown plural is "", nil. +type KindLookup interface { + Kind(plural string) (string, error) +} + +// ApplicationPluralIndexers indexes ApplicationDefinitions by +// spec.application.plural; the informer feeding ApplicationKinds needs it. +func ApplicationPluralIndexers() cache.Indexers { + return cache.Indexers{pluralIndex: func(obj any) ([]string, error) { + u, ok := obj.(*unstructured.Unstructured) + if !ok { + return nil, nil + } + plural, _, _ := unstructured.NestedString(u.Object, "spec", "application", "plural") + if plural == "" { + return nil, nil + } + return []string{plural}, nil + }} +} + +// ApplicationKinds is the one place the proxy maps a plural to its kind; +// both application roots use it. +type ApplicationKinds struct { + indexer cache.Indexer + hasSynced func() bool +} + +func NewApplicationKinds(indexer cache.Indexer, hasSynced func() bool) *ApplicationKinds { + return &ApplicationKinds{indexer: indexer, hasSynced: hasSynced} +} + +func (k *ApplicationKinds) Kind(plural string) (string, error) { + if !k.hasSynced() { + return "", ErrKindsNotSynced + } + objs, err := k.indexer.ByIndex(pluralIndex, plural) + if err != nil { + return "", err + } + var kinds []string + for _, obj := range objs { + kind, _, _ := unstructured.NestedString(obj.(*unstructured.Unstructured).Object, "spec", "application", "kind") + if kind != "" && !slices.Contains(kinds, kind) { + kinds = append(kinds, kind) + } + } + switch len(kinds) { + case 0: + return "", nil + case 1: + return kinds[0], nil + } + // Picking one would scope the request to whichever definition the + // cache happens to list first. + slices.Sort(kinds) + return "", fmt.Errorf("plural %q is declared for kinds %v", plural, kinds) +} diff --git a/internal/observabilityproxy/appkinds_test.go b/internal/observabilityproxy/appkinds_test.go new file mode 100644 index 000000000..6eb8d60bc --- /dev/null +++ b/internal/observabilityproxy/appkinds_test.go @@ -0,0 +1,77 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "errors" + "testing" + + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/client-go/tools/cache" +) + +func appDefinition(name, kind, plural string) *unstructured.Unstructured { + return &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "cozystack.io/v1alpha1", + "kind": "ApplicationDefinition", + "metadata": map[string]any{"name": name}, + "spec": map[string]any{"application": map[string]any{"kind": kind, "plural": plural, "singular": name}}, + }} +} + +func newTestKinds(t *testing.T, synced bool, defs ...*unstructured.Unstructured) *ApplicationKinds { + t.Helper() + indexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, ApplicationPluralIndexers()) + for _, d := range defs { + if err := indexer.Add(d); err != nil { + t.Fatal(err) + } + } + return NewApplicationKinds(indexer, func() bool { return synced }) +} + +func TestApplicationKinds(t *testing.T) { + kinds := newTestKinds(t, true, + appDefinition("postgres", "Postgres", "postgreses"), + appDefinition("vm-instance", "VMInstance", "vminstances"), + appDefinition("broken", "", "brokens"), + ) + for plural, want := range map[string]string{"postgreses": "Postgres", "vminstances": "VMInstance", "postgres": "", "brokens": "", "": ""} { + got, err := kinds.Kind(plural) + if err != nil || got != want { + t.Errorf("Kind(%q) = %q, %v; want %q", plural, got, err, want) + } + } +} + +func TestApplicationKindsNotSynced(t *testing.T) { + kinds := newTestKinds(t, false, appDefinition("postgres", "Postgres", "postgreses")) + if _, err := kinds.Kind("postgreses"); !errors.Is(err, ErrKindsNotSynced) { + t.Fatalf("Kind before sync: err = %v, want ErrKindsNotSynced", err) + } +} + +func TestApplicationKindsAmbiguous(t *testing.T) { + kinds := newTestKinds(t, true, appDefinition("a", "Postgres", "postgreses"), appDefinition("b", "Postgresql", "postgreses")) + if kind, err := kinds.Kind("postgreses"); err == nil { + t.Fatalf("two kinds for one plural resolved to %q", kind) + } + same := newTestKinds(t, true, appDefinition("a", "Postgres", "postgreses"), appDefinition("b", "Postgres", "postgreses")) + if kind, err := same.Kind("postgreses"); err != nil || kind != "Postgres" { + t.Fatalf("duplicate definitions of one kind: %q, %v", kind, err) + } +} From 2f600d5354f672b5e53d382c248d867624298dc2 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 05:59:44 +0700 Subject: [PATCH 335/518] feat(observability-proxy): serve the application logs root A viewer of one application gets that application's lines without writing a namespace query and filtering by hand, and a grant narrowed to one application by resourceNames is honoured on every logs endpoint, tail and the pickers included. The namespace filter stays and the application's identity is ANDed next to it as a second stream filter, so the result can only be narrower than the namespace root. The plural is mapped to its kind only after the caller is authorized, so an anonymous client cannot probe which applications exist, and the proxy is not ready until the ApplicationDefinitions cache is. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 20 ++- internal/observabilityproxy/server.go | 43 +++++-- .../observabilityproxy/server_app_test.go | 114 ++++++++++++++++++ .../observabilityproxy/server_metrics_test.go | 2 +- internal/observabilityproxy/server_test.go | 21 +++- .../observability-proxy/templates/rbac.yaml | 5 + .../observability-proxy/tests/rbac_test.yaml | 5 +- 7 files changed, 192 insertions(+), 18 deletions(-) create mode 100644 internal/observabilityproxy/server_app_test.go diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index a3687fe08..1de6e6e14 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -39,6 +39,8 @@ import ( "github.com/prometheus/client_golang/prometheus/collectors" "github.com/prometheus/client_golang/prometheus/promhttp" "k8s.io/apimachinery/pkg/util/validation" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/dynamic/dynamicinformer" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/cache" @@ -178,13 +180,24 @@ func run(o options) error { return fmt.Errorf("building client: %w", err) } + dynClient, err := dynamic.NewForConfig(restConfig) + if err != nil { + return fmt.Errorf("building dynamic client: %w", err) + } + factory := informers.NewSharedInformerFactory(client, 0) namespaces := factory.Core().V1().Namespaces() lister := namespaces.Lister() + appDefs := dynamicinformer.NewDynamicSharedInformerFactory(dynClient, 0). + ForResource(observabilityproxy.ApplicationDefinitionsResource).Informer() + if err := appDefs.AddIndexers(observabilityproxy.ApplicationPluralIndexers()); err != nil { + return fmt.Errorf("indexing application definitions: %w", err) + } factory.Start(ctx.Done()) + go appDefs.Run(ctx.Done()) var ready atomic.Bool go func() { - if cache.WaitForCacheSync(ctx.Done(), namespaces.Informer().HasSynced) { + if cache.WaitForCacheSync(ctx.Done(), namespaces.Informer().HasSynced, appDefs.HasSynced) { ready.Store(true) } }() @@ -198,13 +211,14 @@ func run(o options) error { Limiter: o.limiter, RequestTimeout: o.requestTimeout, TailIdleTimeout: o.tailIdleTimeout, - }, observabilityproxy.NewAuthorizer(client, o.authCacheTTL), observabilityproxy.NewStoreResolver(lister, namespaces.Informer().HasSynced, o.rootTenant), reg) + }, observabilityproxy.NewAuthorizer(client, o.authCacheTTL), observabilityproxy.NewStoreResolver(lister, namespaces.Informer().HasSynced, o.rootTenant), + observabilityproxy.NewApplicationKinds(appDefs.GetIndexer(), appDefs.HasSynced), reg) probes := http.NewServeMux() probes.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { _, _ = io.WriteString(w, "ok") }) probes.HandleFunc("/readyz", func(w http.ResponseWriter, _ *http.Request) { if !ready.Load() { - http.Error(w, "namespace cache not synced", http.StatusServiceUnavailable) + http.Error(w, "namespace or application definition cache not synced", http.StatusServiceUnavailable) return } _, _ = io.WriteString(w, "ok") diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index eb540a271..617971bf2 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -73,13 +73,14 @@ type Server struct { cfg Config auth *Authorizer stores StoreLookup + kinds KindLookup client *http.Client limits *limiter now func() time.Time requests *prometheus.CounterVec } -func NewServer(cfg Config, auth *Authorizer, stores StoreLookup, reg prometheus.Registerer) *Server { +func NewServer(cfg Config, auth *Authorizer, stores StoreLookup, kinds KindLookup, reg prometheus.Registerer) *Server { if cfg.UpstreamURL == nil { domain := cfg.ClusterDomain cfg.UpstreamURL = func(store string) string { @@ -96,6 +97,7 @@ func NewServer(cfg Config, auth *Authorizer, stores StoreLookup, reg prometheus. cfg: cfg, auth: auth, stores: stores, + kinds: kinds, // No client timeout: it would cut live tails. Non-tail requests are // bounded by a context deadline instead. client: &http.Client{Transport: http.DefaultTransport.(*http.Transport).Clone()}, @@ -172,12 +174,7 @@ func writeError(w http.ResponseWriter, e *HTTPError) { http.Error(w, e.Message, e.Code) } -// serve answers a root it has no handler for before authenticating, so an -// unknown path costs no review calls. func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTTPError { - if route.Kind != NamespaceLogs && route.Kind != NamespaceMetrics { - return errorf(http.StatusNotFound, "not found") - } user, herr := s.authorize(r, route) if herr != nil { return herr @@ -190,8 +187,17 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT w.Header().Set("Retry-After", strconv.FormatInt(int64(math.Ceil(wait.Seconds())), 10)) return errorf(http.StatusTooManyRequests, "request rate limit exceeded") } - if route.Kind == NamespaceMetrics { + var kind string + if route.Kind == ApplicationLog || route.Kind == ApplicationMetrics { + if kind, herr = s.applicationKind(route.Plural); herr != nil { + return herr + } + } + switch route.Kind { + case NamespaceMetrics: return s.serveMetrics(w, r, route, user) + case ApplicationMetrics: + return errorf(http.StatusNotFound, "not found") } endpoint, ok := strings.CutPrefix(route.Endpoint, "select/logsql/") if !ok { @@ -214,6 +220,9 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT if herr != nil { return herr } + if kind != "" { + params.Add("extra_stream_filters", applicationStreamFilter(kind, route.Name)) + } store, err := s.stores.Resolve(route.Namespace) if err != nil { @@ -322,6 +331,26 @@ func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Rout return nil } +func (s *Server) applicationKind(plural string) (string, *HTTPError) { + kind, err := s.kinds.Kind(plural) + if err != nil { + slog.Error("mapping application plural to kind", "plural", plural, "error", err) + return "", errorf(http.StatusServiceUnavailable, "cannot resolve the application kind") + } + if kind == "" { + return "", errorf(http.StatusNotFound, "unknown application resource %q", plural) + } + return kind, nil +} + +// applicationStreamFilter selects the streams Fluent Bit labels with the +// pod's lineage. vlselect ANDs each extra_stream_filters value into the +// query and every subquery (Query.AddExtraFilters in lib/logstorage), so +// it narrows the namespace filter sent next to it. +func applicationStreamFilter(kind, name string) string { + return "{apps_cozystack_io_application_kind=" + strconv.Quote(kind) + ",apps_cozystack_io_application_name=" + strconv.Quote(name) + "}" +} + // post sends params as a form with only the given headers, so nothing the // client sent reaches the store, and tells an unreachable store (502) from // one that missed its deadline or, for a tail, its idle timeout (504). diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go new file mode 100644 index 000000000..0a76ab9a5 --- /dev/null +++ b/internal/observabilityproxy/server_app_test.go @@ -0,0 +1,114 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "io" + "net/http" + "net/url" + "slices" + "testing" + + authorizationv1 "k8s.io/api/authorization/v1" +) + +const appRoot = "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/" + +// appPolicy lets alice read every postgres in tenant-a, as the view role's +// apps.cozystack.io wildcard does, and bob only "a", as a grant narrowed +// with resourceNames does. +func appPolicy(user string, a *authorizationv1.ResourceAttributes) bool { + if a.Namespace != "tenant-a" || a.Verb != "get" || a.Group != "apps.cozystack.io" || a.Resource != "postgreses" || + (a.Subresource != "log" && a.Subresource != "metrics") { + return false + } + return user == "alice" || a.Name == "a" +} + +func TestServerApplicationLogsScope(t *testing.T) { + h := newPolicyHarness(t, Config{}, nil, appPolicy, nil) + hostile := url.Values{ + "query": {"error"}, + "field": {"level"}, + "extra_filters": {`{kubernetes_namespace_name="tenant-b"}`}, + "extra_stream_filters": {`{apps_cozystack_io_application_name=~".*"}`}, + "AccountID": {"9"}, + } + headers := map[string]string{"AccountID": "9", "ProjectID": "9"} + endpoints := []string{"query", "hits", "stats_query_range", "tail", "field_values", "stream_field_values", "facets"} + for _, ep := range endpoints { + resp := h.do(http.MethodPost, appRoot+"db.main/log/select/logsql/"+ep+"?extra_stream_filters=x", "good", hostile, headers) + _, _ = io.Copy(io.Discard, resp.Body) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusOK { + t.Fatalf("%s answered %d", ep, resp.StatusCode) + } + } + want := authorizationv1.ResourceAttributes{Verb: "get", Group: "apps.cozystack.io", Resource: "postgreses", Subresource: "log", Name: "db.main", Namespace: "tenant-a"} + if got := h.kube.lastSAR.Load().ResourceAttributes; *got != want { + t.Fatalf("SubjectAccessReview = %+v, want %+v", *got, want) + } + reqs := h.requests() + if len(reqs) != len(endpoints) { + t.Fatalf("upstream saw %d requests, want %d", len(reqs), len(endpoints)) + } + // Two values: vlselect ANDs every extra_stream_filters value into the + // query and each of its subqueries. + wantFilters := []string{wantScope, `{apps_cozystack_io_application_kind="Postgres",apps_cozystack_io_application_name="db.main"}`} + for i, r := range reqs { + if r.path != "/select/logsql/"+endpoints[i] { + t.Errorf("upstream path = %s, want %s", r.path, endpoints[i]) + } + if got := r.form["extra_stream_filters"]; !slices.Equal(got, wantFilters) { + t.Errorf("%s: extra_stream_filters = %q, want %q", endpoints[i], got, wantFilters) + } + if r.form.Has("extra_filters") || r.form.Has("AccountID") || r.headers.Get("AccountID") != "0" || r.headers.Get("ProjectID") != "0" { + t.Errorf("%s: client scope reached the store: %v %v", endpoints[i], r.form, r.headers) + } + } +} + +func TestServerApplicationKinds(t *testing.T) { + tests := []struct { + name string + kinds KindLookup + path string + want int + }{ + {"unknown plural logs", nil, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/mysqls/a/log/select/logsql/query", http.StatusNotFound}, + {"unknown plural metrics", nil, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/mysqls/a/metrics/api/v1/query?query=up", http.StatusNotFound}, + {"not synced logs", newTestKinds(t, false), appRoot + "a/log/select/logsql/query", http.StatusServiceUnavailable}, + {"not synced metrics", newTestKinds(t, false), appRoot + "a/metrics/api/v1/query?query=up", http.StatusServiceUnavailable}, + {"ambiguous plural", newTestKinds(t, true, appDefinition("x", "Postgres", "postgreses"), appDefinition("y", "Pg", "postgreses")), appRoot + "a/log/select/logsql/query", http.StatusServiceUnavailable}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + h := newPolicyHarness(t, Config{}, nil, func(string, *authorizationv1.ResourceAttributes) bool { return true }, tt.kinds) + if got := h.status(http.MethodGet, tt.path, "good"); got != tt.want { + t.Fatalf("status = %d, want %d", got, tt.want) + } + if n := len(h.requests()); n != 0 { + t.Fatalf("upstream saw %d requests", n) + } + }) + } + // The kind is looked up only for an authorized caller. + h := newPolicyHarness(t, Config{}, nil, appPolicy, newTestKinds(t, false)) + if got := h.status(http.MethodGet, appRoot+"b/log/select/logsql/query", "bob-token"); got != http.StatusForbidden { + t.Fatalf("denied caller answered %d, want 403", got) + } +} diff --git a/internal/observabilityproxy/server_metrics_test.go b/internal/observabilityproxy/server_metrics_test.go index 1ef5e761e..1a3d1de10 100644 --- a/internal/observabilityproxy/server_metrics_test.go +++ b/internal/observabilityproxy/server_metrics_test.go @@ -203,7 +203,7 @@ func TestServerMetricsConcurrency(t *testing.T) { } func TestDefaultMetricsUpstreamURL(t *testing.T) { - s := NewServer(Config{ClusterDomain: "cozy.local", Metrics: MetricsConfig{RootTenant: "tenant-root"}}, nil, nil, nil) + s := NewServer(Config{ClusterDomain: "cozy.local", Metrics: MetricsConfig{RootTenant: "tenant-root"}}, nil, nil, nil, nil) if got, want := s.cfg.Metrics.UpstreamURL("shortterm"), "http://vmselect-shortterm.tenant-root.svc.cozy.local:8481/select/0/prometheus"; got != want { t.Fatalf("upstream = %s, want %s", got, want) } diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index 2670d0e3e..b541a18e1 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -55,10 +55,19 @@ func (m mapStores) Resolve(ns string) (string, error) { return m[ns], nil } // (alice) and "bob-token" (bob), both allowed list logs and metrics in // tenant-a only, and to an upstream served by handler. func newHarness(t *testing.T, cfg Config, handler http.HandlerFunc) *harness { - h := &harness{t: t} - h.kube = newFakeKube(map[string]string{"good": "alice", "bob-token": "bob"}, func(_ string, a *authorizationv1.ResourceAttributes) bool { + return newPolicyHarness(t, cfg, handler, func(_ string, a *authorizationv1.ResourceAttributes) bool { return a.Namespace == "tenant-a" && (a.Resource == "logs" || a.Resource == "metrics") - }) + }, nil) +} + +// newPolicyHarness is newHarness with the given SubjectAccessReview policy +// and plural-to-kind mapping; nil kinds knows only postgreses. +func newPolicyHarness(t *testing.T, cfg Config, handler http.HandlerFunc, allow func(string, *authorizationv1.ResourceAttributes) bool, kinds KindLookup) *harness { + h := &harness{t: t} + h.kube = newFakeKube(map[string]string{"good": "alice", "bob-token": "bob"}, allow) + if kinds == nil { + kinds = newTestKinds(t, true, appDefinition("postgres", "Postgres", "postgreses")) + } h.upstream = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if err := r.ParseForm(); err != nil { t.Errorf("upstream ParseForm: %v", err) @@ -90,7 +99,7 @@ func newHarness(t *testing.T, cfg Config, handler http.HandlerFunc) *harness { if cfg.Logs == (LogsLimits{}) { cfg.Logs = testLimits() } - h.server = NewServer(cfg, NewAuthorizer(h.kube.client, 10*time.Second), mapStores{"tenant-a": "tenant-a-store"}, nil) + h.server = NewServer(cfg, NewAuthorizer(h.kube.client, 10*time.Second), mapStores{"tenant-a": "tenant-a-store"}, kinds, nil) h.server.now = func() time.Time { return testNow } h.proxy = httptest.NewServer(h.server) t.Cleanup(h.proxy.Close) @@ -154,8 +163,8 @@ func TestServerStatusCodes(t *testing.T) { {"unknown metrics endpoint", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/api/v1/status/tsdb", "good", http.StatusNotFound}, {"metrics in another namespace", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-b/metrics/api/v1/query?query=up", "good", http.StatusForbidden}, {"metrics without token", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/metrics/api/v1/query?query=up", "", http.StatusUnauthorized}, - {"application log not served yet", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/log/select/logsql/query", "good", http.StatusNotFound}, - {"application metrics not served yet", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/metrics/api/v1/query", "good", http.StatusNotFound}, + {"application log without a grant", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/log/select/logsql/query", "good", http.StatusForbidden}, + {"application metrics without a grant", http.MethodGet, "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/db/metrics/api/v1/query", "good", http.StatusForbidden}, {"tenant_ids not proxied", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/select/tenant_ids", "good", http.StatusNotFound}, {"unknown logsql endpoint", http.MethodGet, logsRoot + "stats_query", "good", http.StatusNotFound}, {"logs root itself", http.MethodGet, "/apis/cozystack.io/v1alpha1/namespaces/tenant-a/logs/", "good", http.StatusNotFound}, diff --git a/packages/system/observability-proxy/templates/rbac.yaml b/packages/system/observability-proxy/templates/rbac.yaml index 3c5392903..f3f67127c 100644 --- a/packages/system/observability-proxy/templates/rbac.yaml +++ b/packages/system/observability-proxy/templates/rbac.yaml @@ -15,6 +15,11 @@ rules: - apiGroups: [""] resources: ["namespaces"] verbs: ["list", "watch"] +# Application roots map the plural in the path to the kind the lineage +# labels carry, from an informer cache as well. +- apiGroups: ["cozystack.io"] + resources: ["applicationdefinitions"] + verbs: ["list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding diff --git a/packages/system/observability-proxy/tests/rbac_test.yaml b/packages/system/observability-proxy/tests/rbac_test.yaml index ef96c65d7..5382c2ab6 100644 --- a/packages/system/observability-proxy/tests/rbac_test.yaml +++ b/packages/system/observability-proxy/tests/rbac_test.yaml @@ -12,7 +12,7 @@ release: namespace: cozy-observability-proxy tests: - - it: lets the proxy review tokens and access, and list and watch namespaces + - it: lets the proxy review tokens and access, and list and watch namespaces and application definitions template: templates/rbac.yaml documentSelector: path: kind @@ -30,6 +30,9 @@ tests: - apiGroups: [""] resources: ["namespaces"] verbs: ["list", "watch"] + - apiGroups: ["cozystack.io"] + resources: ["applicationdefinitions"] + verbs: ["list", "watch"] - it: binds the proxy role to the proxy ServiceAccount in the release namespace template: templates/rbac.yaml From dcdf0adab3171c0139766f0999cfb975d0ff97c1 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 06:03:40 +0700 Subject: [PATCH 336/518] feat(observability-proxy): serve the application metrics root Graphs of one application need its series without the caller knowing which pods and volumes it ran on. Exporter series carry the identity the scrape class stamps, but cAdvisor and kubelet volume series do not, so the scope ORs three branches: the identity, and the application's pods and PVCs as kube-state-metrics labelled them over the window. Each branch also carries the namespace, so the scope is never wider than the namespace root. The lookup reaches back by the query's own lookback, bounded by a flag, so pods replaced inside an expression's range still count. A failed lookup only drops its branch. A list over its cap answers 422 instead of being truncated, which would hide series without saying so. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 5 + cmd/observability-proxy/main_test.go | 10 +- internal/observabilityproxy/appscope.go | 158 ++++++++++++++++ internal/observabilityproxy/appscope_test.go | 116 ++++++++++++ internal/observabilityproxy/metricsparams.go | 20 ++- internal/observabilityproxy/server.go | 18 +- .../observabilityproxy/server_app_test.go | 170 ++++++++++++++++++ internal/observabilityproxy/server_test.go | 3 + 8 files changed, 492 insertions(+), 8 deletions(-) create mode 100644 internal/observabilityproxy/appscope.go create mode 100644 internal/observabilityproxy/appscope_test.go diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index 1de6e6e14..4a99aa4cc 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -94,6 +94,9 @@ func parseFlags(args []string) (options, error) { fs.DurationVar(&o.metrics.Limits.MaxRange, "metrics-max-range", 30*24*time.Hour, "longest [start, end] and step a metrics request may use") fs.IntVar(&o.metrics.Limits.MaxPoints, "metrics-max-points", 11000, "most points per series a range query may return") fs.DurationVar(&o.metrics.Limits.MaxTimeout, "metrics-max-timeout", 30*time.Second, "timeout forwarded to vmselect; a longer client timeout is clamped; must be below --request-timeout when that is set") + fs.DurationVar(&o.metrics.App.Lookback, "metrics-app-lookback", 24*time.Hour, "how far before the window the application root looks up the application's pods and PVCs at most") + fs.IntVar(&o.metrics.App.MaxPods, "metrics-app-max-pods", 500, "most pods of one application in the lookup window; more answer 422") + fs.IntVar(&o.metrics.App.MaxPVCs, "metrics-app-max-pvcs", 500, "most PVCs of one application in the lookup window; more answer 422") fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables") fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained requests per second per user; 0 disables") fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user") @@ -134,6 +137,8 @@ func parseFlags(args []string) (options, error) { return o, errors.New("--metrics-shortterm-retention, --metrics-max-range and --metrics-max-timeout must be positive") case o.metrics.Limits.MaxPoints < 2: return o, errors.New("--metrics-max-points must be at least 2") + case o.metrics.App.Lookback < 0 || o.metrics.App.MaxPods < 1 || o.metrics.App.MaxPVCs < 1: + return o, errors.New("--metrics-app-lookback must not be negative and --metrics-app-max-pods and --metrics-app-max-pvcs must be positive") case o.logs.MaxTailResume < 0: return o, errors.New("--logs-max-tail-resume must not be negative") // The proxy's deadline starts before the request reaches vmselect, so diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index bf7ac396b..6b5c60d7d 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -40,6 +40,9 @@ func TestParseFlagsDefaults(t *testing.T) { if m.Limits.MaxRange != 30*24*time.Hour || m.Limits.MaxPoints != 11000 || m.Limits.MaxTimeout != 30*time.Second { t.Errorf("metrics limit defaults changed: %+v", m.Limits) } + if m.App.Lookback != 24*time.Hour || m.App.MaxPods != 500 || m.App.MaxPVCs != 500 { + t.Errorf("application scope defaults changed: %+v", m.App) + } if m.RootTenant != o.rootTenant { t.Errorf("metrics root tenant = %q, want --root-tenant", m.RootTenant) } @@ -63,13 +66,15 @@ func TestParseFlagsMetricsOverrides(t *testing.T) { o, err := parseFlags([]string{ "--root-tenant=tenant-x", "--metrics-shortterm-store=fast", "--metrics-longterm-store=slow", "--metrics-shortterm-retention=24h", "--metrics-max-range=168h", "--metrics-max-points=500", "--metrics-max-timeout=10s", + "--metrics-app-lookback=0s", "--metrics-app-max-pods=7", "--metrics-app-max-pvcs=3", }) if err != nil { t.Fatal(err) } m := o.metrics if m.RootTenant != "tenant-x" || m.ShortTermStore != "fast" || m.LongTermStore != "slow" || m.ShortTermRetention != 24*time.Hour || - m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second { + m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second || + m.App.Lookback != 0 || m.App.MaxPods != 7 || m.App.MaxPVCs != 3 { t.Errorf("overrides not applied: %+v", m) } } @@ -101,6 +106,9 @@ func TestParseFlagsInvalid(t *testing.T) { {"--metrics-max-range=0s"}, {"--metrics-max-points=1"}, {"--metrics-max-timeout=0s"}, + {"--metrics-app-lookback=-1h"}, + {"--metrics-app-max-pods=0"}, + {"--metrics-app-max-pvcs=0"}, {"--unknown"}, // The proxy would cut the request while vmselect is still within // the timeout it was given. diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go new file mode 100644 index 000000000..b969cc502 --- /dev/null +++ b/internal/observabilityproxy/appscope.go @@ -0,0 +1,158 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "context" + "encoding/json" + "io" + "log/slog" + "net/http" + "net/url" + "regexp" + "strconv" + "strings" + "time" +) + +// AppScopeLimits bound the pod and PVC lookup of the application metrics +// root. +type AppScopeLimits struct { + // Lookback is how far before the window the lookup reaches at most, so + // pods gone before the window but inside an expression's range count. + Lookback time.Duration + MaxPods int + MaxPVCs int +} + +// maxLookupBytes bounds a lookup answer; the caps keep a valid one far +// below it. +const maxLookupBytes = 8 << 20 + +// The lineage labels as the scrape class writes them on series, and as +// kube-state-metrics writes them on kube_pod_labels and +// kube_persistentvolumeclaim_labels. +const ( + seriesKindLabel = "apps_cozystack_io_application_kind" + seriesNameLabel = "apps_cozystack_io_application_name" + ksmKindLabel = "label_" + seriesKindLabel + ksmNameLabel = "label_" + seriesNameLabel +) + +func identitySelector(ns, kindLabel, nameLabel, kind, name string) string { + return "{namespace=" + strconv.Quote(ns) + "," + kindLabel + "=" + strconv.Quote(kind) + "," + nameLabel + "=" + strconv.Quote(name) + "}" +} + +// nameBranch matches label against any of names within ns. With no names +// it is "": a branch of pod=~"^()$" would match every series without a pod +// label. +func nameBranch(ns, label string, names []string) string { + if len(names) == 0 { + return "" + } + quoted := make([]string, len(names)) + for i, n := range names { + quoted[i] = regexp.QuoteMeta(n) + } + return "{namespace=" + strconv.Quote(ns) + "," + label + "=~" + strconv.Quote("^("+strings.Join(quoted, "|")+")$") + "}" +} + +// applicationSelectors is the application scope: series stamped with the +// application's identity, plus series of its pods and PVCs for those not +// stamped (cAdvisor, kubelet volume stats). Every branch carries the +// namespace, so their OR stays inside it. +// +// Within the namespace, both the identity and the pod and PVC lists come +// from lineage labels on pods and PVCs. They separate applications for a +// grant narrowed by resourceNames only while whoever creates a pod or PVC +// cannot choose those labels; the lineage webhook keeps them as set on an +// object it finds no owner for, and the logs scope rests on the same. +func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { + out := []string{identitySelector(ns, seriesKindLabel, seriesNameLabel, kind, name)} + for _, b := range []string{nameBranch(ns, "pod", pods), nameBranch(ns, "persistentvolumeclaim", pvcs)} { + if b != "" { + out = append(out, b) + } + } + return out +} + +// applicationScope looks the application's pods and PVCs up in store over +// the request window, reaching back by the query's own lookback up to the +// configured limit. A failed lookup drops its branch, which only narrows +// the answer; a list over its cap is refused, because truncating it would +// silently drop series. +func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) { + lim := s.cfg.Metrics.App + start := req.from.Add(-lim.Lookback) + if req.start.After(start) { + start = req.start + } + var lists [2][]string + for i, l := range []struct { + label, metric string + max int + }{{"pod", "kube_pod_labels", lim.MaxPods}, {"persistentvolumeclaim", "kube_persistentvolumeclaim_labels", lim.MaxPVCs}} { + match := l.metric + identitySelector(ns, ksmKindLabel, ksmNameLabel, kind, name) + names, err := s.lookupNames(ctx, r, store, l.label, match, ns, start, req.to, l.max) + if err != nil { + slog.Warn("application lookup failed; its branch is left out", "label", l.label, "namespace", ns, "kind", kind, "name", name, "error", err) + continue + } + if len(names) > l.max { + return nil, errorf(http.StatusUnprocessableEntity, "the application has more than %d %ss in this window; use a shorter range", l.max, l.label) + } + lists[i] = names + } + return applicationSelectors(ns, kind, name, lists[0], lists[1]), nil +} + +// lookupNames asks for at most max+1 values, enough to tell a list over +// the cap without reading all of it. +func (s *Server) lookupNames(ctx context.Context, r *http.Request, store, label, match, ns string, start, end time.Time, max int) ([]string, error) { + params := url.Values{} + params.Set("match[]", match) + params.Set("extra_filters[]", namespaceSelector(ns)) + params.Set("start", formatTime(start)) + params.Set("end", formatTime(end)) + params.Set("limit", strconv.Itoa(max+1)) + resp, herr := s.post(ctx, r, s.cfg.Metrics.UpstreamURL(store)+"/api/v1/label/"+label+"/values", params, "metrics store", nil) + if herr != nil { + return nil, herr + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != http.StatusOK { + return nil, errorf(resp.StatusCode, "metrics store answered %s", resp.Status) + } + var body struct { + Status string `json:"status"` + Data []string `json:"data"` + } + if err := json.NewDecoder(io.LimitReader(resp.Body, maxLookupBytes)).Decode(&body); err != nil { + return nil, err + } + if body.Status != "success" { + return nil, errorf(http.StatusBadGateway, "metrics store answered status %q", body.Status) + } + names := body.Data[:0] + for _, n := range body.Data { + if n != "" { + names = append(names, n) + } + } + return names, nil +} diff --git a/internal/observabilityproxy/appscope_test.go b/internal/observabilityproxy/appscope_test.go new file mode 100644 index 000000000..af028b867 --- /dev/null +++ b/internal/observabilityproxy/appscope_test.go @@ -0,0 +1,116 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "net/url" + "regexp" + "slices" + "strconv" + "testing" + "time" +) + +const wantIdentityBranch = `{namespace="tenant-a",apps_cozystack_io_application_kind="Postgres",apps_cozystack_io_application_name="db.main"}` + +func TestApplicationSelectors(t *testing.T) { + tests := []struct { + name string + pods, pvcs []string + want []string + }{ + {"identity only", nil, nil, []string{wantIdentityBranch}}, + {"pods", []string{"db-main-1", "db.main-2"}, nil, []string{ + wantIdentityBranch, + `{namespace="tenant-a",pod=~"^(db-main-1|db\\.main-2)$"}`, + }}, + {"pvcs", nil, []string{"data-db-main-1"}, []string{ + wantIdentityBranch, + `{namespace="tenant-a",persistentvolumeclaim=~"^(data-db-main-1)$"}`, + }}, + {"all three", []string{"p"}, []string{"v"}, []string{ + wantIdentityBranch, + `{namespace="tenant-a",pod=~"^(p)$"}`, + `{namespace="tenant-a",persistentvolumeclaim=~"^(v)$"}`, + }}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := applicationSelectors("tenant-a", "Postgres", "db.main", tt.pods, tt.pvcs); !slices.Equal(got, tt.want) { + t.Fatalf("got %q\nwant %q", got, tt.want) + } + }) + } +} + +// DNS-1123 keeps most metacharacters out of names, but the names come +// back from the store, so the encoder must not rely on that. +func TestNameBranchEscapes(t *testing.T) { + names := []string{`a.b`, `x|.*`, `)|(.*`, `q"} or {namespace=~".+`, `back\slash`, `$^+?[]{}`} + sel := nameBranch("tenant-a", "pod", names) + m := regexp.MustCompile(`^\{namespace="tenant-a",pod=~(".*")\}$`).FindStringSubmatch(sel) + if m == nil { + t.Fatalf("selector %s is not one namespace and one pod matcher", sel) + } + pattern, err := strconv.Unquote(m[1]) + if err != nil { + t.Fatalf("pod matcher %s does not unquote: %v", m[1], err) + } + re := regexp.MustCompile(pattern) + for _, n := range names { + if !re.MatchString(n) { + t.Errorf("%q does not match its own branch", n) + } + } + for _, other := range []string{"axb", "x", "anything", "tenant-b", "", `back\\slash`} { + if re.MatchString(other) { + t.Errorf("branch matches %q", other) + } + } + if nameBranch("tenant-a", "pod", nil) != "" { + t.Error("an empty list produced a branch, which would match series without the label") + } +} + +func TestMetricsRequestScope(t *testing.T) { + branches := []string{"{a}", "{b}"} + labels := buildMetrics(t, "api/v1/labels", url.Values{}) + labels.setScope(branches) + if got := labels.params["extra_filters[]"]; !slices.Equal(got, branches) { + t.Errorf("extra_filters[] = %q", got) + } + if got := labels.params["match[]"]; !slices.Equal(got, branches) { + t.Errorf("default match[] = %q, want the branches", got) + } + client := buildMetrics(t, "api/v1/labels", url.Values{"match[]": {"up"}}) + client.setScope(branches) + if got := client.params["match[]"]; !slices.Equal(got, []string{"up"}) { + t.Errorf("client match[] replaced: %q", got) + } + query := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}) + query.setScope(branches) + if query.params.Has("match[]") || !slices.Equal(query.params["extra_filters[]"], branches) { + t.Errorf("query params = %v", query.params) + } + if !query.from.Equal(testNow) || !query.to.Equal(testNow) { + t.Errorf("instant window = [%s, %s], want now", query.from, query.to) + } + rng := buildMetrics(t, "api/v1/query_range", url.Values{"query": {"up"}, "start": {unix(testNow.Add(-time.Hour))}, "end": {unix(testNow)}, "step": {"60"}}) + if !rng.from.Equal(testNow.Add(-time.Hour)) || !rng.to.Equal(testNow) { + t.Errorf("range window = [%s, %s]", rng.from, rng.to) + } +} diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 8beb347d0..a0bb42f0d 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -46,6 +46,7 @@ type MetricsConfig struct { LongTermStore string ShortTermRetention time.Duration Limits MetricsLimits + App AppScopeLimits // UpstreamURL maps a store name to its Prometheus API base URL, without // a trailing slash; nil means the in-cluster vmselect Service. UpstreamURL func(store string) string @@ -77,6 +78,20 @@ type metricsRequest struct { // start is the earliest instant the request reads, for the store choice; // zero when that is unknown, which picks the long-term store. start time.Time + // from and to are the window the query is evaluated over, without the + // lookback of its expression. + from, to time.Time + // scopedMatch is set when match[] holds the scope, not client matchers. + scopedMatch bool +} + +// setScope replaces the namespace scope with the given selectors, which +// vmselect ORs. +func (req *metricsRequest) setScope(selectors []string) { + req.params["extra_filters[]"] = selectors + if req.scopedMatch { + req.params["match[]"] = selectors + } } // namespaceSelector quotes ns the way MetricsQL unquotes a label value @@ -127,6 +142,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti out.Set("step", formatSeconds(step)) } req.start = queryStart(out.Get("query"), at.Add(-latencyOffset), step) + req.from, req.to = at, at return req, nil case "api/v1/query_range": @@ -150,6 +166,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti return req, errorf(http.StatusUnprocessableEntity, "%d points per series exceed the limit of %d; raise step", points, lim.MaxPoints) } req.start = queryStart(out.Get("query"), start, step) + req.from, req.to = start, end out.Set("start", formatTime(start)) out.Set("end", formatTime(end)) out.Set("step", formatSeconds(step)) @@ -172,7 +189,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti if herr != nil { return req, herr } - req.start = start + req.start, req.from, req.to = start, start, end out.Set("start", formatTime(start)) out.Set("end", formatTime(end)) @@ -194,6 +211,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti // extra_filters[] on the label APIs unless a match[] is present // (getCommonParamsInternal), so the scope is repeated here. out.Set("match[]", namespaceSelector(namespace)) + req.scopedMatch = true } return req, nil } diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index 617971bf2..fd74491b5 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -193,11 +193,8 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT return herr } } - switch route.Kind { - case NamespaceMetrics: - return s.serveMetrics(w, r, route, user) - case ApplicationMetrics: - return errorf(http.StatusNotFound, "not found") + if route.Kind == NamespaceMetrics || route.Kind == ApplicationMetrics { + return s.serveMetrics(w, r, route, user, kind) } endpoint, ok := strings.CutPrefix(route.Endpoint, "select/logsql/") if !ok { @@ -288,7 +285,9 @@ func (s *Server) serve(w http.ResponseWriter, r *http.Request, route Route) *HTT return nil } -func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Route, user string) *HTTPError { +// serveMetrics scopes to the namespace, or with kind set to the application +// route.Name of that kind within it. +func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Route, user, kind string) *HTTPError { release, ok := s.limits.acquire(user) if !ok { return tooManyConcurrent(w) @@ -310,6 +309,13 @@ func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Rout defer cancel() } store := s.cfg.Metrics.store(req.start, now) + if kind != "" { + selectors, herr := s.applicationScope(ctx, r, store, req, route.Namespace, kind, route.Name) + if herr != nil { + return herr + } + req.setScope(selectors) + } resp, herr := s.post(ctx, r, s.cfg.Metrics.UpstreamURL(store)+"/"+req.path, req.params, "metrics store", nil) if herr != nil { return herr diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 0a76ab9a5..14807637e 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -17,11 +17,14 @@ limitations under the License. package observabilityproxy import ( + "encoding/json" "io" "net/http" "net/url" "slices" + "strings" "testing" + "time" authorizationv1 "k8s.io/api/authorization/v1" ) @@ -112,3 +115,170 @@ func TestServerApplicationKinds(t *testing.T) { t.Fatalf("denied caller answered %d, want 403", got) } } + +const wantPodLookup = `kube_pod_labels{namespace="tenant-a",label_apps_cozystack_io_application_kind="Postgres",label_apps_cozystack_io_application_name="db.main"}` + +// appMetricsUpstream answers the pod and PVC lookups with the given names, +// or with status when it is not 200, and every other path like vmselect. +func appMetricsUpstream(pods, pvcs []string, podStatus int) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + names := map[string][]string{"/api/v1/label/pod/values": pods, "/api/v1/label/persistentvolumeclaim/values": pvcs} + list, lookup := names[r.URL.Path] + if !lookup { + promUpstream(w, r) + return + } + if r.URL.Path == "/api/v1/label/pod/values" && podStatus != http.StatusOK { + http.Error(w, "unavailable", podStatus) + return + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{"status": "success", "data": list}) + } +} + +func TestServerApplicationMetricsScope(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream([]string{"db-main-0", "db.main-1"}, []string{"data-db-main-0"}, http.StatusOK), appPolicy, nil) + hostile := url.Values{ + "query": {"sum(rate(container_cpu_usage_seconds_total[5m]))"}, + "start": {unix(testNow.Add(-time.Hour))}, + "end": {unix(testNow)}, + "step": {"60"}, + "extra_filters": {`{namespace=~".+"}`}, + "extra_filters[]": {`{namespace="tenant-a"}`}, + "extra_label": {"apps_cozystack_io_application_name=b"}, + } + resp := h.do(http.MethodPost, appRoot+"db.main/metrics/api/v1/query_range", "good", hostile, nil) + body, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusOK { + t.Fatalf("answered %d %q", resp.StatusCode, body) + } + want := authorizationv1.ResourceAttributes{Verb: "get", Group: "apps.cozystack.io", Resource: "postgreses", Subresource: "metrics", Name: "db.main", Namespace: "tenant-a"} + if got := h.kube.lastSAR.Load().ResourceAttributes; *got != want { + t.Fatalf("SubjectAccessReview = %+v, want %+v", *got, want) + } + reqs := h.requests() + if len(reqs) != 3 { + t.Fatalf("upstream saw %d requests, want two lookups and the query", len(reqs)) + } + lookups := map[string]string{ + "/api/v1/label/pod/values": wantPodLookup, + "/api/v1/label/persistentvolumeclaim/values": `kube_persistentvolumeclaim_labels{namespace="tenant-a",label_apps_cozystack_io_application_kind="Postgres",label_apps_cozystack_io_application_name="db.main"}`, + } + for _, r := range reqs[:2] { + if got := r.form["match[]"]; len(got) != 1 || got[0] != lookups[r.path] { + t.Errorf("%s match[] = %q", r.path, got) + } + if got := r.form["extra_filters[]"]; !slices.Equal(got, []string{wantMetricsScope}) { + t.Errorf("%s extra_filters[] = %q", r.path, got) + } + start, herr := parseMetricsTime("start", r.form.Get("start")) + if herr != nil || start.After(testNow.Add(-time.Hour)) || start.Before(testNow.Add(-25*time.Hour)) { + t.Errorf("%s start = %s, want within the lookback before the window", r.path, r.form.Get("start")) + } + if r.form.Get("end") != formatTime(testNow) || r.form.Get("limit") != "101" { + t.Errorf("%s end/limit = %s/%s", r.path, r.form.Get("end"), r.form.Get("limit")) + } + } + main := reqs[2] + wantBranches := []string{ + wantIdentityBranch, + `{namespace="tenant-a",pod=~"^(db-main-0|db\\.main-1)$"}`, + `{namespace="tenant-a",persistentvolumeclaim=~"^(data-db-main-0)$"}`, + } + if main.path != "/api/v1/query_range" || !slices.Equal(main.form["extra_filters[]"], wantBranches) { + t.Fatalf("query %s extra_filters[] = %q, want %q", main.path, main.form["extra_filters[]"], wantBranches) + } + if main.form.Has("extra_filters") || main.form.Has("extra_label") { + t.Fatalf("client scope reached the store: %v", main.form) + } + if !slices.Equal(h.stores, []string{"shortterm", "shortterm", "shortterm"}) { + t.Fatalf("stores = %q, want the lookups on the query's store", h.stores) + } +} + +func TestServerApplicationMetricsBranches(t *testing.T) { + podBranch := `{namespace="tenant-a",pod=~"^(p)$"}` + tests := []struct { + name string + pods, pvcs []string + podStatus int + cfg Config + path string + wantCode int + want []string + }{ + {name: "no pods or pvcs", path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch}}, + {name: "pod lookup fails", pods: []string{"p"}, pvcs: []string{"v"}, podStatus: http.StatusServiceUnavailable, path: "api/v1/query?query=up", wantCode: http.StatusOK, + want: []string{wantIdentityBranch, `{namespace="tenant-a",persistentvolumeclaim=~"^(v)$"}`}}, + {name: "pods over the cap", pods: []string{"p", "q"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, + path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, + {name: "pvcs over the cap", pods: []string{"p"}, pvcs: []string{"v", "w"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, + path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, + {name: "labels get the branches as match[]", pods: []string{"p"}, path: "api/v1/labels", wantCode: http.StatusOK, want: []string{wantIdentityBranch, podBranch}}, + {name: "series needs match[]", path: "api/v1/series", wantCode: http.StatusBadRequest}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if tt.podStatus == 0 { + tt.podStatus = http.StatusOK + } + h := newPolicyHarness(t, tt.cfg, appMetricsUpstream(tt.pods, tt.pvcs, tt.podStatus), appPolicy, nil) + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/"+tt.path, "good"); got != tt.wantCode { + t.Fatalf("status = %d, want %d", got, tt.wantCode) + } + var main *upstreamRequest + for _, r := range h.requests() { + if !strings.HasPrefix(r.path, "/api/v1/label/pod/") && !strings.HasPrefix(r.path, "/api/v1/label/persistentvolumeclaim/") { + main = &r + } + } + if tt.want == nil { + if main != nil { + t.Fatalf("query reached the store: %s %v", main.path, main.form) + } + return + } + if main == nil || !slices.Equal(main.form["extra_filters[]"], tt.want) { + t.Fatalf("extra_filters[] = %v, want %q", main, tt.want) + } + if strings.HasSuffix(tt.path, "labels") && !slices.Equal(main.form["match[]"], tt.want) { + t.Fatalf("match[] = %q, want the branches", main.form["match[]"]) + } + }) + } +} + +// A grant narrowed to application "a" by resourceNames must not reach "b" +// through any endpoint of either root. +func TestServerApplicationNarrowedGrant(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream([]string{"a-0"}, nil, http.StatusOK), appPolicy, nil) + paths := []string{ + "metrics/api/v1/query?query=up", + "metrics/api/v1/query_range?query=up&start=" + unix(testNow.Add(-time.Hour)) + "&end=" + unix(testNow) + "&step=60", + "metrics/api/v1/series?match[]=up", + "metrics/api/v1/labels", + "metrics/api/v1/label/pod/values", + "log/select/logsql/query", + "log/select/logsql/hits", + "log/select/logsql/stats_query_range?query=" + url.QueryEscape("* | stats count()"), + "log/select/logsql/tail", + "log/select/logsql/field_values?field=level", + "log/select/logsql/stream_field_values?field=level", + "log/select/logsql/facets", + } + for _, p := range paths { + if got := h.status(http.MethodGet, appRoot+"b/"+p, "bob-token"); got != http.StatusForbidden { + t.Errorf("b/%s answered %d, want 403", p, got) + } + } + if n := len(h.requests()); n != 0 { + t.Fatalf("upstream saw %d requests for a denied application", n) + } + for _, p := range []string{"metrics/api/v1/labels", "log/select/logsql/query"} { + if got := h.status(http.MethodGet, appRoot+"a/"+p, "bob-token"); got != http.StatusOK { + t.Errorf("a/%s answered %d, want 200", p, got) + } + } +} diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index b541a18e1..168c402fe 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -96,6 +96,9 @@ func newPolicyHarness(t *testing.T, cfg Config, handler http.HandlerFunc, allow if cfg.Metrics.Limits == (MetricsLimits{}) { cfg.Metrics.Limits = testMetricsLimits() } + if cfg.Metrics.App == (AppScopeLimits{}) { + cfg.Metrics.App = AppScopeLimits{Lookback: 24 * time.Hour, MaxPods: 100, MaxPVCs: 100} + } if cfg.Logs == (LogsLimits{}) { cfg.Logs = testLimits() } From e6697ea4f176d1b059a40bab9bb87b31340d4e08 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 06:11:18 +0700 Subject: [PATCH 337/518] test(observability-proxy): cover malformed pod lookups and the lookup window An empty name in a lookup answer would turn the pod branch into a pattern that matches every series without a pod label, widening the application scope to the namespace; nothing pinned the guard against it. Malformed answers must only drop their branch, and the lookup must start at the query's own lookback, or at the configured limit when that lookback is unknown. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appkinds.go | 6 ++- internal/observabilityproxy/appscope.go | 6 +-- .../observabilityproxy/server_app_test.go | 39 +++++++++++++++++-- 3 files changed, 43 insertions(+), 8 deletions(-) diff --git a/internal/observabilityproxy/appkinds.go b/internal/observabilityproxy/appkinds.go index 5e545f390..d58368a13 100644 --- a/internal/observabilityproxy/appkinds.go +++ b/internal/observabilityproxy/appkinds.go @@ -79,7 +79,11 @@ func (k *ApplicationKinds) Kind(plural string) (string, error) { } var kinds []string for _, obj := range objs { - kind, _, _ := unstructured.NestedString(obj.(*unstructured.Unstructured).Object, "spec", "application", "kind") + u, ok := obj.(*unstructured.Unstructured) + if !ok { + continue + } + kind, _, _ := unstructured.NestedString(u.Object, "spec", "application", "kind") if kind != "" && !slices.Contains(kinds, kind) { kinds = append(kinds, kind) } diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index b969cc502..3b192d056 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -121,15 +121,15 @@ func (s *Server) applicationScope(ctx context.Context, r *http.Request, store st return applicationSelectors(ns, kind, name, lists[0], lists[1]), nil } -// lookupNames asks for at most max+1 values, enough to tell a list over +// lookupNames asks for at most limit+1 values, enough to tell a list over // the cap without reading all of it. -func (s *Server) lookupNames(ctx context.Context, r *http.Request, store, label, match, ns string, start, end time.Time, max int) ([]string, error) { +func (s *Server) lookupNames(ctx context.Context, r *http.Request, store, label, match, ns string, start, end time.Time, limit int) ([]string, error) { params := url.Values{} params.Set("match[]", match) params.Set("extra_filters[]", namespaceSelector(ns)) params.Set("start", formatTime(start)) params.Set("end", formatTime(end)) - params.Set("limit", strconv.Itoa(max+1)) + params.Set("limit", strconv.Itoa(limit+1)) resp, herr := s.post(ctx, r, s.cfg.Metrics.UpstreamURL(store)+"/api/v1/label/"+label+"/values", params, "metrics store", nil) if herr != nil { return nil, herr diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 14807637e..25d8f2a7f 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -173,9 +173,9 @@ func TestServerApplicationMetricsScope(t *testing.T) { if got := r.form["extra_filters[]"]; !slices.Equal(got, []string{wantMetricsScope}) { t.Errorf("%s extra_filters[] = %q", r.path, got) } - start, herr := parseMetricsTime("start", r.form.Get("start")) - if herr != nil || start.After(testNow.Add(-time.Hour)) || start.Before(testNow.Add(-25*time.Hour)) { - t.Errorf("%s start = %s, want within the lookback before the window", r.path, r.form.Get("start")) + // The query's own lookback is known and shorter than the 24h cap. + if want := formatTime(buildMetrics(t, "api/v1/query_range", hostile).start); r.form.Get("start") != want { + t.Errorf("%s start = %s, want the query's earliest read %s", r.path, r.form.Get("start"), want) } if r.form.Get("end") != formatTime(testNow) || r.form.Get("limit") != "101" { t.Errorf("%s end/limit = %s/%s", r.path, r.form.Get("end"), r.form.Get("limit")) @@ -204,6 +204,7 @@ func TestServerApplicationMetricsBranches(t *testing.T) { name string pods, pvcs []string podStatus int + podBody string cfg Config path string wantCode int @@ -218,13 +219,29 @@ func TestServerApplicationMetricsBranches(t *testing.T) { path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, {name: "labels get the branches as match[]", pods: []string{"p"}, path: "api/v1/labels", wantCode: http.StatusOK, want: []string{wantIdentityBranch, podBranch}}, {name: "series needs match[]", path: "api/v1/series", wantCode: http.StatusBadRequest}, + // An empty name would make pod=~"^(|p)$", which matches every + // series without a pod label. + {name: "empty pod name dropped", pods: []string{"", "p"}, path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch, podBranch}}, + {name: "only empty pod names", pods: []string{""}, path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch}}, + {name: "lookup status error", podBody: `{"status":"error","data":["p"]}`, path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch}}, + {name: "truncated lookup answer", podBody: `{"status":"success","data":["p"`, path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch}}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if tt.podStatus == 0 { tt.podStatus = http.StatusOK } - h := newPolicyHarness(t, tt.cfg, appMetricsUpstream(tt.pods, tt.pvcs, tt.podStatus), appPolicy, nil) + upstream := appMetricsUpstream(tt.pods, tt.pvcs, tt.podStatus) + if tt.podBody != "" { + upstream = func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/api/v1/label/pod/values" { + _, _ = io.WriteString(w, tt.podBody) + return + } + appMetricsUpstream(nil, nil, http.StatusOK)(w, r) + } + } + h := newPolicyHarness(t, tt.cfg, upstream, appPolicy, nil) if got := h.status(http.MethodGet, appRoot+"db.main/metrics/"+tt.path, "good"); got != tt.wantCode { t.Fatalf("status = %d, want %d", got, tt.wantCode) } @@ -282,3 +299,17 @@ func TestServerApplicationNarrowedGrant(t *testing.T) { } } } + +// An expression whose lookback cannot be bounded looks pods up from the +// configured lookback before the window. +func TestServerApplicationLookupUnknownLookback(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + q := url.Values{"query": {"up @ end()"}, "time": {unix(testNow)}} + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK { + t.Fatalf("status = %d", got) + } + r := h.requests()[0] + if want := formatTime(testNow.Add(-24 * time.Hour)); r.path != "/api/v1/label/pod/values" || r.form.Get("start") != want { + t.Fatalf("lookup %s start = %s, want %s", r.path, r.form.Get("start"), want) + } +} From dd7116ea451d11e782852a545dee03b5e83c81e4 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Wed, 4 Nov 2026 06:14:16 +0700 Subject: [PATCH 338/518] fix(observability-proxy): check the lookup cap before dropping empty names The store cuts a lookup at the cap plus one value. Dropping empty values first could bring such a cut list back under the cap and serve it as complete, hiding the application's other pods without the 422 that asks for a shorter range. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 5 +++++ internal/observabilityproxy/server_app_test.go | 2 ++ 2 files changed, 7 insertions(+) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 3b192d056..e259831da 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -148,6 +148,11 @@ func (s *Server) lookupNames(ctx context.Context, r *http.Request, store, label, if body.Status != "success" { return nil, errorf(http.StatusBadGateway, "metrics store answered status %q", body.Status) } + if len(body.Data) > limit { + // Over the cap before dropping empty values: the store cut the + // list at limit+1, so a shorter filtered list could hide names. + return body.Data, nil + } names := body.Data[:0] for _, n := range body.Data { if n != "" { diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 25d8f2a7f..6d353d00f 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -215,6 +215,8 @@ func TestServerApplicationMetricsBranches(t *testing.T) { want: []string{wantIdentityBranch, `{namespace="tenant-a",persistentvolumeclaim=~"^(v)$"}`}}, {name: "pods over the cap", pods: []string{"p", "q"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, + {name: "over the cap with an empty name", pods: []string{"", "p"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, + path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, {name: "pvcs over the cap", pods: []string{"p"}, pvcs: []string{"v", "w"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, {name: "labels get the branches as match[]", pods: []string{"p"}, path: "api/v1/labels", wantCode: http.StatusOK, want: []string{wantIdentityBranch, podBranch}}, From c4593d4c3edcb4b999be72f6bace5dc707b8af50 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:34:41 +0700 Subject: [PATCH 339/518] fix(observability-proxy): require namespace access on application roots The application roots asked only for the application subresource. Their pod and PVC branches select by namespace and name across the whole lookup window, so a caller granted only application "a" received the older series of application "b" whenever "b" had used one of the same pod or PVC names. The lineage labels the scope rests on can also be set by whoever creates an unowned pod. Until the scope carries application identity on every branch, the application roots also require read access to the namespace root, the same review the namespace root asks. Every tenant role that reads applications already holds it, so no current grant loses access; a grant narrowed by resourceNames alone is refused instead of trusted. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 11 ++++---- internal/observabilityproxy/route.go | 15 ++++++++++ internal/observabilityproxy/route_test.go | 24 ++++++++++++++++ internal/observabilityproxy/server.go | 16 ++++++----- .../observabilityproxy/server_app_test.go | 28 +++++++++++++------ internal/observabilityproxy/server_test.go | 2 +- 6 files changed, 75 insertions(+), 21 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index e259831da..55b51afe6 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -76,11 +76,12 @@ func nameBranch(ns, label string, names []string) string { // stamped (cAdvisor, kubelet volume stats). Every branch carries the // namespace, so their OR stays inside it. // -// Within the namespace, both the identity and the pod and PVC lists come -// from lineage labels on pods and PVCs. They separate applications for a -// grant narrowed by resourceNames only while whoever creates a pod or PVC -// cannot choose those labels; the lineage webhook keeps them as set on an -// object it finds no owner for, and the logs scope rests on the same. +// Within the namespace this narrows, it does not isolate: the identity and +// the pod and PVC lists come from lineage labels that the lineage webhook +// keeps as set on an object it finds no owner for, and the pod and PVC +// branches match by name, so a name another application held within the +// lookup window pulls that application's series in. Route.Reviews makes +// the application roots require read access to the namespace root too. func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { out := []string{identitySelector(ns, seriesKindLabel, seriesNameLabel, kind, name)} for _, b := range []string{nameBranch(ns, "pod", pods), nameBranch(ns, "persistentvolumeclaim", pvcs)} { diff --git a/internal/observabilityproxy/route.go b/internal/observabilityproxy/route.go index 56ee9e496..1baff4578 100644 --- a/internal/observabilityproxy/route.go +++ b/internal/observabilityproxy/route.go @@ -133,6 +133,21 @@ func ParseRoute(escapedPath string) (Route, *HTTPError) { // ResourceAttributes is the single SubjectAccessReview a route requires. It // depends on the path only, so a POST query authorizes exactly like a GET. +// Reviews lists every access a request on the route needs. An application +// root also needs read access to its namespace root: its pod and PVC +// branches match by name across the window, so a name another application +// held earlier reaches that application's series, and a grant narrowed by +// resourceNames would read more than it names. +func (r Route) Reviews() []authorizationv1.ResourceAttributes { + switch r.Kind { + case ApplicationLog: + return []authorizationv1.ResourceAttributes{Route{Kind: NamespaceLogs, Namespace: r.Namespace}.ResourceAttributes(), r.ResourceAttributes()} + case ApplicationMetrics: + return []authorizationv1.ResourceAttributes{Route{Kind: NamespaceMetrics, Namespace: r.Namespace}.ResourceAttributes(), r.ResourceAttributes()} + } + return []authorizationv1.ResourceAttributes{r.ResourceAttributes()} +} + func (r Route) ResourceAttributes() authorizationv1.ResourceAttributes { switch r.Kind { case NamespaceLogs: diff --git a/internal/observabilityproxy/route_test.go b/internal/observabilityproxy/route_test.go index f34311e0f..a8e85b377 100644 --- a/internal/observabilityproxy/route_test.go +++ b/internal/observabilityproxy/route_test.go @@ -18,6 +18,7 @@ package observabilityproxy import ( "net/http" + "slices" "testing" authorizationv1 "k8s.io/api/authorization/v1" @@ -130,3 +131,26 @@ func TestRouteResourceAttributes(t *testing.T) { } } } + +func TestRouteReviews(t *testing.T) { + nsLogs := authorizationv1.ResourceAttributes{Verb: "list", Group: "cozystack.io", Resource: "logs", Namespace: "a"} + nsMetrics := authorizationv1.ResourceAttributes{Verb: "list", Group: "cozystack.io", Resource: "metrics", Namespace: "a"} + tests := []struct { + route Route + want []authorizationv1.ResourceAttributes + }{ + {Route{Kind: NamespaceLogs, Namespace: "a"}, []authorizationv1.ResourceAttributes{nsLogs}}, + {Route{Kind: NamespaceMetrics, Namespace: "a"}, []authorizationv1.ResourceAttributes{nsMetrics}}, + {Route{Kind: ApplicationLog, Namespace: "a", Plural: "postgreses", Name: "db"}, []authorizationv1.ResourceAttributes{ + nsLogs, {Verb: "get", Group: "apps.cozystack.io", Resource: "postgreses", Subresource: "log", Name: "db", Namespace: "a"}, + }}, + {Route{Kind: ApplicationMetrics, Namespace: "a", Plural: "postgreses", Name: "db"}, []authorizationv1.ResourceAttributes{ + nsMetrics, {Verb: "get", Group: "apps.cozystack.io", Resource: "postgreses", Subresource: "metrics", Name: "db", Namespace: "a"}, + }}, + } + for _, tt := range tests { + if got := tt.route.Reviews(); !slices.Equal(got, tt.want) { + t.Errorf("%v.Reviews() = %+v, want %+v", tt.route.Kind, got, tt.want) + } + } +} diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index fd74491b5..cf873af72 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -451,13 +451,15 @@ func (s *Server) authorize(r *http.Request, route Route) (string, *HTTPError) { slog.Error("token review failed", "error", err) return "", errorf(http.StatusServiceUnavailable, "authentication unavailable") } - allowed, err := s.auth.Authorize(ctx, user, route.ResourceAttributes()) - if err != nil { - slog.Error("subject access review failed", "error", err) - return "", errorf(http.StatusServiceUnavailable, "authorization unavailable") - } - if !allowed { - return "", errorf(http.StatusForbidden, "forbidden") + for _, attrs := range route.Reviews() { + allowed, err := s.auth.Authorize(ctx, user, attrs) + if err != nil { + slog.Error("subject access review failed", "error", err) + return "", errorf(http.StatusServiceUnavailable, "authorization unavailable") + } + if !allowed { + return "", errorf(http.StatusForbidden, "forbidden") + } } return user.Username, nil } diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 6d353d00f..a767e7f9e 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -31,11 +31,18 @@ import ( const appRoot = "/apis/apps.cozystack.io/v1alpha1/namespaces/tenant-a/postgreses/" -// appPolicy lets alice read every postgres in tenant-a, as the view role's -// apps.cozystack.io wildcard does, and bob only "a", as a grant narrowed -// with resourceNames does. +// appPolicy lets alice read every postgres in tenant-a and the namespace +// roots, as the view role does. bob and carol may read only postgres "a", +// as a grant narrowed with resourceNames does; carol may also read the +// namespace roots. func appPolicy(user string, a *authorizationv1.ResourceAttributes) bool { - if a.Namespace != "tenant-a" || a.Verb != "get" || a.Group != "apps.cozystack.io" || a.Resource != "postgreses" || + if a.Namespace != "tenant-a" { + return false + } + if a.Verb == "list" && a.Group == "cozystack.io" && (a.Resource == "logs" || a.Resource == "metrics") && a.Name == "" && a.Subresource == "" { + return user == "alice" || user == "carol" + } + if a.Verb != "get" || a.Group != "apps.cozystack.io" || a.Resource != "postgreses" || (a.Subresource != "log" && a.Subresource != "metrics") { return false } @@ -270,7 +277,10 @@ func TestServerApplicationMetricsBranches(t *testing.T) { } // A grant narrowed to application "a" by resourceNames must not reach "b" -// through any endpoint of either root. +// through any endpoint of either root. Without read access to the +// namespace roots it does not reach "a" either: the pod and PVC branches +// match by name, and a name another application used earlier in the +// window would pull its series in. func TestServerApplicationNarrowedGrant(t *testing.T) { h := newPolicyHarness(t, Config{}, appMetricsUpstream([]string{"a-0"}, nil, http.StatusOK), appPolicy, nil) paths := []string{ @@ -288,15 +298,17 @@ func TestServerApplicationNarrowedGrant(t *testing.T) { "log/select/logsql/facets", } for _, p := range paths { - if got := h.status(http.MethodGet, appRoot+"b/"+p, "bob-token"); got != http.StatusForbidden { - t.Errorf("b/%s answered %d, want 403", p, got) + for _, denied := range []struct{ app, token string }{{"b", "bob-token"}, {"b", "carol-token"}, {"a", "bob-token"}} { + if got := h.status(http.MethodGet, appRoot+denied.app+"/"+p, denied.token); got != http.StatusForbidden { + t.Errorf("%s on %s/%s answered %d, want 403", denied.token, denied.app, p, got) + } } } if n := len(h.requests()); n != 0 { t.Fatalf("upstream saw %d requests for a denied application", n) } for _, p := range []string{"metrics/api/v1/labels", "log/select/logsql/query"} { - if got := h.status(http.MethodGet, appRoot+"a/"+p, "bob-token"); got != http.StatusOK { + if got := h.status(http.MethodGet, appRoot+"a/"+p, "carol-token"); got != http.StatusOK { t.Errorf("a/%s answered %d, want 200", p, got) } } diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index 168c402fe..ce2ba669b 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -64,7 +64,7 @@ func newHarness(t *testing.T, cfg Config, handler http.HandlerFunc) *harness { // and plural-to-kind mapping; nil kinds knows only postgreses. func newPolicyHarness(t *testing.T, cfg Config, handler http.HandlerFunc, allow func(string, *authorizationv1.ResourceAttributes) bool, kinds KindLookup) *harness { h := &harness{t: t} - h.kube = newFakeKube(map[string]string{"good": "alice", "bob-token": "bob"}, allow) + h.kube = newFakeKube(map[string]string{"good": "alice", "bob-token": "bob", "carol-token": "carol"}, allow) if kinds == nil { kinds = newTestKinds(t, true, appDefinition("postgres", "Postgres", "postgreses")) } From 07f4f187cef1a96e6f39247eeb2bf8ae7ff2fd79 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:36:03 +0700 Subject: [PATCH 340/518] fix(observability-proxy): cover forward-shifted samples in the app lookup The pod and PVC lookup of the application metrics root ended at the evaluation time, but a negative offset reads samples after it. An instant query evaluated two hours ago with "offset -1h" reads samples from one hour ago, so a pod created in between was missing from the scope and its series were dropped while the request succeeded. The lookup now ends at the latest instant the query reads, its window end plus the sum of its negative offsets. A query whose latest sample cannot be bounded, an @ modifier other than start() or end() or an offset that is not a plain duration, is refused with 422 on the application root; the namespace root has no lookup and still serves it. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 12 ++-- .../observabilityproxy/metricslookback.go | 64 +++++++++++++++---- internal/observabilityproxy/metricsparams.go | 11 +++- .../observabilityproxy/metricsparams_test.go | 37 +++++++++++ .../observabilityproxy/server_app_test.go | 36 +++++++++++ 5 files changed, 142 insertions(+), 18 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 55b51afe6..88acb4ba4 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -94,10 +94,14 @@ func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { // applicationScope looks the application's pods and PVCs up in store over // the request window, reaching back by the query's own lookback up to the -// configured limit. A failed lookup drops its branch, which only narrows -// the answer; a list over its cap is refused, because truncating it would -// silently drop series. +// configured limit and forward to the latest sample a negative offset +// reads. A failed lookup drops its branch, which only narrows the answer; +// a list over its cap, or a query whose latest sample is unknown, is +// refused, because the answer would silently miss series. func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) { + if req.last.IsZero() { + return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ other than start() or end(), or an offset that is not a plain duration); use the namespace root") + } lim := s.cfg.Metrics.App start := req.from.Add(-lim.Lookback) if req.start.After(start) { @@ -109,7 +113,7 @@ func (s *Server) applicationScope(ctx context.Context, r *http.Request, store st max int }{{"pod", "kube_pod_labels", lim.MaxPods}, {"persistentvolumeclaim", "kube_persistentvolumeclaim_labels", lim.MaxPVCs}} { match := l.metric + identitySelector(ns, ksmKindLabel, ksmNameLabel, kind, name) - names, err := s.lookupNames(ctx, r, store, l.label, match, ns, start, req.to, l.max) + names, err := s.lookupNames(ctx, r, store, l.label, match, ns, start, req.last, l.max) if err != nil { slog.Warn("application lookup failed; its branch is left out", "label", l.label, "namespace", ns, "kind", kind, "name", name, "error", err) continue diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index e55d189a5..442e9c8d9 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -113,10 +113,55 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { sum += d } lower := strings.ToLower(query) + back, _, ok := scanOffsets(lower) + if !ok { + return 0, false + } + sum += back + if hasKeyword(lower, "with") { + return 0, false + } + n := float64(subqueries + strings.Count(query, "(")) + sum += (2+3*n)*maxStep + (1+n)*float64(silenceInterval) + if sum > float64(maxMetricsDuration) { + return 0, false + } + return time.Duration(sum), true +} + +// queryLookahead is an upper bound on how far after its evaluation window +// a query reads, from the same lexical scan. Only a negative offset reads +// forward; the negative offsets are summed, which covers nesting. An @ +// modifier pins samples to any instant unless it names start() or end(), +// which lie in the window, and WITH can apply one offset several times; +// either makes the bound unknown. +func queryLookahead(query string) (time.Duration, bool) { + lower := strings.ToLower(query) + for rest := lower; ; { + at := strings.IndexByte(rest, '@') + if at < 0 { + break + } + rest = strings.TrimLeft(rest[at+1:], " \t\n\r") + if !strings.HasPrefix(rest, "start()") && !strings.HasPrefix(rest, "end()") { + return 0, false + } + } + _, ahead, ok := scanOffsets(lower) + if !ok || ahead > 0 && hasKeyword(lower, "with") || ahead > float64(maxMetricsDuration) { + return 0, false + } + return time.Duration(ahead), true +} + +// scanOffsets sums every offset of the lowercased query, the positive ones +// into back and the negative ones into ahead, in nanoseconds; ok is false +// when one is not a plain duration. +func scanOffsets(lower string) (back, ahead float64, ok bool) { for i := 0; ; { k := strings.Index(lower[i:], "offset") if k < 0 { - break + return back, ahead, true } k += i i = k + len("offset") @@ -134,21 +179,14 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { } d, ok := parseQueryDuration(arg[:end]) if !ok { - return 0, false + return 0, 0, false } - if !negative { - sum += d + if negative { + ahead += d + } else { + back += d } } - if hasKeyword(lower, "with") { - return 0, false - } - n := float64(subqueries + strings.Count(query, "(")) - sum += (2+3*n)*maxStep + (1+n)*float64(silenceInterval) - if sum > float64(maxMetricsDuration) { - return 0, false - } - return time.Duration(sum), true } // isKeyword reports whether lower[start:end] stands on its own rather than diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index a0bb42f0d..b55f09318 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -81,6 +81,9 @@ type metricsRequest struct { // from and to are the window the query is evaluated over, without the // lookback of its expression. from, to time.Time + // last is the latest instant the request reads, past to by a negative + // offset; zero when that is unknown. + last time.Time // scopedMatch is set when match[] holds the scope, not client matchers. scopedMatch bool } @@ -143,6 +146,9 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti } req.start = queryStart(out.Get("query"), at.Add(-latencyOffset), step) req.from, req.to = at, at + if ahead, ok := queryLookahead(out.Get("query")); ok { + req.last = at.Add(ahead) + } return req, nil case "api/v1/query_range": @@ -167,6 +173,9 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti } req.start = queryStart(out.Get("query"), start, step) req.from, req.to = start, end + if ahead, ok := queryLookahead(out.Get("query")); ok { + req.last = end.Add(ahead) + } out.Set("start", formatTime(start)) out.Set("end", formatTime(end)) out.Set("step", formatSeconds(step)) @@ -189,7 +198,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti if herr != nil { return req, herr } - req.start, req.from, req.to = start, start, end + req.start, req.from, req.to, req.last = start, start, end, end out.Set("start", formatTime(start)) out.Set("end", formatTime(end)) diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index a4a435567..5c39fdfff 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -429,3 +429,40 @@ func TestBuildMetricsParamsPointsAsVmselectCounts(t *testing.T) { t.Fatalf("11001 points answered %d, want 422", got) } } + +func TestQueryLookahead(t *testing.T) { + tests := []struct { + query string + want time.Duration + }{ + {`up`, 0}, + {`x offset 1h`, 0}, + {`x offset -1h`, time.Hour}, + {`x OFFSET - 90m`, 90 * time.Minute}, + {`rate(x[5m] offset -30m) offset -1h`, 90 * time.Minute}, + {`x offset -1h + y offset 2h`, time.Hour}, + {`x @ end()`, 0}, + {`x @ start() offset -1h`, time.Hour}, + {`node_offset_seconds - 1`, 0}, + } + for _, tt := range tests { + t.Run(tt.query, func(t *testing.T) { + got, ok := queryLookahead(tt.query) + if !ok || got != tt.want { + t.Fatalf("queryLookahead = %s, %v; want %s", got, ok, tt.want) + } + }) + } + for _, q := range []string{ + `x @ 1700000000`, + `x @ (end() - 1h)`, + `x{a="a@b"}`, + `x offset -5i`, + `x offset -$__interval`, + `with (f(a) = a offset -1h) f(f(x))`, + } { + if got, ok := queryLookahead(q); ok { + t.Errorf("queryLookahead(%q) = %s, want unknown", q, got) + } + } +} diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index a767e7f9e..c7e7a67ef 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -327,3 +327,39 @@ func TestServerApplicationLookupUnknownLookback(t *testing.T) { t.Fatalf("lookup %s start = %s, want %s", r.path, r.form.Get("start"), want) } } + +// TestServerApplicationLookupCoversNegativeOffset evaluates two hours ago +// a selector shifted an hour forward: the pod lookup must reach the time +// of the samples, or a pod created in between is left out. +func TestServerApplicationLookupCoversNegativeOffset(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + at := testNow.Add(-2 * time.Hour) + q := url.Values{"query": {"container_cpu_usage_seconds_total offset -1h"}, "time": {unix(at)}} + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK { + t.Fatalf("status = %d", got) + } + r := h.requests()[0] + if want := formatTime(at.Add(time.Hour)); r.path != "/api/v1/label/pod/values" || r.form.Get("end") != want { + t.Fatalf("lookup %s end = %s, want %s", r.path, r.form.Get("end"), want) + } +} + +// TestServerApplicationRefusesUnboundedSampleTime refuses a query whose +// samples may lie at a time the lookup cannot be sized for, rather than +// answer with series silently left out. +func TestServerApplicationRefusesUnboundedSampleTime(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + for _, query := range []string{"up @ 1700000000", "up offset -5i"} { + q := url.Values{"query": {query}} + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusUnprocessableEntity { + t.Errorf("%s answered %d, want 422", query, got) + } + } + if n := len(h.requests()); n != 0 { + t.Fatalf("upstream saw %d requests for a refused query", n) + } + // The namespace root has no lookup to size and still serves it. + if got := h.status(http.MethodGet, metricsRoot+"api/v1/query?"+url.Values{"query": {"up @ 1700000000"}}.Encode(), "good"); got != http.StatusOK { + t.Fatalf("namespace root answered %d", got) + } +} From 97a55a167bf786809a88d50d65da2f2a2225f55d Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:44:23 +0700 Subject: [PATCH 341/518] docs(observability-proxy): put the route review comments on their functions Reviews was inserted between ResourceAttributes and its doc comment, so go doc showed the ResourceAttributes text on Reviews, and that text still called it the single review a route needs, which application roots no longer are. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/route.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/route.go b/internal/observabilityproxy/route.go index 1baff4578..49a7c1770 100644 --- a/internal/observabilityproxy/route.go +++ b/internal/observabilityproxy/route.go @@ -131,8 +131,6 @@ func ParseRoute(escapedPath string) (Route, *HTTPError) { return r, nil } -// ResourceAttributes is the single SubjectAccessReview a route requires. It -// depends on the path only, so a POST query authorizes exactly like a GET. // Reviews lists every access a request on the route needs. An application // root also needs read access to its namespace root: its pod and PVC // branches match by name across the window, so a name another application @@ -148,6 +146,9 @@ func (r Route) Reviews() []authorizationv1.ResourceAttributes { return []authorizationv1.ResourceAttributes{r.ResourceAttributes()} } +// ResourceAttributes is the SubjectAccessReview of the route's own +// resource. It depends on the path only, so a POST query authorizes +// exactly like a GET. func (r Route) ResourceAttributes() authorizationv1.ResourceAttributes { switch r.Kind { case NamespaceLogs: From 1a3517fa81d6ca87f4387ce4e50e831d981d60e7 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:55:50 +0700 Subject: [PATCH 342/518] fix(observability-proxy): fail an application query whose lookup failed When the pod or PVC lookup failed, its branch was dropped and the query still answered 200, without the cAdvisor and volume series of the application, so a store error showed up as an empty graph. That is the same silently incomplete answer that a lookup over its cap is refused for. A failed lookup now fails the request: 504 when the store missed the deadline, 502 otherwise. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 13 ++++++++----- internal/observabilityproxy/server_app_test.go | 9 +++++---- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 88acb4ba4..ff1955dc9 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -95,9 +95,9 @@ func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { // applicationScope looks the application's pods and PVCs up in store over // the request window, reaching back by the query's own lookback up to the // configured limit and forward to the latest sample a negative offset -// reads. A failed lookup drops its branch, which only narrows the answer; -// a list over its cap, or a query whose latest sample is unknown, is -// refused, because the answer would silently miss series. +// reads. A failed lookup, a list over its cap, or a query whose latest +// sample is unknown fails the request, because a successful answer would +// silently miss the application's series. func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) { if req.last.IsZero() { return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ other than start() or end(), or an offset that is not a plain duration); use the namespace root") @@ -115,8 +115,11 @@ func (s *Server) applicationScope(ctx context.Context, r *http.Request, store st match := l.metric + identitySelector(ns, ksmKindLabel, ksmNameLabel, kind, name) names, err := s.lookupNames(ctx, r, store, l.label, match, ns, start, req.last, l.max) if err != nil { - slog.Warn("application lookup failed; its branch is left out", "label", l.label, "namespace", ns, "kind", kind, "name", name, "error", err) - continue + slog.Warn("application lookup failed", "label", l.label, "namespace", ns, "kind", kind, "name", name, "error", err) + if herr, ok := err.(*HTTPError); ok && herr.Code == http.StatusGatewayTimeout { + return nil, herr + } + return nil, errorf(http.StatusBadGateway, "the application's %s lookup failed", l.label) } if len(names) > l.max { return nil, errorf(http.StatusUnprocessableEntity, "the application has more than %d %ss in this window; use a shorter range", l.max, l.label) diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index c7e7a67ef..268ea7b8f 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -218,8 +218,9 @@ func TestServerApplicationMetricsBranches(t *testing.T) { want []string }{ {name: "no pods or pvcs", path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch}}, - {name: "pod lookup fails", pods: []string{"p"}, pvcs: []string{"v"}, podStatus: http.StatusServiceUnavailable, path: "api/v1/query?query=up", wantCode: http.StatusOK, - want: []string{wantIdentityBranch, `{namespace="tenant-a",persistentvolumeclaim=~"^(v)$"}`}}, + // A failed lookup would leave the application's own series out of + // a successful answer, so it fails the request instead. + {name: "pod lookup fails", pods: []string{"p"}, pvcs: []string{"v"}, podStatus: http.StatusServiceUnavailable, path: "api/v1/query?query=up", wantCode: http.StatusBadGateway}, {name: "pods over the cap", pods: []string{"p", "q"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, {name: "over the cap with an empty name", pods: []string{"", "p"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, @@ -232,8 +233,8 @@ func TestServerApplicationMetricsBranches(t *testing.T) { // series without a pod label. {name: "empty pod name dropped", pods: []string{"", "p"}, path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch, podBranch}}, {name: "only empty pod names", pods: []string{""}, path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch}}, - {name: "lookup status error", podBody: `{"status":"error","data":["p"]}`, path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch}}, - {name: "truncated lookup answer", podBody: `{"status":"success","data":["p"`, path: "api/v1/query?query=up", wantCode: http.StatusOK, want: []string{wantIdentityBranch}}, + {name: "lookup status error", podBody: `{"status":"error","data":["p"]}`, path: "api/v1/query?query=up", wantCode: http.StatusBadGateway}, + {name: "truncated lookup answer", podBody: `{"status":"success","data":["p"`, path: "api/v1/query?query=up", wantCode: http.StatusBadGateway}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { From 66e2b99f0981cd440a6d9b538c29f51fdaacadf0 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 00:56:32 +0700 Subject: [PATCH 343/518] fix(observability-proxy): reach the aligned end in the app lookup vmselect rounds a range query's end up to a whole step before evaluating it, so the last point can read samples up to one step past the end the client sent. The pod and PVC lookup stopped at that end and could miss a pod created within the last step. The lookup end of a range query now adds one step. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricsparams.go | 3 ++- internal/observabilityproxy/server_app_test.go | 18 +++++++++++++++++- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index b55f09318..721496fe3 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -173,8 +173,9 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti } req.start = queryStart(out.Get("query"), start, step) req.from, req.to = start, end + // vmselect rounds end up to a whole step (AdjustStartEnd). if ahead, ok := queryLookahead(out.Get("query")); ok { - req.last = end.Add(ahead) + req.last = end.Add(step + ahead) } out.Set("start", formatTime(start)) out.Set("end", formatTime(end)) diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 268ea7b8f..395deb18b 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -184,7 +184,8 @@ func TestServerApplicationMetricsScope(t *testing.T) { if want := formatTime(buildMetrics(t, "api/v1/query_range", hostile).start); r.form.Get("start") != want { t.Errorf("%s start = %s, want the query's earliest read %s", r.path, r.form.Get("start"), want) } - if r.form.Get("end") != formatTime(testNow) || r.form.Get("limit") != "101" { + // One step past end, which vmselect rounds up to a whole step. + if r.form.Get("end") != formatTime(testNow.Add(time.Minute)) || r.form.Get("limit") != "101" { t.Errorf("%s end/limit = %s/%s", r.path, r.form.Get("end"), r.form.Get("limit")) } } @@ -364,3 +365,18 @@ func TestServerApplicationRefusesUnboundedSampleTime(t *testing.T) { t.Fatalf("namespace root answered %d", got) } } + +// TestServerApplicationLookupCoversAlignedEnd: vmselect rounds a range +// query's end up to a whole step, so the lookup reaches one step past it. +func TestServerApplicationLookupCoversAlignedEnd(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + end := testNow.Add(-90 * time.Minute) + q := url.Values{"query": {"up"}, "start": {unix(end.Add(-6 * time.Hour))}, "end": {unix(end)}, "step": {"3600"}} + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query_range?"+q.Encode(), "good"); got != http.StatusOK { + t.Fatalf("status = %d", got) + } + r := h.requests()[0] + if want := formatTime(end.Add(time.Hour)); r.path != "/api/v1/label/pod/values" || r.form.Get("end") != want { + t.Fatalf("lookup %s end = %s, want %s", r.path, r.form.Get("end"), want) + } +} From bc823b704e5994301f62d6add1e28971768e7ff9 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:01:57 +0700 Subject: [PATCH 344/518] fix(observability-proxy): answer 504 to a lookup that misses the deadline Only a deadline hit before the lookup's response headers was reported as 504. When the store sent its headers and then stalled, the deadline expired while the body was decoded, and the decode error came back as a generic 502, so a slow store looked like a broken one. The request deadline is now checked whichever step of the lookup it cut, and both cases are covered by a test. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 7 +++-- .../observabilityproxy/server_app_test.go | 28 +++++++++++++++++++ 2 files changed, 33 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index ff1955dc9..80d12cbe2 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -19,6 +19,7 @@ package observabilityproxy import ( "context" "encoding/json" + "errors" "io" "log/slog" "net/http" @@ -116,8 +117,10 @@ func (s *Server) applicationScope(ctx context.Context, r *http.Request, store st names, err := s.lookupNames(ctx, r, store, l.label, match, ns, start, req.last, l.max) if err != nil { slog.Warn("application lookup failed", "label", l.label, "namespace", ns, "kind", kind, "name", name, "error", err) - if herr, ok := err.(*HTTPError); ok && herr.Code == http.StatusGatewayTimeout { - return nil, herr + // The deadline can also expire while the answer is read, after + // post has handed it over. + if errors.Is(context.Cause(ctx), context.DeadlineExceeded) { + return nil, errorf(http.StatusGatewayTimeout, "the application's %s lookup did not answer in time", l.label) } return nil, errorf(http.StatusBadGateway, "the application's %s lookup failed", l.label) } diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 395deb18b..53216e575 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -380,3 +380,31 @@ func TestServerApplicationLookupCoversAlignedEnd(t *testing.T) { t.Fatalf("lookup %s end = %s, want %s", r.path, r.form.Get("end"), want) } } + +// TestServerApplicationLookupDeadline: a lookup that misses the request +// deadline answers 504, whether the store sent nothing or stalled after +// its headers, and the query itself is not sent. +func TestServerApplicationLookupDeadline(t *testing.T) { + for _, headers := range []bool{false, true} { + h := newPolicyHarness(t, Config{RequestTimeout: 50 * time.Millisecond}, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/v1/label/pod/values" { + appMetricsUpstream(nil, nil, http.StatusOK)(w, r) + return + } + if headers { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"status":"success","data":[`) + w.(http.Flusher).Flush() + } + <-r.Context().Done() + }, appPolicy, nil) + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?query=up", "good"); got != http.StatusGatewayTimeout { + t.Fatalf("headers sent %v: status = %d, want 504", headers, got) + } + for _, r := range h.requests() { + if !strings.HasPrefix(r.path, "/api/v1/label/") { + t.Fatalf("headers sent %v: the query reached the store after a failed lookup", headers) + } + } + } +} From 84665a42a798d935a1b078d6fbdd3c720c62bffc Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:28:12 +0700 Subject: [PATCH 345/518] fix(observability-proxy): refuse app queries that read past the lookup The pod and PVC lookup never started earlier than --metrics-app-lookback before the window. A query that read further back, such as rate(x[7d]) with the 24h default, or whose lookback is unknown (an @ modifier, WITH, a step-multiple window), answered 200 without the series of pods that existed only before the lookup start. Such queries now answer 422 on the application root, matching how a query whose latest sample cannot be covered is handled; the namespace root has no lookup and still serves them. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 2 +- internal/observabilityproxy/appscope.go | 17 ++++++----- .../observabilityproxy/server_app_test.go | 29 ++++++++++++++----- 3 files changed, 31 insertions(+), 17 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index 4a99aa4cc..93f4c4553 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -94,7 +94,7 @@ func parseFlags(args []string) (options, error) { fs.DurationVar(&o.metrics.Limits.MaxRange, "metrics-max-range", 30*24*time.Hour, "longest [start, end] and step a metrics request may use") fs.IntVar(&o.metrics.Limits.MaxPoints, "metrics-max-points", 11000, "most points per series a range query may return") fs.DurationVar(&o.metrics.Limits.MaxTimeout, "metrics-max-timeout", 30*time.Second, "timeout forwarded to vmselect; a longer client timeout is clamped; must be below --request-timeout when that is set") - fs.DurationVar(&o.metrics.App.Lookback, "metrics-app-lookback", 24*time.Hour, "how far before the window the application root looks up the application's pods and PVCs at most") + fs.DurationVar(&o.metrics.App.Lookback, "metrics-app-lookback", 24*time.Hour, "how far before the window the application root looks up the application's pods and PVCs; a query reading further back answers 422") fs.IntVar(&o.metrics.App.MaxPods, "metrics-app-max-pods", 500, "most pods of one application in the lookup window; more answer 422") fs.IntVar(&o.metrics.App.MaxPVCs, "metrics-app-max-pvcs", 500, "most PVCs of one application in the lookup window; more answer 422") fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables") diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 80d12cbe2..835d29b84 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -94,20 +94,21 @@ func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { } // applicationScope looks the application's pods and PVCs up in store over -// the request window, reaching back by the query's own lookback up to the -// configured limit and forward to the latest sample a negative offset -// reads. A failed lookup, a list over its cap, or a query whose latest -// sample is unknown fails the request, because a successful answer would -// silently miss the application's series. +// the samples the request reads: back by the query's own lookback, and +// forward to the latest sample a negative offset reads. A query that reads +// further back than the configured lookback, or whose earliest or latest +// sample is unknown, a failed lookup, and a list over its cap all fail the +// request, because a successful answer would silently miss the +// application's series. func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) { if req.last.IsZero() { return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ other than start() or end(), or an offset that is not a plain duration); use the namespace root") } lim := s.cfg.Metrics.App - start := req.from.Add(-lim.Lookback) - if req.start.After(start) { - start = req.start + if req.start.IsZero() || req.start.Before(req.from.Add(-lim.Lookback)) { + return nil, errorf(http.StatusUnprocessableEntity, "the query reads further back than the application scope looks up pods (%s before the window, or an unknown lookback); use a shorter window or the namespace root", lim.Lookback) } + start := req.start var lists [2][]string for i, l := range []struct { label, metric string diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 53216e575..d3dcb683a 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -316,17 +316,30 @@ func TestServerApplicationNarrowedGrant(t *testing.T) { } } -// An expression whose lookback cannot be bounded looks pods up from the -// configured lookback before the window. -func TestServerApplicationLookupUnknownLookback(t *testing.T) { - h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) - q := url.Values{"query": {"up @ end()"}, "time": {unix(testNow)}} +// TestServerApplicationLookbackBeyondCap: the lookup reaches back at most +// --metrics-app-lookback, so a query that reads further back, or whose +// lookback is unknown, would miss pods that existed only earlier; it is +// refused rather than answered without their series. +func TestServerApplicationLookbackBeyondCap(t *testing.T) { + cfg := Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 100, MaxPVCs: 100}}} + h := newPolicyHarness(t, cfg, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + for _, query := range []string{"rate(x[7d])", "x[30d] @ end()", "rate(x[5i])"} { + q := url.Values{"query": {query}, "time": {unix(testNow)}} + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusUnprocessableEntity { + t.Errorf("%s answered %d, want 422", query, got) + } + } + if n := len(h.requests()); n != 0 { + t.Fatalf("upstream saw %d requests for refused queries", n) + } + q := url.Values{"query": {"rate(x[5m])"}, "time": {unix(testNow)}} if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK { - t.Fatalf("status = %d", got) + t.Fatalf("rate(x[5m]) answered %d, want 200", got) } r := h.requests()[0] - if want := formatTime(testNow.Add(-24 * time.Hour)); r.path != "/api/v1/label/pod/values" || r.form.Get("start") != want { - t.Fatalf("lookup %s start = %s, want %s", r.path, r.form.Get("start"), want) + want := formatTime(buildMetrics(t, "api/v1/query", q).start) + if r.path != "/api/v1/label/pod/values" || r.form.Get("start") != want { + t.Fatalf("lookup %s start = %s, want the query's earliest read %s", r.path, r.form.Get("start"), want) } } From 84d5b4ff0aa6e5056d4ea244eb5ba18baff6a0b8 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:28:34 +0700 Subject: [PATCH 346/518] fix(observability-proxy): require a positive application lookback Every query reads some time before its window, at least the implicit rollup window, so with the lookback refusal a zero --metrics-app-lookback would turn every application metrics query into a 422. It is now refused at startup instead. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 4 ++-- cmd/observability-proxy/main_test.go | 7 +++++-- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index 93f4c4553..7fe145229 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -137,8 +137,8 @@ func parseFlags(args []string) (options, error) { return o, errors.New("--metrics-shortterm-retention, --metrics-max-range and --metrics-max-timeout must be positive") case o.metrics.Limits.MaxPoints < 2: return o, errors.New("--metrics-max-points must be at least 2") - case o.metrics.App.Lookback < 0 || o.metrics.App.MaxPods < 1 || o.metrics.App.MaxPVCs < 1: - return o, errors.New("--metrics-app-lookback must not be negative and --metrics-app-max-pods and --metrics-app-max-pvcs must be positive") + case o.metrics.App.Lookback <= 0 || o.metrics.App.MaxPods < 1 || o.metrics.App.MaxPVCs < 1: + return o, errors.New("--metrics-app-lookback, --metrics-app-max-pods and --metrics-app-max-pvcs must be positive") case o.logs.MaxTailResume < 0: return o, errors.New("--logs-max-tail-resume must not be negative") // The proxy's deadline starts before the request reaches vmselect, so diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index 6b5c60d7d..bf8e032c2 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -66,7 +66,7 @@ func TestParseFlagsMetricsOverrides(t *testing.T) { o, err := parseFlags([]string{ "--root-tenant=tenant-x", "--metrics-shortterm-store=fast", "--metrics-longterm-store=slow", "--metrics-shortterm-retention=24h", "--metrics-max-range=168h", "--metrics-max-points=500", "--metrics-max-timeout=10s", - "--metrics-app-lookback=0s", "--metrics-app-max-pods=7", "--metrics-app-max-pvcs=3", + "--metrics-app-lookback=2h", "--metrics-app-max-pods=7", "--metrics-app-max-pvcs=3", }) if err != nil { t.Fatal(err) @@ -74,7 +74,7 @@ func TestParseFlagsMetricsOverrides(t *testing.T) { m := o.metrics if m.RootTenant != "tenant-x" || m.ShortTermStore != "fast" || m.LongTermStore != "slow" || m.ShortTermRetention != 24*time.Hour || m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second || - m.App.Lookback != 0 || m.App.MaxPods != 7 || m.App.MaxPVCs != 3 { + m.App.Lookback != 2*time.Hour || m.App.MaxPods != 7 || m.App.MaxPVCs != 3 { t.Errorf("overrides not applied: %+v", m) } } @@ -107,6 +107,9 @@ func TestParseFlagsInvalid(t *testing.T) { {"--metrics-max-points=1"}, {"--metrics-max-timeout=0s"}, {"--metrics-app-lookback=-1h"}, + // Every query reads some time before its window, so a zero lookback + // would refuse every application metrics query. + {"--metrics-app-lookback=0s"}, {"--metrics-app-max-pods=0"}, {"--metrics-app-max-pvcs=0"}, {"--unknown"}, From 1a3e1827ef2104e7f0ae881bd03f284442e4a5d5 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:29:13 +0700 Subject: [PATCH 347/518] fix(observability-proxy): pass a store 422 from the app lookup through vmselect answers 422 when a label lookup exceeds its own series limits. The application root turned that into a 502, which reads as a broken store, while the client could have fixed it by narrowing the window. The 422 and vmselect's message now reach the client. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 9 +++++++++ internal/observabilityproxy/server_app_test.go | 3 +++ 2 files changed, 12 insertions(+) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 835d29b84..eec1f0eb6 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -120,6 +120,9 @@ func (s *Server) applicationScope(ctx context.Context, r *http.Request, store st slog.Warn("application lookup failed", "label", l.label, "namespace", ns, "kind", kind, "name", name, "error", err) // The deadline can also expire while the answer is read, after // post has handed it over. + if herr, ok := err.(*HTTPError); ok && herr.Code == http.StatusUnprocessableEntity { + return nil, herr + } if errors.Is(context.Cause(ctx), context.DeadlineExceeded) { return nil, errorf(http.StatusGatewayTimeout, "the application's %s lookup did not answer in time", l.label) } @@ -147,6 +150,12 @@ func (s *Server) lookupNames(ctx context.Context, r *http.Request, store, label, return nil, herr } defer func() { _ = resp.Body.Close() }() + if resp.StatusCode == http.StatusUnprocessableEntity { + // vmselect's refusal of a lookup over its own limits; the client + // can act on it by narrowing the window. + msg, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<10)) + return nil, errorf(http.StatusUnprocessableEntity, "the application's %s lookup: %s", label, strings.TrimSpace(string(msg))) + } if resp.StatusCode != http.StatusOK { return nil, errorf(resp.StatusCode, "metrics store answered %s", resp.Status) } diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index d3dcb683a..d1d691b00 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -222,6 +222,9 @@ func TestServerApplicationMetricsBranches(t *testing.T) { // A failed lookup would leave the application's own series out of // a successful answer, so it fails the request instead. {name: "pod lookup fails", pods: []string{"p"}, pvcs: []string{"v"}, podStatus: http.StatusServiceUnavailable, path: "api/v1/query?query=up", wantCode: http.StatusBadGateway}, + // vmselect refuses a lookup over its own series limits with 422; + // the client can act on that by narrowing the window. + {name: "pod lookup over the store's limits", pods: []string{"p"}, podStatus: http.StatusUnprocessableEntity, path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, {name: "pods over the cap", pods: []string{"p", "q"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, {name: "over the cap with an empty name", pods: []string{"", "p"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, From 86dddb447a81f584f9e017cb4fd9223c9e63137b Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:29:14 +0700 Subject: [PATCH 348/518] docs(observability-proxy): note the store calls behind one app request The per-user rate counts client requests, but an application metrics request makes three store calls, so an operator sizing the rate against store capacity needs to know the multiplier. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 3 +++ 1 file changed, 3 insertions(+) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index 7fe145229..a3fa9b259 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -98,6 +98,9 @@ func parseFlags(args []string) (options, error) { fs.IntVar(&o.metrics.App.MaxPods, "metrics-app-max-pods", 500, "most pods of one application in the lookup window; more answer 422") fs.IntVar(&o.metrics.App.MaxPVCs, "metrics-app-max-pvcs", 500, "most PVCs of one application in the lookup window; more answer 422") fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables") + // Counted per client request: an application metrics request costs the + // store three calls (the pod and PVC lookups and the query) for one + // token, so the store load per user can be three times this rate. fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained requests per second per user; 0 disables") fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user") fs.IntVar(&o.limiter.MaxTailsPerUser, "max-tails-per-user", 5, "open tails per user; 0 disables") From 85fb98dd2cd5029adca903da074ab156877abb2a Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:36:52 +0700 Subject: [PATCH 349/518] fix(observability-proxy): stop refusing ordinary app queries on lookback With the lookback refusal, the 24h default turned common application queries into 422: a day's increase, and month-long panels at the step a dashboard picks for them. The lookback estimate is a loose upper bound, made for the store choice, and counts every "(" as an implicit subquery, so it far exceeds what such queries read. The lookback is only the refusal threshold, since the lookup itself spans the window the query reads, so its default now matches --metrics-max-range. The chart passes it from values, so an operator can tune it. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 6 ++++- cmd/observability-proxy/main_test.go | 2 +- .../observabilityproxy/server_app_test.go | 25 +++++++++++++++++++ .../templates/deployment.yaml | 1 + .../tests/deployment_test.yaml | 7 ++++++ .../system/observability-proxy/values.yaml | 3 +++ 6 files changed, 42 insertions(+), 2 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index a3fa9b259..487b6d04d 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -94,7 +94,11 @@ func parseFlags(args []string) (options, error) { fs.DurationVar(&o.metrics.Limits.MaxRange, "metrics-max-range", 30*24*time.Hour, "longest [start, end] and step a metrics request may use") fs.IntVar(&o.metrics.Limits.MaxPoints, "metrics-max-points", 11000, "most points per series a range query may return") fs.DurationVar(&o.metrics.Limits.MaxTimeout, "metrics-max-timeout", 30*time.Second, "timeout forwarded to vmselect; a longer client timeout is clamped; must be below --request-timeout when that is set") - fs.DurationVar(&o.metrics.App.Lookback, "metrics-app-lookback", 24*time.Hour, "how far before the window the application root looks up the application's pods and PVCs; a query reading further back answers 422") + // Only the refusal threshold: the lookup itself spans the window the + // query reads, and that estimate is a loose upper bound (every "(" + // counts as an implicit subquery), so the default matches + // --metrics-max-range to serve month-long panels at a dashboard step. + fs.DurationVar(&o.metrics.App.Lookback, "metrics-app-lookback", 30*24*time.Hour, "how far before the window an application metrics query may read; one reading further back, or by an unknown amount, answers 422") fs.IntVar(&o.metrics.App.MaxPods, "metrics-app-max-pods", 500, "most pods of one application in the lookup window; more answer 422") fs.IntVar(&o.metrics.App.MaxPVCs, "metrics-app-max-pvcs", 500, "most PVCs of one application in the lookup window; more answer 422") fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables") diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index bf8e032c2..616d25c51 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -40,7 +40,7 @@ func TestParseFlagsDefaults(t *testing.T) { if m.Limits.MaxRange != 30*24*time.Hour || m.Limits.MaxPoints != 11000 || m.Limits.MaxTimeout != 30*time.Second { t.Errorf("metrics limit defaults changed: %+v", m.Limits) } - if m.App.Lookback != 24*time.Hour || m.App.MaxPods != 500 || m.App.MaxPVCs != 500 { + if m.App.Lookback != 30*24*time.Hour || m.App.MaxPods != 500 || m.App.MaxPVCs != 500 { t.Errorf("application scope defaults changed: %+v", m.App) } if m.RootTenant != o.rootTenant { diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index d1d691b00..0b894b8fc 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -424,3 +424,28 @@ func TestServerApplicationLookupDeadline(t *testing.T) { } } } + +// TestServerApplicationDefaultLookbackServesLongPanels: the lookback is +// only the refusal threshold, and the lookback estimate is a loose upper +// bound, so the default must serve a day's increase and a month-long +// panel at the step a dashboard picks for it. +func TestServerApplicationDefaultLookbackServesLongPanels(t *testing.T) { + cfg := Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: 30 * 24 * time.Hour, MaxPods: 100, MaxPVCs: 100}}} + h := newPolicyHarness(t, cfg, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + month := url.Values{"start": {unix(testNow.Add(-29 * 24 * time.Hour))}, "end": {unix(testNow)}} + for _, tc := range []struct{ path, query, step string }{ + {"api/v1/query", "increase(x[24h])", ""}, + {"api/v1/query_range", "rate(x[5m])", "21600"}, + {"api/v1/query_range", "histogram_quantile(0.99, sum by (le) (rate(x[5m])))", "7200"}, + } { + q := url.Values{"query": {tc.query}} + if tc.step != "" { + q.Set("step", tc.step) + q.Set("start", month.Get("start")) + q.Set("end", month.Get("end")) + } + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/"+tc.path+"?"+q.Encode(), "good"); got != http.StatusOK { + t.Errorf("%s step %s answered %d, want 200", tc.query, tc.step, got) + } + } +} diff --git a/packages/system/observability-proxy/templates/deployment.yaml b/packages/system/observability-proxy/templates/deployment.yaml index d49de140b..5cb32833f 100644 --- a/packages/system/observability-proxy/templates/deployment.yaml +++ b/packages/system/observability-proxy/templates/deployment.yaml @@ -41,6 +41,7 @@ spec: - --metrics-shortterm-store={{ .Values.observabilityProxy.metrics.shorttermStore }} - --metrics-longterm-store={{ .Values.observabilityProxy.metrics.longtermStore }} - --metrics-shortterm-retention={{ .Values.observabilityProxy.metrics.shorttermRetention }} + - --metrics-app-lookback={{ .Values.observabilityProxy.metrics.appLookback }} ports: - name: http containerPort: 8080 diff --git a/packages/system/observability-proxy/tests/deployment_test.yaml b/packages/system/observability-proxy/tests/deployment_test.yaml index a5e948920..0f0104660 100644 --- a/packages/system/observability-proxy/tests/deployment_test.yaml +++ b/packages/system/observability-proxy/tests/deployment_test.yaml @@ -51,6 +51,9 @@ tests: - contains: path: spec.template.spec.containers[0].args content: --metrics-shortterm-retention=72h + - contains: + path: spec.template.spec.containers[0].args + content: --metrics-app-lookback=720h - it: follows renamed metrics stores set: @@ -58,6 +61,7 @@ tests: shorttermStore: fast longtermStore: slow shorttermRetention: 24h + appLookback: 48h asserts: - contains: path: spec.template.spec.containers[0].args @@ -68,6 +72,9 @@ tests: - contains: path: spec.template.spec.containers[0].args content: --metrics-shortterm-retention=24h + - contains: + path: spec.template.spec.containers[0].args + content: --metrics-app-lookback=48h - it: falls back to cozy.local when the platform passes no cluster domain asserts: diff --git a/packages/system/observability-proxy/values.yaml b/packages/system/observability-proxy/values.yaml index 67334a237..3e3ac70fe 100644 --- a/packages/system/observability-proxy/values.yaml +++ b/packages/system/observability-proxy/values.yaml @@ -14,6 +14,9 @@ observabilityProxy: shorttermStore: shortterm longtermStore: longterm shorttermRetention: 72h + # How far before its window an application metrics query may read; + # one reading further back answers 422 on the application root. + appLookback: 720h resources: requests: cpu: 10m From caa4e8d0e3d5ab9f05fce28c96e0c54e4aed35b1 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:37:59 +0700 Subject: [PATCH 350/518] fix(observability-proxy): keep only the series-limit 422 of a lookup vmselect answers 422 to every error that carries no status of its own, its query timeout and storage failures included, so passing every lookup 422 through told the client to narrow a window when the store was failing, and bypassed the 504 for a missed deadline. Only vmselect's series-limit message stays a 422, with the message taken from its JSON answer; its timeout becomes 504 and anything else 502. The request deadline is checked before the store's status. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 35 ++++++++++++---- .../observabilityproxy/server_app_test.go | 41 +++++++++++++++++-- 2 files changed, 66 insertions(+), 10 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index eec1f0eb6..796280eb6 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -120,12 +120,12 @@ func (s *Server) applicationScope(ctx context.Context, r *http.Request, store st slog.Warn("application lookup failed", "label", l.label, "namespace", ns, "kind", kind, "name", name, "error", err) // The deadline can also expire while the answer is read, after // post has handed it over. - if herr, ok := err.(*HTTPError); ok && herr.Code == http.StatusUnprocessableEntity { - return nil, herr - } if errors.Is(context.Cause(ctx), context.DeadlineExceeded) { return nil, errorf(http.StatusGatewayTimeout, "the application's %s lookup did not answer in time", l.label) } + if herr, ok := err.(*HTTPError); ok && (herr.Code == http.StatusUnprocessableEntity || herr.Code == http.StatusGatewayTimeout) { + return nil, herr + } return nil, errorf(http.StatusBadGateway, "the application's %s lookup failed", l.label) } if len(names) > l.max { @@ -151,10 +151,7 @@ func (s *Server) lookupNames(ctx context.Context, r *http.Request, store, label, } defer func() { _ = resp.Body.Close() }() if resp.StatusCode == http.StatusUnprocessableEntity { - // vmselect's refusal of a lookup over its own limits; the client - // can act on it by narrowing the window. - msg, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<10)) - return nil, errorf(http.StatusUnprocessableEntity, "the application's %s lookup: %s", label, strings.TrimSpace(string(msg))) + return nil, lookupRefusal(label, resp.Body) } if resp.StatusCode != http.StatusOK { return nil, errorf(resp.StatusCode, "metrics store answered %s", resp.Status) @@ -182,3 +179,27 @@ func (s *Server) lookupNames(ctx context.Context, r *http.Request, store, label, } return names, nil } + +// seriesLimitError is the message vmstorage gives when a search matches +// more series than the -search.max* flags allow (lib/storage/index_db.go); +// cluster vmselect wraps it in its own context, hence a substring match. +const seriesLimitError = "the number of matching timeseries exceeds" + +// lookupRefusal reads a vmselect 422. vmselect answers 422 to any error +// without a status of its own (httpserver.SendPrometheusError), storage +// failures and its own timeout included, so only the series limit, which +// a narrower window fixes, stays a 422 for the client. +func lookupRefusal(label string, body io.Reader) *HTTPError { + var answer struct { + Error string `json:"error"` + } + _ = json.NewDecoder(io.LimitReader(body, 64<<10)).Decode(&answer) + switch { + case strings.Contains(answer.Error, seriesLimitError): + return errorf(http.StatusUnprocessableEntity, "the application's %s lookup: %s", label, answer.Error) + case strings.Contains(answer.Error, "timeout exceeded"): + return errorf(http.StatusGatewayTimeout, "the application's %s lookup did not answer in time", label) + default: + return errorf(http.StatusBadGateway, "the application's %s lookup failed: %s", label, answer.Error) + } +} diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 0b894b8fc..2cd2a0eda 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -222,9 +222,6 @@ func TestServerApplicationMetricsBranches(t *testing.T) { // A failed lookup would leave the application's own series out of // a successful answer, so it fails the request instead. {name: "pod lookup fails", pods: []string{"p"}, pvcs: []string{"v"}, podStatus: http.StatusServiceUnavailable, path: "api/v1/query?query=up", wantCode: http.StatusBadGateway}, - // vmselect refuses a lookup over its own series limits with 422; - // the client can act on that by narrowing the window. - {name: "pod lookup over the store's limits", pods: []string{"p"}, podStatus: http.StatusUnprocessableEntity, path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, {name: "pods over the cap", pods: []string{"p", "q"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, {name: "over the cap with an empty name", pods: []string{"", "p"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, @@ -449,3 +446,41 @@ func TestServerApplicationDefaultLookbackServesLongPanels(t *testing.T) { } } } + +// TestServerApplicationLookupStoreErrors: vmselect answers 422 to any error +// without a status of its own (SendPrometheusError), so only its series +// limit, which a narrower window fixes, reaches the client as 422. +func TestServerApplicationLookupStoreErrors(t *testing.T) { + for _, tc := range []struct { + label, message string + want int + }{ + {"pod", "the number of matching timeseries exceeds 300000; either narrow down the search or increase -search.max* command-line flag values", http.StatusUnprocessableEntity}, + {"persistentvolumeclaim", "the number of matching timeseries exceeds 300000; either narrow down the search", http.StatusUnprocessableEntity}, + {"pod", "cannot fetch label values from vmstorage nodes: the number of matching timeseries exceeds 300000", http.StatusUnprocessableEntity}, + {"pod", "timeout exceeded before starting the query processing: 30s", http.StatusGatewayTimeout}, + {"pod", "error when searching for label values: cannot obtain data from vmstorage", http.StatusBadGateway}, + } { + h := newPolicyHarness(t, Config{}, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/api/v1/label/"+tc.label+"/values" { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusUnprocessableEntity) + _ = json.NewEncoder(w).Encode(map[string]string{"status": "error", "errorType": "422", "error": tc.message}) + return + } + appMetricsUpstream([]string{"p"}, nil, http.StatusOK)(w, r) + }, appPolicy, nil) + resp := h.do(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?query=up", "good", nil, nil) + body, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if resp.StatusCode != tc.want { + t.Errorf("%s lookup %q answered %d, want %d", tc.label, tc.message, resp.StatusCode, tc.want) + } + if tc.want == http.StatusUnprocessableEntity && !strings.Contains(string(body), "the number of matching timeseries exceeds 300000") { + t.Errorf("the store's limit message did not reach the client: %q", body) + } + if strings.Contains(string(body), `"errorType"`) { + t.Errorf("the store's raw JSON reached the client: %q", body) + } + } +} From 336961d7cd6dc7293caebd670c96b459b4ade6fc Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:38:10 +0700 Subject: [PATCH 351/518] docs(observability-proxy): put the store-call multiplier in the flag help The note that an application metrics request makes three store calls sat in a source comment, where an operator reading --help never sees it. It is now part of the --rate-limit-per-user help text. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index 487b6d04d..f5676b6d5 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -102,10 +102,7 @@ func parseFlags(args []string) (options, error) { fs.IntVar(&o.metrics.App.MaxPods, "metrics-app-max-pods", 500, "most pods of one application in the lookup window; more answer 422") fs.IntVar(&o.metrics.App.MaxPVCs, "metrics-app-max-pvcs", 500, "most PVCs of one application in the lookup window; more answer 422") fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables") - // Counted per client request: an application metrics request costs the - // store three calls (the pod and PVC lookups and the query) for one - // token, so the store load per user can be three times this rate. - fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained requests per second per user; 0 disables") + fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained client requests per second per user; an application metrics request makes three store calls (pod lookup, PVC lookup, query); 0 disables") fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user") fs.IntVar(&o.limiter.MaxTailsPerUser, "max-tails-per-user", 5, "open tails per user; 0 disables") fs.IntVar(&o.limiter.MaxTails, "max-tails", 200, "open tails in total; 0 disables") From c9c02b2d52e31f70e2a28a2d24c2929c8d042d74 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:45:13 +0700 Subject: [PATCH 352/518] test(observability-proxy): cover a store 504 on the app lookup The lookup now passes a 504 from the store through, where it used to answer 502, but only the 504 read from a 422 body was tested. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server_app_test.go | 1 + 1 file changed, 1 insertion(+) diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 2cd2a0eda..dcbf1e2a2 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -222,6 +222,7 @@ func TestServerApplicationMetricsBranches(t *testing.T) { // A failed lookup would leave the application's own series out of // a successful answer, so it fails the request instead. {name: "pod lookup fails", pods: []string{"p"}, pvcs: []string{"v"}, podStatus: http.StatusServiceUnavailable, path: "api/v1/query?query=up", wantCode: http.StatusBadGateway}, + {name: "pod lookup times out at the store", pods: []string{"p"}, podStatus: http.StatusGatewayTimeout, path: "api/v1/query?query=up", wantCode: http.StatusGatewayTimeout}, {name: "pods over the cap", pods: []string{"p", "q"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, path: "api/v1/query?query=up", wantCode: http.StatusUnprocessableEntity}, {name: "over the cap with an empty name", pods: []string{"", "p"}, cfg: Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}}, From 501f0d98f7c82e02169c33219d2772be49c137fa Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 01:45:13 +0700 Subject: [PATCH 353/518] docs(observability-proxy): drop the claimed link between two flag defaults The comment said the application lookback default matches --metrics-max-range, but nothing ties the two: changing the range flag leaves the lookback where it is. It now gives only the reason for the length. Assisted-by: LLM Signed-off-by: ZverGuy --- cmd/observability-proxy/main.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index f5676b6d5..66cc614f0 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -96,8 +96,8 @@ func parseFlags(args []string) (options, error) { fs.DurationVar(&o.metrics.Limits.MaxTimeout, "metrics-max-timeout", 30*time.Second, "timeout forwarded to vmselect; a longer client timeout is clamped; must be below --request-timeout when that is set") // Only the refusal threshold: the lookup itself spans the window the // query reads, and that estimate is a loose upper bound (every "(" - // counts as an implicit subquery), so the default matches - // --metrics-max-range to serve month-long panels at a dashboard step. + // counts as an implicit subquery), so the default is long enough to + // serve month-long panels at a dashboard step. fs.DurationVar(&o.metrics.App.Lookback, "metrics-app-lookback", 30*24*time.Hour, "how far before the window an application metrics query may read; one reading further back, or by an unknown amount, answers 422") fs.IntVar(&o.metrics.App.MaxPods, "metrics-app-max-pods", 500, "most pods of one application in the lookup window; more answer 422") fs.IntVar(&o.metrics.App.MaxPVCs, "metrics-app-max-pvcs", 500, "most PVCs of one application in the lookup window; more answer 422") From 0bb98ced05bfeca07e15edf31c4470d05ae3ecfc Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:24:13 +0700 Subject: [PATCH 354/518] fix(observability-proxy): say that every @ is refused on app roots The lookahead scan let @ start() and @ end() through and the 422 named them as allowed, but the lookback scan cannot bound any @, so every @ query was refused anyway, with a message pointing at forms that would not pass. Both scans now treat any @ as unknown and the message says so. The lookback-cap test now exercises the cap with a window just past it rather than an @ query. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 2 +- .../observabilityproxy/metricslookback.go | 20 +++++----------- .../observabilityproxy/metricsparams_test.go | 6 +++-- .../observabilityproxy/server_app_test.go | 24 ++++++++++++++++++- 4 files changed, 34 insertions(+), 18 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 796280eb6..e41dd1330 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -102,7 +102,7 @@ func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { // application's series. func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) { if req.last.IsZero() { - return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ other than start() or end(), or an offset that is not a plain duration); use the namespace root") + return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ modifier, or an offset that is not a plain duration); use the namespace root") } lim := s.cfg.Metrics.App if req.start.IsZero() || req.start.Before(req.from.Add(-lim.Lookback)) { diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index 442e9c8d9..e9e7cb0e9 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -131,22 +131,14 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { // queryLookahead is an upper bound on how far after its evaluation window // a query reads, from the same lexical scan. Only a negative offset reads -// forward; the negative offsets are summed, which covers nesting. An @ -// modifier pins samples to any instant unless it names start() or end(), -// which lie in the window, and WITH can apply one offset several times; -// either makes the bound unknown. +// forward; the negative offsets are summed, which covers nesting. Any @ +// makes the bound unknown, as it does for queryLookback, and so does WITH, +// which can apply one offset several times. func queryLookahead(query string) (time.Duration, bool) { - lower := strings.ToLower(query) - for rest := lower; ; { - at := strings.IndexByte(rest, '@') - if at < 0 { - break - } - rest = strings.TrimLeft(rest[at+1:], " \t\n\r") - if !strings.HasPrefix(rest, "start()") && !strings.HasPrefix(rest, "end()") { - return 0, false - } + if strings.Contains(query, "@") { + return 0, false } + lower := strings.ToLower(query) _, ahead, ok := scanOffsets(lower) if !ok || ahead > 0 && hasKeyword(lower, "with") || ahead > float64(maxMetricsDuration) { return 0, false diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 5c39fdfff..5e0d548b7 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -441,8 +441,6 @@ func TestQueryLookahead(t *testing.T) { {`x OFFSET - 90m`, 90 * time.Minute}, {`rate(x[5m] offset -30m) offset -1h`, 90 * time.Minute}, {`x offset -1h + y offset 2h`, time.Hour}, - {`x @ end()`, 0}, - {`x @ start() offset -1h`, time.Hour}, {`node_offset_seconds - 1`, 0}, } for _, tt := range tests { @@ -455,6 +453,10 @@ func TestQueryLookahead(t *testing.T) { } for _, q := range []string{ `x @ 1700000000`, + // The lookback scan cannot bound any @, so the application root + // refuses every one; the lookahead does not tell them apart. + `x @ end()`, + `x @ start() offset -1h`, `x @ (end() - 1h)`, `x{a="a@b"}`, `x offset -5i`, diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index dcbf1e2a2..8ae395d45 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -324,7 +324,9 @@ func TestServerApplicationNarrowedGrant(t *testing.T) { func TestServerApplicationLookbackBeyondCap(t *testing.T) { cfg := Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 100, MaxPVCs: 100}}} h := newPolicyHarness(t, cfg, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) - for _, query := range []string{"rate(x[7d])", "x[30d] @ end()", "rate(x[5i])"} { + // max_over_time(x[2h]) reads just past the 1h cap; rate(x[5i]) has an + // unknown lookback. + for _, query := range []string{"rate(x[7d])", "max_over_time(x[2h])", "rate(x[5i])"} { q := url.Values{"query": {query}, "time": {unix(testNow)}} if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusUnprocessableEntity { t.Errorf("%s answered %d, want 422", query, got) @@ -485,3 +487,23 @@ func TestServerApplicationLookupStoreErrors(t *testing.T) { } } } + +// TestServerApplicationRefusesEveryAt: no @ form can be bounded, and the +// answer says so rather than naming forms that would pass. +func TestServerApplicationRefusesEveryAt(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + for _, query := range []string{"up @ end()", "up @ start()", "up @ 1700000000"} { + resp := h.do(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+url.Values{"query": {query}}.Encode(), "good", nil, nil) + body, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusUnprocessableEntity { + t.Errorf("%s answered %d, want 422", query, resp.StatusCode) + } + if strings.Contains(string(body), "start() or end()") { + t.Errorf("%s: the refusal names @ forms as allowed: %q", query, body) + } + } + if n := len(h.requests()); n != 0 { + t.Fatalf("upstream saw %d requests for refused queries", n) + } +} From 41e61cb288b5d5a0c2cd250c5effb67601927bfb Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:24:54 +0700 Subject: [PATCH 355/518] fix(observability-proxy): keep store error text out of lookup answers A store error other than the series limit was built into the lookup's 502 message, and that text can name vmstorage nodes and addresses. The handler happened to replace the message before it reached the client, but nothing held it there. The text now goes to the log only. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 4 +++- internal/observabilityproxy/appscope_test.go | 13 +++++++++++++ internal/observabilityproxy/server_app_test.go | 5 +++++ 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index e41dd1330..561a92ab8 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -200,6 +200,8 @@ func lookupRefusal(label string, body io.Reader) *HTTPError { case strings.Contains(answer.Error, "timeout exceeded"): return errorf(http.StatusGatewayTimeout, "the application's %s lookup did not answer in time", label) default: - return errorf(http.StatusBadGateway, "the application's %s lookup failed: %s", label, answer.Error) + // The text may name vmstorage nodes, so it stays in the log. + slog.Warn("application lookup refused by the store", "label", label, "error", answer.Error) + return errorf(http.StatusBadGateway, "the application's %s lookup failed", label) } } diff --git a/internal/observabilityproxy/appscope_test.go b/internal/observabilityproxy/appscope_test.go index af028b867..c4b8e5d69 100644 --- a/internal/observabilityproxy/appscope_test.go +++ b/internal/observabilityproxy/appscope_test.go @@ -17,10 +17,12 @@ limitations under the License. package observabilityproxy import ( + "net/http" "net/url" "regexp" "slices" "strconv" + "strings" "testing" "time" ) @@ -114,3 +116,14 @@ func TestMetricsRequestScope(t *testing.T) { t.Errorf("range window = [%s, %s]", rng.from, rng.to) } } + +// TestLookupRefusalKeepsStoreErrorsInternal: a store error other than the +// series limit may name vmstorage nodes; it goes to the log, not into a +// message that could reach the client. +func TestLookupRefusalKeepsStoreErrorsInternal(t *testing.T) { + body := strings.NewReader(`{"status":"error","error":"cannot fetch from vmstorage-longterm-0.vmstorage:8401: connection refused"}`) + herr := lookupRefusal("pod", body) + if herr.Code != http.StatusBadGateway || strings.Contains(herr.Message, "vmstorage") { + t.Fatalf("lookupRefusal = %d %q, want 502 without the store's text", herr.Code, herr.Message) + } +} diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 8ae395d45..9a56e9e18 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -485,6 +485,11 @@ func TestServerApplicationLookupStoreErrors(t *testing.T) { if strings.Contains(string(body), `"errorType"`) { t.Errorf("the store's raw JSON reached the client: %q", body) } + // Other store errors may name vmstorage nodes; only the limit + // message, which the client can act on, is passed on. + if tc.want != http.StatusUnprocessableEntity && strings.Contains(string(body), "vmstorage") { + t.Errorf("the store's error text reached the client: %q", body) + } } } From 5fc9e5a0b9fa83f925b62e68783e047b6106869c Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:33:22 +0700 Subject: [PATCH 356/518] fix(observability-proxy): pass on only the limit text of a lookup 422 The series-limit refusal of a lookup reached the client with all of cluster vmselect's wrapping, and that context can name vmstorage nodes, the leak the generic 502 had just closed for other errors. The client now gets the message from the limit on. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 5 ++++- internal/observabilityproxy/server_app_test.go | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 561a92ab8..526bd0b52 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -196,7 +196,10 @@ func lookupRefusal(label string, body io.Reader) *HTTPError { _ = json.NewDecoder(io.LimitReader(body, 64<<10)).Decode(&answer) switch { case strings.Contains(answer.Error, seriesLimitError): - return errorf(http.StatusUnprocessableEntity, "the application's %s lookup: %s", label, answer.Error) + // From the limit on: what precedes it is cluster vmselect's context, + // which can name vmstorage nodes. + limit := answer.Error[strings.Index(answer.Error, seriesLimitError):] + return errorf(http.StatusUnprocessableEntity, "the application's %s lookup: %s", label, limit) case strings.Contains(answer.Error, "timeout exceeded"): return errorf(http.StatusGatewayTimeout, "the application's %s lookup did not answer in time", label) default: diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 9a56e9e18..52cfaa255 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -487,7 +487,7 @@ func TestServerApplicationLookupStoreErrors(t *testing.T) { } // Other store errors may name vmstorage nodes; only the limit // message, which the client can act on, is passed on. - if tc.want != http.StatusUnprocessableEntity && strings.Contains(string(body), "vmstorage") { + if strings.Contains(string(body), "vmstorage") { t.Errorf("the store's error text reached the client: %q", body) } } From 486f13dce690507a12319a63166b1ac85b0ba398 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 02:33:22 +0700 Subject: [PATCH 357/518] docs(observability-proxy): name every query form the app 422 refuses The refusal named only the @ modifier and non-literal offsets, but the same answer covers an @ inside a label value and WITH with a negative offset, so a client could not tell why such a query was refused. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 526bd0b52..6c3a9fb4f 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -102,7 +102,7 @@ func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { // application's series. func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) { if req.last.IsZero() { - return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ modifier, or an offset that is not a plain duration); use the namespace root") + return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ anywhere in the query, WITH with a negative offset, or an offset that is not a plain duration); use the namespace root") } lim := s.cfg.Metrics.App if req.start.IsZero() || req.start.Before(req.from.Add(-lim.Lookback)) { From 5418d6b40e864022c32914e57f6b2084e6ccdc81 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:05:54 +0700 Subject: [PATCH 358/518] fix(observability-proxy): refuse compound offsets on the app root The offset scan read a duration up to its first sign and ignored the rest. metricsql lexes 1h-30m as one duration and DurationValue makes every part after the first negative one subtract, so offset -1h-30m shifts samples 90 minutes forward while the scan covered 60, and offset 1h-2h shifts them an hour forward while the scan counted it as looking back. The pod and PVC lookup then missed pods created in the gap and the query answered 200 without them. A duration followed directly by a minus now makes the scan unknown, which answers 422 on the application root. A plus or a spaced sign is an operator after the offset in metricsql, not part of it, and is still read. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observabilityproxy/metricslookback.go | 6 +++++ .../observabilityproxy/metricsparams_test.go | 13 +++++++++++ .../observabilityproxy/server_app_test.go | 22 +++++++++++++++++++ 3 files changed, 41 insertions(+) diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index e9e7cb0e9..886d9e43b 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -169,6 +169,12 @@ func scanOffsets(lower string) (back, ahead float64, ok bool) { if end < 0 { end = len(arg) } + // metricsql lexes 1h-30m as one duration whose parts after the + // first negative one all subtract (DurationValue), so -1h-30m is + // -90m and 1h-2h is -1h; the scan does not read such compounds. + if end < len(arg) && arg[end] == '-' { + return 0, 0, false + } d, ok := parseQueryDuration(arg[:end]) if !ok { return 0, 0, false diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 5e0d548b7..29802a220 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -442,6 +442,12 @@ func TestQueryLookahead(t *testing.T) { {`rate(x[5m] offset -30m) offset -1h`, 90 * time.Minute}, {`x offset -1h + y offset 2h`, time.Hour}, {`node_offset_seconds - 1`, 0}, + // metricsql lexes a duration as one token, so a sign with spaces, + // or a plus, is an operator after the offset, not part of it. + {`x offset 1h+30m`, 0}, + {`x offset 1h - 30m`, 0}, + {`x offset -1h + 30m`, time.Hour}, + {`x offset -1h30m`, 90 * time.Minute}, } for _, tt := range tests { t.Run(tt.query, func(t *testing.T) { @@ -460,6 +466,13 @@ func TestQueryLookahead(t *testing.T) { `x @ (end() - 1h)`, `x{a="a@b"}`, `x offset -5i`, + // Compound durations with a negative part (metricsql DurationValue): + // -1h-30m is -90m and 1h-2h is -1h, neither of which the scan reads. + `x offset -1h-30m`, + `x offset - 1h-30m`, + `x offset 1h-2h`, + `x offset 1h-30m`, + `rate(x[5m] offset 1h-30m)`, `x offset -$__interval`, `with (f(a) = a offset -1h) f(f(x))`, } { diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 52cfaa255..e4f0ca34f 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -512,3 +512,25 @@ func TestServerApplicationRefusesEveryAt(t *testing.T) { t.Fatalf("upstream saw %d requests for refused queries", n) } } + +// TestServerApplicationRefusesForwardReads: queries that read past the +// evaluation time by an amount the scan cannot bound answer 422 and send +// nothing upstream, rather than answer 200 without pods created in that +// time. +func TestServerApplicationRefusesForwardReads(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + for _, query := range []string{ + "container_cpu_usage_seconds_total offset -1h-30m", + "container_cpu_usage_seconds_total offset 1h-2h", + } { + q := url.Values{"query": {query}, "time": {unix(testNow.Add(-2 * time.Hour))}} + resp := h.do(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good", nil, nil) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusUnprocessableEntity { + t.Errorf("%s answered %d, want 422", query, resp.StatusCode) + } + } + if n := len(h.requests()); n != 0 { + t.Fatalf("upstream saw %d requests for refused queries", n) + } +} From cb35399ee70f49399c467bb2a4be9117b923255d Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:06:04 +0700 Subject: [PATCH 359/518] fix(observability-proxy): refuse rollup_candlestick on the app root vmselect evaluates rollup_candlestick one step ahead of the evaluation time, with no offset in the query: evalRollupFuncWithoutAt adds the step in effect to the window's start and end (app/vmselect/promql/ eval.go in v1.136.0-cluster, unchanged in v1.139.0). The lookahead scan returned zero for it, so the lookup stopped at the evaluation time and pods created during that step were left out of a 200. The step in effect is the subquery's when the call is nested, so rather than size it the application root answers 422. No other function reads ahead: the subquery evaluation past its end in evalRollupFuncWithSubquery is outside the outer rollup's window. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 2 +- internal/observabilityproxy/metricslookback.go | 8 +++++++- internal/observabilityproxy/metricsparams_test.go | 5 +++++ internal/observabilityproxy/server_app_test.go | 1 + 4 files changed, 14 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 6c3a9fb4f..9911abeeb 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -102,7 +102,7 @@ func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { // application's series. func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) { if req.last.IsZero() { - return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ anywhere in the query, WITH with a negative offset, or an offset that is not a plain duration); use the namespace root") + return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ anywhere in the query, rollup_candlestick, WITH with a negative offset, or an offset that is not a plain duration); use the namespace root") } lim := s.cfg.Metrics.App if req.start.IsZero() || req.start.Before(req.from.Add(-lim.Lookback)) { diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index 886d9e43b..379a42be0 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -133,12 +133,18 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { // a query reads, from the same lexical scan. Only a negative offset reads // forward; the negative offsets are summed, which covers nesting. Any @ // makes the bound unknown, as it does for queryLookback, and so does WITH, -// which can apply one offset several times. +// which can apply one offset several times. vmselect also evaluates +// rollup_candlestick one step ahead of the evaluation time +// (evalRollupFuncWithoutAt), with the step of whatever subquery it sits +// in, so it makes the bound unknown too. func queryLookahead(query string) (time.Duration, bool) { if strings.Contains(query, "@") { return 0, false } lower := strings.ToLower(query) + if hasKeyword(lower, "rollup_candlestick") { + return 0, false + } _, ahead, ok := scanOffsets(lower) if !ok || ahead > 0 && hasKeyword(lower, "with") || ahead > float64(maxMetricsDuration) { return 0, false diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 29802a220..d320333fd 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -473,6 +473,11 @@ func TestQueryLookahead(t *testing.T) { `x offset 1h-2h`, `x offset 1h-30m`, `rate(x[5m] offset 1h-30m)`, + // vmselect evaluates rollup_candlestick one step ahead + // (evalRollupFuncWithoutAt), whatever the offset. + `rollup_candlestick(x)`, + `ROLLUP_CANDLESTICK(x[5m])`, + `sum(rollup_candlestick(x[1h:5m]))`, `x offset -$__interval`, `with (f(a) = a offset -1h) f(f(x))`, } { diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index e4f0ca34f..3ede9f239 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -522,6 +522,7 @@ func TestServerApplicationRefusesForwardReads(t *testing.T) { for _, query := range []string{ "container_cpu_usage_seconds_total offset -1h-30m", "container_cpu_usage_seconds_total offset 1h-2h", + "rollup_candlestick(container_cpu_usage_seconds_total)", } { q := url.Values{"query": {query}, "time": {unix(testNow.Add(-2 * time.Hour))}} resp := h.do(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good", nil, nil) From 84f19e14d88b7a6567ad6b98f67120128e71465b Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:11:42 +0700 Subject: [PATCH 360/518] fix(observability-proxy): reach the subquery steps in the app lookup vmselect evaluates a subquery up to its end plus one subquery step, rounded up to the step (evalRollupFuncWithSubquery in app/vmselect/promql/eval.go), and range_* transforms (setLastValues) and keep_next_value carry those later values back onto earlier points. A series that starts after the evaluation time, such as a pod created within that reach, can therefore appear in the answer, which the earlier claim that only rollup_candlestick reads ahead missed. The lookup stopped at the evaluation time and left such a pod out of a 200. The lookahead now adds two steps per subquery level, counting every "(" as a possible implicit subquery and using the largest step seen, as the lookback already does. Overreaching only widens the lookup over the application's own pods. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observabilityproxy/metricslookback.go | 52 +++++++++++++++---- internal/observabilityproxy/metricsparams.go | 4 +- .../observabilityproxy/metricsparams_test.go | 22 ++++++-- .../observabilityproxy/server_app_test.go | 22 +++++++- 4 files changed, 84 insertions(+), 16 deletions(-) diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index 379a42be0..ffff274a2 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -130,14 +130,19 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { } // queryLookahead is an upper bound on how far after its evaluation window -// a query reads, from the same lexical scan. Only a negative offset reads -// forward; the negative offsets are summed, which covers nesting. Any @ -// makes the bound unknown, as it does for queryLookback, and so does WITH, -// which can apply one offset several times. vmselect also evaluates -// rollup_candlestick one step ahead of the evaluation time -// (evalRollupFuncWithoutAt), with the step of whatever subquery it sits -// in, so it makes the bound unknown too. -func queryLookahead(query string) (time.Duration, bool) { +// a query reads, from the same lexical scan, with step the query's step. +// A negative offset reads forward; the negative offsets are summed, which +// covers nesting. vmselect also evaluates a subquery up to its end plus +// its step, rounded up to the step (evalRollupFuncWithSubquery), and +// range_* transforms and keep_next_value carry those later values back +// onto earlier points, so each subquery level adds two steps; as for the +// lookback, every "(" counts as a possible implicit subquery and each +// term uses the largest step seen. Any @ makes the bound unknown, as it +// does for queryLookback, and so does WITH, which can apply one offset +// several times. vmselect also evaluates rollup_candlestick one step ahead +// of the evaluation time (evalRollupFuncWithoutAt), so it makes the bound +// unknown too. +func queryLookahead(query string, step time.Duration) (time.Duration, bool) { if strings.Contains(query, "@") { return 0, false } @@ -146,7 +151,36 @@ func queryLookahead(query string) (time.Duration, bool) { return 0, false } _, ahead, ok := scanOffsets(lower) - if !ok || ahead > 0 && hasKeyword(lower, "with") || ahead > float64(maxMetricsDuration) { + if !ok || ahead > 0 && hasKeyword(lower, "with") { + return 0, false + } + subqueries := 0 + maxStep := float64(step) + for rest := query; ; { + open := strings.IndexByte(rest, '[') + if open < 0 { + break + } + closing := strings.IndexByte(rest[open:], ']') + if closing < 0 { + return 0, false + } + _, subStep, isSubquery := strings.Cut(rest[open+1:open+closing], ":") + rest = rest[open+closing+1:] + if !isSubquery { + continue + } + subqueries++ + if strings.TrimSpace(subStep) != "" { + d, ok := parseQueryDuration(subStep) + if !ok { + return 0, false + } + maxStep = max(maxStep, d) + } + } + ahead += 2 * float64(subqueries+strings.Count(query, "(")) * maxStep + if ahead > float64(maxMetricsDuration) { return 0, false } return time.Duration(ahead), true diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 721496fe3..63be3c497 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -146,7 +146,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti } req.start = queryStart(out.Get("query"), at.Add(-latencyOffset), step) req.from, req.to = at, at - if ahead, ok := queryLookahead(out.Get("query")); ok { + if ahead, ok := queryLookahead(out.Get("query"), step); ok { req.last = at.Add(ahead) } return req, nil @@ -174,7 +174,7 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti req.start = queryStart(out.Get("query"), start, step) req.from, req.to = start, end // vmselect rounds end up to a whole step (AdjustStartEnd). - if ahead, ok := queryLookahead(out.Get("query")); ok { + if ahead, ok := queryLookahead(out.Get("query"), step); ok { req.last = end.Add(step + ahead) } out.Set("start", formatTime(start)) diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index d320333fd..95ee047d8 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -439,7 +439,9 @@ func TestQueryLookahead(t *testing.T) { {`x offset 1h`, 0}, {`x offset -1h`, time.Hour}, {`x OFFSET - 90m`, 90 * time.Minute}, - {`rate(x[5m] offset -30m) offset -1h`, 90 * time.Minute}, + // Each "(" may be an implicit subquery, which vmselect evaluates + // up to two steps past its end; here the step is 1m. + {`rate(x[5m] offset -30m) offset -1h`, 92 * time.Minute}, {`x offset -1h + y offset 2h`, time.Hour}, {`node_offset_seconds - 1`, 0}, // metricsql lexes a duration as one token, so a sign with spaces, @@ -451,7 +453,7 @@ func TestQueryLookahead(t *testing.T) { } for _, tt := range tests { t.Run(tt.query, func(t *testing.T) { - got, ok := queryLookahead(tt.query) + got, ok := queryLookahead(tt.query, time.Minute) if !ok || got != tt.want { t.Fatalf("queryLookahead = %s, %v; want %s", got, ok, tt.want) } @@ -481,8 +483,22 @@ func TestQueryLookahead(t *testing.T) { `x offset -$__interval`, `with (f(a) = a offset -1h) f(f(x))`, } { - if got, ok := queryLookahead(q); ok { + if got, ok := queryLookahead(q, time.Minute); ok { t.Errorf("queryLookahead(%q) = %s, want unknown", q, got) } } } + +// TestQueryLookaheadSubquery: vmselect evaluates a subquery up to its end +// plus one subquery step, rounded up to the step (evalRollupFuncWithSubquery), +// and range_last and the other range_* transforms copy the last of those +// values back onto earlier points, so a series that starts after the +// evaluation time can reach the answer. +func TestQueryLookaheadSubquery(t *testing.T) { + for _, q := range []string{`max_over_time(range_last(x)[1h:1h])`, `max_over_time(keep_next_value(x)[1h:1h])`} { + got, ok := queryLookahead(q, time.Minute) + if !ok || got < 2*time.Hour { + t.Errorf("queryLookahead(%q) = %s, %v; want at least two subquery steps", q, got, ok) + } + } +} diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 3ede9f239..2d0d9b56e 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -184,8 +184,9 @@ func TestServerApplicationMetricsScope(t *testing.T) { if want := formatTime(buildMetrics(t, "api/v1/query_range", hostile).start); r.form.Get("start") != want { t.Errorf("%s start = %s, want the query's earliest read %s", r.path, r.form.Get("start"), want) } - // One step past end, which vmselect rounds up to a whole step. - if r.form.Get("end") != formatTime(testNow.Add(time.Minute)) || r.form.Get("limit") != "101" { + // One step past end, which vmselect rounds up to a whole step, and + // two steps for each "(" that may be an implicit subquery. + if r.form.Get("end") != formatTime(testNow.Add(5*time.Minute)) || r.form.Get("limit") != "101" { t.Errorf("%s end/limit = %s/%s", r.path, r.form.Get("end"), r.form.Get("limit")) } } @@ -535,3 +536,20 @@ func TestServerApplicationRefusesForwardReads(t *testing.T) { t.Fatalf("upstream saw %d requests for refused queries", n) } } + +// TestServerApplicationLookupCoversSubquery: a pod created during the two +// subquery steps past the evaluation time can show up through range_last, +// so the lookup reaches that far. +func TestServerApplicationLookupCoversSubquery(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + at := testNow.Add(-6 * time.Hour) + q := url.Values{"query": {"max_over_time(range_last(container_cpu_usage_seconds_total)[1h:1h])"}, "time": {unix(at)}} + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK { + t.Fatalf("status = %d", got) + } + r := h.requests()[0] + end, err := time.Parse(time.RFC3339, r.form.Get("end")) + if err != nil || r.path != "/api/v1/label/pod/values" || end.Before(at.Add(2*time.Hour)) { + t.Fatalf("lookup %s end = %s, want at least %s", r.path, r.form.Get("end"), formatTime(at.Add(2*time.Hour))) + } +} From 56fc64575a4bcc58d5cf5d66732dcdb190040188 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:11:54 +0700 Subject: [PATCH 361/518] fix(observability-proxy): read a minus after an offset as an operator The compound-offset refusal took any minus right after an offset's duration for a compound, so x offset 1h-1 or x offset 5m-y answered 422. metricsql continues a duration token only with a minus followed by a duration part (scanSingleDuration), so only that form is refused now; anything else is the subtraction it is. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricslookback.go | 7 ++++++- internal/observabilityproxy/metricsparams_test.go | 3 +++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index ffff274a2..16f1f4cff 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -49,6 +49,10 @@ var ( querySeconds = regexp.MustCompile(`^\d+(?:\.\d+)?$`) ) +// negativeDurationPart is what metricsql's scanSingleDuration accepts +// after a minus inside one duration token. +var negativeDurationPart = regexp.MustCompile(`^(?:\d+(?:\.\d+)?(?:ms|s|m|h|d|w|y|i)|\$__interval)`) + var queryDurationUnits = map[string]time.Duration{ "ms": time.Millisecond, "s": time.Second, "m": time.Minute, "h": time.Hour, "d": 24 * time.Hour, "w": 7 * 24 * time.Hour, "y": 365 * 24 * time.Hour, @@ -212,7 +216,8 @@ func scanOffsets(lower string) (back, ahead float64, ok bool) { // metricsql lexes 1h-30m as one duration whose parts after the // first negative one all subtract (DurationValue), so -1h-30m is // -90m and 1h-2h is -1h; the scan does not read such compounds. - if end < len(arg) && arg[end] == '-' { + // A minus not followed by a duration part is an operator. + if end < len(arg) && arg[end] == '-' && negativeDurationPart.MatchString(arg[end+1:]) { return 0, 0, false } d, ok := parseQueryDuration(arg[:end]) diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 95ee047d8..26db536cf 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -450,6 +450,9 @@ func TestQueryLookahead(t *testing.T) { {`x offset 1h - 30m`, 0}, {`x offset -1h + 30m`, time.Hour}, {`x offset -1h30m`, 90 * time.Minute}, + // A minus not followed by a duration is an operator. + {`x offset 1h-1`, 0}, + {`x offset 5m-y`, 0}, } for _, tt := range tests { t.Run(tt.query, func(t *testing.T) { From b876368a3fd057debcb8147fa96a1cb00d594522 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:16:24 +0700 Subject: [PATCH 362/518] fix(observability-proxy): refuse any minus right after an offset again Telling a compound duration from a subtraction by a duration-part pattern missed forms metricsql accepts after the minus: offset 5m-1.h is 55 minutes forward and offset 5m-.5h is 25, yet both read as five minutes back, and the lookup silently missed pods created in between. A pattern that followed scanSingleDuration exactly would still be a second copy of the grammar to keep in step, so any minus right after an offset's duration is refused again; offset 1h-1 and offset 5m-y now answer 422 as well, and metricsql cannot parse the latter anyway. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricslookback.go | 10 ++++------ internal/observabilityproxy/metricsparams_test.go | 10 +++++++--- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index 16f1f4cff..9f01936b7 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -49,10 +49,6 @@ var ( querySeconds = regexp.MustCompile(`^\d+(?:\.\d+)?$`) ) -// negativeDurationPart is what metricsql's scanSingleDuration accepts -// after a minus inside one duration token. -var negativeDurationPart = regexp.MustCompile(`^(?:\d+(?:\.\d+)?(?:ms|s|m|h|d|w|y|i)|\$__interval)`) - var queryDurationUnits = map[string]time.Duration{ "ms": time.Millisecond, "s": time.Second, "m": time.Minute, "h": time.Hour, "d": 24 * time.Hour, "w": 7 * 24 * time.Hour, "y": 365 * 24 * time.Hour, @@ -216,8 +212,10 @@ func scanOffsets(lower string) (back, ahead float64, ok bool) { // metricsql lexes 1h-30m as one duration whose parts after the // first negative one all subtract (DurationValue), so -1h-30m is // -90m and 1h-2h is -1h; the scan does not read such compounds. - // A minus not followed by a duration part is an operator. - if end < len(arg) && arg[end] == '-' && negativeDurationPart.MatchString(arg[end+1:]) { + // metricsql continues a duration after a minus in more forms than + // the scan can tell from a subtraction (-1.h and -.5h among them), + // so any minus right after the duration is refused. + if end < len(arg) && arg[end] == '-' { return 0, 0, false } d, ok := parseQueryDuration(arg[:end]) diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 26db536cf..15067ed6c 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -450,9 +450,6 @@ func TestQueryLookahead(t *testing.T) { {`x offset 1h - 30m`, 0}, {`x offset -1h + 30m`, time.Hour}, {`x offset -1h30m`, 90 * time.Minute}, - // A minus not followed by a duration is an operator. - {`x offset 1h-1`, 0}, - {`x offset 5m-y`, 0}, } for _, tt := range tests { t.Run(tt.query, func(t *testing.T) { @@ -477,6 +474,12 @@ func TestQueryLookahead(t *testing.T) { `x offset - 1h-30m`, `x offset 1h-2h`, `x offset 1h-30m`, + // metricsql continues a duration with -1.h or -.5h too, so any minus + // right after the duration is refused rather than told apart. + `x offset 5m-1.h`, + `x offset 5m-.5h`, + `x offset 1h-1`, + `x offset 5m-y`, `rate(x[5m] offset 1h-30m)`, // vmselect evaluates rollup_candlestick one step ahead // (evalRollupFuncWithoutAt), whatever the offset. @@ -498,6 +501,7 @@ func TestQueryLookahead(t *testing.T) { // values back onto earlier points, so a series that starts after the // evaluation time can reach the answer. func TestQueryLookaheadSubquery(t *testing.T) { + for _, q := range []string{`max_over_time(range_last(x)[1h:1h])`, `max_over_time(keep_next_value(x)[1h:1h])`} { got, ok := queryLookahead(q, time.Minute) if !ok || got < 2*time.Hour { From d164790946a4b3a557c1bc1204ca6b4f8b78e57c Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:16:31 +0700 Subject: [PATCH 363/518] test(observability-proxy): pin the subquery lookahead exactly The subquery sizing was tested only as "at least two hours", so which step drives it, the fallback to the query step for an empty subquery step, and its refusals (an unreadable step, an unclosed bracket, a sum past the duration range) were not held by any test. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observabilityproxy/metricsparams_test.go | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 15067ed6c..75b88d7f3 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -480,6 +480,12 @@ func TestQueryLookahead(t *testing.T) { `x offset 5m-.5h`, `x offset 1h-1`, `x offset 5m-y`, + // A subquery step the scan cannot read, an unclosed bracket, and a + // sum past the duration range. + `rate(x[1h:5i])`, + `rate(x[1h:$__interval])`, + `rate(x[1h`, + `max_over_time(x[1h:100y])`, `rate(x[5m] offset 1h-30m)`, // vmselect evaluates rollup_candlestick one step ahead // (evalRollupFuncWithoutAt), whatever the offset. @@ -501,7 +507,16 @@ func TestQueryLookahead(t *testing.T) { // values back onto earlier points, so a series that starts after the // evaluation time can reach the answer. func TestQueryLookaheadSubquery(t *testing.T) { - + // One explicit subquery and one "(": two levels of two steps, at the + // subquery's step, or at the query's step when the subquery has none. + for q, want := range map[string]time.Duration{ + `max_over_time(x[1h:1h])`: 4 * time.Hour, + `max_over_time(x[1h:])`: 4 * time.Minute, + } { + if got, ok := queryLookahead(q, time.Minute); !ok || got != want { + t.Errorf("queryLookahead(%q) = %s, %v; want %s", q, got, ok, want) + } + } for _, q := range []string{`max_over_time(range_last(x)[1h:1h])`, `max_over_time(keep_next_value(x)[1h:1h])`} { got, ok := queryLookahead(q, time.Minute) if !ok || got < 2*time.Hour { From 054fac6ab0fa4c3e618080744d168c3a57f4bcc9 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:16:59 +0700 Subject: [PATCH 364/518] docs(observability-proxy): name subquery steps in the app lookup texts Since the lookup reaches subquery steps past the evaluation time, the applicationScope comment still described only negative offsets, and a query refused for a subquery step the scan cannot read got a 422 that listed every other cause but that one. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 5 +++-- internal/observabilityproxy/server_app_test.go | 12 ++++++++++++ 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 9911abeeb..4918c9588 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -95,14 +95,15 @@ func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { // applicationScope looks the application's pods and PVCs up in store over // the samples the request reads: back by the query's own lookback, and -// forward to the latest sample a negative offset reads. A query that reads +// forward to the latest sample a negative offset or a subquery's +// evaluation past its end reads. A query that reads // further back than the configured lookback, or whose earliest or latest // sample is unknown, a failed lookup, and a list over its cap all fail the // request, because a successful answer would silently miss the // application's series. func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) { if req.last.IsZero() { - return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ anywhere in the query, rollup_candlestick, WITH with a negative offset, or an offset that is not a plain duration); use the namespace root") + return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ anywhere in the query, rollup_candlestick, WITH with a negative offset, an offset that is not a plain duration, or a subquery step that is not one); use the namespace root") } lim := s.cfg.Metrics.App if req.start.IsZero() || req.start.Before(req.from.Add(-lim.Lookback)) { diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 2d0d9b56e..c451d35af 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -553,3 +553,15 @@ func TestServerApplicationLookupCoversSubquery(t *testing.T) { t.Fatalf("lookup %s end = %s, want at least %s", r.path, r.form.Get("end"), formatTime(at.Add(2*time.Hour))) } } + +// TestServerApplicationNamesUnreadableSubqueryStep: the refusal names the +// cause when a subquery step is what the scan cannot read. +func TestServerApplicationNamesUnreadableSubqueryStep(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + resp := h.do(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+url.Values{"query": {"max_over_time(x[1h:5i])"}}.Encode(), "good", nil, nil) + body, _ := io.ReadAll(resp.Body) + _ = resp.Body.Close() + if resp.StatusCode != http.StatusUnprocessableEntity || !strings.Contains(string(body), "subquery step") { + t.Fatalf("answered %d %q, want 422 naming the subquery step", resp.StatusCode, body) + } +} From 8e491d15ddeb9dad0b82a9b5940342bcf2d666c5 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:21:16 +0700 Subject: [PATCH 365/518] fix(observability-proxy): accept only a token end after an offset metricsql also reads a trailing $__interval as part of an offset's duration: offset -1h$__interval is -(1h + step). The scan stopped at the "$" and read one hour, so the lookup missed pods created in the last step and the query answered 200 without them. Refusing a minus alone left every other continuation open. Only what certainly ends the duration token is now accepted after it: whitespace, a closing bracket or separator, or a binary operator other than minus; anything else answers 422 on the application root. The grammar comment no longer claims upper-case offset units are refused, as offsets are scanned lower-cased like metricsql reads them. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observabilityproxy/metricslookback.go | 19 +++++++++++++------ .../observabilityproxy/metricsparams_test.go | 4 ++++ .../observabilityproxy/server_app_test.go | 1 + 3 files changed, 18 insertions(+), 6 deletions(-) diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index 9f01936b7..55e78a0c2 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -41,14 +41,20 @@ const latencyOffset = 30 * time.Second // queryDurationPart is the subset of the MetricsQL duration grammar // (scanSingleDuration in metricsql) the scan understands. Step multiples -// (5i, $__interval), negative parts and upper-case units are left out on -// purpose: anything else makes the bound unknown. +// (5i, $__interval) and negative parts are left out on purpose: anything +// else makes the bound unknown. Offsets are scanned lower-cased, as +// metricsql reads units in any case; windows and steps in brackets are +// not, so an upper-case unit there is unknown. var ( queryDurationPart = regexp.MustCompile(`(\d+(?:\.\d+)?)(ms|s|m|h|d|w|y)`) queryDuration = regexp.MustCompile(`^(?:\d+(?:\.\d+)?(?:ms|s|m|h|d|w|y))+$`) querySeconds = regexp.MustCompile(`^\d+(?:\.\d+)?$`) ) +// offsetTerminators may follow an offset's duration: whitespace, a +// closing bracket or separator, or a binary operator other than minus. +const offsetTerminators = " \t\n\r)]},+*/^%<>=!" + var queryDurationUnits = map[string]time.Duration{ "ms": time.Millisecond, "s": time.Second, "m": time.Minute, "h": time.Hour, "d": 24 * time.Hour, "w": 7 * 24 * time.Hour, "y": 365 * 24 * time.Hour, @@ -212,10 +218,11 @@ func scanOffsets(lower string) (back, ahead float64, ok bool) { // metricsql lexes 1h-30m as one duration whose parts after the // first negative one all subtract (DurationValue), so -1h-30m is // -90m and 1h-2h is -1h; the scan does not read such compounds. - // metricsql continues a duration after a minus in more forms than - // the scan can tell from a subtraction (-1.h and -.5h among them), - // so any minus right after the duration is refused. - if end < len(arg) && arg[end] == '-' { + // metricsql continues a duration token in more forms than the scan + // reads: after a minus (-30m, -1.h, -.5h) and with a trailing + // $__interval, all summed by DurationValue. Only what certainly + // ends the token is accepted after the duration. + if end < len(arg) && !strings.ContainsRune(offsetTerminators, rune(arg[end])) { return 0, 0, false } d, ok := parseQueryDuration(arg[:end]) diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 75b88d7f3..0fbbcd7a9 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -480,6 +480,10 @@ func TestQueryLookahead(t *testing.T) { `x offset 5m-.5h`, `x offset 1h-1`, `x offset 5m-y`, + // metricsql also reads a trailing $__interval as part of the + // duration: -1h$__interval is -(1h + step). + `x offset -1h$__interval`, + `rate(x[5m]) offset -1h$__interval`, // A subquery step the scan cannot read, an unclosed bracket, and a // sum past the duration range. `rate(x[1h:5i])`, diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index c451d35af..8b3a4e587 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -523,6 +523,7 @@ func TestServerApplicationRefusesForwardReads(t *testing.T) { for _, query := range []string{ "container_cpu_usage_seconds_total offset -1h-30m", "container_cpu_usage_seconds_total offset 1h-2h", + "container_cpu_usage_seconds_total offset -1h$__interval", "rollup_candlestick(container_cpu_usage_seconds_total)", } { q := url.Values{"query": {query}, "time": {unix(testNow.Add(-2 * time.Hour))}} From 3198de893ad0869995111f8879a62a3155e14225 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:37:04 +0700 Subject: [PATCH 366/518] fix(observability-proxy): refuse escapes outside strings on the app root metricsql unescapes identifiers (unescapeIdent in parser.go), so rollup\_candlestick(x + y) is rollup_candlestick to vmselect while the lexical scan saw another name: the lookup ended ten minutes past the evaluation time and vmselect read nearly thirteen. Any escaped keyword the scan looks for, offset included, had the same hole. A backslash outside a string literal now makes the scan unknown, which answers 422 on the application root; escapes inside label values are unaffected. With rollup_candlestick refused, two steps per "(" stay enough: a subquery evaluates at most one step past its end plus the alignment of that end to the step (evalRollupFuncWithSubquery), and the one step a candlestick over a non-selector would add on top cannot occur. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 2 +- .../observabilityproxy/metricslookback.go | 32 ++++++++++++++++--- .../observabilityproxy/metricsparams_test.go | 10 ++++++ 3 files changed, 39 insertions(+), 5 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 4918c9588..2f0f23751 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -103,7 +103,7 @@ func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { // application's series. func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) { if req.last.IsZero() { - return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ anywhere in the query, rollup_candlestick, WITH with a negative offset, an offset that is not a plain duration, or a subquery step that is not one); use the namespace root") + return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ anywhere in the query, a backslash outside a string, rollup_candlestick, WITH with a negative offset, an offset that is not a plain duration, or a subquery step that is not one); use the namespace root") } lim := s.cfg.Metrics.App if req.start.IsZero() || req.start.Before(req.from.Add(-lim.Lookback)) { diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index 55e78a0c2..7f936307f 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -145,11 +145,12 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { // lookback, every "(" counts as a possible implicit subquery and each // term uses the largest step seen. Any @ makes the bound unknown, as it // does for queryLookback, and so does WITH, which can apply one offset -// several times. vmselect also evaluates rollup_candlestick one step ahead -// of the evaluation time (evalRollupFuncWithoutAt), so it makes the bound -// unknown too. +// several times, and a backslash outside a string, which can spell an +// identifier the scan looks for differently. vmselect also evaluates +// rollup_candlestick one step ahead of the evaluation time +// (evalRollupFuncWithoutAt), so it makes the bound unknown too. func queryLookahead(query string, step time.Duration) (time.Duration, bool) { - if strings.Contains(query, "@") { + if strings.Contains(query, "@") || escapeOutsideString(query) { return 0, false } lower := strings.ToLower(query) @@ -237,6 +238,29 @@ func scanOffsets(lower string) (back, ahead float64, ok bool) { } } +// escapeOutsideString reports a backslash outside a string literal. +// metricsql unescapes identifiers (unescapeIdent), so rollup\_candlestick +// or offs\et name what the lexical scan looks for without its spelling. +// Inside double or single quotes a backslash escapes the next character; +// inside backquotes it is literal. +func escapeOutsideString(query string) bool { + var quote byte + for i := 0; i < len(query); i++ { + c := query[i] + switch { + case quote == 0 && c == '\\': + return true + case quote == 0 && (c == '"' || c == '\'' || c == '`'): + quote = c + case quote != 0 && quote != '`' && c == '\\': + i++ + case quote != 0 && c == quote: + quote = 0 + } + } + return false +} + // isKeyword reports whether lower[start:end] stands on its own rather than // inside an identifier. func isKeyword(lower string, start, end int) bool { diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 0fbbcd7a9..cf452a74f 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -444,6 +444,9 @@ func TestQueryLookahead(t *testing.T) { {`rate(x[5m] offset -30m) offset -1h`, 92 * time.Minute}, {`x offset -1h + y offset 2h`, time.Hour}, {`node_offset_seconds - 1`, 0}, + // An escape inside a string literal is only part of a label value. + {`x{a=~"a\\.b"}`, 0}, + {`x{a="\"rollup\\_candlestick\""}`, 0}, // metricsql lexes a duration as one token, so a sign with spaces, // or a plus, is an operator after the offset, not part of it. {`x offset 1h+30m`, 0}, @@ -491,6 +494,13 @@ func TestQueryLookahead(t *testing.T) { `rate(x[1h`, `max_over_time(x[1h:100y])`, `rate(x[5m] offset 1h-30m)`, + // metricsql unescapes identifiers (unescapeIdent), so an escaped + // rollup_candlestick, or any escape outside a string, could hide + // what the scan looks for. + `rollup\_candlestick(x + y)`, + `rollup_candle\stick(x)`, + `rollup\x5fcandlestick(x)`, + `x offs\et -1h`, // vmselect evaluates rollup_candlestick one step ahead // (evalRollupFuncWithoutAt), whatever the offset. `rollup_candlestick(x)`, From a54760fd9cc145b79c6c901011fe5cb6857a530a Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:37:53 +0700 Subject: [PATCH 367/518] fix(observability-proxy): start the instant app lookup a latency early vmselect evaluates an instant query whose time is within 30s of its own clock at that clock minus 30s (queryOffset in prometheus.go), not at time minus 30s. For a time slightly in the future the evaluation starts earlier than the proxy assumed: time=now+29s with a 1s step read from six minutes before, while the lookup started at five minutes 32 seconds, and pods present only in that gap were missing from a 200. The evaluation instant is always within (time-60s, time], so the lookup of an instant query now starts one more latency offset early. That needs no assumption about the proxy's clock matching vmselect's. The store choice on the namespace root is left as it is. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 8 +++++++ .../observabilityproxy/server_app_test.go | 24 ++++++++++++++++++- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 2f0f23751..0fe8b05e9 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -110,6 +110,14 @@ func (s *Server) applicationScope(ctx context.Context, r *http.Request, store st return nil, errorf(http.StatusUnprocessableEntity, "the query reads further back than the application scope looks up pods (%s before the window, or an unknown lookback); use a shorter window or the namespace root", lim.Lookback) } start := req.start + // vmselect evaluates an instant query whose time is within 30s of its + // own clock at that clock minus 30s (queryOffset in prometheus.go), so + // the evaluation instant lies anywhere in (time-60s, time], while + // req.start assumes time-30s. One more latency offset covers it without + // trusting the proxy's clock to match vmselect's. + if req.path == "api/v1/query" { + start = start.Add(-latencyOffset) + } var lists [2][]string for i, l := range []struct { label, metric string diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 8b3a4e587..39b6ac60f 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -341,7 +341,8 @@ func TestServerApplicationLookbackBeyondCap(t *testing.T) { t.Fatalf("rate(x[5m]) answered %d, want 200", got) } r := h.requests()[0] - want := formatTime(buildMetrics(t, "api/v1/query", q).start) + // An instant query may be evaluated up to one more latency offset early. + want := formatTime(buildMetrics(t, "api/v1/query", q).start.Add(-latencyOffset)) if r.path != "/api/v1/label/pod/values" || r.form.Get("start") != want { t.Fatalf("lookup %s start = %s, want the query's earliest read %s", r.path, r.form.Get("start"), want) } @@ -566,3 +567,24 @@ func TestServerApplicationNamesUnreadableSubqueryStep(t *testing.T) { t.Fatalf("answered %d %q, want 422 naming the subquery step", resp.StatusCode, body) } } + +// TestServerApplicationLookupCoversLatencyOffset: vmselect evaluates an +// instant query whose time is within 30s of its own clock at its clock +// minus 30s (prometheus.go, queryOffset), so for a time slightly in the +// future it reads from earlier than time minus 30s. The lookup starts +// early enough for any evaluation instant in (time-60s, time]. +func TestServerApplicationLookupCoversLatencyOffset(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + at := testNow.Add(29 * time.Second) + q := url.Values{"query": {"container_cpu_usage_seconds_total"}, "time": {at.Format(time.RFC3339)}, "step": {"1s"}} + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK { + t.Fatalf("status = %d", got) + } + // The earliest read of the query evaluated at the earliest instant. + want := buildMetrics(t, "api/v1/query", q).start.Add(-latencyOffset) + r := h.requests()[0] + start, err := time.Parse(time.RFC3339, r.form.Get("start")) + if err != nil || start.After(want) { + t.Fatalf("lookup start = %s, want at most %s", r.form.Get("start"), formatTime(want)) + } +} From cf45a23bf3c79772928e606176133857ce800bd1 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:44:21 +0700 Subject: [PATCH 368/518] fix(observability-proxy): refuse comments outside strings on the app root The escape check tracked string literals but not comments: in metricsql a "#" outside a string comments out the rest of the line, so a quote inside a comment made the check treat the next line's code as string content, and rollup\_candlestick there was accepted again. The bracket and offset scans do not understand comments either. A "#" outside a string now makes the lookahead unknown as well, which answers 422 on the application root, rather than teaching every scan about comments; API clients do not send them. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 2 +- .../observabilityproxy/metricslookback.go | 23 +++++++++++-------- .../observabilityproxy/metricsparams_test.go | 5 ++++ 3 files changed, 19 insertions(+), 11 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 0fe8b05e9..53f8d5582 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -103,7 +103,7 @@ func applicationSelectors(ns, kind, name string, pods, pvcs []string) []string { // application's series. func (s *Server) applicationScope(ctx context.Context, r *http.Request, store string, req metricsRequest, ns, kind, name string) ([]string, *HTTPError) { if req.last.IsZero() { - return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ anywhere in the query, a backslash outside a string, rollup_candlestick, WITH with a negative offset, an offset that is not a plain duration, or a subquery step that is not one); use the namespace root") + return nil, errorf(http.StatusUnprocessableEntity, "the query reads samples at a time the application scope cannot cover (an @ anywhere in the query, a backslash or a comment outside a string, rollup_candlestick, WITH with a negative offset, an offset that is not a plain duration, or a subquery step that is not one); use the namespace root") } lim := s.cfg.Metrics.App if req.start.IsZero() || req.start.Before(req.from.Add(-lim.Lookback)) { diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index 7f936307f..b478f582b 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -145,12 +145,12 @@ func queryLookback(query string, step time.Duration) (time.Duration, bool) { // lookback, every "(" counts as a possible implicit subquery and each // term uses the largest step seen. Any @ makes the bound unknown, as it // does for queryLookback, and so does WITH, which can apply one offset -// several times, and a backslash outside a string, which can spell an -// identifier the scan looks for differently. vmselect also evaluates +// several times, and a backslash or a comment outside a string, which can +// hide what the scan looks for. vmselect also evaluates // rollup_candlestick one step ahead of the evaluation time // (evalRollupFuncWithoutAt), so it makes the bound unknown too. func queryLookahead(query string, step time.Duration) (time.Duration, bool) { - if strings.Contains(query, "@") || escapeOutsideString(query) { + if strings.Contains(query, "@") || unscannableOutsideString(query) { return 0, false } lower := strings.ToLower(query) @@ -238,17 +238,20 @@ func scanOffsets(lower string) (back, ahead float64, ok bool) { } } -// escapeOutsideString reports a backslash outside a string literal. -// metricsql unescapes identifiers (unescapeIdent), so rollup\_candlestick -// or offs\et name what the lexical scan looks for without its spelling. -// Inside double or single quotes a backslash escapes the next character; -// inside backquotes it is literal. -func escapeOutsideString(query string) bool { +// unscannableOutsideString reports a backslash or a "#" outside a string +// literal. metricsql unescapes identifiers (unescapeIdent), so +// rollup\_candlestick or offs\et name what the lexical scan looks for +// without its spelling; "#" starts a comment to the end of the line, and +// quotes or brackets inside one would mislead the string tracking here and +// the bracket and offset scans. Inside double or single quotes a backslash +// escapes the next character; inside backquotes it is literal, and a +// backquote escaped there makes metricsql refuse the string anyway. +func unscannableOutsideString(query string) bool { var quote byte for i := 0; i < len(query); i++ { c := query[i] switch { - case quote == 0 && c == '\\': + case quote == 0 && (c == '\\' || c == '#'): return true case quote == 0 && (c == '"' || c == '\'' || c == '`'): quote = c diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index cf452a74f..e201a6fa8 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -501,6 +501,11 @@ func TestQueryLookahead(t *testing.T) { `rollup_candle\stick(x)`, `rollup\x5fcandlestick(x)`, `x offs\et -1h`, + // A comment runs to the end of the line, so quotes in it would + // confuse the string tracking and brackets in it the scans. + "x # '\n+ rollup\\_candlestick(y) # '", + "x # \"\n+ rollup\\_candlestick(y) # \"", + "rate(x[5m]) # comment", // vmselect evaluates rollup_candlestick one step ahead // (evalRollupFuncWithoutAt), whatever the offset. `rollup_candlestick(x)`, From ac5a3c88c418b60216b0c6b93441f26b4229a98d Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 06:45:21 +0700 Subject: [PATCH 369/518] fix(observability-proxy): put the latency window in the instant start The second latency offset was added to the application lookup only, so the other readers of the instant query's earliest read still assumed time minus 30s: the store choice could send a query to the short-term store up to 30 seconds too late near its retention edge, the lookback cap compared against a start the lookup no longer used, and the field no longer meant the earliest instant read. The earliest read of an instant query now starts from time minus two latency offsets, the earliest instant vmselect can evaluate it at, and the lookup uses it as is. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 8 -------- internal/observabilityproxy/metricsparams.go | 6 +++++- .../observabilityproxy/metricsparams_test.go | 18 +++++++++++++++++- internal/observabilityproxy/server_app_test.go | 6 +++--- 4 files changed, 25 insertions(+), 13 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index 53f8d5582..e6df8ba02 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -110,14 +110,6 @@ func (s *Server) applicationScope(ctx context.Context, r *http.Request, store st return nil, errorf(http.StatusUnprocessableEntity, "the query reads further back than the application scope looks up pods (%s before the window, or an unknown lookback); use a shorter window or the namespace root", lim.Lookback) } start := req.start - // vmselect evaluates an instant query whose time is within 30s of its - // own clock at that clock minus 30s (queryOffset in prometheus.go), so - // the evaluation instant lies anywhere in (time-60s, time], while - // req.start assumes time-30s. One more latency offset covers it without - // trusting the proxy's clock to match vmselect's. - if req.path == "api/v1/query" { - start = start.Add(-latencyOffset) - } var lists [2][]string for i, l := range []struct { label, metric string diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 63be3c497..f02620b3c 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -144,7 +144,11 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti } out.Set("step", formatSeconds(step)) } - req.start = queryStart(out.Get("query"), at.Add(-latencyOffset), step) + // vmselect evaluates an instant query whose time is within the + // latency offset of its own clock at that clock minus the offset + // (queryOffset in prometheus.go), so the evaluation instant lies in + // (time-2*offset, time]; the earlier end needs no clock to agree. + req.start = queryStart(out.Get("query"), at.Add(-2*latencyOffset), step) req.from, req.to = at, at if ahead, ok := queryLookahead(out.Get("query"), step); ok { req.last = at.Add(ahead) diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index e201a6fa8..34d36f7e5 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -227,7 +227,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) { if req.params.Get("end") != "2026-11-04T12:00:00.000Z" || req.params.Get("start") != "2026-11-04T11:55:00.000Z" { t.Errorf("labels window = %s..%s, want the last 5m", req.params.Get("start"), req.params.Get("end")) } - if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Has("time") || !q.start.Equal(testNow.Add(-15*time.Minute-30*time.Second)) { + if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Has("time") || !q.start.Equal(testNow.Add(-15*time.Minute-2*latencyOffset)) { t.Errorf("instant query without time: params %v, start %v", q.params, q.start) } } @@ -543,3 +543,19 @@ func TestQueryLookaheadSubquery(t *testing.T) { } } } + +// TestBuildMetricsParamsInstantStartCoversLatencyOffset: vmselect evaluates +// an instant query whose time is within 30s of its clock at that clock +// minus 30s, anywhere in (time-60s, time], so the earliest read the store +// choice and the lookup rely on starts from time-60s. +func TestBuildMetricsParamsInstantStartCoversLatencyOffset(t *testing.T) { + at := testNow.Add(29 * time.Second) + req := buildMetrics(t, "api/v1/query", url.Values{"query": {"x"}, "time": {at.Format(time.RFC3339)}, "step": {"1s"}}) + lookback, ok := queryLookback("x", time.Second) + if !ok { + t.Fatal("lookback of x unknown") + } + if want := at.Add(-2*latencyOffset - lookback); req.start.After(want) { + t.Fatalf("start = %s, want at most %s", req.start, want) + } +} diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 39b6ac60f..e5c0d3710 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -341,8 +341,7 @@ func TestServerApplicationLookbackBeyondCap(t *testing.T) { t.Fatalf("rate(x[5m]) answered %d, want 200", got) } r := h.requests()[0] - // An instant query may be evaluated up to one more latency offset early. - want := formatTime(buildMetrics(t, "api/v1/query", q).start.Add(-latencyOffset)) + want := formatTime(buildMetrics(t, "api/v1/query", q).start) if r.path != "/api/v1/label/pod/values" || r.form.Get("start") != want { t.Fatalf("lookup %s start = %s, want the query's earliest read %s", r.path, r.form.Get("start"), want) } @@ -581,7 +580,8 @@ func TestServerApplicationLookupCoversLatencyOffset(t *testing.T) { t.Fatalf("status = %d", got) } // The earliest read of the query evaluated at the earliest instant. - want := buildMetrics(t, "api/v1/query", q).start.Add(-latencyOffset) + lookback, _ := queryLookback("container_cpu_usage_seconds_total", time.Second) + want := at.Add(-2*latencyOffset - lookback) r := h.requests()[0] start, err := time.Parse(time.RFC3339, r.form.Get("start")) if err != nil || start.After(want) { From 79ea8561de6f419d52830d3822a331d04c77e7c3 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 07:00:58 +0700 Subject: [PATCH 370/518] fix(observability-proxy): end a backquoted string only at an unescaped quote The string tracking treated a backslash inside backquotes as literal, but metricsql's scanString lets a backslash escape the closing quote in all three quote kinds, and a quoted label name keeps that escape ({`\``=""} parses). The proxy ended such a string early, read the rest of the query as string content, and accepted an escaped rollup_candlestick after it with the lookup ending too soon. Inside any quote a backslash now skips the next character, as scanString does, and the comment no longer claims metricsql refuses an escaped backquote. Assisted-by: LLM Signed-off-by: ZverGuy --- .../observabilityproxy/metricslookback.go | 9 +++++---- .../observabilityproxy/metricsparams_test.go | 5 +++++ .../observabilityproxy/server_app_test.go | 19 +++++++++++++++++++ 3 files changed, 29 insertions(+), 4 deletions(-) diff --git a/internal/observabilityproxy/metricslookback.go b/internal/observabilityproxy/metricslookback.go index b478f582b..3571afc4c 100644 --- a/internal/observabilityproxy/metricslookback.go +++ b/internal/observabilityproxy/metricslookback.go @@ -243,9 +243,10 @@ func scanOffsets(lower string) (back, ahead float64, ok bool) { // rollup\_candlestick or offs\et name what the lexical scan looks for // without its spelling; "#" starts a comment to the end of the line, and // quotes or brackets inside one would mislead the string tracking here and -// the bracket and offset scans. Inside double or single quotes a backslash -// escapes the next character; inside backquotes it is literal, and a -// backquote escaped there makes metricsql refuse the string anyway. +// the bracket and offset scans. Inside any of the three quotes a backslash +// skips the next character, as metricsql's scanString does: an escaped +// closing quote does not end the string, backquotes included (a quoted +// label name keeps such an escape). func unscannableOutsideString(query string) bool { var quote byte for i := 0; i < len(query); i++ { @@ -255,7 +256,7 @@ func unscannableOutsideString(query string) bool { return true case quote == 0 && (c == '"' || c == '\'' || c == '`'): quote = c - case quote != 0 && quote != '`' && c == '\\': + case quote != 0 && c == '\\': i++ case quote != 0 && c == quote: quote = 0 diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 34d36f7e5..2cc22a4ae 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -506,6 +506,11 @@ func TestQueryLookahead(t *testing.T) { "x # '\n+ rollup\\_candlestick(y) # '", "x # \"\n+ rollup\\_candlestick(y) # \"", "rate(x[5m]) # comment", + // metricsql lets a backslash escape the closing quote in backquotes + // too (scanString), and quoted label names keep such escapes, so + // a backquoted string must not end at an escaped backquote. + "container_cpu_usage_seconds_total{`\\``=\"\"} or rollup\\_candlestick(container_cpu_usage_seconds_total offset -5m+1h)", + "{`\\`a`} or rollup\\_candlestick(container_cpu_usage_seconds_total offset -5m+1h)", // vmselect evaluates rollup_candlestick one step ahead // (evalRollupFuncWithoutAt), whatever the offset. `rollup_candlestick(x)`, diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index e5c0d3710..ade81ca04 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -588,3 +588,22 @@ func TestServerApplicationLookupCoversLatencyOffset(t *testing.T) { t.Fatalf("lookup start = %s, want at most %s", r.form.Get("start"), formatTime(want)) } } + +// TestServerApplicationRefusesBackquoteEscapeTail: an escaped backquote +// inside a quoted label name does not end the string in metricsql, so the +// escaped rollup_candlestick after it must still be refused. +func TestServerApplicationRefusesBackquoteEscapeTail(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + for _, query := range []string{ + "container_cpu_usage_seconds_total{`\\``=\"\"} or rollup\\_candlestick(container_cpu_usage_seconds_total offset -5m+1h)", + "{`\\`a`} or rollup\\_candlestick(container_cpu_usage_seconds_total offset -5m+1h)", + } { + q := url.Values{"query": {query}, "time": {unix(testNow.Add(-6 * time.Hour))}} + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusUnprocessableEntity { + t.Errorf("%s answered %d, want 422", query, got) + } + } + if n := len(h.requests()); n != 0 { + t.Fatalf("upstream saw %d requests for refused queries", n) + } +} From e8e0cede61222e884441b7203094c18c13133736 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 18:09:46 +0700 Subject: [PATCH 371/518] fix(observability-proxy): forward the default instant query time An instant query without time had its application lookup sized at the proxy's clock, but the time was not forwarded, so vmselect evaluated at its own later clock. With lookups taking most of the request deadline, the evaluation lands well past the end of the lookup and leaves out pods first sampled in between. Forward the instant the lookup was sized for. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/metricsparams.go | 5 ++++- .../observabilityproxy/metricsparams_test.go | 2 +- .../observabilityproxy/server_app_test.go | 21 +++++++++++++++++++ 3 files changed, 26 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index f02620b3c..c40bc058f 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -134,8 +134,11 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti return req, herr } at = t - out.Set("time", formatTime(t)) } + // The default is forwarded too: the application lookup is sized + // for at, and vmselect's own clock reads later, by as long as the + // lookups took. + out.Set("time", formatTime(at)) step := instantDefaultStep if v := in.Get("step"); v != "" { var herr *HTTPError diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index 2cc22a4ae..ba14a9781 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -227,7 +227,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) { if req.params.Get("end") != "2026-11-04T12:00:00.000Z" || req.params.Get("start") != "2026-11-04T11:55:00.000Z" { t.Errorf("labels window = %s..%s, want the last 5m", req.params.Get("start"), req.params.Get("end")) } - if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Has("time") || !q.start.Equal(testNow.Add(-15*time.Minute-2*latencyOffset)) { + if q := buildMetrics(t, "api/v1/query", url.Values{"query": {"up"}}); q.params.Get("time") != "2026-11-04T12:00:00.000Z" || !q.start.Equal(testNow.Add(-15*time.Minute-2*latencyOffset)) { t.Errorf("instant query without time: params %v, start %v", q.params, q.start) } } diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index ade81ca04..7afaa3618 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -363,6 +363,27 @@ func TestServerApplicationLookupCoversNegativeOffset(t *testing.T) { } } +// TestServerApplicationForwardsDefaultTime: without time the lookup is +// sized at the proxy's clock, so that instant is forwarded. Otherwise +// vmselect evaluates at its own later clock, past the end of the lookup by +// however long the lookups took, and leaves out pods first sampled since. +func TestServerApplicationForwardsDefaultTime(t *testing.T) { + h := newPolicyHarness(t, Config{}, appMetricsUpstream(nil, nil, http.StatusOK), appPolicy, nil) + q := url.Values{"query": {"container_cpu_usage_seconds_total"}} + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?"+q.Encode(), "good"); got != http.StatusOK { + t.Fatalf("status = %d", got) + } + reqs := h.requests() + lookup, final := reqs[0], reqs[len(reqs)-1] + want := formatTime(testNow) + if lookup.path != "/api/v1/label/pod/values" || lookup.form.Get("end") != want { + t.Fatalf("lookup %s end = %s, want %s", lookup.path, lookup.form.Get("end"), want) + } + if final.path != "/api/v1/query" || final.form.Get("time") != want { + t.Fatalf("query %s time = %q, want the lookup's instant %s", final.path, final.form.Get("time"), want) + } +} + // TestServerApplicationRefusesUnboundedSampleTime refuses a query whose // samples may lie at a time the lookup cannot be sized for, rather than // answer with series silently left out. From 19dbee15fc7cb2aef22fd36151061d18cb26a207 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Thu, 5 Nov 2026 18:15:09 +0700 Subject: [PATCH 372/518] docs(observability-proxy): explain why both application roots need the namespace root The reason given for the second access review held only for the metrics root. The log root needs it too, because an ownerless pod keeps the lineage labels its creator set and can claim any application. The tenant ClusterRole comment also named only the namespace roots, though the application roots now depend on the same rule. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/route.go | 12 ++++++++---- internal/observabilityproxy/server_app_test.go | 7 ++++--- .../templates/tenant-clusterroles.yaml | 6 +++++- 3 files changed, 17 insertions(+), 8 deletions(-) diff --git a/internal/observabilityproxy/route.go b/internal/observabilityproxy/route.go index 49a7c1770..bc12c6276 100644 --- a/internal/observabilityproxy/route.go +++ b/internal/observabilityproxy/route.go @@ -132,10 +132,14 @@ func ParseRoute(escapedPath string) (Route, *HTTPError) { } // Reviews lists every access a request on the route needs. An application -// root also needs read access to its namespace root: its pod and PVC -// branches match by name across the window, so a name another application -// held earlier reaches that application's series, and a grant narrowed by -// resourceNames would read more than it names. +// root also needs read access to its namespace root, because its scope +// narrows within the namespace but does not isolate: on both roots the +// identity comes from lineage labels that the lineage webhook keeps as set +// on an object it finds no owner for, so anyone who can create a pod there +// can label it as the application; on the metrics root the pod and PVC +// branches also match by name across the window, so a name another +// application held earlier reaches that application's series. A grant +// narrowed by resourceNames would read more than it names. func (r Route) Reviews() []authorizationv1.ResourceAttributes { switch r.Kind { case ApplicationLog: diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 7afaa3618..461cbc16d 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -282,9 +282,10 @@ func TestServerApplicationMetricsBranches(t *testing.T) { // A grant narrowed to application "a" by resourceNames must not reach "b" // through any endpoint of either root. Without read access to the -// namespace roots it does not reach "a" either: the pod and PVC branches -// match by name, and a name another application used earlier in the -// window would pull its series in. +// namespace roots it does not reach "a" either: an ownerless pod keeps the +// lineage labels its creator set, so it can claim to be "a" on both roots, +// and on the metrics root the pod and PVC branches match by name, so a name +// another application used earlier in the window would pull its series in. func TestServerApplicationNarrowedGrant(t *testing.T) { h := newPolicyHarness(t, Config{}, appMetricsUpstream([]string{"a-0"}, nil, http.StatusOK), appPolicy, nil) paths := []string{ diff --git a/packages/system/observability-proxy/templates/tenant-clusterroles.yaml b/packages/system/observability-proxy/templates/tenant-clusterroles.yaml index a2bf94f3c..ff5f3b987 100644 --- a/packages/system/observability-proxy/templates/tenant-clusterroles.yaml +++ b/packages/system/observability-proxy/templates/tenant-clusterroles.yaml @@ -1,7 +1,11 @@ # Aggregated into cozy-tenant-view (and consequently use, admin, super-admin). # No API server serves cozystack.io logs or metrics: the rule exists only for # the SubjectAccessReview the observability proxy asks before it answers a -# read under /apis/cozystack.io/v1alpha1/namespaces//{logs,metrics}/. +# read under /apis/cozystack.io/v1alpha1/namespaces//{logs,metrics}/, +# and before a read under an application root, +# /apis/apps.cozystack.io/v1alpha1/namespaces////{log,metrics}/, +# which needs the namespace root too. Narrowing this role also cuts off the +# application roots. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: From 3a5eb4fbf200005c88da0e3c68da23d1de6679a0 Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 03:40:26 +0700 Subject: [PATCH 373/518] test(observability-proxy): cover stalled reviews on the application roots Application roots take two access reviews before any body is read; a POST to either must answer 503 within the request timeout when kube-apiserver stalls on them. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/server_deadline_test.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/internal/observabilityproxy/server_deadline_test.go b/internal/observabilityproxy/server_deadline_test.go index 6b367229c..5723365b3 100644 --- a/internal/observabilityproxy/server_deadline_test.go +++ b/internal/observabilityproxy/server_deadline_test.go @@ -428,6 +428,8 @@ func TestServerStalledReviewAnswersInTime(t *testing.T) { {"logs subject access review", logsRoot + "query", false}, {"metrics token review", metricsRoot + "api/v1/query", true}, {"metrics subject access review", metricsRoot + "api/v1/query", false}, + {"application logs subject access review", appRoot + "db/log/select/logsql/query", false}, + {"application metrics subject access review", appRoot + "db/metrics/api/v1/query", false}, } { t.Run(tc.name, func(t *testing.T) { h := newHarness(t, Config{RequestTimeout: 200 * time.Millisecond}, nil) From 6fe3e5f6ff831ab87b4af11f14db0e583716e5cb Mon Sep 17 00:00:00 2001 From: ZverGuy Date: Tue, 10 Nov 2026 03:46:57 +0700 Subject: [PATCH 374/518] fix(observability-proxy): refuse a partial application lookup A cluster vmselect answers label values with isPartial when a storage node does not respond, and the list it returns misses that node's names. Scoping the query to it would drop those pods' series without a word, which the lookup exists to prevent, so a partial list now fails the request as any other failed lookup does. Assisted-by: LLM Signed-off-by: ZverGuy --- internal/observabilityproxy/appscope.go | 10 ++++- .../observabilityproxy/server_app_test.go | 37 +++++++++++++++++++ 2 files changed, 45 insertions(+), 2 deletions(-) diff --git a/internal/observabilityproxy/appscope.go b/internal/observabilityproxy/appscope.go index e6df8ba02..659b55052 100644 --- a/internal/observabilityproxy/appscope.go +++ b/internal/observabilityproxy/appscope.go @@ -158,8 +158,9 @@ func (s *Server) lookupNames(ctx context.Context, r *http.Request, store, label, return nil, errorf(resp.StatusCode, "metrics store answered %s", resp.Status) } var body struct { - Status string `json:"status"` - Data []string `json:"data"` + Status string `json:"status"` + IsPartial bool `json:"isPartial"` + Data []string `json:"data"` } if err := json.NewDecoder(io.LimitReader(resp.Body, maxLookupBytes)).Decode(&body); err != nil { return nil, err @@ -170,8 +171,13 @@ func (s *Server) lookupNames(ctx context.Context, r *http.Request, store, label, if len(body.Data) > limit { // Over the cap before dropping empty values: the store cut the // list at limit+1, so a shorter filtered list could hide names. + // A partial list over the cap settles it too. return body.Data, nil } + // A storage node that did not answer leaves names out of the list. + if body.IsPartial { + return nil, errorf(http.StatusBadGateway, "metrics store answered a partial %s list", label) + } names := body.Data[:0] for _, n := range body.Data { if n != "" { diff --git a/internal/observabilityproxy/server_app_test.go b/internal/observabilityproxy/server_app_test.go index 461cbc16d..056e1a555 100644 --- a/internal/observabilityproxy/server_app_test.go +++ b/internal/observabilityproxy/server_app_test.go @@ -516,6 +516,43 @@ func TestServerApplicationLookupStoreErrors(t *testing.T) { } } +// TestServerApplicationRefusesPartialLookup: vmselect answers a label +// values lookup with isPartial when a vmstorage node is down, and a short +// pod list would drop the missing pods' series without a word. +func TestServerApplicationRefusesPartialLookup(t *testing.T) { + for _, label := range []string{"pod", "persistentvolumeclaim"} { + h := newPolicyHarness(t, Config{}, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/api/v1/label/"+label+"/values" { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"status":"success","isPartial":true,"data":["db-0"]}`) + return + } + appMetricsUpstream([]string{"p"}, nil, http.StatusOK)(w, r) + }, appPolicy, nil) + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?query=up", "good"); got != http.StatusBadGateway { + t.Errorf("partial %s lookup answered %d, want 502", label, got) + } + for _, req := range h.requests() { + if req.path == "/api/v1/query" { + t.Errorf("partial %s lookup still sent the query", label) + } + } + } + // A partial list already over the cap settles it: the full one is longer. + cfg := Config{Metrics: MetricsConfig{App: AppScopeLimits{Lookback: time.Hour, MaxPods: 1, MaxPVCs: 1}}} + h := newPolicyHarness(t, cfg, func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/api/v1/label/pod/values" { + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `{"status":"success","isPartial":true,"data":["db-0","db-1"]}`) + return + } + appMetricsUpstream([]string{"p"}, nil, http.StatusOK)(w, r) + }, appPolicy, nil) + if got := h.status(http.MethodGet, appRoot+"db.main/metrics/api/v1/query?query=up", "good"); got != http.StatusUnprocessableEntity { + t.Errorf("partial pod list over the cap answered %d, want 422", got) + } +} + // TestServerApplicationRefusesEveryAt: no @ form can be bounded, and the // answer says so rather than naming forms that would pass. func TestServerApplicationRefusesEveryAt(t *testing.T) { From a6af14ce7486deafee2d95a2f15bbba0a6bc175c Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 10:40:54 +0300 Subject: [PATCH 375/518] fix(postgres): build a backup's artifact from the destination it was written to The WAL archive destination is written by the pass that creates the cnpg.io/Backup and not again while it runs, yet the artifact took the destination, the endpoint and the credentials Secret from the strategy rendered when the Backup completed. A strategy or BackupClass edited while the backup ran gave an artifact that pointed somewhere nothing had been written, and a restore from it found no data. The pass that writes the destination now records it on the cnpg.io/Backup it creates, and the artifact is built from that record. The record lives on the Backup rather than on the BackupJob because a tenant can edit its BackupJobs but has no access to postgresql.cnpg.io objects: a record a tenant could rewrite would point the artifact, and a restore from it, which takes its S3 keys from the strategy, at an endpoint of the tenant's choosing. The artifact size probe measures the same place: the recorded location and credentials Secret, with the Secret keys and the endpoint CA, which the record does not carry, from the strategy. Should those no longer fit after an edit, the probe reports no size instead of measuring somewhere else. A record that cannot be read, or that names no destination path, creates no artifact and fails the job, since the strategy may no longer match the write. A Backup without a record was created by a controller that kept none, and its artifact is built from the strategy as before. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpg_artifact_size_test.go | 73 +++++++- .../cnpgstrategy_controller.go | 97 +++++++++- .../cnpgstrategy_controller_test.go | 169 +++++++++++++++++- 3 files changed, 323 insertions(+), 16 deletions(-) diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index 85e55e8c2..29209c5f1 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -284,7 +284,7 @@ func TestCreateCNPGBackupArtifact_SizeBytes(t *testing.T) { r := reconcilerWithLister(t, l) r.Client = newCNPGStrategyTestClient(t, s3CredsSecret("tenant", "creds")) - got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")) + got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "srv", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant")) if err != nil { t.Fatalf("createCNPGBackupArtifact: %v", err) } @@ -311,7 +311,7 @@ func TestCreateCNPGBackupArtifact_ExistingBackupSkipsProbe(t *testing.T) { cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}} rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()} - if _, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")); err != nil { + if _, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant")); err != nil { t.Fatalf("createCNPGBackupArtifact: %v", err) } if l.calls != 0 { @@ -331,7 +331,7 @@ func TestCreateCNPGBackupArtifact_AlreadyExistsRefetchReadsPastTheCache(t *testi store := newCNPGStrategyTestClient(t, backup) r := &BackupJobReconciler{Client: staleCache(t, store), APIReader: notFoundOnce(t, store)} rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://b/"}} - got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}}, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")) + got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}}, "postgres-pg", "srv", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant")) if err != nil || got == nil || got.Name != "bj-race" { t.Fatalf("createCNPGBackupArtifact = %v, %v; want the Backup that already exists", got, err) } @@ -349,7 +349,7 @@ func TestCreateCNPGBackupArtifact_UnreadableExistingBackupIsRetried(t *testing.T store := newCNPGStrategyTestClient(t, backup) r := &BackupJobReconciler{Client: staleCache(t, store), APIReader: failingReads(t, store)} rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://b/"}} - _, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}}, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")) + _, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}}, "postgres-pg", "srv", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant")) if !errors.Is(err, errBackupNotReadBack) { t.Errorf("err = %v, want errBackupNotReadBack", err) } @@ -409,7 +409,7 @@ func TestCreateCNPGBackupArtifact_FailedExistenceCheckSkipsProbe(t *testing.T) { cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}} rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()} - got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", rendered, newPostgresApp("pg", "tenant")) + got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant")) if err != nil || got == nil { t.Fatalf("createCNPGBackupArtifact = %v, %v; a transient read must not fail the backup", got, err) } @@ -451,3 +451,66 @@ func TestReconcileCNPG_RejectedCreateFailsTheJob(t *testing.T) { t.Errorf("phase = %q, want Failed for a rejected Create", persisted.Status.Phase) } } + +// The probe measures the base backup where it was written: the destination +// recorded on the cnpg.io/Backup when it was created, not the strategy as +// it reads by the time the backup completes. +func TestCreateCNPGBackupArtifact_SizeProbeReadsTheRecordedDestination(t *testing.T) { + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-moved"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"}, + }, + } + cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}} + rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()} + oldCreds := "creds-before-edit" + written := cnpgDestination{DestinationPath: "s3://old-bkt/before/edit/", EndpointURL: "https://old.example:9000", S3SecretRef: &oldCreds} + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: "before/edit/srv/base/20261021T100000/data.tar.gz", Size: 42}}} + r := reconcilerWithLister(t, l) + r.Client = newCNPGStrategyTestClient(t, s3CredsSecret("tenant", oldCreds)) + + got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", written, rendered, newPostgresApp("pg", "tenant")) + if err != nil { + t.Fatalf("createCNPGBackupArtifact: %v", err) + } + if l.bucket != "old-bkt" || l.prefix != "before/edit/srv/base/20261021T100000/" { + t.Errorf("listed %s/%s, want the recorded destination old-bkt/before/edit/srv/base/20261021T100000/", l.bucket, l.prefix) + } + if l.conn.Endpoint != "old.example:9000" { + t.Errorf("probed endpoint %q, want the recorded old.example:9000", l.conn.Endpoint) + } + if got.Status.Artifact == nil || got.Status.Artifact.SizeBytes != 42 { + t.Errorf("Artifact = %#v, want SizeBytes 42", got.Status.Artifact) + } +} + +// A backup recorded without a credentials Secret is not measured with the +// Secret the strategy names now: that Secret belongs to another location. +func TestCreateCNPGBackupArtifact_SizeProbeSkipsARecordWithoutCredentials(t *testing.T) { + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-nocreds"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"}, + }, + } + cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}} + rendered := &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *newSizedBarmanObjectStore()} + written := cnpgDestination{DestinationPath: "s3://bkt/tenant/pg/", EndpointURL: "https://s3.example:8443"} + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: "x", Size: 1}}} + r := reconcilerWithLister(t, l) + r.Client = newCNPGStrategyTestClient(t, s3CredsSecret("tenant", "creds")) + + got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", written, rendered, newPostgresApp("pg", "tenant")) + if err != nil { + t.Fatalf("createCNPGBackupArtifact: %v", err) + } + if l.calls != 0 { + t.Errorf("S3 listed %d times with the strategy's Secret for a backup recorded without one", l.calls) + } + if got.Status.Artifact == nil || got.Status.Artifact.SizeBytes != 0 { + t.Errorf("Artifact = %#v, want no size", got.Status.Artifact) + } +} diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 83ce9b41e..190e2db6c 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -219,6 +219,11 @@ const ( // instance pods still have to postdate the rollout's own start. backupCondDestinationWriteHeld = "DestinationWriteHeld" + // Annotation on a cnpg.io/Backup carrying, as JSON, the WAL archive + // destination written by the pass that created it: the destination + // path, the endpoint URL and the name of the credentials Secret. + cnpgWrittenDestinationAnnotation = "backups.cozystack.io/cnpg-written-destination" + cnpgPodRoleLabel = "cnpg.io/podRole" cnpgPodRoleInstance = "instance" ) @@ -526,7 +531,7 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp "cluster", clusterName, "live", attachment.flavor, "application", specFlavor) } - cnpgBackup, err := r.ensureCNPGBackup(ctx, j, clusterName) + cnpgBackup, err := r.ensureCNPGBackup(ctx, j, clusterName, cnpgDestinationOf(rendered)) if err != nil { return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to ensure cnpg.io/Backup: %v", err)) } @@ -546,7 +551,11 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp if j.Status.BackupRef != nil { return ctrl.Result{}, nil } - artifact, err := r.createCNPGBackupArtifact(ctx, j, resolved, cnpgBackup, clusterName, serverName, attachment.flavor, rendered, app) + destination, err := cnpgArtifactDestination(cnpgBackup, rendered) + if err != nil { + return r.markBackupJobFailed(ctx, j, fmt.Sprintf("refusing to create the Backup artifact: %v", err)) + } + artifact, err := r.createCNPGBackupArtifact(ctx, j, resolved, cnpgBackup, clusterName, serverName, attachment.flavor, destination, rendered, app) if err != nil { return r.failBackupArtifact(ctx, j, err) } @@ -869,7 +878,12 @@ func currentBarmanServerName(c *cnpgtypes.Cluster) string { // ensureCNPGBackup creates a one-shot postgresql.cnpg.io/Backup CR labelled // with the BackupJob, or returns the existing one if a previous reconcile // already created it. Idempotency relies on the OwningJob labels. -func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string) (*cnpgtypes.Backup, error) { +// +// written is the destination this pass wrote to the ObjectStore. A pass +// reaches the create only after that write: a job without a Backup writes +// the destination on every pass, and a pass that wrote nothing, or wrote it +// and waits for the plugin rollout, returns before this point. +func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string, written cnpgDestination) (*cnpgtypes.Backup, error) { existing, err := r.findCNPGBackupForJob(ctx, j) if err != nil { return nil, err @@ -877,6 +891,10 @@ func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1 if existing != nil { return existing, nil } + record, err := json.Marshal(written) + if err != nil { + return nil, fmt.Errorf("encode the WAL archive destination: %w", err) + } obj := &cnpgtypes.Backup{ ObjectMeta: metav1.ObjectMeta{ @@ -887,6 +905,7 @@ func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1 backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace, cnpgClusterLabel: clusterName, }, + Annotations: map[string]string{cnpgWrittenDestinationAnnotation: string(record)}, }, Spec: cnpgtypes.BackupSpec{ Method: cnpgtypes.BackupMethodPlugin, @@ -984,6 +1003,67 @@ func (r *BackupJobReconciler) waitCNPGBackupJob(ctx context.Context, j *backupsv return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil } +// cnpgDestination is the WAL archive destination recorded in +// cnpgWrittenDestinationAnnotation. S3SecretRef is nil when the strategy +// names no credentials Secret. +type cnpgDestination struct { + DestinationPath string `json:"destinationPath"` + EndpointURL string `json:"endpointURL,omitempty"` + S3SecretRef *string `json:"s3SecretRef,omitempty"` +} + +func cnpgDestinationOf(t *strategyv1alpha1.CNPGTemplate) cnpgDestination { + d := cnpgDestination{ + DestinationPath: t.BarmanObjectStore.DestinationPath, + EndpointURL: t.BarmanObjectStore.EndpointURL, + } + if t.BarmanObjectStore.S3Credentials != nil { + name := t.BarmanObjectStore.S3Credentials.SecretRef.Name + d.S3SecretRef = &name + } + return d +} + +// cnpgArtifactDestination returns the destination the artifact of b names: +// the one recorded when b was created. A Backup without the record was +// created by a controller that kept none, and falls back to the strategy +// rendered now. A record that names no destination is an error: the +// strategy may have changed since the write, so it cannot stand in. +func cnpgArtifactDestination(b *cnpgtypes.Backup, rendered *strategyv1alpha1.CNPGTemplate) (cnpgDestination, error) { + raw, ok := b.Annotations[cnpgWrittenDestinationAnnotation] + if !ok { + return cnpgDestinationOf(rendered), nil + } + var d cnpgDestination + if err := json.Unmarshal([]byte(raw), &d); err != nil { + return cnpgDestination{}, fmt.Errorf("read the %s annotation of cnpg.io/Backup %s/%s: %w", cnpgWrittenDestinationAnnotation, b.Namespace, b.Name, err) + } + if d.DestinationPath == "" { + return cnpgDestination{}, fmt.Errorf("the %s annotation of cnpg.io/Backup %s/%s names no destination path", cnpgWrittenDestinationAnnotation, b.Namespace, b.Name) + } + return d, nil +} + +// cnpgProbeStore is the object store the size probe reads: the location +// and the credentials Secret recorded for the backup, with the Secret keys +// and the endpoint CA the record does not carry taken from the strategy. +// After a strategy edit those may not fit the recorded location; the probe +// then reports no size rather than measuring somewhere else. +func cnpgProbeStore(rendered *strategyv1alpha1.BarmanObjectStoreTemplate, destination cnpgDestination) *strategyv1alpha1.BarmanObjectStoreTemplate { + store := rendered.DeepCopy() + store.DestinationPath = destination.DestinationPath + store.EndpointURL = destination.EndpointURL + if destination.S3SecretRef == nil { + store.S3Credentials = nil + } else { + if store.S3Credentials == nil { + store.S3Credentials = &strategyv1alpha1.S3CredentialsTemplate{} + } + store.S3Credentials.SecretRef.Name = *destination.S3SecretRef + } + return store +} + // createCNPGBackupArtifact materialises a Cozystack Backup resource carrying // the metadata callers need to drive a future restore. The source app's // spec.databases / spec.users are snapshotted into Status.UnderlyingResources @@ -995,6 +1075,7 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact( resolved *ResolvedBackupConfig, cnpgBackup *cnpgtypes.Backup, clusterName, serverName, flavor string, + destination cnpgDestination, rendered *strategyv1alpha1.CNPGTemplate, sourceApp *postgresapp.Postgres, ) (*backupsv1alpha1.Backup, error) { @@ -1010,12 +1091,12 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact( driverMD := map[string]string{ cnpgBackupNameKey: cnpgBackup.Name, cnpgServerNameKey: serverName, - cnpgDestinationPathKey: rendered.BarmanObjectStore.DestinationPath, - cnpgEndpointURLKey: rendered.BarmanObjectStore.EndpointURL, + cnpgDestinationPathKey: destination.DestinationPath, + cnpgEndpointURLKey: destination.EndpointURL, cnpgClusterNameKey: clusterName, } - if rendered.BarmanObjectStore.S3Credentials != nil { - driverMD[cnpgS3SecretRefKey] = rendered.BarmanObjectStore.S3Credentials.SecretRef.Name + if destination.S3SecretRef != nil { + driverMD[cnpgS3SecretRefKey] = *destination.S3SecretRef } // Copied rather than read back at restore time: barman retention deletes // the cnpg.io/Backup while the artifact can still be restored from. @@ -1048,7 +1129,7 @@ func (r *BackupJobReconciler) createCNPGBackupArtifact( }, } if probe { - backup.Status.Artifact.SizeBytes = r.cnpgArtifactSizeBytes(ctx, j.Namespace, &rendered.BarmanObjectStore, serverName, cnpgBackup.Status.BackupID) + backup.Status.Artifact.SizeBytes = r.cnpgArtifactSizeBytes(ctx, j.Namespace, cnpgProbeStore(&rendered.BarmanObjectStore, destination), serverName, cnpgBackup.Status.BackupID) } if j.Spec.PlanRef != nil { backup.Spec.PlanRef = j.Spec.PlanRef diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 8f0035aa6..23e4c182f 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -1075,7 +1075,7 @@ func TestCreateCNPGBackupArtifact_AlreadyExistsReturnsExisting(t *testing.T) { } sourceApp := newPostgresApp("pg", "tenant") - got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "postgres-pg", "postgresql", rendered, sourceApp) + got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "postgres-pg", "postgresql", cnpgDestinationOf(rendered), rendered, sourceApp) if err != nil { t.Fatalf("expected AlreadyExists to be swallowed, got error %v", err) } @@ -2207,7 +2207,7 @@ func TestEnsureCNPGBackup_UsesPluginMethod(t *testing.T) { c := newCNPGStrategyTestClient(t, j) r := &BackupJobReconciler{Client: c} - bk, err := r.ensureCNPGBackup(context.Background(), j, "postgres-app") + bk, err := r.ensureCNPGBackup(context.Background(), j, "postgres-app", cnpgDestination{DestinationPath: "s3://bucket/"}) if err != nil { t.Fatalf("ensureCNPGBackup: %v", err) } @@ -5311,7 +5311,7 @@ func TestCreateCNPGBackupArtifact_RecordsMajorVersion(t *testing.T) { rendered := &strategyv1alpha1.CNPGTemplate{ BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://b/"}, } - got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "postgres-pg", "postgresql", rendered, newPostgresApp("pg", "tenant")) + got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "postgres-pg", "postgresql", cnpgDestinationOf(rendered), rendered, newPostgresApp("pg", "tenant")) if err != nil { t.Fatalf("createCNPGBackupArtifact: %v", err) } @@ -5492,3 +5492,166 @@ func TestReconcileCNPGRestore_RefusesCrossMajorRestore(t *testing.T) { }) } } + +func (f *cnpgRolloutFixture) setStrategyDestination(destination, endpoint, secret string) { + f.t.Helper() + strategy := &strategyv1alpha1.CNPG{} + if err := f.c.Get(context.Background(), client.ObjectKey{Name: "cnpg-strategy"}, strategy); err != nil { + f.t.Fatalf("get strategy: %v", err) + } + strategy.Spec.Template.BarmanObjectStore.DestinationPath = destination + strategy.Spec.Template.BarmanObjectStore.EndpointURL = endpoint + strategy.Spec.Template.BarmanObjectStore.S3Credentials = &strategyv1alpha1.S3CredentialsTemplate{SecretRef: corev1.LocalObjectReference{Name: secret}} + if err := f.c.Update(context.Background(), strategy); err != nil { + f.t.Fatalf("update strategy: %v", err) + } +} + +// completeOwnBackup applies edit, when given, to the job's cnpg.io/Backup and +// marks it completed. +func (f *cnpgRolloutFixture) completeOwnBackup(edit func(b *cnpgtypes.Backup)) { + f.t.Helper() + backups := &cnpgtypes.BackupList{} + if err := f.c.List(context.Background(), backups, client.InNamespace(rolloutNS)); err != nil { + f.t.Fatalf("list cnpg.io/Backups: %v", err) + } + for i := range backups.Items { + b := &backups.Items[i] + if b.Labels[backupsv1alpha1.OwningJobNameLabel] != "bj" { + continue + } + if edit != nil { + edit(b) + } + b.Status.Phase = cnpgBackupPhaseComplete + if err := f.c.Update(context.Background(), b); err != nil { + f.t.Fatalf("complete cnpg.io/Backup: %v", err) + } + return + } + f.t.Fatalf("the job has no cnpg.io/Backup") +} + +func (f *cnpgRolloutFixture) artifact() *backupsv1alpha1.Backup { + f.t.Helper() + artifact := &backupsv1alpha1.Backup{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, artifact); err != nil { + if apierrors.IsNotFound(err) { + return nil + } + f.t.Fatalf("get Backup artifact: %v", err) + } + return artifact +} + +// The destination is written by the pass that creates the cnpg.io/Backup and +// not again while it runs, so the artifact has to name what that pass wrote: +// a strategy edited while the backup runs does not reach the backup, and must +// not reach its artifact either. The BackupJob, which a tenant can edit, +// carries no part of it. +func TestReconcileCNPG_ArtifactNamesTheDestinationItsBackupWasCreatedWith(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f.setStrategyDestination("s3://bucket/", "https://s3.one", "creds-one") + + j, backups := f.pass() + if len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups after the first pass, want 1; Ready = %+v", len(backups), apimeta.FindStatusCondition(j.Status.Conditions, "Ready")) + } + if got := f.liveDestination(); got != "s3://bucket/" { + t.Fatalf("destination = %q, want s3://bucket/", got) + } + want := `{"destinationPath":"s3://bucket/","endpointURL":"https://s3.one","s3SecretRef":"creds-one"}` + if got := backups[0].Annotations[cnpgWrittenDestinationAnnotation]; got != want { + t.Fatalf("%s on the created cnpg.io/Backup = %q, want %q", cnpgWrittenDestinationAnnotation, got, want) + } + + f.setStrategyDestination("s3://edited/", "https://s3.two", "creds-two") + j.Annotations = map[string]string{cnpgWrittenDestinationAnnotation: `{"destinationPath":"s3://tenant/","endpointURL":"https://tenant","s3SecretRef":"creds-one"}`} + if err := f.c.Update(context.Background(), j); err != nil { + t.Fatalf("annotate BackupJob: %v", err) + } + f.completeOwnBackup(nil) + j, _ = f.pass() + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseSucceeded { + t.Fatalf("phase = %q (%s), want Succeeded", j.Status.Phase, j.Status.Message) + } + artifact := f.artifact() + if artifact == nil { + t.Fatalf("no Backup artifact") + } + md := artifact.Spec.DriverMetadata + if md[cnpgDestinationPathKey] != "s3://bucket/" || md[cnpgEndpointURLKey] != "https://s3.one" || md[cnpgS3SecretRefKey] != "creds-one" { + t.Fatalf("artifact driver metadata = %v, want the destination written with the Backup (s3://bucket/, https://s3.one, creds-one)", md) + } +} + +// A strategy without credentials records none, and the artifact names none. +func TestReconcileCNPG_ArtifactOfARecordWithoutCredentials(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://bucket/") + if _, backups := f.pass(); len(backups) != 1 || backups[0].Annotations[cnpgWrittenDestinationAnnotation] != `{"destinationPath":"s3://bucket/"}` { + t.Fatalf("cnpg.io/Backups after the first pass = %+v, want one recording only s3://bucket/", backups) + } + f.setStrategyDestination("s3://edited/", "https://s3.two", "creds-two") + f.completeOwnBackup(nil) + j, _ := f.pass() + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseSucceeded { + t.Fatalf("phase = %q (%s), want Succeeded", j.Status.Phase, j.Status.Message) + } + md := f.artifact().Spec.DriverMetadata + if _, ok := md[cnpgS3SecretRefKey]; ok || md[cnpgDestinationPathKey] != "s3://bucket/" || md[cnpgEndpointURLKey] != "" { + t.Fatalf("artifact driver metadata = %v, want s3://bucket/ with no endpoint and no credentials", md) + } +} + +// A record that names no usable destination cannot say where the backup is, +// so no artifact is created rather than one pointing at the strategy as it +// reads now. +func TestReconcileCNPG_ArtifactRefusedForAnUnusableDestinationRecord(t *testing.T) { + for name, value := range map[string]string{ + "empty": "", + "unreadable": "{not json", + "not an object": `"s3://bucket/"`, + "null": "null", + "no fields": "{}", + "empty destination": `{"destinationPath":"","endpointURL":"https://s3.one"}`, + // Decoding sets the destination path and skips the mistyped + // endpoint, which would leave the artifact on the default one. + "mistyped endpoint": `{"destinationPath":"s3://bucket/","endpointURL":7}`, + } { + t.Run(name, func(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + if _, backups := f.pass(); len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups after the first pass, want 1", len(backups)) + } + f.completeOwnBackup(func(b *cnpgtypes.Backup) { + if b.Annotations == nil { + b.Annotations = map[string]string{} + } + b.Annotations[cnpgWrittenDestinationAnnotation] = value + }) + j, _ := f.pass() + if f.artifact() != nil { + t.Fatalf("an artifact was created from the record %q", value) + } + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed || !strings.Contains(j.Status.Message, cnpgWrittenDestinationAnnotation) { + t.Fatalf("phase = %q (%s), want Failed naming %s", j.Status.Phase, j.Status.Message, cnpgWrittenDestinationAnnotation) + } + }) + } +} + +// A cnpg.io/Backup created by a controller that recorded nothing keeps +// getting its artifact from the strategy. +func TestReconcileCNPG_ArtifactOfABackupWithoutARecord(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://bucket/") + f.seedCNPGBackup("bj-legacy", "bj", rolloutCluster, cnpgBackupPhaseComplete) + + j, _ := f.pass() + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseSucceeded { + t.Fatalf("phase = %q (%s), want Succeeded", j.Status.Phase, j.Status.Message) + } + artifact := f.artifact() + if artifact == nil || artifact.Spec.DriverMetadata[cnpgDestinationPathKey] != "s3://bucket/" { + t.Fatalf("artifact = %+v, want one built from the strategy", artifact) + } +} From e5ccde82e2258a55e97b0dd6e5375caed5b59b86 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 10:47:21 +0300 Subject: [PATCH 376/518] fix(postgres): read cnpg Backups from the apiserver before writing the destination The queue that keeps two BackupJobs of one application from writing the WAL archive destination at once listed cnpg.io/Backups from the informer cache. The controller runs one reconcile at a time under leader election, so a Backup created by one job's pass already exists when the next pass starts, but the cache can still miss it. The next job then took the queue as empty, rewrote the destination under the running backup and left that backup's recorded destination naming a place its WAL no longer went. A job could likewise miss its own fresh Backup, write the destination again and create a second Backup. The queue now lists Backups from the apiserver, and a job that finds no Backup of its own in the cache checks the apiserver before it writes or creates one. Only a leader handover, when two processes reconcile for a moment, can still let two jobs through. The pass that creates the Backup no longer looks it up a second time right before the create. Nothing can create it in between, and once the lookup can reach the apiserver, a failed second lookup would come after the destination write and fail the job instead of retrying. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 35 +++-- .../cnpgstrategy_controller_test.go | 130 +++++++++++++++++- 2 files changed, 150 insertions(+), 15 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 190e2db6c..3eb3421d6 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -531,7 +531,7 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp "cluster", clusterName, "live", attachment.flavor, "application", specFlavor) } - cnpgBackup, err := r.ensureCNPGBackup(ctx, j, clusterName, cnpgDestinationOf(rendered)) + cnpgBackup, err := r.ensureCNPGBackup(ctx, j, clusterName, existingBackup, cnpgDestinationOf(rendered)) if err != nil { return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to ensure cnpg.io/Backup: %v", err)) } @@ -876,18 +876,16 @@ func currentBarmanServerName(c *cnpgtypes.Cluster) string { } // ensureCNPGBackup creates a one-shot postgresql.cnpg.io/Backup CR labelled -// with the BackupJob, or returns the existing one if a previous reconcile -// already created it. Idempotency relies on the OwningJob labels. +// with the BackupJob, or returns existing, the job's Backup as looked up at +// the start of the pass. It does not look again: a reconcile of the same +// job cannot run in between, and a failed lookup here would come after the +// destination write and fail the job. // // written is the destination this pass wrote to the ObjectStore. A pass // reaches the create only after that write: a job without a Backup writes // the destination on every pass, and a pass that wrote nothing, or wrote it // and waits for the plugin rollout, returns before this point. -func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string, written cnpgDestination) (*cnpgtypes.Backup, error) { - existing, err := r.findCNPGBackupForJob(ctx, j) - if err != nil { - return nil, err - } +func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string, existing *cnpgtypes.Backup, written cnpgDestination) (*cnpgtypes.Backup, error) { if existing != nil { return existing, nil } @@ -923,17 +921,27 @@ func (r *BackupJobReconciler) ensureCNPGBackup(ctx context.Context, j *backupsv1 // findCNPGBackupForJob returns the cnpg.io/Backup labelled with the // BackupJob's OwningJob{Name,Namespace}, if any. Returns (nil, nil) when // no match is found. +// +// A miss in the cache is checked against the apiserver: the cache can lag +// behind a Backup the previous pass created, and taking that miss as no +// Backup would write the destination under it and create a second one. func (r *BackupJobReconciler) findCNPGBackupForJob(ctx context.Context, j *backupsv1alpha1.BackupJob) (*cnpgtypes.Backup, error) { - list := &cnpgtypes.BackupList{} - if err := r.List(ctx, list, + opts := []client.ListOption{ client.InNamespace(j.Namespace), client.MatchingLabels{ backupsv1alpha1.OwningJobNameLabel: j.Name, backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace, }, - ); err != nil { + } + list := &cnpgtypes.BackupList{} + if err := r.List(ctx, list, opts...); err != nil { return nil, err } + if len(list.Items) == 0 { + if err := r.apiReader().List(ctx, list, opts...); err != nil { + return nil, err + } + } if len(list.Items) == 0 { return nil, nil } @@ -944,9 +952,12 @@ func (r *BackupJobReconciler) findCNPGBackupForJob(ctx context.Context, j *backu // running and whose cnpg.io/Backup of clusterName has not finished, or "" when // there is none. A Backup left behind by a finished or deleted job does not // hold the queue. +// +// The Backups are read from the apiserver, not the cache, which can still +// miss a Backup another job's earlier pass created. func (r *BackupJobReconciler) otherCNPGBackupInFlight(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string) (string, error) { list := &cnpgtypes.BackupList{} - if err := r.List(ctx, list, client.InNamespace(j.Namespace)); err != nil { + if err := r.apiReader().List(ctx, list, client.InNamespace(j.Namespace)); err != nil { return "", err } var holders []string diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 23e4c182f..de525f0a7 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -5,6 +5,7 @@ import ( "context" "errors" "fmt" + "slices" "strings" "testing" "time" @@ -2207,7 +2208,7 @@ func TestEnsureCNPGBackup_UsesPluginMethod(t *testing.T) { c := newCNPGStrategyTestClient(t, j) r := &BackupJobReconciler{Client: c} - bk, err := r.ensureCNPGBackup(context.Background(), j, "postgres-app", cnpgDestination{DestinationPath: "s3://bucket/"}) + bk, err := r.ensureCNPGBackup(context.Background(), j, "postgres-app", nil, cnpgDestination{DestinationPath: "s3://bucket/"}) if err != nil { t.Fatalf("ensureCNPGBackup: %v", err) } @@ -2567,8 +2568,8 @@ func (f *cnpgRolloutFixture) seedOtherBackupJob(name string, phase backupsv1alph func TestReconcileCNPG_RunningBackupDoesNotReapplyObjectStore(t *testing.T) { f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") f.seedCNPGBackup("bj-cnpg", "bj", rolloutCluster, "running") - // Two jobs can pass the queue together, since it reads the cache; once - // both have a Backup neither may wait on the other. + // Two jobs can still pass the queue together, across a leader handover; + // once both have a Backup neither may wait on the other. f.seedOtherBackupJob("bj-other", backupsv1alpha1.BackupJobPhaseRunning) f.seedCNPGBackup("bj-other-cnpg", "bj-other", rolloutCluster, "running") writes := 0 @@ -5655,3 +5656,126 @@ func TestReconcileCNPG_ArtifactOfABackupWithoutARecord(t *testing.T) { t.Fatalf("artifact = %+v, want one built from the strategy", artifact) } } + +// cacheLaggingBehind makes the reconciler's cached client miss the named +// cnpg.io/Backups, as an informer does right after they are created, while +// its uncached reader sees them. +func (f *cnpgRolloutFixture) cacheLaggingBehind(hidden ...string) { + f.r.APIReader = f.c + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if err := cl.List(ctx, list, opts...); err != nil { + return err + } + if backups, ok := list.(*cnpgtypes.BackupList); ok { + backups.Items = slices.DeleteFunc(backups.Items, func(b cnpgtypes.Backup) bool { + return slices.Contains(hidden, b.Name) + }) + } + return nil + }, + }) +} + +// Reconciles run one at a time, so a Backup created by one pass exists by +// the next; the queue has to see it even when the cache does not yet, or +// the next job rewrites the destination under it and the record on that +// Backup no longer names where its WAL goes. +func TestReconcileCNPG_QueueSeesABackupTheCacheHasNotCaught(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f.seedOtherBackupJob("bj-other", backupsv1alpha1.BackupJobPhaseRunning) + f.seedCNPGBackup("bj-other-cnpg", "bj-other", rolloutCluster, "running") + f.cacheLaggingBehind("bj-other-cnpg") + + j, backups := f.pass() + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForOtherBackup" { + t.Fatalf("Ready = %+v, want WaitingForOtherBackup", ready) + } + if got := f.liveDestination(); got != "s3://other/" { + t.Fatalf("destination = %q, rewritten under the other job's Backup", got) + } + if len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups, want only the other job's", len(backups)) + } +} + +// The job's own Backup, missing from the cache right after it was created, +// is neither created again nor written under. +func TestReconcileCNPG_OwnBackupTheCacheHasNotCaught(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f.seedCNPGBackup("bj-cnpg", "bj", rolloutCluster, "running") + f.cacheLaggingBehind("bj-cnpg") + + _, backups := f.pass() + if len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups, want the job's one", len(backups)) + } + if got := f.liveDestination(); got != "s3://other/" { + t.Fatalf("destination = %q, rewritten under the job's own Backup", got) + } +} + +// failingBackupLists makes the uncached reader fail the cnpg.io/Backup lists +// for which fail, given whether the list selects the job's own Backup by +// label and how many such lists came before, returns true. +func (f *cnpgRolloutFixture) failingBackupLists(fail func(own bool, n int) bool) { + counts := map[bool]int{} + f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*cnpgtypes.BackupList); ok { + o := &client.ListOptions{} + o.ApplyOptions(opts) + own := o.LabelSelector != nil && !o.LabelSelector.Empty() + counts[own]++ + if fail(own, counts[own]) { + return apierrors.NewServiceUnavailable("apiserver unavailable") + } + } + return cl.List(ctx, list, opts...) + }, + }) +} + +// A failed apiserver read of the Backups is retried, not taken as the job +// failing: nothing has been written yet that the retry would not redo. +func TestReconcileCNPG_BackupListErrorIsReturned(t *testing.T) { + for name, own := range map[string]bool{"own Backup": true, "queue": false} { + t.Run(name, func(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f.failingBackupLists(func(o bool, _ int) bool { return o == own }) + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := f.r.reconcileCNPG(context.Background(), j, f.resolved); err == nil { + t.Fatalf("reconcileCNPG swallowed the Backup list error") + } + got := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, got); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if got.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("BackupJob failed on a Backup list error: %s", got.Status.Message) + } + if dest := f.liveDestination(); dest != "s3://other/" { + t.Fatalf("destination = %q, written although the Backups could not be read", dest) + } + }) + } +} + +// The pass that writes the destination looks its own Backup up once, at the +// start: a second lookup before the create could only fail, after the write, +// and fail the job with it. +func TestReconcileCNPG_LooksUpItsOwnBackupOncePerPass(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f.failingBackupLists(func(own bool, n int) bool { return own && n > 1 }) + + j, backups := f.pass() + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("BackupJob failed: %s", j.Status.Message) + } + if len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups, want 1", len(backups)) + } +} From 325d2b815c6b310c33d415957442d08a79197b77 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 14:34:04 +0300 Subject: [PATCH 377/518] fix(postgres): keep the plugin attach time when its status write is lost The rollout check took the attach time from the PluginRolledOut condition on the BackupJob, which the pass that attaches the plugin writes only after the Cluster apply went through. When that status write is lost to a conflict or a controller restart, the next pass finds the plugin attached and falls back to the latch written before the attach. An instance recreated between the latch and the attach then counts as one that rolled with the plugin, and the cnpg.io/Backup is created against an instance that has no plugin sidecar; a latch older than the deadline also fails a rollout that has only just begun. The apply that attaches the plugin now records the time on the Cluster itself, so the record cannot be lost apart from the attach. The rollout check and its deadline use the later of the two times. A record that does not parse fails the job rather than guess which instances rolled. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 56 +++++++-- .../cnpgstrategy_controller_test.go | 112 ++++++++++++++++++ 2 files changed, 157 insertions(+), 11 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 3eb3421d6..5869103ca 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -177,14 +177,15 @@ const ( // Cap on the wall-clock time a BackupJob spends before its // cnpg.io/Backup exists on each of two waits. The plugin rollout is - // counted from the PluginRolledOut condition, or from the release + // counted from the attach, the later of the PluginRolledOut condition and + // the Cluster's cnpgPluginAttachedAtAnnotation, or from the release // recorded in DestinationWriteHeld when that is later. Retrying the WAL // archive destination write is counted from the first failure since the // last successful write or the last wait for another backup or a - // restore. A Backup that exists is - // not capped: CloudNativePG keeps a healthy one pending while another - // backup of the same Cluster runs, or while it restarts one on another - // instance, so no fixed deadline tells those apart from a stuck one. + // restore. A Backup that exists is not capped: CloudNativePG keeps a + // healthy one pending while another backup of the same Cluster runs, or + // while it restarts one on another instance, so no fixed deadline tells + // those apart from a stuck one. cnpgDefaultBackupDeadline = 30 * time.Minute cnpgClusterHealthyPhase = "Cluster in healthy state" @@ -195,9 +196,10 @@ const ( // instance pod spec (the plugin injects its sidecar), so CNPG rolls every // instance, and a cnpg.io/Backup created meanwhile fails with "requested // plugin is not available" or "instance manager was restarted during - // backup" and never recovers. False holds the Backup back; its - // LastTransitionTime is what the instance pods must postdate (see - // cnpgReasonAttachingPlugin and cnpgPluginRolloutPending). + // backup" and never recovers. False holds the Backup back; the instance + // pods must postdate its LastTransitionTime and the Cluster's + // cnpgPluginAttachedAtAnnotation (see cnpgReasonAttachingPlugin and + // cnpgPluginRolloutPending). backupCondPluginRolledOut = "PluginRolledOut" // Reason of the PluginRolledOut latch recorded before the write that @@ -224,6 +226,12 @@ const ( // path, the endpoint URL and the name of the credentials Secret. cnpgWrittenDestinationAnnotation = "backups.cozystack.io/cnpg-written-destination" + // Annotation on a cnpg.io/Cluster carrying, as RFC 3339, when a backup + // job attached the barman-cloud plugin. It is written by the same apply + // that attaches the plugin, so unlike the PluginRolledOut condition it + // cannot be lost to a failed BackupJob status write after the attach. + cnpgPluginAttachedAtAnnotation = "backups.cozystack.io/barman-plugin-attached-at" + cnpgPodRoleLabel = "cnpg.io/podRole" cnpgPodRoleInstance = "instance" ) @@ -487,12 +495,25 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } return ctrl.Result{RequeueAfter: cnpgPollInterval}, nil } - pending, err := r.cnpgPluginRolloutPending(ctx, j.Namespace, clusterName, attachment, rollout.LastTransitionTime) + // The Cluster's record covers an attach whose status write was lost. + attachedAt := rollout.LastTransitionTime + if attachment.attachedAt != "" { + recorded, err := time.Parse(time.RFC3339, attachment.attachedAt) + if err != nil { + return r.markBackupJobFailed(ctx, j, fmt.Sprintf( + "cannot tell when the barman-cloud plugin was attached: the %s annotation of cnpg.io/Cluster %s/%s: %v", + cnpgPluginAttachedAtAnnotation, j.Namespace, clusterName, err)) + } + if recorded.After(attachedAt.Time) { + attachedAt = metav1.NewTime(recorded) + } + } + pending, err := r.cnpgPluginRolloutPending(ctx, j.Namespace, clusterName, attachment, attachedAt) if err != nil { return ctrl.Result{}, err } if pending != "" { - deadlineFrom := rollout.LastTransitionTime + deadlineFrom := attachedAt if held := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationWriteHeld); held != nil && held.LastTransitionTime.After(deadlineFrom.Time) { deadlineFrom = held.LastTransitionTime @@ -744,6 +765,7 @@ func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, name flavor: cnpgImageFlavor(existing.Spec.ImageName), clusterStatus: existing.Status, } + attachment.attachedAt = existing.Annotations[cnpgPluginAttachedAtAnnotation] if live := currentBarmanServerName(existing); live != "" { attachment.serverName = live } else { @@ -788,6 +810,15 @@ func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, name patch := newCNPGClusterPatch(namespace, clusterName) patch.Spec.Plugins = []cnpgtypes.PluginConfiguration{buildBarmanPlugin(objStoreName, serverName)} + // Re-sent unchanged on later applies: the field manager owns it, and an + // apply that left it out would delete it. + stamp := existing.Annotations[cnpgPluginAttachedAtAnnotation] + if attachment.missing { + stamp = metav1.Now().UTC().Format(time.RFC3339) + } + if stamp != "" { + patch.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: stamp} + } if err := r.Patch(ctx, patch, client.Apply, client.FieldOwner(cnpgFieldManager), client.ForceOwnership); err != nil { return cnpgPluginAttachment{}, err } @@ -802,7 +833,10 @@ type cnpgPluginAttachment struct { // flavor is the image flavor the live Cluster runs. flavor string // missing reports a Cluster that carried no barman-cloud plugin. - missing bool + missing bool + // attachedAt is the cnpgPluginAttachedAtAnnotation the Cluster carried, + // "" when no backup job attached the plugin. + attachedAt string clusterStatus cnpgtypes.ClusterStatus } diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index de525f0a7..255be9a27 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2924,6 +2924,118 @@ func TestReconcileCNPG_PluginRolloutCountsFromTheSuccessfulAttach(t *testing.T) } } +// The status write that records the attach can be lost after the attach +// itself went through, by a conflict or a restart; an instance recreated +// between the latch and the attach still has to roll, and the rollout +// deadline still counts from the attach. +func TestReconcileCNPG_LostAttachRecordStillWaitsForTheRollout(t *testing.T) { + latchedAt := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)).Truncate(time.Second) + oldPod := newRolloutInstancePod(latchedAt.Add(-time.Hour)) + f := newCNPGRolloutFixture(t, newRolloutBackupJob(latchedAt.Add(-time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase), oldPod) + + j, _ := f.pass() // latch + latch := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) + if latch == nil { + t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) + } + latch.LastTransitionTime = metav1.NewTime(latchedAt) + if err := f.c.Status().Update(context.Background(), j); err != nil { + t.Fatalf("age the latch: %v", err) + } + f.replacePod(oldPod, newRolloutInstancePod(latchedAt.Add(time.Minute))) + + base := f.r.Client + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + SubResourceUpdate: func(ctx context.Context, cl client.Client, sub string, obj client.Object, opts ...client.SubResourceUpdateOption) error { + return apierrors.NewConflict(backupsv1alpha1.GroupVersion.WithResource("backupjobs").GroupResource(), obj.GetName(), errors.New("the object has been modified")) + }, + }) + j = &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := f.r.reconcileCNPG(context.Background(), j, f.resolved); err == nil { + t.Fatalf("the pass that attached the plugin reported no lost status write") + } + f.r.Client = base + + attached := &cnpgtypes.Cluster{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: rolloutCluster}, attached); err != nil { + t.Fatalf("get Cluster: %v", err) + } + if len(attached.Spec.Plugins) == 0 { + t.Fatalf("the plugin was not attached") + } + live := newRolloutCluster(cnpgClusterHealthyPhase, attached.Spec.Plugins...) + live.Annotations = attached.Annotations + f.observeCNPG(live) + + var backups []cnpgtypes.Backup + for range 3 { + if j, backups = f.pass(); len(backups) != 0 || j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + break + } + } + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created against an instance recreated before the attach: %+v", backups) + } + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("the job failed instead of waiting for the rollout: %s", j.Status.Message) + } +} + +// A later write to a Cluster that already carries the plugin re-sends the +// attach record as it is: moving it would make the instances that rolled +// with the plugin look older than the attach. +func TestReconcileCNPG_LaterWriteKeepsTheAttachRecord(t *testing.T) { + const stamp = "2026-01-02T03:04:05Z" + cluster := newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)) + cluster.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: stamp} + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), cluster, + newRolloutInstancePod(time.Now().Add(-time.Hour))) + + var sent []map[string]string + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + if _, ok := obj.(*cnpgtypes.Cluster); ok { + sent = append(sent, obj.GetAnnotations()) + } + return cl.Patch(ctx, obj, patch, opts...) + }, + }) + if _, backups := f.pass(); len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups, want the one this pass creates", len(backups)) + } + if len(sent) != 1 { + t.Fatalf("got %d Cluster writes, want 1", len(sent)) + } + if got := sent[0][cnpgPluginAttachedAtAnnotation]; got != stamp { + t.Errorf("the write sent %s = %q, want the recorded %q", cnpgPluginAttachedAtAnnotation, got, stamp) + } +} + +// An attach record that does not parse cannot tell which instances rolled +// with the plugin, so the job fails naming it instead of guessing. +func TestReconcileCNPG_UnreadableAttachRecordFailsTheJob(t *testing.T) { + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase), newRolloutInstancePod(time.Now().Add(-time.Hour))) + if j, _ := f.pass(); apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) == nil { + t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) + } + live := newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)) + live.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: "yesterday"} + f.observeCNPG(live) + + j, backups := f.pass() + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created with an unreadable attach record: %+v", backups) + } + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed || !strings.Contains(j.Status.Message, cnpgPluginAttachedAtAnnotation) { + t.Errorf("phase %q, message %q; want Failed naming %s", j.Status.Phase, j.Status.Message, cnpgPluginAttachedAtAnnotation) + } +} + // A plugin attached by someone else after the latch, another backup job or // the chart re-rendering a restored Cluster outside the platform flow, // started the rollout before this job's write, which only finds it there: From 8d92aee796edaa1d4347b964e1d06de9d8fef655 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 14:40:44 +0300 Subject: [PATCH 378/518] test(postgres): pin that an older attach record keeps the latch The rollout check takes the later of the latch and the Cluster's attach record. A record left by an earlier attach of a plugin since removed is older than the latch when someone who keeps no record attaches the plugin again, and taking it would accept an instance recreated before the latch as one that rolled with the plugin. The record is absent not only when no backup job attached the plugin but also when the chart did, or a controller that kept no record, so the comment on the field says that. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 3 +- .../cnpgstrategy_controller_test.go | 38 +++++++++++++++++++ 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 5869103ca..37597678d 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -835,7 +835,8 @@ type cnpgPluginAttachment struct { // missing reports a Cluster that carried no barman-cloud plugin. missing bool // attachedAt is the cnpgPluginAttachedAtAnnotation the Cluster carried, - // "" when no backup job attached the plugin. + // "" when it carries no record: the plugin was attached by the chart, or + // before the record existed. attachedAt string clusterStatus cnpgtypes.ClusterStatus } diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 255be9a27..b9a10b56e 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2985,6 +2985,44 @@ func TestReconcileCNPG_LostAttachRecordStillWaitsForTheRollout(t *testing.T) { } } +// A record older than the latch is left from an earlier attach of a plugin +// since removed; the plugin found now was attached by someone who keeps no +// record, after the latch, so an instance recreated between the record and +// the latch still has to roll. +func TestReconcileCNPG_AttachRecordOlderThanTheLatchKeepsTheLatch(t *testing.T) { + latchedAt := time.Now().Add(-10 * time.Minute).Truncate(time.Second) + oldPod := newRolloutInstancePod(latchedAt.Add(-2 * time.Hour)) + f := newCNPGRolloutFixture(t, newRolloutBackupJob(latchedAt.Add(-2*time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase), oldPod) + + j, _ := f.pass() // latch + latch := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) + if latch == nil { + t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) + } + latch.LastTransitionTime = metav1.NewTime(latchedAt) + if err := f.c.Status().Update(context.Background(), j); err != nil { + t.Fatalf("age the latch: %v", err) + } + f.replacePod(oldPod, newRolloutInstancePod(latchedAt.Add(-time.Minute))) + live := newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)) + live.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: latchedAt.Add(-time.Hour).UTC().Format(time.RFC3339)} + f.observeCNPG(live) + + var backups []cnpgtypes.Backup + for range 3 { + if j, backups = f.pass(); len(backups) != 0 || j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + break + } + } + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created against an instance recreated before the latch: %+v", backups) + } + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("the job failed instead of waiting for the rollout: %s", j.Status.Message) + } +} + // A later write to a Cluster that already carries the plugin re-sends the // attach record as it is: moving it would make the instances that rolled // with the plugin look older than the attach. From fe03589d9136e2d8c6152c3a518d1e443aae0855 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 15:42:34 +0300 Subject: [PATCH 379/518] test(postgres): cover a lost attach record on the retry of a failed attach A status write that fails with the apiserver unavailable takes the same path as one lost to a conflict, but the attach it follows can itself be the retry of an apply that failed first, which leaves its own condition on the job. The test now runs that sequence as well, so the rollout check is held to it whichever way the record was lost. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller_test.go | 133 +++++++++++------- 1 file changed, 81 insertions(+), 52 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index b9a10b56e..4753dda68 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2925,63 +2925,92 @@ func TestReconcileCNPG_PluginRolloutCountsFromTheSuccessfulAttach(t *testing.T) } // The status write that records the attach can be lost after the attach -// itself went through, by a conflict or a restart; an instance recreated -// between the latch and the attach still has to roll, and the rollout -// deadline still counts from the attach. +// itself went through, by a conflict, an apiserver that is briefly +// unavailable, or a restart, including on the retry of an attach that +// failed; an instance recreated between the latch and the attach still has +// to roll, and the rollout deadline still counts from the attach. func TestReconcileCNPG_LostAttachRecordStillWaitsForTheRollout(t *testing.T) { - latchedAt := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)).Truncate(time.Second) - oldPod := newRolloutInstancePod(latchedAt.Add(-time.Hour)) - f := newCNPGRolloutFixture(t, newRolloutBackupJob(latchedAt.Add(-time.Hour)), - newRolloutCluster(cnpgClusterHealthyPhase), oldPod) + jobs := backupsv1alpha1.GroupVersion.WithResource("backupjobs").GroupResource() + for _, tc := range []struct { + name string + failFirstAttach bool + statusWriteError error + }{ + {name: "conflict", statusWriteError: apierrors.NewConflict(jobs, "bj", errors.New("the object has been modified"))}, + {name: "retried attach, apiserver unavailable", failFirstAttach: true, statusWriteError: apierrors.NewServiceUnavailable("the server is currently unable to handle the request")}, + } { + t.Run(tc.name, func(t *testing.T) { + latchedAt := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)).Truncate(time.Second) + oldPod := newRolloutInstancePod(latchedAt.Add(-time.Hour)) + f := newCNPGRolloutFixture(t, newRolloutBackupJob(latchedAt.Add(-time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase), oldPod) - j, _ := f.pass() // latch - latch := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) - if latch == nil { - t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) - } - latch.LastTransitionTime = metav1.NewTime(latchedAt) - if err := f.c.Status().Update(context.Background(), j); err != nil { - t.Fatalf("age the latch: %v", err) - } - f.replacePod(oldPod, newRolloutInstancePod(latchedAt.Add(time.Minute))) + j, _ := f.pass() // latch + latch := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) + if latch == nil { + t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) + } + latch.LastTransitionTime = metav1.NewTime(latchedAt) + if err := f.c.Status().Update(context.Background(), j); err != nil { + t.Fatalf("age the latch: %v", err) + } + f.replacePod(oldPod, newRolloutInstancePod(latchedAt.Add(time.Minute))) - base := f.r.Client - f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ - SubResourceUpdate: func(ctx context.Context, cl client.Client, sub string, obj client.Object, opts ...client.SubResourceUpdateOption) error { - return apierrors.NewConflict(backupsv1alpha1.GroupVersion.WithResource("backupjobs").GroupResource(), obj.GetName(), errors.New("the object has been modified")) - }, - }) - j = &backupsv1alpha1.BackupJob{} - if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { - t.Fatalf("get BackupJob: %v", err) - } - if _, err := f.r.reconcileCNPG(context.Background(), j, f.resolved); err == nil { - t.Fatalf("the pass that attached the plugin reported no lost status write") - } - f.r.Client = base + base := f.r.Client + if tc.failFirstAttach { + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + if _, ok := obj.(*cnpgtypes.Cluster); ok { + return apierrors.NewServiceUnavailable("the server is currently unable to handle the request") + } + return cl.Patch(ctx, obj, patch, opts...) + }, + }) + if j, _ = f.pass(); apimeta.IsStatusConditionTrue(j.Status.Conditions, backupCondPluginRolledOut) { + t.Fatalf("a failed attach reported the rollout done: %+v", j.Status.Conditions) + } + if applied := apimeta.FindStatusCondition(j.Status.Conditions, backupCondDestinationApplied); applied == nil || applied.Reason != "ClusterPluginApplyRetrying" { + t.Fatalf("%s = %+v, want the attach retrying", backupCondDestinationApplied, applied) + } + } + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + SubResourceUpdate: func(ctx context.Context, cl client.Client, sub string, obj client.Object, opts ...client.SubResourceUpdateOption) error { + return tc.statusWriteError + }, + }) + j = &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := f.r.reconcileCNPG(context.Background(), j, f.resolved); err == nil { + t.Fatalf("the pass that attached the plugin reported no lost status write") + } + f.r.Client = base - attached := &cnpgtypes.Cluster{} - if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: rolloutCluster}, attached); err != nil { - t.Fatalf("get Cluster: %v", err) - } - if len(attached.Spec.Plugins) == 0 { - t.Fatalf("the plugin was not attached") - } - live := newRolloutCluster(cnpgClusterHealthyPhase, attached.Spec.Plugins...) - live.Annotations = attached.Annotations - f.observeCNPG(live) + attached := &cnpgtypes.Cluster{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: rolloutCluster}, attached); err != nil { + t.Fatalf("get Cluster: %v", err) + } + if len(attached.Spec.Plugins) == 0 { + t.Fatalf("the plugin was not attached") + } + live := newRolloutCluster(cnpgClusterHealthyPhase, attached.Spec.Plugins...) + live.Annotations = attached.Annotations + f.observeCNPG(live) - var backups []cnpgtypes.Backup - for range 3 { - if j, backups = f.pass(); len(backups) != 0 || j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { - break - } - } - if len(backups) != 0 { - t.Fatalf("a cnpg.io/Backup was created against an instance recreated before the attach: %+v", backups) - } - if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { - t.Fatalf("the job failed instead of waiting for the rollout: %s", j.Status.Message) + var backups []cnpgtypes.Backup + for range 3 { + if j, backups = f.pass(); len(backups) != 0 || j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + break + } + } + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created against an instance recreated before the attach: %+v", backups) + } + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("the job failed instead of waiting for the rollout: %s", j.Status.Message) + } + }) } } From 8a631311a7ac4f751c14e11742182801730ad4f2 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 15:51:56 +0300 Subject: [PATCH 380/518] fix(postgres): fail a backup job whose plugin attach record is in the future The rollout check takes the later of the latch and the Cluster's attach record, and the record is never rewritten while the plugin stays attached. A record that parses but lies in the future made every instance look older than the plugin and counted the rollout deadline from that time, so the job waited past the deadline by as far as the record lay ahead, without end for one far enough out. Such a record says no more about when the plugin was attached than one that does not parse, and fails the job the same way. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 4 ++ .../cnpgstrategy_controller_test.go | 39 ++++++++++++------- 2 files changed, 28 insertions(+), 15 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 37597678d..ac58fa4bc 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -499,6 +499,10 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp attachedAt := rollout.LastTransitionTime if attachment.attachedAt != "" { recorded, err := time.Parse(time.RFC3339, attachment.attachedAt) + if recorded.After(time.Now()) { + // Every instance would predate it, and the deadline would count from it. + err = fmt.Errorf("%s is in the future", attachment.attachedAt) + } if err != nil { return r.markBackupJobFailed(ctx, j, fmt.Sprintf( "cannot tell when the barman-cloud plugin was attached: the %s annotation of cnpg.io/Cluster %s/%s: %v", diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 4753dda68..2f116734b 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -3083,23 +3083,32 @@ func TestReconcileCNPG_LaterWriteKeepsTheAttachRecord(t *testing.T) { } // An attach record that does not parse cannot tell which instances rolled -// with the plugin, so the job fails naming it instead of guessing. +// with the plugin, so the job fails naming it instead of guessing. One in the +// future would hold every instance as older than the plugin and push the +// rollout deadline out by as far as it lies ahead, so it fails the same way. func TestReconcileCNPG_UnreadableAttachRecordFailsTheJob(t *testing.T) { - f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), - newRolloutCluster(cnpgClusterHealthyPhase), newRolloutInstancePod(time.Now().Add(-time.Hour))) - if j, _ := f.pass(); apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) == nil { - t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) - } - live := newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)) - live.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: "yesterday"} - f.observeCNPG(live) + for _, tc := range []struct{ name, record string }{ + {name: "does not parse", record: "yesterday"}, + {name: "in the future", record: time.Now().Add(time.Hour).UTC().Format(time.RFC3339)}, + } { + t.Run(tc.name, func(t *testing.T) { + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase), newRolloutInstancePod(time.Now().Add(-time.Hour))) + if j, _ := f.pass(); apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) == nil { + t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) + } + live := newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)) + live.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: tc.record} + f.observeCNPG(live) - j, backups := f.pass() - if len(backups) != 0 { - t.Fatalf("a cnpg.io/Backup was created with an unreadable attach record: %+v", backups) - } - if j.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed || !strings.Contains(j.Status.Message, cnpgPluginAttachedAtAnnotation) { - t.Errorf("phase %q, message %q; want Failed naming %s", j.Status.Phase, j.Status.Message, cnpgPluginAttachedAtAnnotation) + j, backups := f.pass() + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created with an unreadable attach record: %+v", backups) + } + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed || !strings.Contains(j.Status.Message, cnpgPluginAttachedAtAnnotation) { + t.Errorf("phase %q, message %q; want Failed naming %s", j.Status.Phase, j.Status.Message, cnpgPluginAttachedAtAnnotation) + } + }) } } From b4551eb379f22b9e2121daa96bf53bf5999e8120 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 23:58:57 +0300 Subject: [PATCH 381/518] fix(postgres): read restores from the apiserver before writing the destination A backup job that is about to write the WAL archive destination first checks for a restore of its application, because a write landing during the restore puts the restored Cluster back on the source's WAL prefix and the restore fails with an expected-empty-archive error after it has purged the target. That check read RestoreJobs from the informer cache, which can still miss one created a moment ago. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backupjob_restore_lock.go | 4 +-- .../cnpgstrategy_controller.go | 5 ++-- .../cnpgstrategy_controller_test.go | 28 +++++++++++++++++++ 3 files changed, 33 insertions(+), 4 deletions(-) diff --git a/internal/backupcontroller/backupjob_restore_lock.go b/internal/backupcontroller/backupjob_restore_lock.go index 3782527d9..6b662cad5 100644 --- a/internal/backupcontroller/backupjob_restore_lock.go +++ b/internal/backupcontroller/backupjob_restore_lock.go @@ -38,7 +38,7 @@ const ConditionReasonRestoreInProgress = "RestoreInProgress" // restored cluster back on the source's WAL prefix, where its recovery fails // with "Expected empty archive". Plan-created runs are the ones that hit // this, since nothing else looks at restores before creating a BackupJob. -func activeRestoreTargeting(ctx context.Context, c client.Client, namespace string, app corev1.TypedLocalObjectReference) (string, error) { +func activeRestoreTargeting(ctx context.Context, c client.Reader, namespace string, app corev1.TypedLocalObjectReference) (string, error) { app = NormalizeApplicationRef(app) list := &backupsv1alpha1.RestoreJobList{} if err := c.List(ctx, list, client.InNamespace(namespace)); err != nil { @@ -80,7 +80,7 @@ func activeRestoreTargeting(ctx context.Context, c client.Client, namespace stri // reads options.targetNamespace, so the option is looked at only when the // Backup's strategy is Velero — the CNPG and Job-based drivers ignore it and // restore in place. -func restoreJobTarget(ctx context.Context, c client.Client, rj *backupsv1alpha1.RestoreJob) (corev1.TypedLocalObjectReference, bool, error) { +func restoreJobTarget(ctx context.Context, c client.Reader, rj *backupsv1alpha1.RestoreJob) (corev1.TypedLocalObjectReference, bool, error) { backup := &backupsv1alpha1.Backup{} if err := c.Get(ctx, client.ObjectKey{Namespace: rj.Namespace, Name: rj.Spec.BackupRef.Name}, backup); err != nil { if client.IgnoreNotFound(err) == nil { diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index ac58fa4bc..cae1a95d1 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -365,8 +365,9 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp if writeDestination { // The gate in Reconcile sees only a job that has not started, and // this one may have waited in the queue for as long as a restore - // takes to begin. - restore, err := activeRestoreTargeting(ctx, r.Client, j.Namespace, j.Spec.ApplicationRef) + // takes to begin. Read from the apiserver: the cache can still miss + // a RestoreJob created a moment ago. + restore, err := activeRestoreTargeting(ctx, r.apiReader(), j.Namespace, j.Spec.ApplicationRef) if err != nil { return ctrl.Result{}, err } diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 2f116734b..0b0531f59 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -3209,6 +3209,34 @@ func (f *cnpgRolloutFixture) seedRestoreOfApp() *backupsv1alpha1.RestoreJob { return restore } +// The restore check before the destination write reads the apiserver: the +// cache can still miss a RestoreJob created a moment ago, and a write landing +// in that gap can put the restored Cluster back on the source's WAL prefix. +func TestReconcileCNPG_RestoreCheckReadsPastTheCache(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + f.seedRestoreOfApp() + f.r.APIReader = f.c + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*backupsv1alpha1.RestoreJobList); ok { + return nil + } + return cl.List(ctx, list, opts...) + }, + }) + + j, backups := f.pass() + if got := f.liveDestination(); got != "s3://other/" { + t.Errorf("destination written during a restore the cache had not caught: %q", got) + } + if len(backups) != 0 { + t.Errorf("a cnpg.io/Backup was created during a restore the cache had not caught: %+v", backups) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != ConditionReasonRestoreInProgress { + t.Errorf("Ready = %+v, want %s", ready, ConditionReasonRestoreInProgress) + } +} + // A job past its start writes the WAL archive destination only while no // restore rewrites its application. The restore gate in Reconcile looks only // at a job that has not started, and a job released from the queue, or one From 9a8025ca1432f73fcdca79ed0584f009160aa9fa Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Tue, 10 Nov 2026 00:00:52 +0300 Subject: [PATCH 382/518] fix(postgres): record the serverName a backup was archived under The artifact took its destination from the record written when the cnpg.io/Backup was created, but its serverName, the archive prefix under that destination, from the live Cluster when the backup completed. A restore that re-renders the Cluster in between gives it another serverName, and the artifact then pairs the original destination with a prefix the backup was never written to, so a later restore searches the wrong archive. The record now carries the serverName too; a record written before it did still falls back to the live Cluster. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 16 +++++-- .../cnpgstrategy_controller_test.go | 46 ++++++++++++++++++- 2 files changed, 57 insertions(+), 5 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index cae1a95d1..75c3b3f06 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -223,7 +223,8 @@ const ( // Annotation on a cnpg.io/Backup carrying, as JSON, the WAL archive // destination written by the pass that created it: the destination - // path, the endpoint URL and the name of the credentials Secret. + // path, the endpoint URL, the name of the credentials Secret and the + // serverName the Cluster archived under. cnpgWrittenDestinationAnnotation = "backups.cozystack.io/cnpg-written-destination" // Annotation on a cnpg.io/Cluster carrying, as RFC 3339, when a backup @@ -557,7 +558,9 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp "cluster", clusterName, "live", attachment.flavor, "application", specFlavor) } - cnpgBackup, err := r.ensureCNPGBackup(ctx, j, clusterName, existingBackup, cnpgDestinationOf(rendered)) + written := cnpgDestinationOf(rendered) + written.ServerName = serverName + cnpgBackup, err := r.ensureCNPGBackup(ctx, j, clusterName, existingBackup, written) if err != nil { return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to ensure cnpg.io/Backup: %v", err)) } @@ -581,6 +584,11 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp if err != nil { return r.markBackupJobFailed(ctx, j, fmt.Sprintf("refusing to create the Backup artifact: %v", err)) } + // A restore can re-render the Cluster under another serverName + // between the backup and this pass. + if destination.ServerName != "" { + serverName = destination.ServerName + } artifact, err := r.createCNPGBackupArtifact(ctx, j, resolved, cnpgBackup, clusterName, serverName, attachment.flavor, destination, rendered, app) if err != nil { return r.failBackupArtifact(ctx, j, err) @@ -1056,11 +1064,13 @@ func (r *BackupJobReconciler) waitCNPGBackupJob(ctx context.Context, j *backupsv // cnpgDestination is the WAL archive destination recorded in // cnpgWrittenDestinationAnnotation. S3SecretRef is nil when the strategy -// names no credentials Secret. +// names no credentials Secret. ServerName is empty in a record written +// before it carried one. type cnpgDestination struct { DestinationPath string `json:"destinationPath"` EndpointURL string `json:"endpointURL,omitempty"` S3SecretRef *string `json:"s3SecretRef,omitempty"` + ServerName string `json:"serverName,omitempty"` } func cnpgDestinationOf(t *strategyv1alpha1.CNPGTemplate) cnpgDestination { diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 0b0531f59..8604d8c1a 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -5777,7 +5777,7 @@ func TestReconcileCNPG_ArtifactNamesTheDestinationItsBackupWasCreatedWith(t *tes if got := f.liveDestination(); got != "s3://bucket/" { t.Fatalf("destination = %q, want s3://bucket/", got) } - want := `{"destinationPath":"s3://bucket/","endpointURL":"https://s3.one","s3SecretRef":"creds-one"}` + want := `{"destinationPath":"s3://bucket/","endpointURL":"https://s3.one","s3SecretRef":"creds-one","serverName":"` + rolloutCluster + `"}` if got := backups[0].Annotations[cnpgWrittenDestinationAnnotation]; got != want { t.Fatalf("%s on the created cnpg.io/Backup = %q, want %q", cnpgWrittenDestinationAnnotation, got, want) } @@ -5802,10 +5802,52 @@ func TestReconcileCNPG_ArtifactNamesTheDestinationItsBackupWasCreatedWith(t *tes } } +// The serverName is the archive prefix under the destination, so the record +// carries it too: a restore that re-renders the Cluster between the backup +// and its artifact gives the live Cluster another one. A record written +// before it carried a serverName falls back to the live Cluster. +func TestReconcileCNPG_ArtifactNamesTheServerNameItsBackupWasCreatedWith(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://bucket/") + _, backups := f.pass() + want := `{"destinationPath":"s3://bucket/","serverName":"` + rolloutCluster + `"}` + if len(backups) != 1 || backups[0].Annotations[cnpgWrittenDestinationAnnotation] != want { + t.Fatalf("cnpg.io/Backups after the first pass = %+v, want one recording %s", backups, want) + } + f.observeCNPG(newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, "restored-server"))) + f.completeOwnBackup(nil) + j, _ := f.pass() + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseSucceeded { + t.Fatalf("phase = %q (%s), want Succeeded", j.Status.Phase, j.Status.Message) + } + artifact := f.artifact() + if got := artifact.Spec.DriverMetadata[cnpgServerNameKey]; got != rolloutCluster { + t.Errorf("artifact serverName = %q, want %q recorded with the Backup", got, rolloutCluster) + } + if !strings.HasPrefix(artifact.Status.Artifact.URI, "cnpg://"+rolloutCluster+"/") { + t.Errorf("artifact URI = %q, want it under %s", artifact.Status.Artifact.URI, rolloutCluster) + } +} + +func TestReconcileCNPG_ArtifactOfARecordWithoutServerName(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://bucket/") + f.pass() + f.observeCNPG(newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, "live-server"))) + f.completeOwnBackup(func(b *cnpgtypes.Backup) { + b.Annotations[cnpgWrittenDestinationAnnotation] = `{"destinationPath":"s3://bucket/"}` + }) + j, _ := f.pass() + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseSucceeded { + t.Fatalf("phase = %q (%s), want Succeeded", j.Status.Phase, j.Status.Message) + } + if got := f.artifact().Spec.DriverMetadata[cnpgServerNameKey]; got != "live-server" { + t.Errorf("artifact serverName = %q, want the live Cluster's live-server", got) + } +} + // A strategy without credentials records none, and the artifact names none. func TestReconcileCNPG_ArtifactOfARecordWithoutCredentials(t *testing.T) { f := newAttachedObjectStoreFixture(t, time.Now(), "s3://bucket/") - if _, backups := f.pass(); len(backups) != 1 || backups[0].Annotations[cnpgWrittenDestinationAnnotation] != `{"destinationPath":"s3://bucket/"}` { + if _, backups := f.pass(); len(backups) != 1 || backups[0].Annotations[cnpgWrittenDestinationAnnotation] != `{"destinationPath":"s3://bucket/","serverName":"`+rolloutCluster+`"}` { t.Fatalf("cnpg.io/Backups after the first pass = %+v, want one recording only s3://bucket/", backups) } f.setStrategyDestination("s3://edited/", "https://s3.two", "creds-two") From c69787ff49badeed6907cb69fec244ec84bbdb67 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Tue, 10 Nov 2026 00:03:02 +0300 Subject: [PATCH 383/518] perf(postgres): list only the Cluster's own backups for the queue Every pass of a backup job waiting for another one reads the cnpg.io/Backups from the apiserver, and the read covered every Backup in the namespace, so a namespace with a long backup history paid for it on each poll of each waiting job. The queue only looks at Backups this controller created for the Cluster, which have carried the cluster and owning-job labels since the driver was introduced, so it now asks the apiserver for those alone. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 9 ++- .../cnpgstrategy_controller_test.go | 61 ++++++++++++++++--- 2 files changed, 60 insertions(+), 10 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 75c3b3f06..a4bb0a3b9 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -1002,10 +1002,15 @@ func (r *BackupJobReconciler) findCNPGBackupForJob(ctx context.Context, j *backu // hold the queue. // // The Backups are read from the apiserver, not the cache, which can still -// miss a Backup another job's earlier pass created. +// miss a Backup another job's earlier pass created. Only the ones this +// controller created for the Cluster are asked for, by the labels +// ensureCNPGBackup has always set, so a waiting job's poll does not fetch +// the namespace's whole Backup history. func (r *BackupJobReconciler) otherCNPGBackupInFlight(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string) (string, error) { list := &cnpgtypes.BackupList{} - if err := r.apiReader().List(ctx, list, client.InNamespace(j.Namespace)); err != nil { + if err := r.apiReader().List(ctx, list, client.InNamespace(j.Namespace), + client.MatchingLabels{cnpgClusterLabel: clusterName}, + client.HasLabels{backupsv1alpha1.OwningJobNameLabel}); err != nil { return "", err } var holders []string diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 8604d8c1a..25a72b746 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -15,6 +15,7 @@ import ( apimeta "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" @@ -2518,19 +2519,20 @@ func (f *cnpgRolloutFixture) liveDestination() string { } // seedCNPGBackup creates a cnpg.io/Backup of cluster owned by the BackupJob -// named owner; an empty owner leaves it unlabelled, as a Backup created by -// hand. +// named owner, labelled as ensureCNPGBackup labels it; an empty owner leaves +// it unlabelled, as a Backup created by hand. func (f *cnpgRolloutFixture) seedCNPGBackup(name, owner, cluster, phase string) { f.t.Helper() - var labels map[string]string + var backupLabels map[string]string if owner != "" { - labels = map[string]string{ + backupLabels = map[string]string{ backupsv1alpha1.OwningJobNameLabel: owner, backupsv1alpha1.OwningJobNamespaceLabel: rolloutNS, + cnpgClusterLabel: cluster, } } b := &cnpgtypes.Backup{ - ObjectMeta: metav1.ObjectMeta{Namespace: rolloutNS, Name: name, Labels: labels}, + ObjectMeta: metav1.ObjectMeta{Namespace: rolloutNS, Name: name, Labels: backupLabels}, Spec: cnpgtypes.BackupSpec{Cluster: cnpgtypes.ClusterReference{Name: cluster}}, Status: cnpgtypes.BackupStatus{Phase: phase}, } @@ -2595,6 +2597,43 @@ func TestReconcileCNPG_RunningBackupDoesNotReapplyObjectStore(t *testing.T) { } } +// The queue reads the apiserver on every pass of every waiting job, so it +// asks only for the Backups this controller created for the Cluster rather +// than the namespace's whole Backup history. +func TestReconcileCNPG_QueueListsOnlyTheClustersOwnedBackups(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://bucket/") + var selectors []labels.Selector + f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*cnpgtypes.BackupList); ok { + lo := &client.ListOptions{} + lo.ApplyOptions(opts) + selectors = append(selectors, lo.LabelSelector) + } + return cl.List(ctx, list, opts...) + }, + }) + f.pass() + owned := labels.Set{cnpgClusterLabel: rolloutCluster, backupsv1alpha1.OwningJobNameLabel: "bj-other"} + foreignCluster := labels.Set{cnpgClusterLabel: "postgres-other", backupsv1alpha1.OwningJobNameLabel: "bj-other"} + unowned := labels.Set{cnpgClusterLabel: rolloutCluster} + var queue labels.Selector + for _, s := range selectors { + if s == nil { + s = labels.Everything() + } + if s.Matches(owned) { + queue = s + } + } + if queue == nil { + t.Fatalf("no cnpg.io/Backup list would find another job's Backup; selectors = %v", selectors) + } + if queue.Matches(foreignCluster) || queue.Matches(unowned) { + t.Errorf("queue selector %q, want it to match only owned Backups of %s", queue, rolloutCluster) + } +} + func TestReconcileCNPG_WaitsForAnotherRunningBackupOfTheApp(t *testing.T) { cases := []struct { name string @@ -5974,16 +6013,22 @@ func TestReconcileCNPG_OwnBackupTheCacheHasNotCaught(t *testing.T) { } // failingBackupLists makes the uncached reader fail the cnpg.io/Backup lists -// for which fail, given whether the list selects the job's own Backup by -// label and how many such lists came before, returns true. +// for which fail, given whether the list selects only the job's own Backup +// (it would not find another job's) and how many such lists came before, +// returns true. func (f *cnpgRolloutFixture) failingBackupLists(fail func(own bool, n int) bool) { counts := map[bool]int{} + anotherJobs := labels.Set{ + backupsv1alpha1.OwningJobNameLabel: "bj-other", + backupsv1alpha1.OwningJobNamespaceLabel: rolloutNS, + cnpgClusterLabel: rolloutCluster, + } f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { if _, ok := list.(*cnpgtypes.BackupList); ok { o := &client.ListOptions{} o.ApplyOptions(opts) - own := o.LabelSelector != nil && !o.LabelSelector.Empty() + own := o.LabelSelector != nil && !o.LabelSelector.Matches(anotherJobs) counts[own]++ if fail(own, counts[own]) { return apierrors.NewServiceUnavailable("apiserver unavailable") From d37701a32747a3b2b6a82e7637efc0c09135ec1d Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Tue, 10 Nov 2026 00:07:01 +0300 Subject: [PATCH 384/518] fix(postgres): tolerate an attach record slightly ahead of the clock An attach record in the future failed the backup job at once, because it would hold every instance as older than the plugin. A record a few minutes ahead is also what a leadership change between controllers whose clocks differ leaves behind after an attach that worked, and failing then throws away a backup that would have run. Within that margin the job now keeps waiting: the instances predate the record, so the rollout check holds the backup, and the deadline counted from the record is not spent by the wait. Only a record further ahead is taken as corrupt and fails the job. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 44 +++++++++++++------ .../cnpgstrategy_controller_test.go | 35 +++++++++++++-- 2 files changed, 62 insertions(+), 17 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index a4bb0a3b9..a9eb6fabc 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -233,6 +233,11 @@ const ( // cannot be lost to a failed BackupJob status write after the attach. cnpgPluginAttachedAtAnnotation = "backups.cozystack.io/barman-plugin-attached-at" + // How far ahead of this controller's clock an attach record may be + // before it is taken as corrupt rather than as written by a leader whose + // clock ran ahead. + cnpgAttachClockSkewTolerance = 5 * time.Minute + cnpgPodRoleLabel = "cnpg.io/podRole" cnpgPodRoleInstance = "instance" ) @@ -499,20 +504,18 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } // The Cluster's record covers an attach whose status write was lost. attachedAt := rollout.LastTransitionTime - if attachment.attachedAt != "" { - recorded, err := time.Parse(time.RFC3339, attachment.attachedAt) - if recorded.After(time.Now()) { - // Every instance would predate it, and the deadline would count from it. - err = fmt.Errorf("%s is in the future", attachment.attachedAt) - } - if err != nil { - return r.markBackupJobFailed(ctx, j, fmt.Sprintf( - "cannot tell when the barman-cloud plugin was attached: the %s annotation of cnpg.io/Cluster %s/%s: %v", - cnpgPluginAttachedAtAnnotation, j.Namespace, clusterName, err)) - } - if recorded.After(attachedAt.Time) { - attachedAt = metav1.NewTime(recorded) - } + recorded, ok, err := cnpgRecordedAttach(attachment.attachedAt) + if err == nil && ok && time.Until(recorded) > cnpgAttachClockSkewTolerance { + // Every instance would predate it, and the deadline would count from it. + err = fmt.Errorf("%s is in the future", attachment.attachedAt) + } + if err != nil { + return r.markBackupJobFailed(ctx, j, fmt.Sprintf( + "cannot tell when the barman-cloud plugin was attached: the %s annotation of cnpg.io/Cluster %s/%s: %v", + cnpgPluginAttachedAtAnnotation, j.Namespace, clusterName, err)) + } + if ok && recorded.After(attachedAt.Time) { + attachedAt = metav1.NewTime(recorded) } pending, err := r.cnpgPluginRolloutPending(ctx, j.Namespace, clusterName, attachment, attachedAt) if err != nil { @@ -897,6 +900,19 @@ func (r *BackupJobReconciler) cnpgPluginRolloutPending(ctx context.Context, name return "", nil } +// cnpgRecordedAttach parses a cnpgPluginAttachedAtAnnotation value; ok is +// false for "", a Cluster that carries no record. +func cnpgRecordedAttach(raw string) (recorded time.Time, ok bool, err error) { + if raw == "" { + return time.Time{}, false, nil + } + recorded, err = time.Parse(time.RFC3339, raw) + if err != nil { + return time.Time{}, false, err + } + return recorded, true, nil +} + func podReady(pod *corev1.Pod) bool { for _, c := range pod.Status.Conditions { if c.Type == corev1.PodReady { diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 25a72b746..85afa0c65 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -3122,9 +3122,10 @@ func TestReconcileCNPG_LaterWriteKeepsTheAttachRecord(t *testing.T) { } // An attach record that does not parse cannot tell which instances rolled -// with the plugin, so the job fails naming it instead of guessing. One in the -// future would hold every instance as older than the plugin and push the -// rollout deadline out by as far as it lies ahead, so it fails the same way. +// with the plugin, so the job fails naming it instead of guessing. One further +// ahead than controller clocks drift apart would hold every instance as older +// than the plugin and push the rollout deadline out by as far as it lies +// ahead, so it fails the same way. func TestReconcileCNPG_UnreadableAttachRecordFailsTheJob(t *testing.T) { for _, tc := range []struct{ name, record string }{ {name: "does not parse", record: "yesterday"}, @@ -3151,6 +3152,34 @@ func TestReconcileCNPG_UnreadableAttachRecordFailsTheJob(t *testing.T) { } } +// A record a little ahead of this controller's clock is what a leadership +// change between controllers whose clocks differ leaves behind, so the job +// waits for the clocks to agree rather than failing an attach that worked. +func TestReconcileCNPG_AttachRecordSlightlyAheadWaits(t *testing.T) { + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase), newRolloutInstancePod(time.Now().Add(-time.Hour))) + if j, _ := f.pass(); apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) == nil { + t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) + } + live := newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)) + live.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: time.Now().Add(time.Minute).UTC().Format(time.RFC3339)} + f.observeCNPG(live) + + j, backups, res := f.passResult() + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("the job failed on a record a minute ahead: %s", j.Status.Message) + } + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created before the rollout could be judged: %+v", backups) + } + if res.RequeueAfter == 0 { + t.Errorf("a waiting job is not requeued: %+v", res) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForPluginRollout" { + t.Errorf("Ready = %+v, want WaitingForPluginRollout", ready) + } +} + // A plugin attached by someone else after the latch, another backup job or // the chart re-rendering a restored Cluster outside the platform flow, // started the rollout before this job's write, which only finds it there: From a27de054cabac5bdc15f7113bc772bf5963d3726 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Tue, 10 Nov 2026 00:07:37 +0300 Subject: [PATCH 385/518] fix(postgres): wait for the rollout another backup job's attach started Only the job that attaches the barman-cloud plugin waited for the instance rollout it causes. A second job of the same application that started meanwhile found the plugin already there, had no rollout of its own to wait for and created its cnpg.io/Backup at once, against instances CNPG was still recreating, and such a Backup fails for good. The queue does not hold it either, since the attaching job has no Backup yet. The attach record on the Cluster says when the plugin was attached, so a job without a rollout of its own now waits for instances newer than it while the attach is recent enough for that rollout to be running. An older attach has had its rollout judged by the job that made it. The record belongs to that job, so one this job cannot read, or one further ahead than controller clocks drift, is skipped rather than failing it; the latter would otherwise hold every later job of the application for as long as it lies ahead. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 23 ++++++++++ .../cnpgstrategy_controller_test.go | 44 +++++++++++++++++++ 2 files changed, 67 insertions(+) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index a9eb6fabc..bdc04bb0e 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -551,6 +551,29 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp }) apimeta.RemoveStatusCondition(&j.Status.Conditions, "Ready") rolledOut = true + } else if existingBackup == nil && attachment.attachedAt != "" { + // Another backup job attached the plugin. Its rollout can still be + // running while the attach is recent, and later than that the + // attaching job has failed it already. The record is that job's: + // one this job cannot read does not fail it, and neither does one + // further ahead than clocks drift, which every instance would + // predate for as long as it lies ahead. + recorded, _, err := cnpgRecordedAttach(attachment.attachedAt) + if err == nil && time.Until(recorded) > cnpgAttachClockSkewTolerance { + err = fmt.Errorf("%s is in the future", attachment.attachedAt) + } + if err != nil { + logger.Info("ignoring an unusable attach record another backup job left", + "cluster", clusterName, "annotation", cnpgPluginAttachedAtAnnotation, "error", err) + } else if time.Since(recorded) < cnpgDefaultBackupDeadline { + pending, err := r.cnpgPluginRolloutPending(ctx, j.Namespace, clusterName, attachment, metav1.NewTime(recorded)) + if err != nil { + return ctrl.Result{}, err + } + if pending != "" { + return r.waitCNPGBackupJob(ctx, j, "WaitingForPluginRollout", pending) + } + } } // The chart refuses a flavor change on a live Cluster, but the app keeps diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 85afa0c65..9c41d3931 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -3180,6 +3180,50 @@ func TestReconcileCNPG_AttachRecordSlightlyAheadWaits(t *testing.T) { } } +// A job that attached nothing itself still finds the plugin another backup +// job attached a moment ago, while CNPG rolls the instances for it; a Backup +// created then fails for good. It waits for that rollout as long as the +// attach is recent enough for it to be running, and an attach record it +// cannot read is not its own to fail on. +func TestReconcileCNPG_JobWithoutLatchWaitsForAnotherJobsRollout(t *testing.T) { + cases := []struct { + name string + record string + podCreated time.Time + wantBackup bool + }{ + {name: "recent attach, instance not rolled yet", record: time.Now().Add(-5 * time.Minute).UTC().Format(time.RFC3339), podCreated: time.Now().Add(-10 * time.Minute), wantBackup: false}, + {name: "recent attach, instance rolled", record: time.Now().Add(-5 * time.Minute).UTC().Format(time.RFC3339), podCreated: time.Now().Add(-time.Minute), wantBackup: true}, + {name: "attach older than the rollout deadline", record: time.Now().Add(-2 * cnpgDefaultBackupDeadline).UTC().Format(time.RFC3339), podCreated: time.Now().Add(-3 * cnpgDefaultBackupDeadline), wantBackup: true}, + {name: "unreadable record", record: "yesterday", podCreated: time.Now().Add(-time.Hour), wantBackup: true}, + {name: "record further ahead than clocks drift", record: time.Now().Add(time.Hour).UTC().Format(time.RFC3339), podCreated: time.Now().Add(-time.Hour), wantBackup: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + cluster := newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)) + cluster.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: tc.record} + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), cluster, + newRolloutInstancePod(tc.podCreated.Truncate(time.Second))) + + j, backups, res := f.passResult() + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("the job failed: %s", j.Status.Message) + } + if got := len(backups) == 1; got != tc.wantBackup { + t.Fatalf("cnpg.io/Backup created = %t, want %t; Ready = %+v", got, tc.wantBackup, apimeta.FindStatusCondition(j.Status.Conditions, "Ready")) + } + if !tc.wantBackup { + if res.RequeueAfter == 0 { + t.Errorf("a waiting job is not requeued: %+v", res) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForPluginRollout" { + t.Errorf("Ready = %+v, want WaitingForPluginRollout", ready) + } + } + }) + } +} + // A plugin attached by someone else after the latch, another backup job or // the chart re-rendering a restored Cluster outside the platform flow, // started the rollout before this job's write, which only finds it there: From 5635d9d815cb7adfe13798649dfd07f0d06bd1c8 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Tue, 10 Nov 2026 00:08:24 +0300 Subject: [PATCH 386/518] test(postgres): pin that a rollout dated by the attach record runs out The deadline tests dated every rollout by the job's own condition, so a change that let a rollout dated by the Cluster's attach record wait forever would have passed them. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller_test.go | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 9c41d3931..78c9e9966 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2887,6 +2887,36 @@ func TestReconcileCNPG_PluginRolloutDeadlineCountsFromAttach(t *testing.T) { } } +// A rollout dated by the Cluster's attach record runs out like one dated by +// the job's own condition: a record newer than the condition but older than +// the deadline fails the job too. +func TestReconcileCNPG_PluginRolloutDeadlineCountsFromTheAttachRecord(t *testing.T) { + expired := time.Now().Add(-(cnpgDefaultBackupDeadline + time.Minute)).Truncate(time.Second) + f := newCNPGRolloutFixture(t, newRolloutBackupJob(expired.Add(-time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase), newRolloutInstancePod(expired.Add(-time.Hour))) + + j, _ := f.pass() // latch + latch := apimeta.FindStatusCondition(j.Status.Conditions, backupCondPluginRolledOut) + if latch == nil { + t.Fatalf("no %s latch: %+v", backupCondPluginRolledOut, j.Status.Conditions) + } + latch.LastTransitionTime = metav1.NewTime(expired.Add(-10 * time.Minute)) + if err := f.c.Status().Update(context.Background(), j); err != nil { + t.Fatalf("age the latch: %v", err) + } + live := newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)) + live.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: expired.UTC().Format(time.RFC3339)} + f.observeCNPG(live) + + j, backups := f.pass() + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created against an instance older than the attach: %+v", backups) + } + if j.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed || !strings.Contains(j.Status.Message, "rollout did not finish") { + t.Fatalf("phase = %q (%s), want Failed on a rollout recorded longer ago than the deadline", j.Status.Phase, j.Status.Message) + } +} + // The rollout is measured from the write that attached the plugin, not from // the latch recorded before it: retries of a failing write can lie between // the two, and an instance recreated in that time runs without the plugin. From c8cdefc665f6d80c50fea832813ba8ea53c15783 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Wed, 21 Oct 2026 19:11:09 +0400 Subject: [PATCH 387/518] feat(backups): record the RDB snapshot location and size for Redis backups Backups made by the Redis strategy, which also serves Valkey, carried no Backup.status.artifact at all: neither a URI nor a size. The snapshot is one object per backup, //.rdb, whose key the driver already records, and the default strategy uploads it with the S3 coordinates of the backup credentials Secret the controller projects into the application namespace. On completion the driver now stats that object with the same coordinates and records its s3:// URI and size. The artifact is set only when the object is found, so a custom strategy that writes elsewhere gets no URI pointing at a missing object. A Backup that already exists is returned without probing, as for the MariaDB and CNPG drivers. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 9 + .../backupcontroller/artifact_size_test.go | 15 ++ .../backupcontroller/redis_artifact_size.go | 80 +++++++++ .../redis_artifact_size_test.go | 166 ++++++++++++++++++ .../redisstrategy_controller.go | 13 +- 5 files changed, 280 insertions(+), 3 deletions(-) create mode 100644 internal/backupcontroller/redis_artifact_size.go create mode 100644 internal/backupcontroller/redis_artifact_size_test.go diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 941b00323..dc23542b9 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -36,6 +36,7 @@ type s3ObjectInfo struct { type s3ObjectLister interface { ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) + StatObject(ctx context.Context, bucket, key string) (int64, error) } // s3Connection is what a driver resolves from its rendered strategy to @@ -148,6 +149,14 @@ func (l *minioObjectLister) ListObjects(ctx context.Context, bucket, prefix stri return collectObjects(ctx, l.client.ListObjects(ctx, bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: recursive})) } +func (l *minioObjectLister) StatObject(ctx context.Context, bucket, key string) (int64, error) { + info, err := l.client.StatObject(ctx, bucket, key, minio.StatObjectOptions{}) + if err != nil { + return 0, err + } + return info.Size, nil +} + // collectObjects drains a minio listing. minio-go ends a listing whose // context is cancelled between pages by closing the channel without an // error item, so a cancelled context is checked once the channel closes: diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go index 3d9c35b23..55073e742 100644 --- a/internal/backupcontroller/artifact_size_test.go +++ b/internal/backupcontroller/artifact_size_test.go @@ -37,6 +37,16 @@ type fakeS3Lister struct { bucket, prefix string recursive bool calls int + + statSize int64 + statErr error + statKey string +} + +func (f *fakeS3Lister) StatObject(_ context.Context, bucket, key string) (int64, error) { + f.calls++ + f.bucket, f.statKey = bucket, key + return f.statSize, f.statErr } func (f *fakeS3Lister) ListObjects(_ context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) { @@ -167,6 +177,11 @@ func (blockingLister) ListObjects(ctx context.Context, _, _ string, _ bool) ([]s return nil, ctx.Err() } +func (blockingLister) StatObject(ctx context.Context, _, _ string) (int64, error) { + <-ctx.Done() + return 0, ctx.Err() +} + // withShortProbeTimeout shortens the probe timeout and fails the test if fn // is not back well before the default timeout would have expired. func withShortProbeTimeout(t *testing.T, fn func()) { diff --git a/internal/backupcontroller/redis_artifact_size.go b/internal/backupcontroller/redis_artifact_size.go new file mode 100644 index 000000000..0e76d0b9c --- /dev/null +++ b/internal/backupcontroller/redis_artifact_size.go @@ -0,0 +1,80 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "errors" + "fmt" + "net/url" + "strings" + + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/types" + + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" +) + +// redisArtifact locates the RDB snapshot the backup Job uploaded and returns +// an Artifact carrying its s3:// URI and size. The default Redis strategy +// takes its S3 coordinates from the projected backup credentials Secret, so +// the object is looked up there. Returns nil - leaving Status.Artifact +// absent - when the Secret cannot be read or the object is not found: a +// custom strategy that writes elsewhere must not get a URI pointing at an +// object that does not exist. +func (r *BackupJobReconciler) redisArtifact(ctx context.Context, namespace, objectKey string) *backupsv1alpha1.BackupArtifact { + logger := getLogger(ctx) + ctx, cancel := context.WithTimeout(ctx, artifactSizeProbeTimeout) + defer cancel() + + bucket, conn, err := r.projectedS3Connection(ctx, namespace) + if err != nil { + logger.Info("cannot resolve backup credentials to locate the Redis snapshot", "objectKey", objectKey, "error", err) + return nil + } + lister, err := r.s3Lister(conn) + if err != nil { + logger.Info("cannot build S3 client to locate the Redis snapshot", "objectKey", objectKey, "error", err) + return nil + } + size, err := lister.StatObject(ctx, bucket, objectKey) + if err != nil { + logger.Info("cannot stat the Redis snapshot; leaving the artifact unset", "bucket", bucket, "objectKey", objectKey, "error", err) + return nil + } + return &backupsv1alpha1.BackupArtifact{ + URI: fmt.Sprintf("s3://%s/%s", bucket, objectKey), + SizeBytes: size, + } +} + +// projectedS3Connection reads the backup credentials Secret the controller +// projects into the application namespace. Its endpoint is a bare host, +// which the strategy scripts reach over https. +func (r *BackupJobReconciler) projectedS3Connection(ctx context.Context, namespace string) (string, s3Connection, error) { + var conn s3Connection + name := r.CredentialsConfig.TargetSecretName + if name == "" { + return "", conn, errors.New("no projected credentials Secret is configured") + } + secret := &corev1.Secret{} + if err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, secret); err != nil { + return "", conn, err + } + value := func(key string) string { return string(secret.Data[key]) } + + bucket := value("bucketName") + conn.AccessKey, conn.SecretKey = value("AWS_ACCESS_KEY_ID"), value("AWS_SECRET_ACCESS_KEY") + if bucket == "" || value("endpoint") == "" || conn.AccessKey == "" || conn.SecretKey == "" { + return "", conn, fmt.Errorf("secret %s/%s lacks bucketName, endpoint or access keys", namespace, name) + } + conn.Endpoint, conn.Secure = value("endpoint"), true + if strings.Contains(conn.Endpoint, "://") { + u, err := url.Parse(conn.Endpoint) + if err != nil || u.Host == "" { + return "", conn, fmt.Errorf("invalid endpoint %q in secret %s/%s", conn.Endpoint, namespace, name) + } + conn.Endpoint, conn.Secure = u.Host, u.Scheme == "https" + } + conn.Region = value("region") + return bucket, conn, nil +} diff --git a/internal/backupcontroller/redis_artifact_size_test.go b/internal/backupcontroller/redis_artifact_size_test.go new file mode 100644 index 000000000..458726b67 --- /dev/null +++ b/internal/backupcontroller/redis_artifact_size_test.go @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "sigs.k8s.io/controller-runtime/pkg/client" + + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" +) + +const redisTestKey = "tenant/cache/bj-1.rdb" + +func projectedCredsSecret(data map[string]string) *corev1.Secret { + s := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "cozy-backups-creds"}, + Data: map[string][]byte{}, + } + for k, v := range map[string]string{ + "endpoint": "s3.example.org", + "bucketName": "cozy-backups", + "region": "us-east-1", + "AWS_ACCESS_KEY_ID": "AKID", + "AWS_SECRET_ACCESS_KEY": "secret", + } { + s.Data[k] = []byte(v) + } + for k, v := range data { + if v == "" { + delete(s.Data, k) + continue + } + s.Data[k] = []byte(v) + } + return s +} + +func redisReconciler(t *testing.T, l *fakeS3Lister, objs ...client.Object) *BackupJobReconciler { + r := reconcilerWithLister(t, l, objs...) + r.CredentialsConfig.TargetSecretName = "cozy-backups-creds" + return r +} + +func TestRedisArtifact(t *testing.T) { + t.Run("stats the snapshot with the projected credentials", func(t *testing.T) { + l := &fakeS3Lister{statSize: 2048} + r := redisReconciler(t, l, projectedCredsSecret(nil)) + got := r.redisArtifact(context.Background(), "tenant", redisTestKey) + if got == nil || got.URI != "s3://cozy-backups/"+redisTestKey || got.SizeBytes != 2048 { + t.Fatalf("artifact = %#v", got) + } + if l.bucket != "cozy-backups" || l.statKey != redisTestKey { + t.Errorf("stat %q/%q, want cozy-backups/%q", l.bucket, l.statKey, redisTestKey) + } + if l.conn.Endpoint != "s3.example.org" || !l.conn.Secure || l.conn.AccessKey != "AKID" || l.conn.Region != "us-east-1" { + t.Errorf("connection = %+v", l.conn) + } + }) + + t.Run("an endpoint with a scheme sets the transport", func(t *testing.T) { + l := &fakeS3Lister{statSize: 1} + r := redisReconciler(t, l, projectedCredsSecret(map[string]string{"endpoint": "http://seaweedfs:8333"})) + if got := r.redisArtifact(context.Background(), "tenant", redisTestKey); got == nil { + t.Fatal("artifact = nil") + } + if l.conn.Endpoint != "seaweedfs:8333" || l.conn.Secure { + t.Errorf("connection = %+v", l.conn) + } + }) + + unset := map[string]struct { + lister *fakeS3Lister + objs []client.Object + noTarget bool + }{ + "missing object": {lister: &fakeS3Lister{statErr: errors.New("NoSuchKey")}, objs: []client.Object{projectedCredsSecret(nil)}}, + "missing Secret": {lister: &fakeS3Lister{statSize: 1}}, + "missing bucket": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(map[string]string{"bucketName": ""})}}, + "missing keys": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(map[string]string{"AWS_SECRET_ACCESS_KEY": ""})}}, + "no target name": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(nil)}, noTarget: true}, + "broken endpoint": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(map[string]string{"endpoint": "https://"})}}, + } + for name, tc := range unset { + t.Run(name+" leaves the artifact unset", func(t *testing.T) { + r := redisReconciler(t, tc.lister, tc.objs...) + if tc.noTarget { + r.CredentialsConfig.TargetSecretName = "" + } + if got := r.redisArtifact(context.Background(), "tenant", redisTestKey); got != nil { + t.Errorf("artifact = %#v, want nil", got) + } + }) + } +} + +// TestRedisArtifact_MinioClient drives the real minio client against a stub +// that answers the HEAD request, so addressing, signing and the +// Content-Length parsing are exercised together. +func TestRedisArtifact_MinioClient(t *testing.T) { + var gotMethod, gotPath, gotAuth string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { + gotMethod, gotPath, gotAuth = req.Method, req.URL.Path, req.Header.Get("Authorization") + w.Header().Set("Content-Length", "4096") + w.Header().Set("ETag", `"abc"`) + w.Header().Set("Last-Modified", "Wed, 21 Oct 2026 10:00:00 GMT") + w.WriteHeader(http.StatusOK) + })) + defer srv.Close() + + creds := projectedCredsSecret(map[string]string{"endpoint": srv.URL}) + c := newMariaDBStrategyTestClient(t, creds) + r := &BackupJobReconciler{Client: c, Scheme: c.Scheme()} + r.CredentialsConfig.TargetSecretName = "cozy-backups-creds" + + got := r.redisArtifact(context.Background(), "tenant", redisTestKey) + if got == nil || got.SizeBytes != 4096 { + t.Fatalf("artifact = %#v, want SizeBytes 4096", got) + } + if gotMethod != http.MethodHead || gotPath != "/cozy-backups/"+redisTestKey { + t.Errorf("request %s %s, want HEAD /cozy-backups/%s", gotMethod, gotPath, redisTestKey) + } + if !strings.Contains(gotAuth, "Credential=AKID/") { + t.Errorf("request not signed with the Secret's access key: %q", gotAuth) + } +} + +func TestCreateRedisBackupArtifact_Artifact(t *testing.T) { + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-1"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: "Redis", Name: "cache"}, + }, + } + + t.Run("records the snapshot location and size", func(t *testing.T) { + l := &fakeS3Lister{statSize: 777} + r := redisReconciler(t, l, j, projectedCredsSecret(nil)) + got, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey) + if err != nil { + t.Fatalf("createRedisBackupArtifact: %v", err) + } + if got.Status.Artifact == nil || got.Status.Artifact.SizeBytes != 777 || got.Status.Artifact.URI != "s3://cozy-backups/"+redisTestKey { + t.Fatalf("Artifact = %#v", got.Status.Artifact) + } + }) + + t.Run("an existing Backup is returned without probing", func(t *testing.T) { + existing := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-1"}} + l := &fakeS3Lister{statSize: 777} + r := redisReconciler(t, l, j, existing, projectedCredsSecret(nil)) + if _, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey); err != nil { + t.Fatalf("createRedisBackupArtifact: %v", err) + } + if l.calls != 0 { + t.Errorf("S3 probed %d times for a Backup that already exists", l.calls) + } + }) +} diff --git a/internal/backupcontroller/redisstrategy_controller.go b/internal/backupcontroller/redisstrategy_controller.go index 7d287d385..40f3a99aa 100644 --- a/internal/backupcontroller/redisstrategy_controller.go +++ b/internal/backupcontroller/redisstrategy_controller.go @@ -368,7 +368,7 @@ func (r *BackupJobReconciler) reconcileRedis(ctx context.Context, j *backupsv1al } artifact, err := r.createRedisBackupArtifact(ctx, j, resolved, objectKey) if err != nil { - return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to create Backup artifact: %v", err)) + return r.failBackupArtifact(ctx, j, err) } now := metav1.Now() j.Status.BackupRef = &corev1.LocalObjectReference{Name: artifact.Name} @@ -472,6 +472,10 @@ func (r *BackupJobReconciler) createRedisBackupArtifact( resolved *ResolvedBackupConfig, objectKey string, ) (*backupsv1alpha1.Backup, error) { + existing, probe := r.existingBackup(ctx, j.Namespace, j.Name) + if existing != nil { + return existing, nil + } driverMD := map[string]string{redisObjectMetaKey: objectKey} for k, v := range resolved.Parameters { driverMD[redisParamPrefix+k] = v @@ -492,6 +496,9 @@ func (r *BackupJobReconciler) createRedisBackupArtifact( Phase: backupsv1alpha1.BackupPhaseReady, }, } + if probe { + backup.Status.Artifact = r.redisArtifact(ctx, j.Namespace, objectKey) + } if j.Spec.PlanRef != nil { backup.Spec.PlanRef = j.Spec.PlanRef } @@ -500,8 +507,8 @@ func (r *BackupJobReconciler) createRedisBackupArtifact( return nil, err } existing := &backupsv1alpha1.Backup{} - if getErr := r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: backup.Name}, existing); getErr != nil { - return nil, getErr + if getErr := r.apiReader().Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: backup.Name}, existing); getErr != nil { + return nil, fmt.Errorf("%w: %v", errBackupNotReadBack, getErr) } return existing, nil } From 2ff547348d5578f420a3833e904d16352d23dd24 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Wed, 21 Oct 2026 19:59:46 +0400 Subject: [PATCH 388/518] fix(backups): probe the Redis snapshot with the trust its Job used The default Redis strategy script verifies the S3 endpoint against ca.crt, then tls.crt, from the Secret mounted as the s3-ca volume (backupStorage.endpointCASecretName), and only without either honours S3_INSECURE_SKIP_VERIFY=true. The probe always verified against the system roots, so on a cluster with a private CA, such as a self-signed SeaweedFS, every Redis and Valkey backup succeeded with no artifact. Read the same settings from the pod spec of the completed backup Job, in the order the script applies them, and pass them to the S3 connection. The shared transport gains an InsecureSkipVerify switch that is set only from that explicit opt-in. The projector always stores the endpoint as a bare host, so the branch that parsed a scheme out of it was unreachable; drop it and its test. Cover the endpoint and access-key terms of the Secret completeness check. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 13 +- .../backupcontroller/redis_artifact_size.go | 86 +++++-- .../redis_artifact_size_test.go | 209 +++++++++++++++--- .../redisstrategy_controller.go | 5 +- 4 files changed, 263 insertions(+), 50 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index dc23542b9..230ca3ce8 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -54,6 +54,9 @@ type s3Connection struct { // what the writer of the backup trusted, no more. CAPEM []byte CAOnly bool + // InsecureSkipVerify mirrors an explicit admin opt-in a strategy + // already applies to its own uploads; it is never a fallback. + InsecureSkipVerify bool } // objectStore reads back what a backup wrote: the Secrets it was written @@ -105,14 +108,18 @@ func newMinioObjectLister(conn s3Connection) (s3ObjectLister, error) { Region: conn.Region, Secure: conn.Secure, } - if conn.Secure && len(conn.CAPEM) > 0 { + if conn.Secure && (len(conn.CAPEM) > 0 || conn.InsecureSkipVerify) { // Start from minio's transport to keep its proxy and timeout settings. tr, err := minio.DefaultTransport(true) if err != nil { return nil, err } - if tr.TLSClientConfig.RootCAs, err = endpointTrustPool(conn, tr.TLSClientConfig.RootCAs); err != nil { - return nil, err + if len(conn.CAPEM) > 0 { + if tr.TLSClientConfig.RootCAs, err = endpointTrustPool(conn, tr.TLSClientConfig.RootCAs); err != nil { + return nil, err + } + } else { + tr.TLSClientConfig.InsecureSkipVerify = true } opts.Transport = tr } diff --git a/internal/backupcontroller/redis_artifact_size.go b/internal/backupcontroller/redis_artifact_size.go index 0e76d0b9c..1c66f49af 100644 --- a/internal/backupcontroller/redis_artifact_size.go +++ b/internal/backupcontroller/redis_artifact_size.go @@ -5,10 +5,9 @@ import ( "context" "errors" "fmt" - "net/url" - "strings" corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/types" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" @@ -17,11 +16,12 @@ import ( // redisArtifact locates the RDB snapshot the backup Job uploaded and returns // an Artifact carrying its s3:// URI and size. The default Redis strategy // takes its S3 coordinates from the projected backup credentials Secret, so -// the object is looked up there. Returns nil - leaving Status.Artifact -// absent - when the Secret cannot be read or the object is not found: a -// custom strategy that writes elsewhere must not get a URI pointing at an -// object that does not exist. -func (r *BackupJobReconciler) redisArtifact(ctx context.Context, namespace, objectKey string) *backupsv1alpha1.BackupArtifact { +// the object is looked up there, with the TLS trust the Job's pod applied +// (see redisS3Trust). Returns nil - leaving Status.Artifact absent - when +// the Secret cannot be read or the object is not found: a custom strategy +// that writes elsewhere must not get a URI pointing at an object that does +// not exist. +func (r *BackupJobReconciler) redisArtifact(ctx context.Context, namespace, objectKey string, podSpec *corev1.PodSpec) *backupsv1alpha1.BackupArtifact { logger := getLogger(ctx) ctx, cancel := context.WithTimeout(ctx, artifactSizeProbeTimeout) defer cancel() @@ -31,6 +31,10 @@ func (r *BackupJobReconciler) redisArtifact(ctx context.Context, namespace, obje logger.Info("cannot resolve backup credentials to locate the Redis snapshot", "objectKey", objectKey, "error", err) return nil } + if conn.CAPEM, conn.InsecureSkipVerify, err = r.redisS3Trust(ctx, namespace, podSpec); err != nil { + logger.Info("cannot read the S3 endpoint CA to locate the Redis snapshot", "objectKey", objectKey, "error", err) + return nil + } lister, err := r.s3Lister(conn) if err != nil { logger.Info("cannot build S3 client to locate the Redis snapshot", "objectKey", objectKey, "error", err) @@ -48,8 +52,8 @@ func (r *BackupJobReconciler) redisArtifact(ctx context.Context, namespace, obje } // projectedS3Connection reads the backup credentials Secret the controller -// projects into the application namespace. Its endpoint is a bare host, -// which the strategy scripts reach over https. +// projects into the application namespace. The projector always stores the +// endpoint as a bare host, which the strategy scripts reach over https. func (r *BackupJobReconciler) projectedS3Connection(ctx context.Context, namespace string) (string, s3Connection, error) { var conn s3Connection name := r.CredentialsConfig.TargetSecretName @@ -68,13 +72,63 @@ func (r *BackupJobReconciler) projectedS3Connection(ctx context.Context, namespa return "", conn, fmt.Errorf("secret %s/%s lacks bucketName, endpoint or access keys", namespace, name) } conn.Endpoint, conn.Secure = value("endpoint"), true - if strings.Contains(conn.Endpoint, "://") { - u, err := url.Parse(conn.Endpoint) - if err != nil || u.Host == "" { - return "", conn, fmt.Errorf("invalid endpoint %q in secret %s/%s", conn.Endpoint, namespace, name) - } - conn.Endpoint, conn.Secure = u.Host, u.Scheme == "https" - } conn.Region = value("region") return bucket, conn, nil } + +const ( + redisS3CAVolume = "s3-ca" + redisS3InsecureSkipEnv = "S3_INSECURE_SKIP_VERIFY" +) + +// redisS3Trust mirrors how the default strategy script verifies the S3 +// endpoint: it trusts ca.crt, then tls.crt, from the Secret mounted as the +// s3-ca volume, and only when neither exists does it honour +// S3_INSECURE_SKIP_VERIFY=true. Both are read from the pod spec the backup +// Job ran with. A missing CA Secret is not an error, because the script +// mounts it optionally. +func (r *BackupJobReconciler) redisS3Trust(ctx context.Context, namespace string, podSpec *corev1.PodSpec) ([]byte, bool, error) { + if podSpec == nil { + return nil, false, nil + } + for _, v := range podSpec.Volumes { + if v.Name != redisS3CAVolume || v.Secret == nil { + continue + } + secret := &corev1.Secret{} + err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: v.Secret.SecretName}, secret) + if apierrors.IsNotFound(err) { + break + } + if err != nil { + return nil, false, err + } + for _, file := range []string{"ca.crt", "tls.crt"} { + if pem := secret.Data[secretKeyForFile(v.Secret.Items, file)]; len(pem) > 0 { + return pem, false, nil + } + } + } + for _, c := range podSpec.Containers { + for _, e := range c.Env { + if e.Name == redisS3InsecureSkipEnv && e.Value == "true" { + return nil, true, nil + } + } + } + return nil, false, nil +} + +// secretKeyForFile returns the Secret key a secret volume projects to file: +// the key itself when the volume lists no items. +func secretKeyForFile(items []corev1.KeyToPath, file string) string { + if len(items) == 0 { + return file + } + for _, it := range items { + if it.Path == file { + return it.Key + } + } + return "" +} diff --git a/internal/backupcontroller/redis_artifact_size_test.go b/internal/backupcontroller/redis_artifact_size_test.go index 458726b67..a167d8f4a 100644 --- a/internal/backupcontroller/redis_artifact_size_test.go +++ b/internal/backupcontroller/redis_artifact_size_test.go @@ -3,15 +3,18 @@ package backupcontroller import ( "context" + "encoding/pem" "errors" "net/http" "net/http/httptest" "strings" "testing" + batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "sigs.k8s.io/controller-runtime/pkg/client" + clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" ) @@ -52,7 +55,7 @@ func TestRedisArtifact(t *testing.T) { t.Run("stats the snapshot with the projected credentials", func(t *testing.T) { l := &fakeS3Lister{statSize: 2048} r := redisReconciler(t, l, projectedCredsSecret(nil)) - got := r.redisArtifact(context.Background(), "tenant", redisTestKey) + got := r.redisArtifact(context.Background(), "tenant", redisTestKey, nil) if got == nil || got.URI != "s3://cozy-backups/"+redisTestKey || got.SizeBytes != 2048 { t.Fatalf("artifact = %#v", got) } @@ -64,28 +67,18 @@ func TestRedisArtifact(t *testing.T) { } }) - t.Run("an endpoint with a scheme sets the transport", func(t *testing.T) { - l := &fakeS3Lister{statSize: 1} - r := redisReconciler(t, l, projectedCredsSecret(map[string]string{"endpoint": "http://seaweedfs:8333"})) - if got := r.redisArtifact(context.Background(), "tenant", redisTestKey); got == nil { - t.Fatal("artifact = nil") - } - if l.conn.Endpoint != "seaweedfs:8333" || l.conn.Secure { - t.Errorf("connection = %+v", l.conn) - } - }) - unset := map[string]struct { lister *fakeS3Lister objs []client.Object noTarget bool }{ - "missing object": {lister: &fakeS3Lister{statErr: errors.New("NoSuchKey")}, objs: []client.Object{projectedCredsSecret(nil)}}, - "missing Secret": {lister: &fakeS3Lister{statSize: 1}}, - "missing bucket": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(map[string]string{"bucketName": ""})}}, - "missing keys": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(map[string]string{"AWS_SECRET_ACCESS_KEY": ""})}}, - "no target name": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(nil)}, noTarget: true}, - "broken endpoint": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(map[string]string{"endpoint": "https://"})}}, + "missing object": {lister: &fakeS3Lister{statErr: errors.New("NoSuchKey")}, objs: []client.Object{projectedCredsSecret(nil)}}, + "missing Secret": {lister: &fakeS3Lister{statSize: 1}}, + "missing bucket": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(map[string]string{"bucketName": ""})}}, + "missing keys": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(map[string]string{"AWS_SECRET_ACCESS_KEY": ""})}}, + "no target name": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(nil)}, noTarget: true}, + "missing endpoint": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(map[string]string{"endpoint": ""})}}, + "missing access key": {lister: &fakeS3Lister{statSize: 1}, objs: []client.Object{projectedCredsSecret(map[string]string{"AWS_ACCESS_KEY_ID": ""})}}, } for name, tc := range unset { t.Run(name+" leaves the artifact unset", func(t *testing.T) { @@ -93,19 +86,20 @@ func TestRedisArtifact(t *testing.T) { if tc.noTarget { r.CredentialsConfig.TargetSecretName = "" } - if got := r.redisArtifact(context.Background(), "tenant", redisTestKey); got != nil { + if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, nil); got != nil { t.Errorf("artifact = %#v, want nil", got) } }) } } -// TestRedisArtifact_MinioClient drives the real minio client against a stub -// that answers the HEAD request, so addressing, signing and the -// Content-Length parsing are exercised together. +// TestRedisArtifact_MinioClient runs the real minio client against a TLS +// stub with a self-signed certificate that answers the HEAD request. It +// covers, in one pass, the trust settings the backup Job used, request +// signing and Content-Length parsing. func TestRedisArtifact_MinioClient(t *testing.T) { var gotMethod, gotPath, gotAuth string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { + srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { gotMethod, gotPath, gotAuth = req.Method, req.URL.Path, req.Header.Get("Authorization") w.Header().Set("Content-Length", "4096") w.Header().Set("ETag", `"abc"`) @@ -113,15 +107,24 @@ func TestRedisArtifact_MinioClient(t *testing.T) { w.WriteHeader(http.StatusOK) })) defer srv.Close() + caPEM := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: srv.Certificate().Raw}) - creds := projectedCredsSecret(map[string]string{"endpoint": srv.URL}) - c := newMariaDBStrategyTestClient(t, creds) + creds := projectedCredsSecret(map[string]string{"endpoint": strings.TrimPrefix(srv.URL, "https://")}) + ca := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "s3-endpoint-ca"}, + Data: map[string][]byte{"ca.crt": caPEM}, + } + c := newMariaDBStrategyTestClient(t, creds, ca) r := &BackupJobReconciler{Client: c, Scheme: c.Scheme()} r.CredentialsConfig.TargetSecretName = "cozy-backups-creds" - got := r.redisArtifact(context.Background(), "tenant", redisTestKey) + if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, &corev1.PodSpec{}); got != nil { + t.Errorf("without the Job's trust settings: artifact = %#v, want nil (untrusted certificate)", got) + } + + got := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("s3-endpoint-ca", "")) if got == nil || got.SizeBytes != 4096 { - t.Fatalf("artifact = %#v, want SizeBytes 4096", got) + t.Fatalf("with the s3-ca volume: artifact = %#v, want SizeBytes 4096", got) } if gotMethod != http.MethodHead || gotPath != "/cozy-backups/"+redisTestKey { t.Errorf("request %s %s, want HEAD /cozy-backups/%s", gotMethod, gotPath, redisTestKey) @@ -129,6 +132,84 @@ func TestRedisArtifact_MinioClient(t *testing.T) { if !strings.Contains(gotAuth, "Credential=AKID/") { t.Errorf("request not signed with the Secret's access key: %q", gotAuth) } + + if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("", "true")); got == nil || got.SizeBytes != 4096 { + t.Errorf("with S3_INSECURE_SKIP_VERIFY=true: artifact = %#v, want SizeBytes 4096", got) + } +} + +// redisPodSpec builds the parts of the backup Job's pod spec that +// redisS3Trust reads: the s3-ca volume when caSecret is set, and the +// S3_INSECURE_SKIP_VERIFY variable. +func redisPodSpec(caSecret, skipVerify string) *corev1.PodSpec { + spec := &corev1.PodSpec{Containers: []corev1.Container{{ + Name: "backup", + Env: []corev1.EnvVar{{Name: "S3_INSECURE_SKIP_VERIFY", Value: skipVerify}}, + }}} + if caSecret != "" { + spec.Volumes = []corev1.Volume{{ + Name: "s3-ca", + VolumeSource: corev1.VolumeSource{Secret: &corev1.SecretVolumeSource{SecretName: caSecret}}, + }} + } + return spec +} + +func TestRedisS3Trust(t *testing.T) { + caSecret := func(data map[string]string) *corev1.Secret { + s := &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "s3-endpoint-ca"}, Data: map[string][]byte{}} + for k, v := range data { + s.Data[k] = []byte(v) + } + return s + } + cases := map[string]struct { + spec *corev1.PodSpec + objs []client.Object + wantCA string + wantSkip bool + }{ + "ca.crt wins over tls.crt and skip-verify": { + spec: redisPodSpec("s3-endpoint-ca", "true"), + objs: []client.Object{caSecret(map[string]string{"ca.crt": "CA", "tls.crt": "TLS"})}, + wantCA: "CA", + }, + "tls.crt when there is no ca.crt": { + spec: redisPodSpec("s3-endpoint-ca", ""), + objs: []client.Object{caSecret(map[string]string{"tls.crt": "TLS"})}, + wantCA: "TLS", + }, + "volume items map the file to another key": { + spec: func() *corev1.PodSpec { + s := redisPodSpec("s3-endpoint-ca", "") + s.Volumes[0].Secret.Items = []corev1.KeyToPath{{Key: "bundle.pem", Path: "ca.crt"}} + return s + }(), + objs: []client.Object{caSecret(map[string]string{"bundle.pem": "MAPPED", "ca.crt": "UNMAPPED"})}, + wantCA: "MAPPED", + }, + "missing CA Secret falls through to skip-verify": { + spec: redisPodSpec("s3-endpoint-ca", "true"), + wantSkip: true, + }, + "skip-verify only when it is exactly true": { + spec: redisPodSpec("", "false"), + }, + "no pod spec": {}, + } + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + c := newMariaDBStrategyTestClient(t, tc.objs...) + r := &BackupJobReconciler{Client: c} + ca, skip, err := r.redisS3Trust(context.Background(), "tenant", tc.spec) + if err != nil { + t.Fatalf("redisS3Trust: %v", err) + } + if string(ca) != tc.wantCA || skip != tc.wantSkip { + t.Errorf("trust = (%q, %v), want (%q, %v)", ca, skip, tc.wantCA, tc.wantSkip) + } + }) + } } func TestCreateRedisBackupArtifact_Artifact(t *testing.T) { @@ -143,7 +224,7 @@ func TestCreateRedisBackupArtifact_Artifact(t *testing.T) { t.Run("records the snapshot location and size", func(t *testing.T) { l := &fakeS3Lister{statSize: 777} r := redisReconciler(t, l, j, projectedCredsSecret(nil)) - got, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey) + got, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil) if err != nil { t.Fatalf("createRedisBackupArtifact: %v", err) } @@ -156,7 +237,7 @@ func TestCreateRedisBackupArtifact_Artifact(t *testing.T) { existing := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-1"}} l := &fakeS3Lister{statSize: 777} r := redisReconciler(t, l, j, existing, projectedCredsSecret(nil)) - if _, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey); err != nil { + if _, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil); err != nil { t.Fatalf("createRedisBackupArtifact: %v", err) } if l.calls != 0 { @@ -164,3 +245,73 @@ func TestCreateRedisBackupArtifact_Artifact(t *testing.T) { } }) } + +// TestReconcileRedis_ArtifactUsesTheJobTrust pins the wiring from the +// completed backup Job to the probe: the s3-ca volume of the pod the Job +// ran must reach the S3 connection. +func TestReconcileRedis_ArtifactUsesTheJobTrust(t *testing.T) { + app := newRedisApp("cache", "tenant-test") + strategy := newRedisStrategy("redis-strategy") + now := metav1.Now() + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newRedisAppRef("cache"), + BackupClassName: "cozy-default", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now, Phase: backupsv1alpha1.BackupJobPhaseRunning}, + } + completedK8sJob := &batchv1.Job{ + ObjectMeta: metav1.ObjectMeta{ + Name: jobNameForBackupJob(backupJob), + Namespace: backupJob.Namespace, + Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: backupJob.Name, + backupsv1alpha1.OwningJobNamespaceLabel: backupJob.Namespace, + }, + }, + Spec: batchv1.JobSpec{Template: corev1.PodTemplateSpec{Spec: *redisPodSpec("s3-endpoint-ca", "")}}, + Status: batchv1.JobStatus{ + CompletionTime: &now, + Conditions: []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}}, + }, + } + creds := projectedCredsSecret(nil) + creds.Namespace = "tenant-test" + ca := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant-test", Name: "s3-endpoint-ca"}, + Data: map[string][]byte{"ca.crt": []byte("CA")}, + } + resolved := newRedisResolved("redis-strategy", map[string]string{"bucketName": "redis-bucket"}) + + r, _ := newRedisTestEnv(t, app, clientfake.NewClientBuilder().WithObjects(backupJob, strategy, completedK8sJob, creds, ca)) + l := &fakeS3Lister{statSize: 512} + r.CredentialsConfig.TargetSecretName = "cozy-backups-creds" + r.newS3ObjectLister = func(conn s3Connection) (s3ObjectLister, error) { + l.conn = conn + return l, nil + } + ctx := context.Background() + + if _, err := r.reconcileRedis(ctx, backupJob, resolved); err != nil { + t.Fatalf("reconcileRedis() error = %v", err) + } + if string(l.conn.CAPEM) != "CA" { + t.Errorf("connection CA = %q, want the s3-ca volume's ca.crt", l.conn.CAPEM) + } + + updated := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, client.ObjectKeyFromObject(backupJob), updated); err != nil { + t.Fatalf("get backupjob: %v", err) + } + if updated.Status.BackupRef == nil { + t.Fatalf("expected BackupRef set") + } + created := &backupsv1alpha1.Backup{} + if err := r.Get(ctx, client.ObjectKey{Namespace: backupJob.Namespace, Name: updated.Status.BackupRef.Name}, created); err != nil { + t.Fatalf("get Backup: %v", err) + } + if created.Status.Artifact == nil || created.Status.Artifact.SizeBytes != 512 { + t.Errorf("Artifact = %#v, want SizeBytes 512", created.Status.Artifact) + } +} diff --git a/internal/backupcontroller/redisstrategy_controller.go b/internal/backupcontroller/redisstrategy_controller.go index 40f3a99aa..76030cb10 100644 --- a/internal/backupcontroller/redisstrategy_controller.go +++ b/internal/backupcontroller/redisstrategy_controller.go @@ -366,7 +366,7 @@ func (r *BackupJobReconciler) reconcileRedis(ctx context.Context, j *backupsv1al if j.Status.BackupRef != nil { return ctrl.Result{}, nil } - artifact, err := r.createRedisBackupArtifact(ctx, j, resolved, objectKey) + artifact, err := r.createRedisBackupArtifact(ctx, j, resolved, objectKey, &batchJob.Spec.Template.Spec) if err != nil { return r.failBackupArtifact(ctx, j, err) } @@ -471,6 +471,7 @@ func (r *BackupJobReconciler) createRedisBackupArtifact( j *backupsv1alpha1.BackupJob, resolved *ResolvedBackupConfig, objectKey string, + podSpec *corev1.PodSpec, ) (*backupsv1alpha1.Backup, error) { existing, probe := r.existingBackup(ctx, j.Namespace, j.Name) if existing != nil { @@ -497,7 +498,7 @@ func (r *BackupJobReconciler) createRedisBackupArtifact( }, } if probe { - backup.Status.Artifact = r.redisArtifact(ctx, j.Namespace, objectKey) + backup.Status.Artifact = r.redisArtifact(ctx, j.Namespace, objectKey, podSpec) } if j.Spec.PlanRef != nil { backup.Spec.PlanRef = j.Spec.PlanRef From bfea468b6c80d343c3a71e52741056da443ca0c1 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Wed, 21 Oct 2026 20:14:38 +0400 Subject: [PATCH 389/518] fix(backups): never skip TLS verification in place of a vanished CA The Redis probe reads the CA Secret when it runs, not when the backup Job did. If the pod mounted a CA Secret that has since been deleted and the strategy also opts into S3_INSECURE_SKIP_VERIFY, the probe skipped verification against an endpoint the Job may have verified. Refuse to probe in that case. Without skip-verify a missing CA Secret still falls back to the system roots, as the script does. The endpoint comment claimed a bare host; it can carry a path, which the shared client now refuses. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/redis_artifact_size.go | 16 ++++++++++++---- .../backupcontroller/redis_artifact_size_test.go | 14 ++++++++++++-- 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/internal/backupcontroller/redis_artifact_size.go b/internal/backupcontroller/redis_artifact_size.go index 1c66f49af..c01545c4a 100644 --- a/internal/backupcontroller/redis_artifact_size.go +++ b/internal/backupcontroller/redis_artifact_size.go @@ -52,8 +52,9 @@ func (r *BackupJobReconciler) redisArtifact(ctx context.Context, namespace, obje } // projectedS3Connection reads the backup credentials Secret the controller -// projects into the application namespace. The projector always stores the -// endpoint as a bare host, which the strategy scripts reach over https. +// projects into the application namespace. The projector stores the endpoint +// without a scheme, and the strategy scripts reach it over https. It can +// still carry a path, which minio-go refuses. func (r *BackupJobReconciler) projectedS3Connection(ctx context.Context, namespace string) (string, s3Connection, error) { var conn s3Connection name := r.CredentialsConfig.TargetSecretName @@ -85,12 +86,15 @@ const ( // endpoint: it trusts ca.crt, then tls.crt, from the Secret mounted as the // s3-ca volume, and only when neither exists does it honour // S3_INSECURE_SKIP_VERIFY=true. Both are read from the pod spec the backup -// Job ran with. A missing CA Secret is not an error, because the script -// mounts it optionally. +// Job ran with, but the Secret is read now, so a CA Secret the pod mounts +// and that is gone by now never lets the probe skip verification: the Job +// may have verified against it. Without skip-verify a missing CA Secret +// leaves the system roots, as in the script, which mounts it optionally. func (r *BackupJobReconciler) redisS3Trust(ctx context.Context, namespace string, podSpec *corev1.PodSpec) ([]byte, bool, error) { if podSpec == nil { return nil, false, nil } + caMissing := false for _, v := range podSpec.Volumes { if v.Name != redisS3CAVolume || v.Secret == nil { continue @@ -98,6 +102,7 @@ func (r *BackupJobReconciler) redisS3Trust(ctx context.Context, namespace string secret := &corev1.Secret{} err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: v.Secret.SecretName}, secret) if apierrors.IsNotFound(err) { + caMissing = true break } if err != nil { @@ -112,6 +117,9 @@ func (r *BackupJobReconciler) redisS3Trust(ctx context.Context, namespace string for _, c := range podSpec.Containers { for _, e := range c.Env { if e.Name == redisS3InsecureSkipEnv && e.Value == "true" { + if caMissing { + return nil, false, fmt.Errorf("CA Secret of the %s volume is gone; not skipping TLS verification in its place", redisS3CAVolume) + } return nil, true, nil } } diff --git a/internal/backupcontroller/redis_artifact_size_test.go b/internal/backupcontroller/redis_artifact_size_test.go index a167d8f4a..964533546 100644 --- a/internal/backupcontroller/redis_artifact_size_test.go +++ b/internal/backupcontroller/redis_artifact_size_test.go @@ -188,8 +188,11 @@ func TestRedisS3Trust(t *testing.T) { objs: []client.Object{caSecret(map[string]string{"bundle.pem": "MAPPED", "ca.crt": "UNMAPPED"})}, wantCA: "MAPPED", }, - "missing CA Secret falls through to skip-verify": { - spec: redisPodSpec("s3-endpoint-ca", "true"), + "missing CA Secret falls back to the system roots": { + spec: redisPodSpec("s3-endpoint-ca", ""), + }, + "skip-verify without a CA volume": { + spec: redisPodSpec("", "true"), wantSkip: true, }, "skip-verify only when it is exactly true": { @@ -197,6 +200,13 @@ func TestRedisS3Trust(t *testing.T) { }, "no pod spec": {}, } + t.Run("a mounted CA Secret that is gone never turns into skip-verify", func(t *testing.T) { + r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t)} + ca, skip, err := r.redisS3Trust(context.Background(), "tenant", redisPodSpec("s3-endpoint-ca", "true")) + if err == nil || skip || ca != nil { + t.Errorf("trust = (%q, %v, %v), want an error and no skip-verify", ca, skip, err) + } + }) for name, tc := range cases { t.Run(name, func(t *testing.T) { c := newMariaDBStrategyTestClient(t, tc.objs...) From 50b1a2e5eee2559a5ef3e7f62da9577cb18c8667 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Thu, 22 Oct 2026 19:41:48 +0400 Subject: [PATCH 390/518] test(backups): pin the Redis retry check Cover the two retry-check paths the Redis driver shares with MariaDB and CNPG: a failed existence check creates the Backup without probing the bucket, and an existing Backup that cannot be read back is returned as errBackupNotReadBack so the reconcile is retried. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- .../redis_artifact_size_test.go | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/internal/backupcontroller/redis_artifact_size_test.go b/internal/backupcontroller/redis_artifact_size_test.go index 964533546..a6dbd52ee 100644 --- a/internal/backupcontroller/redis_artifact_size_test.go +++ b/internal/backupcontroller/redis_artifact_size_test.go @@ -325,3 +325,36 @@ func TestReconcileRedis_ArtifactUsesTheJobTrust(t *testing.T) { t.Errorf("Artifact = %#v, want SizeBytes 512", created.Status.Artifact) } } + +func TestCreateRedisBackupArtifact_RetryCheck(t *testing.T) { + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-1"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: "Redis", Name: "cache"}, + }, + } + + t.Run("a failed existence check creates the Backup without probing", func(t *testing.T) { + l := &fakeS3Lister{statSize: 777} + r := redisReconciler(t, l, j, projectedCredsSecret(nil)) + r.APIReader = failingReads(t, newMariaDBStrategyTestClient(t)) + got, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil) + if err != nil || got == nil { + t.Fatalf("createRedisBackupArtifact = %v, %v; a transient read must not fail the backup", got, err) + } + if l.calls != 0 { + t.Errorf("S3 probed %d times after the existence check failed", l.calls) + } + }) + + t.Run("an existing Backup that cannot be read back is retried", func(t *testing.T) { + backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-1"}} + store := newMariaDBStrategyTestClient(t, j, backup, projectedCredsSecret(nil)) + r := redisReconciler(t, &fakeS3Lister{statSize: 777}) + r.Client, r.APIReader = staleCache(t, store), failingReads(t, store) + if _, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil); !errors.Is(err, errBackupNotReadBack) { + t.Errorf("err = %v, want errBackupNotReadBack", err) + } + }) +} From 631f6cfc0dc0e5172302034747850249f722ad1f Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Thu, 22 Oct 2026 20:13:47 +0400 Subject: [PATCH 391/518] fix(backups): mirror the Redis script's CA trust exactly The script passes the endpoint CA as curl --cacert, which replaces the default trust store, while the probe trusted the system roots on top of it. Trust only the CA, as the script does. A mounted CA Secret that is still there but no longer carries ca.crt or tls.crt let the probe skip TLS verification when the strategy opts into it, although the Job may have verified against a key since removed. Treat a volume that yields no CA now like a vanished Secret and refuse to skip verification in its place. Pin the probe timeout and the reconcile's retry of an unreadable Backup for Redis with tests. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- .../backupcontroller/redis_artifact_size.go | 6 +- .../redis_artifact_size_test.go | 81 +++++++++++++++++++ 2 files changed, 86 insertions(+), 1 deletion(-) diff --git a/internal/backupcontroller/redis_artifact_size.go b/internal/backupcontroller/redis_artifact_size.go index c01545c4a..22fad244f 100644 --- a/internal/backupcontroller/redis_artifact_size.go +++ b/internal/backupcontroller/redis_artifact_size.go @@ -35,6 +35,9 @@ func (r *BackupJobReconciler) redisArtifact(ctx context.Context, namespace, obje logger.Info("cannot read the S3 endpoint CA to locate the Redis snapshot", "objectKey", objectKey, "error", err) return nil } + // The script passes the CA as curl --cacert, which replaces the default + // trust store rather than adding to it. + conn.CAOnly = true lister, err := r.s3Lister(conn) if err != nil { logger.Info("cannot build S3 client to locate the Redis snapshot", "objectKey", objectKey, "error", err) @@ -113,12 +116,13 @@ func (r *BackupJobReconciler) redisS3Trust(ctx context.Context, namespace string return pem, false, nil } } + caMissing = true } for _, c := range podSpec.Containers { for _, e := range c.Env { if e.Name == redisS3InsecureSkipEnv && e.Value == "true" { if caMissing { - return nil, false, fmt.Errorf("CA Secret of the %s volume is gone; not skipping TLS verification in its place", redisS3CAVolume) + return nil, false, fmt.Errorf("the %s volume yields no CA now; not skipping TLS verification in its place", redisS3CAVolume) } return nil, true, nil } diff --git a/internal/backupcontroller/redis_artifact_size_test.go b/internal/backupcontroller/redis_artifact_size_test.go index a6dbd52ee..4f88525ea 100644 --- a/internal/backupcontroller/redis_artifact_size_test.go +++ b/internal/backupcontroller/redis_artifact_size_test.go @@ -200,6 +200,13 @@ func TestRedisS3Trust(t *testing.T) { }, "no pod spec": {}, } + t.Run("a mounted CA Secret without CA keys never turns into skip-verify", func(t *testing.T) { + r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t, caSecret(map[string]string{"other": "x"}))} + ca, skip, err := r.redisS3Trust(context.Background(), "tenant", redisPodSpec("s3-endpoint-ca", "true")) + if err == nil || skip || ca != nil { + t.Errorf("trust = (%q, %v, %v), want an error and no skip-verify", ca, skip, err) + } + }) t.Run("a mounted CA Secret that is gone never turns into skip-verify", func(t *testing.T) { r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t)} ca, skip, err := r.redisS3Trust(context.Background(), "tenant", redisPodSpec("s3-endpoint-ca", "true")) @@ -358,3 +365,77 @@ func TestCreateRedisBackupArtifact_RetryCheck(t *testing.T) { } }) } + +func TestRedisArtifact_HangingEndpointIsGivenUp(t *testing.T) { + r := redisReconciler(t, &fakeS3Lister{}, projectedCredsSecret(nil)) + r.newS3ObjectLister = func(s3Connection) (s3ObjectLister, error) { return blockingLister{}, nil } + withShortProbeTimeout(t, func() { + if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, nil); got != nil { + t.Errorf("artifact = %#v, want nil", got) + } + }) +} + +func TestRedisArtifact_TrustsOnlyTheCA(t *testing.T) { + l := &fakeS3Lister{statSize: 1} + ca := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "s3-endpoint-ca"}, + Data: map[string][]byte{"ca.crt": []byte("PEM")}, + } + r := redisReconciler(t, l, projectedCredsSecret(nil), ca) + if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("s3-endpoint-ca", "")); got == nil { + t.Fatal("artifact = nil") + } + if !l.conn.CAOnly { + t.Error("the probe trusts the system roots on top of the CA, which curl --cacert does not") + } +} + +// TestReconcileRedis_UnreadableBackupIsRetried drives a completed backup +// through the reconcile: when the Backup exists but cannot be read back, +// the reconcile is retried and the BackupJob is not failed. +func TestReconcileRedis_UnreadableBackupIsRetried(t *testing.T) { + app := newRedisApp("cache", "tenant-test") + strategy := newRedisStrategy("redis-strategy") + now := metav1.Now() + backupJob := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: newRedisAppRef("cache"), + BackupClassName: "cozy-default", + }, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &now, Phase: backupsv1alpha1.BackupJobPhaseRunning}, + } + completedK8sJob := &batchv1.Job{ + ObjectMeta: metav1.ObjectMeta{ + Name: jobNameForBackupJob(backupJob), + Namespace: backupJob.Namespace, + Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: backupJob.Name, + backupsv1alpha1.OwningJobNamespaceLabel: backupJob.Namespace, + }, + }, + Status: batchv1.JobStatus{ + CompletionTime: &now, + Conditions: []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}}, + }, + } + existing := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant-test", Name: "test-bj"}} + resolved := newRedisResolved("redis-strategy", map[string]string{"bucketName": "redis-bucket"}) + + r, _ := newRedisTestEnv(t, app, clientfake.NewClientBuilder().WithObjects(backupJob, strategy, completedK8sJob, existing)) + store := r.Client + r.Client, r.APIReader = staleCache(t, store), failingReads(t, store) + ctx := context.Background() + + if _, err := r.reconcileRedis(ctx, backupJob, resolved); !errors.Is(err, errBackupNotReadBack) { + t.Fatalf("reconcileRedis err = %v, want errBackupNotReadBack so the reconcile is retried", err) + } + persisted := &backupsv1alpha1.BackupJob{} + if err := store.Get(ctx, client.ObjectKeyFromObject(backupJob), persisted); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if persisted.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Error("the BackupJob was failed although its Backup exists") + } +} From e03986e8d09ab4b154fad6c4e47158c600a93b78 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Tue, 10 Nov 2026 11:01:18 +0700 Subject: [PATCH 392/518] fix(backups): treat an S3 object without a reported size as unmeasured minio-go returns size -1 when a HEAD answer carries no Content-Length, which would land in Backup.status.artifact.sizeBytes as -1. The probe now reports such an object as unmeasured, so the field stays unset. The shared client's trust settings also gain a test against an endpoint whose certificate the configured CA did not sign, with the CA alone and on top of the system roots: nothing pinned that a configured CA is actually enforced. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- internal/backupcontroller/artifact_size.go | 4 + .../backupcontroller/artifact_size_test.go | 84 ++++++++++++++++--- 2 files changed, 76 insertions(+), 12 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 230ca3ce8..6376c8e87 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -161,6 +161,10 @@ func (l *minioObjectLister) StatObject(ctx context.Context, bucket, key string) if err != nil { return 0, err } + // minio-go reports -1 when the response carries no Content-Length. + if info.Size < 0 { + return 0, fmt.Errorf("object %s/%s: the endpoint reported no size", bucket, key) + } return info.Size, nil } diff --git a/internal/backupcontroller/artifact_size_test.go b/internal/backupcontroller/artifact_size_test.go index 55073e742..aff417344 100644 --- a/internal/backupcontroller/artifact_size_test.go +++ b/internal/backupcontroller/artifact_size_test.go @@ -14,6 +14,7 @@ import ( "math/big" "net/http" "net/http/httptest" + "strings" "testing" "time" @@ -248,6 +249,76 @@ func TestFailBackupArtifact(t *testing.T) { }) } +// unrelatedCAPEM mints a CA that signed nothing an httptest server serves: +// every httptest server shares one certificate. +func unrelatedCAPEM(t *testing.T) []byte { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + tmpl := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "unrelated CA"}, NotBefore: time.Now(), NotAfter: time.Now().Add(time.Hour), IsCA: true, BasicConstraintsValid: true} + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + if err != nil { + t.Fatal(err) + } + return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}) +} + +// TestMinioObjectLister_EnforcesTheCA pins that a configured CA is the +// trust the client verifies with, whether alone or on top of the system +// roots: an endpoint whose certificate it did not sign is refused. +func TestMinioObjectLister_EnforcesTheCA(t *testing.T) { + srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Length", "1") + w.Header().Set("ETag", `"abc"`) + w.Header().Set("Last-Modified", "Wed, 21 Oct 2026 10:00:00 GMT") + })) + defer srv.Close() + // minio-go retries a TLS failure with backoff; a refusal needs no more + // than the first attempt. + stat := func(ca []byte, caOnly bool) error { + l, err := newMinioObjectLister(s3Connection{ + Endpoint: strings.TrimPrefix(srv.URL, "https://"), Secure: true, + AccessKey: "AKID", SecretKey: "secret", Region: "us-east-1", + CAPEM: ca, CAOnly: caOnly, + }) + if err != nil { + t.Fatalf("newMinioObjectLister: %v", err) + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + _, err = l.StatObject(ctx, "cozy-backups", "k") + return err + } + signer := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: srv.Certificate().Raw}) + for _, caOnly := range []bool{true, false} { + if err := stat(signer, caOnly); err != nil { + t.Errorf("CAOnly=%v: the CA that signed the endpoint: %v", caOnly, err) + } + if err := stat(unrelatedCAPEM(t), caOnly); err == nil { + t.Errorf("CAOnly=%v: StatObject succeeded against a certificate the CA did not sign, want a TLS error", caOnly) + } + } +} + +// TestMinioObjectLister_StatWithoutSize pins that a HEAD answer without +// Content-Length, which minio-go turns into size -1, is not a size. +func TestMinioObjectLister_StatWithoutSize(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("ETag", `"abc"`) + w.Header().Set("Last-Modified", "Wed, 21 Oct 2026 10:00:00 GMT") + })) + defer srv.Close() + l, err := newMinioObjectLister(s3Connection{Endpoint: strings.TrimPrefix(srv.URL, "http://"), AccessKey: "AKID", SecretKey: "secret", Region: "us-east-1"}) + if err != nil { + t.Fatalf("newMinioObjectLister: %v", err) + } + if size, err := l.StatObject(context.Background(), "cozy-backups", "k"); err == nil { + t.Errorf("StatObject = %d, nil; want an error for a response without Content-Length", size) + } +} + // TestEndpointTrustPool_CAOnly tells apart "the CA on top of the system // roots" and "the CA alone". The system pool holds the endpoint's // certificate, and the configured CA is an unrelated one, so the endpoint @@ -258,18 +329,7 @@ func TestEndpointTrustPool_CAOnly(t *testing.T) { system := x509.NewCertPool() system.AddCert(endpoint.Certificate()) - // Every httptest server shares one certificate, so the unrelated CA is - // minted here. - key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) - if err != nil { - t.Fatal(err) - } - tmpl := &x509.Certificate{SerialNumber: big.NewInt(1), Subject: pkix.Name{CommonName: "unrelated CA"}, NotBefore: time.Now(), NotAfter: time.Now().Add(time.Hour), IsCA: true, BasicConstraintsValid: true} - der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) - if err != nil { - t.Fatal(err) - } - unrelatedCA := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}) + unrelatedCA := unrelatedCAPEM(t) verifies := func(caOnly bool) bool { pool, err := endpointTrustPool(s3Connection{CAPEM: unrelatedCA, CAOnly: caOnly}, system.Clone()) From 43bed8f6d932925e80715d4491924174387e2ac7 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Tue, 10 Nov 2026 11:01:18 +0700 Subject: [PATCH 393/518] fix(backups): keep the Redis probe verifying TLS unless every container skips The probe read S3_INSECURE_SKIP_VERIFY from any container of the backup Job, so a custom strategy whose sidecar sets it while the uploading container verifies made the probe skip verification the upload did not. The probe cannot tell which container uploaded, so it now skips only when every container sets the variable to true, the last entry winning as in the kubelet. An s3-ca volume of a source other than a Secret yields no CA the probe can read; it is now treated like a CA Secret that is gone, so it never turns into skip-verify. Tests pin the guards that decide the probe's trust: the Redis CA volume mounted next to the S3 one, a decoy variable set to true, a CA that did not sign the endpoint, and the AlreadyExists read-back past the cache. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- .../backupcontroller/redis_artifact_size.go | 57 +++++++++----- .../redis_artifact_size_test.go | 78 +++++++++++++++++++ 2 files changed, 117 insertions(+), 18 deletions(-) diff --git a/internal/backupcontroller/redis_artifact_size.go b/internal/backupcontroller/redis_artifact_size.go index 22fad244f..5977f42aa 100644 --- a/internal/backupcontroller/redis_artifact_size.go +++ b/internal/backupcontroller/redis_artifact_size.go @@ -86,26 +86,31 @@ const ( ) // redisS3Trust mirrors how the default strategy script verifies the S3 -// endpoint: it trusts ca.crt, then tls.crt, from the Secret mounted as the -// s3-ca volume, and only when neither exists does it honour -// S3_INSECURE_SKIP_VERIFY=true. Both are read from the pod spec the backup -// Job ran with, but the Secret is read now, so a CA Secret the pod mounts -// and that is gone by now never lets the probe skip verification: the Job -// may have verified against it. Without skip-verify a missing CA Secret -// leaves the system roots, as in the script, which mounts it optionally. +// endpoint: it trusts ca.crt, then tls.crt, from the s3-ca volume, and only +// when neither exists does it honour S3_INSECURE_SKIP_VERIFY=true. Both are +// read from the pod spec the backup Job ran with, but the Secret is read +// now, so an s3-ca volume that yields no CA now - a Secret that is gone or +// lacks the keys, or a source other than a Secret - never lets the probe +// skip verification: the Job may have verified against it. Without +// skip-verify such a volume leaves the system roots, as in the script, +// which mounts it optionally. func (r *BackupJobReconciler) redisS3Trust(ctx context.Context, namespace string, podSpec *corev1.PodSpec) ([]byte, bool, error) { if podSpec == nil { return nil, false, nil } - caMissing := false + caUnknown := false for _, v := range podSpec.Volumes { - if v.Name != redisS3CAVolume || v.Secret == nil { + if v.Name != redisS3CAVolume { continue } + if v.Secret == nil { + caUnknown = true + break + } secret := &corev1.Secret{} err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: v.Secret.SecretName}, secret) if apierrors.IsNotFound(err) { - caMissing = true + caUnknown = true break } if err != nil { @@ -116,19 +121,35 @@ func (r *BackupJobReconciler) redisS3Trust(ctx context.Context, namespace string return pem, false, nil } } - caMissing = true + caUnknown = true } - for _, c := range podSpec.Containers { + if !redisSkipsVerify(podSpec.Containers) { + return nil, false, nil + } + if caUnknown { + return nil, false, fmt.Errorf("the %s volume yields no CA now; not skipping TLS verification in its place", redisS3CAVolume) + } + return nil, true, nil +} + +// redisSkipsVerify reports whether every container sets +// S3_INSECURE_SKIP_VERIFY=true: the probe cannot tell which container +// uploaded, so one that verified is enough to keep verification on. When a +// container lists the variable twice, the last entry wins, as in the +// kubelet. +func redisSkipsVerify(containers []corev1.Container) bool { + for _, c := range containers { + value := "" for _, e := range c.Env { - if e.Name == redisS3InsecureSkipEnv && e.Value == "true" { - if caMissing { - return nil, false, fmt.Errorf("the %s volume yields no CA now; not skipping TLS verification in its place", redisS3CAVolume) - } - return nil, true, nil + if e.Name == redisS3InsecureSkipEnv { + value = e.Value } } + if value != "true" { + return false + } } - return nil, false, nil + return len(containers) > 0 } // secretKeyForFile returns the Secret key a secret volume projects to file: diff --git a/internal/backupcontroller/redis_artifact_size_test.go b/internal/backupcontroller/redis_artifact_size_test.go index 4f88525ea..9f95735e9 100644 --- a/internal/backupcontroller/redis_artifact_size_test.go +++ b/internal/backupcontroller/redis_artifact_size_test.go @@ -9,6 +9,7 @@ import ( "net/http/httptest" "strings" "testing" + "time" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" @@ -98,6 +99,11 @@ func TestRedisArtifact(t *testing.T) { // covers, in one pass, the trust settings the backup Job used, request // signing and Content-Length parsing. func TestRedisArtifact_MinioClient(t *testing.T) { + // minio-go retries a TLS failure with backoff; a refusal needs no more + // than the first attempt. + saved := artifactSizeProbeTimeout + artifactSizeProbeTimeout = time.Second + defer func() { artifactSizeProbeTimeout = saved }() var gotMethod, gotPath, gotAuth string srv := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { gotMethod, gotPath, gotAuth = req.Method, req.URL.Path, req.Header.Get("Authorization") @@ -136,6 +142,14 @@ func TestRedisArtifact_MinioClient(t *testing.T) { if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("", "true")); got == nil || got.SizeBytes != 4096 { t.Errorf("with S3_INSECURE_SKIP_VERIFY=true: artifact = %#v, want SizeBytes 4096", got) } + + ca.Data["ca.crt"] = unrelatedCAPEM(t) + if err := c.Update(context.Background(), ca); err != nil { + t.Fatalf("update CA Secret: %v", err) + } + if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("s3-endpoint-ca", "true")); got != nil { + t.Errorf("with a CA that did not sign the endpoint and S3_INSECURE_SKIP_VERIFY=true: artifact = %#v, want nil", got) + } } // redisPodSpec builds the parts of the backup Job's pod spec that @@ -155,6 +169,22 @@ func redisPodSpec(caSecret, skipVerify string) *corev1.PodSpec { return spec } +func withVolume(spec *corev1.PodSpec, name, secret string) *corev1.PodSpec { + spec.Volumes = append(spec.Volumes, corev1.Volume{ + Name: name, + VolumeSource: corev1.VolumeSource{Secret: &corev1.SecretVolumeSource{SecretName: secret}}, + }) + return spec +} + +func configMapS3CA(spec *corev1.PodSpec) *corev1.PodSpec { + spec.Volumes = append(spec.Volumes, corev1.Volume{ + Name: "s3-ca", + VolumeSource: corev1.VolumeSource{ConfigMap: &corev1.ConfigMapVolumeSource{LocalObjectReference: corev1.LocalObjectReference{Name: "s3-endpoint-ca"}}}, + }) + return spec +} + func TestRedisS3Trust(t *testing.T) { caSecret := func(data map[string]string) *corev1.Secret { s := &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "s3-endpoint-ca"}, Data: map[string][]byte{}} @@ -199,7 +229,44 @@ func TestRedisS3Trust(t *testing.T) { spec: redisPodSpec("", "false"), }, "no pod spec": {}, + "the Redis CA volume is not the S3 CA": { + spec: withVolume(redisPodSpec("", "true"), "redis-ca", "redis-tls"), + objs: []client.Object{caSecret(map[string]string{"ca.crt": "CA"}), + &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "redis-tls"}, Data: map[string][]byte{"ca.crt": []byte("REDIS")}}}, + wantSkip: true, + }, + "the S3 CA next to the Redis CA": { + spec: withVolume(withVolume(&corev1.PodSpec{Containers: redisPodSpec("", "").Containers}, "redis-ca", "redis-tls"), "s3-ca", "s3-endpoint-ca"), + objs: []client.Object{caSecret(map[string]string{"ca.crt": "CA"}), + &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "redis-tls"}, Data: map[string][]byte{"ca.crt": []byte("REDIS")}}}, + wantCA: "CA", + }, + "another variable set to true is not skip-verify": { + spec: &corev1.PodSpec{Containers: []corev1.Container{{Name: "backup", Env: []corev1.EnvVar{{Name: "REDIS_TLS", Value: "true"}}}}}, + }, + "skip-verify only when every container skips": { + spec: &corev1.PodSpec{Containers: []corev1.Container{ + {Name: "upload", Env: []corev1.EnvVar{{Name: "S3_INSECURE_SKIP_VERIFY", Value: "false"}}}, + {Name: "sidecar", Env: []corev1.EnvVar{{Name: "S3_INSECURE_SKIP_VERIFY", Value: "true"}}}, + }}, + }, + "the last entry of a repeated variable wins": { + spec: &corev1.PodSpec{Containers: []corev1.Container{{Name: "backup", Env: []corev1.EnvVar{ + {Name: "S3_INSECURE_SKIP_VERIFY", Value: "true"}, + {Name: "S3_INSECURE_SKIP_VERIFY", Value: "false"}, + }}}}, + }, + "an s3-ca volume of another source leaves the system roots": { + spec: configMapS3CA(redisPodSpec("", "")), + }, } + t.Run("an s3-ca volume of another source never turns into skip-verify", func(t *testing.T) { + r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t)} + ca, skip, err := r.redisS3Trust(context.Background(), "tenant", configMapS3CA(redisPodSpec("", "true"))) + if err == nil || skip || ca != nil { + t.Errorf("trust = (%q, %v, %v), want an error and no skip-verify", ca, skip, err) + } + }) t.Run("a mounted CA Secret without CA keys never turns into skip-verify", func(t *testing.T) { r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t, caSecret(map[string]string{"other": "x"}))} ca, skip, err := r.redisS3Trust(context.Background(), "tenant", redisPodSpec("s3-endpoint-ca", "true")) @@ -355,6 +422,17 @@ func TestCreateRedisBackupArtifact_RetryCheck(t *testing.T) { } }) + t.Run("a Create collision is resolved from the API server past the cache", func(t *testing.T) { + backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-1"}} + store := newMariaDBStrategyTestClient(t, j, backup, projectedCredsSecret(nil)) + r := redisReconciler(t, &fakeS3Lister{statSize: 777}) + r.Client, r.APIReader = staleCache(t, store), notFoundOnce(t, store) + got, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil) + if err != nil || got == nil || got.Name != "bj-1" { + t.Fatalf("createRedisBackupArtifact = %v, %v; want the Backup that already exists", got, err) + } + }) + t.Run("an existing Backup that cannot be read back is retried", func(t *testing.T) { backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-1"}} store := newMariaDBStrategyTestClient(t, j, backup, projectedCredsSecret(nil)) From d61a5054a1455be41c0e287f8573c5d9fa49a603 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Tue, 10 Nov 2026 11:21:46 +0700 Subject: [PATCH 394/518] fix(backups): refuse to probe a Redis snapshot behind a CA that is gone The default strategy mounts the s3-ca volume only when the admin configured an endpoint CA, and the script then trusts that CA alone. When the volume yields no CA by the time the probe runs, because the Secret was deleted or emptied after the Job or the volume has another source, the probe used to fall back to the system roots, trusting more than the Job may have. It now leaves the artifact unset instead, with or without skip-verify. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- .../backupcontroller/redis_artifact_size.go | 15 +++--- .../redis_artifact_size_test.go | 49 +++++++++---------- 2 files changed, 29 insertions(+), 35 deletions(-) diff --git a/internal/backupcontroller/redis_artifact_size.go b/internal/backupcontroller/redis_artifact_size.go index 5977f42aa..c49ee3960 100644 --- a/internal/backupcontroller/redis_artifact_size.go +++ b/internal/backupcontroller/redis_artifact_size.go @@ -90,10 +90,10 @@ const ( // when neither exists does it honour S3_INSECURE_SKIP_VERIFY=true. Both are // read from the pod spec the backup Job ran with, but the Secret is read // now, so an s3-ca volume that yields no CA now - a Secret that is gone or -// lacks the keys, or a source other than a Secret - never lets the probe -// skip verification: the Job may have verified against it. Without -// skip-verify such a volume leaves the system roots, as in the script, -// which mounts it optionally. +// lacks the keys, or a source other than a Secret - is refused: the Job may +// have verified against a CA that is gone, and neither the system roots nor +// skip-verify would trust as narrowly. The default strategy mounts s3-ca +// only when the admin configured an endpoint CA. func (r *BackupJobReconciler) redisS3Trust(ctx context.Context, namespace string, podSpec *corev1.PodSpec) ([]byte, bool, error) { if podSpec == nil { return nil, false, nil @@ -123,13 +123,10 @@ func (r *BackupJobReconciler) redisS3Trust(ctx context.Context, namespace string } caUnknown = true } - if !redisSkipsVerify(podSpec.Containers) { - return nil, false, nil - } if caUnknown { - return nil, false, fmt.Errorf("the %s volume yields no CA now; not skipping TLS verification in its place", redisS3CAVolume) + return nil, false, fmt.Errorf("the %s volume yields no CA now", redisS3CAVolume) } - return nil, true, nil + return nil, redisSkipsVerify(podSpec.Containers), nil } // redisSkipsVerify reports whether every container sets diff --git a/internal/backupcontroller/redis_artifact_size_test.go b/internal/backupcontroller/redis_artifact_size_test.go index 9f95735e9..24050baf1 100644 --- a/internal/backupcontroller/redis_artifact_size_test.go +++ b/internal/backupcontroller/redis_artifact_size_test.go @@ -218,9 +218,6 @@ func TestRedisS3Trust(t *testing.T) { objs: []client.Object{caSecret(map[string]string{"bundle.pem": "MAPPED", "ca.crt": "UNMAPPED"})}, wantCA: "MAPPED", }, - "missing CA Secret falls back to the system roots": { - spec: redisPodSpec("s3-endpoint-ca", ""), - }, "skip-verify without a CA volume": { spec: redisPodSpec("", "true"), wantSkip: true, @@ -256,31 +253,31 @@ func TestRedisS3Trust(t *testing.T) { {Name: "S3_INSECURE_SKIP_VERIFY", Value: "false"}, }}}}, }, - "an s3-ca volume of another source leaves the system roots": { - spec: configMapS3CA(redisPodSpec("", "")), - }, } - t.Run("an s3-ca volume of another source never turns into skip-verify", func(t *testing.T) { - r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t)} - ca, skip, err := r.redisS3Trust(context.Background(), "tenant", configMapS3CA(redisPodSpec("", "true"))) - if err == nil || skip || ca != nil { - t.Errorf("trust = (%q, %v, %v), want an error and no skip-verify", ca, skip, err) + // Each s3-ca volume that yields no CA now is refused, with or without + // skip-verify: the Job may have verified against a CA that is gone. + refused := map[string]struct { + spec func(skip string) *corev1.PodSpec + objs []client.Object + }{ + "a CA Secret that is gone": {spec: func(skip string) *corev1.PodSpec { return redisPodSpec("s3-endpoint-ca", skip) }}, + "a CA Secret without CA keys": { + spec: func(skip string) *corev1.PodSpec { return redisPodSpec("s3-endpoint-ca", skip) }, + objs: []client.Object{caSecret(map[string]string{"other": "x"})}, + }, + "an s3-ca volume of another source": {spec: func(skip string) *corev1.PodSpec { return configMapS3CA(redisPodSpec("", skip)) }}, + } + for name, tc := range refused { + for _, skip := range []string{"", "true"} { + t.Run(name+" is refused, skip="+skip, func(t *testing.T) { + r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t, tc.objs...)} + ca, gotSkip, err := r.redisS3Trust(context.Background(), "tenant", tc.spec(skip)) + if err == nil || gotSkip || ca != nil { + t.Errorf("trust = (%q, %v, %v), want an error, no CA and no skip-verify", ca, gotSkip, err) + } + }) } - }) - t.Run("a mounted CA Secret without CA keys never turns into skip-verify", func(t *testing.T) { - r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t, caSecret(map[string]string{"other": "x"}))} - ca, skip, err := r.redisS3Trust(context.Background(), "tenant", redisPodSpec("s3-endpoint-ca", "true")) - if err == nil || skip || ca != nil { - t.Errorf("trust = (%q, %v, %v), want an error and no skip-verify", ca, skip, err) - } - }) - t.Run("a mounted CA Secret that is gone never turns into skip-verify", func(t *testing.T) { - r := &BackupJobReconciler{Client: newMariaDBStrategyTestClient(t)} - ca, skip, err := r.redisS3Trust(context.Background(), "tenant", redisPodSpec("s3-endpoint-ca", "true")) - if err == nil || skip || ca != nil { - t.Errorf("trust = (%q, %v, %v), want an error and no skip-verify", ca, skip, err) - } - }) + } for name, tc := range cases { t.Run(name, func(t *testing.T) { c := newMariaDBStrategyTestClient(t, tc.objs...) From 6f3d26c4914bb024deaeada3fd96633fa25b6f4c Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Tue, 10 Nov 2026 11:21:46 +0700 Subject: [PATCH 395/518] feat(backups): tell on the BackupJob why a Redis artifact was not recorded A Backup without status.artifact looked the same whether a custom strategy wrote elsewhere, the credentials Secret was unusable or the endpoint CA was gone; only the controller log told them apart. The probe now returns why it gave up, and the controller records it as a Normal ArtifactNotMeasured Event on the BackupJob. The Event lives in the application namespace, so it carries a fixed reason only; the raw error, which may name the platform's endpoint and bucket, stays in the log. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- .../backupcontroller/redis_artifact_size.go | 36 +++++++------ .../redis_artifact_size_test.go | 51 ++++++++++++++++--- .../redisstrategy_controller.go | 11 +++- 3 files changed, 74 insertions(+), 24 deletions(-) diff --git a/internal/backupcontroller/redis_artifact_size.go b/internal/backupcontroller/redis_artifact_size.go index c49ee3960..49a73d292 100644 --- a/internal/backupcontroller/redis_artifact_size.go +++ b/internal/backupcontroller/redis_artifact_size.go @@ -17,43 +17,49 @@ import ( // an Artifact carrying its s3:// URI and size. The default Redis strategy // takes its S3 coordinates from the projected backup credentials Secret, so // the object is looked up there, with the TLS trust the Job's pod applied -// (see redisS3Trust). Returns nil - leaving Status.Artifact absent - when -// the Secret cannot be read or the object is not found: a custom strategy -// that writes elsewhere must not get a URI pointing at an object that does -// not exist. -func (r *BackupJobReconciler) redisArtifact(ctx context.Context, namespace, objectKey string, podSpec *corev1.PodSpec) *backupsv1alpha1.BackupArtifact { - logger := getLogger(ctx) +// (see redisS3Trust). It returns an artifactProbeError, and no Artifact, +// when the Secret cannot be read or the object is not found: a custom +// strategy that writes elsewhere must not get a URI pointing at an object +// that does not exist. +func (r *BackupJobReconciler) redisArtifact(ctx context.Context, namespace, objectKey string, podSpec *corev1.PodSpec) (*backupsv1alpha1.BackupArtifact, error) { ctx, cancel := context.WithTimeout(ctx, artifactSizeProbeTimeout) defer cancel() bucket, conn, err := r.projectedS3Connection(ctx, namespace) if err != nil { - logger.Info("cannot resolve backup credentials to locate the Redis snapshot", "objectKey", objectKey, "error", err) - return nil + return nil, &artifactProbeError{reason: "the backup credentials Secret is unusable", err: err} } if conn.CAPEM, conn.InsecureSkipVerify, err = r.redisS3Trust(ctx, namespace, podSpec); err != nil { - logger.Info("cannot read the S3 endpoint CA to locate the Redis snapshot", "objectKey", objectKey, "error", err) - return nil + return nil, &artifactProbeError{reason: "the S3 endpoint CA the Job mounted cannot be read", err: err} } // The script passes the CA as curl --cacert, which replaces the default // trust store rather than adding to it. conn.CAOnly = true lister, err := r.s3Lister(conn) if err != nil { - logger.Info("cannot build S3 client to locate the Redis snapshot", "objectKey", objectKey, "error", err) - return nil + return nil, &artifactProbeError{reason: "the S3 endpoint is not usable", err: err} } size, err := lister.StatObject(ctx, bucket, objectKey) if err != nil { - logger.Info("cannot stat the Redis snapshot; leaving the artifact unset", "bucket", bucket, "objectKey", objectKey, "error", err) - return nil + return nil, &artifactProbeError{reason: "the snapshot could not be found or read in the bucket", err: err} } return &backupsv1alpha1.BackupArtifact{ URI: fmt.Sprintf("s3://%s/%s", bucket, objectKey), SizeBytes: size, - } + }, nil } +// artifactProbeError carries a fixed reason fit for an Event in the +// application namespace; err may name the platform's endpoint and bucket, +// so it goes to the controller log only. +type artifactProbeError struct { + reason string + err error +} + +func (e *artifactProbeError) Error() string { return e.reason + ": " + e.err.Error() } +func (e *artifactProbeError) Unwrap() error { return e.err } + // projectedS3Connection reads the backup credentials Secret the controller // projects into the application namespace. The projector stores the endpoint // without a scheme, and the strategy scripts reach it over https. It can diff --git a/internal/backupcontroller/redis_artifact_size_test.go b/internal/backupcontroller/redis_artifact_size_test.go index 24050baf1..f733086e6 100644 --- a/internal/backupcontroller/redis_artifact_size_test.go +++ b/internal/backupcontroller/redis_artifact_size_test.go @@ -14,6 +14,7 @@ import ( batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/tools/record" "sigs.k8s.io/controller-runtime/pkg/client" clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake" @@ -56,7 +57,7 @@ func TestRedisArtifact(t *testing.T) { t.Run("stats the snapshot with the projected credentials", func(t *testing.T) { l := &fakeS3Lister{statSize: 2048} r := redisReconciler(t, l, projectedCredsSecret(nil)) - got := r.redisArtifact(context.Background(), "tenant", redisTestKey, nil) + got, _ := r.redisArtifact(context.Background(), "tenant", redisTestKey, nil) if got == nil || got.URI != "s3://cozy-backups/"+redisTestKey || got.SizeBytes != 2048 { t.Fatalf("artifact = %#v", got) } @@ -87,7 +88,7 @@ func TestRedisArtifact(t *testing.T) { if tc.noTarget { r.CredentialsConfig.TargetSecretName = "" } - if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, nil); got != nil { + if got, _ := r.redisArtifact(context.Background(), "tenant", redisTestKey, nil); got != nil { t.Errorf("artifact = %#v, want nil", got) } }) @@ -124,11 +125,11 @@ func TestRedisArtifact_MinioClient(t *testing.T) { r := &BackupJobReconciler{Client: c, Scheme: c.Scheme()} r.CredentialsConfig.TargetSecretName = "cozy-backups-creds" - if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, &corev1.PodSpec{}); got != nil { + if got, _ := r.redisArtifact(context.Background(), "tenant", redisTestKey, &corev1.PodSpec{}); got != nil { t.Errorf("without the Job's trust settings: artifact = %#v, want nil (untrusted certificate)", got) } - got := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("s3-endpoint-ca", "")) + got, _ := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("s3-endpoint-ca", "")) if got == nil || got.SizeBytes != 4096 { t.Fatalf("with the s3-ca volume: artifact = %#v, want SizeBytes 4096", got) } @@ -139,7 +140,7 @@ func TestRedisArtifact_MinioClient(t *testing.T) { t.Errorf("request not signed with the Secret's access key: %q", gotAuth) } - if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("", "true")); got == nil || got.SizeBytes != 4096 { + if got, _ := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("", "true")); got == nil || got.SizeBytes != 4096 { t.Errorf("with S3_INSECURE_SKIP_VERIFY=true: artifact = %#v, want SizeBytes 4096", got) } @@ -147,7 +148,7 @@ func TestRedisArtifact_MinioClient(t *testing.T) { if err := c.Update(context.Background(), ca); err != nil { t.Fatalf("update CA Secret: %v", err) } - if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("s3-endpoint-ca", "true")); got != nil { + if got, _ := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("s3-endpoint-ca", "true")); got != nil { t.Errorf("with a CA that did not sign the endpoint and S3_INSECURE_SKIP_VERIFY=true: artifact = %#v, want nil", got) } } @@ -314,6 +315,40 @@ func TestCreateRedisBackupArtifact_Artifact(t *testing.T) { } }) + t.Run("an unmeasured artifact is reported on the BackupJob without the raw error", func(t *testing.T) { + l := &fakeS3Lister{statErr: errors.New("Get https://s3.internal.example/cozy-backups: NoSuchKey")} + r := redisReconciler(t, l, j, projectedCredsSecret(nil)) + rec := record.NewFakeRecorder(4) + r.Recorder = rec + got, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil) + if err != nil || got == nil || got.Status.Artifact != nil { + t.Fatalf("createRedisBackupArtifact = %#v, %v; want a Backup without an artifact", got, err) + } + select { + case ev := <-rec.Events: + if !strings.HasPrefix(ev, "Normal ArtifactNotMeasured ") || !strings.Contains(ev, "could not be found or read") { + t.Errorf("event = %q, want Normal ArtifactNotMeasured with the reason", ev) + } + if strings.Contains(ev, "s3.internal.example") { + t.Errorf("event = %q leaks the raw S3 error into the application namespace", ev) + } + default: + t.Error("no event on the BackupJob for an artifact left unset") + } + }) + + t.Run("a measured artifact raises no event", func(t *testing.T) { + r := redisReconciler(t, &fakeS3Lister{statSize: 1}, j, projectedCredsSecret(nil)) + rec := record.NewFakeRecorder(4) + r.Recorder = rec + if _, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil); err != nil { + t.Fatalf("createRedisBackupArtifact: %v", err) + } + if len(rec.Events) != 0 { + t.Errorf("event %q for an artifact that was recorded", <-rec.Events) + } + }) + t.Run("an existing Backup is returned without probing", func(t *testing.T) { existing := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-1"}} l := &fakeS3Lister{statSize: 777} @@ -445,7 +480,7 @@ func TestRedisArtifact_HangingEndpointIsGivenUp(t *testing.T) { r := redisReconciler(t, &fakeS3Lister{}, projectedCredsSecret(nil)) r.newS3ObjectLister = func(s3Connection) (s3ObjectLister, error) { return blockingLister{}, nil } withShortProbeTimeout(t, func() { - if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, nil); got != nil { + if got, _ := r.redisArtifact(context.Background(), "tenant", redisTestKey, nil); got != nil { t.Errorf("artifact = %#v, want nil", got) } }) @@ -458,7 +493,7 @@ func TestRedisArtifact_TrustsOnlyTheCA(t *testing.T) { Data: map[string][]byte{"ca.crt": []byte("PEM")}, } r := redisReconciler(t, l, projectedCredsSecret(nil), ca) - if got := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("s3-endpoint-ca", "")); got == nil { + if got, _ := r.redisArtifact(context.Background(), "tenant", redisTestKey, redisPodSpec("s3-endpoint-ca", "")); got == nil { t.Fatal("artifact = nil") } if !l.conn.CAOnly { diff --git a/internal/backupcontroller/redisstrategy_controller.go b/internal/backupcontroller/redisstrategy_controller.go index 76030cb10..8d57d1144 100644 --- a/internal/backupcontroller/redisstrategy_controller.go +++ b/internal/backupcontroller/redisstrategy_controller.go @@ -3,6 +3,7 @@ package backupcontroller import ( "context" + "errors" "fmt" "maps" "strings" @@ -498,7 +499,15 @@ func (r *BackupJobReconciler) createRedisBackupArtifact( }, } if probe { - backup.Status.Artifact = r.redisArtifact(ctx, j.Namespace, objectKey, podSpec) + artifact, err := r.redisArtifact(ctx, j.Namespace, objectKey, podSpec) + if err != nil { + getLogger(ctx).Info("leaving the Redis backup artifact unset", "backupjob", j.Name, "objectKey", objectKey, "error", err) + var probeErr *artifactProbeError + if r.Recorder != nil && errors.As(err, &probeErr) { + r.Recorder.Eventf(j, corev1.EventTypeNormal, "ArtifactNotMeasured", "The backup succeeded, but its location and size were not recorded: %s", probeErr.reason) + } + } + backup.Status.Artifact = artifact } if j.Spec.PlanRef != nil { backup.Spec.PlanRef = j.Spec.PlanRef From 2013e624ca81b3a3cf8b2741d9130ef79ecb3d9d Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Tue, 10 Nov 2026 11:44:33 +0700 Subject: [PATCH 396/518] fix(backups): raise ArtifactNotMeasured only for a Backup this pass created The Event was recorded before the Backup was created, so a Create that then failed left the BackupJob Failed next to an Event saying the backup succeeded, and a Create that collided with a Backup another pass had made reported on that pass's Backup. The Event is now recorded once Create succeeds. It also covers the one unmeasured path that had none: a failed existence check, which skips the probe. A test now also pins that a CA Secret key the s3-ca volume's items do not project is never trusted. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- .../redis_artifact_size_test.go | 51 +++++++++++++++++++ .../redisstrategy_controller.go | 11 +++- 2 files changed, 60 insertions(+), 2 deletions(-) diff --git a/internal/backupcontroller/redis_artifact_size_test.go b/internal/backupcontroller/redis_artifact_size_test.go index f733086e6..ade7cbc9b 100644 --- a/internal/backupcontroller/redis_artifact_size_test.go +++ b/internal/backupcontroller/redis_artifact_size_test.go @@ -186,6 +186,16 @@ func configMapS3CA(spec *corev1.PodSpec) *corev1.PodSpec { return spec } +// nextEvent returns the next recorded event, or "" when there is none. +func nextEvent(rec *record.FakeRecorder) string { + select { + case ev := <-rec.Events: + return ev + default: + return "" + } +} + func TestRedisS3Trust(t *testing.T) { caSecret := func(data map[string]string) *corev1.Secret { s := &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "s3-endpoint-ca"}, Data: map[string][]byte{}} @@ -267,6 +277,14 @@ func TestRedisS3Trust(t *testing.T) { objs: []client.Object{caSecret(map[string]string{"other": "x"})}, }, "an s3-ca volume of another source": {spec: func(skip string) *corev1.PodSpec { return configMapS3CA(redisPodSpec("", skip)) }}, + "a CA Secret whose keys the volume does not project": { + spec: func(skip string) *corev1.PodSpec { + s := redisPodSpec("s3-endpoint-ca", skip) + s.Volumes[0].Secret.Items = []corev1.KeyToPath{{Key: "bundle.pem", Path: "ca.crt"}} + return s + }, + objs: []client.Object{caSecret(map[string]string{"tls.crt": "UNPROJECTED"})}, + }, } for name, tc := range refused { for _, skip := range []string{"", "true"} { @@ -445,6 +463,8 @@ func TestCreateRedisBackupArtifact_RetryCheck(t *testing.T) { l := &fakeS3Lister{statSize: 777} r := redisReconciler(t, l, j, projectedCredsSecret(nil)) r.APIReader = failingReads(t, newMariaDBStrategyTestClient(t)) + rec := record.NewFakeRecorder(4) + r.Recorder = rec got, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil) if err != nil || got == nil { t.Fatalf("createRedisBackupArtifact = %v, %v; a transient read must not fail the backup", got, err) @@ -452,6 +472,37 @@ func TestCreateRedisBackupArtifact_RetryCheck(t *testing.T) { if l.calls != 0 { t.Errorf("S3 probed %d times after the existence check failed", l.calls) } + if ev := nextEvent(rec); !strings.HasPrefix(ev, "Normal ArtifactNotMeasured ") || !strings.Contains(ev, "could not be checked") { + t.Errorf("event = %q, want ArtifactNotMeasured saying the existence check failed", ev) + } + }) + + t.Run("no event when Create fails", func(t *testing.T) { + r := redisReconciler(t, &fakeS3Lister{statErr: errors.New("NoSuchKey")}, j, projectedCredsSecret(nil)) + r.Client = forbidBackupCreate(t, r.Client) + rec := record.NewFakeRecorder(4) + r.Recorder = rec + if _, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil); err == nil { + t.Fatal("createRedisBackupArtifact succeeded although Create was rejected") + } + if ev := nextEvent(rec); ev != "" { + t.Errorf("event %q although the BackupJob fails", ev) + } + }) + + t.Run("no event when Create collides with a Backup another pass made", func(t *testing.T) { + backup := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-1"}} + store := newMariaDBStrategyTestClient(t, j, backup, projectedCredsSecret(nil)) + r := redisReconciler(t, &fakeS3Lister{statErr: errors.New("NoSuchKey")}) + r.Client, r.APIReader = staleCache(t, store), notFoundOnce(t, store) + rec := record.NewFakeRecorder(4) + r.Recorder = rec + if _, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil); err != nil { + t.Fatalf("createRedisBackupArtifact: %v", err) + } + if ev := nextEvent(rec); ev != "" { + t.Errorf("event %q about a Backup this pass did not create", ev) + } }) t.Run("a Create collision is resolved from the API server past the cache", func(t *testing.T) { diff --git a/internal/backupcontroller/redisstrategy_controller.go b/internal/backupcontroller/redisstrategy_controller.go index 8d57d1144..036d1218a 100644 --- a/internal/backupcontroller/redisstrategy_controller.go +++ b/internal/backupcontroller/redisstrategy_controller.go @@ -498,13 +498,15 @@ func (r *BackupJobReconciler) createRedisBackupArtifact( Phase: backupsv1alpha1.BackupPhaseReady, }, } + unmeasured := "whether the Backup already exists could not be checked" if probe { artifact, err := r.redisArtifact(ctx, j.Namespace, objectKey, podSpec) + unmeasured = "" if err != nil { getLogger(ctx).Info("leaving the Redis backup artifact unset", "backupjob", j.Name, "objectKey", objectKey, "error", err) var probeErr *artifactProbeError - if r.Recorder != nil && errors.As(err, &probeErr) { - r.Recorder.Eventf(j, corev1.EventTypeNormal, "ArtifactNotMeasured", "The backup succeeded, but its location and size were not recorded: %s", probeErr.reason) + if errors.As(err, &probeErr) { + unmeasured = probeErr.reason } } backup.Status.Artifact = artifact @@ -522,6 +524,11 @@ func (r *BackupJobReconciler) createRedisBackupArtifact( } return existing, nil } + // Only once the Backup exists: a terminal Create error fails the + // BackupJob, and a collision returns a Backup another pass measured. + if unmeasured != "" && r.Recorder != nil { + r.Recorder.Eventf(j, corev1.EventTypeNormal, "ArtifactNotMeasured", "The backup succeeded, but its location and size were not recorded: %s", unmeasured) + } return backup, nil } From 1093f390aee123cde1d146a2d004f20441178bec Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Tue, 10 Nov 2026 12:18:07 +0700 Subject: [PATCH 397/518] feat(backups): warn on the BackupJob when the Redis snapshot is empty A valid RDB snapshot always carries a header, so a 0-byte object means the backup is broken. sizeBytes is omitempty, though, so such a Backup read exactly like one whose size was never measured. The controller still records the artifact, and now raises a Warning EmptySnapshot Event on the BackupJob once the Backup is created. A test now also pins that a retry over an existing Backup returns it without calling Create. Assisted-by: LLM Signed-off-by: Andrey Kolkov --- .../redis_artifact_size_test.go | 26 ++++++++++++++++++- .../redisstrategy_controller.go | 11 ++++++-- 2 files changed, 34 insertions(+), 3 deletions(-) diff --git a/internal/backupcontroller/redis_artifact_size_test.go b/internal/backupcontroller/redis_artifact_size_test.go index ade7cbc9b..47d357c4e 100644 --- a/internal/backupcontroller/redis_artifact_size_test.go +++ b/internal/backupcontroller/redis_artifact_size_test.go @@ -17,6 +17,7 @@ import ( "k8s.io/client-go/tools/record" "sigs.k8s.io/controller-runtime/pkg/client" clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" ) @@ -367,16 +368,39 @@ func TestCreateRedisBackupArtifact_Artifact(t *testing.T) { } }) - t.Run("an existing Backup is returned without probing", func(t *testing.T) { + t.Run("an existing Backup is returned without probing or creating", func(t *testing.T) { existing := &backupsv1alpha1.Backup{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-1"}} l := &fakeS3Lister{statSize: 777} r := redisReconciler(t, l, j, existing, projectedCredsSecret(nil)) + creates := 0 + r.Client = interceptor.NewClient(r.Client.(client.WithWatch), interceptor.Funcs{ + Create: func(ctx context.Context, cl client.WithWatch, obj client.Object, opts ...client.CreateOption) error { + creates++ + return cl.Create(ctx, obj, opts...) + }, + }) if _, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil); err != nil { t.Fatalf("createRedisBackupArtifact: %v", err) } if l.calls != 0 { t.Errorf("S3 probed %d times for a Backup that already exists", l.calls) } + if creates != 0 { + t.Errorf("Create called %d times for a Backup that already exists", creates) + } + }) + + t.Run("an empty snapshot is recorded and warned about", func(t *testing.T) { + r := redisReconciler(t, &fakeS3Lister{statSize: 0}, j, projectedCredsSecret(nil)) + rec := record.NewFakeRecorder(4) + r.Recorder = rec + got, err := r.createRedisBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, redisTestKey, nil) + if err != nil || got == nil || got.Status.Artifact == nil || got.Status.Artifact.URI != "s3://cozy-backups/"+redisTestKey { + t.Fatalf("createRedisBackupArtifact = %#v, %v; want the artifact recorded", got, err) + } + if ev := nextEvent(rec); !strings.HasPrefix(ev, "Warning EmptySnapshot ") { + t.Errorf("event = %q, want Warning EmptySnapshot", ev) + } }) } diff --git a/internal/backupcontroller/redisstrategy_controller.go b/internal/backupcontroller/redisstrategy_controller.go index 036d1218a..4169e6cef 100644 --- a/internal/backupcontroller/redisstrategy_controller.go +++ b/internal/backupcontroller/redisstrategy_controller.go @@ -526,8 +526,15 @@ func (r *BackupJobReconciler) createRedisBackupArtifact( } // Only once the Backup exists: a terminal Create error fails the // BackupJob, and a collision returns a Backup another pass measured. - if unmeasured != "" && r.Recorder != nil { - r.Recorder.Eventf(j, corev1.EventTypeNormal, "ArtifactNotMeasured", "The backup succeeded, but its location and size were not recorded: %s", unmeasured) + if r.Recorder != nil { + switch { + case unmeasured != "": + r.Recorder.Eventf(j, corev1.EventTypeNormal, "ArtifactNotMeasured", "The backup succeeded, but its location and size were not recorded: %s", unmeasured) + case backup.Status.Artifact != nil && backup.Status.Artifact.SizeBytes == 0: + // sizeBytes is omitempty, so a 0 reads as "not measured" on the + // Backup; a valid RDB snapshot always carries a header. + r.Recorder.Event(j, corev1.EventTypeWarning, "EmptySnapshot", "The backup's RDB snapshot is empty (0 bytes); a valid snapshot is never empty, so this backup is likely broken.") + } } return backup, nil } From 5a997fb7d435e0498c82d02ae895257dc0ad375f Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 9 Nov 2026 15:19:04 +0000 Subject: [PATCH 398/518] feat(release): post an immutable report of every classification and write record version 2 Each run rewrote the batch body whole, so the only account of what a run found was gone at the next run, and a merged batch kept only its last one. Nothing said what a classification was made of either: not the heads it read, the policy, the tool or the flags. A run now makes a report: every original merged into main since the line branched, in exactly one category, with what the run read. Its ID hashes the classification and names no pull request, so a new batch's report is whole before GitHub numbers the pull request, and a run that finds the same as the last one makes the same report, which is posted once and keeps its first bytes. The body carries the latest report under the record; a comment the tool never edits keeps each one, so report reads any of them back, from merged batches too. The comment goes up before the push and the body edit on an open batch, and after the pull request a new batch opens with the report in its body; a later run posts a report the body kept but no comment did, byte for byte, before its own. Every write is preceded by a read of the pull request, and nothing is written once it is merged or closed. Both the comment and the body are rendered and measured before the first write, so an oversized one stops the run with nothing written. The record becomes version 2: what it was built from, with the run's --no-workflow-files, which --defer keeps rather than resets, and the report it goes with. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 21 +- cmd/backport-batch/assemble.go | 153 +++++- cmd/backport-batch/body.go | 32 +- cmd/backport-batch/body_test.go | 20 +- cmd/backport-batch/defer.go | 7 + cmd/backport-batch/defer_test.go | 4 +- cmd/backport-batch/fixture_test.go | 105 +++- cmd/backport-batch/github.go | 35 +- cmd/backport-batch/main.go | 46 +- cmd/backport-batch/notify.go | 5 +- cmd/backport-batch/notify_test.go | 21 + cmd/backport-batch/overrides_test.go | 16 +- cmd/backport-batch/record.go | 21 + cmd/backport-batch/record_test.go | 74 +++ cmd/backport-batch/report.go | 734 ++++++++++++++++++++++++++ cmd/backport-batch/report_test.go | 687 ++++++++++++++++++++++++ cmd/backport-batch/workflows_test.go | 3 +- docs/release/backport-tasks/README.md | 4 +- docs/release/backports.md | 8 +- 19 files changed, 1898 insertions(+), 98 deletions(-) create mode 100644 cmd/backport-batch/report.go create mode 100644 cmd/backport-batch/report_test.go diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index 945d6fac1..7a1acabf1 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -13,12 +13,13 @@ go build ./cmd/backport-batch ./backport-batch fold --line release-1.7 --original 4321 --commit SHA [--policy REF] ./backport-batch verify --line release-1.7 [--head SHA] ./backport-batch notify --line release-1.7 [--mention org/team|login] [--push] +./backport-batch report --pr 5246 [--id ID] [--json] ./backport-batch lines [--policy REF] ``` [`hack/backport-batch-scheduled.sh`](../../hack/backport-batch-scheduled.sh) runs `lines`, then `run --continue --push`, `check` and `notify` for each line; [`backport-batch.yaml`](../../.github/workflows/backport-batch.yaml) runs it on schedule as the cozystack-ci App, with `--no-workflow-files`, which it hands to `run`. -Requires `git` and an authenticated `gh` on PATH, and a remote called `origin`. Every command but `notify` fetches `origin` first; `notify` reads only GitHub. The batch runs only for the lines `lines` prints. +Requires `git` and an authenticated `gh` on PATH, and a remote called `origin`. Every command but `notify` and `report` fetches `origin` first; those two read only GitHub. The batch runs only for the lines `lines` prints. `run`, `check`, `fold` and `lines` read the `backport-policy` block of the policy page from `origin/main`, or from the ref `--policy` names, which is how a change to the page is tried before it merges; `verify` and `notify` do not read it and take no `--policy`. A line the block lists takes nothing merged into `main` before the day it gives, unless the original carries `backport/include`; any other line starts at its fork point. A page without exactly one readable block stops the run with exit 2. @@ -28,7 +29,7 @@ Build the binary wherever an exit code matters: `go run` turns any non-zero exit Switches the current working tree to `backport-batch/-`, created at `origin/`, and picks onto it one original at a time, in the order they merged into `main`, as one unit each on the branch's first-parent line. A pull request merged into `main` with a merge commit `M` goes on as its own commits, `git rev-list --reverse --topo-order --no-merges M^1..M^2`, each picked with `git cherry-pick -x --signoff`, so each keeps its author, message and sign-offs and gains the `(cherry picked from commit …)` line; the merges that brought `main` into its branch are left out, since `main` already has what they brought. A merge commit then joins the picks to the batch: its first parent is the batch before the original, its second the last pick, and its message is `M`'s with a paragraph of its own, `Backported-from: M`. A pull request squash-merged into `main` goes on as one pick. When any commit of an original conflicts, the pick is aborted, the branch goes back to where it was before that original, and the original is recorded with the commit that failed and its files; the run goes on with the next one. A commit the branch already carries is skipped as empty, and an original none of whose commits changes anything is reported as carried. The tree has to be clean. A local branch of that name is reset to where the run starts; when that drops commits only it has, the run first prints its tip and the `git branch` command that restores them, and so does `--defer`. Signing follows your git configuration; hooks are off for the picks and the merge, so each commit carries its message as git writes it. -The pull request body goes to stdout, built from the line's own `.github/PULL_REQUEST_TEMPLATE.md`: the originals on the branch, each with how many commits it brought and which of them differ from the commit they were picked from by `git patch-id`, a resolution or a pick git had to fit, the deferred conflicts with the commit that failed and its files, the skipped pull requests counted by rule with the lists collapsed, the originals' release notes, and a `backport-batch` block of JSON that lists the deferred conflicts for `check`. The title is `fix(backport): batch for , `. A body that would pass GitHub's 65536-byte limit leaves out its longest skipped lists, and the run prints them on stderr instead. +The pull request body goes to stdout, built from the line's own `.github/PULL_REQUEST_TEMPLATE.md`: the originals on the branch, each with how many commits it brought and which of them differ from the commit they were picked from by `git patch-id`, a resolution or a pick git had to fit, the deferred conflicts with the commit that failed and its files, the skipped pull requests counted by rule with the lists collapsed, the originals' release notes, and two blocks of JSON: the [line's record](#the-lines-record-and-reports) in `backport-batch`, which later runs, `check` and `notify` act on, and the run's report in `backport-batch-report`. The title is `fix(backport): batch for , `. A body that would pass GitHub's 65536-byte limit leaves out its longest skipped lists, and the run prints them on stderr instead; one that what it has to carry still puts over the limit is refused before anything is pushed. `--onto SHA` continues a batch already pushed instead of starting a new one. SHA has to be the head the remote has for a batch branch of the line right now, read with `git ls-remote`, so a head someone pushed onto since is refused. What that branch carries stays, and only what it does not carry yet is picked on top, so the next push is a fast-forward. `--continue` takes SHA from the open batch pull request, and starts a new batch when none is open, which is what the scheduled run does. @@ -36,11 +37,11 @@ The pull request body goes to stdout, built from the line's own `.github/PULL_RE `--defer N[,N] --reason TEXT` takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off `origin/` without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got `backport/skip` after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the `backport-batch` block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. -`--push` pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with `gh pr create --draft`, or updates the open batch pull request with `gh pr edit`, and prints its URL on stderr. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the `run --onto --push` that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no `backport-batch` block either: someone wrote that body by hand, and the batch is theirs to merge or close. After `--defer` alone it replaces the branch, with `--force-with-lease=refs/heads/:`, where head is what `git ls-remote` reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. +`--push` pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with `gh pr create --draft`, or updates the open batch pull request with `gh pr edit`, and prints its URL on stderr. On the open batch it first posts the run's report as a comment, so a body never names a report nobody can read; a new batch carries the report in the body it opens with, and the comment follows. Before every write to the pull request it reads the pull request again, and stops with exit 2 once it is merged or closed. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the `run --onto --push` that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no `backport-batch` block either: someone wrote that body by hand, and the batch is theirs to merge or close. After `--defer` alone it replaces the branch, with `--force-with-lease=refs/heads/:`, where head is what `git ls-remote` reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. ### check -Exits 1 and prints the URL of every pull request the rules take that is neither on the line nor on the open batch pull request, 0 when there is none, and 2 when it cannot tell. A pull request that the [line's record](#the-lines-record) lists as deferred, a conflict or one taken out with `--defer`, prints as a warning and does not fail the check: deferring is a maintainer's call. That holds once the batch that recorded it merged or closed too. So does one the open batch lists as waiting for a maintainer because it changes workflows; once that batch is gone, nothing lists it any more, and it is missing. +Exits 1 and prints the URL of every pull request the rules take that is neither on the line nor on the open batch pull request, 0 when there is none, and 2 when it cannot tell. A pull request that the [line's record](#the-lines-record-and-reports) lists as deferred, a conflict or one taken out with `--defer`, prints as a warning and does not fail the check: deferring is a maintainer's call. That holds once the batch that recorded it merged or closed too. So does one the open batch lists as waiting for a maintainer because it changes workflows; once that batch is gone, nothing lists it any more, and it is missing. `--cut` makes it the gate before a patch release or an rc is cut from a line. A cut tags the line, so a pull request that is only on the open batch pull request fails it too, with one line each: `#N is only on open batch PR #M, merge it before the cut`. One that waits there for a maintainer fails it as well, since nobody decided to leave it out, also when it is all the batch has left: bring it over, or take it out with `run --defer` and a reason. Everything else is as without it, and a cut that passes says `no change the policy takes is missing before the cut`. The scheduled run checks without `--cut`, since its batch is open by design. @@ -56,12 +57,16 @@ Holds every unit of a batch to the shape fold checks for one, the clean picks as ### notify -Reads the [line's record](#the-lines-record) first, as the other commands do, and exits 2 when it cannot read it or when it says the open batch is being closed, with a batch open or not. Then it reads the open batch pull request of the line, trusted as described below, and prints one line for each thing that waits on it: a batch open for three days or more, the shortest gap between two scheduled runs, so one the previous run already saw; the conflicts it deferred; and the originals waiting for a maintainer because they change workflows. Nothing open prints nothing. +Reads the [line's record](#the-lines-record-and-reports) first, as the other commands do, and exits 2 when it cannot read it or when it says the open batch is being closed, with a batch open or not. Then it reads the open batch pull request of the line, trusted as described below, and prints one line for each thing that waits on it: a batch open for three days or more, the shortest gap between two scheduled runs, so one the previous run already saw; the conflicts it deferred; and the originals waiting for a maintainer because they change workflows. Nothing open prints nothing. With `--push` it writes the same into one comment on the batch pull request: the conflicts, each to be resolved as the [conflict task](../../docs/release/backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows. The comment starts with `` and is one per pull request: a later run edits it in place rather than adding another, and once nothing waits it says so. Only a comment whose author the batch trusts counts as it, so one someone else wrote with the same first line is neither edited nor believed. It mentions `--mention`, an `org/team` or a login, with or without `@`, only while something waits. A failed batch CI is not among what it reports: the run has no trigger on the batch's CI finishing. +### report + +Prints a report a batch pull request keeps: the one its body carries, or the one `--id` names. It reads the report comments the batch trusts and falls back to the body, recomputes the report's ID from what it says, and exits 2 when a trusted comment does not match its ID, when a report says anything a run does not write or is for another line than the pull request's, or when the pull request keeps no such report. A report comment counts only when its author is one the batch trusts with a record, so anyone else's comment, whatever ID it carries, is not read. It prints what the classification was made of, whether the pull request's head or body shows the report was published, which it calls unconfirmed when neither does, and one line per original with its category and what the report recorded about it; `--json` prints the report as stored. A report whose comment the run never got to post is printed from the body, with a note that the next `run --push` posts it, or, on a merged or closed batch, that it is kept in the body only. It reads GitHub alone and writes nothing. + ### lines Prints the release lines the batch runs for, newest first: of the `release-X.Y` branches on `origin`, the newest that have a stable `vX.Y.Z` tag there, as many as `released-lines` in the policy block says, two when it says nothing, and every branch newer than the newest of them. The tags are read with `git ls-remote`; a tag with a pre-release suffix does not count. `run`, `check` and `fold` refuse any other line, and rule 6 holds each line to the next newer one among them. @@ -96,7 +101,7 @@ Only the line's own commits count, the ones after the point where it branched fr Titles and bodies are read from the merge commits on `main`, which hold them as a maintainer merged them; the live pull request stays editable by its author. -## The line's record +## The line's record and reports The line's record is the `backport-batch` block of its newest batch pull request: the newest pull request against the line from a `backport-batch/-` branch of this repository, opened by an author the batch trusts as above, whether it is open, merged or closed. `run`, `check`, `fold` and `notify` find it in GitHub's list of the line's pull requests, newest first, reading at most ten pages of 100. A batch pull request a person opened without the block is passed over. A block whose author the batch does not trust, a bot's batch without its block, a block that cannot be read whole, a closed record newer than the open batch, or ten pages with no record stop the command with exit 2, naming the pull request, rather than fall back to an older record. @@ -104,10 +109,12 @@ The record is written only while its pull request is open: a merged or closed ba The record is only as trustworthy as the body it sits in. Anyone with write access can edit any body, the bot's included, and neither the bot's authorship nor anything in the block proves who edited it last. A hand edit that leaves a readable record is believed; one that breaks it stops every command with exit 2, naming the pull request. When the newest batch pull request is deleted and no later one exists, nothing shows it was there: the older record governs, and what was recorded since is lost. +The record holds only what later commands act on. Every run that classifies the line also makes a report, one line of JSON in a `backport-batch-report` block: every original merged into `main` since the line branched in exactly one category, picked, missing, deferred with what it was deferred for, or skipped with the rule that skipped it, the commits on `main` that merged no pull request, the labels, and what the classification was made of: the heads of `main`, the line and the newer line, the policy, the batch head it describes, the tool's commit, the Actions run and the flags. Its ID is a hash of all that but the time and the run, so two runs that find the same make the same report, and it names no pull request, so a new batch's report is whole before GitHub numbers the pull request. The body carries the latest report beside the record, and a comment on the open batch pull request keeps each one: `` and the same block, folded away. A report already posted is not posted again, and the tool never edits or deletes such a comment, so a report the next run replaces in the body is still read back by `report`, also after the batch merged. A comment that looks like one but whose author the batch does not trust is not read. When a run stops after writing the body but before posting its comment, the next `run --push` on the open batch posts that report first, byte for byte; once the batch is closed nothing is written to it, and the report stays in its body. A run that picks nothing while no batch is open keeps no report. + ## What CI verifies The tool does not build or test anything. CI on the batch pull request runs the line's own build and tests against what was picked, and `verify` against its shape, and a maintainer reviews and merges it. A deferred conflict waits for the next batch unless someone resolves it, checks the resolution with `fold` and pushes it, and then CI runs again. ## Untrusted text -Titles, release notes and file names in the body come from whoever opened the originals. A title, a reason or a file name goes into a code span whose fence no backtick inside can close, and the release notes go into the `release-note` block with every run of three backticks or tildes defused, so GitHub acts on no mention, reference, link or HTML in them. Inside that block, in case a note is copied out of it, links lose any target outside this repository, and closing keywords, cross-repository references and mentions have their `@`, `#` and `://` swapped for fullwidth signs, which act on nothing however the backticks around them pair up. Zero-width, bidi and control characters are dropped first, and everything the tool prints passes the same filter, so nothing it relays can drive a terminal. Commit messages are read through `git cat-file --batch`, which gives each object's length, and end at their first NUL as `git log` shows them, so a message cannot pass for a second commit or hide a trailer from a reader of the log. Every value handed to git comes after `--end-of-options` or `--`, refs pass `git check-ref-format`, SHAs have to be 40 hex digits, and nothing runs through a shell. +Titles, release notes and file names in the body come from whoever opened the originals, and a reason from whoever ran `--defer`. A title, a reason or a file name goes into a code span whose fence no backtick inside can close, and the release notes go into the `release-note` block with every run of three backticks or tildes defused, so GitHub acts on no mention, reference, link or HTML in them. Inside that block, in case a note is copied out of it, links lose any target outside this repository, and closing keywords, cross-repository references and mentions have their `@`, `#` and `://` swapped for fullwidth signs, which act on nothing however the backticks around them pair up. The record and the report are one line of JSON each, whose encoding leaves no newline, `<`, `>` or `&` in them, so nothing in a file name or a reason ends their block or opens HTML, and a report comment names no one: numbers are bare, and its only text outside the block is the line, the date and the report's ID. Zero-width, bidi and control characters are dropped first, and everything the tool prints passes the same filter, so nothing it relays can drive a terminal. Commit messages are read through `git cat-file --batch`, which gives each object's length, and end at their first NUL as `git log` shows them, so a message cannot pass for a second commit or hide a trailer from a reader of the log. Every value handed to git comes after `--end-of-options` or `--`, refs pass `git check-ref-format`, SHAs have to be 40 hex digits, and nothing runs through a shell. diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index ad6de0a1b..a2adc5c1d 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -158,7 +158,9 @@ func (a *app) run(args []string) int { } r := newReport(b) + r.opts = runOptions{NoWorkflowFiles: *noWorkflows, Mode: modeNew} if carried != nil { + r.opts.Mode = modeContinue if r.stale, err = keptOut(b, carried); err != nil { return a.fail(err) } @@ -264,12 +266,25 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r if r.shapes, err = summarize(b, head, r.onBranch); err != nil { return a.fail(err) } - title := prTitle(b.line, date.Format(time.DateOnly)) + r.built = builtFrom{Main: b.mainHead, Line: b.lineHead, Newer: b.newerHead, Policy: b.policyAt, Batch: head, Tool: a.tool, + Run: runURL(os.LookupEnv), At: a.now().UTC().Format(time.RFC3339), Options: r.opts} + doc, err := buildReport(b, r) + if err != nil { + return a.fail(err) + } + p := publication{branch: branch, head: head, title: prTitle(b.line, date.Format(time.DateOnly)), report: storedReport{doc: doc, raw: reportJSON(doc)}} + p.record = r.lineRecord(doc) + // A record the next command would refuse is never written. + if err := p.record.validate(b.line); err != nil { + return a.fail(fmt.Errorf("the record this run made %w", err)) + } + r.blocks = recordBlock(p.record) + "\n\n" + reportBlock(p.report.raw) template, err := lineTemplate(b.lineHead) if err != nil { return a.fail(err) } body, left := r.body(template) + p.body = body fmt.Fprint(a.stdout, body) skipped := 0 for _, items := range r.skipped { @@ -283,8 +298,8 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r for _, p := range r.waiting { fmt.Fprintf(a.stderr, "waiting for a maintainer: %s %s changes %s\n", p.v.o.url(), terminalSafe(p.v.o.title), terminalSafe(strings.Join(p.files, ", "))) } - fmt.Fprintf(a.stderr, "%s at %.12s: %d on the branch, %d deferred, %d waiting for a maintainer, %d skipped\ntitle: %s\n", - branch, head, len(r.onBranch), len(r.deferred)+len(r.held), len(r.waiting), skipped, title) + fmt.Fprintf(a.stderr, "%s at %.12s: %d on the branch, %d deferred, %d waiting for a maintainer, %d skipped\ntitle: %s\nreport: %s\n", + branch, head, len(r.onBranch), len(r.deferred)+len(r.held), len(r.waiting), skipped, p.title, doc.ID) if !push { return 0 } @@ -296,12 +311,19 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r fmt.Fprintf(a.stderr, "no batch pull request is open on %s; this run's classification is not retained\n", b.line.branch()) return 0 } - if err := a.publish(b, branch, head, title, body, rebuilt); err != nil { + if err := a.publish(b, p, rebuilt); err != nil { return a.fail(err) } return 0 } +// lineRecord is the record a run leaves for the line: what later commands +// act on, with the full classification left to the report it names. +func (r *report) lineRecord(doc reportDoc) lineRecord { + built := doc.Built + return lineRecord{Version: 2, Line: doc.Line, Built: &built, Report: doc.ID, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: []droppedEntry{}, Stale: slices.Clone(doc.Overlays.Stale)} +} + // branchDate is the date a batch branch was opened under. func branchDate(branch string) (time.Time, error) { m := batchBranchRE.FindStringSubmatch(branch) @@ -446,51 +468,64 @@ func lineTemplate(lineHead string) (string, error) { return git("show", "--end-of-options", lineHead+":"+template) } +// publication is what a run publishes: the branch at head, and the body that +// carries the record and the report. +type publication struct { + branch, head, title, body string + report storedReport + record lineRecord +} + // publish pushes the batch branch, then opens the draft batch pull request or // updates the open one. A run pushes only a fast-forward of what the remote // has. A rebuild replaces the branch, and only the head it was rebuilt from: // the lease is that head, as ls-remote reads it now, so a commit pushed since // is never dropped. -func (a *app) publish(b *batch, branch, head, title, body string, rebuilt bool) error { - if b.pr != nil && b.pr.Head.Ref != branch { +// +// The report goes up as a comment before the push and the body that names +// it, so no write leaves a body naming a report nobody can read; a new batch +// carries the report in the body it is opened with, and the comment follows. +// Everything is rendered and measured before the first write. +func (a *app) publish(b *batch, p publication, rebuilt bool) error { + if b.pr != nil && b.pr.Head.Ref != p.branch { return fmt.Errorf("the open batch pull request #%d is on %s; continue it with --onto %s", b.pr.Number, b.pr.Head.Ref, b.prHead) } // A body without the block was written by hand, and the tool's would // replace it whole, title included; such a batch is the maintainer's. - if b.pr != nil && !strings.Contains("\n"+strings.ReplaceAll(b.pr.Body, "\r\n", "\n")+"\n", "\n"+deferredFence+"\n") { + if b.pr != nil && !b.src.open { return fmt.Errorf("batch pull request #%d has no %s block, so backport-batch did not write it; merge or close it, and the next run opens a batch of its own", b.pr.Number, deferredFence) } - if head == b.lineHead { + if p.head == b.lineHead { if rebuilt { - return fmt.Errorf("without what is deferred, %s carries nothing beyond %s; close batch pull request #%d instead", branch, b.line.branch(), b.pr.Number) + return fmt.Errorf("without what is deferred, %s carries nothing beyond %s; close batch pull request #%d instead", p.branch, b.line.branch(), b.pr.Number) } - return fmt.Errorf("%s carries nothing beyond %s; nothing to push", branch, b.line.branch()) + return fmt.Errorf("%s carries nothing beyond %s; nothing to push", p.branch, b.line.branch()) } heads, err := remoteBatchHeads(b.line) if err != nil { return err } - pushed, onRemote := heads[branch] + pushed, onRemote := heads[p.branch] lease := "" switch { case rebuilt: if !onRemote || pushed != b.prHead { - return fmt.Errorf("%s on %s is at %.12s, not at %.12s, which the rebuild started from; run again", branch, remote, pushed, b.prHead) + return fmt.Errorf("%s on %s is at %.12s, not at %.12s, which the rebuild started from; run again", p.branch, remote, pushed, b.prHead) } lease = pushed - case onRemote && pushed != head: + case onRemote && pushed != p.head: have, err := lookup(pushed) if err != nil { return err } descends := false if have != "" { - if descends, err = isAncestor(pushed, head); err != nil { + if descends, err = isAncestor(pushed, p.head); err != nil { return err } } if !descends { - refusal := fmt.Sprintf("%s on %s is at %.12s, which %.12s does not descend from; refusing a non-fast-forward push", branch, remote, pushed, head) + refusal := fmt.Sprintf("%s on %s is at %.12s, which %.12s does not descend from; refusing a non-fast-forward push", p.branch, remote, pushed, p.head) // A run that stopped between its push and opening the pull request // leaves the branch, and the next run that day builds it anew. if b.pr == nil { @@ -499,38 +534,100 @@ func (a *app) publish(b *batch, branch, head, title, body string, rebuilt bool) return errors.New(refusal) } } - if pushed != head { - if _, err := git(pushArgs(branch, head, lease)...); err != nil { + if err := p.fits(); err != nil { + return err + } + if b.pr != nil { + if err := a.keepReports(b, &p); err != nil { return err } } - f, err := os.CreateTemp("", "backport-batch-body-*.md") + if pushed != p.head { + if _, err := git(pushArgs(p.branch, p.head, lease)...); err != nil { + return err + } + } + name, err := writeTemp("backport-batch-body-*.md", p.body) if err != nil { return err } - defer os.Remove(f.Name()) - if _, err := f.WriteString(body); err != nil { - f.Close() - return err - } - if err := f.Close(); err != nil { - return err - } + defer os.Remove(name) if b.pr == nil { - url, err := a.gh.createPull(b.line.branch(), branch, title, f.Name()) + n, url, err := a.gh.createPull(b.line.branch(), p.branch, p.title, name) if err != nil { return err } fmt.Fprintf(a.stderr, "opened draft %s\n", url) + if err := a.postReport(n, p.report); err != nil { + return fmt.Errorf("posting report %.12s on #%d: %w; the body keeps it, and the next run --push posts it", p.report.doc.ID, n, err) + } return nil } - if err := a.gh.editPull(b.pr.Number, title, f.Name()); err != nil { + if err := a.stillOpen(b.pr.Number); err != nil { + return err + } + if err := a.gh.editPull(b.pr.Number, p.title, name); err != nil { return err } fmt.Fprintf(a.stderr, "updated %s\n", pullURL(b.pr)) return nil } +// fits says why GitHub would not take the comment or the body. +func (p *publication) fits() error { + if _, err := reportComment(p.report); err != nil { + return err + } + if len(p.body) > bodyLimit { + return fmt.Errorf("the body is %d bytes with only what it cannot leave out, over GitHub's %d; nothing is published", len(p.body), bodyLimit) + } + return nil +} + +// keepReports makes sure the open batch's comments keep both the report its +// body carries now and the one this run made, in that order, each once. +// When this run's report is one already kept, the body carries the kept +// bytes, which differ only in when and in which job it was made. +func (a *app) keepReports(b *batch, p *publication) error { + pr := b.pr + posted, bad, err := findReports(a.gh, pr.Number, b.line) + if err != nil { + return err + } + for _, why := range bad { + a.warn("batch pull request #%d: %s; it does not count as a report", pr.Number, why) + } + earlier, err := bodyReport(pr.Body, b.line) + if err != nil { + return fmt.Errorf("batch pull request #%d: %w", pr.Number, err) + } + kept, ok := posted[p.report.doc.ID] + if !ok && earlier != nil && earlier.doc.ID == p.report.doc.ID { + kept, ok = *earlier, true + } + if ok { + p.body = strings.Replace(p.body, reportBlock(p.report.raw), reportBlock(kept.raw), 1) + p.report = kept + if err := p.fits(); err != nil { + return err + } + } + if earlier != nil { + if _, ok := posted[earlier.doc.ID]; !ok { + if err := a.postReport(pr.Number, *earlier); err != nil { + return fmt.Errorf("posting the earlier report %.12s on #%d: %w", earlier.doc.ID, pr.Number, err) + } + posted[earlier.doc.ID] = *earlier + } + } + if _, ok := posted[p.report.doc.ID]; !ok { + if err := a.postReport(pr.Number, p.report); err != nil { + return fmt.Errorf("posting report %.12s on #%d: %w", p.report.doc.ID, pr.Number, err) + } + } + return nil +} + // pullURL is where a pull request is on GitHub. func pullURL(pr *pullRequest) string { if pullURLRE.MatchString(pr.HTMLURL) { diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index bfb908d2d..ae34be876 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -190,25 +190,11 @@ type deferredRecord struct { Deferred []deferredEntry `json:"deferred"` } -// deferredBlock is the machine-readable list of what the batch defers. One -// line of JSON: encoding/json escapes newlines and <, > and &, so nothing in a -// file name or a reason can end the fence or open HTML. It leaves bidi and -// zero-width characters as they are, so file names go through clean first, -// as the reason does. +// deferredBlock is a version 1 record, which batches carried before records +// had a version. encoding/json leaves bidi and zero-width characters as they +// are, so file names go through clean first, as the reason does. func deferredBlock(l line, conflicts, held, waiting []pick) string { - rec := deferredRecord{Line: l.branch(), Deferred: []deferredEntry{}} - for _, p := range conflicts { - files := cleanFiles(p.files[:min(len(p.files), filesKept)]) - rec.Deferred = append(rec.Deferred, deferredEntry{Original: p.v.o.number, Commit: p.failed, Files: files, More: len(p.files) - len(files) + p.more}) - } - for _, p := range held { - rec.Deferred = append(rec.Deferred, deferredEntry{Original: p.v.o.number, Files: []string{}, Reason: capped(oneLine(clean(p.reason, false)), titleLimit)}) - } - for _, p := range waiting { - files := cleanFiles(p.files[:min(len(p.files), filesKept)]) - rec.Deferred = append(rec.Deferred, deferredEntry{Original: p.v.o.number, Files: files, More: len(p.files) - len(files) + p.more, Workflows: true}) - } - data, _ := json.Marshal(rec) + data, _ := json.Marshal(deferredRecord{Line: l.branch(), Deferred: deferredEntries(conflicts, held, waiting)}) return deferredFence + "\n" + string(data) + "\n```" } @@ -254,6 +240,14 @@ type report struct { start time.Time // shapes says, for an original on the branch, what it brought. shapes map[*verdict]string + // missing are originals the policy takes that the run neither tried nor + // found anywhere. + missing []*verdict + // opts is how the run was asked to classify, and built what it read. + opts runOptions + built builtFrom + // blocks are the record and the report the body ends with. + blocks string } // ruleName is how output names what kept an original out: the policy's @@ -368,7 +362,7 @@ func (r *report) what(details map[int]bool) string { b.WriteString("\n
\n") } } - b.WriteString("\n" + deferredBlock(r.line, r.deferred, r.held, r.waiting) + "\n") + b.WriteString("\n" + r.blocks + "\n") return b.String() } diff --git a/cmd/backport-batch/body_test.go b/cmd/backport-batch/body_test.go index 07f2f8582..3f746d17d 100644 --- a/cmd/backport-batch/body_test.go +++ b/cmd/backport-batch/body_test.go @@ -286,7 +286,7 @@ func TestBodyFollowsTheTemplate(t *testing.T) { l, _ := parseLine("release-1.6") template := "\n\n## What this PR does\n\n\n### Screenshots\n\n\n\n### Downstream repositories\n\n- [ ] No downstream repository is affected by this change\n\n### Release note\n\n```release-note\n\n```\n" o := &original{number: 7, title: "fix(x): y", body: "```release-note\nfix(x): note\n```"} - r := &report{line: l, onBranch: []*verdict{{o: o}}, skipped: map[int][]*verdict{}} + r := &report{line: l, onBranch: []*verdict{{o: o}}, skipped: map[int][]*verdict{}, blocks: deferredBlock(l, nil, nil, nil)} body, _ := r.body(template) for _, want := range []string{"## What this PR does\n\nBackports to", "- [#7](https://github.com/cozystack/cozystack/pull/7) `fix(x): y`", "### Downstream repositories\n\n- [ ] No downstream", "```release-note\nfix(x): note\n```", deferredFence} { if !strings.Contains(body, want) { @@ -305,7 +305,7 @@ func TestBodyFollowsTheTemplate(t *testing.T) { for i := range 40 { many = append(many, &verdict{o: &original{number: i + 1, title: "fix(x): y", body: "```release-note\n" + long + "\n```"}}) } - r = &report{line: l, onBranch: many, skipped: map[int][]*verdict{ruleType: many}} + r = &report{line: l, onBranch: many, skipped: map[int][]*verdict{ruleType: many}, blocks: deferredBlock(l, nil, nil, nil)} if body, _ := r.body(template); len(body) > bodyLimit || !strings.Contains(body, "truncated") || !strings.Contains(body, deferredFence) { t.Errorf("a body over the limit is %d bytes (limit %d), truncated=%v", len(body), bodyLimit, strings.Contains(body, "truncated")) } @@ -322,7 +322,7 @@ func TestBodyFollowsTheTemplate(t *testing.T) { for i := range 80 { conflict.files = append(conflict.files, fmt.Sprintf("chart/f%02d.yaml", i)) } - r = &report{line: l, onBranch: many[:1], deferred: []pick{conflict}, skipped: map[int][]*verdict{ruleType: lots, ruleOnLine: few}} + r = &report{line: l, onBranch: many[:1], deferred: []pick{conflict}, skipped: map[int][]*verdict{ruleType: lots, ruleOnLine: few}, blocks: deferredBlock(l, []pick{conflict}, nil, nil)} body, left := r.body(template) if !slices.Equal(left, []int{ruleType}) { t.Errorf("left out the lists of rules %v, want only rule 2", left) @@ -366,3 +366,17 @@ func TestBodyLinksTaskPagesThatExist(t *testing.T) { } } } + +// A body that what it has to carry pushes over GitHub's limit is refused +// before anything is written, the replay of an earlier report included, +// although the report alone would fit a comment. +func TestBodyRefusesMandatoryContentOverTheLimit(t *testing.T) { + f, gh, head := bigWaits(t, 4) + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--no-workflow-files", "--push") + if code != 2 || !strings.Contains(stderr, "with only what it cannot leave out, over GitHub's 65536") { + t.Errorf("run --push = %d, want a refusal of the body:\n%s", code, stderr) + } + if f.remoteHead(batchBranch1) != head || len(gh.issueComments) != 0 || len(gh.edited) != 0 { + t.Errorf("the refused run moved the batch to %.12s, wrote %d comments and edited %q", f.remoteHead(batchBranch1), len(gh.issueComments), gh.edited) + } +} diff --git a/cmd/backport-batch/defer.go b/cmd/backport-batch/defer.go index 5f2b29c6a..cdc17a78c 100644 --- a/cmd/backport-batch/defer.go +++ b/cmd/backport-batch/defer.go @@ -114,6 +114,9 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { return a.fail(err) } r := newReport(b) + // Taking an original out changes nothing about how the batch is + // assembled, so the record keeps the flag the batch was assembled with. + r.opts = runOptions{NoWorkflowFiles: b.src.savedNoWorkflowFiles(), Mode: modeDefer} if r.stale, err = keptOut(b, picks); err != nil { return a.fail(err) } @@ -128,6 +131,10 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { r.waiting = append(r.waiting, pick{v: v, status: "waiting", files: e.Files, more: e.More}) case len(e.Files) > 0: r.deferred = append(r.deferred, pick{v: v, status: "conflict", files: e.Files, more: e.More, failed: e.Commit}) + default: + // Merged into main since the batch was assembled, or recorded + // nowhere: a rebuild adds no original, the next run tries it. + r.missing = append(r.missing, v) } } return a.finish(b, r, branch, date, push, true) diff --git a/cmd/backport-batch/defer_test.go b/cmd/backport-batch/defer_test.go index 856c9909a..59c736083 100644 --- a/cmd/backport-batch/defer_test.go +++ b/cmd/backport-batch/defer_test.go @@ -261,8 +261,8 @@ func TestDeferLeasesTheHeadItRebuilt(t *testing.T) { pr := f.batchPR(90, "release-1.1", batchBranch1, gh.prs[0].Body) a := &app{gh: gh} - b := &batch{line: l, lineHead: lineHead, pr: &pr, prHead: head} - err := a.publish(b, batchBranch1, rebuilt, "title", "body", true) + b := &batch{line: l, lineHead: lineHead, pr: &pr, prHead: head, src: recordSource{carrier: &pr, open: true, code: &pr}} + err := a.publish(b, publication{branch: batchBranch1, head: rebuilt, title: "title", body: "body"}, true) if err == nil || !strings.Contains(err.Error(), "which the rebuild started from") { t.Errorf("publish over a head pushed since = %v, want a refusal", err) } diff --git a/cmd/backport-batch/fixture_test.go b/cmd/backport-batch/fixture_test.go index 23e157f9f..1a633134f 100644 --- a/cmd/backport-batch/fixture_test.go +++ b/cmd/backport-batch/fixture_test.go @@ -18,6 +18,7 @@ package main import ( "bytes" + "cmp" "fmt" "os" "os/exec" @@ -36,6 +37,8 @@ type fixture struct { dir string remote string now time.Time + // tool is the commit the tool says it was built from. + tool string } func newFixture(t *testing.T) *fixture { @@ -256,6 +259,48 @@ type fakeGH struct { // closedWrites names every write the tool attempted on a pull request // that is no longer open; exec fails the test on one. closedWrites []string + // fail makes the next call of an operation, by its method name, fail. + fail map[string]error + // nextPR is the number createPull gives, 90 when unset. + nextPR int + // remoteHead reads a branch's head off the fixture's remote: GitHub + // reports an open pull request at what its branch has. + remoteHead func(string) string + // commented lists the pull request each comment went to, in order. + commented []int + // after runs once an operation, by its method name, has succeeded: what + // someone else does on GitHub meanwhile. + after map[string]func() +} + +// live is pr as GitHub reports it now: an open one at the head its branch +// has on the remote. +func (g *fakeGH) live(pr pullRequest) pullRequest { + if g.remoteHead != nil && (pr.State == "" || pr.State == "open") { + if head := g.remoteHead(pr.Head.Ref); head != "" { + pr.Head.SHA = head + } + } + return pr +} + +// inject is the failure a test set for the next call of op, once. +func (g *fakeGH) inject(op string) error { + err := g.fail[op] + delete(g.fail, op) + return err +} + +func (g *fakeGH) pull(n int) (pullRequest, error) { + if err := g.inject("pull"); err != nil { + return pullRequest{}, err + } + for _, pr := range g.prs { + if pr.Number == n { + return g.live(pr), nil + } + } + return pullRequest{}, fmt.Errorf("HTTP 404: no pull request #%d", n) } // isOpen says whether the fake has pull request n open, or does not know it. @@ -296,6 +341,11 @@ func (g *fakeGH) recentPulls(base string, page int) ([]pullRequest, error) { func (g *fakeGH) batchBot() string { return g.bot } func (g *fakeGH) comments(int) ([]issueComment, error) { + defer func() { + if f := g.after["comments"]; f != nil { + f() + } + }() return slices.Clone(g.issueComments), nil } @@ -303,6 +353,15 @@ func (g *fakeGH) createComment(number int, bodyFile string) error { if err := g.write(number, "a comment"); err != nil { return err } + if err := g.inject("createComment"); err != nil { + return err + } + g.commented = append(g.commented, number) + defer func() { + if f := g.after["createComment"]; f != nil { + f() + } + }() data, err := os.ReadFile(bodyFile) var c issueComment c.ID, c.Body = int64(1000+len(g.issueComments)), string(data) @@ -346,23 +405,56 @@ func (g *fakeGH) openPulls(base string) ([]pullRequest, error) { var out []pullRequest for _, pr := range g.prs { if pr.Base.Ref == base && (pr.State == "" || pr.State == "open") { - out = append(out, pr) + out = append(out, g.live(pr)) } } return out, g.err } -func (g *fakeGH) createPull(base, head, title, bodyFile string) (string, error) { +// createPull opens the pull request on the fake, as the maintainer, at the +// head the remote has, numbered nextPR. +func (g *fakeGH) createPull(base, head, title, bodyFile string) (int, string, error) { + if err := g.inject("createPull"); err != nil { + return 0, "", err + } g.created = append(g.created, base+" "+head+" "+title) - return fmt.Sprintf("https://github.com/%s/pull/90", upstreamRepo), g.readBody(bodyFile) + if err := g.readBody(bodyFile); err != nil { + return 0, "", err + } + n := cmp.Or(g.nextPR, 90) + var pr pullRequest + pr.Number, pr.Body, pr.State = n, g.bodies[len(g.bodies)-1], "open" + pr.HTMLURL = fmt.Sprintf("https://github.com/%s/pull/%d", upstreamRepo, n) + pr.User.Login, pr.User.Type = "maintainer", "User" + pr.Head.Ref = head + if g.remoteHead != nil { + pr.Head.SHA = g.remoteHead(head) + } + pr.Head.Repo = &struct { + FullName string `json:"full_name"` + }{upstreamRepo} + pr.Base.Ref, pr.Base.Repo.FullName = base, upstreamRepo + g.prs = append(slices.DeleteFunc(g.prs, func(p pullRequest) bool { return p.Number == n }), pr) + return n, pr.HTMLURL, nil } func (g *fakeGH) editPull(number int, title, bodyFile string) error { if err := g.write(number, "an edit"); err != nil { return err } + if err := g.inject("editPull"); err != nil { + return err + } g.edited = append(g.edited, fmt.Sprintf("#%d %s", number, title)) - return g.readBody(bodyFile) + if err := g.readBody(bodyFile); err != nil { + return err + } + for i := range g.prs { + if g.prs[i].Number == number { + g.prs[i].Body = g.bodies[len(g.bodies)-1] + } + } + return nil } func (g *fakeGH) readBody(path string) error { @@ -423,8 +515,11 @@ func (f *fixture) exec(gh *fakeGH, args ...string) (int, string, string) { if gh == nil { gh = &fakeGH{} } + if gh.remoteHead == nil { + gh.remoteHead = f.remoteHead + } var stdout, stderr bytes.Buffer - a := &app{gh: gh, stdout: &stdout, stderr: &stderr, now: func() time.Time { return f.now }} + a := &app{gh: gh, stdout: &stdout, stderr: &stderr, now: func() time.Time { return f.now }, tool: f.tool} written := len(gh.closedWrites) code := a.main(args) if len(gh.closedWrites) > written { diff --git a/cmd/backport-batch/github.go b/cmd/backport-batch/github.go index 3a7fa39d4..b1f3b5ce3 100644 --- a/cmd/backport-batch/github.go +++ b/cmd/backport-batch/github.go @@ -52,7 +52,7 @@ var ( batchBranchRE = regexp.MustCompile(`^backport-batch/(release-\d+\.\d+)-(\d{8})$`) loginRE = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?$`) botLoginRE = regexp.MustCompile(`^[A-Za-z0-9](?:[A-Za-z0-9-]*[A-Za-z0-9])?\[bot\]$`) - pullURLRE = regexp.MustCompile(`^https://[^/\s]+/[^/\s]+/[^/\s]+/pull/[1-9][0-9]*$`) + pullURLRE = regexp.MustCompile(`^https://[^/\s]+/[^/\s]+/[^/\s]+/pull/([1-9][0-9]{0,8})$`) ) // batchBotEnv names the GitHub App bot the scheduled batch opens its pull @@ -122,8 +122,10 @@ type github interface { permission(login string) (string, error) // labelled lists the pull requests merged into main that carry label. labelled(label string) ([]int, error) - // createPull opens the draft and returns its URL. - createPull(base, head, title, bodyFile string) (string, error) + // pull reads one pull request as it is now. + pull(number int) (pullRequest, error) + // createPull opens the draft and returns its number and URL. + createPull(base, head, title, bodyFile string) (int, string, error) editPull(number int, title, bodyFile string) error comments(number int) ([]issueComment, error) createComment(number int, bodyFile string) error @@ -223,16 +225,33 @@ func (ghCLI) labelled(label string) ([]int, error) { return numbers, nil } -func (ghCLI) createPull(base, head, title, bodyFile string) (string, error) { +func (ghCLI) pull(number int) (pullRequest, error) { + var pr pullRequest + out, err := capture("gh", "api", "repos/{owner}/{repo}/pulls/"+strconv.Itoa(number)) + if err != nil { + return pr, err + } + if err := json.Unmarshal([]byte(out), &pr); err != nil { + return pr, fmt.Errorf("reading pull request #%d: %w", number, err) + } + if pr.Number != number { + return pr, fmt.Errorf("asked for pull request #%d, GitHub answered with #%d", number, pr.Number) + } + return pr, nil +} + +func (ghCLI) createPull(base, head, title, bodyFile string) (int, string, error) { out, err := capture("gh", "pr", "create", "--draft", "--base", base, "--head", head, "--title", title, "--body-file", bodyFile) if err != nil { - return "", err + return 0, "", err } url := strings.TrimSpace(out) - if !pullURLRE.MatchString(url) { - return "", fmt.Errorf("gh pr create printed %q, not the URL of a pull request", url) + m := pullURLRE.FindStringSubmatch(url) + if m == nil { + return 0, "", fmt.Errorf("gh pr create printed %q, not the URL of a pull request", url) } - return url, nil + n, err := strconv.Atoi(m[1]) + return n, url, err } func (ghCLI) editPull(number int, title, bodyFile string) error { diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index b921656f7..98ed2b2ac 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -38,6 +38,7 @@ Usage: backport-batch fold --line release-X.Y --original N --commit SHA [--policy REF] backport-batch verify --line release-X.Y [--head SHA] backport-batch notify --line release-X.Y [--mention org/team|login] [--push] + backport-batch report --pr N [--id ID] [--json] backport-batch lines [--policy REF] run Builds backport-batch/- in this working tree, off @@ -69,11 +70,15 @@ notify Prints what waits on the open batch PR for a maintainer: a batch open for 3 days or more, its deferred conflicts, and the originals that change workflows. --push keeps one comment on the PR that says what waits, mentioning --mention. +report Prints a report batch PR N keeps: the one its body carries, or the + one --id names, read from its comments or its body. --json prints the + report as it is stored. It writes nothing, so it reads merged and + closed batches too. lines Prints the release lines the batch runs for, newest first. The policy is docs/release/backports.md. run, check, fold and lines read its policy block from that page on origin/main, or on --policy REF. Every command -but notify, which reads only GitHub, fetches origin first. +but notify and report, which read only GitHub, fetches origin first. BACKPORT_BATCH_BOT names the bot the scheduled batch writes as, [bot], cozystack-ci[bot] when unset: a batch pull request or a notify @@ -93,10 +98,12 @@ type app struct { now func() time.Time // policyRef is where the policy page is read from; origin/main when empty. policyRef string + // tool is the commit backport-batch was built from, for the record. + tool string } func main() { - a := &app{gh: ghCLI{bot: batchBotLogin(os.LookupEnv)}, stdout: os.Stdout, stderr: os.Stderr, now: time.Now} + a := &app{gh: ghCLI{bot: batchBotLogin(os.LookupEnv)}, stdout: os.Stdout, stderr: os.Stderr, now: time.Now, tool: toolVersion()} os.Exit(a.main(os.Args[1:])) } @@ -121,6 +128,8 @@ func (a *app) main(args []string) int { return a.verify(args[1:]) case "notify": return a.notify(args[1:]) + case "report": + return a.report(args[1:]) case "lines": return a.lines(args[1:]) case "-h", "--help", "help": @@ -199,11 +208,15 @@ type batch struct { newer *line lineHead string mainHead string - base string - other []string - verdicts []*verdict - pr *pullRequest - prHead string + // newerHead is the newer line's head, and policyAt the commit the + // policy was read from. + newerHead string + policyAt string + base string + other []string + verdicts []*verdict + pr *pullRequest + prHead string // picks is what the open batch pull request carries beyond the line. picks *history // src is the line's record and where it was found. @@ -229,18 +242,20 @@ func fetch() error { return err } -func (a *app) policy(mainHead string) (*policy, error) { +// policy reads the policy and says from which commit. +func (a *app) policy(mainHead string) (*policy, string, error) { at := mainHead if a.policyRef != "" { if strings.HasPrefix(a.policyRef, "-") { - return nil, fmt.Errorf("--policy %q is not a ref", a.policyRef) + return nil, "", fmt.Errorf("--policy %q is not a ref", a.policyRef) } var err error if at, err = resolve(a.policyRef); err != nil { - return nil, err + return nil, "", err } } - return readPolicy(at) + pol, err := readPolicy(at) + return pol, at, err } // prepare reads the line, main and the open batch pull request, each pinned to @@ -251,7 +266,7 @@ func (a *app) prepare(l line) (*batch, error) { if b.mainHead, err = resolve("refs/remotes/" + remote + "/main"); err != nil { return nil, err } - pol, err := a.policy(b.mainHead) + pol, policyAt, err := a.policy(b.mainHead) if err != nil { return nil, err } @@ -265,7 +280,7 @@ func (a *app) prepare(l line) (*batch, error) { if b.lineHead, err = resolve("refs/remotes/" + remote + "/" + l.branch()); err != nil { return nil, err } - b.start, b.textUntil = pol.starts[l.branch()], pol.batchOnly + b.start, b.textUntil, b.policyAt = pol.starts[l.branch()], pol.batchOnly, policyAt out, err := git("merge-base", "--end-of-options", b.mainHead, b.lineHead) if err != nil { return nil, fmt.Errorf("%s shares no history with main: %w", l.branch(), err) @@ -282,11 +297,10 @@ func (a *app) prepare(l line) (*batch, error) { } var newerHist *history if b.newer != nil { - newerHead, err := resolve("refs/remotes/" + remote + "/" + b.newer.branch()) - if err != nil { + if b.newerHead, err = resolve("refs/remotes/" + remote + "/" + b.newer.branch()); err != nil { return nil, err } - if newerHist, err = readHistory(newerHead, b.mainHead, b.textUntil); err != nil { + if newerHist, err = readHistory(b.newerHead, b.mainHead, b.textUntil); err != nil { return nil, err } } diff --git a/cmd/backport-batch/notify.go b/cmd/backport-batch/notify.go index 80c5aa1da..7191f42b4 100644 --- a/cmd/backport-batch/notify.go +++ b/cmd/backport-batch/notify.go @@ -182,6 +182,9 @@ func (a *app) comment(pr *pullRequest, mention string, today time.Time, text []s if err := f.Close(); err != nil { return a.fail(err) } + if err := a.stillOpen(pr.Number); err != nil { + return a.fail(err) + } if ours != nil { err = a.gh.editComment(ours.ID, f.Name()) } else { @@ -207,7 +210,7 @@ func (a *app) lines(args []string) int { if err != nil { return a.fail(err) } - pol, err := a.policy(mainHead) + pol, _, err := a.policy(mainHead) if err != nil { return a.fail(err) } diff --git a/cmd/backport-batch/notify_test.go b/cmd/backport-batch/notify_test.go index 6b91a4159..4bfb650e9 100644 --- a/cmd/backport-batch/notify_test.go +++ b/cmd/backport-batch/notify_test.go @@ -313,3 +313,24 @@ func TestRunWithNothingToPick(t *testing.T) { t.Errorf("pushed %q and opened %q with nothing picked", f.remoteHead(batchBranch1), gh.created) } } + +// notify writes its comment only after GitHub says the batch is still open, +// so a batch closed while notify reads it is not written to, whether the +// comment would be new or an update of its own. +func TestNotifyWritesNothingOnABatchClosedMeanwhile(t *testing.T) { + for _, update := range []bool{false, true} { + t.Run(map[bool]string{false: "a new comment", true: "an update"}[update], func(t *testing.T) { + gh := &fakeGH{prs: []pullRequest{notifyPR("20261023", recordBody(5))}} + if update { + ours := issueComment{ID: 7, Body: notifyMarker + "\nearlier"} + ours.User.Login, ours.User.Type = "maintainer", "User" + gh.issueComments = []issueComment{ours} + } + gh.after = map[string]func(){"comments": func() { gh.find(90).State = "closed" }} + code, _, stderr := notifyRun(gh, "--push") + if code != 2 || !strings.Contains(stderr, "batch pull request #90 is closed now; nothing is written to it") || len(gh.closedWrites) != 0 || update && gh.issueComments[0].Body != notifyMarker+"\nearlier" { + t.Errorf("notify = %d, closed writes %q, comments %+v; want 2 and nothing written:\n%s", code, gh.closedWrites, gh.issueComments, stderr) + } + }) + } +} diff --git a/cmd/backport-batch/overrides_test.go b/cmd/backport-batch/overrides_test.go index 740713359..a759fd59c 100644 --- a/cmd/backport-batch/overrides_test.go +++ b/cmd/backport-batch/overrides_test.go @@ -238,16 +238,26 @@ func TestGHCLI(t *testing.T) { // A batch pull request carries no label: the release workflows act on a // pull request cozystack-ci[bot] opened only when it has the release label. write("https://github.com/cozystack/cozystack/pull/5300\n") - if url, err := (ghCLI{}).createPull("release-1.7", batchBranch1, "fix(backport): batch", "/tmp/body.md"); err != nil || url != "https://github.com/cozystack/cozystack/pull/5300" { - t.Errorf("createPull = %q, %v; want the URL gh printed", url, err) + if n, url, err := (ghCLI{}).createPull("release-1.7", batchBranch1, "fix(backport): batch", "/tmp/body.md"); err != nil || n != 5300 || url != "https://github.com/cozystack/cozystack/pull/5300" { + t.Errorf("createPull = %d, %q, %v; want the pull request gh printed", n, url, err) } if want := "pr create --draft --base release-1.7 --head " + batchBranch1 + " --title fix(backport): batch --body-file /tmp/body.md"; called() != want { t.Errorf("gh %q, want gh %q", called(), want) } write("Warning: 1 uncommitted change\n") - if _, err := (ghCLI{}).createPull("release-1.7", batchBranch1, "fix(backport): batch", "/tmp/body.md"); err == nil { + if _, _, err := (ghCLI{}).createPull("release-1.7", batchBranch1, "fix(backport): batch", "/tmp/body.md"); err == nil { t.Error("createPull took output that is no pull request URL") } + write(`{"number":90,"state":"closed","merged_at":"2026-10-26T12:00:00Z","head":{"ref":"x","sha":"y"}}`) + if pr, err := (ghCLI{}).pull(90); err != nil || pr.State != "closed" || pr.MergedAt == nil { + t.Errorf("pull = %+v, %v; want #90 merged", pr, err) + } + if want := "api repos/{owner}/{repo}/pulls/90"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } + if _, err := (ghCLI{}).pull(91); err == nil { + t.Error("pull took GitHub's answer about another pull request") + } write("") if err := (ghCLI{}).editPull(90, "fix(backport): batch", "/tmp/body.md"); err != nil { t.Fatal(err) diff --git a/cmd/backport-batch/record.go b/cmd/backport-batch/record.go index 648b000ba..4f504ee55 100644 --- a/cmd/backport-batch/record.go +++ b/cmd/backport-batch/record.go @@ -285,6 +285,27 @@ func (b *builtFrom) validate() error { var runURLRE = regexp.MustCompile(`^https://[^/\s]+/[^/\s]+/[^/\s]+/actions/runs/[0-9]+(/attempts/[0-9]+)?$`) +// recordBlock is the block that holds rec. One line of JSON: encoding/json +// escapes newlines and <, > and &, so nothing in a file name or a reason can +// end the fence or open HTML. +func recordBlock(rec lineRecord) string { + data, _ := json.Marshal(rec) + return deferredFence + "\n" + string(data) + "\n```" +} + +// savedNoWorkflowFiles is the --no-workflow-files the open batch was last +// assembled with, which an operation that only changes the record keeps. A +// record from before it was saved says so by listing a wait. +func (s recordSource) savedNoWorkflowFiles() bool { + if !s.open || s.rec == nil { + return false + } + if s.rec.Built != nil { + return s.rec.Built.Options.NoWorkflowFiles + } + return slices.ContainsFunc(s.rec.Deferred, func(d deferredEntry) bool { return d.Workflows }) +} + // deferred maps the originals the record defers to their entries. func (rec *lineRecord) deferred() map[int]deferredEntry { out := map[int]deferredEntry{} diff --git a/cmd/backport-batch/record_test.go b/cmd/backport-batch/record_test.go index b512a2cda..830685474 100644 --- a/cmd/backport-batch/record_test.go +++ b/cmd/backport-batch/record_test.go @@ -242,3 +242,77 @@ func TestRecordAsksForAPermissionOnce(t *testing.T) { t.Errorf("asked GitHub for %q, want the author once", gh.asked) } } + +// The record and the report name what the classification was made of: the +// heads it read, the policy, the tool, the job, the time and the flags. +func TestRecordNamesItsInputs(t *testing.T) { + f := newFixture(t) + f.lines() + mc5 := f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.onLine("release-1.1", "fix(app): five\n\n(cherry picked from commit "+mc5+")", map[string]string{"app/five.txt": "5\n"}) + f.tool = strings.Repeat("c", 40) + "+dirty" + t.Setenv("GITHUB_SERVER_URL", "https://github.com") + t.Setenv("GITHUB_REPOSITORY", "cozystack/cozystack") + t.Setenv("GITHUB_RUN_ID", "123") + t.Setenv("GITHUB_RUN_ATTEMPT", "2") + gh := &fakeGH{} + stdout := mustRun(t, f, gh, "run", "--line", "release-1.0", "--no-workflow-files", "--push") + l, _ := parseLine("release-1.0") + const branch = "backport-batch/release-1.0-20261026" + rec, err := parseRecord(stdout, l) + if err != nil { + t.Fatal(err) + } + want := builtFrom{Main: f.git("rev-parse", "origin/main"), Line: f.git("rev-parse", "origin/release-1.0"), Newer: f.git("rev-parse", "origin/release-1.1"), + Policy: f.git("rev-parse", "origin/main"), Batch: f.remoteHead(branch), Tool: f.tool, Run: "https://github.com/cozystack/cozystack/actions/runs/123/attempts/2", + At: "2026-10-26T06:00:00Z", Options: runOptions{NoWorkflowFiles: true, Mode: modeNew}} + if rec.Version != 2 || *rec.Built != want || bodyReportOf(t, stdout, l).doc.Built != want || rec.Report != bodyReportOf(t, stdout, l).doc.ID { + t.Errorf("record = %+v, built %+v; want built %+v and the report it names", rec, rec.Built, want) + } + + // A policy from another ref, a run by hand that continues the batch. + f.git("push", "-q", "origin", "origin/main:refs/heads/policy-draft") + f.git("fetch", "-q", "origin") + t.Setenv("GITHUB_RUN_ID", "") + gh.prs = []pullRequest{f.batchPR(90, "release-1.0", branch, stdout)} + mc6 := f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.onLine("release-1.1", "fix(app): six\n\n(cherry picked from commit "+mc6+")", map[string]string{"app/six.txt": "6\n"}) + stdout = mustRun(t, f, gh, "run", "--line", "release-1.0", "--continue", "--policy", "origin/policy-draft", "--push") + if rec, err = parseRecord(stdout, l); err != nil { + t.Fatal(err) + } + if b := rec.Built; b.Policy != f.git("rev-parse", "origin/policy-draft") || b.Main != f.git("rev-parse", "origin/main") || b.Run != "" || b.Options != (runOptions{Mode: modeContinue}) || b.Batch != f.remoteHead(branch) { + t.Errorf("built %+v; want the draft's policy, no job and a continuation without the flag", b) + } +} + +// What a run assembled with --no-workflow-files stays saved through the +// operations that only change the record, which take no such flag: check +// keeps failing the cut on the wait they leave in place. +func TestMetadataOperationsKeepTheMode(t *testing.T) { + for _, v1 := range []bool{false, true} { + t.Run(fmt.Sprintf("an open record of version 1: %v", v1), func(t *testing.T) { + f := workflowWorld(t) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + gh := &fakeGH{} + stdout := mustRun(t, f, gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push") + reopen(f, gh) + if v1 { + waits := []pick{{v: &verdict{o: &original{number: 5}}, files: []string{".github/workflows/five.yaml"}}} + gh.prs[0].Body = strings.Replace(stdout, strings.Split(strings.Split(stdout, deferredFence+"\n")[1], "\n")[0], strings.Split(deferredBlock(notifyLine, nil, nil, waits), "\n")[1], 1) + } else if rec, err := parseRecord(stdout, notifyLine); err != nil || !rec.Built.Options.NoWorkflowFiles { + t.Fatalf("the scheduled run saved %+v, %v", rec, err) + } + stdout = mustRun(t, f, gh, "run", "--line", "release-1.1", "--defer", "7", "--reason", "fails the e2e", "--push") + rec, err := parseRecord(stdout, notifyLine) + if err != nil || rec.Built.Options != (runOptions{NoWorkflowFiles: true, Mode: modeDefer}) { + t.Errorf("after --defer the record is %+v, %v; want the flag kept", rec.Built, err) + } + reopen(f, gh) + if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#5 changes .github/workflows/") { + t.Errorf("check --cut = %d, want #5 still waiting:\n%s", code, stdout) + } + }) + } +} diff --git a/cmd/backport-batch/report.go b/cmd/backport-batch/report.go new file mode 100644 index 000000000..78c460479 --- /dev/null +++ b/cmd/backport-batch/report.go @@ -0,0 +1,734 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "cmp" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "os" + "regexp" + "runtime/debug" + "slices" + "strconv" + "strings" + "time" +) + +// A report is one classification of a line: where every original merged into +// main since the line branched stands, and what it was made of. The batch +// pull request's body carries the latest one beside the record, and an issue +// comment the tool never edits keeps each one, so a later run that replaces +// the body loses none of them. Its ID hashes the classification, not the pull +// request it lands on, which a new batch does not have yet when the report is +// made. + +const ( + reportFence = "```backport-batch-report" + reportMarker = "\n" + + "
Backport batch report for " + s.doc.Line + ", " + date + " (" + s.doc.ID[:12] + ")\n\n" + + reportBlock(s.raw) + "\n
\n" + if len(c) > reportLimit { + return "", fmt.Errorf("the report comment is %d bytes, over the %d it may have", len(c), reportLimit) + } + return c, nil +} + +var reportMarkerRE = regexp.MustCompile(`^` + regexp.QuoteMeta(reportMarker) + `([0-9a-f]{64}) -->\n`) + +// findReports reads the report comments on batch pull request n, against l, +// by authors the batch trusts, as it trusts a record's author, by ID. One +// whose content does not match the ID its marker gives, or is not what a run +// writes, is returned apart, with why; a look-alike by anyone else is not +// read, whatever it says. +func findReports(gh github, n int, l line) (map[string]storedReport, map[string]string, error) { + comments, err := gh.comments(n) + if err != nil { + return nil, nil, err + } + found, bad := map[string]storedReport{}, map[string]string{} + for _, c := range comments { + if !strings.HasPrefix(c.Body, reportMarker) { + continue + } + why, err := trustedAuthor(gh, c.User.Login, c.User.Type) + if err != nil { + return nil, nil, err + } + if why != "" { + continue + } + m := reportMarkerRE.FindStringSubmatch(c.Body) + if m == nil { + bad[strconv.FormatInt(c.ID, 10)] = fmt.Sprintf("comment %d starts like a report but names none", c.ID) + continue + } + blocks, err := fencedLines(c.Body, reportFence) + if err == nil && len(blocks) != 1 { + err = fmt.Errorf("it holds %d reports", len(blocks)) + } + var doc reportDoc + if err == nil { + doc, err = parseReport(blocks[0], l) + } + if err == nil && doc.ID != m[1] { + err = fmt.Errorf("its marker names %.12s", m[1]) + } + if err != nil { + bad[m[1]] = fmt.Sprintf("comment %d: %v", c.ID, err) + continue + } + found[m[1]] = storedReport{doc: doc, raw: blocks[0]} + } + return found, bad, nil +} + +// toolVersion is the commit this binary was built from, with +dirty when its +// tree had changes, or "" when the build did not record it, as go run does +// not. +func toolVersion() string { + info, ok := debug.ReadBuildInfo() + if !ok { + return "" + } + rev, dirty := "", false + for _, s := range info.Settings { + switch s.Key { + case "vcs.revision": + rev = s.Value + case "vcs.modified": + dirty = s.Value == "true" + } + } + if checkSHA(rev) != nil { + return "" + } + if dirty { + rev += "+dirty" + } + return rev +} + +// runURL is the Actions run the command runs in, from the variables GitHub +// sets for it, or "" outside one. +func runURL(lookup func(string) (string, bool)) string { + get := func(key string) string { v, _ := lookup(key); return v } + url := get("GITHUB_SERVER_URL") + "/" + get("GITHUB_REPOSITORY") + "/actions/runs/" + get("GITHUB_RUN_ID") + if attempt := get("GITHUB_RUN_ATTEMPT"); attempt != "" && attempt != "1" { + url += "/attempts/" + attempt + } + if !runURLRE.MatchString(url) { + return "" + } + return url +} + +// report prints a report a batch pull request keeps, reading GitHub alone and +// writing nothing, so it works on merged and closed batches too. +func (a *app) report(args []string) int { + fs := a.flags("report") + number := fs.Int("pr", 0, "the batch pull request") + id := fs.String("id", "", "the report, by its ID (default: the one the body carries)") + asJSON := fs.Bool("json", false, "print the report's JSON") + if err := parse(fs, args); err != nil { + return 2 + } + if *number <= 0 { + return a.fail(fmt.Errorf("--pr %d is not a pull request number", *number)) + } + if *id != "" && !reportIDRE.MatchString(*id) { + return a.fail(fmt.Errorf("--id %q is not a report ID", *id)) + } + pr, err := a.gh.pull(*number) + if err != nil { + return a.fail(err) + } + l, err := parseLine(pr.Base.Ref) + if m := batchBranchRE.FindStringSubmatch(pr.Head.Ref); err != nil || m == nil || m[1] != l.branch() || pr.Head.Repo == nil || pr.Head.Repo.FullName != pr.Base.Repo.FullName { + return a.fail(fmt.Errorf("#%d is not a batch pull request of this repository", pr.Number)) + } + why, err := trusted(a.gh, pr) + if err != nil { + return a.fail(err) + } + if why != "" { + return a.fail(fmt.Errorf("batch pull request #%d was %s, so what it says is not the batch's", pr.Number, why)) + } + inBody, err := bodyReport(pr.Body, l) + if err != nil { + return a.fail(fmt.Errorf("batch pull request #%d: %w", pr.Number, err)) + } + if *id == "" { + if inBody == nil { + return a.fail(fmt.Errorf("batch pull request #%d carries no report in its body; name one with --id", pr.Number)) + } + *id = inBody.doc.ID + } + posted, bad, err := findReports(a.gh, pr.Number, l) + if err != nil { + return a.fail(err) + } + if why := bad[*id]; why != "" { + return a.fail(fmt.Errorf("batch pull request #%d: the report %.12s is not what its comment says: %s", pr.Number, *id, why)) + } + s, inComment := posted[*id] + switch { + case inComment: + case inBody != nil && inBody.doc.ID == *id: + s = *inBody + default: + return a.fail(fmt.Errorf("batch pull request #%d keeps no report %.12s", pr.Number, *id)) + } + if *asJSON { + fmt.Fprintln(a.stdout, s.raw) + return 0 + } + a.printReport(&pr, s, inBody, inComment) + return 0 +} + +// printReport prints a report. It was checked when it was read, and every +// field still goes through terminalSafe, so nothing it carries spans lines +// or drives a terminal. +func (a *app) printReport(pr *pullRequest, s storedReport, inBody *storedReport, inComment bool) { + d, b := s.doc, s.doc.Built + safe := terminalSafe + fmt.Fprintf(a.stdout, "report %s for %s, made %s by a %s run", safe(d.ID), safe(d.Line), safe(b.At), safe(b.Options.Mode)) + if b.Options.NoWorkflowFiles { + fmt.Fprint(a.stdout, " with --no-workflow-files") + } + fmt.Fprintln(a.stdout) + fmt.Fprintf(a.stdout, " main %s, line %s, newer line %s, policy %s\n batch %s, tool %s, run %s\n", safe(b.Main), safe(b.Line), safe(or(b.Newer, "none")), safe(b.Policy), safe(b.Batch), safe(or(b.Tool, "unknown")), safe(or(b.Run, "none"))) + current := "none" + if inBody != nil { + current = safe(inBody.doc.ID) + } + switch { + case inBody != nil && inBody.doc.ID == d.ID: + fmt.Fprintf(a.stdout, " #%d's body carries this report\n", pr.Number) + case pr.Head.SHA == b.Batch: + fmt.Fprintf(a.stdout, " #%d's head is %.12s, the batch this report describes, but its body carries report %.12s\n", pr.Number, safe(b.Batch), current) + default: + fmt.Fprintf(a.stdout, " publication unconfirmed: #%d's head is %.12s, not %.12s, and its body carries report %.12s\n", pr.Number, safe(pr.Head.SHA), safe(b.Batch), current) + } + if !inComment { + if prState(pr) == "open" { + fmt.Fprintln(a.stdout, " comment not posted; the next backport-batch run --push posts it") + } else { + fmt.Fprintf(a.stdout, " comment not posted; kept in the body only, since nothing is written to a %s pull request\n", prState(pr)) + } + } + type line struct { + n int + text string + } + var lines []line + for _, n := range d.Primary.Picked { + lines = append(lines, line{n, "picked"}) + } + for _, n := range d.Primary.Missing { + lines = append(lines, line{n, "missing"}) + } + for _, e := range d.Primary.Deferred { + text := "deferred: " + switch { + case e.Workflows: + text += "waits for a maintainer, changes " + strings.Join(e.Files, ", ") + case e.Reason != "": + text += e.Reason + default: + text += fmt.Sprintf("conflict at %.12s in %s", e.Commit, strings.Join(e.Files, ", ")) + } + if e.More > 0 { + text += fmt.Sprintf(" and %d more", e.More) + } + lines = append(lines, line{e.Original, text}) + } + for _, l := range d.Primary.Skipped.lists() { + for _, n := range *l.list { + lines = append(lines, line{n, "skipped." + l.name}) + } + } + slices.SortFunc(lines, func(x, y line) int { return cmp.Compare(x.n, y.n) }) + for _, l := range lines { + fmt.Fprintf(a.stdout, "#%d %s\n", l.n, terminalSafe(l.text)) + } + overlay := func(name string, list []int) { + if len(list) > 0 { + fmt.Fprintf(a.stdout, "%s: %s\n", name, strings.Trim(fmt.Sprint(list), "[]")) + } + } + overlay("stale, still on the batch", d.Overlays.Stale) + for _, e := range d.Overlays.Dropped { + fmt.Fprintf(a.stdout, "dropped: #%d, merged as %.12s, on #%d: %s\n", e.Original, terminalSafe(e.Merge), e.On, terminalSafe(e.Reason)) + } + overlay("changed in the pick", d.Overlays.Changed) + if len(d.Other) > 0 { + other := make([]string, len(d.Other)) + for i, sha := range d.Other { + other[i] = safe(sha) + } + fmt.Fprintf(a.stdout, "commits on main that merged no pull request: %s\n", strings.Join(other, ", ")) + } +} + +func or(s, otherwise string) string { + if s == "" { + return otherwise + } + return s +} + +// postReport writes a report comment on pull request n, after GitHub says +// it is still open: nothing is written to a merged or closed one. +func (a *app) postReport(n int, s storedReport) error { + if err := a.stillOpen(n); err != nil { + return err + } + c, err := reportComment(s) + if err != nil { + return err + } + name, err := writeTemp("backport-batch-report-*.md", c) + if err != nil { + return err + } + defer os.Remove(name) + return a.gh.createComment(n, name) +} + +// stillOpen reads pull request n again and refuses to go on unless it is +// open. +func (a *app) stillOpen(n int) error { + pr, err := a.gh.pull(n) + if err != nil { + return err + } + if st := prState(&pr); st != "open" { + return fmt.Errorf("batch pull request #%d is %s now; nothing is written to it", n, st) + } + return nil +} + +func writeTemp(pattern, content string) (string, error) { + f, err := os.CreateTemp("", pattern) + if err != nil { + return "", err + } + if _, err := f.WriteString(content); err != nil { + f.Close() + os.Remove(f.Name()) + return "", err + } + if err := f.Close(); err != nil { + os.Remove(f.Name()) + return "", err + } + return f.Name(), nil +} diff --git a/cmd/backport-batch/report_test.go b/cmd/backport-batch/report_test.go new file mode 100644 index 000000000..a0b2942ff --- /dev/null +++ b/cmd/backport-batch/report_test.go @@ -0,0 +1,687 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "slices" + "strings" + "testing" + "time" + "unicode" +) + +// reportsPosted lists the report IDs the comments carry, in the order they +// were posted, whoever wrote them. +func reportsPosted(gh *fakeGH) []string { + var ids []string + for _, c := range gh.issueComments { + if m := reportMarkerRE.FindStringSubmatch(c.Body); m != nil { + ids = append(ids, m[1]) + } + } + return ids +} + +// bodyReportOf is the report a body on release-1.1, or on the line given, +// carries, failing the test without one. +func bodyReportOf(t *testing.T, body string, on ...line) storedReport { + t.Helper() + l := notifyLine + if len(on) > 0 { + l = on[0] + } + s, err := bodyReport(body, l) + if err != nil || s == nil { + t.Fatalf("the body carries no readable report: %v\n%s", err, body) + } + return *s +} + +func (g *fakeGH) find(n int) *pullRequest { + for i := range g.prs { + if g.prs[i].Number == n { + return &g.prs[i] + } + } + return nil +} + +// refusePushes makes the remote refuse every push, as a lost connection or +// a protected branch would, until it is called with false. +func (f *fixture) refusePushes(on bool) { + f.t.Helper() + hook := filepath.Join(f.remote, "hooks", "pre-receive") + if !on { + if err := os.Remove(hook); err != nil { + f.t.Fatal(err) + } + return + } + if err := os.WriteFile(hook, []byte("#!/bin/sh\necho refused >&2\nexit 1\n"), 0o755); err != nil { + f.t.Fatal(err) + } +} + +func mustRun(t *testing.T, f *fixture, gh *fakeGH, args ...string) string { + t.Helper() + code, stdout, stderr := f.exec(gh, args...) + if code != 0 { + t.Fatalf("%q = %d\n%s", args, code, stderr) + } + f.git("switch", "-q", "main") + return stdout +} + +// A report comment keeps each classification: one the next run replaced in +// the body is still read back by its ID, also once the batch merged, which +// nothing writes to. +func TestReportSurvivesContinuation(t *testing.T) { + f := newFixture(t) + f.lines() + f.onLine("release-1.1", "fix(app): a line-only five", map[string]string{"app/five.txt": "line\n"}) + mc5 := f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + r1 := bodyReportOf(t, gh.prs[0].Body).doc.ID + f.onLine("release-1.1", "fix(app): five, by hand\n\n(cherry picked from commit "+mc5+")", map[string]string{"app/five.txt": "5\n"}) + f.now = f.now.AddDate(0, 0, 1) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + r2 := bodyReportOf(t, gh.find(90).Body).doc.ID + if r1 == r2 || !slices.Equal(reportsPosted(gh), []string{r1, r2}) { + t.Fatalf("reports %q; want R1 %.12s, then R2 %.12s", reportsPosted(gh), r1, r2) + } + read := func(id, want string) { + t.Helper() + code, stdout, stderr := f.exec(gh, "report", "--pr", "90", "--id", id) + if code != 0 || !strings.Contains(stdout, want) { + t.Errorf("report --id %.12s = %d, want %q:\n%s%s", id, code, want, stdout, stderr) + } + } + for _, merged := range []bool{false, true} { + if merged { + mergeBatch(f, gh) + } + read(r1, "#5 deferred: conflict at ") + read(r2, "#5 skipped.on-line") + read(r2, "#6 picked") + } +} + +// Two runs that find the same post one report, and the body carries the one +// kept, whatever time and job the second ran in. A look-alike someone else +// wrote is no report, and a kept report edited since stops the reader. +func TestReportDedupAndTrust(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + t.Setenv("GITHUB_SERVER_URL", "https://github.com") + t.Setenv("GITHUB_REPOSITORY", "cozystack/cozystack") + t.Setenv("GITHUB_RUN_ID", "100") + + // The report the next run makes, read off a run that publishes nothing, + // planted by someone the batch does not trust. + preview := bodyReportOf(t, mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue")) + planted, err := reportComment(preview) + if err != nil { + t.Fatal(err) + } + look := issueComment{ID: 7, Body: planted} + look.User.Login, look.User.Type = "someone", "User" + gh.issueComments = append(gh.issueComments, look) + + f.now = f.now.AddDate(0, 0, 1) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + kept := bodyReportOf(t, gh.find(90).Body) + if kept.doc.ID != preview.doc.ID { + t.Fatalf("the run made %.12s, the preview %.12s", kept.doc.ID, preview.doc.ID) + } + f.now = f.now.AddDate(0, 0, 1) + t.Setenv("GITHUB_RUN_ID", "101") + mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + posted := reportsPosted(gh) + if n := len(slices.DeleteFunc(slices.Clone(posted), func(id string) bool { return id != kept.doc.ID })); n != 2 || len(posted) != 3 { + t.Errorf("reports %q; want the first, the look-alike and the run's own once", posted) + } + body := gh.find(90).Body + again := bodyReportOf(t, body) + rec, err := parseRecord(body, notifyLine) + if err != nil || again.raw != kept.raw || rec.Report != again.doc.ID || !strings.Contains(again.raw, `"run":"https://github.com/cozystack/cozystack/actions/runs/100"`) { + t.Errorf("the second run's body carries %s with record report %v (%v); want the kept bytes of run 100", again.raw, rec, err) + } + + // The reader takes the run's own comment, not the look-alike. + if code, stdout, stderr := f.exec(gh, "report", "--pr", "90", "--id", kept.doc.ID); code != 0 || !strings.Contains(stdout, "#6 picked") { + t.Errorf("report = %d:\n%s%s", code, stdout, stderr) + } + for i := range gh.issueComments { + c := &gh.issueComments[i] + if c.User.Login == "maintainer" && strings.Contains(c.Body, kept.doc.ID) { + c.Body = strings.Replace(c.Body, `"picked":[5,6]`, `"picked":[5]`, 1) + } + } + if code, stdout, stderr := f.exec(gh, "report", "--pr", "90", "--id", kept.doc.ID); code != 2 || !strings.Contains(stderr, "does not match") { + t.Errorf("report of an edited comment = %d, want 2:\n%s%s", code, stdout, stderr) + } +} + +// The report holds no pull request number, so a new batch's report is whole +// before GitHub numbers the pull request, and the reader finds it on +// whatever number it got. +func TestFirstReportIDIsKnownBeforeTheNumber(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{nextPR: 4242} + mustRun(t, f, gh, "run", "--line", "release-1.1", "--push") + inBody := bodyReportOf(t, gh.bodies[0]) + // Every list is one, empty or not, so equal reports are equal bytes: + // main here has no commit that merged no pull request. + if !strings.Contains(inBody.raw, `"other":[]`) { + t.Errorf("the report writes its empty list of other commits as something else:\n%s", inBody.raw) + } + if !slices.Equal(gh.commented, []int{4242}) || !slices.Equal(reportsPosted(gh), []string{inBody.doc.ID}) { + t.Fatalf("commented on %v with %q, want #4242 with %.12s", gh.commented, reportsPosted(gh), inBody.doc.ID) + } + found, bad, err := findReports(gh, 4242, notifyLine) + if err != nil || len(bad) != 0 || found[inBody.doc.ID].raw != inBody.raw { + t.Errorf("findReports = %v, %v, %v; want the body's report", found, bad, err) + } + if code, stdout, stderr := f.exec(gh, "report", "--pr", "4242"); code != 0 || !strings.Contains(stdout, "report "+inBody.doc.ID) || !strings.Contains(stdout, "#4242's body carries this report") { + t.Errorf("report --pr 4242 = %d:\n%s%s", code, stdout, stderr) + } +} + +// Every original main merged since the line branched is in exactly one +// category of the report, a commit that merged no pull request is counted +// apart, and the report says so whatever the run had to do. +func TestReportPartition(t *testing.T) { + f := newFixture(t) + f.lines() + gh := &fakeGH{labels: map[string][]int{labelSkip: {9}}} + // onNewer puts an original on release-1.1, as the newer line's batch + // would, so rule 6 lets it onto release-1.0. + onNewer := func(mc string, files map[string]string) { + f.onLine("release-1.1", "fix: by the newer batch\n\n(cherry picked from commit "+mc+")", files) + } + pr := func(n int, title string, files map[string]string, newer bool) { + mc := f.pr(n, title, "", files) + if newer { + onNewer(mc, files) + } + } + t.Setenv("GIT_COMMITTER_DATE", "2026-01-01T00:00:00Z") + f.pr(4, "fix(app): four, before the start", "", map[string]string{"app/four.txt": "4\n"}) + os.Unsetenv("GIT_COMMITTER_DATE") + f.setPolicy("release-1.0: 2026-06-01\n") + f.onLine("release-1.0", "fix(app): a line-only eleven", map[string]string{"app/eleven.txt": "line\n"}) + pr(5, "fix(app): five", map[string]string{"app/five.txt": "5\n"}, true) + pr(6, "fix(app): six, not on the newer line", map[string]string{"app/six.txt": "6\n"}, false) + pr(7, "fix(app)!: seven", map[string]string{"app/seven.txt": "7\n"}, true) + pr(8, "fix(platform): eight", map[string]string{migrationsDir + "42": "#!/bin/sh\n"}, true) + pr(9, "fix(app): nine", map[string]string{"app/nine.txt": "9\n"}, true) + mc10 := f.pr(10, "fix(app): ten", "", map[string]string{"app/ten.txt": "10\n"}) + onNewer(mc10, map[string]string{"app/ten.txt": "10\n"}) + f.onLine("release-1.0", "fix(app): ten\n\n(cherry picked from commit "+mc10+")", map[string]string{"app/ten.txt": "10\n"}) + pr(11, "fix(app): eleven", map[string]string{"app/eleven.txt": "11\n"}, true) + pr(12, "fix(ci): twelve", map[string]string{".github/workflows/twelve.yaml": "on: push\n"}, true) + pr(13, "fix(app): thirteen", map[string]string{"app/thirteen.txt": "13\n"}, true) + f.git("switch", "-q", "main") + f.write("app/direct.txt", "pushed\n") + f.git("add", "-A") + f.git("commit", "-q", "-m", "chore: a direct push") + direct := f.git("rev-parse", "HEAD") + f.git("push", "-q", "origin", "main") + + stdout := mustRun(t, f, gh, "run", "--line", "release-1.0", "--no-workflow-files", "--push") + const branch = "backport-batch/release-1.0-20261026" + gh.prs = []pullRequest{f.batchPR(90, "release-1.0", branch, stdout)} + pr(14, "fix(app): fourteen, after the batch", map[string]string{"app/fourteen.txt": "14\n"}, true) + stdout = mustRun(t, f, gh, "run", "--line", "release-1.0", "--defer", "13", "--reason", "fails the e2e", "--push") + + l, _ := parseLine("release-1.0") + s := bodyReportOf(t, stdout, l) + got, err := s.doc.categories() + if err != nil { + t.Fatal(err) + } + want := map[int]string{1: "skipped.type", 4: "skipped.before-start", 5: "picked", 6: "skipped.newer", 7: "skipped.breaking", + 8: "skipped.touches", 9: "skipped.skip-label", 10: "skipped.on-line", 11: "deferred", 12: "deferred", 13: "deferred", + 14: "missing", 9000: "skipped.type"} + for n, category := range want { + if got[n] != category { + t.Errorf("#%d is %q, want %q", n, got[n], category) + } + delete(got, n) + } + if len(got) != 0 { + t.Errorf("the report also names %v", got) + } + if !slices.Contains(s.doc.Other, direct) { + t.Errorf("the report does not count the direct push %.12s: %v", direct, s.doc.Other) + } + kinds := map[int]string{} + for _, e := range s.doc.Primary.Deferred { + kinds[e.Original] = fmt.Sprintf("commit=%v reason=%q workflows=%v", e.Commit != "", e.Reason, e.Workflows) + } + if kinds[11] != `commit=true reason="" workflows=false` || kinds[12] != `commit=false reason="" workflows=true` || kinds[13] != `commit=false reason="fails the e2e" workflows=false` { + t.Errorf("the deferred entries are %v", kinds) + } +} + +// A report too big for a comment is refused before anything is written: no +// push, no comment, no edit, and no replay of the earlier report either. +func TestReportTooLargeIsRefusedBeforePush(t *testing.T) { + f, gh, head := bigWaits(t, 7) + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--no-workflow-files", "--push") + if code != 2 || !strings.Contains(stderr, "the report comment is") { + t.Errorf("run --push = %d, want a refusal of the report:\n%s", code, stderr) + } + if f.remoteHead(batchBranch1) != head || len(gh.issueComments) != 0 || len(gh.edited) != 0 { + t.Errorf("the refused run moved the batch to %.12s, wrote %d comments and edited %q", f.remoteHead(batchBranch1), len(gh.issueComments), gh.edited) + } +} + +// bigWaits opens a batch whose report comment failed, so its body keeps a +// report no comment does, and then merges n originals that each wait on 50 +// workflow files with long names, which both the record and the report list +// in full. +func bigWaits(t *testing.T, n int) (*fixture, *fakeGH, string) { + t.Helper() + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{fail: map[string]error{"createComment": errString("HTTP 502")}} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 2 || len(gh.created) != 1 { + t.Fatalf("the first run = %d, want its comment to fail after the pull request opened\n%s", code, stderr) + } + f.git("switch", "-q", "main") + for i := range n { + files := map[string]string{} + for j := range filesKept { + files[fmt.Sprintf(".github/workflows/%s-%02d-%02d.yaml", strings.Repeat("w", 170), i, j)] = "on: push\n" + } + f.pr(10+i, fmt.Sprintf("fix(ci): wait %d", i), "", files) + } + f.now = f.now.AddDate(0, 0, 1) + return f, gh, f.remoteHead(batchBranch1) +} + +// A failure between any two writes leaves what was written readable, and the +// next run, whatever changed meanwhile, replays a report the body kept but no +// comment did, byte for byte, and posts nothing twice. A batch closed in +// between is never written again. +func TestReportReplayAfterEachFailure(t *testing.T) { + setup := func(t *testing.T) (*fixture, *fakeGH) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + return f, &fakeGH{} + } + // change moves a label, main, the clock and the run's flags between + // the failure and the retry, so the retry's report is another one. + change := func(f *fixture, gh *fakeGH) []string { + gh.labels = map[string][]int{labelSkip: {6}} + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + f.now = f.now.AddDate(0, 0, 1) + return []string{"run", "--line", "release-1.1", "--continue", "--no-workflow-files", "--push"} + } + once := func(t *testing.T, gh *fakeGH, ids ...string) { + t.Helper() + posted := reportsPosted(gh) + for _, id := range ids { + if n := len(slices.DeleteFunc(slices.Clone(posted), func(p string) bool { return p != id })); n != 1 { + t.Errorf("report %.12s is posted %d times: %q", id, n, posted) + } + } + } + + t.Run("a new batch whose pull request was not opened", func(t *testing.T) { + f, gh := setup(t) + gh.fail = map[string]error{"createPull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + if len(gh.prs) != 0 || len(gh.issueComments) != 0 || f.remoteHead(batchBranch1) == "" { + t.Errorf("after the failure: %d pull requests, %d comments, branch %q; want the branch alone", len(gh.prs), len(gh.issueComments), f.remoteHead(batchBranch1)) + } + mustRun(t, f, gh, change(f, gh)...) + if len(gh.prs) != 1 || len(reportsPosted(gh)) != 1 || bodyReportOf(t, gh.prs[0].Body).doc.ID != reportsPosted(gh)[0] { + t.Errorf("the retry opened %d pull requests with reports %q", len(gh.prs), reportsPosted(gh)) + } + }) + t.Run("a new batch whose report comment failed", func(t *testing.T) { + f, gh := setup(t) + gh.fail = map[string]error{"createComment": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 2 || !strings.Contains(stderr, "the body keeps it") { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + first := bodyReportOf(t, gh.find(90).Body) + mustRun(t, f, gh, change(f, gh)...) + second := bodyReportOf(t, gh.find(90).Body) + want, err := reportComment(first) + if err != nil || len(gh.issueComments) != 2 || gh.issueComments[0].Body != want || reportsPosted(gh)[1] != second.doc.ID || first.doc.ID == second.doc.ID { + t.Errorf("comments %q; want the first report replayed as it was, then the retry's", reportsPosted(gh)) + } + }) + t.Run("a continuation whose push failed", func(t *testing.T) { + f, gh := setup(t) + openBatch(t, f, gh) + first := reportsPosted(gh)[0] + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + f.refusePushes(true) + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push"); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + f.refusePushes(false) + attempted := reportsPosted(gh)[1] + if bodyReportOf(t, gh.find(90).Body).doc.ID != first { + t.Error("the body moved on although the push failed") + } + mustRun(t, f, gh, change(f, gh)...) + once(t, gh, first, attempted) + if len(reportsPosted(gh)) != 3 { + t.Errorf("reports %q; want the first, the attempted and the retry's", reportsPosted(gh)) + } + }) + t.Run("a continuation whose body edit failed", func(t *testing.T) { + f, gh := setup(t) + openBatch(t, f, gh) + first := reportsPosted(gh)[0] + before := f.remoteHead(batchBranch1) + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + gh.fail = map[string]error{"editPull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push"); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + attempted := reportsPosted(gh)[1] + if f.remoteHead(batchBranch1) == before || bodyReportOf(t, gh.find(90).Body).doc.ID != first { + t.Error("the push or the body is not where the failure left them") + } + mustRun(t, f, gh, change(f, gh)...) + once(t, gh, first, attempted) + if last := reportsPosted(gh)[len(reportsPosted(gh))-1]; bodyReportOf(t, gh.find(90).Body).doc.ID != last || len(reportsPosted(gh)) != 3 { + t.Errorf("reports %q; want the body to carry the retry's, posted last", reportsPosted(gh)) + } + }) + t.Run("a batch closed while the run writes to it", func(t *testing.T) { + f, gh := setup(t) + openBatch(t, f, gh) + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + gh.after = map[string]func(){"createComment": func() { gh.find(90).State = "closed" }} + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push") + if code != 2 || !strings.Contains(stderr, "batch pull request #90 is closed now; nothing is written to it") || len(gh.edited) != 0 { + t.Errorf("run = %d, edited %q; want 2 and no edit of the closed pull request:\n%s", code, gh.edited, stderr) + } + }) + t.Run("a batch closed before its report was posted", func(t *testing.T) { + f, gh := setup(t) + gh.fail = map[string]error{"createComment": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + kept := bodyReportOf(t, gh.find(90).Body) + gh.find(90).State = "closed" + gh.nextPR = 91 + mustRun(t, f, gh, change(f, gh)...) + if slices.Contains(gh.commented, 90) || slices.Contains(reportsPosted(gh), kept.doc.ID) { + t.Errorf("commented on %v, posting %q; want nothing on the closed #90", gh.commented, reportsPosted(gh)) + } + code, stdout, stderr := f.exec(gh, "report", "--pr", "90") + if code != 0 || !strings.Contains(stdout, "report "+kept.doc.ID) || !strings.Contains(stdout, "kept in the body only") { + t.Errorf("report --pr 90 = %d:\n%s%s", code, stdout, stderr) + } + }) + // Every comment is posted only after GitHub says the batch is still + // open, the replay's included, so a close between the two stops the + // second. + t.Run("a batch closed between the replay and the new report", func(t *testing.T) { + f, gh := setup(t) + gh.fail = map[string]error{"createComment": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + kept := bodyReportOf(t, gh.find(90).Body) + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + gh.after = map[string]func(){"createComment": func() { gh.find(90).State = "closed" }} + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push") + if code != 2 || !strings.Contains(stderr, "batch pull request #90 is closed now; nothing is written to it") || !slices.Equal(reportsPosted(gh), []string{kept.doc.ID}) || len(gh.edited) != 0 { + t.Errorf("run = %d, posted %q, edited %q; want 2 after the replay alone:\n%s", code, reportsPosted(gh), gh.edited, stderr) + } + }) +} + +// The reader says how a report stands against the pull request as GitHub +// shows it now, and claims publication only when the head or the body shows +// it: a failed push, a push whose answer was lost, a body that was not +// updated and a rebuild since all read as they are. +func TestReportPublicationStatus(t *testing.T) { + setup := func(t *testing.T) (*fixture, *fakeGH, string) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + return f, gh, reportsPosted(gh)[0] + } + status := func(t *testing.T, f *fixture, gh *fakeGH, id, want string) { + t.Helper() + code, stdout, stderr := f.exec(gh, "report", "--pr", "90", "--id", id) + if code != 0 || !strings.Contains(stdout, want) { + t.Errorf("report --id %.12s = %d, want %q:\n%s%s", id, code, want, stdout, stderr) + } + } + t.Run("a push the remote refused, and one it took whose answer was lost", func(t *testing.T) { + f, gh, first := setup(t) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + f.refusePushes(true) + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push"); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.refusePushes(false) + attempted := reportsPosted(gh)[1] + status(t, f, gh, attempted, "publication unconfirmed: #90's head is ") + status(t, f, gh, attempted, "its body carries report "+first[:12]) + // The same head reaching the remote is what a push GitHub took looks + // like when its answer never came back. + f.git("push", "-q", "origin", "HEAD:refs/heads/"+batchBranch1) + f.git("switch", "-q", "main") + status(t, f, gh, attempted, "the batch this report describes, but its body carries report "+first[:12]) + }) + t.Run("a push followed by a failed body edit", func(t *testing.T) { + f, gh, first := setup(t) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + gh.fail = map[string]error{"editPull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push"); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + status(t, f, gh, reportsPosted(gh)[1], "the batch this report describes, but its body carries report "+first[:12]) + status(t, f, gh, first, "#90's body carries this report") + }) + t.Run("a rebuild since", func(t *testing.T) { + f, gh, first := setup(t) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--defer", "6", "--reason", "x", "--push") + second := reportsPosted(gh)[1] + status(t, f, gh, first, "publication unconfirmed: #90's head is ") + status(t, f, gh, second, "#90's body carries this report") + }) +} + +// forged is a report on release-1.1 with what change sets, under an ID that +// matches it: only checking what it says stands in its way. +func forged(change func(d *reportDoc)) storedReport { + sha := strings.Repeat("a", 40) + d := reportDoc{Format: reportFormat, Line: "release-1.1", Diag: map[string][]string{}, Other: []string{sha}, + Built: builtFrom{Main: sha, Line: sha, Policy: sha, Batch: sha, At: "2026-10-26T06:00:00Z", Options: runOptions{Mode: modeNew}}} + d.Primary.Picked = []int{5} + change(&d) + d.ID = reportID(d) + return storedReport{doc: d, raw: reportJSON(d)} +} + +// reportRun runs report with gh faked; it reads nothing but GitHub. +func reportRun(gh *fakeGH, args ...string) (int, string, string) { + var stdout, stderr strings.Builder + a := &app{gh: gh, stdout: &stdout, stderr: &stderr, now: func() time.Time { return time.Date(2026, 10, 26, 6, 0, 0, 0, time.UTC) }} + code := a.main(append([]string{"report"}, args...)) + return code, stdout.String(), stderr.String() +} + +// A report is read only when everything it says is what a run writes, and +// it is for the line of the pull request it sits on: an ID that matches its +// content proves only that nobody changed it since it was hashed. +func TestReportRefusesWhatItCannotVouchFor(t *testing.T) { + for name, change := range map[string]func(d *reportDoc){ + "a line that is no release line": func(d *reportDoc) { d.Line = "release-1.1\n\x1b[2Jpicked" }, + "another line": func(d *reportDoc) { d.Line = "release-1.0" }, + "a main that is no commit": func(d *reportDoc) { d.Built.Main = "main\n#5 picked" }, + "a time that is none": func(d *reportDoc) { d.Built.At = "2026-10-26T" }, + "an unknown operation": func(d *reportDoc) { d.Built.Options.Mode = "new\x1b[2J" }, + "a run that is no Actions run": func(d *reportDoc) { d.Built.Run = "javascript:alert(1)" }, + "a tool that is no commit": func(d *reportDoc) { d.Built.Tool = "v1\n" }, + "a commit on main that is none": func(d *reportDoc) { d.Other = []string{"HEAD\x1b]0;x\x07"} }, + "a deferred commit that is none": func(d *reportDoc) { + d.Primary.Deferred = []deferredEntry{{Original: 6, Commit: "-x", Files: []string{"a"}}} + }, + "an original named twice": func(d *reportDoc) { d.Primary.Missing = []int{5} }, + "an original that is none": func(d *reportDoc) { d.Primary.Skipped.Type = []int{-1} }, + // Fields that speak of one another have to agree. + "a drop category with no drop": func(d *reportDoc) { d.Primary.Skipped.Dropped = []int{6} }, + "a stale original it has missing": func(d *reportDoc) { + d.Primary.Missing, d.Overlays.Stale = []int{7}, []int{7} + }, + "a stale original it puts nowhere": func(d *reportDoc) { d.Overlays.Stale = []int{8} }, + "a changed original it defers": func(d *reportDoc) { + d.Primary.Deferred = []deferredEntry{{Original: 9, Files: []string{}, Reason: "fails the e2e"}} + d.Overlays.Changed = []int{9} + }, + "a skip label on a picked original": func(d *reportDoc) { d.Overrides.Skip = []int{5} }, + "a skip-label category with no label": func(d *reportDoc) { d.Primary.Skipped.SkipLabel = []int{10} }, + "an include label on a rule 2 skip": func(d *reportDoc) { + d.Primary.Skipped.Type, d.Overrides.Include = []int{11}, []int{11} + }, + "both labels on one original": func(d *reportDoc) { + d.Primary.Skipped.SkipLabel, d.Overrides.Include, d.Overrides.Skip = []int{12}, []int{12}, []int{12} + }, + "a label on no original": func(d *reportDoc) { d.Overrides.Include = []int{13} }, + "a diagnostic of an original it puts nowhere": func(d *reportDoc) { d.Diag = map[string][]string{"14": {"incomplete"}} }, + "a workflow wait without the flag": func(d *reportDoc) { + d.Primary.Deferred = []deferredEntry{{Original: 15, Files: []string{".github/workflows/ci.yaml"}, Workflows: true}} + }, + "a close that still picks": func(d *reportDoc) { d.Built.Options.Mode = modeClose }, + "a parked original": func(d *reportDoc) { d.Primary.Parked = []json.RawMessage{json.RawMessage(`{"original":16}`)} }, + } { + s := forged(change) + c, _ := reportComment(storedReport{doc: forged(func(*reportDoc) {}).doc, raw: s.raw}) + comment := issueComment{ID: 1, Body: strings.Replace(c, forged(func(*reportDoc) {}).doc.ID, s.doc.ID, 1)} + comment.User.Login, comment.User.Type = "maintainer", "User" + for where, gh := range map[string]*fakeGH{ + "in the body": {prs: []pullRequest{notifyPR("20261026", "text\n\n"+reportBlock(s.raw)+"\n")}}, + "in a comment": {prs: []pullRequest{notifyPR("20261026", "text")}, issueComments: []issueComment{comment}}, + } { + if code, stdout, stderr := reportRun(gh, "--pr", "90", "--id", s.doc.ID); code != 2 || stdout != "" { + t.Errorf("%s, %s: report = %d, want 2 and nothing printed:\n%s%s", name, where, code, stdout, stderr) + } + } + } + if code, stdout, stderr := reportRun(&fakeGH{prs: []pullRequest{notifyPR("20261026", "text\n\n"+reportBlock(forged(func(*reportDoc) {}).raw)+"\n")}}, "--pr", "90"); code != 0 || !strings.Contains(stdout, "#5 picked") { + t.Errorf("report of a sound one = %d:\n%s%s", code, stdout, stderr) + } +} + +// What report prints stays one line per field and carries nothing a +// terminal acts on, whatever reached it. +func TestReportPrintsNothingThatDrivesATerminal(t *testing.T) { + const evil = "\n\x1b[2Jfake: #9 picked\u202e" + s := forged(func(d *reportDoc) { + d.Line, d.Built.At, d.Built.Options.Mode = "release-1.1"+evil, "2026-10-26"+evil, "new"+evil + d.Built.Main, d.Built.Tool, d.Built.Run, d.Built.Newer = "main"+evil, "tool"+evil, "run"+evil, "newer"+evil + d.Other = []string{"other" + evil} + d.Primary.Deferred = []deferredEntry{{Original: 6, Reason: "why" + evil}} + }) + var stdout strings.Builder + a := &app{stdout: &stdout} + pr := notifyPR("20261026", "") + a.printReport(&pr, s, nil, true) + for _, ln := range strings.Split(stdout.String(), "\n") { + if strings.HasPrefix(ln, "fake") || strings.ContainsFunc(ln, func(r rune) bool { return r != '\t' && unicode.IsControl(r) || unicode.Is(unicode.Cf, r) }) { + t.Errorf("report printed %q", ln) + } + } +} + +// The comment is made of what was checked: a line, a time and an ID that +// carry no markup into the summary GitHub renders. +func TestReportCommentIsBuiltFromCheckedFields(t *testing.T) { + for name, change := range map[string]func(d *reportDoc){ + "markup in the line": func(d *reportDoc) { d.Line = "release-1.1" }, + "markup in the time": func(d *reportDoc) { d.Built.At = "2026-10-26T06:00:00Z" }, + "a mention": func(d *reportDoc) { d.Line = "release-1.1 @cozystack/maintainers" }, + } { + if c, err := reportComment(forged(change)); err == nil { + t.Errorf("%s: reportComment made\n%s", name, c) + } + } + if _, err := reportComment(forged(func(*reportDoc) {})); err != nil { + t.Errorf("reportComment of a sound report: %v", err) + } +} + +// Anyone can comment on a public pull request: the reader takes a report +// from a comment only when the batch trusts its author, whatever ID it +// carries. +func TestReportReaderTakesOnlyTrustedComments(t *testing.T) { + s := forged(func(*reportDoc) {}) + c, err := reportComment(s) + if err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + login, kind string + read bool + }{ + {"someone", "User", false}, + {"renovate[bot]", "Bot", false}, + {"github-actions[bot]", "Bot", false}, + {"maintainer", "User", true}, + {defaultBatchBot, "Bot", true}, + } { + comment := issueComment{ID: 1, Body: c} + comment.User.Login, comment.User.Type = tc.login, tc.kind + gh := &fakeGH{prs: []pullRequest{notifyPR("20261026", "text")}, issueComments: []issueComment{comment}, bot: defaultBatchBot} + code, stdout, stderr := reportRun(gh, "--pr", "90", "--id", s.doc.ID) + if read := code == 0 && strings.Contains(stdout, "#5 picked"); read != tc.read || !tc.read && (code != 2 || !strings.Contains(stderr, "keeps no report")) { + t.Errorf("a report by %s: report = %d, read %v, want %v:\n%s%s", tc.login, code, read, tc.read, stdout, stderr) + } + } +} diff --git a/cmd/backport-batch/workflows_test.go b/cmd/backport-batch/workflows_test.go index 35f2d9f14..97547f6b5 100644 --- a/cmd/backport-batch/workflows_test.go +++ b/cmd/backport-batch/workflows_test.go @@ -110,7 +110,8 @@ func TestWorkflowChangesWaitUntilOnTheLine(t *testing.T) { if code != 0 || !strings.Contains(stdout, "1 originals on #90 change .github/workflows/ and wait for a maintainer") || strings.Contains(stdout, "deferred conflicts") { t.Errorf("notify = %d, %q; want #5 for a maintainer and no conflict\n%s", code, stdout, stderr) } - if len(gh.issueComments) != 1 || !strings.Contains(gh.issueComments[0].Body, "[#5](") || !strings.Contains(gh.issueComments[0].Body, "#changes-to-workflows") || strings.Contains(gh.issueComments[0].Body, "These conflicts") { + notes := slices.DeleteFunc(slices.Clone(gh.issueComments), func(c issueComment) bool { return !strings.HasPrefix(c.Body, notifyMarker) }) + if len(notes) != 1 || !strings.Contains(notes[0].Body, "[#5](") || !strings.Contains(notes[0].Body, "#changes-to-workflows") || strings.Contains(notes[0].Body, "These conflicts") { t.Errorf("the comment does not list #5 for a maintainer: %+v", gh.issueComments) } diff --git a/docs/release/backport-tasks/README.md b/docs/release/backport-tasks/README.md index eba2c3738..e4395480e 100644 --- a/docs/release/backport-tasks/README.md +++ b/docs/release/backport-tasks/README.md @@ -12,7 +12,7 @@ The [backport batch](../backports.md) is built by [`backport-batch`](../../../cm - Push nothing but the branch of the open batch pull request, `backport-batch/-` in this repository, and only what the task's check accepted. Never push to `main` or a `release-*` branch, and never open a second batch pull request: the scheduled run continues the open one. - Change no file the original pull request does not change. A conflict that cannot be resolved inside those files is not resolved: the original stays deferred. - Never rewrite the batch branch by hand. A plain push to it is a fast-forward; only `backport-batch run --defer --push` replaces it, leased on the head it rebuilt. -- Leave the batch pull request's title and body to the tool: it rewrites both on every run, and a hand edit is lost or, inside the `backport-batch` block, makes `check` refuse the body. +- Leave the batch pull request's title and body to the tool: it rewrites both on every run, and a hand edit is lost or, inside the `backport-batch` block, makes `run`, `check`, `fold` and `notify` refuse the body, and inside the `backport-batch-report` block makes `run --push` and `backport-batch report` refuse it. Leave the tool's report comments as they are too: each keeps one classification of the line, and an edited one makes `backport-batch report` refuse it. - When the task cannot be done within these rules, push nothing and leave a comment on the batch pull request saying why, so the next maintainer to look finds it. Nothing is lost by waiting: a red batch does not merge, and a deferred original goes to the next batch. ### Getting the tool @@ -26,4 +26,4 @@ cd ../backport-task go build -o /tmp/backport-batch ./cmd/backport-batch ``` -Run it from that worktree. Every command but `notify` fetches `origin` first, and `run`, `check`, `fold` and `lines` read the policy from `origin/main`. +Run it from that worktree. Every command but `notify` and `report` fetches `origin` first, and `run`, `check`, `fold` and `lines` read the policy from `origin/main`. diff --git a/docs/release/backports.md b/docs/release/backports.md index f63b6e089..0cc8db3b6 100644 --- a/docs/release/backports.md +++ b/docs/release/backports.md @@ -81,15 +81,17 @@ A run by hand trusts the open batch pull request the scheduled run opened as `co `./backport-batch lines` prints the lines, newest first. `run`, `check`, `fold` and `lines` read the policy block from this page on `main`, so until the page with its block is on `main` they need `--policy ` naming a ref that has it, and the scheduled run, which passes none, fails. -`run` picks what the rules take onto `backport-batch/-` and opens a draft pull request against the line. A pull request merged into `main` with a merge commit goes onto the batch as its own commits, oldest first, each picked with `git cherry-pick -x --signoff` so it keeps its author, its message and its sign-offs and gains the sign-off of whoever runs the batch; commits that merged `main` into the pull request's branch are left out. A merge commit joins them to the batch: it carries the original's merge message from `main` and ends in a `Backported-from: ` trailer. A pull request squash-merged into `main` goes on as one pick. When any commit of an original conflicts, nothing of that original stays on the batch. The batch pull request is merged into the line with a merge commit, so all of this reaches the line as it is. Its body lists the originals with how many commits each brought and which of them differ from the commit they came from, the conflicts it deferred with the commit that failed and its files, and what it skipped under which rule. While that pull request is open, the next run continues it, `--continue` reading its head, rather than opening another. CI on the batch pull request runs the line's build and tests, and a maintainer merges it. Since rule 6 reads the newer line itself, an older line takes a fix in the batch after the newer line's batch with that fix has merged. +`run` picks what the rules take onto `backport-batch/-` and opens a draft pull request against the line. A pull request merged into `main` with a merge commit goes onto the batch as its own commits, oldest first, each picked with `git cherry-pick -x --signoff` so it keeps its author, its message and its sign-offs and gains the sign-off of whoever runs the batch; commits that merged `main` into the pull request's branch are left out. A merge commit joins them to the batch: it carries the original's merge message from `main` and ends in a `Backported-from: ` trailer. A pull request squash-merged into `main` goes on as one pick. When any commit of an original conflicts, nothing of that original stays on the batch. The batch pull request is merged into the line with a merge commit, so all of this reaches the line as it is. Its body lists the originals with how many commits each brought and which of them differ from the commit they came from, the conflicts it deferred with the commit that failed and its files, and what it skipped under which rule, and ends with the [line's record and the run's report](#the-lines-record-and-reports); `--push` prints the pull request's URL. While that pull request is open, the next run continues it, `--continue` reading its head, rather than opening another. CI on the batch pull request runs the line's build and tests, and a maintainer merges it. Since rule 6 reads the newer line itself, an older line takes a fix in the batch after the newer line's batch with that fix has merged. A deferred conflict waits for the next batch. To land it sooner, pick the original's commits onto the batch head with `git cherry-pick -x`, resolving each conflict, and join them to the batch head with a merge commit that names the original; an original squash-merged into `main` is one pick of its commit there, with no merge. Check that merge or that pick with `./backport-batch fold --line --original --commit `, and push it onto the batch branch, as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes. CI runs `backport-batch verify` on every push to a batch pull request, which holds every original on it to the same shape. When CI on the batch pull request fails, the original it comes from is taken out with `./backport-batch run --line --defer --reason --push` and goes to the next batch, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the rest of the batch merges without it. Before a patch release or an rc is cut from a line, `./backport-batch check --line --cut` has to exit 0. A cut tags the line, so the check fails while a pull request the rules take is not on the line, including one that is only on the open batch pull request: merge the batch pull request first. It prints the URLs, and prints what the batch deferred as warnings, since deferring is a maintainer's call; so is a pull request labelled `backport/skip` that the open batch still carries. An original that waits on the batch for a maintainer fails the cut, also when it is all the batch has left. A cut that passes says so. Without `--cut`, as the scheduled run uses it, a change on the open batch pull request counts as carried. -### The line's record +### The line's record and reports -What the batch decided about a line, the conflicts it deferred, the originals taken out with `--defer` and those waiting for a maintainer, is kept in a `backport-batch` block in the body of the line's newest batch pull request, its record. `backport-batch` reads the record whether that pull request is open, merged or closed, and writes it only while the pull request is open: a merged or closed batch pull request is never edited or commented on again. So a conflict the batch deferred is still a warning in `check` after the batch merged, until a later batch tries it again. A run that picks nothing while no batch is open writes nothing, and says so. A record `backport-batch` cannot read, or one written by someone it does not trust, stops it rather than let it fall back to an older one; how it decides is in [its README](../../cmd/backport-batch/README.md#the-lines-record). +What the batch decided about a line, the conflicts it deferred, the originals taken out with `--defer` and those waiting for a maintainer, is kept in a `backport-batch` block in the body of the line's newest batch pull request, its record. `backport-batch` reads the record whether that pull request is open, merged or closed, and writes it only while the pull request is open: a merged or closed batch pull request is never edited or commented on again. So a conflict the batch deferred is still a warning in `check` after the batch merged, until a later batch tries it again. A run that picks nothing while no batch is open writes nothing, and says so. A record `backport-batch` cannot read, or one written by someone it does not trust, stops it rather than let it fall back to an older one; how it decides is in [its README](../../cmd/backport-batch/README.md#the-lines-record-and-reports). + +Each run also reports where every pull request merged into `main` since the line branched stands, and what it read to decide. The body carries the latest report under the record, and a comment the tool posts on the open batch pull request keeps each one, so an earlier classification is still there after a later run replaced the body, and after the batch merged. `./backport-batch report --pr ` prints the report the body carries, and `--id ` an earlier one; it reads GitHub only and writes nothing. A run that picks nothing while no batch is open has nowhere to keep its report: the next batch's report has the line's state. ### Changes to workflows From 79b335e461797d411caee4456caf3d61ff2d8d8f Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 9 Nov 2026 15:21:02 +0000 Subject: [PATCH 399/518] fix(release): close the batch --defer leaves empty, recording why while it is open docs/release/backport-tasks/failed-ci.md says to defer the culprit with run --defer --push. When the culprit was all the batch carried, the push stopped with "close batch pull request #N instead", and closing it by hand left nothing that said why: once closed, the deferred original was simply missing. The run now closes the batch itself, and does every write while the pull request is open: the report, then the body, whose record keeps the deferrals and says the batch is closing, at the head the rebuild started from. The close and the deletion of the branch go only while GitHub and the remote still show that head, and the deletion is leased on it. A close that stops part way is finished by the next run --push, before anything else, and by no other command; a head the close did not freeze is refused, never adopted. The run that finishes it stops there, and fails when it was asked for --defer, since no open batch is left to defer on. Recovery never writes the closed pull request. A same-day batch that reuses the branch name and fails to open its pull request leaves a branch the next run names for a person to delete. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 8 +- cmd/backport-batch/assemble.go | 163 +++++++++++- cmd/backport-batch/body.go | 3 + cmd/backport-batch/check.go | 28 +++ cmd/backport-batch/defer_test.go | 306 +++++++++++++++++++++++ cmd/backport-batch/fixture_test.go | 30 ++- cmd/backport-batch/github.go | 6 + cmd/backport-batch/main.go | 3 +- cmd/backport-batch/overrides_test.go | 7 + cmd/backport-batch/record_test.go | 11 + cmd/backport-batch/report_test.go | 9 - docs/release/backport-mechanics.md | 2 +- docs/release/backport-tasks/failed-ci.md | 2 + docs/release/backports.md | 2 +- 14 files changed, 560 insertions(+), 20 deletions(-) diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index 7a1acabf1..768f21b2f 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -35,9 +35,9 @@ The pull request body goes to stdout, built from the line's own `.github/PULL_RE `--no-workflow-files` is for a token that may not push a change to `.github/workflows/`, which the scheduled run's App token may not. An original that changes anything there is not picked: the body lists it under "Waiting for a maintainer" with those files, and the `backport-batch` block records it with `"workflows": true`, so `check`, `notify` and a run that continues the batch know it waits. It is never a conflict. A run without the flag, a maintainer's with their own token, picks it like any other original, onto the open batch when it continues one. -`--defer N[,N] --reason TEXT` takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off `origin/` without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got `backport/skip` after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the `backport-batch` block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. +`--defer N[,N] --reason TEXT` takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off `origin/` without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got `backport/skip` after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the `backport-batch` block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. When it takes out everything the batch carries, there is nothing left to push, and `--push` closes the batch pull request instead, as [the line's record](#the-lines-record-and-reports) describes. -`--push` pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with `gh pr create --draft`, or updates the open batch pull request with `gh pr edit`, and prints its URL on stderr. On the open batch it first posts the run's report as a comment, so a body never names a report nobody can read; a new batch carries the report in the body it opens with, and the comment follows. Before every write to the pull request it reads the pull request again, and stops with exit 2 once it is merged or closed. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the `run --onto --push` that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no `backport-batch` block either: someone wrote that body by hand, and the batch is theirs to merge or close. After `--defer` alone it replaces the branch, with `--force-with-lease=refs/heads/:`, where head is what `git ls-remote` reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. +`--push` pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with `gh pr create --draft`, or updates the open batch pull request with `gh pr edit`, and prints its URL on stderr. On the open batch it first posts the run's report as a comment, so a body never names a report nobody can read; a new batch carries the report in the body it opens with, and the comment follows. Before every write to the pull request it reads the pull request again, and stops with exit 2 once it is merged or closed. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the `run --onto --push` that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no `backport-batch` block either: someone wrote that body by hand, and the batch is theirs to merge or close. After `--defer` alone it replaces the branch, with `--force-with-lease=refs/heads/:`, where head is what `git ls-remote` reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. A record that says the batch is being closed is acted on before anything else, and only by `run --push`. ### check @@ -111,6 +111,10 @@ The record is only as trustworthy as the body it sits in. Anyone with write acce The record holds only what later commands act on. Every run that classifies the line also makes a report, one line of JSON in a `backport-batch-report` block: every original merged into `main` since the line branched in exactly one category, picked, missing, deferred with what it was deferred for, or skipped with the rule that skipped it, the commits on `main` that merged no pull request, the labels, and what the classification was made of: the heads of `main`, the line and the newer line, the policy, the batch head it describes, the tool's commit, the Actions run and the flags. Its ID is a hash of all that but the time and the run, so two runs that find the same make the same report, and it names no pull request, so a new batch's report is whole before GitHub numbers the pull request. The body carries the latest report beside the record, and a comment on the open batch pull request keeps each one: `` and the same block, folded away. A report already posted is not posted again, and the tool never edits or deletes such a comment, so a report the next run replaces in the body is still read back by `report`, also after the batch merged. A comment that looks like one but whose author the batch does not trust is not read. When a run stops after writing the body but before posting its comment, the next `run --push` on the open batch posts that report first, byte for byte; once the batch is closed nothing is written to it, and the report stays in its body. A run that picks nothing while no batch is open keeps no report. +A `--defer` that leaves the batch carrying nothing closes it, and writes everything while the pull request is still open: the report comment, then the body, whose record keeps the deferrals and says the batch is closing, with its number, branch and the head the rebuild started from. It then reads the pull request and the branch again, closes the pull request while both are at that head, and deletes the branch with `--force-with-lease` on it, printing `closed `. The next batch tries what was deferred again. A close that stops part way is finished by the next `run --push`, before it does anything else: it closes and deletes when the pull request is open at the frozen head, deletes the branch when the pull request is closed and the branch still at that head, and goes on as usual once both are gone. A run that finishes a close stops there, with exit 0, or with exit 2 when it was asked for `--defer`, which it does not apply. It never takes a head it did not freeze: with the branch moved, it stops with exit 2 and names both heads. `run` without `--push`, `check` and `fold` stop with exit 2 while the pull request is still open; once it is closed they go on, warning about a branch the close left, and `notify` says nothing. A closed pull request is never written to, so its record says it was closing for good, and a closed pull request whose branch is gone counts as closed. + +One way for the branch to move is a new batch of the same day: it reuses the branch name, and if its push succeeds and opening its pull request fails, the closed batch is still the newest record while its branch now holds the new batch. The next run stops on it, naming the branch, and so does `--onto` with its head. Check what it carries before deleting it with `git push origin --delete backport-batch/-`: the next run picks again what the tool put there from the line, but not a commit someone pushed onto it, such as a resolved conflict. Separate writes to GitHub are not atomic: the re-reads and the leases refuse a race they see, but cannot prevent one. + ## What CI verifies The tool does not build or test anything. CI on the batch pull request runs the line's own build and tests against what was picked, and `verify` against its shape, and a maintainer reviews and merges it. A deferred conflict waits for the next batch unless someone resolves it, checks the resolution with `fold` and pushes it, and then CI runs again. diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index a2adc5c1d..3ac19c4f9 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -119,6 +119,9 @@ func (a *app) run(args []string) int { if err != nil { return a.fail(err) } + if goOn, code := a.resumeClose(b, *push, deferred != nil); !goOn { + return code + } if deferred != nil { return a.rebuild(b, deferred, reason, *push) } @@ -266,6 +269,13 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r if r.shapes, err = summarize(b, head, r.onBranch); err != nil { return a.fail(err) } + if rebuilt && head == b.lineHead && b.pr != nil { + // Deferring everything the batch carries leaves nothing to push: the + // batch is closed, and its record says so first, with the head the + // close is leased on. + r.closing = &closing{PR: b.pr.Number, Branch: branch, Head: b.prHead} + r.opts.Mode = modeClose + } r.built = builtFrom{Main: b.mainHead, Line: b.lineHead, Newer: b.newerHead, Policy: b.policyAt, Batch: head, Tool: a.tool, Run: runURL(os.LookupEnv), At: a.now().UTC().Format(time.RFC3339), Options: r.opts} doc, err := buildReport(b, r) @@ -321,7 +331,7 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r // act on, with the full classification left to the report it names. func (r *report) lineRecord(doc reportDoc) lineRecord { built := doc.Built - return lineRecord{Version: 2, Line: doc.Line, Built: &built, Report: doc.ID, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: []droppedEntry{}, Stale: slices.Clone(doc.Overlays.Stale)} + return lineRecord{Version: 2, Line: doc.Line, Built: &built, Report: doc.ID, Closing: r.closing, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: []droppedEntry{}, Stale: slices.Clone(doc.Overlays.Stale)} } // branchDate is the date a batch branch was opened under. @@ -496,8 +506,8 @@ func (a *app) publish(b *batch, p publication, rebuilt bool) error { return fmt.Errorf("batch pull request #%d has no %s block, so backport-batch did not write it; merge or close it, and the next run opens a batch of its own", b.pr.Number, deferredFence) } if p.head == b.lineHead { - if rebuilt { - return fmt.Errorf("without what is deferred, %s carries nothing beyond %s; close batch pull request #%d instead", p.branch, b.line.branch(), b.pr.Number) + if rebuilt && p.record.Closing != nil { + return a.closeBatch(b, p) } return fmt.Errorf("%s carries nothing beyond %s; nothing to push", p.branch, b.line.branch()) } @@ -528,7 +538,13 @@ func (a *app) publish(b *batch, p publication, rebuilt bool) error { refusal := fmt.Sprintf("%s on %s is at %.12s, which %.12s does not descend from; refusing a non-fast-forward push", p.branch, remote, pushed, p.head) // A run that stopped between its push and opening the pull request // leaves the branch, and the next run that day builds it anew. - if b.pr == nil { + switch c := b.src.carrier; { + case b.pr != nil: + case c != nil && !b.src.open && c.Head.Ref == p.branch: + // Continuing a batch that merged or closed would publish + // what nobody can merge any more. + refusal += fmt.Sprintf("; it is the branch of batch pull request #%d, which is %s; the next run picks again what the tool put on it, but not a commit someone pushed there, such as a resolved conflict: check what %.12s carries, then delete it with git push origin --delete %s and run again", c.Number, prState(c), pushed, p.branch) + default: refusal += fmt.Sprintf("; no batch pull request is open on it, and backport-batch run --line %s --onto %s --push continues it and opens one", b.line.branch(), pushed) } return errors.New(refusal) @@ -636,6 +652,140 @@ func pullURL(pr *pullRequest) string { return fmt.Sprintf("%spull/%d", ourRepoURL, pr.Number) } +// closeBatch closes the open batch that --defer left carrying nothing. Every +// write happens while it is open: the report, then the body whose record +// says it is closing and at which head. The close and the deletion of its +// branch then go only while both are still at that head. +func (a *app) closeBatch(b *batch, p publication) error { + c := *p.record.Closing + if err := p.fits(); err != nil { + return err + } + if err := a.keepReports(b, &p); err != nil { + return err + } + name, err := writeTemp("backport-batch-body-*.md", p.body) + if err != nil { + return err + } + defer os.Remove(name) + if err := a.stillOpen(c.PR); err != nil { + return err + } + if err := a.gh.editPull(c.PR, p.title, name); err != nil { + return fmt.Errorf("recording the close of #%d: %w", c.PR, err) + } + return a.finishClose(b.line, pullURL(b.pr), c) +} + +// finishClose closes the batch pull request a record says is closing, and +// deletes its branch, each only while it is at the head the record froze. +func (a *app) finishClose(l line, url string, c closing) error { + pr, err := a.gh.pull(c.PR) + if err != nil { + return fmt.Errorf("closing #%d, reading it again: %w", c.PR, err) + } + if st := prState(&pr); st != "open" || pr.Head.SHA != c.Head { + return fmt.Errorf("closing #%d: it is %s at %.12s, not open at %.12s, which its record froze", c.PR, st, pr.Head.SHA, c.Head) + } + heads, err := remoteBatchHeads(l) + if err != nil { + return err + } + if h := heads[c.Branch]; h != c.Head { + return fmt.Errorf("closing #%d: %s on %s is at %.12s, not at %.12s, which its record froze", c.PR, c.Branch, remote, h, c.Head) + } + if err := a.gh.closePull(c.PR); err != nil { + return fmt.Errorf("closing #%d: %w; backport-batch run --line %s --push finishes it", c.PR, err, l.branch()) + } + if err := a.deleteBranch(l, c); err != nil { + return err + } + fmt.Fprintf(a.stderr, "closed %s: without what is deferred it carries nothing beyond %s; the next batch tries them again\n", url, l.branch()) + return nil +} + +// deleteBranch deletes the branch of a closed batch, leased on the head it +// was closed at: one someone pushed since is a person's to delete or keep. +func (a *app) deleteBranch(l line, c closing) error { + heads, err := remoteBatchHeads(l) + if err != nil { + return err + } + switch h, ok := heads[c.Branch]; { + case !ok: + return nil + case h != c.Head: + return movedBranch(c, h) + } + if _, err := git(deleteArgs(c.Branch, c.Head)...); err != nil { + return fmt.Errorf("deleting %s, left by the close of #%d: %w; backport-batch run --line %s --push deletes it", c.Branch, c.PR, err, l.branch()) + } + return nil +} + +// takenBranch is the refusal for a branch a close left that an open batch +// pull request now uses, whatever it holds, or nil. +func takenBranch(b *batch, c closing) error { + if b.pr == nil || b.pr.Head.Ref != c.Branch { + return nil + } + return fmt.Errorf("batch pull request #%d is open on %s, which the close of #%d left; the branch is that pull request's now, and the close leaves it alone", b.pr.Number, c.Branch, c.PR) +} + +func movedBranch(c closing, h string) error { + return fmt.Errorf("batch pull request #%d was closed at %.12s, but %s is at %.12s now; the next run picks again what the tool put on it, but not a commit someone pushed there, such as a resolved conflict: check what %.12s carries, then delete it with git push origin --delete %s, and run again", c.PR, c.Head, c.Branch, h, h, c.Branch) +} + +// resumeClose finishes a close the record says is under way, before a run +// does anything else, and only under --push. It says whether the run may go +// on: only once the batch is closed and its branch gone. A run that finishes +// one stops there, and fails when it was asked to change the batch. +func (a *app) resumeClose(b *batch, push, changing bool) (bool, int) { + if b.src.rec == nil || b.src.rec.Closing == nil { + return true, 0 + } + c := *b.src.rec.Closing + heads, err := remoteBatchHeads(b.line) + if err != nil { + return false, a.fail(err) + } + h, onRemote := heads[c.Branch] + switch { + case b.src.open && h != c.Head: + return false, a.fail(fmt.Errorf("batch pull request #%d is being closed, but %s is at %.12s, not at %.12s, which its record froze; a person closes it, or pushes the branch back, by hand", c.PR, c.Branch, h, c.Head)) + case b.src.open && !push: + return false, a.fail(fmt.Errorf("batch pull request #%d is being closed: closing it and deleting %s are left; backport-batch run --line %s --push finishes it", c.PR, c.Branch, b.line.branch())) + case b.src.open: + if err := a.finishClose(b.line, pullURL(b.src.carrier), c); err != nil { + return false, a.fail(err) + } + return false, a.finished(c, changing) + case !onRemote: + return true, 0 + case takenBranch(b, c) != nil: + return false, a.fail(takenBranch(b, c)) + case h != c.Head: + return false, a.fail(movedBranch(c, h)) + case !push: + return false, a.fail(fmt.Errorf("batch pull request #%d was closed, but %s is still at %.12s; backport-batch run --line %s --push deletes it", c.PR, c.Branch, h, b.line.branch())) + } + if err := a.deleteBranch(b.line, c); err != nil { + return false, a.fail(err) + } + fmt.Fprintf(a.stderr, "deleted %s, which the close of #%d left\n", c.Branch, c.PR) + return false, a.finished(c, changing) +} + +// finished is the exit of a run that finished a close: a change it was asked +// for needs an open batch pull request, and none is left to make it on. +func (a *app) finished(c closing, changing bool) int { + if !changing { + return 0 + } + return a.fail(fmt.Errorf("finished the close of #%d and made no other change: what this run was asked to change needs an open batch pull request, which the next run opens", c.PR)) +} + // pushArgs pushes head to branch: as a fast-forward, which the remote checks, // or, given a lease, over exactly that commit and no other. func pushArgs(branch, head, lease string) []string { @@ -645,3 +795,8 @@ func pushArgs(branch, head, lease string) []string { } return append(args, "--end-of-options", remote, head+":refs/heads/"+branch) } + +// deleteArgs deletes branch, only while the remote has it at lease. +func deleteArgs(branch, lease string) []string { + return []string{"push", "--quiet", "--force-with-lease=refs/heads/" + branch + ":" + lease, "--end-of-options", remote, ":refs/heads/" + branch} +} diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index ae34be876..1bd2024f7 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -248,6 +248,9 @@ type report struct { built builtFrom // blocks are the record and the report the body ends with. blocks string + // closing is the close of the open batch this run leaves carrying + // nothing. + closing *closing } // ruleName is how output names what kept an original out: the policy's diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index 9398a37e6..d7bc7e213 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -47,6 +47,9 @@ func (a *app) check(args []string) int { if err != nil { return a.fail(err) } + if err := a.warnLeftBranch(b); err != nil { + return a.fail(err) + } disp, err := b.dispositions(*cut) if err != nil { return a.fail(err) @@ -90,6 +93,28 @@ func (a *app) check(args []string) int { return 1 } +// warnLeftBranch names the branch a closed batch's close left behind. +func (a *app) warnLeftBranch(b *batch) error { + if b.src.open || b.src.rec == nil || b.src.rec.Closing == nil { + return nil + } + c := b.src.rec.Closing + heads, err := remoteBatchHeads(b.line) + if err != nil { + return err + } + switch h, ok := heads[c.Branch]; { + case !ok: + case takenBranch(b, *c) != nil: + a.warn("%v", takenBranch(b, *c)) + case h == c.Head: + a.warn("batch pull request #%d is closed, but its branch %s is still at %.12s; backport-batch run --line %s --push deletes it", c.PR, c.Branch, h, b.line.branch()) + default: + a.warn("%v", movedBranch(*c, h)) + } + return nil +} + // outcome is what check makes of one original. type outcome int @@ -196,6 +221,9 @@ func (a *app) fold(args []string) int { if c := b.src.closingOpen(); c != nil { return a.fail(fmt.Errorf("batch pull request #%d is being closed; backport-batch run --line %s --push finishes it", c.PR, l.branch())) } + if err := a.warnLeftBranch(b); err != nil { + return a.fail(err) + } why, err := refuseFold(b, *number, *commit) if err != nil { return a.fail(err) diff --git a/cmd/backport-batch/defer_test.go b/cmd/backport-batch/defer_test.go index 59c736083..d0f8798c8 100644 --- a/cmd/backport-batch/defer_test.go +++ b/cmd/backport-batch/defer_test.go @@ -378,3 +378,309 @@ func TestSkipAfterThePick(t *testing.T) { t.Errorf("check = %d\n%s", code, stderr) } } + +// closeWorld is a batch carrying #5 alone, opened and pushed. +func closeWorld(t *testing.T) (*fixture, *fakeGH, string) { + t.Helper() + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{} + h1 := openBatch(t, f, gh) + return f, gh, h1 +} + +// Deferring everything a batch carries closes it: the record says why and +// what it froze while the pull request is still open, then the pull request +// is closed and its branch deleted, leased on that head. The next batch +// tries what was deferred again. +func TestDeferEverythingClosesTheBatch(t *testing.T) { + f, gh, h1 := closeWorld(t) + var log []string + gh.after = map[string]func(){ + "createComment": func() { log = append(log, "comment") }, + "editPull": func() { log = append(log, "edit") }, + "closePull": func() { log = append(log, "close, branch at "+f.remoteHead(batchBranch1)) }, + } + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "fails the app e2e", "--push") + if code != 0 || !strings.Contains(stderr, "closed https://github.com/cozystack/cozystack/pull/90: without what is deferred it carries nothing beyond release-1.1; the next batch tries them again") { + t.Fatalf("run --defer of the last pick = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + if want := []string{"comment", "edit", "close, branch at " + h1}; !slices.Equal(log, want) || f.remoteHead(batchBranch1) != "" || gh.find(90).State != "closed" { + t.Errorf("writes %q, branch at %q, #90 %s; want %q, the branch deleted and #90 closed", log, f.remoteHead(batchBranch1), gh.find(90).State, want) + } + rec, err := parseRecord(gh.find(90).Body, notifyLine) + if err != nil || rec.Closing == nil || *rec.Closing != (closing{PR: 90, Branch: batchBranch1, Head: h1}) || rec.Built.Options.Mode != modeClose || rec.deferred()[5].Reason != "fails the app e2e" { + t.Errorf("the closed batch's record is %+v, %v; want #5 deferred and the close it froze", rec, err) + } + if !strings.Contains(stdout, `"closing":{"pr":90,"branch":"`+batchBranch1+`","head":"`+h1+`"}`) { + t.Errorf("the body printed does not say what the close froze:\n%s", stdout) + } + code, _, stderr = f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "warning: deferred on #90 (closed) to the next batch: https://github.com/cozystack/cozystack/pull/5 ") { + t.Errorf("check = %d, want 0 with #5 deferred:\n%s", code, stderr) + } + gh.nextPR = 91 + mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + if got := f.subjects("origin/release-1.1", f.remoteHead(batchBranch1)); !slices.Equal(got, []string{"fix(app): five (#5)"}) || gh.find(91) == nil { + t.Errorf("the next batch carries %q, want #5 tried again on a new pull request", got) + } +} + +// A close that stopped part way is finished by run --push, and by nothing +// else, and never past a head it did not freeze; nothing is ever written to +// the closed pull request. +func TestCloseRecovery(t *testing.T) { + deferAll := []string{"run", "--line", "release-1.1", "--defer", "5", "--reason", "fails the app e2e", "--push"} + push := []string{"run", "--line", "release-1.1", "--continue", "--push"} + // moveBranch pushes a commit onto the batch branch, or recreates it, as + // someone else would. + moveBranch := func(f *fixture, base string) string { + f.git("switch", "-q", "--detach", base) + f.git("commit", "-q", "--allow-empty", "-m", "pushed since") + h2 := f.git("rev-parse", "HEAD") + f.git("push", "-q", "origin", "HEAD:refs/heads/"+batchBranch1) + f.git("switch", "-q", "main") + return h2 + } + writes := func(gh *fakeGH) int { return len(gh.issueComments) + len(gh.edited) } + + t.Run("stopped after the record, then the branch moved", func(t *testing.T) { + f, gh, h1 := closeWorld(t) + gh.after = map[string]func(){"editPull": func() { gh.fail = map[string]error{"pull": errString("HTTP 502")} }} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 || !strings.Contains(stderr, "HTTP 502") { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + h2 := moveBranch(f, h1) + before := writes(gh) + if code, _, stderr := f.exec(gh, push...); code != 2 || !strings.Contains(stderr, h2[:12]) { + t.Errorf("run --push over a moved branch = %d, want 2 naming it:\n%s", code, stderr) + } + if gh.find(90).State != "open" || f.remoteHead(batchBranch1) != h2 || writes(gh) != before { + t.Error("the run acted on a branch that moved since the close froze it") + } + }) + // GitHub and the remote are read apart, and either may be the first to + // show that someone pushed while the record was written. + for _, seen := range []string{"by the remote", "by GitHub"} { + t.Run("the branch moved while the record was written, seen "+seen, func(t *testing.T) { + f, gh, h1 := closeWorld(t) + gh.after = map[string]func(){"editPull": func() { + gh.remoteHead = func(string) string { return "" } + if seen == "by GitHub" { + gh.find(90).Head.SHA = strings.Repeat("b", 40) + } else { + moveBranch(f, h1) + } + }} + code, _, stderr := f.exec(gh, deferAll...) + f.git("switch", "-q", "main") + if code != 2 || !strings.Contains(stderr, "which its record froze") || gh.find(90).State != "open" || f.remoteHead(batchBranch1) == "" { + t.Errorf("run = %d, #90 %s, branch %q; want 2, #90 open and the branch kept:\n%s", code, gh.find(90).State, f.remoteHead(batchBranch1), stderr) + } + }) + } + t.Run("the close failed", func(t *testing.T) { + f, gh, _ := closeWorld(t) + gh.fail = map[string]error{"closePull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + before := writes(gh) + // Anything but run --push leaves it as it is. + for _, args := range [][]string{ + {"run", "--line", "release-1.1", "--continue"}, + {"check", "--line", "release-1.1"}, + {"fold", "--line", "release-1.1", "--original", "5", "--commit", strings.Repeat("a", 40)}, + {"notify", "--line", "release-1.1", "--push"}, + } { + if code, _, stderr := f.exec(gh, args...); code != 2 || !strings.Contains(stderr, "being closed") { + t.Errorf("%q = %d, want 2:\n%s", args, code, stderr) + } + f.git("switch", "-q", "main") + } + if writes(gh) != before || gh.find(90).State != "open" { + t.Error("a command other than run --push wrote to the batch") + } + code, _, stderr := f.exec(gh, push...) + if code != 0 || gh.find(90).State != "closed" || f.remoteHead(batchBranch1) != "" || writes(gh) != before { + t.Errorf("run --push = %d, #90 %s, branch %q, %d writes; want the close finished and nothing else\n%s", code, gh.find(90).State, f.remoteHead(batchBranch1), writes(gh)-before, stderr) + } + }) + // A run that finishes a close stops there: the change it was asked for + // needs an open batch pull request, and none is left to make it on. + t.Run("a change asked for while the close was left", func(t *testing.T) { + f, gh, _ := closeWorld(t) + gh.fail = map[string]error{"closePull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + code, _, stderr := f.exec(gh, deferAll...) + f.git("switch", "-q", "main") + if code != 2 || !strings.Contains(stderr, "finished the close of #90 and made no other change") || gh.find(90).State != "closed" || f.remoteHead(batchBranch1) != "" { + t.Errorf("run --defer = %d, #90 %s, branch %q; want 2, the close finished and nothing deferred:\n%s", code, gh.find(90).State, f.remoteHead(batchBranch1), stderr) + } + }) + t.Run("closed by hand while the close writes", func(t *testing.T) { + f, gh, h1 := closeWorld(t) + gh.after = map[string]func(){"createComment": func() { gh.find(90).State = "closed" }} + code, _, stderr := f.exec(gh, deferAll...) + f.git("switch", "-q", "main") + if code != 2 || !strings.Contains(stderr, "batch pull request #90 is closed now; nothing is written to it") || len(gh.edited) != 0 || f.remoteHead(batchBranch1) != h1 { + t.Errorf("run = %d, edited %q, branch %q; want 2, no edit and the branch kept:\n%s", code, gh.edited, f.remoteHead(batchBranch1), stderr) + } + }) + t.Run("the delete failed", func(t *testing.T) { + f, gh, h1 := closeWorld(t) + gh.after = map[string]func(){"closePull": func() { f.refusePushes(true) }} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + f.refusePushes(false) + code, _, stderr := f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "warning: batch pull request #90 is closed, but its branch "+batchBranch1+" is still at "+h1[:12]) { + t.Errorf("check = %d, want 0 naming the branch:\n%s", code, stderr) + } + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue"); code != 2 { + t.Errorf("run without --push = %d, want 2:\n%s", code, stderr) + } + f.git("switch", "-q", "main") + if code, _, stderr := f.exec(gh, push...); code != 0 || f.remoteHead(batchBranch1) != "" { + t.Errorf("run --push = %d, branch %q; want it deleted\n%s", code, f.remoteHead(batchBranch1), stderr) + } + mustRun(t, f, gh, "check", "--line", "release-1.1") + }) + // A batch pull request a person opens on the branch a close left owns + // it: neither recovery nor check offers to delete it, moved or not. One + // on another branch does not stop the delete. + for _, tc := range []struct { + name string + branch string + moved bool + }{ + {"a batch opened on the branch the close left", batchBranch1, false}, + {"a batch opened on the branch the close left, moved since", batchBranch1, true}, + {"a batch opened on another branch", "backport-batch/release-1.1-20261027", false}, + } { + t.Run(tc.name, func(t *testing.T) { + f, gh, h1 := closeWorld(t) + gh.after = map[string]func(){"closePull": func() { f.refusePushes(true) }} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + f.refusePushes(false) + gh.after = nil + if tc.branch != batchBranch1 { + f.git("push", "-q", "origin", h1+":refs/heads/"+tc.branch) + } + left := h1 + if tc.moved { + left = moveBranch(f, h1) + } + gh.prs = append(gh.prs, f.batchPR(91, "release-1.1", tc.branch, "a maintainer's batch")) + taken := tc.branch == batchBranch1 + _, _, stderr := f.exec(gh, "check", "--line", "release-1.1") + f.git("switch", "-q", "main") + if named := strings.Contains(stderr, "batch pull request #91 is open on "+batchBranch1); named != taken || taken == strings.Contains(stderr, "--push deletes it") || taken && strings.Contains(stderr, "git push origin --delete") { + t.Errorf("check names the branch as #91's: %v, want %v, and offers no delete of it:\n%s", named, taken, stderr) + } + code, _, stderr := f.exec(gh, push...) + f.git("switch", "-q", "main") + if taken && (code != 2 || !strings.Contains(stderr, "batch pull request #91 is open on "+batchBranch1) || strings.Contains(stderr, "git push origin --delete") || f.remoteHead(batchBranch1) != left || gh.find(91).State != "open") { + t.Errorf("run --push = %d, branch %q; want 2, no delete offered and #91's branch kept:\n%s", code, f.remoteHead(batchBranch1), stderr) + } + if !taken && (code != 0 || f.remoteHead(batchBranch1) != "") { + t.Errorf("run --push = %d, branch %q; want the left branch deleted:\n%s", code, f.remoteHead(batchBranch1), stderr) + } + }) + } + t.Run("the branch moved after the close", func(t *testing.T) { + f, gh, h1 := closeWorld(t) + var h2 string + gh.after = map[string]func(){"closePull": func() { h2 = moveBranch(f, h1) }} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 || !strings.Contains(stderr, "is at "+h2[:12]+" now") { + t.Fatalf("run = %d, want 2 naming where the branch moved\n%s", code, stderr) + } + f.git("switch", "-q", "main") + // What was pushed onto the branch may be work the next run cannot + // pick again, so the refusal promises nothing about deleting it. + if code, _, stderr := f.exec(gh, push...); code != 2 || !strings.Contains(stderr, h1[:12]) || !strings.Contains(stderr, h2[:12]) || !strings.Contains(stderr, "#90") || strings.Contains(stderr, "nothing on it is lost") || !strings.Contains(stderr, "check what "+h2[:12]+" carries") { + t.Errorf("run --push = %d, want 2 naming both heads and #90, and what to check before deleting:\n%s", code, stderr) + } + if f.remoteHead(batchBranch1) != h2 { + t.Error("the run moved a branch it did not freeze") + } + if _, err := git(deleteArgs(batchBranch1, h1)...); err == nil || f.remoteHead(batchBranch1) != h2 { + t.Error("the remote took a delete whose lease names a head it no longer has") + } + }) + t.Run("a batch of the same day that opened no pull request", func(t *testing.T) { + f, gh, _ := closeWorld(t) + if code, _, stderr := f.exec(gh, deferAll...); code != 0 || f.remoteHead(batchBranch1) != "" { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + // The same day, a new batch reuses the branch name, pushes, and + // fails to open its pull request. + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh.fail = map[string]error{"createPull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, push...); code != 2 { + t.Fatalf("the same-day run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + h2 := f.remoteHead(batchBranch1) + for _, args := range [][]string{push, {"run", "--line", "release-1.1", "--onto", h2, "--push"}} { + if code, _, stderr := f.exec(gh, args...); code != 2 || !strings.Contains(stderr, "git push origin --delete "+batchBranch1) { + t.Errorf("%q = %d, want 2 naming the branch to delete:\n%s", args, code, stderr) + } + f.git("switch", "-q", "main") + } + if f.remoteHead(batchBranch1) != h2 || len(gh.created) != 1 { + t.Errorf("the branch moved to %.12s or a pull request was opened: %q", f.remoteHead(batchBranch1), gh.created) + } + f.git("push", "-q", "origin", "--delete", batchBranch1) + gh.nextPR = 91 + mustRun(t, f, gh, push...) + if gh.find(91) == nil { + t.Error("after the branch was deleted the next run opened no batch") + } + }) + t.Run("a preview", func(t *testing.T) { + f, gh, _ := closeWorld(t) + gh.fail = map[string]error{"closePull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + before := writes(gh) + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "x") + if code != 2 || !strings.Contains(stderr, "run --line release-1.1 --push finishes") || writes(gh) != before || gh.find(90).State != "open" { + t.Errorf("a preview = %d, want 2 naming the step left and nothing written:\n%s", code, stderr) + } + }) +} + +// A batch that merged on GitHub and left its branch behind is named when a +// run of the same day refuses to push over that branch. +func TestRefusalNamesTheMergedBatchsBranch(t *testing.T) { + f, gh, _ := closeWorld(t) + head := f.remoteHead(batchBranch1) + mergeBatch(f, gh) + // A commit pushed onto the batch after it merged is left on its branch. + f.git("switch", "-q", "--detach", head) + f.git("commit", "-q", "--allow-empty", "-m", "pushed after the merge") + pushed := f.git("rev-parse", "HEAD") + f.git("push", "-q", "origin", "HEAD:refs/heads/"+batchBranch1) + f.git("switch", "-q", "main") + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push") + if code != 2 || !strings.Contains(stderr, "it is the branch of batch pull request #90, which is merged") || strings.Contains(stderr, "--onto") || !strings.Contains(stderr, "check what "+pushed[:12]+" carries") { + t.Errorf("run --push = %d, want a refusal naming #90:\n%s", code, stderr) + } +} diff --git a/cmd/backport-batch/fixture_test.go b/cmd/backport-batch/fixture_test.go index 1a633134f..4ae5731b6 100644 --- a/cmd/backport-batch/fixture_test.go +++ b/cmd/backport-batch/fixture_test.go @@ -449,9 +449,35 @@ func (g *fakeGH) editPull(number int, title, bodyFile string) error { if err := g.readBody(bodyFile); err != nil { return err } + if pr := g.find(number); pr != nil { + pr.Body = g.bodies[len(g.bodies)-1] + } + if f := g.after["editPull"]; f != nil { + f() + } + return nil +} + +func (g *fakeGH) closePull(number int) error { + if err := g.write(number, "a close"); err != nil { + return err + } + if err := g.inject("closePull"); err != nil { + return err + } + if pr := g.find(number); pr != nil { + pr.State = "closed" + } + if f := g.after["closePull"]; f != nil { + f() + } + return nil +} + +func (g *fakeGH) find(n int) *pullRequest { for i := range g.prs { - if g.prs[i].Number == number { - g.prs[i].Body = g.bodies[len(g.bodies)-1] + if g.prs[i].Number == n { + return &g.prs[i] } } return nil diff --git a/cmd/backport-batch/github.go b/cmd/backport-batch/github.go index b1f3b5ce3..072673a77 100644 --- a/cmd/backport-batch/github.go +++ b/cmd/backport-batch/github.go @@ -127,6 +127,7 @@ type github interface { // createPull opens the draft and returns its number and URL. createPull(base, head, title, bodyFile string) (int, string, error) editPull(number int, title, bodyFile string) error + closePull(number int) error comments(number int) ([]issueComment, error) createComment(number int, bodyFile string) error editComment(id int64, bodyFile string) error @@ -259,6 +260,11 @@ func (ghCLI) editPull(number int, title, bodyFile string) error { return err } +func (ghCLI) closePull(number int) error { + _, err := capture("gh", "pr", "close", strconv.Itoa(number)) + return err +} + func (ghCLI) comments(number int) ([]issueComment, error) { out, err := capture("gh", "api", "--paginate", "--slurp", "repos/{owner}/{repo}/issues/"+strconv.Itoa(number)+"/comments") if err != nil { diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index 98ed2b2ac..bd0e02d30 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -54,7 +54,8 @@ run Builds backport-batch/- in this working tree, off --defer rebuilds the open batch off origin/ without those originals and records them as deferred, for --reason: the batch keeps them out, the next one tries them again. --push replaces the branch, - leased on the head the rebuild started from. + leased on the head the rebuild started from; when nothing is left it + closes the batch PR instead, after recording why in it. check Exits 1 while a change the policy takes is neither on the line nor on the open batch PR, 0 when nothing is, 2 when it cannot tell. With --cut, the gate before an rc or a patch is cut, it also exits 1 while a change diff --git a/cmd/backport-batch/overrides_test.go b/cmd/backport-batch/overrides_test.go index a759fd59c..0bcdc426e 100644 --- a/cmd/backport-batch/overrides_test.go +++ b/cmd/backport-batch/overrides_test.go @@ -259,6 +259,13 @@ func TestGHCLI(t *testing.T) { t.Error("pull took GitHub's answer about another pull request") } write("") + if err := (ghCLI{}).closePull(90); err != nil { + t.Fatal(err) + } + if want := "pr close 90"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } + write("") if err := (ghCLI{}).editPull(90, "fix(backport): batch", "/tmp/body.md"); err != nil { t.Fatal(err) } diff --git a/cmd/backport-batch/record_test.go b/cmd/backport-batch/record_test.go index 830685474..10b3b3f7f 100644 --- a/cmd/backport-batch/record_test.go +++ b/cmd/backport-batch/record_test.go @@ -313,6 +313,17 @@ func TestMetadataOperationsKeepTheMode(t *testing.T) { if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#5 changes .github/workflows/") { t.Errorf("check --cut = %d, want #5 still waiting:\n%s", code, stdout) } + + // Deferring the last picks closes the batch, and its record + // still says how the batch was assembled. + mustRun(t, f, gh, "run", "--line", "release-1.1", "--defer", "6,8", "--reason", "fails the e2e", "--push") + rec, err = parseRecord(gh.find(90).Body, notifyLine) + if err != nil || gh.find(90).State != "closed" || rec.Closing == nil || rec.Built.Options != (runOptions{NoWorkflowFiles: true, Mode: modeClose}) { + t.Errorf("after the close the record is %+v, %v; want the flag kept", rec.Built, err) + } + if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "/pull/5 ") { + t.Errorf("check --cut = %d, want #5 missing once no batch lists it:\n%s", code, stdout) + } }) } } diff --git a/cmd/backport-batch/report_test.go b/cmd/backport-batch/report_test.go index a0b2942ff..46a76669f 100644 --- a/cmd/backport-batch/report_test.go +++ b/cmd/backport-batch/report_test.go @@ -55,15 +55,6 @@ func bodyReportOf(t *testing.T, body string, on ...line) storedReport { return *s } -func (g *fakeGH) find(n int) *pullRequest { - for i := range g.prs { - if g.prs[i].Number == n { - return &g.prs[i] - } - } - return nil -} - // refusePushes makes the remote refuse every push, as a lost connection or // a protected branch would, until it is called with false. func (f *fixture) refusePushes(on bool) { diff --git a/docs/release/backport-mechanics.md b/docs/release/backport-mechanics.md index f713d20aa..70da94e85 100644 --- a/docs/release/backport-mechanics.md +++ b/docs/release/backport-mechanics.md @@ -14,7 +14,7 @@ go build ./cmd/backport-batch ./backport-batch check --line release-X.Y --cut ``` -[`backport-batch check`](../../cmd/backport-batch/README.md) applies the rules to every pull request merged into `main` since the line branched, and looks for each one they take on the line and then on the open batch pull request: by picks whose `-x` lines name its merge commit or every commit of its own, by a pick that would change nothing, and, on commits from before the policy's batch-only day, by what their text says, such as `Backport of #N`. With `--cut` a change found only on the open batch pull request fails the check too, since a cut tags the line and would not carry it: merge the batch pull request first. A conflict the batch deferred, or an original deferred to the next batch, prints as a warning without failing the check. A change that reached the line by hand counts like any other, by what reached the line. +[`backport-batch check`](../../cmd/backport-batch/README.md) applies the rules to every pull request merged into `main` since the line branched, and looks for each one they take on the line and then on the open batch pull request: by picks whose `-x` lines name its merge commit or every commit of its own, by a pick that would change nothing, and, on commits from before the policy's batch-only day, by what their text says, such as `Backport of #N`. With `--cut` a change found only on the open batch pull request fails the check too, since a cut tags the line and would not carry it: merge the batch pull request first. A conflict the batch deferred, or an original deferred to the next batch, prints as a warning without failing the check, also once that batch merged or closed. An original that waits on the open batch for a maintainer fails the cut. A change that reached the line by hand counts like any other, by what reached the line. ### Backporting code that touches monotonic counters diff --git a/docs/release/backport-tasks/failed-ci.md b/docs/release/backport-tasks/failed-ci.md index e845eff99..18b26f4dc 100644 --- a/docs/release/backport-tasks/failed-ci.md +++ b/docs/release/backport-tasks/failed-ci.md @@ -33,6 +33,8 @@ It rebuilds the branch locally without those originals and prints the new body, It replaces the batch branch only if nobody pushed onto it since the rebuild read it, and updates the pull request body. CI runs again on the new head; if it fails again on another original, repeat from step 1. +When the culprits are everything the batch carries, there is no branch left to push: the body printed without `--push` says the batch is closing, and `--push` records the deferrals in the batch pull request, closes it and deletes its branch, still only if nobody pushed onto it since. The next batch tries them again. If it stops part way, the next `run --push` on the line finishes the close, and says so. + ### Never - Defer an original the failure does not reach, to make the batch green by taking out more than the culprit. diff --git a/docs/release/backports.md b/docs/release/backports.md index 0cc8db3b6..37e2e89f9 100644 --- a/docs/release/backports.md +++ b/docs/release/backports.md @@ -83,7 +83,7 @@ A run by hand trusts the open batch pull request the scheduled run opened as `co `run` picks what the rules take onto `backport-batch/-` and opens a draft pull request against the line. A pull request merged into `main` with a merge commit goes onto the batch as its own commits, oldest first, each picked with `git cherry-pick -x --signoff` so it keeps its author, its message and its sign-offs and gains the sign-off of whoever runs the batch; commits that merged `main` into the pull request's branch are left out. A merge commit joins them to the batch: it carries the original's merge message from `main` and ends in a `Backported-from: ` trailer. A pull request squash-merged into `main` goes on as one pick. When any commit of an original conflicts, nothing of that original stays on the batch. The batch pull request is merged into the line with a merge commit, so all of this reaches the line as it is. Its body lists the originals with how many commits each brought and which of them differ from the commit they came from, the conflicts it deferred with the commit that failed and its files, and what it skipped under which rule, and ends with the [line's record and the run's report](#the-lines-record-and-reports); `--push` prints the pull request's URL. While that pull request is open, the next run continues it, `--continue` reading its head, rather than opening another. CI on the batch pull request runs the line's build and tests, and a maintainer merges it. Since rule 6 reads the newer line itself, an older line takes a fix in the batch after the newer line's batch with that fix has merged. -A deferred conflict waits for the next batch. To land it sooner, pick the original's commits onto the batch head with `git cherry-pick -x`, resolving each conflict, and join them to the batch head with a merge commit that names the original; an original squash-merged into `main` is one pick of its commit there, with no merge. Check that merge or that pick with `./backport-batch fold --line --original --commit `, and push it onto the batch branch, as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes. CI runs `backport-batch verify` on every push to a batch pull request, which holds every original on it to the same shape. When CI on the batch pull request fails, the original it comes from is taken out with `./backport-batch run --line --defer --reason --push` and goes to the next batch, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the rest of the batch merges without it. +A deferred conflict waits for the next batch. To land it sooner, pick the original's commits onto the batch head with `git cherry-pick -x`, resolving each conflict, and join them to the batch head with a merge commit that names the original; an original squash-merged into `main` is one pick of its commit there, with no merge. Check that merge or that pick with `./backport-batch fold --line --original --commit `, and push it onto the batch branch, as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes. CI runs `backport-batch verify` on every push to a batch pull request, which holds every original on it to the same shape. When CI on the batch pull request fails, the original it comes from is taken out with `./backport-batch run --line --defer --reason --push` and goes to the next batch, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the rest of the batch merges without it. When it is all the batch carries, `--push` records why in the batch pull request, closes it and deletes its branch, and the next batch tries it again. Before a patch release or an rc is cut from a line, `./backport-batch check --line --cut` has to exit 0. A cut tags the line, so the check fails while a pull request the rules take is not on the line, including one that is only on the open batch pull request: merge the batch pull request first. It prints the URLs, and prints what the batch deferred as warnings, since deferring is a maintainer's call; so is a pull request labelled `backport/skip` that the open batch still carries. An original that waits on the batch for a maintainer fails the cut, also when it is all the batch has left. A cut that passes says so. Without `--cut`, as the scheduled run uses it, a change on the open batch pull request counts as carried. From 2c1cb2f41cd6b63b7e2c8c445c416b1889b2ad0e Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 9 Nov 2026 18:32:20 +0000 Subject: [PATCH 400/518] feat(release): drop an original from one line with run --drop A fix the rules take for every line can still be wrong for one of them, typically because it builds on code that line does not have. The only ways to keep it out were backport/skip, which keeps it off every line, and --defer, which holds it out of one batch: the next batch tried it again, and a conflict that would never resolve came back every run. run --drop N --reason R records a drop in the line's record, bound to the original's merge commit on main, and every later record carries it, a new batch's included. What the open batch carries of a dropped original comes off it the way --defer takes one off; any other drop changes only the record, so a batch on a line that moved since stays where it is. --undrop withdraws one. Both need an open batch, as every write to the record does. A drop comes after the rules and the carries: backport/skip, rules 2 to 4 or the line itself decide first and the drop stays on record, and backport/include does not lift it. check notes a dropped original, the body lists it, fold refuses it, and a record whose drop names another original's merge stops every command. Each report keeps the record's drops with the merge commit each is bound to and its reason, held on read to the same rules, so an earlier report still says why after an undrop. A reason now travels from a body to terminals, bodies and comments, so a reason or a file name in a record is believed only in the form a run writes it, one clean line, and so is a report's diagnostic. Output that repeats a reason, file names, a report ID or a state GitHub reports escapes it. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 16 +- cmd/backport-batch/assemble.go | 52 +- cmd/backport-batch/body.go | 20 +- cmd/backport-batch/check.go | 5 +- cmd/backport-batch/check_test.go | 14 + cmd/backport-batch/defer.go | 65 +- cmd/backport-batch/defer_test.go | 23 + cmd/backport-batch/drop.go | 269 ++++++++ cmd/backport-batch/drop_test.go | 576 ++++++++++++++++++ cmd/backport-batch/fold_test.go | 18 + cmd/backport-batch/lines.go | 32 +- cmd/backport-batch/main.go | 60 +- cmd/backport-batch/record.go | 26 +- cmd/backport-batch/record_test.go | 47 +- cmd/backport-batch/report.go | 24 +- cmd/backport-batch/report_test.go | 34 +- cmd/backport-batch/rules.go | 3 + .../backport-tasks/resolve-conflict.md | 4 +- docs/release/backports.md | 10 +- 19 files changed, 1217 insertions(+), 81 deletions(-) create mode 100644 cmd/backport-batch/drop.go create mode 100644 cmd/backport-batch/drop_test.go diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index 768f21b2f..b1837f489 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -9,6 +9,8 @@ go build ./cmd/backport-batch ./backport-batch run --line release-1.7 [--onto SHA] [--date YYYY-MM-DD] [--no-workflow-files] [--push] [--policy REF] ./backport-batch run --line release-1.7 --continue [--no-workflow-files] [--push] [--policy REF] ./backport-batch run --line release-1.7 --defer 4321[,4322] --reason "fails the e2e of the app" [--push] [--policy REF] +./backport-batch run --line release-1.7 --drop 4321[,4322] --reason "builds on the 1.8 storage rework" [--push] [--policy REF] +./backport-batch run --line release-1.7 --undrop 4321[,4322] [--push] [--policy REF] ./backport-batch check --line release-1.7 [--cut] [--policy REF] ./backport-batch fold --line release-1.7 --original 4321 --commit SHA [--policy REF] ./backport-batch verify --line release-1.7 [--head SHA] @@ -19,7 +21,7 @@ go build ./cmd/backport-batch [`hack/backport-batch-scheduled.sh`](../../hack/backport-batch-scheduled.sh) runs `lines`, then `run --continue --push`, `check` and `notify` for each line; [`backport-batch.yaml`](../../.github/workflows/backport-batch.yaml) runs it on schedule as the cozystack-ci App, with `--no-workflow-files`, which it hands to `run`. -Requires `git` and an authenticated `gh` on PATH, and a remote called `origin`. Every command but `notify` and `report` fetches `origin` first; those two read only GitHub. The batch runs only for the lines `lines` prints. +Requires `git` and an authenticated `gh` on PATH, and a remote called `origin`. Every command but `notify` and `report` fetches `origin` first; `notify` reads only GitHub, and `report` reads GitHub and, for a report that keeps drops, the clone's history. The batch runs only for the lines `lines` prints. `run`, `check`, `fold` and `lines` read the `backport-policy` block of the policy page from `origin/main`, or from the ref `--policy` names, which is how a change to the page is tried before it merges; `verify` and `notify` do not read it and take no `--policy`. A line the block lists takes nothing merged into `main` before the day it gives, unless the original carries `backport/include`; any other line starts at its fork point. A page without exactly one readable block stops the run with exit 2. @@ -37,11 +39,13 @@ The pull request body goes to stdout, built from the line's own `.github/PULL_RE `--defer N[,N] --reason TEXT` takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off `origin/` without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got `backport/skip` after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the `backport-batch` block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. When it takes out everything the batch carries, there is nothing left to push, and `--push` closes the batch pull request instead, as [the line's record](#the-lines-record-and-reports) describes. +`--drop N[,N] --reason TEXT` drops originals from the line: where `--defer` holds one out of this batch, a drop keeps it out of every later batch of the line too, until `--undrop N[,N]` withdraws it. A drop is recorded in the [line's record](#the-lines-record-and-reports), so both need an open batch pull request that carries one, and exit 2 without it; once a batch merges, its drops are carried into the next batch's record. A dropped original the open batch carries is taken off it as `--defer` would, and the rebuild is pushed the same way; any other drop or undrop changes only the record, so the batch stays as it is even when the line moved since. An undropped original is missing until the next run tries it. A drop names the original's merge commit on `main` as well as its number, and a record whose drop does not match what that merge commit merged stops every command with exit 2. An original the line already carries cannot be dropped, and neither can a number that merged into `main` before the line branched. The order is the rules', then the drop: what `backport/skip`, rules 2 to 4 or the line itself decide comes first, with the drop kept on record beside it, and `backport/include` does not lift a drop. The body lists dropped originals under their own heading among the skipped ones, `check` prints a note for each, and `fold` refuses one. + `--push` pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with `gh pr create --draft`, or updates the open batch pull request with `gh pr edit`, and prints its URL on stderr. On the open batch it first posts the run's report as a comment, so a body never names a report nobody can read; a new batch carries the report in the body it opens with, and the comment follows. Before every write to the pull request it reads the pull request again, and stops with exit 2 once it is merged or closed. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the `run --onto --push` that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no `backport-batch` block either: someone wrote that body by hand, and the batch is theirs to merge or close. After `--defer` alone it replaces the branch, with `--force-with-lease=refs/heads/:`, where head is what `git ls-remote` reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. A record that says the batch is being closed is acted on before anything else, and only by `run --push`. ### check -Exits 1 and prints the URL of every pull request the rules take that is neither on the line nor on the open batch pull request, 0 when there is none, and 2 when it cannot tell. A pull request that the [line's record](#the-lines-record-and-reports) lists as deferred, a conflict or one taken out with `--defer`, prints as a warning and does not fail the check: deferring is a maintainer's call. That holds once the batch that recorded it merged or closed too. So does one the open batch lists as waiting for a maintainer because it changes workflows; once that batch is gone, nothing lists it any more, and it is missing. +Exits 1 and prints the URL of every pull request the rules take that is neither on the line nor on the open batch pull request, 0 when there is none, and 2 when it cannot tell. A pull request that the [line's record](#the-lines-record-and-reports) lists as deferred, a conflict or one taken out with `--defer`, prints as a warning and does not fail the check: deferring is a maintainer's call. That holds once the batch that recorded it merged or closed too. So does one the open batch lists as waiting for a maintainer because it changes workflows; once that batch is gone, nothing lists it any more, and it is missing. One a maintainer dropped from the line prints as a note, `#N is dropped from on #M ()`. `--cut` makes it the gate before a patch release or an rc is cut from a line. A cut tags the line, so a pull request that is only on the open batch pull request fails it too, with one line each: `#N is only on open batch PR #M, merge it before the cut`. One that waits there for a maintainer fails it as well, since nobody decided to leave it out, also when it is all the batch has left: bring it over, or take it out with `run --defer` and a reason. Everything else is as without it, and a cut that passes says `no change the policy takes is missing before the cut`. The scheduled run checks without `--cut`, since its batch is open by design. @@ -65,7 +69,7 @@ The comment starts with `` and is one per pull req ### report -Prints a report a batch pull request keeps: the one its body carries, or the one `--id` names. It reads the report comments the batch trusts and falls back to the body, recomputes the report's ID from what it says, and exits 2 when a trusted comment does not match its ID, when a report says anything a run does not write or is for another line than the pull request's, or when the pull request keeps no such report. A report comment counts only when its author is one the batch trusts with a record, so anyone else's comment, whatever ID it carries, is not read. It prints what the classification was made of, whether the pull request's head or body shows the report was published, which it calls unconfirmed when neither does, and one line per original with its category and what the report recorded about it; `--json` prints the report as stored. A report whose comment the run never got to post is printed from the body, with a note that the next `run --push` posts it, or, on a merged or closed batch, that it is kept in the body only. It reads GitHub alone and writes nothing. +Prints a report a batch pull request keeps: the one its body carries, or the one `--id` names. It reads the report comments the batch trusts and falls back to the body, recomputes the report's ID from what it says, and exits 2 when a trusted comment does not match its ID, when a report says anything a run does not write or is for another line than the pull request's, or when the pull request keeps no such report. A report comment counts only when its author is one the batch trusts with a record, so anyone else's comment, whatever ID it carries, is not read. It prints what the classification was made of, whether the pull request's head or body shows the report was published, which it calls unconfirmed when neither does, and one line per original with its category and what the report recorded about it; `--json` prints the report as stored. A report whose comment the run never got to post is printed from the body, with a note that the next `run --push` posts it, or, on a merged or closed batch, that it is kept in the body only. It writes nothing. It reads GitHub, and for a report that keeps drops also the clone's history, without fetching: each drop is bound to `main` as the record's drops are, over the heads of `main` and the line the report was made from, so a drop of an original the report classified has to name that original's merge, and any other a merge of its number on `main`'s first-parent line. A clone without those heads exits 2, asking for a fetch. ### lines @@ -107,11 +111,11 @@ The line's record is the `backport-batch` block of its newest batch pull request The record is written only while its pull request is open: a merged or closed batch pull request is never edited or commented on again. What it recorded keeps counting, so a conflict or a deferral a merged batch recorded still prints as a warning. A workflow original it listed as waiting is missing, since no open batch lists it. A run that picks nothing while no batch is open has nowhere to write: it prints the body, says that its classification is not retained, and exits 0. -The record is only as trustworthy as the body it sits in. Anyone with write access can edit any body, the bot's included, and neither the bot's authorship nor anything in the block proves who edited it last. A hand edit that leaves a readable record is believed; one that breaks it stops every command with exit 2, naming the pull request. When the newest batch pull request is deleted and no later one exists, nothing shows it was there: the older record governs, and what was recorded since is lost. +The record is only as trustworthy as the body it sits in. Anyone with write access can edit any body, the bot's included, and neither the bot's authorship nor anything in the block proves who edited it last. A hand edit that leaves a readable record is believed; one that breaks it stops every command with exit 2, naming the pull request. A reason or a file name in the record counts only in the form a run writes it, one line with no control, zero-width or bidi character, so text someone typed into the body never reaches a terminal, a body or a comment as they typed it. When the newest batch pull request is deleted and no later one exists, nothing shows it was there: the older record governs, and what was recorded since is lost. -The record holds only what later commands act on. Every run that classifies the line also makes a report, one line of JSON in a `backport-batch-report` block: every original merged into `main` since the line branched in exactly one category, picked, missing, deferred with what it was deferred for, or skipped with the rule that skipped it, the commits on `main` that merged no pull request, the labels, and what the classification was made of: the heads of `main`, the line and the newer line, the policy, the batch head it describes, the tool's commit, the Actions run and the flags. Its ID is a hash of all that but the time and the run, so two runs that find the same make the same report, and it names no pull request, so a new batch's report is whole before GitHub numbers the pull request. The body carries the latest report beside the record, and a comment on the open batch pull request keeps each one: `` and the same block, folded away. A report already posted is not posted again, and the tool never edits or deletes such a comment, so a report the next run replaces in the body is still read back by `report`, also after the batch merged. A comment that looks like one but whose author the batch does not trust is not read. When a run stops after writing the body but before posting its comment, the next `run --push` on the open batch posts that report first, byte for byte; once the batch is closed nothing is written to it, and the report stays in its body. A run that picks nothing while no batch is open keeps no report. +The record holds only what later commands act on. Every run that classifies the line also makes a report, one line of JSON in a `backport-batch-report` block: every original merged into `main` since the line branched in exactly one category, picked, missing, deferred with what it was deferred for, or skipped with the rule that skipped it, each drop the record keeps with the merge commit it is bound to, its reason and the batch pull request that recorded it, the commits on `main` that merged no pull request, the labels, and what the classification was made of: the heads of `main`, the line and the newer line, the policy, the batch head it describes, the tool's commit, the Actions run and the flags. Its ID is a hash of all that but the time and the run, so two runs that find the same make the same report, and it names no pull request, so a new batch's report is whole before GitHub numbers the pull request. The body carries the latest report beside the record, and a comment on the open batch pull request keeps each one: `` and the same block, folded away. A report already posted is not posted again, and the tool never edits or deletes such a comment, so a report the next run replaces in the body is still read back by `report`, also after the batch merged. A comment that looks like one but whose author the batch does not trust is not read. When a run stops after writing the body but before posting its comment, the next `run --push` on the open batch posts that report first, byte for byte; once the batch is closed nothing is written to it, and the report stays in its body. A run that picks nothing while no batch is open keeps no report. -A `--defer` that leaves the batch carrying nothing closes it, and writes everything while the pull request is still open: the report comment, then the body, whose record keeps the deferrals and says the batch is closing, with its number, branch and the head the rebuild started from. It then reads the pull request and the branch again, closes the pull request while both are at that head, and deletes the branch with `--force-with-lease` on it, printing `closed `. The next batch tries what was deferred again. A close that stops part way is finished by the next `run --push`, before it does anything else: it closes and deletes when the pull request is open at the frozen head, deletes the branch when the pull request is closed and the branch still at that head, and goes on as usual once both are gone. A run that finishes a close stops there, with exit 0, or with exit 2 when it was asked for `--defer`, which it does not apply. It never takes a head it did not freeze: with the branch moved, it stops with exit 2 and names both heads. `run` without `--push`, `check` and `fold` stop with exit 2 while the pull request is still open; once it is closed they go on, warning about a branch the close left, and `notify` says nothing. A closed pull request is never written to, so its record says it was closing for good, and a closed pull request whose branch is gone counts as closed. +A `--defer` that leaves the batch carrying nothing closes it, and writes everything while the pull request is still open: the report comment, then the body, whose record keeps the deferrals and says the batch is closing, with its number, branch and the head the rebuild started from. It then reads the pull request and the branch again, closes the pull request while both are at that head, and deletes the branch with `--force-with-lease` on it, printing `closed `. The next batch tries what was deferred again. A close that stops part way is finished by the next `run --push`, before it does anything else: it closes and deletes when the pull request is open at the frozen head, deletes the branch when the pull request is closed and the branch still at that head, and goes on as usual once both are gone. A run that finishes a close stops there, with exit 0, or with exit 2 when it was asked for `--defer`, `--drop` or `--undrop`, which it does not apply. It never takes a head it did not freeze: with the branch moved, it stops with exit 2 and names both heads. `run` without `--push`, `check` and `fold` stop with exit 2 while the pull request is still open; once it is closed they go on, warning about a branch the close left, and `notify` says nothing. A closed pull request is never written to, so its record says it was closing for good, and a closed pull request whose branch is gone counts as closed. One way for the branch to move is a new batch of the same day: it reuses the branch name, and if its push succeeds and opening its pull request fails, the closed batch is still the newest record while its branch now holds the new batch. The next run stops on it, naming the branch, and so does `--onto` with its head. Check what it carries before deleting it with `git push origin --delete backport-batch/-`: the next run picks again what the tool put there from the line, but not a commit someone pushed onto it, such as a resolved conflict. Separate writes to GitHub are not atomic: the re-reads and the leases refuse a race they see, but cannot prevent one. diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index 3ac19c4f9..7f36ff3f4 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -75,8 +75,10 @@ func (a *app) run(args []string) int { cont := fs.Bool("continue", false, "continue the open batch pull request, or start a batch when none is open") dateArg := fs.String("date", "", "batch date, YYYY-MM-DD (default: today, UTC)") deferArg := fs.String("defer", "", "originals to take out of the open batch, N[,N]; rebuilds it without them") + dropArg := fs.String("drop", "", "originals to drop from the line, N[,N]; recorded on the open batch, and taken off it") + undropArg := fs.String("undrop", "", "originals whose drop to withdraw, N[,N]") noWorkflows := fs.Bool("no-workflow-files", false, "leave an original that changes "+workflowsDir+" to a maintainer, for a token that cannot push one") - reasonArg := fs.String("reason", "", "why --defer takes them out, recorded in the batch body") + reasonArg := fs.String("reason", "", "why --defer takes them out or --drop drops them, recorded in the batch body") push := fs.Bool("push", false, "push the branch and open or update the draft batch PR") if err := parse(fs, args); err != nil { return 2 @@ -96,9 +98,30 @@ func (a *app) run(args []string) int { return a.fail(fmt.Errorf("--onto: %w", err)) } } - var deferred []int - reason := "" + var deferred, drops []int + reason, ops := "", 0 + for _, op := range []string{*deferArg, *dropArg, *undropArg} { + if op != "" { + ops++ + } + } switch { + case ops > 1: + return a.fail(errors.New("--defer, --drop and --undrop each change the open batch pull request on their own; give one")) + case *dropArg != "" || *undropArg != "": + if *onto != "" || *cont || *dateArg != "" { + return a.fail(errors.New("--drop and --undrop change the open batch pull request; each takes no --onto, --continue or --date")) + } + if *undropArg != "" && *reasonArg != "" { + return a.fail(errors.New("--undrop takes no --reason")) + } + list := *dropArg + *undropArg + if drops, err = parseNumbers(map[bool]string{true: "--drop", false: "--undrop"}[*dropArg != ""], list); err != nil { + return a.fail(err) + } + if reason = capped(oneLine(clean(*reasonArg, false)), titleLimit); *dropArg != "" && reason == "" { + return a.fail(errors.New("--drop needs a --reason, which the line's record keeps")) + } case *deferArg != "" || *reasonArg != "": if deferred, reason, err = parseDefer(*deferArg, *reasonArg); err != nil { return a.fail(err) @@ -119,9 +142,12 @@ func (a *app) run(args []string) int { if err != nil { return a.fail(err) } - if goOn, code := a.resumeClose(b, *push, deferred != nil); !goOn { + if goOn, code := a.resumeClose(b, *push, deferred != nil || drops != nil); !goOn { return code } + if drops != nil { + return a.redrop(b, drops, reason, *undropArg != "", *push) + } if deferred != nil { return a.rebuild(b, deferred, reason, *push) } @@ -162,6 +188,9 @@ func (a *app) run(args []string) int { r := newReport(b) r.opts = runOptions{NoWorkflowFiles: *noWorkflows, Mode: modeNew} + // A drop is a decision about the line, not about one batch: a new batch + // keeps the ones the closed batch recorded. + r.dropped = b.src.rec.dropped() if carried != nil { r.opts.Mode = modeContinue if r.stale, err = keptOut(b, carried); err != nil { @@ -221,12 +250,12 @@ func (a *app) run(args []string) int { return a.finish(b, r, branch, date, *push, false) } -// keptOut lists the originals with backport/skip that the batch picked before -// the label came: a run never takes a commit off the batch, --defer does. +// keptOut lists the originals with backport/skip, or dropped, that the batch +// still carries: a run never takes a commit off the batch, --defer does. func keptOut(b *batch, carried *history) ([]*verdict, error) { var out []*verdict for _, v := range b.verdicts { - if v.label != labelSkip { + if v.rule != ruleSkipLabel && v.rule != ruleDropped { continue } if err := b.load(v); err != nil { @@ -331,7 +360,12 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r // act on, with the full classification left to the report it names. func (r *report) lineRecord(doc reportDoc) lineRecord { built := doc.Built - return lineRecord{Version: 2, Line: doc.Line, Built: &built, Report: doc.ID, Closing: r.closing, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: []droppedEntry{}, Stale: slices.Clone(doc.Overlays.Stale)} + dropped := slices.Clone(r.dropped) + if dropped == nil { + dropped = []droppedEntry{} + } + slices.SortFunc(dropped, func(a, b droppedEntry) int { return a.Original - b.Original }) + return lineRecord{Version: 2, Line: doc.Line, Built: &built, Report: doc.ID, Closing: r.closing, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: dropped, Stale: slices.Clone(doc.Overlays.Stale)} } // branchDate is the date a batch branch was opened under. @@ -686,7 +720,7 @@ func (a *app) finishClose(l line, url string, c closing) error { return fmt.Errorf("closing #%d, reading it again: %w", c.PR, err) } if st := prState(&pr); st != "open" || pr.Head.SHA != c.Head { - return fmt.Errorf("closing #%d: it is %s at %.12s, not open at %.12s, which its record froze", c.PR, st, pr.Head.SHA, c.Head) + return fmt.Errorf("closing #%d: it is %s at %.12s, not open at %.12s, which its record froze", c.PR, st, terminalSafe(pr.Head.SHA), c.Head) } heads, err := remoteBatchHeads(l) if err != nil { diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index 1bd2024f7..da4691450 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -221,7 +221,7 @@ func prTitle(l line, date string) string { } // skipRules are the rules a skipped original is reported under, in order. -var skipRules = []int{ruleBeforeStart, ruleOnLine, ruleType, ruleBreaking, ruleTouches, ruleNewer} +var skipRules = []int{ruleBeforeStart, ruleOnLine, ruleType, ruleBreaking, ruleTouches, ruleNewer, ruleDropped} // report is what one run put on the batch branch and what it left out. type report struct { @@ -251,6 +251,8 @@ type report struct { // closing is the close of the open batch this run leaves carrying // nothing. closing *closing + // dropped are the maintainers' drops the record keeps. + dropped []droppedEntry } // ruleName is how output names what kept an original out: the policy's @@ -261,10 +263,20 @@ func ruleName(rule int) string { return labelSkip case ruleBeforeStart: return "rule 1, before the line's start" + case ruleDropped: + return "a maintainer's drop" } return "rule " + strconv.Itoa(rule) } +// listTitle titles the list of what a rule skipped, a drop among them. +func (r *report) listTitle(rule int) string { + if rule == ruleDropped { + return "Dropped from " + codeSpan(r.line.branch()) + " by a maintainer" + } + return r.ruleTitle(rule) +} + func (r *report) ruleTitle(rule int) string { switch rule { case ruleBeforeStart: @@ -350,7 +362,7 @@ func (r *report) what(details map[int]bool) string { if !details[rule] { left = ", too many to list here; the run's output lists them" } - fmt.Fprintf(&b, "- %s: %d%s\n", r.ruleTitle(rule), len(items), left) + fmt.Fprintf(&b, "- %s: %d%s\n", r.listTitle(rule), len(items), left) } if r.other > 0 { fmt.Fprintf(&b, "- Commits on `main` that merged no pull request: %d\n", r.other) @@ -358,7 +370,7 @@ func (r *report) what(details map[int]bool) string { for _, rule := range skipRules { items := r.skipped[rule] if len(items) > 0 && details[rule] { - fmt.Fprintf(&b, "\n
%s\n\n", r.ruleTitle(rule)) + fmt.Fprintf(&b, "\n
%s\n\n", r.listTitle(rule)) for _, v := range items { fmt.Fprintf(&b, "- %s %s, %s\n", prLink(v.o.number), titleSpan(v.o.title), titleSpan(v.reason)) } @@ -372,6 +384,8 @@ func (r *report) what(details map[int]bool) string { // outcome is what became of an original a label overrode. func (r *report) outcome(v *verdict) string { switch { + case slices.Contains(r.stale, v) && v.rule == ruleDropped: + return "dropped, but still on this branch; take it out with `backport-batch run --defer`" case slices.Contains(r.stale, v): return "still on this branch, picked before the label; take it out with `backport-batch run --defer`" case v.label == labelSkip: diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index d7bc7e213..ae5395d45 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -165,6 +165,9 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { switch { case slices.Contains(stale, v): out[v] = disposition{outcomeWarning, fmt.Sprintf("#%d is still on batch #%d; take it out with backport-batch run --defer: %s", v.o.number, b.pr.Number, named)} + case v.rule == ruleDropped: + d := b.drops[v.o.number] + out[v] = disposition{outcomeNote, fmt.Sprintf("#%d is dropped from %s on #%d (%s): %s", v.o.number, b.line.branch(), d.On, terminalSafe(d.Reason), named)} case v.rule != 0: case v.onBatch != "": if cut { @@ -250,7 +253,7 @@ func refuseFold(b *batch, n int, commit string) (string, error) { case v.rule == ruleSkipLabel: return fmt.Sprintf("#%d carries %s", n, labelSkip), nil case v.rule != 0: - return fmt.Sprintf("#%d is not one the batch takes: %s, %s", n, ruleName(v.rule), v.reason), nil + return fmt.Sprintf("#%d is not one the batch takes: %s, %s", n, ruleName(v.rule), terminalSafe(v.reason)), nil case b.pr == nil: return fmt.Sprintf("no batch pull request is open on %s", b.line.branch()), nil case v.onBatch != "": diff --git a/cmd/backport-batch/check_test.go b/cmd/backport-batch/check_test.go index b011b18b8..cf66d865e 100644 --- a/cmd/backport-batch/check_test.go +++ b/cmd/backport-batch/check_test.go @@ -396,6 +396,20 @@ func TestOutputNamesNoInternalRule(t *testing.T) { } } +// A refusal fold prints is one line with no control sequence, whatever the +// reason it repeats says: a drop's reason comes from a pull request body, a +// rule 4 reason from file names an original's author chose. +func TestFoldRefusalPrintsNothingThatDrivesATerminal(t *testing.T) { + l, _ := parseLine("release-1.1") + for _, reason := range []string{"dropped on #90: x\n\x1b[2Jrefused: forged", "touches \"app/a\x1b]0;x\x07\nok\""} { + b := &batch{line: l, verdicts: []*verdict{{o: &original{number: 5}, rule: ruleDropped, reason: reason}}} + why, err := refuseFold(b, 5, strings.Repeat("a", 40)) + if err != nil || why == "" || strings.ContainsFunc(why, func(r rune) bool { return unicode.IsControl(r) || unicode.Is(unicode.Cf, r) }) { + t.Errorf("refuseFold = %q, %v; want one clean line", why, err) + } + } +} + type errString string func (e errString) Error() string { return string(e) } diff --git a/cmd/backport-batch/defer.go b/cmd/backport-batch/defer.go index cdc17a78c..52fa8cf62 100644 --- a/cmd/backport-batch/defer.go +++ b/cmd/backport-batch/defer.go @@ -27,23 +27,29 @@ import ( // parseDefer reads --defer N[,N] and the --reason that has to come with it. func parseDefer(list, reason string) ([]int, string, error) { if list == "" { - return nil, "", errors.New("--reason goes with --defer") + return nil, "", errors.New("--reason goes with --defer or --drop") } reason = capped(oneLine(clean(reason, false)), titleLimit) if reason == "" { return nil, "", errors.New("--defer needs a --reason, which the batch body records") } + numbers, err := parseNumbers("--defer", list) + return numbers, reason, err +} + +// parseNumbers reads the N[,N] list a flag takes, each once. +func parseNumbers(flag, list string) ([]int, error) { var numbers []int for _, s := range strings.Split(list, ",") { n, err := strconv.Atoi(strings.TrimPrefix(strings.TrimSpace(s), "#")) if err != nil || n <= 0 { - return nil, "", fmt.Errorf("--defer %q is not a list of pull request numbers", list) + return nil, fmt.Errorf("%s %q is not a list of pull request numbers", flag, list) } if !slices.Contains(numbers, n) { numbers = append(numbers, n) } } - return numbers, reason, nil + return numbers, nil } // rebuild takes the originals in numbers out of the open batch: it builds the @@ -54,17 +60,8 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { if b.pr == nil { return a.fail(fmt.Errorf("no batch pull request is open on %s; --defer rebuilds the open one", b.line.branch())) } - branch := b.pr.Head.Ref - date, err := branchDate(branch) - if err != nil { - return a.fail(err) - } - recorded := map[int]deferredEntry{} - if b.src.open { - recorded = b.src.rec.deferred() - } hold := map[int]string{} - for n, e := range recorded { + for n, e := range b.recorded() { if e.Reason != "" { hold[n] = e.Reason } @@ -86,7 +83,15 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { } hold[n] = reason } - start, replay, err := rebuildPlan(b, hold) + return a.reassemble(b, hold, hold, b.src.rec.dropped(), modeDefer, push) +} + +// reassemble builds the open batch again off the line's head without the +// units of the originals in out, and reports it with hold deferred for their +// reasons and drops kept. +func (a *app) reassemble(b *batch, out, hold map[int]string, drops []droppedEntry, mode string, push bool) int { + branch := b.pr.Head.Ref + start, replay, err := rebuildPlan(b, out) if err != nil { return a.fail(err) } @@ -102,7 +107,7 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { case err != nil: return a.fail(err) case failed != "": - return a.fail(fmt.Errorf("without what is deferred, %.12s%s no longer applies (%s); defer it as well", failed, originName(b, u), strings.Join(files, ", "))) + return a.fail(fmt.Errorf("without what is deferred, %.12s%s no longer applies (%s); defer it as well", failed, originName(b, u), terminalSafe(strings.Join(files, ", ")))) } } head, err := resolve("HEAD") @@ -113,10 +118,23 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { if err != nil { return a.fail(err) } + return a.reclassify(b, picks, hold, drops, mode, push) +} + +// reclassify reports the open batch as picks shows it, after an operation +// that adds no original: what the record held stays held, and an original +// the batch neither carries nor records is missing until a run tries it. +func (a *app) reclassify(b *batch, picks *history, hold map[int]string, drops []droppedEntry, mode string, push bool) int { + date, err := branchDate(b.pr.Head.Ref) + if err != nil { + return a.fail(err) + } + recorded := b.recorded() r := newReport(b) - // Taking an original out changes nothing about how the batch is - // assembled, so the record keeps the flag the batch was assembled with. - r.opts = runOptions{NoWorkflowFiles: b.src.savedNoWorkflowFiles(), Mode: modeDefer} + // None of these changes how the batch is assembled, so the record keeps + // the flag the batch was assembled with. + r.opts = runOptions{NoWorkflowFiles: b.src.savedNoWorkflowFiles(), Mode: mode} + r.dropped = drops if r.stale, err = keptOut(b, picks); err != nil { return a.fail(err) } @@ -137,7 +155,16 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { r.missing = append(r.missing, v) } } - return a.finish(b, r, branch, date, push, true) + return a.finish(b, r, b.pr.Head.Ref, date, push, true) +} + +// recorded is what the open batch's record defers; a closed batch's record +// holds nothing back on this one. +func (b *batch) recorded() map[int]deferredEntry { + if !b.src.open { + return map[int]deferredEntry{} + } + return b.src.rec.deferred() } // rebuildPlan is where the rebuild of the open batch without the originals in diff --git a/cmd/backport-batch/defer_test.go b/cmd/backport-batch/defer_test.go index d0f8798c8..66f7807a0 100644 --- a/cmd/backport-batch/defer_test.go +++ b/cmd/backport-batch/defer_test.go @@ -684,3 +684,26 @@ func TestRefusalNamesTheMergedBatchsBranch(t *testing.T) { t.Errorf("run --push = %d, want a refusal naming #90:\n%s", code, stderr) } } + +// A rebuild that stops names the files that no longer apply on the line of +// its refusal, whatever the files are called. +func TestDeferRefusalNamesFilesOnOneLine(t *testing.T) { + f := newFixture(t) + f.lines() + name := "app/x\nwarning: forged.txt" + f.pr(5, "fix(app): five", "", map[string]string{name: "five\n"}) + f.pr(6, "fix(app): six", "", map[string]string{name: "five and six\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + f.onLine("release-1.1", "fix(chart): a line-only value", map[string]string{"chart/values.yaml": "replicas: 3\nimage: app:1\n"}) + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "x", "--push") + f.git("switch", "-q", "main") + if code != 2 || !strings.Contains(stderr, "no longer applies") { + t.Fatalf("run --defer 5 = %d, want the refusal:\n%s", code, stderr) + } + for _, ln := range strings.Split(stderr, "\n") { + if strings.HasPrefix(ln, "warning: forged") { + t.Errorf("the refusal split onto a line of its own: %q", ln) + } + } +} diff --git a/cmd/backport-batch/drop.go b/cmd/backport-batch/drop.go new file mode 100644 index 000000000..b374081cf --- /dev/null +++ b/cmd/backport-batch/drop.go @@ -0,0 +1,269 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "cmp" + "fmt" + "maps" + "slices" + "strings" + "time" +) + +// A drop is a maintainer's decision that an original stays off one line, +// for a reason, where the rules would take it. It is kept in the line's +// record and carried from batch to batch, bound to the original's merge +// commit on main, so a record cannot point it at another original. It is +// written only to the open batch's record, like everything else there. + +// onLine says whether the line carries v, whatever rule decided it. +func (b *batch) onLine(v *verdict) (bool, error) { + if v.rule == ruleOnLine { + return true, nil + } + if err := b.load(v); err != nil { + return false, err + } + how, err := b.lineHist.carries(v.o, v.commits) + return how != "", err +} + +// carried says whether the open batch has a unit of v, by the mapping the +// rebuild takes units off with: found needs every commit of a merged +// original, and one whose pick came out empty is missing from it. +func (b *batch) carried(v *verdict) (bool, error) { + units, err := batchUnits(b.prHead, b.lineHead) + if err != nil { + return false, err + } + for _, u := range units { + if o := b.unitOriginal(u); o != nil && o.o.number == v.o.number { + return true, nil + } + } + return false, nil +} + +// redrop drops the originals in numbers from the line, for reason, or with +// undrop withdraws their drops. What the open batch carries of a dropped +// original comes off it, as --defer takes it off; otherwise only the record +// changes, and the batch stays as it is even when the line moved since. +func (a *app) redrop(b *batch, numbers []int, reason string, undrop bool, push bool) int { + if !b.src.open { + return a.fail(fmt.Errorf("no open batch pull request on %s carries a record; a drop is recorded on one", b.line.branch())) + } + drops := map[int]droppedEntry{} + for _, d := range b.src.rec.Dropped { + drops[d.Original] = d + } + mode, today := modeDrop, a.now().UTC().Format(time.DateOnly) + if undrop { + mode = modeUndrop + } + for _, n := range numbers { + v := b.find(n) + _, isDropped := drops[n] + switch { + case undrop && !isDropped: + return a.fail(fmt.Errorf("#%d is not dropped from %s", n, b.line.branch())) + case undrop: + delete(drops, n) + case v == nil: + return a.fail(fmt.Errorf("#%d did not merge into main after %s branched", n, b.line.branch())) + default: + // backport/skip and rules 1 to 4 decide before the line is + // looked at, so whether it carries the original is asked here. + on, err := b.onLine(v) + if err != nil { + return a.fail(err) + } + if on { + return a.fail(fmt.Errorf("#%d is already on %s; a drop cannot remove code the line has", n, b.line.branch())) + } + drops[n] = droppedEntry{Original: n, Merge: v.o.merge, Reason: reason, On: b.pr.Number, Date: today} + } + } + active := map[int]droppedEntry{} + for n, d := range drops { + if b.find(n) != nil { + active[n] = d + } + } + b.drops = active + out := map[int]string{} + for _, n := range numbers { + v := b.find(n) + if v == nil { + continue + } + if err := b.judge(v, active); err != nil { + return a.fail(err) + } + if err := b.load(v); err != nil { + return a.fail(err) + } + if undrop { + continue + } + on, err := b.carried(v) + if err != nil { + return a.fail(err) + } + if on { + out[n] = "dropped" + } + } + hold := map[int]string{} + for n, e := range b.recorded() { + if e.Reason != "" { + hold[n] = e.Reason + } + } + list := slices.SortedFunc(maps.Values(drops), func(x, y droppedEntry) int { return cmp.Compare(x.Original, y.Original) }) + if len(out) > 0 { + maps.Copy(out, hold) + return a.reassemble(b, out, hold, list, mode, push) + } + if err := a.warnReset(b.pr.Head.Ref, b.prHead); err != nil { + return a.fail(err) + } + if _, err := git("switch", "--quiet", "--no-track", "-C", b.pr.Head.Ref, "--end-of-options", b.prHead); err != nil { + return a.fail(err) + } + return a.reclassify(b, b.picks, hold, list, mode, push) +} + +// bindDrops holds each drop the record keeps to the original it names: a +// drop of an original merged since the line branched has to name that +// original's merge commit, and one of an earlier original a merge of it on +// main's first-parent line. The first apply; the others are kept and carried +// on, and decide nothing. +func (b *batch) bindDrops(merges []*original) (map[int]droppedEntry, error) { + return bindEntries(b.src.rec.dropped(), merges, b.mainHead, func(d droppedEntry) string { + return fmt.Sprintf("batch pull request #%d drops #%d by %.12s", b.src.carrier.Number, d.Original, d.Merge) + }) +} + +// bindEntries binds drops to main: one whose original is among merges, the +// originals main merged since the line branched, has to name that +// original's merge exactly; any other has to be a merge of its number on +// main's first-parent line up to mainHead, as an inherited drop is. It +// returns the drops whose originals are among merges; about says what an +// error is about. +func bindEntries(drops []droppedEntry, merges []*original, mainHead string, about func(droppedEntry) string) (map[int]droppedEntry, error) { + byMerge, byNumber := map[string]*original{}, map[int]*original{} + for _, o := range merges { + byMerge[o.merge], byNumber[o.number] = o, o + } + active := map[int]droppedEntry{} + for _, d := range drops { + at := about(d) + if o := byMerge[d.Merge]; o != nil { + if o.number != d.Original { + return nil, fmt.Errorf("%s, which merged #%d", at, o.number) + } + active[d.Original] = d + continue + } + if o := byNumber[d.Original]; o != nil { + return nil, fmt.Errorf("%s, but #%d merged as %.12s", at, d.Original, o.merge) + } + ok, err := mergedOnMain(d.Merge, mainHead, d.Original) + if err != nil { + return nil, err + } + if !ok { + return nil, fmt.Errorf("%s, which is no merge of it on main", at) + } + } + return active, nil +} + +// bindDrops binds each drop a report keeps to main exactly as the record's +// drops are bound, over the window the report classified: main and the +// line at the heads it was made from. Its errors read after "report ". +func (d *reportDoc) bindDrops() error { + if len(d.Overlays.Dropped) == 0 { + return nil + } + for _, h := range []struct{ name, head string }{{"main", d.Built.Main}, {d.Line, d.Built.Line}} { + have, err := lookup(h.head) + if err != nil { + return err + } + if have == "" { + return fmt.Errorf("was made on %s at %.12s, which this clone does not have; fetch %s and run again", h.name, h.head, h.name) + } + } + out, err := git("merge-base", "--end-of-options", d.Built.Main, d.Built.Line) + if err != nil { + return fmt.Errorf("was made from a line that shares no history with main: %w", err) + } + merges, _, err := mainMerges(strings.TrimSpace(out), d.Built.Main) + if err != nil { + return err + } + active, err := bindEntries(d.Overlays.Dropped, merges, d.Built.Main, func(e droppedEntry) string { + return fmt.Sprintf("drops #%d by %.12s", e.Original, e.Merge) + }) + if err != nil { + return err + } + // A drop decides only for an original the report classified, and only + // once every earlier rule has passed it by; an inherited one decides + // nothing. + where, err := d.categories() + if err != nil { + return err + } + for _, n := range d.Primary.Skipped.Dropped { + if _, ok := active[n]; !ok { + return fmt.Errorf("has #%d dropped, but no drop of it binds to an original it classified", n) + } + } + for n := range active { + switch c := where[n]; c { + case "skipped.dropped", "skipped.skip-label", "skipped.before-start", "skipped.type", "skipped.breaking", "skipped.touches", "skipped.on-line": + default: + return fmt.Errorf("drops #%d, which it has %s, though a drop decides before that", n, or(c, "nowhere")) + } + } + return nil +} + +// mergedOnMain says whether merge is on main's first-parent line up to +// mainHead and its message names pull request n, as mainMerges reads one. +func mergedOnMain(merge, mainHead string, n int) (bool, error) { + have, err := lookup(merge) + if err != nil || have == "" { + return false, err + } + if ok, err := isAncestor(merge, mainHead); err != nil || !ok { + return false, err + } + line, err := revList("--first-parent", "--end-of-options", mainHead) + if err != nil || !slices.Contains(line, merge) { + return false, err + } + msgs, err := commitMessages([]string{merge}) + if err != nil { + return false, err + } + number, _, _ := mergedPR(msgs[merge].message) + return number == n, nil +} diff --git a/cmd/backport-batch/drop_test.go b/cmd/backport-batch/drop_test.go new file mode 100644 index 000000000..f88b2d21c --- /dev/null +++ b/cmd/backport-batch/drop_test.go @@ -0,0 +1,576 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "slices" + "strings" + "testing" +) + +// recordOf is the record the open batch #90 carries now. +func recordOf(t *testing.T, gh *fakeGH) *lineRecord { + t.Helper() + rec, err := parseRecord(gh.find(90).Body, notifyLine) + if err != nil || rec == nil { + t.Fatalf("#90 carries no readable record: %v", err) + } + return rec +} + +func dropped(rec *lineRecord) []int { + var out []int + for _, d := range rec.Dropped { + out = append(out, d.Original) + } + return out +} + +// A drop takes a picked original off the batch and keeps it off the line: +// a run that continues the batch does not pick it, and nor does the next +// batch, which carries the drop over from the merged one. +func TestDropTakesAnOriginalOffTheLine(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + mc6 := f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + stdout := mustRun(t, f, gh, "run", "--line", "release-1.1", "--drop", "6", "--reason", "needs the rework of #4", "--push") + if got := f.subjects("origin/release-1.1", f.remoteHead(batchBranch1)); !slices.Equal(got, []string{"fix(app): five (#5)"}) { + t.Errorf("after the drop the batch carries %q, want #5 alone", got) + } + rec := recordOf(t, gh) + if want := (droppedEntry{Original: 6, Merge: mc6, Reason: "needs the rework of #4", On: 90, Date: "2026-10-26"}); len(rec.Dropped) != 1 || rec.Dropped[0] != want || rec.Built.Options.Mode != modeDrop { + t.Errorf("the record drops %+v (%s), want %+v", rec.Dropped, rec.Built.Options.Mode, want) + } + if s := bodyReportOf(t, stdout); !slices.Equal(s.doc.Primary.Skipped.Dropped, []int{6}) || !slices.Equal(droppedNumbers(s.doc.Overlays.Dropped), []int{6}) { + t.Errorf("the report has %v dropped and %v as the decision", s.doc.Primary.Skipped.Dropped, s.doc.Overlays.Dropped) + } + if !strings.Contains(stdout, "
Dropped from `release-1.1` by a maintainer") { + t.Errorf("the body does not list #6 as dropped:\n%s", stdout) + } + code, _, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut") + if code != 1 || !strings.Contains(stderr, "note: #6 is dropped from release-1.1 on #90 (needs the rework of #4): https://github.com/cozystack/cozystack/pull/6 ") { + t.Errorf("check --cut = %d, want #5 to fail it and a note on #6:\n%s", code, stderr) + } + + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + if got := f.subjects("origin/release-1.1", f.remoteHead(batchBranch1)); !slices.Equal(got, []string{"fix(app): five (#5)", "fix(app): seven (#7)"}) || !slices.Equal(dropped(recordOf(t, gh)), []int{6}) { + t.Errorf("the continued batch carries %q and drops %v", got, dropped(recordOf(t, gh))) + } + + mergeBatch(f, gh) + f.now = f.now.AddDate(0, 0, 3) + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + gh.nextPR = 91 + stdout = mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + rec, err := parseRecord(stdout, notifyLine) + if got := f.subjects("origin/release-1.1", f.remoteHead("backport-batch/release-1.1-20261029")); err != nil || !slices.Equal(got, []string{"fix(app): eight (#8)"}) || !slices.Equal(dropped(rec), []int{6}) { + t.Errorf("the next batch carries %q and drops %v (%v); want #8 alone, #6 still dropped", got, rec, err) + } +} + +// Dropping an original the batch does not carry changes only the record: +// the batch is not rebuilt, so a line that moved since does not move it. +func TestDropIsRecordOnlyWhenTheLineMoved(t *testing.T) { + f := newFixture(t) + f.lines() + f.onLine("release-1.1", "fix(app): a line-only seven", map[string]string{"app/seven.txt": "line\n"}) + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + gh := &fakeGH{} + head := openBatch(t, f, gh) + if recordOf(t, gh).deferred()[7].Commit == "" { + t.Fatal("#7 is not a recorded conflict") + } + f.onLine("release-1.1", "fix(chart): a line-only value", map[string]string{"chart/values.yaml": "replicas: 3\nimage: app:1\n"}) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--drop", "7", "--reason", "needs #4", "--push") + rec := recordOf(t, gh) + if f.remoteHead(batchBranch1) != head || !slices.Equal(dropped(rec), []int{7}) || len(rec.deferred()) != 0 || len(gh.edited) != 1 { + t.Errorf("the batch moved to %.12s (from %.12s), the record drops %v and defers %v; want only the record changed", f.remoteHead(batchBranch1), head, dropped(rec), rec.deferred()) + } +} + +// A drop is written to the open batch's record, and to no other: with no +// batch open, or none ever, it is refused and nothing is written. +func TestDropNeedsAnOpenBatch(t *testing.T) { + f := newFixture(t) + f.lines() + f.onLine("release-1.1", "fix(app): a line-only seven", map[string]string{"app/seven.txt": "line\n"}) + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--drop", "5", "--reason", "r", "--push"); code != 2 || !strings.Contains(stderr, "no open batch pull request on release-1.1 carries a record; a drop is recorded on one") { + t.Errorf("a drop on a line that never had a batch = %d:\n%s", code, stderr) + } + openBatch(t, f, gh) + mergeBatch(f, gh) + written := len(gh.edited) + len(gh.issueComments) + for _, args := range [][]string{{"--drop", "7", "--reason", "r", "--push"}, {"--undrop", "7", "--push"}} { + if code, _, stderr := f.exec(gh, append([]string{"run", "--line", "release-1.1"}, args...)...); code != 2 || !strings.Contains(stderr, "carries a record; a drop is recorded on one") { + t.Errorf("%q after the batch merged = %d:\n%s", args, code, stderr) + } + } + if len(gh.edited)+len(gh.issueComments) != written { + t.Error("a refused drop wrote to GitHub") + } +} + +// A drop is a decision kept beside whatever else decides an original: the +// rule, label or carry that applies first is its category, and the drop +// stays on record. backport/include does not lift it. +func TestDropOverlay(t *testing.T) { + setup := func(t *testing.T) (*fixture, *fakeGH) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.pr(7, "feat(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + gh := &fakeGH{labels: map[string][]int{labelInclude: {7}}} + openBatch(t, f, gh) + return f, gh + } + category := func(t *testing.T, gh *fakeGH, n int) string { + t.Helper() + s := bodyReportOf(t, gh.find(90).Body) + where, err := s.doc.categories() + if err != nil { + t.Fatal(err) + } + return where[n] + } + merge := func(f *fixture, n int) string { + mc, _, _ := strings.Cut(f.git("log", "--first-parent", "--format=%H", "--grep", fmt.Sprintf("(#%d)$", n), "main"), "\n") + return mc + } + drop := func(t *testing.T, f *fixture, gh *fakeGH, n string) { + t.Helper() + mustRun(t, f, gh, "run", "--line", "release-1.1", "--drop", n, "--reason", "not for 1.1", "--push") + } + t.Run("then backport/skip", func(t *testing.T) { + f, gh := setup(t) + drop(t, f, gh, "6") + gh.labels[labelSkip] = []int{6} + mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + if got := category(t, gh, 6); got != "skipped.skip-label" || !slices.Equal(dropped(recordOf(t, gh)), []int{6}) { + t.Errorf("#6 is %q, dropped %v", got, dropped(recordOf(t, gh))) + } + }) + t.Run("then picked onto the line by hand", func(t *testing.T) { + f, gh := setup(t) + drop(t, f, gh, "6") + pickOnLine(f, 6, map[string]string{"app/six.txt": "6\n"}) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + if got := category(t, gh, 6); got != "skipped.on-line" || !slices.Equal(dropped(recordOf(t, gh)), []int{6}) { + t.Errorf("#6 is %q, dropped %v", got, dropped(recordOf(t, gh))) + } + }) + t.Run("a feat backport/include let in", func(t *testing.T) { + f, gh := setup(t) + drop(t, f, gh, "7") + if got := category(t, gh, 7); got != "skipped.dropped" || slices.Contains(f.subjects("origin/release-1.1", f.remoteHead(batchBranch1)), "feat(app): seven (#7)") { + t.Errorf("#7 is %q; want it dropped and off the batch", got) + } + if !strings.Contains(gh.find(90).Body, "`backport/include`, skipped by a maintainer's drop") { + t.Errorf("the overrides do not say what became of #7:\n%s", gh.find(90).Body) + } + }) + t.Run("undropped", func(t *testing.T) { + f, gh := setup(t) + drop(t, f, gh, "6") + head := f.remoteHead(batchBranch1) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--undrop", "6", "--push") + if got := category(t, gh, 6); got != "missing" || len(recordOf(t, gh).Dropped) != 0 || f.remoteHead(batchBranch1) != head || recordOf(t, gh).Built.Options.Mode != modeUndrop { + t.Errorf("#6 is %q, dropped %v; want it missing until the next run, the batch where it was", got, dropped(recordOf(t, gh))) + } + if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/6 ") { + t.Errorf("check = %d, want #6 missing:\n%s", code, stdout) + } + mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + if got := category(t, gh, 6); got != "picked" { + t.Errorf("after the next run #6 is %q, want it picked again", got) + } + }) + t.Run("a new original during a change to the record alone", func(t *testing.T) { + f, gh := setup(t) + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + drop(t, f, gh, "7") + mustRun(t, f, gh, "run", "--line", "release-1.1", "--undrop", "7", "--push") + if got := category(t, gh, 8); got != "missing" { + t.Errorf("#8 is %q, want missing until a run tries it", got) + } + }) + // A stored drop is bound to the merge it names: one that names another + // original's, or one of no merge of it, is refused; one of an original + // merged before the line branched is kept and decides nothing. + t.Run("a stored drop of an earlier commit that names the original", func(t *testing.T) { + // Before release-1.1 branched, a commit on main already ended in + // (#6); the drop has to name the merge of #6 the line can take. + f := newFixture(t) + f.release("release-1.0") + f.tag("v1.0.0") + early := f.squash(6, "docs: what six will change", "", map[string]string{"docs/six.md": "6\n"}) + f.release("release-1.1") + f.tag("v1.1.0") + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + drop(t, f, gh, "6") + gh.find(90).Body = strings.Replace(gh.find(90).Body, `{"original":6,"merge":"`+merge(f, 6)+`"`, `{"original":6,"merge":"`+early+`"`, 1) + if code, _, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 2 || !strings.Contains(stderr, "but #6 merged as") { + t.Errorf("check = %d, want 2 naming the merge of #6:\n%s", code, stderr) + } + }) + t.Run("a stored drop of a commit inside a pull request", func(t *testing.T) { + // A commit of a pull request's own branch can end in its number too, + // but only the merge on main's first-parent line is the original. + f := newFixture(t) + f.release("release-1.0") + f.tag("v1.0.0") + mc3 := f.prCommits(3, "feat(app): three", "", change{files: map[string]string{"app/three.txt": "3\n"}, message: "feat(app): the part of three (#3)"}) + inner := f.git("rev-parse", mc3+"^2") + f.release("release-1.1") + f.tag("v1.1.0") + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + drop(t, f, gh, "6") + gh.find(90).Body = strings.Replace(gh.find(90).Body, `{"original":6,"merge":"`+merge(f, 6)+`"`, `{"original":3,"merge":"`+inner+`"`, 1) + if code, _, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 2 || !strings.Contains(stderr, "which is no merge of it on main") { + t.Errorf("check = %d, want 2:\n%s", code, stderr) + } + }) + for _, tc := range []struct { + name string + original func(f *fixture) (int, string) + refused bool + }{ + {"another original's merge", func(f *fixture) (int, string) { return 6, merge(f, 5) }, true}, + {"a commit that merged no pull request", func(f *fixture) (int, string) { return 6, f.git("rev-list", "--max-parents=0", "main") }, true}, + {"a merge from before the line branched, of another pull request", func(f *fixture) (int, string) { return 2, merge(f, 1) }, true}, + {"a merge from before the line branched", func(f *fixture) (int, string) { return 1, merge(f, 1) }, false}, + } { + t.Run("a stored drop of "+tc.name, func(t *testing.T) { + f, gh := setup(t) + drop(t, f, gh, "6") + n, mc := tc.original(f) + gh.find(90).Body = strings.Replace(gh.find(90).Body, `{"original":6,"merge":"`+merge(f, 6)+`"`, fmt.Sprintf(`{"original":%d,"merge":"%s"`, n, mc), 1) + code, _, stderr := f.exec(gh, "check", "--line", "release-1.1") + switch { + case tc.refused && (code != 2 || !strings.Contains(stderr, fmt.Sprintf("drops #%d by %.12s", n, mc))): + t.Errorf("check = %d, want 2 naming the drop:\n%s", code, stderr) + case !tc.refused && code == 2: + t.Errorf("check = %d, want the drop kept:\n%s", code, stderr) + case !tc.refused: + mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + if !slices.Equal(dropped(recordOf(t, gh)), []int{1}) || category(t, gh, 6) != "picked" { + t.Errorf("the next run drops %v and puts #6 in %q; want #1's drop carried on and #6 picked", dropped(recordOf(t, gh)), category(t, gh, 6)) + } + } + }) + } +} + +// What a drop cannot do is refused before anything is touched. +func TestDropRefuses(t *testing.T) { + f := newFixture(t) + f.lines() + mc5 := f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{labels: map[string][]int{labelSkip: {7, 9}}} + head := openBatch(t, f, gh) + f.onLine("release-1.1", "fix(app): five\n\n(cherry picked from commit "+mc5+")", map[string]string{"app/five.txt": "5\n"}) + // The line carries these too, by hand, while backport/skip and rule 2 + // decide them before the line is looked at. + mc7 := f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + mc8 := f.pr(8, "feat(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + f.onLine("release-1.1", "fix(app): seven\n\n(cherry picked from commit "+mc7+")", map[string]string{"app/seven.txt": "7\n"}) + f.onLine("release-1.1", "feat(app): eight\n\n(cherry picked from commit "+mc8+")", map[string]string{"app/eight.txt": "8\n"}) + // #9 is on the line only through a pick of its own commit, which a later + // line change has since moved away from what main has. + mc9 := f.pr(9, "fix(app): nine", "", map[string]string{"app/nine.txt": "9\n"}) + f.onLine("release-1.1", "fix(app): nine\n\n(cherry picked from commit "+f.git("rev-parse", mc9+"^2")+")", map[string]string{"app/nine.txt": "9\n"}) + f.onLine("release-1.1", "fix(app): nine, as the line needs it", map[string]string{"app/nine.txt": "nine on release-1.1\n"}) + for _, tc := range []struct { + args []string + why string + }{ + {[]string{"--drop", "4242", "--reason", "r"}, "#4242 did not merge into main"}, + {[]string{"--drop", "5", "--reason", "r"}, "already on release-1.1; a drop cannot remove code the line has"}, + {[]string{"--drop", "7", "--reason", "r"}, "#7 is already on release-1.1; a drop cannot remove code the line has"}, + {[]string{"--drop", "8", "--reason", "r"}, "#8 is already on release-1.1; a drop cannot remove code the line has"}, + {[]string{"--drop", "9", "--reason", "r"}, "#9 is already on release-1.1; a drop cannot remove code the line has"}, + {[]string{"--drop", "6"}, "--drop needs a --reason"}, + {[]string{"--drop", "6", "--reason", "r", "--defer", "6"}, "give one"}, + {[]string{"--drop", "6", "--reason", "r", "--continue"}, "takes no --onto, --continue or --date"}, + {[]string{"--drop", "6", "--reason", "r", "--onto", head}, "takes no --onto, --continue or --date"}, + {[]string{"--drop", "6", "--reason", "r", "--date", "2026-10-26"}, "takes no --onto, --continue or --date"}, + {[]string{"--undrop", "6", "--reason", "r"}, "--undrop takes no --reason"}, + {[]string{"--undrop", "6"}, "#6 is not dropped from release-1.1"}, + {[]string{"--drop", "6", "--undrop", "6", "--reason", "r"}, "give one"}, + {[]string{"--drop", "six", "--reason", "r"}, "not a list of pull request numbers"}, + } { + code, _, stderr := f.exec(gh, append(append([]string{"run", "--line", "release-1.1"}, tc.args...), "--push")...) + f.git("switch", "-q", "main") + if code != 2 || !strings.Contains(stderr, tc.why) { + t.Errorf("run %q = %d, want 2 with %q:\n%s", tc.args, code, tc.why, stderr) + } + } + if f.remoteHead(batchBranch1) != head || len(gh.edited) != 0 || len(reportsPosted(gh)) != 1 { + t.Errorf("a refused drop moved the batch to %.12s or wrote %q", f.remoteHead(batchBranch1), gh.edited) + } +} + +// A drop or an undrop asked for while a close was left changes nothing: the +// run finishes the close, and no open batch pull request is left to record +// it on. +func TestDropDuringAnUnfinishedClose(t *testing.T) { + deferAll := []string{"run", "--line", "release-1.1", "--defer", "5", "--reason", "fails the app e2e", "--push"} + for _, tc := range []struct { + name string + stop func(f *fixture, gh *fakeGH) + refused bool + args []string + }{ + {"a drop after the close failed", func(f *fixture, gh *fakeGH) { gh.fail = map[string]error{"closePull": errString("HTTP 502")} }, false, []string{"--drop", "6", "--reason", "not for 1.1"}}, + {"an undrop after the delete failed", func(f *fixture, gh *fakeGH) { + gh.after = map[string]func(){"closePull": func() { f.refusePushes(true) }} + }, true, []string{"--undrop", "6"}}, + } { + t.Run(tc.name, func(t *testing.T) { + f, gh, _ := closeWorld(t) + tc.stop(f, gh) + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + if tc.refused { + f.refusePushes(false) + } + gh.after = nil + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + code, _, stderr := f.exec(gh, append([]string{"run", "--line", "release-1.1", "--push"}, tc.args...)...) + f.git("switch", "-q", "main") + rec, err := parseRecord(gh.find(90).Body, notifyLine) + if code != 2 || !strings.Contains(stderr, "finished the close of #90 and made no other change") || gh.find(90).State != "closed" || f.remoteHead(batchBranch1) != "" || err != nil || len(rec.Dropped) != 0 { + t.Errorf("run %q = %d, #90 %s, branch %q, record %+v, %v; want 2, the close finished and nothing dropped:\n%s", tc.args, code, gh.find(90).State, f.remoteHead(batchBranch1), rec, err, stderr) + } + }) + } +} + +// The batch carries an original whose first commit the line already had, so +// its pick came out empty and its unit holds the rest: a drop still takes +// that unit off. +func TestDropTakesOffAUnitWithAnEmptyPick(t *testing.T) { + f := newFixture(t) + f.lines() + f.onLine("release-1.1", "fix(app): the first part, on its own", map[string]string{"app/already.txt": "first\n"}) + f.prCommits(5, "fix(app): a fix in two parts", "", + change{files: map[string]string{"app/already.txt": "first\n"}}, + change{files: map[string]string{"app/new.txt": "second\n"}}, + ) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--drop", "5", "--reason", "not for 1.1", "--push") + head := f.remoteHead(batchBranch1) + if got := f.subjects("origin/release-1.1", head); !slices.Equal(got, []string{"fix(app): six (#6)"}) || f.git("ls-tree", "--name-only", head, "--", "app/new.txt") != "" { + t.Errorf("after the drop the batch carries %q, want #6 alone and nothing of #5", got) + } +} + +// A report keeps each drop the record keeps, with the merge commit it is +// bound to, its reason and the batch pull request that recorded it, so an +// earlier report still says why after the drop is withdrawn. +func TestReportKeepsEachDrop(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + mc6 := f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + stdout := mustRun(t, f, gh, "run", "--line", "release-1.1", "--drop", "6", "--reason", "needs the rework of #4", "--push") + dropped := bodyReportOf(t, stdout) + want := `"overlays":{"dropped":[{"original":6,"merge":"` + mc6 + `","reason":"needs the rework of #4","on":90,"date":"2026-10-26"}]` + if !strings.Contains(dropped.raw, want) { + t.Errorf("the report does not keep the drop with its merge and reason:\n%s", dropped.raw) + } + mustRun(t, f, gh, "run", "--line", "release-1.1", "--undrop", "6", "--push") + if raw := bodyReportOf(t, gh.find(90).Body).raw; !strings.Contains(raw, `"overlays":{"dropped":[],`) { + t.Errorf("the report after the undrop still keeps a drop:\n%s", raw) + } + code, out, stderr := f.exec(gh, "report", "--pr", "90", "--id", dropped.doc.ID) + if code != 0 || !strings.Contains(out, "dropped: #6, merged as "+mc6[:12]+", on #90: needs the rework of #4\n") { + t.Errorf("report --id of the earlier report = %d, want it to show the drop:\n%s%s", code, out, stderr) + } +} + +// A report's drop is held to the record's rules and bound to main as the +// record's drops are: a reason a run does not write, another original's +// merge, or an earlier commit on main that names the same number is +// refused, however the ID was made to match, in the body and in a trusted +// report comment alike. +func TestReportRefusesAForgedDrop(t *testing.T) { + f := newFixture(t) + f.release("release-1.0") + f.tag("v1.0.0") + early := f.squash(6, "docs: what six will change", "", map[string]string{"docs/six.md": "6\n"}) + f.release("release-1.1") + f.tag("v1.1.0") + mc5 := f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + mc6 := f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--drop", "6", "--reason", "not for 1.1", "--push") + body, comments := gh.find(90).Body, slices.Clone(gh.issueComments) + s := bodyReportOf(t, body) + for _, tc := range []struct{ name, old, new, why string }{ + {"a reason of two lines", `"reason":"not for 1.1"`, `"reason":"not for 1.1\nwarning: forged"`, "which is no reason a run writes"}, + {"another original's merge", `"merge":"` + mc6 + `"`, `"merge":"` + mc5 + `"`, "which merged #5"}, + {"an earlier commit that names the same number", `"merge":"` + mc6 + `"`, `"merge":"` + early + `"`, "but #6 merged as " + mc6[:12]}, + } { + raw := strings.Replace(s.raw, tc.old, tc.new, 1) + if raw == s.raw { + t.Fatalf("the report holds no %s to change:\n%s", tc.old, s.raw) + } + var d reportDoc + if err := decodeStrict(raw, &d); err != nil { + t.Fatalf("the changed report does not decode: %v", err) + } + d.ID = reportID(d) + forged := storedReport{doc: d, raw: reportJSON(d)} + refused := func(t *testing.T, args ...string) { + t.Helper() + code, out, stderr := f.exec(gh, append([]string{"report", "--pr", "90"}, args...)...) + if code != 2 || !strings.Contains(stderr, tc.why) || strings.Contains(out+stderr, "\nwarning: forged") { + t.Errorf("report of a forged drop = %d, want 2 with %q:\n%s%s", code, tc.why, out, stderr) + } + } + t.Run(tc.name+", in the body", func(t *testing.T) { + gh.find(90).Body = strings.Replace(body, reportBlock(s.raw), reportBlock(forged.raw), 1) + defer func() { gh.find(90).Body = body }() + refused(t) + }) + t.Run(tc.name+", in a comment", func(t *testing.T) { + text, err := reportComment(forged) + if err != nil { + t.Fatal(err) + } + c := issueComment{ID: 9000, Body: text} + c.User.Login, c.User.Type = "maintainer", "User" + gh.issueComments = append(slices.Clone(comments), c) + defer func() { gh.issueComments = comments }() + refused(t, "--id", d.ID) + }) + } +} + +// droppedNumbers are the originals a report's drops name. +func droppedNumbers(es []droppedEntry) []int { + var out []int + for _, e := range es { + out = append(out, e.Original) + } + return out +} + +// dropWorld has #3 merged before release-1.1 branched, an earlier commit on +// main that names #6, and a batch with #5 and #6 from which #6 is dropped. +func dropWorld(t *testing.T) (f *fixture, gh *fakeGH, inherited, mc5, mc6 string) { + t.Helper() + f = newFixture(t) + f.release("release-1.0") + f.tag("v1.0.0") + inherited = f.prCommits(3, "feat(app): three", "", change{files: map[string]string{"app/three.txt": "3\n"}, message: "feat(app): part of three (#3)"}) + f.squash(6, "docs: what six will change", "", map[string]string{"docs/six.md": "6\n"}) + f.release("release-1.1") + f.tag("v1.1.0") + mc5 = f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + mc6 = f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh = &fakeGH{} + openBatch(t, f, gh) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--drop", "6", "--reason", "not for 1.1", "--push") + return f, gh, inherited, mc5, mc6 +} + +// A report's drops and its categories hold each other up: an original it +// has dropped needs a drop that binds to it, and a drop of an original it +// classified needs that original dropped or decided by an earlier rule. A +// drop inherited from before the line branched decides nothing and stays +// readable. Each holds in the body and in a trusted report comment. +func TestReportTiesDropsToTheirCategory(t *testing.T) { + f, gh, inherited, mc5, _ := dropWorld(t) + body, comments := gh.find(90).Body, slices.Clone(gh.issueComments) + s := bodyReportOf(t, body) + kept := func(n int, merge string) droppedEntry { + return droppedEntry{Original: n, Merge: merge, Reason: "carried from an earlier batch", On: 89, Date: "2026-10-25"} + } + for _, tc := range []struct { + name string + change func(d *reportDoc) + why string + }{ + {"an inherited drop in place of the classified one", func(d *reportDoc) { d.Overlays.Dropped[0] = kept(3, inherited) }, "has #6 dropped, but keeps no drop of it"}, + {"an inherited drop classified as dropped", func(d *reportDoc) { + d.Primary.Skipped.Dropped = append(d.Primary.Skipped.Dropped, 3) + d.Overlays.Dropped = append(d.Overlays.Dropped, kept(3, inherited)) + }, "has #3 dropped, but no drop of it binds to an original it classified"}, + {"a drop of an original it picked", func(d *reportDoc) { d.Overlays.Dropped = append(d.Overlays.Dropped, kept(5, mc5)) }, "drops #5, which it has picked"}, + {"an inherited drop beside the classified one", func(d *reportDoc) { d.Overlays.Dropped = append(d.Overlays.Dropped, kept(3, inherited)) }, ""}, + } { + var d reportDoc + if err := decodeStrict(s.raw, &d); err != nil { + t.Fatal(err) + } + tc.change(&d) + d.canonical() + d.ID = reportID(d) + forged := storedReport{doc: d, raw: reportJSON(d)} + check := func(t *testing.T, args ...string) { + t.Helper() + code, out, stderr := f.exec(gh, append([]string{"report", "--pr", "90"}, args...)...) + switch { + case tc.why == "" && (code != 0 || !strings.Contains(out, "#6 skipped.dropped") || !strings.Contains(out, "dropped: #3, merged as "+inherited[:12]) || strings.Contains(out, "#3 skipped.dropped")): + t.Errorf("report of a sound report = %d, want it read with #3's drop beside #6's:\n%s%s", code, out, stderr) + case tc.why != "" && (code != 2 || out != "" || !strings.Contains(stderr, tc.why)): + t.Errorf("report = %d, want 2 with %q:\n%s%s", code, tc.why, out, stderr) + } + } + t.Run(tc.name+", in the body", func(t *testing.T) { + gh.find(90).Body = strings.Replace(body, reportBlock(s.raw), reportBlock(forged.raw), 1) + defer func() { gh.find(90).Body = body }() + check(t) + }) + t.Run(tc.name+", in a comment", func(t *testing.T) { + text, err := reportComment(forged) + if err != nil { + t.Fatal(err) + } + c := issueComment{ID: 9001, Body: text} + c.User.Login, c.User.Type = "maintainer", "User" + gh.issueComments = append(slices.Clone(comments), c) + defer func() { gh.issueComments = comments }() + check(t, "--id", d.ID) + }) + } +} diff --git a/cmd/backport-batch/fold_test.go b/cmd/backport-batch/fold_test.go index 87f46f681..89797d0d5 100644 --- a/cmd/backport-batch/fold_test.go +++ b/cmd/backport-batch/fold_test.go @@ -330,3 +330,21 @@ func TestFold(t *testing.T) { } }) } + +// A resolution of an original a maintainer dropped from the line is refused: +// the drop is the decision that it does not go there. +func TestFoldRefusesADroppedOriginal(t *testing.T) { + w := newFoldWorld(t) + sha := strings.Repeat("a", 40) + rec := strings.Replace(v2(""), `"dropped":[{"original":8,"merge":"`+sha+`"`, `"dropped":[{"original":5,"merge":"`+w.mc5+`"`, 1) + rec = strings.Replace(rec, `"deferred":[{"original":5,"commit":"`+sha+`","files":["app/five.txt"]},`, `"deferred":[`, 1) + rec = strings.Replace(rec, `"stale":[9]`, `"stale":[]`, 1) + w.gh.prs[0].Body = "text\n\n" + deferredFence + "\n" + rec + "\n```\n" + if _, err := parseRecord(w.gh.prs[0].Body, notifyLine); err != nil { + t.Fatal(err) + } + code, stdout, stderr := w.f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", w.resolution()) + if code != 1 || !strings.Contains(stdout, "#5 is not one the batch takes: a maintainer's drop, dropped on #90: needs #4") { + t.Errorf("fold = %d, %q, %s; want a refusal naming the drop", code, stdout, stderr) + } +} diff --git a/cmd/backport-batch/lines.go b/cmd/backport-batch/lines.go index 2e3c618f2..275be00f9 100644 --- a/cmd/backport-batch/lines.go +++ b/cmd/backport-batch/lines.go @@ -207,29 +207,35 @@ func mainMerges(base, mainHead string) ([]*original, []string, error) { var merges []*original var other []string for _, m := range list { - // The subject is the first paragraph, its lines joined, as git reads it. - msg := strings.TrimRight(msgs[m.oid].message, "\n") - subject, body, _ := strings.Cut(msg, "\n\n") - subject = strings.ReplaceAll(subject, "\n", " ") o := &original{merge: m.oid, parents: m.parents, merged: msgs[m.oid].committed} - if pr := mergePRRE.FindStringSubmatch(subject); pr != nil { - o.number, _ = strconv.Atoi(pr[1]) - body = strings.TrimLeft(body, "\n") - o.title, body, _ = strings.Cut(body, "\n") - } else if pr := prNumberRE.FindStringSubmatchIndex(subject); pr != nil { - o.number, _ = strconv.Atoi(subject[pr[2]:pr[3]]) - o.title = strings.TrimSpace(subject[:pr[0]]) - } + o.number, o.title, o.body = mergedPR(msgs[m.oid].message) if o.number <= 0 || o.parents > 2 { other = append(other, m.oid) continue } - o.body = strings.TrimLeft(body, "\n") merges = append(merges, o) } return merges, other, nil } +// mergedPR reads the pull request a commit on main's first-parent line +// merged from its message, with its title and body; 0 when it names none. +func mergedPR(message string) (int, string, string) { + // The subject is the first paragraph, its lines joined, as git reads it. + subject, body, _ := strings.Cut(strings.TrimRight(message, "\n"), "\n\n") + subject = strings.ReplaceAll(subject, "\n", " ") + number, title := 0, "" + if pr := mergePRRE.FindStringSubmatch(subject); pr != nil { + number, _ = strconv.Atoi(pr[1]) + body = strings.TrimLeft(body, "\n") + title, body, _ = strings.Cut(body, "\n") + } else if pr := prNumberRE.FindStringSubmatchIndex(subject); pr != nil { + number, _ = strconv.Atoi(subject[pr[2]:pr[3]]) + title = strings.TrimSpace(subject[:pr[0]]) + } + return number, title, strings.TrimLeft(body, "\n") +} + // prCommits lists what an original brings to a line, one pick each: for a // merged pull request every commit of its own, oldest first, leaving out the // merges that brought main into its branch, whose content main already has; diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index bd0e02d30..c02601240 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -34,6 +34,8 @@ Usage: backport-batch run --line release-X.Y [--onto SHA] [--date YYYY-MM-DD] [--no-workflow-files] [--push] [--policy REF] backport-batch run --line release-X.Y --continue [--no-workflow-files] [--push] [--policy REF] backport-batch run --line release-X.Y --defer N[,N] --reason TEXT [--push] [--policy REF] + backport-batch run --line release-X.Y --drop N[,N] --reason TEXT [--push] [--policy REF] + backport-batch run --line release-X.Y --undrop N[,N] [--push] [--policy REF] backport-batch check --line release-X.Y [--cut] [--policy REF] backport-batch fold --line release-X.Y --original N --commit SHA [--policy REF] backport-batch verify --line release-X.Y [--head SHA] @@ -56,6 +58,11 @@ run Builds backport-batch/- in this working tree, off them out, the next one tries them again. --push replaces the branch, leased on the head the rebuild started from; when nothing is left it closes the batch PR instead, after recording why in it. + --drop drops originals from the line for --reason: the line's record + keeps them out of this batch and every later one. One the open batch + carries is taken off it as --defer would; otherwise only the record + changes. --undrop withdraws a drop. Both need an open batch PR whose + record they change. check Exits 1 while a change the policy takes is neither on the line nor on the open batch PR, 0 when nothing is, 2 when it cannot tell. With --cut, the gate before an rc or a patch is cut, it also exits 1 while a change @@ -79,7 +86,10 @@ lines Prints the release lines the batch runs for, newest first. The policy is docs/release/backports.md. run, check, fold and lines read its policy block from that page on origin/main, or on --policy REF. Every command -but notify and report, which read only GitHub, fetches origin first. +but notify and report fetches origin first. notify reads only GitHub; report +reads GitHub, and for a report that keeps drops also this clone's history, +which has to hold the main and line heads the report was made from: report +never fetches on its own. BACKPORT_BATCH_BOT names the bot the scheduled batch writes as, [bot], cozystack-ci[bot] when unset: a batch pull request or a notify @@ -220,8 +230,13 @@ type batch struct { prHead string // picks is what the open batch pull request carries beyond the line. picks *history - // src is the line's record and where it was found. - src recordSource + // src is the line's record and where it was found, and drops its drops + // that name an original of this batch. + src recordSource + drops map[int]droppedEntry + // lineHist and newerHist are what the line and the newer line carry, + // read once for every original judged. + lineHist, newerHist *history // start is when the line started under the policy, zero for its fork point. start time.Time // textUntil is the batch-only day: what a commit's text says counts as @@ -292,16 +307,14 @@ func (a *app) prepare(l line) (*batch, error) { return nil, err } b.other = other - lineHist, err := readHistory(b.lineHead, b.mainHead, b.textUntil) - if err != nil { + if b.lineHist, err = readHistory(b.lineHead, b.mainHead, b.textUntil); err != nil { return nil, err } - var newerHist *history if b.newer != nil { if b.newerHead, err = resolve("refs/remotes/" + remote + "/" + b.newer.branch()); err != nil { return nil, err } - if newerHist, err = readHistory(b.newerHead, b.mainHead, b.textUntil); err != nil { + if b.newerHist, err = readHistory(b.newerHead, b.mainHead, b.textUntil); err != nil { return nil, err } } @@ -332,19 +345,33 @@ func (a *app) prepare(l line) (*batch, error) { if b.src, err = findLineRecord(a.gh, l, pr); err != nil { return nil, err } + if b.drops, err = b.bindDrops(merges); err != nil { + return nil, err + } for _, o := range merges { - v, err := judge(o, labels[o.number], b.start, lineHist, b.newer, newerHist) - if err != nil { + v := &verdict{o: o, label: labels[o.number]} + if err := b.judge(v, b.drops); err != nil { return nil, err } - if v.rule == 0 && b.picks != nil { - v.onBatch, _ = b.picks.found(v.o, v.commits) - } b.verdicts = append(b.verdicts, v) } return b, nil } +// judge judges v again under drops, and finds it on the open batch when the +// policy takes it. +func (b *batch) judge(v *verdict, drops map[int]droppedEntry) error { + j, err := judge(v.o, v.label, b.start, b.lineHist, b.newer, b.newerHist, drops) + if err != nil { + return err + } + *v = *j + if v.rule == 0 && b.picks != nil { + v.onBatch, _ = b.picks.found(v.o, v.commits) + } + return nil +} + // overrides maps each pull request merged into main that carries a backport/ // label to the one that decides; backport/skip wins over backport/include. func overrides(gh github) (map[int]string, error) { @@ -366,8 +393,9 @@ func overrides(gh github) (map[int]string, error) { // started under the policy after it branched takes nothing merged before that // start. backport/skip keeps an original out whatever the rules say, and // backport/include lifts the start and rules 2 to 4, but not what the line or -// the newer line carries. -func judge(o *original, label string, start time.Time, lineHist *history, newer *line, newerHist *history) (*verdict, error) { +// the newer line carries. A maintainer's drop keeps out what the line does +// not carry, include or not. +func judge(o *original, label string, start time.Time, lineHist *history, newer *line, newerHist *history, drops map[int]droppedEntry) (*verdict, error) { v := &verdict{o: o, label: label} if label == labelSkip { v.rule, v.reason = ruleSkipLabel, labelSkip @@ -410,6 +438,10 @@ func judge(o *original, label string, start time.Time, lineHist *history, newer v.rule, v.reason = ruleOnLine, how return v, nil } + if d, ok := drops[o.number]; ok { + v.rule, v.reason = ruleDropped, fmt.Sprintf("dropped on #%d: %s", d.On, d.Reason) + return v, nil + } if newer != nil { how, err := newerHist.carries(o, v.commits) if err != nil { diff --git a/cmd/backport-batch/record.go b/cmd/backport-batch/record.go index 4f504ee55..4ab43933e 100644 --- a/cmd/backport-batch/record.go +++ b/cmd/backport-batch/record.go @@ -102,6 +102,11 @@ var ( reportIDRE = regexp.MustCompile(`^[0-9a-f]{64}$`) ) +// oneLineText is s as a run writes a reason: one line, cleaned and capped. +// A record holding anything else was not written by a run, and its text +// would reach a terminal, a body or a comment as someone else wrote it. +func oneLineText(s string) string { return capped(oneLine(clean(s, false)), titleLimit) } + // fencedLines lists the one line of each block that fence opens. func fencedLines(body, fence string) ([]string, error) { var blocks []string @@ -220,6 +225,10 @@ func (rec *lineRecord) validate(l line) error { return fmt.Errorf("does not say whether #%d is a conflict, deferred for a reason or waiting", d.Original) case d.Reason != "" && strings.TrimSpace(d.Reason) == "": return fmt.Errorf("defers #%d for an empty reason", d.Original) + case d.Reason != oneLineText(d.Reason): + return fmt.Errorf("defers #%d for %q, which is no reason a run writes", d.Original, d.Reason) + case !slices.Equal(d.Files, cleanFiles(d.Files)): + return fmt.Errorf("names files of #%d that are not as a run writes them: %q", d.Original, d.Files) case d.More < 0: return fmt.Errorf("counts %d more files for #%d", d.More, d.Original) } @@ -231,6 +240,9 @@ func (rec *lineRecord) validate(l line) error { if _, err := time.Parse(time.DateOnly, d.Date); err != nil || checkSHA(d.Merge) != nil || strings.TrimSpace(d.Reason) == "" || d.On <= 0 { return fmt.Errorf("drops #%d with a merge, reason, pull request or date it cannot have", d.Original) } + if d.Reason != oneLineText(d.Reason) { + return fmt.Errorf("drops #%d for %q, which is no reason a run writes", d.Original, d.Reason) + } } staleSeen := map[int]bool{} for _, n := range rec.Stale { @@ -306,6 +318,14 @@ func (s recordSource) savedNoWorkflowFiles() bool { return slices.ContainsFunc(s.rec.Deferred, func(d deferredEntry) bool { return d.Workflows }) } +// dropped is what the record drops, which every later record keeps. +func (rec *lineRecord) dropped() []droppedEntry { + if rec == nil { + return nil + } + return slices.Clone(rec.Dropped) +} + // deferred maps the originals the record defers to their entries. func (rec *lineRecord) deferred() map[int]deferredEntry { out := map[int]deferredEntry{} @@ -343,10 +363,12 @@ func prState(pr *pullRequest) string { switch { case pr.MergedAt != nil: return "merged" - case pr.State == "": + case pr.State == "" || pr.State == "open": return "open" + case pr.State == "closed": + return "closed" } - return pr.State + return fmt.Sprintf("in state %q", pr.State) } func hasRecord(body string) bool { diff --git a/cmd/backport-batch/record_test.go b/cmd/backport-batch/record_test.go index 10b3b3f7f..13ab6cc79 100644 --- a/cmd/backport-batch/record_test.go +++ b/cmd/backport-batch/record_test.go @@ -87,6 +87,12 @@ func TestRecordRefuses(t *testing.T) { "an unknown key": v2(`,"parked":[]`), "an entry of no kind": strings.Replace(v2(""), `{"original":6,"files":[],"reason":"fails the e2e"}`, `{"original":6,"files":[]}`, 1), "another line": strings.Replace(v2(""), `"line":"release-1.1"`, `"line":"release-1.0"`, 1), + "an escape in a reason": strings.Replace(v2(""), `"reason":"fails the e2e"`, `"reason":"fails\u001b[2J the e2e"`, 1), + "a newline in a reason": strings.Replace(v2(""), `"reason":"fails the e2e"`, `"reason":"fails\nwarning: forged"`, 1), + "an escape in a drop": strings.Replace(v2(""), `"reason":"needs #4"`, `"reason":"needs\u001b]0;x\u0007 #4"`, 1), + "a newline in a drop": strings.Replace(v2(""), `"reason":"needs #4"`, `"reason":"needs #4\nnote: forged"`, 1), + "a reason past its length": strings.Replace(v2(""), `"reason":"needs #4"`, `"reason":"`+strings.Repeat("x", titleLimit+1)+`"`, 1), + "a bidi file name": strings.Replace(v2(""), `"files":["app/five.txt"]`, `"files":["app/\u202efive.txt"]`, 1), } { if _, err := parseRecord(block(json), l); err == nil { t.Errorf("%s: parseRecord accepted it", name) @@ -293,8 +299,10 @@ func TestMetadataOperationsKeepTheMode(t *testing.T) { for _, v1 := range []bool{false, true} { t.Run(fmt.Sprintf("an open record of version 1: %v", v1), func(t *testing.T) { f := workflowWorld(t) + f.onLine("release-1.1", "fix(app): a line-only nine", map[string]string{"app/nine.txt": "line\n"}) f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + f.pr(9, "fix(app): nine", "", map[string]string{"app/nine.txt": "9\n"}) gh := &fakeGH{} stdout := mustRun(t, f, gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push") reopen(f, gh) @@ -304,11 +312,23 @@ func TestMetadataOperationsKeepTheMode(t *testing.T) { } else if rec, err := parseRecord(stdout, notifyLine); err != nil || !rec.Built.Options.NoWorkflowFiles { t.Fatalf("the scheduled run saved %+v, %v", rec, err) } - stdout = mustRun(t, f, gh, "run", "--line", "release-1.1", "--defer", "7", "--reason", "fails the e2e", "--push") - rec, err := parseRecord(stdout, notifyLine) - if err != nil || rec.Built.Options != (runOptions{NoWorkflowFiles: true, Mode: modeDefer}) { - t.Errorf("after --defer the record is %+v, %v; want the flag kept", rec.Built, err) + for _, op := range []struct { + args []string + mode string + }{ + {[]string{"--drop", "9", "--reason", "needs #4"}, modeDrop}, + {[]string{"--undrop", "9"}, modeUndrop}, + {[]string{"--defer", "7", "--reason", "fails the e2e"}, modeDefer}, + } { + stdout = mustRun(t, f, gh, append(append([]string{"run", "--line", "release-1.1"}, op.args...), "--push")...) + rec, err := parseRecord(stdout, notifyLine) + if err != nil || rec.Built.Options != (runOptions{NoWorkflowFiles: true, Mode: op.mode}) { + t.Errorf("after %q the record is %+v, %v; want the flag kept", op.args, rec.Built, err) + } + reopen(f, gh) } + var rec *lineRecord + var err error reopen(f, gh) if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#5 changes .github/workflows/") { t.Errorf("check --cut = %d, want #5 still waiting:\n%s", code, stdout) @@ -327,3 +347,22 @@ func TestMetadataOperationsKeepTheMode(t *testing.T) { }) } } + +// A state is named only when it is one GitHub uses; anything else is quoted. +func TestPRStateNamesOnlyKnownStates(t *testing.T) { + at := "2026-10-26T12:00:00Z" + for _, tc := range []struct { + pr pullRequest + want string + }{ + {pullRequest{State: "open"}, "open"}, + {pullRequest{State: ""}, "open"}, + {pullRequest{State: "closed"}, "closed"}, + {pullRequest{State: "closed", MergedAt: &at}, "merged"}, + {pullRequest{State: "open\n\x1b[2Jforged"}, `in state "open\n\x1b[2Jforged"`}, + } { + if got := prState(&tc.pr); got != tc.want { + t.Errorf("prState(%q) = %q, want %q", tc.pr.State, got, tc.want) + } + } +} diff --git a/cmd/backport-batch/report.go b/cmd/backport-batch/report.go index 78c460479..46e37c955 100644 --- a/cmd/backport-batch/report.go +++ b/cmd/backport-batch/report.go @@ -110,6 +110,8 @@ func (s *reportSkipped) list(rule int) *[]int { return &s.Newer case ruleSkipLabel: return &s.SkipLabel + case ruleDropped: + return &s.Dropped } return nil } @@ -180,6 +182,7 @@ func buildReport(b *batch, r *report) (reportDoc, error) { *list = append(*list, numbers(vs)...) } doc.Overlays = reportOverlays{Dropped: []droppedEntry{}, Stale: numbers(r.stale), Changed: []int{}} + doc.Overlays.Dropped = append(doc.Overlays.Dropped, r.dropped...) doc.canonical() where, err := doc.categories() if err != nil { @@ -286,11 +289,14 @@ func parseReport(raw string, l line) (reportDoc, error) { return doc, fmt.Errorf("the report is format %d, which this backport-batch does not read", doc.Format) } if !reportIDRE.MatchString(doc.ID) || reportID(doc) != doc.ID { - return doc, fmt.Errorf("report %.12s does not match what it says", doc.ID) + return doc, fmt.Errorf("report %.12s does not match what it says", terminalSafe(doc.ID)) } if err := doc.validate(l); err != nil { return doc, fmt.Errorf("report %.12s %w", doc.ID, err) } + if err := doc.bindDrops(); err != nil { + return doc, fmt.Errorf("report %.12s %w", doc.ID, err) + } return doc, nil } @@ -317,6 +323,16 @@ func (d *reportDoc) validate(l line) error { if err := kept.validate(l); err != nil { return err } + for n, lines := range d.Diag { + if k, err := strconv.Atoi(n); err != nil || k <= 0 || strconv.Itoa(k) != n { + return fmt.Errorf("has diagnostics for %q, which is no original", n) + } + for _, ln := range lines { + if ln != oneLineText(ln) { + return fmt.Errorf("has a diagnostic for #%s that is not one clean line: %q", n, ln) + } + } + } for _, list := range [][]int{d.Overrides.Include, d.Overrides.Skip, d.Overlays.Stale, d.Overlays.Changed} { for _, n := range list { if n <= 0 { @@ -529,8 +545,10 @@ func runURL(lookup func(string) (string, bool)) string { return url } -// report prints a report a batch pull request keeps, reading GitHub alone and -// writing nothing, so it works on merged and closed batches too. +// report prints a report a batch pull request keeps, writing nothing, so it +// works on merged and closed batches too. It reads GitHub, and for a report +// that keeps drops also this clone's history at the heads the report was +// made from, which it never fetches. func (a *app) report(args []string) int { fs := a.flags("report") number := fs.Int("pr", 0, "the batch pull request") diff --git a/cmd/backport-batch/report_test.go b/cmd/backport-batch/report_test.go index 46a76669f..2f6b8ef1a 100644 --- a/cmd/backport-batch/report_test.go +++ b/cmd/backport-batch/report_test.go @@ -250,6 +250,10 @@ func TestReportPartition(t *testing.T) { gh.prs = []pullRequest{f.batchPR(90, "release-1.0", branch, stdout)} pr(14, "fix(app): fourteen, after the batch", map[string]string{"app/fourteen.txt": "14\n"}, true) stdout = mustRun(t, f, gh, "run", "--line", "release-1.0", "--defer", "13", "--reason", "fails the e2e", "--push") + // An active drop, of #5, which the batch carries, and one of a feat, + // which rule 2 keeps out first. + gh.prs = []pullRequest{f.batchPR(90, "release-1.0", branch, stdout)} + stdout = mustRun(t, f, gh, "run", "--line", "release-1.0", "--drop", "5,1", "--reason", "not for 1.0", "--push") l, _ := parseLine("release-1.0") s := bodyReportOf(t, stdout, l) @@ -257,7 +261,7 @@ func TestReportPartition(t *testing.T) { if err != nil { t.Fatal(err) } - want := map[int]string{1: "skipped.type", 4: "skipped.before-start", 5: "picked", 6: "skipped.newer", 7: "skipped.breaking", + want := map[int]string{1: "skipped.type", 4: "skipped.before-start", 5: "skipped.dropped", 6: "skipped.newer", 7: "skipped.breaking", 8: "skipped.touches", 9: "skipped.skip-label", 10: "skipped.on-line", 11: "deferred", 12: "deferred", 13: "deferred", 14: "missing", 9000: "skipped.type"} for n, category := range want { @@ -269,6 +273,12 @@ func TestReportPartition(t *testing.T) { if len(got) != 0 { t.Errorf("the report also names %v", got) } + if gh.find(90).State != "closed" || !strings.Contains(stdout, `"mode":"close"`) { + t.Errorf("dropping the batch's last pick left #90 %s; want it closed", gh.find(90).State) + } + if !slices.Equal(droppedNumbers(s.doc.Overlays.Dropped), []int{1, 5}) || len(s.doc.Primary.Picked) != 0 { + t.Errorf("the report drops %v and picks %v; want #1 and #5 dropped, nothing picked", s.doc.Overlays.Dropped, s.doc.Primary.Picked) + } if !slices.Contains(s.doc.Other, direct) { t.Errorf("the report does not count the direct push %.12s: %v", direct, s.doc.Other) } @@ -564,8 +574,10 @@ func TestReportRefusesWhatItCannotVouchFor(t *testing.T) { "a deferred commit that is none": func(d *reportDoc) { d.Primary.Deferred = []deferredEntry{{Original: 6, Commit: "-x", Files: []string{"a"}}} }, - "an original named twice": func(d *reportDoc) { d.Primary.Missing = []int{5} }, - "an original that is none": func(d *reportDoc) { d.Primary.Skipped.Type = []int{-1} }, + "an original named twice": func(d *reportDoc) { d.Primary.Missing = []int{5} }, + "an original that is none": func(d *reportDoc) { d.Primary.Skipped.Type = []int{-1} }, + "a diagnostic of two lines": func(d *reportDoc) { d.Diag = map[string][]string{"5": {"incomplete\n\x1b[2Jforged"}} }, + "a diagnostic of no original": func(d *reportDoc) { d.Diag = map[string][]string{"five": {"incomplete"}} }, // Fields that speak of one another have to agree. "a drop category with no drop": func(d *reportDoc) { d.Primary.Skipped.Dropped = []int{6} }, "a stale original it has missing": func(d *reportDoc) { @@ -676,3 +688,19 @@ func TestReportReaderTakesOnlyTrustedComments(t *testing.T) { } } } + +// A report whose ID is not one says so without repeating it as written. +func TestReportErrorsRepeatNothingRaw(t *testing.T) { + s := forged(func(*reportDoc) {}) + // Short enough that a cut to 12 characters keeps the forged line. + raw := strings.Replace(s.raw, `"id":"`+s.doc.ID+`"`, `"id":"\nforged"`, 1) + code, _, stderr := reportRun(&fakeGH{prs: []pullRequest{notifyPR("20261026", "text\n\n"+reportBlock(raw)+"\n")}}, "--pr", "90") + for _, ln := range strings.Split(stderr, "\n") { + if strings.HasPrefix(ln, "forged") { + t.Errorf("report = %d, printed the forged line %q", code, ln) + } + } + if code != 2 { + t.Errorf("report = %d, want 2:\n%s", code, stderr) + } +} diff --git a/cmd/backport-batch/rules.go b/cmd/backport-batch/rules.go index 632fd7ea8..e7aa2740a 100644 --- a/cmd/backport-batch/rules.go +++ b/cmd/backport-batch/rules.go @@ -40,6 +40,9 @@ const ( // Rule 1 for a line that started under the policy after it branched: // merged before that start. ruleBeforeStart = 101 + // Not a rule of the policy: a maintainer dropped the original from the + // line with run --drop. + ruleDropped = 102 ) const ( diff --git a/docs/release/backport-tasks/resolve-conflict.md b/docs/release/backport-tasks/resolve-conflict.md index 928bc8966..46e24cf8d 100644 --- a/docs/release/backport-tasks/resolve-conflict.md +++ b/docs/release/backport-tasks/resolve-conflict.md @@ -4,7 +4,7 @@ The batch carries a pull request merged into `main` as its own commits, each pic ### Inputs -- The open batch pull request for the line, its branch `backport-batch/-` and its head commit, as GitHub shows them: `gh pr view --json headRefName,headRefOid`. +- The open batch pull request for the line, its branch `backport-batch/-` and its head commit, as GitHub shows them: `gh pr view --json headRefName,headRefOid`. The resolution lands on that batch, so without an open batch pull request there is nothing to resolve onto: wait for the next batch, which lists the conflict again, or port the change as a pull request of its own against the line, as [fixes that exist only on a release line](../backports.md#fixes-that-exist-only-on-a-release-line) describes, and drop the original from the line. - The original's number `N` from the **Deferred conflicts** list, and its merge commit on `main`: `gh pr view N --json mergeCommit --jq .mergeCommit.oid`. It has two parents when the pull request was merged, one when it was squashed. - For a merged pull request, its own commits, oldest first, without the merges that brought `main` into its branch: `git rev-list --reverse --topo-order --no-merges ^1..^2`. - The files the original changes, `git diff --name-only ^1 `, which are the only files the resolution may touch. @@ -34,7 +34,7 @@ git add git -c core.hooksPath=/dev/null cherry-pick --continue --no-edit ``` -When the original builds on something the line does not have, a helper, a value or another pull request, the resolution would have to bring that along, which this task does not allow: abort with `git cherry-pick --abort`, go back with `git switch --detach "$base"`, and comment on the batch pull request which change the original depends on; it stays deferred until that change is on the line. A commit whose change the line already has stops the pick as empty; skip it with `git cherry-pick --skip`. +When the original builds on something the line does not have, a helper, a value or another pull request, the resolution would have to bring that along, which this task does not allow: abort with `git cherry-pick --abort`, go back with `git switch --detach "$base"`, and comment on the batch pull request which change the original depends on; it stays deferred until that change is on the line. When that change will never reach the line, drop the original there instead, with `backport-batch run --line --drop N --reason "" --push`, so later batches stop trying it. A commit whose change the line already has stops the pick as empty; skip it with `git cherry-pick --skip`. For a merged pull request, join the picks to the batch head with the merge commit: diff --git a/docs/release/backports.md b/docs/release/backports.md index 37e2e89f9..b0cefdd17 100644 --- a/docs/release/backports.md +++ b/docs/release/backports.md @@ -21,6 +21,10 @@ The batch takes a pull request from `main` into release line X.Y when all of the The batch reads the pull requests off `main`'s first-parent history, one per merge commit or squash commit that names its pull request. A commit there that names none, a direct push for instance, and a merge of more than two parents are never taken; the batch pull request counts them. +### What the batch leaves out on one line + +A pull request the rules take for every line can still be wrong for one of them, for instance because it builds on code that line does not have. A maintainer drops it from that line with `./backport-batch run --line --drop --reason --push`: the batch takes it off, if it carries it, and no later batch for that line picks it again. The drop is kept in the [line's record](#the-lines-record-and-reports), so it needs an open batch pull request to record it on; before a line's first batch pull request exists there is nothing to record it on, and the command refuses. `--undrop ` withdraws it, and the next batch tries the pull request again. A pull request the line already carries cannot be dropped. To land a hand-made port instead, open it as a pull request of its own, as [fixes that exist only on a release line](#fixes-that-exist-only-on-a-release-line) describes, and drop the original. + ### What waits for the next minor `feat`, `chore` and `docs` pull requests, and dependency bumps (scope `deps`), ship with the next minor, not in a patch release; only a maintainer's `backport/include`, below, takes one. If a class of fixes should also wait for a minor, it gets a new rule on this page; a single pull request that the rules get wrong is what the overrides below are for. @@ -32,6 +36,8 @@ A maintainer overrides the rules for one pull request with a label on the merged - `backport/include` takes it although rules 2 to 4 keep it out, a `feat` for instance, or although it merged before its line started under this policy. It still has to pick cleanly or have its conflict resolved, pass CI on the batch, and, on an older line, already be on the newer one, and a line that already carries it does not take it again. - `backport/skip` keeps it out although the rules take it, and `backport-batch check` stops asking for it. An open batch that picked it before the label came keeps it until `backport-batch run --defer` takes it out; the batch pull request says so meanwhile. +A label decides for every line. To keep a pull request out of one line only, drop it there with `backport-batch run --drop`, as [what the batch leaves out on one line](#what-the-batch-leaves-out-on-one-line) describes; `backport/include` does not lift a drop. + `backport/skip` wins when a pull request carries both. Each run reads the labels afresh, and the batch pull request lists every override and what became of it. ### Which lines the batch runs for @@ -83,7 +89,7 @@ A run by hand trusts the open batch pull request the scheduled run opened as `co `run` picks what the rules take onto `backport-batch/-` and opens a draft pull request against the line. A pull request merged into `main` with a merge commit goes onto the batch as its own commits, oldest first, each picked with `git cherry-pick -x --signoff` so it keeps its author, its message and its sign-offs and gains the sign-off of whoever runs the batch; commits that merged `main` into the pull request's branch are left out. A merge commit joins them to the batch: it carries the original's merge message from `main` and ends in a `Backported-from: ` trailer. A pull request squash-merged into `main` goes on as one pick. When any commit of an original conflicts, nothing of that original stays on the batch. The batch pull request is merged into the line with a merge commit, so all of this reaches the line as it is. Its body lists the originals with how many commits each brought and which of them differ from the commit they came from, the conflicts it deferred with the commit that failed and its files, and what it skipped under which rule, and ends with the [line's record and the run's report](#the-lines-record-and-reports); `--push` prints the pull request's URL. While that pull request is open, the next run continues it, `--continue` reading its head, rather than opening another. CI on the batch pull request runs the line's build and tests, and a maintainer merges it. Since rule 6 reads the newer line itself, an older line takes a fix in the batch after the newer line's batch with that fix has merged. -A deferred conflict waits for the next batch. To land it sooner, pick the original's commits onto the batch head with `git cherry-pick -x`, resolving each conflict, and join them to the batch head with a merge commit that names the original; an original squash-merged into `main` is one pick of its commit there, with no merge. Check that merge or that pick with `./backport-batch fold --line --original --commit `, and push it onto the batch branch, as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes. CI runs `backport-batch verify` on every push to a batch pull request, which holds every original on it to the same shape. When CI on the batch pull request fails, the original it comes from is taken out with `./backport-batch run --line --defer --reason --push` and goes to the next batch, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the rest of the batch merges without it. When it is all the batch carries, `--push` records why in the batch pull request, closes it and deletes its branch, and the next batch tries it again. +A deferred conflict waits for the next batch. To land it sooner, pick the original's commits onto the batch head with `git cherry-pick -x`, resolving each conflict, and join them to the batch head with a merge commit that names the original; an original squash-merged into `main` is one pick of its commit there, with no merge. Check that merge or that pick with `./backport-batch fold --line --original --commit `, and push it onto the batch branch, as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes. CI runs `backport-batch verify` on every push to a batch pull request, which holds every original on it to the same shape. When CI on the batch pull request fails, the original it comes from is taken out with `./backport-batch run --line --defer --reason --push` and goes to the next batch, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the rest of the batch merges without it. When it is all the batch carries, `--push` records why in the batch pull request, closes it and deletes its branch, and the next batch tries it again. `--drop` and `--undrop` change the record of the open batch pull request in the same way, and need one. Before a patch release or an rc is cut from a line, `./backport-batch check --line --cut` has to exit 0. A cut tags the line, so the check fails while a pull request the rules take is not on the line, including one that is only on the open batch pull request: merge the batch pull request first. It prints the URLs, and prints what the batch deferred as warnings, since deferring is a maintainer's call; so is a pull request labelled `backport/skip` that the open batch still carries. An original that waits on the batch for a maintainer fails the cut, also when it is all the batch has left. A cut that passes says so. Without `--cut`, as the scheduled run uses it, a change on the open batch pull request counts as carried. @@ -91,7 +97,7 @@ Before a patch release or an rc is cut from a line, `./backport-batch check --li What the batch decided about a line, the conflicts it deferred, the originals taken out with `--defer` and those waiting for a maintainer, is kept in a `backport-batch` block in the body of the line's newest batch pull request, its record. `backport-batch` reads the record whether that pull request is open, merged or closed, and writes it only while the pull request is open: a merged or closed batch pull request is never edited or commented on again. So a conflict the batch deferred is still a warning in `check` after the batch merged, until a later batch tries it again. A run that picks nothing while no batch is open writes nothing, and says so. A record `backport-batch` cannot read, or one written by someone it does not trust, stops it rather than let it fall back to an older one; how it decides is in [its README](../../cmd/backport-batch/README.md#the-lines-record-and-reports). -Each run also reports where every pull request merged into `main` since the line branched stands, and what it read to decide. The body carries the latest report under the record, and a comment the tool posts on the open batch pull request keeps each one, so an earlier classification is still there after a later run replaced the body, and after the batch merged. `./backport-batch report --pr ` prints the report the body carries, and `--id ` an earlier one; it reads GitHub only and writes nothing. A run that picks nothing while no batch is open has nowhere to keep its report: the next batch's report has the line's state. +Each run also reports where every pull request merged into `main` since the line branched stands, and what it read to decide. The body carries the latest report under the record, and a comment the tool posts on the open batch pull request keeps each one, so an earlier classification is still there after a later run replaced the body, and after the batch merged. Each drop in a report keeps its reason and the merge commit it is bound to. `./backport-batch report --pr ` prints the report the body carries, and `--id ` an earlier one; it writes nothing and never fetches, and reads GitHub and, for a report that keeps drops, the clone's history at the heads the report was made from. A run that picks nothing while no batch is open has nowhere to keep its report: the next batch's report has the line's state. ### Changes to workflows From c368c48fd8a0dbb858fdf1c1136f20448f0facd8 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Tue, 10 Nov 2026 09:03:55 +0100 Subject: [PATCH 401/518] fix(controller, platform): give unbounded pool members the root's LimitRange The tenant chart renders the tenant-range-limits LimitRange only next to the tenant-quota of a tenant that declares resourceQuotas. Since quotas became hierarchical, an unbounded sub-tenant shares its parent's pool and gets a controller-owned tenant-quota-allocated ResourceQuota, but still no LimitRange. A ResourceQuota on limits.cpu or limits.memory makes admission reject any container that omits that limit, and with no LimitRange nothing fills it in. Pods that ran before the quota was introduced fail as soon as they are recreated; the barman-cloud plugin sidecar, which sets no CPU limit, takes every backed-up postgres instance in such a namespace down on its next restart. The controller already knows which namespaces it clamps, so it copies the pool root's chart LimitRange into each unbounded member, under its own name so that Helm can still create the chart object if the tenant later declares a quota of its own, and removes the copy once the namespace leaves the pool. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- internal/controller/tenantquota/reconciler.go | 85 ++++++++++- .../controller/tenantquota/reconciler_test.go | 136 ++++++++++++++++++ .../cozystack-controller/templates/rbac.yaml | 8 +- 3 files changed, 224 insertions(+), 5 deletions(-) diff --git a/internal/controller/tenantquota/reconciler.go b/internal/controller/tenantquota/reconciler.go index dddd0fbac..6c77eff6d 100644 --- a/internal/controller/tenantquota/reconciler.go +++ b/internal/controller/tenantquota/reconciler.go @@ -52,6 +52,17 @@ const ( // whenever it is below the chart quota, without the controller fighting Flux // over the chart-owned object. allocatedQuotaName = "tenant-quota-allocated" + // chartLimitRangeName is the LimitRange the tenant chart renders next to + // chartQuotaName, giving containers default requests and limits. The chart + // renders both only for a tenant that declares resourceQuotas. + chartLimitRangeName = "tenant-range-limits" + // allocatedLimitRangeName is the controller-owned copy of the pool root's + // chart LimitRange, written into every unbounded pool member. Those members + // render no chart LimitRange, yet carry an allocated quota; without + // container defaults, any pod omitting a limit the quota bounds is rejected + // at admission. A distinct name keeps the copy from colliding with the + // chart-owned object if the member later declares its own quota. + allocatedLimitRangeName = "tenant-range-limits-allocated" // managedByLabel marks the controller-owned ResourceQuotas so they can be // listed and garbage-collected. managedByLabel = "quota.cozystack.io/managed-by" @@ -102,6 +113,9 @@ func (r *Reconciler) Reconcile(ctx context.Context, _ ctrl.Request) (ctrl.Result pools := ComputePools(tenants) desired := map[string]corev1.ResourceList{} + // limitRangeSource maps each unbounded pool member to the pool root whose + // chart LimitRange it must inherit. + limitRangeSource := map[string]string{} for _, p := range pools { if over := p.Overcommitted(); len(over) > 0 { logger.Info("tenant quota pool is overcommitted by sub-tenant carve-outs", "pool", p.Root, "overcommit", over) @@ -120,6 +134,9 @@ func (r *Reconciler) Reconcile(ctx context.Context, _ ctrl.Request) (ctrl.Result for _, ns := range p.Members { if existing[ns] { desired[ns] = buffered.EnforcedHard(ns, usedByNS) + if ns != p.Root { + limitRangeSource[ns] = p.Root + } } } } @@ -134,6 +151,22 @@ func (r *Reconciler) Reconcile(ctx context.Context, _ ctrl.Request) (ctrl.Result logger.Error(err, "failed to garbage-collect stale allocated quotas") } + written := map[string]bool{} + for ns, root := range limitRangeSource { + ok, err := r.upsertAllocatedLimitRange(ctx, ns, root) + if err != nil { + logger.Error(err, "failed to apply allocated limit range", "namespace", ns, "poolRoot", root) + } + // Keep the existing copy on a transient error rather than dropping the + // defaults the namespace's pods rely on. + if ok || err != nil { + written[ns] = true + } + } + if err := r.gcAllocatedLimitRanges(ctx, written); err != nil { + logger.Error(err, "failed to garbage-collect stale allocated limit ranges") + } + return ctrl.Result{RequeueAfter: resyncInterval}, nil } @@ -224,6 +257,50 @@ func (r *Reconciler) gcAllocatedQuotas(ctx context.Context, desired map[string]c return nil } +// upsertAllocatedLimitRange copies the pool root's chart LimitRange into an +// unbounded member namespace. It reports false, without error, when the root +// has no chart LimitRange to copy. +func (r *Reconciler) upsertAllocatedLimitRange(ctx context.Context, namespace, poolRoot string) (bool, error) { + source := &corev1.LimitRange{} + if err := r.Get(ctx, types.NamespacedName{Namespace: poolRoot, Name: chartLimitRangeName}, source); err != nil { + if apierrors.IsNotFound(err) { + return false, nil + } + return false, err + } + lr := &corev1.LimitRange{ObjectMeta: metav1.ObjectMeta{Name: allocatedLimitRangeName, Namespace: namespace}} + _, err := controllerutil.CreateOrUpdate(ctx, r.Client, lr, func() error { + if lr.Labels == nil { + lr.Labels = map[string]string{} + } + lr.Labels[managedByLabel] = managedByValue + lr.Labels["internal.cozystack.io/managed-by-cozystack"] = "" + lr.Spec = *source.Spec.DeepCopy() + return nil + }) + return err == nil, err +} + +// gcAllocatedLimitRanges deletes controller-owned LimitRanges in namespaces +// that are no longer unbounded pool members (e.g. the tenant gained its own +// quota and now renders the chart LimitRange, or was deleted). +func (r *Reconciler) gcAllocatedLimitRanges(ctx context.Context, keep map[string]bool) error { + managed := &corev1.LimitRangeList{} + if err := r.List(ctx, managed, client.MatchingLabels{managedByLabel: managedByValue}); err != nil { + return err + } + for i := range managed.Items { + m := &managed.Items[i] + if keep[m.Namespace] { + continue + } + if err := r.Delete(ctx, m); err != nil && !apierrors.IsNotFound(err) { + return err + } + } + return nil +} + func (r *Reconciler) recordOvercommit(ctx context.Context, rootNamespace string, over corev1.ResourceList) { if r.Recorder == nil { return @@ -237,8 +314,8 @@ func (r *Reconciler) recordOvercommit(ctx context.Context, rootNamespace string, } // SetupWithManager wires the controller. Every relevant change (a tenant -// HelmRelease, a namespace, or a tenant ResourceQuota whose usage moved) -// coalesces into one full-tree sweep. +// HelmRelease, a namespace, a tenant ResourceQuota whose usage moved, or a +// tenant LimitRange) coalesces into one full-tree sweep. func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error { toSweep := handler.EnqueueRequestsFromMapFunc(func(context.Context, client.Object) []reconcile.Request { return []reconcile.Request{sweepKey} @@ -249,11 +326,15 @@ func (r *Reconciler) SetupWithManager(mgr ctrl.Manager) error { tenantQuotas := predicate.NewPredicateFuncs(func(o client.Object) bool { return o.GetName() == chartQuotaName || o.GetName() == allocatedQuotaName }) + tenantLimitRanges := predicate.NewPredicateFuncs(func(o client.Object) bool { + return o.GetName() == chartLimitRangeName || o.GetName() == allocatedLimitRangeName + }) return ctrl.NewControllerManagedBy(mgr). Named("tenantquota-controller"). Watches(&helmv2.HelmRelease{}, toSweep, builder.WithPredicates(tenantReleases)). Watches(&corev1.Namespace{}, toSweep). Watches(&corev1.ResourceQuota{}, toSweep, builder.WithPredicates(tenantQuotas)). + Watches(&corev1.LimitRange{}, toSweep, builder.WithPredicates(tenantLimitRanges)). Complete(r) } diff --git a/internal/controller/tenantquota/reconciler_test.go b/internal/controller/tenantquota/reconciler_test.go index 5f19ed1b1..4624b2e36 100644 --- a/internal/controller/tenantquota/reconciler_test.go +++ b/internal/controller/tenantquota/reconciler_test.go @@ -227,3 +227,139 @@ func TestReconcile_GarbageCollect(t *testing.T) { t.Fatalf("stale allocated quota in tenant-gone should have been garbage-collected") } } + +// chartLimitRange builds the chart-rendered "tenant-range-limits" LimitRange, +// which the tenant chart emits alongside its "tenant-quota". +func chartLimitRange(namespace, defaultCPU string) *corev1.LimitRange { + return &corev1.LimitRange{ + ObjectMeta: metav1.ObjectMeta{Name: chartLimitRangeName, Namespace: namespace}, + Spec: corev1.LimitRangeSpec{Limits: []corev1.LimitRangeItem{{ + Type: corev1.LimitTypeContainer, + Default: resourceList(map[string]string{"cpu": defaultCPU, "memory": "128Mi"}), + DefaultRequest: resourceList(map[string]string{"cpu": "25m", "memory": "128Mi"}), + }}}, + } +} + +func getLimitRange(c client.Client, namespace, name string) (*corev1.LimitRange, bool) { + lr := &corev1.LimitRange{} + if err := c.Get(context.Background(), types.NamespacedName{Namespace: namespace, Name: name}, lr); err != nil { + return nil, false + } + return lr, true +} + +// TestReconcile_UnboundedMemberGetsPoolLimitRange: an unbounded child renders +// no chart quota and therefore no chart LimitRange, yet the controller clamps +// it with an allocated quota. Without container defaults, every pod that omits +// a limit the quota bounds (e.g. a sidecar without a CPU limit) is rejected at +// admission, so the controller must copy the pool root's LimitRange into it. +func TestReconcile_UnboundedMemberGetsPoolLimitRange(t *testing.T) { + r, c := newReconciler(t, + tenantHR("foo", "tenant-root"), + tenantHR("bar", "tenant-foo"), + ns("tenant-foo"), ns("tenant-foo-bar"), + chartQuota("tenant-foo", map[string]string{"limits.cpu": "10"}, nil), + chartLimitRange("tenant-foo", "250m"), + ) + if _, err := r.Reconcile(context.Background(), sweepKey); err != nil { + t.Fatalf("reconcile: %v", err) + } + lr, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName) + if !ok { + t.Fatalf("unbounded member tenant-foo-bar has an allocated quota but no LimitRange") + } + if len(lr.Spec.Limits) != 1 { + t.Fatalf("tenant-foo-bar LimitRange limits = %d, want 1", len(lr.Spec.Limits)) + } + if got := lr.Spec.Limits[0].Default[corev1.ResourceCPU]; got.String() != "250m" { + t.Fatalf("tenant-foo-bar default cpu = %q, want 250m copied from the pool root", got.String()) + } + if lr.Labels[managedByLabel] != managedByValue { + t.Fatalf("tenant-foo-bar LimitRange is not marked as controller-owned") + } + if _, ok := getLimitRange(c, "tenant-foo", allocatedLimitRangeName); ok { + t.Fatalf("pool root tenant-foo already has the chart LimitRange and must not get a second one") + } +} + +// TestReconcile_PoolLimitRangeFollowsRoot: a change to the root's chart +// LimitRange propagates to the members on the next sweep. +func TestReconcile_PoolLimitRangeFollowsRoot(t *testing.T) { + root := chartLimitRange("tenant-foo", "250m") + r, c := newReconciler(t, + tenantHR("foo", "tenant-root"), + tenantHR("bar", "tenant-foo"), + ns("tenant-foo"), ns("tenant-foo-bar"), + chartQuota("tenant-foo", map[string]string{"limits.cpu": "10"}, nil), + root, + ) + if _, err := r.Reconcile(context.Background(), sweepKey); err != nil { + t.Fatalf("reconcile: %v", err) + } + root.Spec.Limits[0].Default[corev1.ResourceCPU] = resource.MustParse("500m") + if err := c.Update(context.Background(), root); err != nil { + t.Fatalf("update root LimitRange: %v", err) + } + if _, err := r.Reconcile(context.Background(), sweepKey); err != nil { + t.Fatalf("reconcile: %v", err) + } + lr, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName) + if !ok { + t.Fatalf("tenant-foo-bar lost its LimitRange") + } + if got := lr.Spec.Limits[0].Default[corev1.ResourceCPU]; got.String() != "500m" { + t.Fatalf("tenant-foo-bar default cpu = %q, want 500m after the root changed", got.String()) + } +} + +// TestReconcile_PoolLimitRangeGarbageCollected: once the child declares its own +// quota, the chart renders its LimitRange and the controller-owned copy goes. +func TestReconcile_PoolLimitRangeGarbageCollected(t *testing.T) { + stale := &corev1.LimitRange{ + ObjectMeta: metav1.ObjectMeta{ + Name: allocatedLimitRangeName, + Namespace: "tenant-foo-bar", + Labels: map[string]string{managedByLabel: managedByValue}, + }, + } + r, c := newReconciler(t, + tenantHR("foo", "tenant-root"), + tenantHR("bar", "tenant-foo"), + ns("tenant-foo"), ns("tenant-foo-bar"), + chartQuota("tenant-foo", map[string]string{"limits.cpu": "10"}, nil), + chartLimitRange("tenant-foo", "250m"), + chartQuota("tenant-foo-bar", map[string]string{"limits.cpu": "4"}, nil), + chartLimitRange("tenant-foo-bar", "250m"), + stale, + ) + if _, err := r.Reconcile(context.Background(), sweepKey); err != nil { + t.Fatalf("reconcile: %v", err) + } + if _, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName); ok { + t.Fatalf("bounded child tenant-foo-bar renders its own LimitRange; the controller copy must be removed") + } + if _, ok := getLimitRange(c, "tenant-foo-bar", chartLimitRangeName); !ok { + t.Fatalf("the chart-owned LimitRange must never be touched") + } +} + +// TestReconcile_PoolRootWithoutLimitRange: if the root has no chart LimitRange +// there is nothing to copy; the quota is still enforced and the sweep succeeds. +func TestReconcile_PoolRootWithoutLimitRange(t *testing.T) { + r, c := newReconciler(t, + tenantHR("foo", "tenant-root"), + tenantHR("bar", "tenant-foo"), + ns("tenant-foo"), ns("tenant-foo-bar"), + chartQuota("tenant-foo", map[string]string{"cpu": "10"}, nil), + ) + if _, err := r.Reconcile(context.Background(), sweepKey); err != nil { + t.Fatalf("reconcile: %v", err) + } + if _, ok := allocatedHard(t, c, "tenant-foo-bar", "cpu"); !ok { + t.Fatalf("tenant-foo-bar must still get its allocated quota") + } + if _, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName); ok { + t.Fatalf("no LimitRange can be copied from a root that has none") + } +} diff --git a/packages/system/cozystack-controller/templates/rbac.yaml b/packages/system/cozystack-controller/templates/rbac.yaml index e40ab9080..da8f71f49 100644 --- a/packages/system/cozystack-controller/templates/rbac.yaml +++ b/packages/system/cozystack-controller/templates/rbac.yaml @@ -49,10 +49,12 @@ rules: resources: ["namespaces"] verbs: ["get", "list", "watch", "patch", "update"] # TenantQuotaReconciler manages the per-namespace "tenant-quota-allocated" -# ResourceQuota that enforces each tenant's share of its parent pool, and emits -# events on the tenant namespace when sub-tenant quotas overcommit the budget. +# ResourceQuota that enforces each tenant's share of its parent pool, copies the +# pool root's "tenant-range-limits" LimitRange into unbounded members as +# "tenant-range-limits-allocated", and emits events on the tenant namespace when +# sub-tenant quotas overcommit the budget. - apiGroups: [""] - resources: ["resourcequotas"] + resources: ["resourcequotas", "limitranges"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["events"] From 0f44cf9e25d85c43c0909a02ba218e5142328c39 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Thu, 15 Oct 2026 15:18:17 +0400 Subject: [PATCH 402/518] feat(backups): add Kubernetes backup strategy for tenant clusters A tenant Kubernetes application had no backup route in cozy-default, and none of the existing strategies fits it. Its API state is a key prefix in the etcd DataStore the whole tenant shares, so the Etcd strategy's snapshot would roll every tenant cluster back at once; its control-plane CA, service account key and Talos secrets cannot be regenerated without breaking every node and credential issued under them; and its PersistentVolumes are DataVolumes in the tenant namespace that the application's own deletion removes. The new Kubernetes strategy backs up the cluster's etcd prefix with a Job that authenticates with the cluster's own datastore certificate, and the PKI Secrets and data DataVolumes with Velero. The HelmReleases of the cluster and its node pools are recorded on the Backup rather than in Velero, because a label selector for them would also pick up the worker system disks. A backup warns when a CA would expire before the backup does. A restore rebuilds a deleted cluster under its old CA, or rolls a live one back after stopping its control plane and workers, and lets the etcd prefix and the volumes be restored separately. Signed-off-by: Andrey Kolkov Assisted-by: LLM --- .../strategy/v1alpha1/kubernetes_types.go | 95 ++ .../v1alpha1/zz_generated.deepcopy.go | 103 ++ examples/backups/kubernetes/00-helpers.sh | 94 ++ .../backups/kubernetes/01-create-cluster.sh | 102 ++ .../backups/kubernetes/02-create-backupjob.sh | 36 + .../backups/kubernetes/03-restore-in-place.sh | 44 + .../kubernetes/04-restore-after-delete.sh | 39 + examples/backups/kubernetes/README.md | 68 + examples/backups/kubernetes/cleanup.sh | 18 + examples/backups/kubernetes/run-all.sh | 29 + .../backupcontroller/backup_controller.go | 3 + .../backupcontroller/backupjob_controller.go | 3 + .../backupjob_controller_test.go | 1 + .../backupcontroller/default_objects_gate.go | 13 +- .../default_objects_gate_test.go | 24 +- .../kubernetesstrategy_controller.go | 862 ++++++++++ .../kubernetesstrategy_controller_test.go | 1483 +++++++++++++++++ .../kubernetesstrategy_restore.go | 1129 +++++++++++++ .../backupcontroller/restorejob_controller.go | 8 + .../system/backupstrategy-controller/Makefile | 15 +- ...ategy.backups.cozystack.io_kubernetes.yaml | 147 ++ .../images/kubernetes-backup/Dockerfile | 14 + .../templates/backupclass-default.yaml | 7 + .../templates/rbac.yaml | 20 +- .../strategy-kubernetes-default.yaml | 304 ++++ .../tests/rendering_test.yaml | 36 +- .../tests/restore_rbac_test.yaml | 4 +- .../backupstrategy-controller/values.yaml | 18 + 28 files changed, 4697 insertions(+), 22 deletions(-) create mode 100644 api/backups/strategy/v1alpha1/kubernetes_types.go create mode 100755 examples/backups/kubernetes/00-helpers.sh create mode 100755 examples/backups/kubernetes/01-create-cluster.sh create mode 100755 examples/backups/kubernetes/02-create-backupjob.sh create mode 100755 examples/backups/kubernetes/03-restore-in-place.sh create mode 100755 examples/backups/kubernetes/04-restore-after-delete.sh create mode 100644 examples/backups/kubernetes/README.md create mode 100755 examples/backups/kubernetes/cleanup.sh create mode 100755 examples/backups/kubernetes/run-all.sh create mode 100644 internal/backupcontroller/kubernetesstrategy_controller.go create mode 100644 internal/backupcontroller/kubernetesstrategy_controller_test.go create mode 100644 internal/backupcontroller/kubernetesstrategy_restore.go create mode 100644 packages/system/backupstrategy-controller/definitions/strategy.backups.cozystack.io_kubernetes.yaml create mode 100644 packages/system/backupstrategy-controller/images/kubernetes-backup/Dockerfile create mode 100644 packages/system/backupstrategy-controller/templates/strategy-kubernetes-default.yaml diff --git a/api/backups/strategy/v1alpha1/kubernetes_types.go b/api/backups/strategy/v1alpha1/kubernetes_types.go new file mode 100644 index 000000000..704fc5d9f --- /dev/null +++ b/api/backups/strategy/v1alpha1/kubernetes_types.go @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: Apache-2.0 +// Package v1alpha1 defines strategy.backups.cozystack.io API types. +// +// Group: strategy.backups.cozystack.io +// Version: v1alpha1 +package v1alpha1 + +import ( + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" +) + +func init() { + SchemeBuilder.Register(func(s *runtime.Scheme) error { + s.AddKnownTypes(GroupVersion, + &Kubernetes{}, + &KubernetesList{}, + ) + return nil + }) +} + +const ( + KubernetesStrategyKind = "Kubernetes" +) + +// +kubebuilder:object:root=true +// +kubebuilder:subresource:status +// +kubebuilder:resource:scope=Cluster + +// Kubernetes defines a backup strategy for a tenant Kubernetes application. +// A tenant cluster keeps its state in two places the platform can reach: the +// cluster's key prefix in the tenant's shared etcd DataStore, and host-side +// objects (control-plane PKI Secrets, the application HelmReleases, and the +// DataVolumes that back the tenant's PersistentVolumes). The strategy captures +// the first with a one-shot Job and the second with Velero. The two are taken +// one after another, so a backup is crash-consistent at best. +type Kubernetes struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec KubernetesSpec `json:"spec,omitempty"` + Status KubernetesStatus `json:"status,omitempty"` +} + +// +kubebuilder:object:root=true + +// KubernetesList contains a list of Kubernetes backup strategies. +type KubernetesList struct { + metav1.TypeMeta `json:",inline"` + metav1.ListMeta `json:"metadata,omitempty"` + Items []Kubernetes `json:"items"` +} + +// KubernetesSpec specifies the desired behavior of a Kubernetes backup strategy. +type KubernetesSpec struct { + // Template holds a PodTemplateSpec for the Job that dumps the cluster's + // key prefix from its etcd DataStore to object storage, writes it back on + // restore, and deletes the object on cleanup. The Job runs in the + // application namespace. Helm-like Go templates are supported in every + // string field and are rendered against: + // - `.Release.Name` and `.Release.Namespace` — the application's name and + // namespace; `.Release.Prefix` is `kubernetes-`. + // - `.Mode` — `"backup"`, `"restore"` or `"cleanup"`. + // - `.ObjectKey` — the object-storage key of this backup's dump. + // - `.Datastore.Prefix` — the cluster's key prefix, without slashes. + // - `.Datastore.Endpoints` — comma-separated etcd client URLs. + // - `.Datastore.CertificateSecret` — the Secret holding the cluster's + // etcd client certificate (`ca.crt`, `server.crt`, `server.key`). + // - `.Parameters` — the resolved BackupClass strategy parameters. + // + // Stored schemaless for the same CRD-size reason as redis_types.go. + // + // +kubebuilder:validation:Schemaless + // +kubebuilder:pruning:PreserveUnknownFields + Template corev1.PodTemplateSpec `json:"template"` + + // Velero holds the Velero Backup and Restore specs for the host-side + // objects and volumes. Rendered against `.Application` and `.Parameters`, + // like the Velero strategy. + // + // +kubebuilder:validation:Schemaless + // +kubebuilder:pruning:PreserveUnknownFields + Velero VeleroTemplate `json:"velero"` + + // CertificateWarningThreshold is how close to expiry a control-plane + // certificate may be before a backup records a warning. Defaults to 720h. + // +optional + CertificateWarningThreshold *metav1.Duration `json:"certificateWarningThreshold,omitempty"` +} + +type KubernetesStatus struct { + Conditions []metav1.Condition `json:"conditions,omitempty"` +} diff --git a/api/backups/strategy/v1alpha1/zz_generated.deepcopy.go b/api/backups/strategy/v1alpha1/zz_generated.deepcopy.go index 3ad54f0fe..9efd1fb9b 100644 --- a/api/backups/strategy/v1alpha1/zz_generated.deepcopy.go +++ b/api/backups/strategy/v1alpha1/zz_generated.deepcopy.go @@ -839,6 +839,109 @@ func (in *KafkaStatus) DeepCopy() *KafkaStatus { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *Kubernetes) DeepCopyInto(out *Kubernetes) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ObjectMeta.DeepCopyInto(&out.ObjectMeta) + in.Spec.DeepCopyInto(&out.Spec) + in.Status.DeepCopyInto(&out.Status) +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Kubernetes. +func (in *Kubernetes) DeepCopy() *Kubernetes { + if in == nil { + return nil + } + out := new(Kubernetes) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *Kubernetes) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *KubernetesList) DeepCopyInto(out *KubernetesList) { + *out = *in + out.TypeMeta = in.TypeMeta + in.ListMeta.DeepCopyInto(&out.ListMeta) + if in.Items != nil { + in, out := &in.Items, &out.Items + *out = make([]Kubernetes, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new KubernetesList. +func (in *KubernetesList) DeepCopy() *KubernetesList { + if in == nil { + return nil + } + out := new(KubernetesList) + in.DeepCopyInto(out) + return out +} + +// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object. +func (in *KubernetesList) DeepCopyObject() runtime.Object { + if c := in.DeepCopy(); c != nil { + return c + } + return nil +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *KubernetesSpec) DeepCopyInto(out *KubernetesSpec) { + *out = *in + in.Template.DeepCopyInto(&out.Template) + in.Velero.DeepCopyInto(&out.Velero) + if in.CertificateWarningThreshold != nil { + in, out := &in.CertificateWarningThreshold, &out.CertificateWarningThreshold + *out = new(v1.Duration) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new KubernetesSpec. +func (in *KubernetesSpec) DeepCopy() *KubernetesSpec { + if in == nil { + return nil + } + out := new(KubernetesSpec) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *KubernetesStatus) DeepCopyInto(out *KubernetesStatus) { + *out = *in + if in.Conditions != nil { + in, out := &in.Conditions, &out.Conditions + *out = make([]v1.Condition, len(*in)) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new KubernetesStatus. +func (in *KubernetesStatus) DeepCopy() *KubernetesStatus { + if in == nil { + return nil + } + out := new(KubernetesStatus) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *MariaDB) DeepCopyInto(out *MariaDB) { *out = *in diff --git a/examples/backups/kubernetes/00-helpers.sh b/examples/backups/kubernetes/00-helpers.sh new file mode 100755 index 000000000..401b5edb8 --- /dev/null +++ b/examples/backups/kubernetes/00-helpers.sh @@ -0,0 +1,94 @@ +#!/bin/bash +# Shared helpers for the tenant Kubernetes backup/restore walkthrough. +# Source this file in other scripts: source "$(dirname "$0")/00-helpers.sh" + +export RED='\033[0;31m' +export GREEN='\033[0;32m' +export YELLOW='\033[1;33m' +export BLUE='\033[0;34m' +export MAGENTA='\033[0;35m' +export CYAN='\033[0;36m' +export NC='\033[0m' +export BOLD='\033[1m' + +export NAMESPACE="${NAMESPACE:-tenant-root}" +export CLUSTER_NAME="${CLUSTER_NAME:-backup-demo}" +export POOL_NAME="${POOL_NAME:-md0}" +export WORKER_DISK_SIZE="${WORKER_DISK_SIZE:-20Gi}" +export BACKUPCLASS_NAME="${BACKUPCLASS_NAME:-cozy-default}" +export BACKUPJOB_NAME="${BACKUPJOB_NAME:-kubernetes-backup-job}" +export RESTOREJOB_INPLACE_NAME="${RESTOREJOB_INPLACE_NAME:-kubernetes-restore-inplace}" +export RESTOREJOB_RECREATE_NAME="${RESTOREJOB_RECREATE_NAME:-kubernetes-restore-recreate}" +export TENANT_API_PORT="${TENANT_API_PORT:-16443}" +export MARKER_VALUE="${MARKER_VALUE:-roundtrip-7c2e91d4}" +export RELEASE="kubernetes-${CLUSTER_NAME}" + +log_info() { echo -e "${BLUE}i${NC} $*" >&2; } +log_success() { echo -e "${GREEN}OK${NC} $*" >&2; } +log_warning() { echo -e "${YELLOW}!${NC} $*" >&2; } +log_error() { echo -e "${RED}x${NC} $*" >&2; } +log_substep() { echo -e "${CYAN} -> $*${NC}" >&2; } + +print_header() { + echo -e "\n${MAGENTA}${BOLD}== $1 ==${NC}\n" >&2 +} + +# shellcheck source-path=SCRIPTDIR source=../_lib/wait-helpers.sh +source "$(dirname "${BASH_SOURCE[0]}")/../_lib/wait-helpers.sh" + +# The admin kubeconfig points at the in-cluster Service, so the walkthrough +# reaches the tenant API through a port-forward to it and keeps the TLS server +# name the certificate is issued for. +TENANT_KUBECONFIG="$(dirname "${BASH_SOURCE[0]}")/.tenant-kubeconfig" +export TENANT_KUBECONFIG + +tenant_connect() { + kubectl -n "$NAMESPACE" get secret "${RELEASE}-admin-kubeconfig" \ + -o jsonpath='{.data.super-admin\.svc}' | base64 -d > "$TENANT_KUBECONFIG" + chmod 600 "$TENANT_KUBECONFIG" + local cluster + cluster=$(kubectl --kubeconfig "$TENANT_KUBECONFIG" config view -o jsonpath='{.clusters[0].name}') + kubectl --kubeconfig "$TENANT_KUBECONFIG" config set-cluster "$cluster" \ + --server="https://127.0.0.1:${TENANT_API_PORT}" \ + --tls-server-name="${RELEASE}.${NAMESPACE}.svc" >/dev/null + pkill -f "port-forward svc/${RELEASE} ${TENANT_API_PORT}:6443" 2>/dev/null || true + kubectl -n "$NAMESPACE" port-forward "svc/${RELEASE}" "${TENANT_API_PORT}:6443" >/dev/null 2>&1 & + for _ in $(seq 1 30); do + tkubectl get --raw /readyz >/dev/null 2>&1 && return 0 + sleep 2 + done + log_error "tenant API of ${RELEASE} is not reachable through the port-forward" + return 1 +} + +tenant_disconnect() { + pkill -f "port-forward svc/${RELEASE} ${TENANT_API_PORT}:6443" 2>/dev/null || true +} + +tkubectl() { + kubectl --kubeconfig "$TENANT_KUBECONFIG" "$@" +} + +# The CA fingerprint is what proves a restore brought back the cluster rather +# than a new one: nodes, kubeconfigs and tokens all trust this CA. +cluster_ca_fingerprint() { + kubectl -n "$NAMESPACE" get secret "${RELEASE}-ca" -o jsonpath='{.data.ca\.crt}' \ + | base64 -d | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2 +} + +# wait_tenant_marker checks both halves of the backup: the ConfigMap comes back +# with etcd, the file on the PVC with the volumes. +wait_tenant_marker() { + local timeout="${1:-900}" elapsed=0 cm data + while true; do + cm=$(tkubectl -n demo get configmap marker -o jsonpath='{.data.marker}' 2>/dev/null || true) + data=$(tkubectl -n demo exec deploy/writer -- cat /data/marker 2>/dev/null || true) + if [[ "$cm" == "$MARKER_VALUE" && "$data" == "$MARKER_VALUE" ]]; then + log_success "marker present in etcd (ConfigMap) and on the volume" + return 0 + fi + (( elapsed >= timeout )) && { log_error "marker not restored: configmap='${cm}' volume='${data}'"; return 1; } + sleep 10 + elapsed=$((elapsed + 10)) + done +} diff --git a/examples/backups/kubernetes/01-create-cluster.sh b/examples/backups/kubernetes/01-create-cluster.sh new file mode 100755 index 000000000..8577ec202 --- /dev/null +++ b/examples/backups/kubernetes/01-create-cluster.sh @@ -0,0 +1,102 @@ +#!/bin/bash +# Step 01: Create a tenant Kubernetes cluster with one worker pool and seed it +# with a marker held in etcd (a ConfigMap) and on a PersistentVolume. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/00-helpers.sh" + +print_header "Step 01: Create Kubernetes '${CLUSTER_NAME}' and seed a marker" + +kubectl apply -f - </dev/null 2>&1 \ + && [[ -n "$(tkubectl get nodes -o name)" ]] \ + && [[ -n "$(tkubectl get storageclass -o name)" ]]; then + kubectl_ok=1 + break + fi + sleep 15 +done +(( kubectl_ok )) || { log_error "worker node or StorageClass did not appear"; exit 1; } + +tkubectl apply -f - < /data/marker; sync; sleep infinity"] + volumeMounts: + - {name: data, mountPath: /data} + volumes: + - name: data + persistentVolumeClaim: {claimName: data} +EOF + +wait_tenant_marker 900 +cluster_ca_fingerprint > "$SCRIPT_DIR/.ca-fingerprint" +log_success "cluster CA $(cat "$SCRIPT_DIR/.ca-fingerprint")" +tenant_disconnect + +echo -e "\n${GREEN}${BOLD}Next:${NC} ./02-create-backupjob.sh" diff --git a/examples/backups/kubernetes/02-create-backupjob.sh b/examples/backups/kubernetes/02-create-backupjob.sh new file mode 100755 index 000000000..c290711c1 --- /dev/null +++ b/examples/backups/kubernetes/02-create-backupjob.sh @@ -0,0 +1,36 @@ +#!/bin/bash +# Step 02: Back the cluster up through the cozy-default BackupClass. The +# controller dumps the cluster's etcd prefix to the platform bucket with a +# Job, then backs up its control-plane Secrets and volumes with Velero. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/00-helpers.sh" + +print_header "Step 02: Create BackupJob '${BACKUPJOB_NAME}'" + +kubectl apply -f - </dev/null || true) +kubectl -n "$NAMESPACE" delete backupjob "$BACKUPJOB_NAME" --ignore-not-found +[[ -n "$backup_ref" ]] && kubectl -n "$NAMESPACE" delete backup "$backup_ref" --ignore-not-found +kubectl -n "$NAMESPACE" delete kubernetesnodeses.apps.cozystack.io "${CLUSTER_NAME}-${POOL_NAME}" --ignore-not-found +kubectl -n "$NAMESPACE" delete kuberneteses.apps.cozystack.io "$CLUSTER_NAME" --ignore-not-found +rm -f "$TENANT_KUBECONFIG" "$SCRIPT_DIR/.ca-fingerprint" +log_success "done" diff --git a/examples/backups/kubernetes/run-all.sh b/examples/backups/kubernetes/run-all.sh new file mode 100755 index 000000000..731448b4d --- /dev/null +++ b/examples/backups/kubernetes/run-all.sh @@ -0,0 +1,29 @@ +#!/bin/bash +# End-to-end tenant Kubernetes backup/restore walkthrough. It proves the +# restore brings back the same cluster: the CA fingerprint is unchanged, and a +# marker kept in etcd and one kept on a PersistentVolume both come back, first +# after rolling a live cluster back, then after rebuilding a deleted one. +# +# Needs the platform default backups (backupstrategy-controller with a Velero +# BSL and the system bucket), so cozy-default routes Kubernetes to +# cozy-default-kubernetes. +# +# Env knobs: see 00-helpers.sh. SKIP_RESTORE=1 stops after the backup. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/00-helpers.sh" + +kubectl wait --for=create kubernetes.strategy.backups.cozystack.io/cozy-default-kubernetes --timeout=15m >/dev/null \ + || { log_error "cozy-default-kubernetes strategy is missing: enable the platform default backups first"; exit 1; } + +steps=(01 02 03 04) +[[ "${SKIP_RESTORE:-0}" == "1" ]] && steps=(01 02) +for step in "${steps[@]}"; do + script="$(find "$SCRIPT_DIR" -maxdepth 1 -type f -name "${step}-*.sh" | sort | head -n1)" + [[ -n "$script" ]] || { echo "no script found for step $step" >&2; exit 1; } + echo "=== Running $(basename "$script") ===" + bash "$script" +done + +print_header "Walkthrough complete" diff --git a/internal/backupcontroller/backup_controller.go b/internal/backupcontroller/backup_controller.go index 1a8ec0b40..6741fbaf6 100644 --- a/internal/backupcontroller/backup_controller.go +++ b/internal/backupcontroller/backup_controller.go @@ -183,6 +183,9 @@ func (r *BackupReconciler) cleanupOnDelete(ctx context.Context, backup *backupsv // deletes the object and WAITS for that delete before the Backup is // removed, so no object is orphaned (see cleanupKafkaBackup). return r.cleanupKafkaBackup(ctx, backup) + case strategyv1alpha1.KubernetesStrategyKind: + // Owns two artifacts: the Velero backup and the etcd dump object. + return r.cleanupKubernetesBackup(ctx, backup) case strategyv1alpha1.VeleroStrategyKind: return ctrl.Result{}, r.cleanupVeleroBackup(ctx, backup) default: diff --git a/internal/backupcontroller/backupjob_controller.go b/internal/backupcontroller/backupjob_controller.go index b86278649..256a9aaf5 100644 --- a/internal/backupcontroller/backupjob_controller.go +++ b/internal/backupcontroller/backupjob_controller.go @@ -214,6 +214,8 @@ func (r *BackupJobReconciler) Reconcile(ctx context.Context, req ctrl.Request) ( return r.reconcileRedis(ctx, j, resolved) case strategyv1alpha1.KafkaStrategyKind: return r.reconcileKafka(ctx, j, resolved) + case strategyv1alpha1.KubernetesStrategyKind: + return r.reconcileKubernetes(ctx, j, resolved) default: logger.V(1).Info("BackupJob resolved StrategyRef.Kind not supported, skipping", "backupjob", j.Name, @@ -240,6 +242,7 @@ func supportedBackupStrategyKinds() []string { strategyv1alpha1.RabbitmqStrategyKind, strategyv1alpha1.RedisStrategyKind, strategyv1alpha1.KafkaStrategyKind, + strategyv1alpha1.KubernetesStrategyKind, } } diff --git a/internal/backupcontroller/backupjob_controller_test.go b/internal/backupcontroller/backupjob_controller_test.go index f802fe374..994654987 100644 --- a/internal/backupcontroller/backupjob_controller_test.go +++ b/internal/backupcontroller/backupjob_controller_test.go @@ -45,6 +45,7 @@ func TestSupportedBackupStrategyKindsMatchesDispatch(t *testing.T) { strategyv1alpha1.RabbitmqStrategyKind, strategyv1alpha1.RedisStrategyKind, strategyv1alpha1.KafkaStrategyKind, + strategyv1alpha1.KubernetesStrategyKind, } sort.Strings(got) sort.Strings(want) diff --git a/internal/backupcontroller/default_objects_gate.go b/internal/backupcontroller/default_objects_gate.go index 9aea96f31..94075651e 100644 --- a/internal/backupcontroller/default_objects_gate.go +++ b/internal/backupcontroller/default_objects_gate.go @@ -593,12 +593,13 @@ func (g *DefaultObjectsGate) inspectObjects(ctx context.Context, backupClass *ba group = *strategy.StrategyRef.APIGroup } // When the Velero BSL is disabled (velero.bslEnabled=false, surfaced - // here as an empty VeleroNamespace) the chart gates the Velero - // Strategy CRs off the SAME flag, so they are never rendered. The - // BackupClass still routes to them unconditionally, so counting them - // as missing would force a Helm upgrade every MinForceInterval - // forever against a render that can never produce them. - if g.VeleroNamespace == "" && group == strategyAPIGroup && kind == "Velero" { + // here as an empty VeleroNamespace) the chart gates the Strategy CRs + // that back up through Velero (Velero, Kubernetes) off the SAME flag, + // so they are never rendered. The BackupClass still routes to them + // unconditionally, so counting them as missing would force a Helm + // upgrade every MinForceInterval forever against a render that can + // never produce them. + if g.VeleroNamespace == "" && group == strategyAPIGroup && (kind == "Velero" || kind == "Kubernetes") { continue } key := fmt.Sprintf("%s/%s", kind, name) diff --git a/internal/backupcontroller/default_objects_gate_test.go b/internal/backupcontroller/default_objects_gate_test.go index e287c9872..a07beea7f 100644 --- a/internal/backupcontroller/default_objects_gate_test.go +++ b/internal/backupcontroller/default_objects_gate_test.go @@ -37,6 +37,7 @@ var gateGVKs = []schema.GroupVersionKind{ {Group: strategyAPIGroup, Version: "v1alpha1", Kind: "Etcd"}, {Group: strategyAPIGroup, Version: "v1alpha1", Kind: "Altinity"}, {Group: strategyAPIGroup, Version: "v1alpha1", Kind: "Velero"}, + {Group: strategyAPIGroup, Version: "v1alpha1", Kind: "Kubernetes"}, {Group: "velero.io", Version: "v1", Kind: "BackupStorageLocation"}, {Group: "helm.toolkit.fluxcd.io", Version: "v2", Kind: "HelmRelease"}, } @@ -109,6 +110,7 @@ func cozyDefaultBackupClass() *backupsv1alpha1.BackupClass { ref("Altinity", "cozy-default-altinity"), ref("Velero", "cozy-default-velero-vminstance"), ref("Velero", "cozy-default-velero-vmdisk"), + ref("Kubernetes", "cozy-default-kubernetes"), }, }, } @@ -149,6 +151,7 @@ func allDefaultObjects() []runtime.Object { strategyObject("Altinity", "cozy-default-altinity"), strategyObject("Velero", "cozy-default-velero-vminstance"), strategyObject("Velero", "cozy-default-velero-vmdisk"), + strategyObject("Kubernetes", "cozy-default-kubernetes"), bslObject(), } } @@ -288,8 +291,8 @@ func TestCheckForcesWhenObjectsMissing(t *testing.T) { t.Fatal("expected a forced Helm upgrade when default objects are missing") } // 6 strategyRefs (Velero twice under different names) + the BSL. - if len(missing) != 7 { - t.Fatalf("missing = %v, want 7 entries", missing) + if len(missing) != 8 { + t.Fatalf("missing = %v, want 8 entries", missing) } ann := forceAnnotations(t, dyn) forceAt, ok := ann["reconcile.fluxcd.io/forceAt"] @@ -319,6 +322,7 @@ func TestCheckNoopWhenAllPresent(t *testing.T) { strategyObject("Altinity", "cozy-default-altinity"), strategyObject("Velero", "cozy-default-velero-vminstance"), strategyObject("Velero", "cozy-default-velero-vmdisk"), + strategyObject("Kubernetes", "cozy-default-kubernetes"), bslObject(), ) @@ -454,8 +458,8 @@ func TestCheckThrottlesTheTwoReleasesIndependently(t *testing.T) { if !forced { t.Fatal("objects release throttled by the earlier credentials force") } - if len(missing) != 7 { - t.Fatalf("missing = %v, want the 6 strategies + the BSL", missing) + if len(missing) != 8 { + t.Fatalf("missing = %v, want the 7 strategies + the BSL", missing) } if ann := forceAnnotations(t, dyn); ann["reconcile.fluxcd.io/forceAt"] == "" { t.Errorf("own HelmRelease not forced: %v", ann) @@ -482,7 +486,7 @@ func TestForceSkipsSuspendedHelmRelease(t *testing.T) { if forced { t.Error("forced reported true for a suspended HelmRelease that ignores the annotations") } - if len(missing) != 7 { + if len(missing) != 8 { t.Fatalf("missing = %v, want the objects still reported", missing) } if ann := forceAnnotations(t, dyn); len(ann) != 0 { @@ -599,6 +603,7 @@ func TestMissingObjectsIgnoresUnmappedKinds(t *testing.T) { strategyObject("Altinity", "cozy-default-altinity"), strategyObject("Velero", "cozy-default-velero-vminstance"), strategyObject("Velero", "cozy-default-velero-vmdisk"), + strategyObject("Kubernetes", "cozy-default-kubernetes"), bslObject(), ) @@ -658,6 +663,7 @@ func TestMissingObjectsSkipsBSLWhenVeleroAPIAbsent(t *testing.T) { strategyObject("Altinity", "cozy-default-altinity"), strategyObject("Velero", "cozy-default-velero-vminstance"), strategyObject("Velero", "cozy-default-velero-vmdisk"), + strategyObject("Kubernetes", "cozy-default-kubernetes"), ) // Rebuild the RESTMapper without BackupStorageLocation: the velero.io API // is not served on this cluster. @@ -1053,8 +1059,8 @@ func TestCheckKeepsForcingForUnannotatedObjects(t *testing.T) { g.lastForce = base.Add(-time.Minute) g.now = func() time.Time { return base } - if missing, forced, err := g.Check(context.Background()); err != nil || forced || len(missing) != 7 { - t.Fatalf("throttled Check: missing = %v, forced = %v, err = %v; want 7 reported, none forced", missing, forced, err) + if missing, forced, err := g.Check(context.Background()); err != nil || forced || len(missing) != len(allDefaultObjects()) { + t.Fatalf("throttled Check: missing = %v, forced = %v, err = %v; want %d reported, none forced", missing, forced, err, len(allDefaultObjects())) } g.now = func() time.Time { return base.Add(5 * time.Minute) } @@ -1065,8 +1071,8 @@ func TestCheckKeepsForcingForUnannotatedObjects(t *testing.T) { g.now = func() time.Time { return base.Add(time.Hour) } missing, forced, err := g.Check(context.Background()) - if err != nil || !forced || len(missing) != 7 { - t.Fatalf("after a render that left them unannotated: missing = %v, forced = %v, err = %v; want 7 reported and forced again", missing, forced, err) + if err != nil || !forced || len(missing) != len(allDefaultObjects()) { + t.Fatalf("after a render that left them unannotated: missing = %v, forced = %v, err = %v; want %d reported and forced again", missing, forced, err, len(allDefaultObjects())) } if again := forceAnnotations(t, dyn)["reconcile.fluxcd.io/forceAt"]; again == first { t.Errorf("not forced again for unannotated objects: forceAt still %q", first) diff --git a/internal/backupcontroller/kubernetesstrategy_controller.go b/internal/backupcontroller/kubernetesstrategy_controller.go new file mode 100644 index 000000000..1fd47f195 --- /dev/null +++ b/internal/backupcontroller/kubernetesstrategy_controller.go @@ -0,0 +1,862 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "crypto/x509" + "encoding/json" + "encoding/pem" + "fmt" + "maps" + "regexp" + "slices" + "sort" + "strconv" + "strings" + "time" + + velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + apimeta "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/dynamic" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + + strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" + "github.com/cozystack/cozystack/internal/template" +) + +const ( + kubernetesAppKind = "Kubernetes" + kubernetesReleasePrefix = "kubernetes-" + + kubernetesLabelMode = "kubernetes.strategy.backups.cozystack.io/mode" + kubernetesModeBackup = "backup" + kubernetesModeRestore = "restore" + kubernetesModeCleanup = "cleanup" + // kubernetesModeVerify checks that the dump object exists before a restore + // changes anything. + kubernetesModeVerify = "verify" + + // kubernetesCleanupAttemptsAnnotation counts failed cleanup Jobs on the + // Backup, so a persistently failing delete (revoked credentials, a removed + // bucket) ends in a released Backup and a Warning rather than a Job storm. + kubernetesCleanupAttemptsAnnotation = "kubernetes.strategy.backups.cozystack.io/cleanup-attempts" + kubernetesCleanupMaxAttempts = 5 + + kubernetesMetaPrefix = "kubernetes.strategy.backups.cozystack.io/" + kubernetesObjectMetaKey = kubernetesMetaPrefix + "object" + kubernetesDatastorePrefixMetaKey = kubernetesMetaPrefix + "datastore-prefix" + kubernetesDatastoreEndpointsMetaKey = kubernetesMetaPrefix + "datastore-endpoints" + kubernetesHelmReleasesMetaKey = kubernetesMetaPrefix + "helmreleases" + kubernetesDataVolumesMetaKey = kubernetesMetaPrefix + "datavolumes" + kubernetesCertWarningsMetaKey = kubernetesMetaPrefix + "certificate-warnings" + kubernetesControlPlaneUIDMetaKey = kubernetesMetaPrefix + "tenantcontrolplane-uid" + kubernetesParamPrefix = kubernetesMetaPrefix + "parameter/" + + // kubernetesCondCertificates is set on the BackupJob once the control-plane + // certificates have been checked; Status=True means at least one expires + // inside the warning window. + kubernetesCondCertificates = "CertificatesExpiring" + + kubernetesPollInterval = 5 * time.Second + // Leaf certificates are rotated by Kamaji and cert-manager, so a near + // expiry is only worth a warning when it is this close. + kubernetesDefaultCertThreshold = 720 * time.Hour + // A CA must outlive every backup that may still be restored, so its + // window is the Velero backup TTL; this is the fallback when none is set. + kubernetesDefaultBackupTTL = 720 * time.Hour + + helmReleaseNameAnnotation = "meta.helm.sh/release-name" + clusterNameLabel = "cluster.x-k8s.io/cluster-name" + helmInstanceLabel = "app.kubernetes.io/instance" + kamajiNameLabel = "kamaji.clastix.io/name" +) + +var ( + tenantControlPlaneGVR = schema.GroupVersionResource{Group: "kamaji.clastix.io", Version: "v1alpha1", Resource: "tenantcontrolplanes"} + kamajiDataStoreGVR = schema.GroupVersionResource{Group: "kamaji.clastix.io", Version: "v1alpha1", Resource: "datastores"} + kamajiControlPlaneGVR = schema.GroupVersionResource{Group: "controlplane.cluster.x-k8s.io", Version: "v1alpha1", Resource: "kamajicontrolplanes"} + machineDeploymentGVR = schema.GroupVersionResource{Group: "cluster.x-k8s.io", Version: "v1beta1", Resource: "machinedeployments"} + machineGVR = schema.GroupVersionResource{Group: "cluster.x-k8s.io", Version: "v1beta1", Resource: "machines"} + deploymentGVR = schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"} +) + +// kubernetesCertificateSecrets lists, by suffix after the release name, the +// Secrets whose certificates a backup checks. Kamaji and cert-manager rotate +// the leaves; the CAs are irreplaceable, so a restore needs them valid. +var kubernetesCertificateSecrets = []string{ + "ca", + "front-proxy-ca-certificate", + "api-server-certificate", + "api-server-kubelet-client-certificate", + "front-proxy-client-certificate", + "talos-ca", + "talos-tls-cert", +} + +// kubernetesUnlabelledSecrets are issued by cert-manager without the Helm +// instance label the Velero selector relies on; the driver adds it before the +// backup so the Talos PKI is not silently left out. +var kubernetesUnlabelledSecrets = []string{"talos-ca", "talos-tls-cert"} + +func kubernetesReleaseName(appName string) string { + return kubernetesReleasePrefix + appName +} + +func validateKubernetesApplicationRef(ref corev1.TypedLocalObjectReference) error { + if ref.Kind != kubernetesAppKind { + return fmt.Errorf("kubernetes strategy supports applicationRef.kind %s, got %q", kubernetesAppKind, ref.Kind) + } + if ref.APIGroup != nil && *ref.APIGroup != "" && *ref.APIGroup != backupsv1alpha1.DefaultApplicationAPIGroup { + return fmt.Errorf("kubernetes strategy supports applicationRef.apiGroup=%q, got %q", backupsv1alpha1.DefaultApplicationAPIGroup, *ref.APIGroup) + } + return nil +} + +func kubernetesObjectKey(namespace, appName, backupName string) string { + return fmt.Sprintf("%s/%s/%s.etcd.json", namespace, appName, backupName) +} + +// kubernetesDatastore is where the tenant cluster keeps its state: a key +// prefix in the etcd DataStore shared by every cluster of the tenant. +type kubernetesDatastore struct { + // ControlPlaneUID identifies the cluster across a delete and re-create + // under the same name, which Prefix does not on a Kamaji that keys the + // schema on the namespace and name. + ControlPlaneUID string + Prefix string + Endpoints []string + CertificateSecret string + // SourcePrefix is the prefix the restored dump was taken under. It + // differs from Prefix when a deleted cluster is rebuilt: Kamaji keys a new + // TenantControlPlane's schema on its UID. + SourcePrefix string +} + +func (d kubernetesDatastore) templateContext() map[string]any { + source := d.SourcePrefix + if source == "" { + source = d.Prefix + } + return map[string]any{ + "Prefix": d.Prefix, + "SourcePrefix": source, + "Endpoints": strings.Join(d.Endpoints, ","), + "CertificateSecret": d.CertificateSecret, + } +} + +// etcdEndpointURL gives a scheme to a DataStore endpoint: Kamaji lists them as +// host:port and always talks TLS to an etcd DataStore. +func etcdEndpointURL(endpoint string) string { + if strings.Contains(endpoint, "://") { + return endpoint + } + return "https://" + endpoint +} + +// resolveKubernetesDatastore reads the cluster's key prefix and client +// certificate from its TenantControlPlane and the client URLs from the +// DataStore it points at. +func resolveKubernetesDatastore(ctx context.Context, dyn dynamic.Interface, namespace, appName string) (*kubernetesDatastore, error) { + release := kubernetesReleaseName(appName) + tcp, err := dyn.Resource(tenantControlPlaneGVR).Namespace(namespace).Get(ctx, release, metav1.GetOptions{}) + if err != nil { + return nil, err + } + if driver, _, _ := unstructured.NestedString(tcp.Object, "status", "storage", "driver"); driver != "" && driver != "etcd" { + return nil, fmt.Errorf("TenantControlPlane %s/%s uses the %q datastore driver; only etcd is supported", namespace, release, driver) + } + prefix, _, _ := unstructured.NestedString(tcp.Object, "status", "storage", "setup", "schema") + if prefix == "" { + prefix, _, _ = unstructured.NestedString(tcp.Object, "spec", "dataStoreSchema") + } + if prefix == "" { + return nil, fmt.Errorf("TenantControlPlane %s/%s reports no datastore schema yet", namespace, release) + } + dsName, _, _ := unstructured.NestedString(tcp.Object, "status", "storage", "dataStoreName") + if dsName == "" { + dsName, _, _ = unstructured.NestedString(tcp.Object, "spec", "dataStore") + } + if dsName == "" { + return nil, fmt.Errorf("TenantControlPlane %s/%s names no DataStore", namespace, release) + } + certSecret, _, _ := unstructured.NestedString(tcp.Object, "status", "storage", "certificate", "secretName") + if certSecret == "" { + certSecret = release + "-datastore-certificate" + } + ds, err := dyn.Resource(kamajiDataStoreGVR).Get(ctx, dsName, metav1.GetOptions{}) + if err != nil { + return nil, fmt.Errorf("get DataStore %s: %w", dsName, err) + } + raw, _, _ := unstructured.NestedStringSlice(ds.Object, "spec", "endpoints") + if len(raw) == 0 { + return nil, fmt.Errorf("DataStore %s lists no endpoints", dsName) + } + endpoints := make([]string, 0, len(raw)) + for _, e := range raw { + endpoints = append(endpoints, etcdEndpointURL(e)) + } + return &kubernetesDatastore{ControlPlaneUID: string(tcp.GetUID()), Prefix: prefix, Endpoints: endpoints, CertificateSecret: certSecret}, nil +} + +// kubernetesCertificateWarnings reports every certificate in the given Secrets +// that expires inside its window: caWindow for CAs, leafWindow for the rest. +// Output is sorted so a condition message is stable across reconciles. +func kubernetesCertificateWarnings(secrets []corev1.Secret, now time.Time, caWindow, leafWindow time.Duration) []string { + seen := map[string]bool{} + var out []string + for _, s := range secrets { + for key, data := range s.Data { + if !strings.HasSuffix(key, ".crt") { + continue + } + for rest := data; ; { + var block *pem.Block + block, rest = pem.Decode(rest) + if block == nil { + break + } + if block.Type != "CERTIFICATE" { + continue + } + cert, err := x509.ParseCertificate(block.Bytes) + if err != nil { + continue + } + window := leafWindow + kind := "certificate" + if cert.IsCA { + window = caWindow + kind = "CA" + } + if !cert.NotAfter.Before(now.Add(window)) { + continue + } + msg := fmt.Sprintf("%s %s/%s (CN=%s) expires %s", kind, s.Name, key, cert.Subject.CommonName, cert.NotAfter.UTC().Format(time.RFC3339)) + if !seen[msg] { + seen[msg] = true + out = append(out, msg) + } + } + } + } + sort.Strings(out) + return out +} + +// kubernetesHelmReleaseNames returns the HelmRelease of the cluster followed +// by those of its worker pools. Pools live in separate kubernetes-nodes +// releases; the MachineDeployments of the cluster name them, which also +// covers the older layout where the cluster release owned its pools. +func kubernetesHelmReleaseNames(mds []unstructured.Unstructured, release string) []string { + names := []string{release} + for _, md := range mds { + if cluster, _, _ := unstructured.NestedString(md.Object, "spec", "clusterName"); cluster != release { + continue + } + if owner := md.GetAnnotations()[helmReleaseNameAnnotation]; owner != "" && !slices.Contains(names, owner) { + names = append(names, owner) + } + } + return names +} + +// sanitizeHelmRelease keeps what is needed to recreate a HelmRelease and +// drops server-set metadata and status. +func sanitizeHelmRelease(hr *unstructured.Unstructured) map[string]any { + labels := map[string]string{} + for k, v := range hr.GetLabels() { + if !strings.HasPrefix(k, "velero.io/") { + labels[k] = v + } + } + meta := map[string]any{ + "name": hr.GetName(), + "namespace": hr.GetNamespace(), + } + if len(labels) > 0 { + meta["labels"] = labels + } + spec, _, _ := unstructured.NestedMap(hr.Object, "spec") + delete(spec, "suspend") + return map[string]any{ + "apiVersion": hr.GetAPIVersion(), + "kind": hr.GetKind(), + "metadata": meta, + "spec": spec, + } +} + +// kubernetesSchemaPattern is every datastore schema Kamaji assigns: a +// TenantControlPlane UID, or _ on an older Kamaji. The Job +// turns a schema into the prefix it runs `etcdctl del --prefix` on in the +// etcd every cluster of the tenant shares, so a value that is empty, holds a +// slash or a dot, or comes from an edited Backup must never reach it. +var kubernetesSchemaPattern = regexp.MustCompile(`^[a-z0-9_-]+$`) + +func validateKubernetesDatastore(ds kubernetesDatastore) error { + if !kubernetesSchemaPattern.MatchString(ds.Prefix) { + return fmt.Errorf("refusing datastore schema %q: want %s", ds.Prefix, kubernetesSchemaPattern) + } + if ds.SourcePrefix != "" && !kubernetesSchemaPattern.MatchString(ds.SourcePrefix) { + return fmt.Errorf("refusing source datastore schema %q: want %s", ds.SourcePrefix, kubernetesSchemaPattern) + } + return nil +} + +func renderKubernetesTemplate( + tmpl corev1.PodTemplateSpec, + appName, namespace, mode, objectKey string, + ds kubernetesDatastore, + parameters map[string]string, +) (*corev1.PodTemplateSpec, error) { + if err := validateKubernetesDatastore(ds); err != nil { + return nil, err + } + return template.Template(&tmpl, map[string]any{ + "Release": map[string]string{ + "Name": appName, + "Namespace": namespace, + "Prefix": kubernetesReleasePrefix, + }, + "Mode": mode, + "ObjectKey": objectKey, + "Datastore": ds.templateContext(), + "Parameters": parameters, + }) +} + +func kubernetesStrategyParameters(b *backupsv1alpha1.Backup) map[string]string { + out := map[string]string{} + for k, v := range b.Spec.DriverMetadata { + if p := strings.TrimPrefix(k, kubernetesParamPrefix); p != k && p != "" { + out[p] = v + } + } + return out +} + +func kubernetesBackupDatastore(b *backupsv1alpha1.Backup) kubernetesDatastore { + ds := kubernetesDatastore{ + Prefix: b.Spec.DriverMetadata[kubernetesDatastorePrefixMetaKey], + CertificateSecret: kubernetesReleaseName(b.Spec.ApplicationRef.Name) + "-datastore-certificate", + } + if e := b.Spec.DriverMetadata[kubernetesDatastoreEndpointsMetaKey]; e != "" { + ds.Endpoints = strings.Split(e, ",") + } + return ds +} + +func kubernetesBackupTTL(s *strategyv1alpha1.Kubernetes) time.Duration { + if s.Spec.Velero.Spec.TTL.Duration > 0 { + return s.Spec.Velero.Spec.TTL.Duration + } + return kubernetesDefaultBackupTTL +} + +func kubernetesCertThreshold(s *strategyv1alpha1.Kubernetes) time.Duration { + if s.Spec.CertificateWarningThreshold != nil && s.Spec.CertificateWarningThreshold.Duration > 0 { + return s.Spec.CertificateWarningThreshold.Duration + } + return kubernetesDefaultCertThreshold +} + +// --------------------------------------------------------------------------- +// BackupJob path +// --------------------------------------------------------------------------- + +// reconcileKubernetes drives a BackupJob for a tenant Kubernetes application in +// three steps, each observed on later reconciles: check the control-plane +// certificates, dump the cluster's etcd prefix with a Job, then back up the +// host-side Secrets and volumes with Velero. The Backup records what a restore +// needs to put both halves back. +func (r *BackupJobReconciler) reconcileKubernetes(ctx context.Context, j *backupsv1alpha1.BackupJob, resolved *ResolvedBackupConfig) (ctrl.Result, error) { + logger := getLogger(ctx) + logger.Debug("reconciling Kubernetes strategy", "backupjob", j.Name, "phase", j.Status.Phase) + + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseSucceeded || + j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + return ctrl.Result{}, nil + } + if err := validateKubernetesApplicationRef(j.Spec.ApplicationRef); err != nil { + return r.markBackupJobFailed(ctx, j, err.Error()) + } + + if j.Status.StartedAt == nil { + fresh := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, types.NamespacedName{Namespace: j.Namespace, Name: j.Name}, fresh); err != nil { + return ctrl.Result{}, err + } + if fresh.Status.StartedAt == nil { + base := fresh.DeepCopy() + now := metav1.Now() + fresh.Status.StartedAt = &now + fresh.Status.Phase = backupsv1alpha1.BackupJobPhaseRunning + if err := r.Status().Patch(ctx, fresh, client.MergeFrom(base)); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + } + j.Status.StartedAt = fresh.Status.StartedAt + j.Status.Phase = fresh.Status.Phase + } + + strategy := &strategyv1alpha1.Kubernetes{} + if err := r.Get(ctx, client.ObjectKey{Name: resolved.StrategyRef.Name}, strategy); err != nil { + if apierrors.IsNotFound(err) { + return r.requeueStrategyNotReady(ctx, j, resolved.StrategyRef.Name) + } + return ctrl.Result{}, err + } + + app, err := r.getApplicationUnstructured(ctx, j.Namespace, j.Spec.ApplicationRef) + if err != nil { + if apierrors.IsNotFound(err) || apimeta.IsNoMatchError(err) { + return r.markBackupJobFailed(ctx, j, fmt.Sprintf("Kubernetes application not found: %s/%s", j.Namespace, j.Spec.ApplicationRef.Name)) + } + return ctrl.Result{}, err + } + + appName := j.Spec.ApplicationRef.Name + release := kubernetesReleaseName(appName) + ds, err := resolveKubernetesDatastore(ctx, r.Interface, j.Namespace, appName) + if err != nil { + if apierrors.IsNotFound(err) { + return r.markBackupJobFailed(ctx, j, fmt.Sprintf("Kubernetes application %s/%s has no TenantControlPlane: %v", j.Namespace, appName, err)) + } + return r.markBackupJobFailed(ctx, j, err.Error()) + } + + // A restore that failed partway leaves the control plane stopped on a + // possibly partial prefix, marked with the replica count it had. Backing + // that up would turn the damage into a recovery point, so a run that has + // not dumped anything yet refuses. + if err := r.Get(ctx, client.ObjectKey{Namespace: j.Namespace, Name: jobNameForBackupJob(j)}, &batchv1.Job{}); apierrors.IsNotFound(err) { + kcp, kerr := r.Interface.Resource(kamajiControlPlaneGVR).Namespace(j.Namespace).Get(ctx, release, metav1.GetOptions{}) + if kerr != nil && !apierrors.IsNotFound(kerr) && !apimeta.IsNoMatchError(kerr) { + return ctrl.Result{}, kerr + } + if kerr == nil { + if _, stopped := kcp.GetAnnotations()[kubernetesReplicasBeforeAnnotation]; stopped { + return r.markBackupJobFailed(ctx, j, fmt.Sprintf("Kubernetes application %s/%s is stopped by an unfinished restore; finish or repair the restore before backing it up", j.Namespace, appName)) + } + } + } else if err != nil { + return ctrl.Result{}, err + } + + if apimeta.FindStatusCondition(j.Status.Conditions, kubernetesCondCertificates) == nil { + return r.checkKubernetesCertificates(ctx, j, strategy, release) + } + + objectKey := kubernetesObjectKey(j.Namespace, appName, j.Name) + rendered, err := renderKubernetesTemplate(strategy.Spec.Template, appName, j.Namespace, kubernetesModeBackup, objectKey, *ds, resolved.Parameters) + if err != nil { + return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to template Kubernetes strategy: %v", err)) + } + dump, err := ensureBackupBatchJob(ctx, r.Client, r.Scheme, j, j.Namespace, jobNameForBackupJob(j), + map[string]string{ + kubernetesLabelMode: kubernetesModeBackup, + backupsv1alpha1.OwningJobNameLabel: j.Name, + backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace, + }, rendered) + if err != nil { + return r.markBackupJobFailed(ctx, j, fmt.Sprintf("failed to ensure etcd dump Job: %v", err)) + } + switch jobConditionState(dump) { + case batchv1.JobFailed: + msg := jobFailureMessage(dump) + if msg == "" { + msg = "etcd dump Job reported Failed" + } + return r.markBackupJobFailed(ctx, j, msg) + case batchv1.JobComplete: + default: + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + } + + veleroBackups := &velerov1.BackupList{} + if err := r.List(ctx, veleroBackups, client.InNamespace(veleroNamespace), client.MatchingLabels{ + backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace, + backupsv1alpha1.OwningJobNameLabel: j.Name, + }); err != nil { + return ctrl.Result{}, err + } + if len(veleroBackups.Items) == 0 { + if err := r.labelKubernetesSecrets(ctx, j.Namespace, release); err != nil { + return ctrl.Result{}, err + } + if err := r.createKubernetesVeleroBackup(ctx, j, strategy, app, resolved); err != nil { + return r.failKubernetesBackupAfterDump(ctx, j, strategy, *ds, objectKey, resolved, fmt.Sprintf("failed to create Velero Backup: %v", err)) + } + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + } + if len(veleroBackups.Items) > 1 { + return r.markBackupJobFailed(ctx, j, "more than one Velero Backup references this BackupJob") + } + vb := &veleroBackups.Items[0] + switch vb.Status.Phase { + case velerov1.BackupPhaseCompleted: + case velerov1.BackupPhaseFailed, velerov1.BackupPhasePartiallyFailed, velerov1.BackupPhaseFailedValidation: + return r.failKubernetesBackupAfterDump(ctx, j, strategy, *ds, objectKey, resolved, formatVeleroBackupFailureMessageForBackupJob(ctx, r.Client, vb)) + default: + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + } + + if j.Status.BackupRef != nil { + return ctrl.Result{}, nil + } + // A failed create is retried rather than failing the run: the dump and + // the Velero backup already exist, and no Backup would own them. + artifact, err := r.createKubernetesBackupArtifact(ctx, j, resolved, vb, *ds, objectKey) + if err != nil { + return ctrl.Result{}, fmt.Errorf("create Backup artifact: %w", err) + } + now := metav1.Now() + j.Status.BackupRef = &corev1.LocalObjectReference{Name: artifact.Name} + j.Status.CompletedAt = &now + j.Status.Phase = backupsv1alpha1.BackupJobPhaseSucceeded + apimeta.SetStatusCondition(&j.Status.Conditions, metav1.Condition{ + Type: "Ready", + Status: metav1.ConditionTrue, + Reason: "BackupCompleted", + Message: "etcd prefix and Velero backup completed", + }) + return ctrl.Result{}, r.Status().Update(ctx, j) +} + +// failKubernetesBackupAfterDump fails a run whose etcd dump is already in the +// bucket. No Backup will own the dump, so a cleanup-mode Job owned by the +// BackupJob removes it; the dump holds every Secret of the cluster. +func (r *BackupJobReconciler) failKubernetesBackupAfterDump(ctx context.Context, j *backupsv1alpha1.BackupJob, strategy *strategyv1alpha1.Kubernetes, ds kubernetesDatastore, objectKey string, resolved *ResolvedBackupConfig, msg string) (ctrl.Result, error) { + rendered, err := renderKubernetesTemplate(strategy.Spec.Template, j.Spec.ApplicationRef.Name, j.Namespace, kubernetesModeCleanup, objectKey, ds, resolved.Parameters) + if err == nil { + _, err = ensureBackupBatchJob(ctx, r.Client, r.Scheme, j, j.Namespace, j.Name+"-discard", + map[string]string{ + kubernetesLabelMode: kubernetesModeCleanup, + backupsv1alpha1.OwningJobNameLabel: j.Name, + backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace, + }, rendered) + } + if err != nil { + msg = fmt.Sprintf("%s; the etcd dump %s could not be scheduled for deletion: %v", msg, objectKey, err) + } + return r.markBackupJobFailed(ctx, j, msg) +} + +// checkKubernetesCertificates records on the BackupJob whether any +// control-plane certificate expires inside its window. It never fails the +// run: an expiring certificate is a reason to act, not to skip the backup. +func (r *BackupJobReconciler) checkKubernetesCertificates(ctx context.Context, j *backupsv1alpha1.BackupJob, strategy *strategyv1alpha1.Kubernetes, release string) (ctrl.Result, error) { + var secrets []corev1.Secret + for _, suffix := range kubernetesCertificateSecrets { + s := corev1.Secret{} + err := r.Get(ctx, types.NamespacedName{Namespace: j.Namespace, Name: release + "-" + suffix}, &s) + if apierrors.IsNotFound(err) { + continue + } + if err != nil { + return ctrl.Result{}, err + } + secrets = append(secrets, s) + } + warnings := kubernetesCertificateWarnings(secrets, time.Now(), kubernetesBackupTTL(strategy), kubernetesCertThreshold(strategy)) + cond := metav1.Condition{ + Type: kubernetesCondCertificates, + Status: metav1.ConditionFalse, + Reason: "CertificatesValid", + Message: "no control-plane certificate expires within the warning window", + } + if len(warnings) > 0 { + cond.Status = metav1.ConditionTrue + cond.Reason = "CertificatesExpiring" + cond.Message = strings.Join(warnings, "; ") + if r.Recorder != nil { + r.Recorder.Event(j, corev1.EventTypeWarning, "CertificatesExpiring", cond.Message) + } + } + apimeta.SetStatusCondition(&j.Status.Conditions, cond) + if err := r.Status().Update(ctx, j); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil +} + +func (r *BackupJobReconciler) labelKubernetesSecrets(ctx context.Context, namespace, release string) error { + for _, suffix := range kubernetesUnlabelledSecrets { + s := &corev1.Secret{} + err := r.Get(ctx, types.NamespacedName{Namespace: namespace, Name: release + "-" + suffix}, s) + if apierrors.IsNotFound(err) { + continue + } + if err != nil { + return err + } + if s.Labels[helmInstanceLabel] == release { + continue + } + base := s.DeepCopy() + if s.Labels == nil { + s.Labels = map[string]string{} + } + s.Labels[helmInstanceLabel] = release + if err := r.Patch(ctx, s, client.MergeFrom(base)); err != nil { + return fmt.Errorf("label Secret %s/%s: %w", namespace, s.Name, err) + } + } + return nil +} + +func (r *BackupJobReconciler) createKubernetesVeleroBackup(ctx context.Context, j *backupsv1alpha1.BackupJob, strategy *strategyv1alpha1.Kubernetes, app map[string]any, resolved *ResolvedBackupConfig) error { + spec, err := template.Template(&strategy.Spec.Velero.Spec, map[string]any{ + "Application": app, + "Parameters": resolved.Parameters, + }) + if err != nil { + return err + } + // The volumes an in-place restore deletes and replaces are listed before + // Velero starts: each of them is then in the Velero backup unless it was + // deleted first, and a volume created while the backup runs, which Velero + // may have missed, is never deleted by a restore of it. + volumes, err := r.kubernetesDataVolumeNames(ctx, j.Namespace, kubernetesReleaseName(j.Spec.ApplicationRef.Name)) + if err != nil { + return err + } + vb := &velerov1.Backup{ + ObjectMeta: metav1.ObjectMeta{ + GenerateName: fmt.Sprintf("%s.%s-", j.Namespace, j.Name), + Namespace: veleroNamespace, + Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: j.Name, + backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace, + }, + Annotations: map[string]string{kubernetesDataVolumesMetaKey: strings.Join(volumes, ",")}, + }, + Spec: *spec, + } + if err := r.Create(ctx, vb); err != nil { + return err + } + if r.Recorder != nil { + r.Recorder.Event(j, corev1.EventTypeNormal, "VeleroBackupCreated", fmt.Sprintf("Created Velero Backup %s/%s", veleroNamespace, vb.Name)) + } + return nil +} + +// kubernetesDataVolumeNames lists the claims backing the cluster's +// PersistentVolumes: the DataVolumes and PVCs the CSI driver labels with the +// cluster name. A PVC is listed even without a DataVolume, so an in-place +// restore clears it too rather than leaving Velero to skip it. +func (r *BackupJobReconciler) kubernetesDataVolumeNames(ctx context.Context, ns, release string) ([]string, error) { + selector := clusterNameLabel + "=" + release + seen := map[string]bool{} + dvs, err := r.Interface.Resource(dataVolumeGVR).Namespace(ns).List(ctx, metav1.ListOptions{LabelSelector: selector}) + if err != nil { + return nil, err + } + for _, dv := range dvs.Items { + seen[dv.GetName()] = true + } + pvcs := &corev1.PersistentVolumeClaimList{} + if err := r.List(ctx, pvcs, client.InNamespace(ns), client.MatchingLabels{clusterNameLabel: release}); err != nil { + return nil, err + } + for _, pvc := range pvcs.Items { + seen[pvc.Name] = true + } + names := make([]string, 0, len(seen)) + for name := range seen { + names = append(names, name) + } + sort.Strings(names) + return names, nil +} + +// createKubernetesBackupArtifact records, besides the Velero backup and the +// etcd object, the cluster's HelmReleases and data DataVolumes. The +// HelmReleases are kept here rather than in Velero because selecting them by +// application label would also select the worker system disks, which the +// lineage webhook labels after the same application. +func (r *BackupJobReconciler) createKubernetesBackupArtifact( + ctx context.Context, + j *backupsv1alpha1.BackupJob, + resolved *ResolvedBackupConfig, + vb *velerov1.Backup, + ds kubernetesDatastore, + objectKey string, +) (*backupsv1alpha1.Backup, error) { + release := kubernetesReleaseName(j.Spec.ApplicationRef.Name) + mds, err := r.Interface.Resource(machineDeploymentGVR).Namespace(j.Namespace).List(ctx, metav1.ListOptions{}) + if err != nil && !apierrors.IsNotFound(err) && !apimeta.IsNoMatchError(err) { + return nil, err + } + var mdItems []unstructured.Unstructured + if mds != nil { + mdItems = mds.Items + } + var releases []map[string]any + for _, name := range kubernetesHelmReleaseNames(mdItems, release) { + hr, err := r.Interface.Resource(helmReleaseGVR).Namespace(j.Namespace).Get(ctx, name, metav1.GetOptions{}) + if err != nil { + if apierrors.IsNotFound(err) { + continue + } + return nil, err + } + releases = append(releases, sanitizeHelmRelease(hr)) + } + releasesJSON, err := json.Marshal(releases) + if err != nil { + return nil, err + } + + driverMD := map[string]string{ + veleroBackupNameMetadataKey: vb.Name, + veleroBackupNamespaceMetadataKey: vb.Namespace, + kubernetesObjectMetaKey: objectKey, + kubernetesDatastorePrefixMetaKey: ds.Prefix, + kubernetesDatastoreEndpointsMetaKey: strings.Join(ds.Endpoints, ","), + kubernetesHelmReleasesMetaKey: string(releasesJSON), + kubernetesDataVolumesMetaKey: vb.Annotations[kubernetesDataVolumesMetaKey], + kubernetesControlPlaneUIDMetaKey: ds.ControlPlaneUID, + } + if c := apimeta.FindStatusCondition(j.Status.Conditions, kubernetesCondCertificates); c != nil && c.Status == metav1.ConditionTrue { + driverMD[kubernetesCertWarningsMetaKey] = c.Message + } + for k, v := range resolved.Parameters { + driverMD[kubernetesParamPrefix+k] = v + } + + backup := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Name: j.Name, Namespace: j.Namespace}, + Spec: backupsv1alpha1.BackupSpec{ + ApplicationRef: j.Spec.ApplicationRef, + StrategyRef: resolved.StrategyRef, + TakenAt: metav1.Now(), + DriverMetadata: driverMD, + PlanRef: j.Spec.PlanRef, + }, + Status: backupsv1alpha1.BackupStatus{ + Phase: backupsv1alpha1.BackupPhaseReady, + Artifact: &backupsv1alpha1.BackupArtifact{URI: fmt.Sprintf("velero://%s/%s", vb.Namespace, vb.Name)}, + }, + } + if err := r.Create(ctx, backup); err != nil { + if !apierrors.IsAlreadyExists(err) { + return nil, err + } + existing := &backupsv1alpha1.Backup{} + if err := r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: backup.Name}, existing); err != nil { + return nil, err + } + return existing, nil + } + return backup, nil +} + +// --------------------------------------------------------------------------- +// Backup deletion cleanup +// --------------------------------------------------------------------------- + +// cleanupKubernetesBackup removes both halves of a deleted Backup: the Velero +// backup through a DeleteBackupRequest, and the etcd dump object through a +// cleanup-mode Job, waiting for the Job before the Backup is released. The +// escape hatches match the Redis driver's. +func (r *BackupReconciler) cleanupKubernetesBackup(ctx context.Context, backup *backupsv1alpha1.Backup) (ctrl.Result, error) { + if err := r.cleanupVeleroBackup(ctx, backup); err != nil { + return ctrl.Result{}, err + } + + jobName := backup.Name + "-cleanup" + if backup.Annotations[redisSkipArtifactCleanupAnnotation] == "true" { + existing := &batchv1.Job{} + if err := r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: jobName}, existing); err == nil { + _ = r.deleteRedisCleanupJob(ctx, existing) + } + return ctrl.Result{}, nil + } + objectKey := backup.Spec.DriverMetadata[kubernetesObjectMetaKey] + if objectKey == "" { + return ctrl.Result{}, nil + } + + strategy := &strategyv1alpha1.Kubernetes{} + if err := r.Get(ctx, client.ObjectKey{Name: backup.Spec.StrategyRef.Name}, strategy); err != nil { + if apierrors.IsNotFound(err) { + return r.releaseRedisCleanup(ctx, backup, objectKey, "strategy CR is gone"), nil + } + return ctrl.Result{}, err + } + + job := &batchv1.Job{} + err := r.Get(ctx, types.NamespacedName{Namespace: backup.Namespace, Name: jobName}, job) + switch { + case apierrors.IsNotFound(err): + terminating, nsErr := r.namespaceTerminating(ctx, backup.Namespace) + if nsErr != nil { + return ctrl.Result{}, nsErr + } + if terminating { + return r.releaseRedisCleanup(ctx, backup, objectKey, "namespace is terminating"), nil + } + if perr := ProjectBackupCredentials(ctx, r.Client, r.CredentialsConfig, backup.Namespace); perr != nil { + return r.releaseRedisCleanup(ctx, backup, objectKey, fmt.Sprintf("cannot project credentials: %v", perr)), nil + } + rendered, rerr := renderKubernetesTemplate(strategy.Spec.Template, backup.Spec.ApplicationRef.Name, backup.Namespace, + kubernetesModeCleanup, objectKey, kubernetesBackupDatastore(backup), kubernetesStrategyParameters(backup)) + if rerr != nil { + return r.releaseRedisCleanup(ctx, backup, objectKey, fmt.Sprintf("cleanup template render failed: %v", rerr)), nil + } + cleanup := buildRedisCleanupJob(backup.Namespace, jobName, rendered) + cleanup.Labels = map[string]string{kubernetesLabelMode: kubernetesModeCleanup} + delete(cleanup.Spec.Template.Labels, redisLabelMode) + maps.Copy(cleanup.Spec.Template.Labels, cleanup.Labels) + if cerr := r.Create(ctx, cleanup); cerr != nil && !apierrors.IsAlreadyExists(cerr) { + if apierrors.IsForbidden(cerr) || apierrors.IsInvalid(cerr) { + return r.releaseRedisCleanup(ctx, backup, objectKey, fmt.Sprintf("cannot create cleanup Job: %v", cerr)), nil + } + return ctrl.Result{}, cerr + } + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + case err != nil: + return ctrl.Result{}, err + } + + if !job.DeletionTimestamp.IsZero() { + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + } + switch jobConditionState(job) { + case batchv1.JobComplete: + _ = r.deleteRedisCleanupJob(ctx, job) + return ctrl.Result{}, nil + case batchv1.JobFailed: + attempts, _ := strconv.Atoi(backup.Annotations[kubernetesCleanupAttemptsAnnotation]) + attempts++ + if attempts >= kubernetesCleanupMaxAttempts { + _ = r.deleteRedisCleanupJob(ctx, job) + return r.releaseRedisCleanup(ctx, backup, objectKey, fmt.Sprintf("the cleanup Job failed %d times: %s", attempts, jobFailureMessage(job))), nil + } + base := backup.DeepCopy() + if backup.Annotations == nil { + backup.Annotations = map[string]string{} + } + backup.Annotations[kubernetesCleanupAttemptsAnnotation] = strconv.Itoa(attempts) + if err := r.Patch(ctx, backup, client.MergeFrom(base)); err != nil { + return ctrl.Result{}, err + } + _ = r.deleteRedisCleanupJob(ctx, job) + return ctrl.Result{RequeueAfter: time.Duration(attempts) * 30 * time.Second}, nil + default: + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + } +} diff --git a/internal/backupcontroller/kubernetesstrategy_controller_test.go b/internal/backupcontroller/kubernetesstrategy_controller_test.go new file mode 100644 index 000000000..590f82ae9 --- /dev/null +++ b/internal/backupcontroller/kubernetesstrategy_controller_test.go @@ -0,0 +1,1483 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/x509" + "crypto/x509/pkix" + "encoding/json" + "encoding/pem" + "fmt" + "math/big" + "strconv" + "strings" + "testing" + "time" + + velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + dynamicfake "k8s.io/client-go/dynamic/fake" + "k8s.io/client-go/kubernetes/scheme" + clienttesting "k8s.io/client-go/testing" + "k8s.io/client-go/tools/record" + "sigs.k8s.io/controller-runtime/pkg/client" + clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake" + + strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" +) + +const ( + k8sTestNS = "tenant-a" + k8sTestApp = "c1" + k8sTestRelease = "kubernetes-c1" +) + +var kubernetesAppGVR = schema.GroupVersionResource{Group: backupsv1alpha1.DefaultApplicationAPIGroup, Version: "v1alpha1", Resource: "kubernetes"} + +func testCertPEM(t *testing.T, isCA bool, notAfter time.Time) []byte { + t.Helper() + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + if err != nil { + t.Fatal(err) + } + tmpl := &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: "test"}, + NotBefore: notAfter.Add(-time.Hour), + NotAfter: notAfter, + IsCA: isCA, + BasicConstraintsValid: true, + } + der, err := x509.CreateCertificate(rand.Reader, tmpl, tmpl, &key.PublicKey, key) + if err != nil { + t.Fatal(err) + } + return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}) +} + +func unstr(gvk schema.GroupVersionKind, ns, name string, fields map[string]any) *unstructured.Unstructured { + u := &unstructured.Unstructured{Object: map[string]any{}} + for k, v := range fields { + u.Object[k] = v + } + u.SetGroupVersionKind(gvk) + u.SetNamespace(ns) + u.SetName(name) + return u +} + +func TestKubernetesCertificateWarnings(t *testing.T) { + now := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) + secrets := []corev1.Secret{ + {ObjectMeta: metav1.ObjectMeta{Name: "ca"}, Data: map[string][]byte{ + "ca.crt": testCertPEM(t, true, now.Add(10*24*time.Hour)), + "ca.key": []byte("not a certificate"), + }}, + {ObjectMeta: metav1.ObjectMeta{Name: "leaf-soon"}, Data: map[string][]byte{ + "tls.crt": testCertPEM(t, false, now.Add(24*time.Hour)), + }}, + {ObjectMeta: metav1.ObjectMeta{Name: "leaf-later"}, Data: map[string][]byte{ + "tls.crt": testCertPEM(t, false, now.Add(10*24*time.Hour)), + }}, + } + + got := kubernetesCertificateWarnings(secrets, now, 30*24*time.Hour, 2*24*time.Hour) + if len(got) != 2 { + t.Fatalf("got %d warnings %v, want 2", len(got), got) + } + if !strings.HasPrefix(got[0], "CA ca/ca.crt") || !strings.HasPrefix(got[1], "certificate leaf-soon/tls.crt") { + t.Fatalf("unexpected warnings %v", got) + } + + if got := kubernetesCertificateWarnings(secrets, now, time.Hour, time.Hour); len(got) != 0 { + t.Fatalf("want no warnings inside a short window, got %v", got) + } +} + +func TestKubernetesHelmReleaseNames(t *testing.T) { + mdGVK := schema.GroupVersionKind{Group: "cluster.x-k8s.io", Version: "v1beta1", Kind: "MachineDeployment"} + md := func(name, cluster, owner string) unstructured.Unstructured { + u := unstr(mdGVK, k8sTestNS, name, map[string]any{"spec": map[string]any{"clusterName": cluster}}) + u.SetAnnotations(map[string]string{helmReleaseNameAnnotation: owner}) + return *u + } + got := kubernetesHelmReleaseNames([]unstructured.Unstructured{ + md("a", k8sTestRelease, "kubernetes-nodes-c1-a"), + md("b", k8sTestRelease, "kubernetes-nodes-c1-b"), + md("legacy", k8sTestRelease, k8sTestRelease), + md("other", "kubernetes-c2", "kubernetes-nodes-c2-a"), + }, k8sTestRelease) + want := []string{k8sTestRelease, "kubernetes-nodes-c1-a", "kubernetes-nodes-c1-b"} + if strings.Join(got, ",") != strings.Join(want, ",") { + t.Fatalf("got %v, want %v", got, want) + } +} + +func TestParseKubernetesRestoreOptions(t *testing.T) { + o, err := parseKubernetesRestoreOptions(nil) + if err != nil || !o.restoreEtcd() || !o.restoreVolumes() { + t.Fatalf("defaults: %+v %v", o, err) + } + o, err = parseKubernetesRestoreOptions(&runtime.RawExtension{Raw: []byte(`{"volumes":false}`)}) + if err != nil || !o.restoreEtcd() || o.restoreVolumes() { + t.Fatalf("etcd only: %+v %v", o, err) + } + if _, err := parseKubernetesRestoreOptions(&runtime.RawExtension{Raw: []byte(`{"etcd":false,"volumes":false}`)}); err == nil { + t.Fatal("want an error when both halves are disabled") + } +} + +func TestKubernetesVeleroRestoreSpecPhases(t *testing.T) { + strategy := &strategyv1alpha1.Kubernetes{} + backup := &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS}, + Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{Kind: kubernetesAppKind, Name: k8sTestApp}}, + } + secrets, err := kubernetesVeleroRestoreSpec(strategy, backup, kubernetesPhaseSecrets, k8sTestRelease) + if err != nil { + t.Fatal(err) + } + if strings.Join(secrets.IncludedResources, ",") != "secrets" || len(secrets.OrLabelSelectors) != 2 { + t.Fatalf("secrets phase: %+v", secrets) + } + volumes, err := kubernetesVeleroRestoreSpec(strategy, backup, kubernetesPhaseVolumes, k8sTestRelease) + if err != nil { + t.Fatal(err) + } + if volumes.OrLabelSelectors[0].MatchLabels[clusterNameLabel] != k8sTestRelease || volumes.RestorePVs == nil || !*volumes.RestorePVs { + t.Fatalf("volumes phase: %+v", volumes) + } + if _, err := kubernetesVeleroRestoreSpec(strategy, backup, "bogus", k8sTestRelease); err == nil { + t.Fatal("want an error for an unknown phase") + } +} + +type kubernetesTestEnv struct { + backup *BackupJobReconciler + restore *RestoreJobReconciler + client client.Client +} + +// newKubernetesTestEnv seeds a live tenant cluster: the application, its +// HelmReleases, TenantControlPlane, DataStore, KamajiControlPlane, one worker +// pool and one data DataVolume. +const k8sTestControlPlaneUID = "6b0c4f1e-2d3a-4e5f-8a9b-0c1d2e3f4a5b" + +func newKubernetesTestEnv(t *testing.T, objs ...client.Object) *kubernetesTestEnv { + t.Helper() + s := runtime.NewScheme() + _ = scheme.AddToScheme(s) + _ = backupsv1alpha1.AddToScheme(s) + _ = strategyv1alpha1.AddToScheme(s) + _ = velerov1.AddToScheme(s) + + listKinds := map[schema.GroupVersionResource]string{ + kubernetesAppGVR: "KubernetesList", + helmReleaseGVR: "HelmReleaseList", + tenantControlPlaneGVR: "TenantControlPlaneList", + kamajiDataStoreGVR: "DataStoreList", + kamajiControlPlaneGVR: "KamajiControlPlaneList", + machineDeploymentGVR: "MachineDeploymentList", + machineGVR: "MachineList", + dataVolumeGVR: "DataVolumeList", + deploymentGVR: "DeploymentList", + } + dyn := dynamicfake.NewSimpleDynamicClientWithCustomListKinds(s, listKinds) + seed := func(gvr schema.GroupVersionResource, u *unstructured.Unstructured) { + if err := dyn.Tracker().Create(gvr, u, u.GetNamespace()); err != nil { + t.Fatalf("seed %s: %v", u.GetName(), err) + } + } + appGVK := schema.GroupVersionKind{Group: backupsv1alpha1.DefaultApplicationAPIGroup, Version: "v1alpha1", Kind: kubernetesAppKind} + seed(kubernetesAppGVR, unstr(appGVK, k8sTestNS, k8sTestApp, map[string]any{"spec": map[string]any{}})) + hrGVK := schema.GroupVersionKind{Group: "helm.toolkit.fluxcd.io", Version: "v2", Kind: "HelmRelease"} + for _, name := range []string{k8sTestRelease, "kubernetes-nodes-c1-md0"} { + hr := unstr(hrGVK, k8sTestNS, name, map[string]any{"spec": map[string]any{"interval": "5m"}}) + hr.SetLabels(map[string]string{"apps.cozystack.io/application.name": name, "velero.io/backup-name": "old"}) + seed(helmReleaseGVR, hr) + } + tcp := unstr(schema.GroupVersionKind{Group: "kamaji.clastix.io", Version: "v1alpha1", Kind: "TenantControlPlane"}, k8sTestNS, k8sTestRelease, map[string]any{ + "status": map[string]any{"storage": map[string]any{ + "driver": "etcd", + "dataStoreName": "tenant-a", + "setup": map[string]any{"schema": "tenant_a_kubernetes_c1"}, + "certificate": map[string]any{"secretName": k8sTestRelease + "-datastore-certificate"}, + }}, + }) + tcp.SetUID(k8sTestControlPlaneUID) + seed(tenantControlPlaneGVR, tcp) + seed(kamajiDataStoreGVR, unstr(schema.GroupVersionKind{Group: "kamaji.clastix.io", Version: "v1alpha1", Kind: "DataStore"}, "", "tenant-a", map[string]any{ + "spec": map[string]any{"endpoints": []any{"etcd.tenant-a.svc:2379"}}, + })) + seed(kamajiControlPlaneGVR, unstr(schema.GroupVersionKind{Group: "controlplane.cluster.x-k8s.io", Version: "v1alpha1", Kind: "KamajiControlPlane"}, k8sTestNS, k8sTestRelease, map[string]any{ + "spec": map[string]any{"replicas": int64(2)}, + })) + md := unstr(schema.GroupVersionKind{Group: "cluster.x-k8s.io", Version: "v1beta1", Kind: "MachineDeployment"}, k8sTestNS, k8sTestRelease+"-md0", map[string]any{ + "spec": map[string]any{"clusterName": k8sTestRelease, "replicas": int64(3)}, + }) + md.SetAnnotations(map[string]string{helmReleaseNameAnnotation: "kubernetes-nodes-c1-md0"}) + seed(machineDeploymentGVR, md) + dv := unstr(schema.GroupVersionKind{Group: "cdi.kubevirt.io", Version: "v1beta1", Kind: "DataVolume"}, k8sTestNS, "pvc-1234", nil) + dv.SetLabels(map[string]string{clusterNameLabel: k8sTestRelease}) + seed(dataVolumeGVR, dv) + seed(deploymentGVR, unstr(schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "Deployment"}, k8sTestNS, k8sTestRelease+"-cluster-autoscaler", map[string]any{ + "spec": map[string]any{"replicas": int64(1)}, + })) + + base := []client.Object{ + &velerov1.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: veleroNamespace, Name: "tenant-a.bj1-xyz"}, + Status: velerov1.BackupStatus{Phase: velerov1.BackupPhaseCompleted}, + }, + &strategyv1alpha1.Kubernetes{ + ObjectMeta: metav1.ObjectMeta{Name: "cozy-default-kubernetes"}, + Spec: strategyv1alpha1.KubernetesSpec{ + Template: corev1.PodTemplateSpec{Spec: corev1.PodSpec{Containers: []corev1.Container{{ + Name: "etcd", + Image: "img", + Args: []string{"{{ .Mode }} {{ .ObjectKey }} {{ .Datastore.Prefix }} {{ .Datastore.Endpoints }} {{ .Datastore.CertificateSecret }} {{ .Datastore.SourcePrefix }}"}, + }}}}, + Velero: strategyv1alpha1.VeleroTemplate{Spec: velerov1.BackupSpec{ + IncludedNamespaces: []string{"{{ .Application.metadata.namespace }}"}, + TTL: metav1.Duration{Duration: 720 * time.Hour}, + }}, + }, + }, + &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: k8sTestRelease + "-talos-ca"}}, + &corev1.Secret{ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: k8sTestRelease + "-ca"}, Data: map[string][]byte{ + "ca.crt": testCertPEM(t, true, time.Now().Add(24*time.Hour)), + }}, + } + c := clientfake.NewClientBuilder().WithScheme(s). + WithObjects(append(base, objs...)...). + WithStatusSubresource(&backupsv1alpha1.BackupJob{}, &backupsv1alpha1.RestoreJob{}, &backupsv1alpha1.Backup{}). + Build() + mapper := &mockRESTMapper{mapping: &meta.RESTMapping{Resource: kubernetesAppGVR, GroupVersionKind: appGVK, Scope: meta.RESTScopeNamespace}} + return &kubernetesTestEnv{ + backup: &BackupJobReconciler{Client: c, Interface: dyn, RESTMapper: mapper, Scheme: s, Recorder: record.NewFakeRecorder(20)}, + restore: &RestoreJobReconciler{Client: c, Interface: dyn, RESTMapper: mapper, Scheme: s, Recorder: record.NewFakeRecorder(20)}, + client: c, + } +} + +func k8sResolved() *ResolvedBackupConfig { + return &ResolvedBackupConfig{ + StrategyRef: corev1.TypedLocalObjectReference{APIGroup: new("strategy.backups.cozystack.io"), Kind: strategyv1alpha1.KubernetesStrategyKind, Name: "cozy-default-kubernetes"}, + Parameters: map[string]string{"p": "v"}, + } +} + +func TestReconcileKubernetesBackup(t *testing.T) { + ctx := context.Background() + bj := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "bj1"}, + Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: corev1.TypedLocalObjectReference{ + APIGroup: new(backupsv1alpha1.DefaultApplicationAPIGroup), Kind: kubernetesAppKind, Name: k8sTestApp, + }}, + } + env := newKubernetesTestEnv(t, bj) + get := func() *backupsv1alpha1.BackupJob { + out := &backupsv1alpha1.BackupJob{} + if err := env.client.Get(ctx, client.ObjectKeyFromObject(bj), out); err != nil { + t.Fatal(err) + } + return out + } + step := func() { + t.Helper() + if _, err := env.backup.reconcileKubernetes(ctx, get(), k8sResolved()); err != nil { + t.Fatal(err) + } + } + + step() // StartedAt + step() // certificates + cond := meta.FindStatusCondition(get().Status.Conditions, kubernetesCondCertificates) + if cond == nil || cond.Status != metav1.ConditionTrue || !strings.Contains(cond.Message, k8sTestRelease+"-ca/ca.crt") { + t.Fatalf("certificate condition: %+v", cond) + } + + step() // dump Job + job := &batchv1.Job{} + if err := env.client.Get(ctx, client.ObjectKey{Namespace: k8sTestNS, Name: "bj1-backup"}, job); err != nil { + t.Fatalf("dump Job: %v", err) + } + args := job.Spec.Template.Spec.Containers[0].Args[0] + want := "backup tenant-a/c1/bj1.etcd.json tenant_a_kubernetes_c1 https://etcd.tenant-a.svc:2379 kubernetes-c1-datastore-certificate tenant_a_kubernetes_c1" + if args != want { + t.Fatalf("rendered args %q, want %q", args, want) + } + step() + vbs := &velerov1.BackupList{} + _ = env.client.List(ctx, vbs, client.MatchingLabels{backupsv1alpha1.OwningJobNameLabel: "bj1"}) + if len(vbs.Items) != 0 { + t.Fatal("Velero Backup created before the dump Job completed") + } + + job.Status.Conditions = []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}} + if err := env.client.Status().Update(ctx, job); err != nil { + t.Fatal(err) + } + // A claim the CSI driver created without a DataVolume is still one an + // in-place restore has to clear before Velero writes it back. + if err := env.client.Create(ctx, &corev1.PersistentVolumeClaim{ObjectMeta: metav1.ObjectMeta{ + Namespace: k8sTestNS, Name: "pvc-plain", Labels: map[string]string{clusterNameLabel: k8sTestRelease}, + }}); err != nil { + t.Fatal(err) + } + step() + if err := env.client.List(ctx, vbs, client.MatchingLabels{backupsv1alpha1.OwningJobNameLabel: "bj1"}); err != nil || len(vbs.Items) != 1 { + t.Fatalf("Velero Backups: %d %v", len(vbs.Items), err) + } + // A volume created while Velero runs may be missing from its backup, so a + // restore of this backup must never delete it. + late := unstr(schema.GroupVersionKind{Group: "cdi.kubevirt.io", Version: "v1beta1", Kind: "DataVolume"}, k8sTestNS, "pvc-late", nil) + late.SetLabels(map[string]string{clusterNameLabel: k8sTestRelease}) + if _, err := env.backup.Interface.Resource(dataVolumeGVR).Namespace(k8sTestNS).Create(ctx, late, metav1.CreateOptions{}); err != nil { + t.Fatal(err) + } + if ns := vbs.Items[0].Spec.IncludedNamespaces; len(ns) != 1 || ns[0] != k8sTestNS { + t.Fatalf("Velero spec not rendered: %v", ns) + } + talosCA := &corev1.Secret{} + _ = env.client.Get(ctx, client.ObjectKey{Namespace: k8sTestNS, Name: k8sTestRelease + "-talos-ca"}, talosCA) + if talosCA.Labels[helmInstanceLabel] != k8sTestRelease { + t.Fatalf("talos CA not labelled for the Velero selector: %v", talosCA.Labels) + } + + vb := vbs.Items[0] + vb.Status.Phase = velerov1.BackupPhaseCompleted + if err := env.client.Update(ctx, &vb); err != nil { + t.Fatal(err) + } + // A pool that is suspended while the backup runs must not come back + // suspended when the cluster is rebuilt from it. + if _, err := env.backup.Interface.Resource(helmReleaseGVR).Namespace(k8sTestNS).Patch(ctx, "kubernetes-nodes-c1-md0", types.MergePatchType, []byte(`{"spec":{"suspend":true}}`), metav1.PatchOptions{}); err != nil { + t.Fatal(err) + } + step() + if got := get(); got.Status.Phase != backupsv1alpha1.BackupJobPhaseSucceeded || got.Status.BackupRef == nil { + t.Fatalf("BackupJob not succeeded: %+v", got.Status) + } + backup := &backupsv1alpha1.Backup{} + if err := env.client.Get(ctx, client.ObjectKey{Namespace: k8sTestNS, Name: "bj1"}, backup); err != nil { + t.Fatal(err) + } + md := backup.Spec.DriverMetadata + if md[veleroBackupNameMetadataKey] != vb.Name || md[kubernetesDataVolumesMetaKey] != "pvc-1234,pvc-plain" || + md[kubernetesDatastorePrefixMetaKey] != "tenant_a_kubernetes_c1" || md[kubernetesParamPrefix+"p"] != "v" || + md[kubernetesCertWarningsMetaKey] == "" || md[kubernetesControlPlaneUIDMetaKey] != k8sTestControlPlaneUID { + t.Fatalf("driver metadata: %v", md) + } + releases, err := kubernetesBackupHelmReleases(backup) + if err != nil || len(releases) != 2 { + t.Fatalf("recorded HelmReleases: %v", releases) + } + raw, _ := json.Marshal(releases) + if strings.Contains(string(raw), "velero.io/") || strings.Contains(string(raw), "suspend") { + t.Fatalf("Velero labels or the suspend flag leaked into the recorded HelmReleases: %s", raw) + } +} + +func k8sBackupForRestore() *backupsv1alpha1.Backup { + return &backupsv1alpha1.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "b1"}, + Spec: backupsv1alpha1.BackupSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: new(backupsv1alpha1.DefaultApplicationAPIGroup), Kind: kubernetesAppKind, Name: k8sTestApp}, + StrategyRef: k8sResolved().StrategyRef, + DriverMetadata: map[string]string{ + veleroBackupNameMetadataKey: "tenant-a.bj1-xyz", + kubernetesObjectMetaKey: "tenant-a/c1/bj1.etcd.json", + kubernetesDatastorePrefixMetaKey: "tenant_a_kubernetes_c1", + kubernetesControlPlaneUIDMetaKey: k8sTestControlPlaneUID, + kubernetesDatastoreEndpointsMetaKey: "https://etcd.tenant-a.svc:2379", + kubernetesDataVolumesMetaKey: "pvc-1234", + kubernetesHelmReleasesMetaKey: `[{"apiVersion":"helm.toolkit.fluxcd.io/v2","kind":"HelmRelease","metadata":{"name":"kubernetes-c1"},"spec":{"interval":"5m"}}]`, + }, + }, + } +} + +// k8sRestoreRun drives one RestoreJob through the test environment. +type k8sRestoreRun struct { + t *testing.T + env *kubernetesTestEnv + rj *backupsv1alpha1.RestoreJob + backup *backupsv1alpha1.Backup +} + +func newK8sRestoreRun(t *testing.T, options string, backup *backupsv1alpha1.Backup) *k8sRestoreRun { + t.Helper() + rj := &backupsv1alpha1.RestoreJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "rj"}, + Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: backup.Name}}, + } + if options != "" { + rj.Spec.Options = &runtime.RawExtension{Raw: []byte(options)} + } + return &k8sRestoreRun{t: t, env: newKubernetesTestEnv(t, backup, rj), rj: rj, backup: backup} +} + +func (r *k8sRestoreRun) get() *backupsv1alpha1.RestoreJob { + r.t.Helper() + out := &backupsv1alpha1.RestoreJob{} + if err := r.env.client.Get(context.Background(), client.ObjectKeyFromObject(r.rj), out); err != nil { + r.t.Fatal(err) + } + return out +} + +func (r *k8sRestoreRun) step() { + r.t.Helper() + if _, err := r.env.restore.reconcileKubernetesRestore(context.Background(), r.get(), r.backup); err != nil { + r.t.Fatal(err) + } +} + +func (r *k8sRestoreRun) finishJob(name string, condType batchv1.JobConditionType) { + r.t.Helper() + job := &batchv1.Job{} + if err := r.env.client.Get(context.Background(), client.ObjectKey{Namespace: k8sTestNS, Name: name}, job); err != nil { + r.t.Fatalf("Job %s: %v", name, err) + } + job.Status.Conditions = []batchv1.JobCondition{{Type: condType, Status: corev1.ConditionTrue, Message: "boom"}} + if err := r.env.client.Status().Update(context.Background(), job); err != nil { + r.t.Fatal(err) + } +} + +func (r *k8sRestoreRun) finishVeleroRestore(phase velerov1.RestorePhase) { + r.t.Helper() + list := &velerov1.RestoreList{} + if err := r.env.client.List(context.Background(), list); err != nil || len(list.Items) == 0 { + r.t.Fatalf("Velero Restores: %v %v", len(list.Items), err) + } + vr := list.Items[len(list.Items)-1] + vr.Status.Phase = phase + if err := r.env.client.Update(context.Background(), &vr); err != nil { + r.t.Fatal(err) + } +} + +func (r *k8sRestoreRun) obj(gvr schema.GroupVersionResource, name string) *unstructured.Unstructured { + r.t.Helper() + u, err := r.env.restore.Interface.Resource(gvr).Namespace(k8sTestNS).Get(context.Background(), name, metav1.GetOptions{}) + if err != nil { + r.t.Fatalf("get %s %s: %v", gvr.Resource, name, err) + } + return u +} + +func (r *k8sRestoreRun) replicas(gvr schema.GroupVersionResource, name string) (int64, bool) { + n, found, _ := unstructured.NestedInt64(r.obj(gvr, name).Object, "spec", "replicas") + return n, found +} + +func (r *k8sRestoreRun) suspended(name string) bool { + s, _, _ := unstructured.NestedBool(r.obj(helmReleaseGVR, name).Object, "spec", "suspend") + return s +} + +func (r *k8sRestoreRun) addMachine(name string) { + r.t.Helper() + m := unstr(schema.GroupVersionKind{Group: "cluster.x-k8s.io", Version: "v1beta1", Kind: "Machine"}, k8sTestNS, name, nil) + m.SetLabels(map[string]string{clusterNameLabel: k8sTestRelease}) + if _, err := r.env.restore.Interface.Resource(machineGVR).Namespace(k8sTestNS).Create(context.Background(), m, metav1.CreateOptions{}); err != nil { + r.t.Fatal(err) + } +} + +// verifyAndCapture runs the preflight and the mode decision. +func (r *k8sRestoreRun) verifyAndCapture(etcd bool) { + r.t.Helper() + r.step() // StartedAt + r.step() // preflight + if etcd { + r.finishJob("rj-verify", batchv1.JobComplete) + r.step() + } + if !meta.IsStatusConditionTrue(r.get().Status.Conditions, kubernetesCondVerified) { + r.t.Fatalf("backup not verified: %+v", r.get().Status) + } + r.step() // capture +} + +func TestReconcileKubernetesRestoreInPlaceDefault(t *testing.T) { + run := newK8sRestoreRun(t, "", k8sBackupForRestore()) + run.addMachine("w-1") + run.verifyAndCapture(true) + + run.step() // stop: a worker is still there + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 2 { + t.Fatal("control plane scaled down while a worker Machine still needs it to delete its Node") + } + if n, _ := run.replicas(machineDeploymentGVR, k8sTestRelease+"-md0"); n != 0 { + t.Fatalf("worker pool not scaled down: %d", n) + } + if n, _ := run.replicas(deploymentGVR, k8sTestRelease+"-cluster-autoscaler"); n != 0 { + t.Fatalf("autoscaler not scaled down: %d", n) + } + if _, ok := run.obj(machineGVR, "w-1").GetAnnotations()[machineExcludeDrainAnnotation]; !ok { + t.Fatal("Machine not released from draining") + } + if c := meta.FindStatusCondition(run.get().Status.Conditions, kubernetesCondStopped); c == nil || c.Status != metav1.ConditionFalse || !strings.Contains(c.Message, "Machine") { + t.Fatalf("stop progress not reported: %+v", c) + } + if _, err := run.env.restore.Interface.Resource(dataVolumeGVR).Namespace(k8sTestNS).Get(context.Background(), "pvc-1234", metav1.GetOptions{}); err != nil { + t.Fatal("data DataVolume deleted while a worker may still mount it") + } + + if err := run.env.restore.Interface.Resource(machineGVR).Namespace(k8sTestNS).Delete(context.Background(), "w-1", metav1.DeleteOptions{}); err != nil { + t.Fatal(err) + } + run.step() // stop settles + if _, err := run.env.restore.Interface.Resource(dataVolumeGVR).Namespace(k8sTestNS).Get(context.Background(), "pvc-1234", metav1.GetOptions{}); err == nil { + t.Fatal("backed-up data DataVolume not deleted once the workers were gone") + } + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 0 { + t.Fatal("control plane not scaled down after the workers were gone") + } + if !meta.IsStatusConditionTrue(run.get().Status.Conditions, kubernetesCondStopped) { + t.Fatal("cluster not reported stopped") + } + + run.step() // etcd Job + run.finishJob("rj-restore", batchv1.JobComplete) + run.step() // etcd done + run.step() // volumes Velero Restore + run.finishVeleroRestore(velerov1.RestorePhaseCompleted) + run.step() // volumes done + run.step() // resume + + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded { + t.Fatalf("RestoreJob not succeeded: %+v", got.Status) + } + for _, c := range []struct { + gvr schema.GroupVersionResource + name string + want int64 + }{ + {kamajiControlPlaneGVR, k8sTestRelease, 2}, + {machineDeploymentGVR, k8sTestRelease + "-md0", 3}, + {deploymentGVR, k8sTestRelease + "-cluster-autoscaler", 1}, + } { + if n, _ := run.replicas(c.gvr, c.name); n != c.want { + t.Errorf("%s %s replicas = %d, want %d", c.gvr.Resource, c.name, n, c.want) + } + if _, left := run.obj(c.gvr, c.name).GetAnnotations()[kubernetesReplicasBeforeAnnotation]; left { + t.Errorf("%s %s keeps the restore's replica record", c.gvr.Resource, c.name) + } + } + if run.suspended(k8sTestRelease) || run.suspended("kubernetes-nodes-c1-md0") { + t.Fatal("HelmRelease left suspended") + } +} + +func TestReconcileKubernetesRestoreInPlaceEtcdOnly(t *testing.T) { + run := newK8sRestoreRun(t, `{"volumes":false}`, k8sBackupForRestore()) + run.verifyAndCapture(true) + run.step() // stop + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 0 { + t.Fatalf("control plane not scaled down: %d", n) + } + if n, _ := run.replicas(machineDeploymentGVR, k8sTestRelease+"-md0"); n != 3 { + t.Fatalf("workers touched by an etcd-only restore: %d", n) + } + run.step() + run.finishJob("rj-restore", batchv1.JobComplete) + run.step() + run.step() + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded { + t.Fatalf("RestoreJob not succeeded: %+v", got.Status) + } + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 2 { + t.Fatalf("control plane replicas not restored: %d", n) + } + list := &velerov1.RestoreList{} + _ = run.env.client.List(context.Background(), list) + if len(list.Items) != 0 { + t.Fatal("an etcd-only restore created a Velero Restore") + } +} + +func TestReconcileKubernetesRestoreKeepsPreSuspendedRelease(t *testing.T) { + run := newK8sRestoreRun(t, `{"volumes":false}`, k8sBackupForRestore()) + patch := []byte(`{"spec":{"suspend":true}}`) + if _, err := run.env.restore.Interface.Resource(helmReleaseGVR).Namespace(k8sTestNS).Patch(context.Background(), "kubernetes-nodes-c1-md0", types.MergePatchType, patch, metav1.PatchOptions{}); err != nil { + t.Fatal(err) + } + run.verifyAndCapture(true) + run.step() + run.step() + run.finishJob("rj-restore", batchv1.JobComplete) + run.step() + run.step() + if run.get().Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded { + t.Fatalf("RestoreJob not succeeded: %+v", run.get().Status) + } + if !run.suspended("kubernetes-nodes-c1-md0") { + t.Fatal("a HelmRelease suspended before the restore was resumed by it") + } + if run.suspended(k8sTestRelease) { + t.Fatal("the cluster HelmRelease the restore suspended was left suspended") + } +} + +func TestReconcileKubernetesRestoreEtcdFailureLeavesControlPlaneStopped(t *testing.T) { + run := newK8sRestoreRun(t, `{"volumes":false}`, k8sBackupForRestore()) + run.verifyAndCapture(true) + run.step() + run.step() + run.addJobPod("rj-restore", kubernetesJobWriting) + run.finishJob("rj-restore", batchv1.JobFailed) + run.step() + got := run.get() + if got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed || !meta.IsStatusConditionTrue(got.Status.Conditions, kubernetesCondEtcdWriteFailed) { + t.Fatalf("want a failed restore with the etcd write marked failed: %+v", got.Status) + } + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 0 { + t.Fatal("control plane brought back up on a possibly partial etcd prefix") + } + if !strings.Contains(got.Status.Message+fmtConditions(got), ".pre-restore") { + t.Fatalf("failure does not point at the pre-restore copy: %+v", got.Status) + } + + run.env.restore.cleanupKubernetesRestore(context.Background(), got) + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 0 { + t.Fatal("deleting the RestoreJob brought the control plane up on a possibly partial prefix") + } +} + +func fmtConditions(rj *backupsv1alpha1.RestoreJob) string { + var b strings.Builder + for _, c := range rj.Status.Conditions { + b.WriteString(c.Message) + } + return b.String() +} + +func TestCleanupKubernetesRestoreResumesStoppedCluster(t *testing.T) { + run := newK8sRestoreRun(t, "", k8sBackupForRestore()) + run.verifyAndCapture(true) + run.step() // stop: workers gone, volumes deleted, control plane stopped + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 0 { + t.Fatalf("control plane not stopped: %d", n) + } + run.env.restore.cleanupKubernetesRestore(context.Background(), run.get()) + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 2 { + t.Fatalf("deleting the RestoreJob left the control plane at %d replicas", n) + } + if n, _ := run.replicas(machineDeploymentGVR, k8sTestRelease+"-md0"); n != 3 { + t.Fatalf("deleting the RestoreJob left the worker pool at %d replicas", n) + } + if run.suspended(k8sTestRelease) { + t.Fatal("deleting the RestoreJob left the HelmRelease suspended") + } +} + +func TestReconcileKubernetesRestoreStopDeadline(t *testing.T) { + run := newK8sRestoreRun(t, "", k8sBackupForRestore()) + run.addMachine("w-1") + run.verifyAndCapture(true) + run.step() // stop starts, a worker remains + rj := run.get() + cond := meta.FindStatusCondition(rj.Status.Conditions, kubernetesCondStopped) + cond.LastTransitionTime = metav1.NewTime(time.Now().Add(-restorePreparationDeadline - time.Minute)) + if err := run.env.client.Status().Update(context.Background(), rj); err != nil { + t.Fatal(err) + } + run.step() + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("a stop that never settles must fail the restore: %+v", got.Status) + } + if n, _ := run.replicas(machineDeploymentGVR, k8sTestRelease+"-md0"); n != 3 { + t.Fatalf("worker pool not resumed after the stop timed out: %d", n) + } + if run.suspended(k8sTestRelease) { + t.Fatal("HelmRelease left suspended after the stop timed out") + } +} + +func TestReconcileKubernetesRestoreVerifiesBeforeChangingAnything(t *testing.T) { + backup := k8sBackupForRestore() + backup.Spec.DriverMetadata[veleroBackupNameMetadataKey] = "expired" + run := newK8sRestoreRun(t, "", backup) + run.step() + run.step() + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("a missing Velero backup must fail the restore: %+v", got.Status) + } + if run.suspended(k8sTestRelease) { + t.Fatal("cluster touched before the backup was verified") + } + if _, err := run.env.restore.Interface.Resource(dataVolumeGVR).Namespace(k8sTestNS).Get(context.Background(), "pvc-1234", metav1.GetOptions{}); err != nil { + t.Fatal("data DataVolume deleted before the backup was verified") + } + + backup = k8sBackupForRestore() + backup.Spec.DriverMetadata[veleroBackupNameMetadataKey] = "partial" + run = newK8sRestoreRun(t, "", backup) + if err := run.env.client.Create(context.Background(), &velerov1.Backup{ + ObjectMeta: metav1.ObjectMeta{Namespace: veleroNamespace, Name: "partial"}, + Status: velerov1.BackupStatus{Phase: velerov1.BackupPhasePartiallyFailed}, + }); err != nil { + t.Fatal(err) + } + run.step() + run.step() + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("a Velero backup that did not complete must fail the restore: %+v", got.Status) + } + + run = newK8sRestoreRun(t, "", k8sBackupForRestore()) + run.step() + run.step() + run.finishJob("rj-verify", batchv1.JobFailed) + run.step() + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("a missing etcd dump must fail the restore: %+v", got.Status) + } + if run.suspended(k8sTestRelease) { + t.Fatal("cluster touched before the dump was verified") + } +} + +func k8sBackupOfDeletedCluster() *backupsv1alpha1.Backup { + backup := k8sBackupForRestore() + backup.Spec.ApplicationRef.Name = "gone" + backup.Spec.DriverMetadata[kubernetesHelmReleasesMetaKey] = `[` + + `{"apiVersion":"helm.toolkit.fluxcd.io/v2","kind":"HelmRelease","metadata":{"name":"kubernetes-gone"},"spec":{"interval":"5m"}},` + + `{"apiVersion":"helm.toolkit.fluxcd.io/v2","kind":"HelmRelease","metadata":{"name":"kubernetes-nodes-gone-md0"},"spec":{"interval":"5m"}}]` + return backup +} + +// TestReconcileKubernetesRestoreRecreate rebuilds a deleted cluster whose new +// TenantControlPlane gets a fresh, UID-keyed schema: the dump must be written +// under that schema, and the worker pools come back only after it is. +func TestReconcileKubernetesRestoreRecreate(t *testing.T) { + ctx := context.Background() + run := newK8sRestoreRun(t, "", k8sBackupOfDeletedCluster()) + run.verifyAndCapture(true) + if state := kubernetesStateFromRestoreJob(run.get()); state == nil || state.Mode != kubernetesRestoreModeRecreate || state.Release != "kubernetes-gone" { + t.Fatalf("want recreate mode, got %+v", state) + } + run.step() // secrets + run.finishVeleroRestore(velerov1.RestorePhaseCompleted) + run.step() + run.step() // volumes + run.finishVeleroRestore(velerov1.RestorePhaseCompleted) + run.step() + + run.step() // cluster HelmRelease, waiting for the TenantControlPlane + dyn := run.env.restore.Interface + if _, err := dyn.Resource(helmReleaseGVR).Namespace(k8sTestNS).Get(ctx, "kubernetes-gone", metav1.GetOptions{}); err != nil { + t.Fatalf("cluster HelmRelease not recreated: %v", err) + } + if _, err := dyn.Resource(helmReleaseGVR).Namespace(k8sTestNS).Get(ctx, "kubernetes-nodes-gone-md0", metav1.GetOptions{}); err == nil { + t.Fatal("worker pool recreated before the etcd state was written back") + } + if c := meta.FindStatusCondition(run.get().Status.Conditions, kubernetesCondRecreated); c == nil || c.Status != metav1.ConditionFalse { + t.Fatalf("wait for the new control plane not reported: %+v", c) + } + run.step() // a later poll, the control plane still coming up + if ann := run.obj(helmReleaseGVR, "kubernetes-gone").GetAnnotations(); ann["reconcile.fluxcd.io/forceAt"] != "" { + t.Fatalf("the cluster HelmRelease the rebuild created is forced through a Helm upgrade on a poll: %v", ann) + } + + const newSchema = "0f2c1e4a-77b1-4c39-9a3e-6d1f2b9c8e10" + if _, err := dyn.Resource(tenantControlPlaneGVR).Namespace(k8sTestNS).Create(ctx, unstr(schema.GroupVersionKind{Group: "kamaji.clastix.io", Version: "v1alpha1", Kind: "TenantControlPlane"}, k8sTestNS, "kubernetes-gone", map[string]any{ + "status": map[string]any{"storage": map[string]any{"driver": "etcd", "dataStoreName": "tenant-a", "setup": map[string]any{"schema": newSchema}}}, + }), metav1.CreateOptions{}); err != nil { + t.Fatal(err) + } + if _, err := dyn.Resource(kamajiControlPlaneGVR).Namespace(k8sTestNS).Create(ctx, unstr(schema.GroupVersionKind{Group: "controlplane.cluster.x-k8s.io", Version: "v1alpha1", Kind: "KamajiControlPlane"}, k8sTestNS, "kubernetes-gone", map[string]any{ + "spec": map[string]any{"replicas": int64(2)}, + }), metav1.CreateOptions{}); err != nil { + t.Fatal(err) + } + run.step() // control plane is up + run.step() // stop it for the etcd write + if n, _ := run.replicas(kamajiControlPlaneGVR, "kubernetes-gone"); n != 0 { + t.Fatalf("new control plane not stopped before the etcd write: %d", n) + } + run.step() // etcd Job + job := &batchv1.Job{} + if err := run.env.client.Get(ctx, client.ObjectKey{Namespace: k8sTestNS, Name: "rj-restore"}, job); err != nil { + t.Fatal(err) + } + args := job.Spec.Template.Spec.Containers[0].Args[0] + if !strings.Contains(args, " "+newSchema+" ") || !strings.HasSuffix(args, " tenant_a_kubernetes_c1") || !strings.Contains(args, "kubernetes-gone-datastore-certificate") { + t.Fatalf("restore Job does not write the dump under the new cluster's schema: %q", args) + } + run.finishJob("rj-restore", batchv1.JobComplete) + run.step() + run.step() // resume, worker pools + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded { + t.Fatalf("rebuild not succeeded: %+v", got.Status) + } + if n, _ := run.replicas(kamajiControlPlaneGVR, "kubernetes-gone"); n != 2 { + t.Fatalf("new control plane not started again: %d", n) + } + if _, err := dyn.Resource(helmReleaseGVR).Namespace(k8sTestNS).Get(ctx, "kubernetes-nodes-gone-md0", metav1.GetOptions{}); err != nil { + t.Fatalf("worker pool not recreated: %v", err) + } +} + +func TestReconcileKubernetesRestoreRecreateSecretsFailure(t *testing.T) { + run := newK8sRestoreRun(t, "", k8sBackupOfDeletedCluster()) + run.verifyAndCapture(true) + run.step() // secrets restore + list := &velerov1.RestoreList{} + if err := run.env.client.List(context.Background(), list); err != nil || len(list.Items) != 1 { + t.Fatalf("Velero Restores: %d %v", len(list.Items), err) + } + vr := list.Items[0] + if vr.Labels[kubernetesRestorePhaseLabel] != kubernetesPhaseSecrets || vr.Spec.BackupName != "tenant-a.bj1-xyz" || vr.Spec.ResourceModifier == nil { + t.Fatalf("secrets Restore: %+v", vr) + } + cm := &corev1.ConfigMap{} + if err := run.env.client.Get(context.Background(), client.ObjectKey{Namespace: veleroNamespace, Name: vr.Spec.ResourceModifier.Name}, cm); err != nil { + t.Fatal(err) + } + if !strings.Contains(cm.Data["resource-modifier-rules.yaml"], "ownerReferences: null") { + t.Fatalf("modifiers do not strip ownerReferences: %s", cm.Data["resource-modifier-rules.yaml"]) + } + run.finishVeleroRestore(velerov1.RestorePhaseFailed) + run.step() + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("a failed Velero Restore must fail the RestoreJob: %+v", got.Status) + } +} + +func TestReconcileKubernetesRestoreRecreateRejectsBadReleaseRecord(t *testing.T) { + backup := k8sBackupForRestore() + backup.Spec.ApplicationRef.Name = "gone" + backup.Spec.DriverMetadata[kubernetesHelmReleasesMetaKey] = "not json" + run := newK8sRestoreRun(t, "", backup) + run.verifyAndCapture(true) + run.step() + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("an unreadable HelmRelease record must fail the rebuild: %+v", got.Status) + } +} + +func TestReconcileKubernetesRestoreRejectsRename(t *testing.T) { + backup := k8sBackupForRestore() + run := newK8sRestoreRun(t, "", backup) + rj := run.get() + rj.Spec.TargetApplicationRef = &corev1.TypedLocalObjectReference{Kind: kubernetesAppKind, Name: "other"} + if _, err := run.env.restore.reconcileKubernetesRestore(context.Background(), rj, backup); err != nil { + t.Fatal(err) + } + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("restore into another application must fail: %+v", got.Status) + } +} + +func TestResolveKubernetesDatastoreRejectsNonEtcd(t *testing.T) { + env := newKubernetesTestEnv(t) + dyn := env.backup.Interface + tcp, _ := dyn.Resource(tenantControlPlaneGVR).Namespace(k8sTestNS).Get(context.Background(), k8sTestRelease, metav1.GetOptions{}) + _ = unstructured.SetNestedField(tcp.Object, "PostgreSQL", "status", "storage", "driver") + if _, err := dyn.Resource(tenantControlPlaneGVR).Namespace(k8sTestNS).Update(context.Background(), tcp, metav1.UpdateOptions{}); err != nil { + t.Fatal(err) + } + if _, err := resolveKubernetesDatastore(context.Background(), dyn, k8sTestNS, k8sTestApp); err == nil { + t.Fatal("want an error for a non-etcd datastore") + } +} + +func TestCleanupKubernetesBackupGivesUpAfterRepeatedFailures(t *testing.T) { + ctx := context.Background() + backup := k8sBackupForRestore() + backup.Annotations = map[string]string{kubernetesCleanupAttemptsAnnotation: strconv.Itoa(kubernetesCleanupMaxAttempts - 1)} + failed := &batchv1.Job{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "b1-cleanup"}, + Status: batchv1.JobStatus{Conditions: []batchv1.JobCondition{{Type: batchv1.JobFailed, Status: corev1.ConditionTrue}}}, + } + env := newKubernetesTestEnv(t, backup, failed) + br := &BackupReconciler{Client: env.client, Scheme: env.backup.Scheme, Recorder: record.NewFakeRecorder(10)} + res, err := br.cleanupKubernetesBackup(ctx, backup) + if err != nil { + t.Fatal(err) + } + if res.RequeueAfter != 0 { + t.Fatalf("cleanup keeps retrying after %d failures: %+v", kubernetesCleanupMaxAttempts, res) + } +} + +func (r *k8sRestoreRun) addJobPod(job, message string) { + r.t.Helper() + pod := &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: job + "-pod", Labels: map[string]string{"job-name": job}}, + Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{ + Name: "etcd", + State: corev1.ContainerState{Terminated: &corev1.ContainerStateTerminated{ExitCode: 1, Message: message + "\n"}}, + }}}, + } + if err := r.env.client.Create(context.Background(), pod); err != nil { + r.t.Fatal(err) + } +} + +func TestReconcileKubernetesRestoreFailureBeforeWriteResumes(t *testing.T) { + run := newK8sRestoreRun(t, `{"volumes":false}`, k8sBackupForRestore()) + run.verifyAndCapture(true) + run.step() // stop + run.step() // etcd Job + run.addJobPod("rj-restore", "") + run.finishJob("rj-restore", batchv1.JobFailed) + run.step() + got := run.get() + if got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed || meta.IsStatusConditionTrue(got.Status.Conditions, kubernetesCondEtcdWriteFailed) { + t.Fatalf("a Job that never touched the prefix must fail without marking it partial: %+v", got.Status) + } + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 2 { + t.Fatalf("control plane left stopped although the prefix was never changed: %d", n) + } +} + +func TestCleanupKubernetesRestoreKeepsClusterStoppedWhileRestoreJobRuns(t *testing.T) { + run := newK8sRestoreRun(t, `{"volumes":false}`, k8sBackupForRestore()) + run.verifyAndCapture(true) + run.step() // stop + run.step() // etcd Job, still running + run.env.restore.cleanupKubernetesRestore(context.Background(), run.get()) + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 0 { + t.Fatal("deleting the RestoreJob brought the control plane up while the restore Job may be rewriting the prefix") + } + job := &batchv1.Job{} + if err := run.env.client.Get(context.Background(), client.ObjectKey{Namespace: k8sTestNS, Name: "rj-restore"}, job); err == nil { + t.Fatal("the running restore Job was left to keep writing") + } +} + +func TestReconcileKubernetesRestoreIgnoresModeAnnotation(t *testing.T) { + run := newK8sRestoreRun(t, "", k8sBackupForRestore()) + rj := run.get() + rj.Annotations = map[string]string{"kubernetes.strategy.backups.cozystack.io/restore-state": `{"mode":"recreate","release":"kubernetes-c1"}`} + if err := run.env.client.Update(context.Background(), rj); err != nil { + t.Fatal(err) + } + run.verifyAndCapture(true) + if state := kubernetesStateFromRestoreJob(run.get()); state == nil || state.Mode != kubernetesRestoreModeInPlace { + t.Fatalf("a live cluster must be restored in place whatever the RestoreJob claims: %+v", state) + } +} + +func TestReconcileKubernetesBackupDiscardsDumpWhenVeleroFails(t *testing.T) { + ctx := context.Background() + bj := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "bj2"}, + Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: corev1.TypedLocalObjectReference{ + APIGroup: new(backupsv1alpha1.DefaultApplicationAPIGroup), Kind: kubernetesAppKind, Name: k8sTestApp, + }}, + } + env := newKubernetesTestEnv(t, bj) + step := func() { + t.Helper() + got := &backupsv1alpha1.BackupJob{} + if err := env.client.Get(ctx, client.ObjectKeyFromObject(bj), got); err != nil { + t.Fatal(err) + } + if _, err := env.backup.reconcileKubernetes(ctx, got, k8sResolved()); err != nil { + t.Fatal(err) + } + } + step() + step() + step() + dump := &batchv1.Job{} + if err := env.client.Get(ctx, client.ObjectKey{Namespace: k8sTestNS, Name: "bj2-backup"}, dump); err != nil { + t.Fatal(err) + } + dump.Status.Conditions = []batchv1.JobCondition{{Type: batchv1.JobComplete, Status: corev1.ConditionTrue}} + if err := env.client.Status().Update(ctx, dump); err != nil { + t.Fatal(err) + } + step() + vbs := &velerov1.BackupList{} + if err := env.client.List(ctx, vbs, client.MatchingLabels{backupsv1alpha1.OwningJobNameLabel: "bj2"}); err != nil || len(vbs.Items) != 1 { + t.Fatalf("Velero Backups: %d %v", len(vbs.Items), err) + } + vb := vbs.Items[0] + vb.Status.Phase = velerov1.BackupPhaseFailed + if err := env.client.Update(ctx, &vb); err != nil { + t.Fatal(err) + } + step() + got := &backupsv1alpha1.BackupJob{} + _ = env.client.Get(ctx, client.ObjectKeyFromObject(bj), got) + if got.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("BackupJob not failed: %+v", got.Status) + } + discard := &batchv1.Job{} + if err := env.client.Get(ctx, client.ObjectKey{Namespace: k8sTestNS, Name: "bj2-discard"}, discard); err != nil { + t.Fatalf("the orphaned etcd dump is not scheduled for deletion: %v", err) + } + if args := discard.Spec.Template.Spec.Containers[0].Args[0]; !strings.HasPrefix(args, "cleanup tenant-a/c1/bj2.etcd.json") { + t.Fatalf("discard Job does not delete this run's dump: %q", args) + } +} + +func TestReconcileKubernetesRestoreRecreateFailureBeforeWriteStartsControlPlane(t *testing.T) { + ctx := context.Background() + run := newK8sRestoreRun(t, `{"volumes":false}`, k8sBackupOfDeletedCluster()) + run.verifyAndCapture(true) + run.step() // secrets + run.finishVeleroRestore(velerov1.RestorePhaseCompleted) + run.step() + run.step() // cluster HelmRelease + dyn := run.env.restore.Interface + if _, err := dyn.Resource(tenantControlPlaneGVR).Namespace(k8sTestNS).Create(ctx, unstr(schema.GroupVersionKind{Group: "kamaji.clastix.io", Version: "v1alpha1", Kind: "TenantControlPlane"}, k8sTestNS, "kubernetes-gone", map[string]any{ + "status": map[string]any{"storage": map[string]any{"driver": "etcd", "dataStoreName": "tenant-a", "setup": map[string]any{"schema": "uid-1"}}}, + }), metav1.CreateOptions{}); err != nil { + t.Fatal(err) + } + if _, err := dyn.Resource(kamajiControlPlaneGVR).Namespace(k8sTestNS).Create(ctx, unstr(schema.GroupVersionKind{Group: "controlplane.cluster.x-k8s.io", Version: "v1alpha1", Kind: "KamajiControlPlane"}, k8sTestNS, "kubernetes-gone", map[string]any{ + "spec": map[string]any{"replicas": int64(2)}, + }), metav1.CreateOptions{}); err != nil { + t.Fatal(err) + } + run.step() // control plane is up + run.step() // stopped + run.step() // etcd Job + run.addJobPod("rj-restore", "") + run.finishJob("rj-restore", batchv1.JobFailed) + run.step() + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("rebuild not failed: %+v", got.Status) + } + if n, _ := run.replicas(kamajiControlPlaneGVR, "kubernetes-gone"); n != 2 { + t.Fatalf("the rebuilt control plane is left stopped although its prefix was never changed: %d", n) + } +} + +func TestReconcileKubernetesRestoreJobThatNeverRanResumes(t *testing.T) { + run := newK8sRestoreRun(t, `{"volumes":false}`, k8sBackupForRestore()) + run.verifyAndCapture(true) + run.step() // stop + run.step() // etcd Job + pod := &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "rj-restore-pod", Labels: map[string]string{"job-name": "rj-restore"}}, + Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{ + Name: "etcd", + State: corev1.ContainerState{Waiting: &corev1.ContainerStateWaiting{Reason: "ImagePullBackOff"}}, + }}}, + } + if err := run.env.client.Create(context.Background(), pod); err != nil { + t.Fatal(err) + } + run.step() // still waiting on the Job + if meta.IsStatusConditionTrue(run.get().Status.Conditions, kubernetesCondEtcdJobStarted) { + t.Fatal("a container that never ran was taken for a started Job") + } + run.finishJob("rj-restore", batchv1.JobFailed) + run.step() + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 2 { + t.Fatalf("control plane left stopped after a restore Job that never ran: %d", n) + } +} + +func TestReconcileKubernetesRestoreJobPodsGoneAfterStart(t *testing.T) { + run := newK8sRestoreRun(t, `{"volumes":false}`, k8sBackupForRestore()) + run.verifyAndCapture(true) + run.step() // stop + run.step() // etcd Job + pod := &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "rj-restore-pod", Labels: map[string]string{"job-name": "rj-restore"}}, + Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{{ + Name: "etcd", + State: corev1.ContainerState{Running: &corev1.ContainerStateRunning{}}, + }}}, + } + if err := run.env.client.Create(context.Background(), pod); err != nil { + t.Fatal(err) + } + run.step() + if !meta.IsStatusConditionTrue(run.get().Status.Conditions, kubernetesCondEtcdJobStarted) { + t.Fatal("a running restore Job was not recorded as started") + } + // The Job deadline deletes its Pods and fails it. + if err := run.env.client.Delete(context.Background(), pod); err != nil { + t.Fatal(err) + } + run.finishJob("rj-restore", batchv1.JobFailed) + run.step() + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 0 { + t.Fatal("control plane brought up after a restore Job that ran and left no trace of how far it got") + } +} + +func TestReconcileKubernetesRestoreRetriesStopAfterAPIError(t *testing.T) { + run := newK8sRestoreRun(t, `{"volumes":false}`, k8sBackupForRestore()) + run.verifyAndCapture(true) + failed := false + run.env.restore.Interface.(*dynamicfake.FakeDynamicClient).PrependReactor("patch", "kamajicontrolplanes", func(clienttesting.Action) (bool, runtime.Object, error) { + if failed { + return false, nil, nil + } + failed = true + return true, nil, fmt.Errorf("etcdserver: request timed out") + }) + run.step() + got := run.get() + if got.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("one API error aborted the restore: %+v", got.Status) + } + if c := meta.FindStatusCondition(got.Status.Conditions, kubernetesCondStopped); c == nil || !strings.Contains(c.Message, "API error") { + t.Fatalf("the retried API error is not reported: %+v", c) + } + run.step() + if !meta.IsStatusConditionTrue(run.get().Status.Conditions, kubernetesCondStopped) { + t.Fatal("the stop did not go through once the API answered again") + } +} + +// driveRecreateToEtcdJob runs a rebuild up to the etcd restore Job, the new +// TenantControlPlane reporting schema. +func driveRecreateToEtcdJob(t *testing.T, run *k8sRestoreRun, schemaName string) { + t.Helper() + ctx := context.Background() + run.verifyAndCapture(true) + run.step() // secrets + run.finishVeleroRestore(velerov1.RestorePhaseCompleted) + run.step() + run.step() // volumes + run.finishVeleroRestore(velerov1.RestorePhaseCompleted) + run.step() + run.step() // cluster HelmRelease + dyn := run.env.restore.Interface + if _, err := dyn.Resource(tenantControlPlaneGVR).Namespace(k8sTestNS).Create(ctx, unstr(schema.GroupVersionKind{Group: "kamaji.clastix.io", Version: "v1alpha1", Kind: "TenantControlPlane"}, k8sTestNS, "kubernetes-gone", map[string]any{ + "status": map[string]any{"storage": map[string]any{"driver": "etcd", "dataStoreName": "tenant-a", "setup": map[string]any{"schema": schemaName}}}, + }), metav1.CreateOptions{}); err != nil { + t.Fatal(err) + } + if _, err := dyn.Resource(kamajiControlPlaneGVR).Namespace(k8sTestNS).Create(ctx, unstr(schema.GroupVersionKind{Group: "controlplane.cluster.x-k8s.io", Version: "v1alpha1", Kind: "KamajiControlPlane"}, k8sTestNS, "kubernetes-gone", map[string]any{ + "spec": map[string]any{"replicas": int64(2)}, + }), metav1.CreateOptions{}); err != nil { + t.Fatal(err) + } + run.step() // control plane is up + run.step() // stopped + run.step() // etcd Job +} + +func TestReconcileKubernetesRestoreRecreateForcesSurvivingPool(t *testing.T) { + ctx := context.Background() + run := newK8sRestoreRun(t, "", k8sBackupOfDeletedCluster()) + // Deleting only the Kubernetes application leaves its KubernetesNodes + // pool's HelmRelease behind, while the pool's MachineDeployment went + // with the CAPI Cluster. + pool := unstr(schema.GroupVersionKind{Group: "helm.toolkit.fluxcd.io", Version: "v2", Kind: "HelmRelease"}, k8sTestNS, "kubernetes-nodes-gone-md0", map[string]any{"spec": map[string]any{"interval": "5m"}}) + if _, err := run.env.restore.Interface.Resource(helmReleaseGVR).Namespace(k8sTestNS).Create(ctx, pool, metav1.CreateOptions{}); err != nil { + t.Fatal(err) + } + driveRecreateToEtcdJob(t, run, "uid-1") + run.finishJob("rj-restore", batchv1.JobComplete) + run.step() + run.step() + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded { + t.Fatalf("rebuild not succeeded: %+v", got.Status) + } + ann := run.obj(helmReleaseGVR, "kubernetes-nodes-gone-md0").GetAnnotations() + if ann["reconcile.fluxcd.io/forceAt"] == "" || ann["reconcile.fluxcd.io/forceAt"] != ann["reconcile.fluxcd.io/requestedAt"] { + t.Fatalf("the surviving pool is not forced to recreate its MachineDeployment: %v", ann) + } +} + +func TestReconcileKubernetesBackupRefusesClusterStoppedByRestore(t *testing.T) { + ctx := context.Background() + bj := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "bj3"}, + Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: corev1.TypedLocalObjectReference{ + APIGroup: new(backupsv1alpha1.DefaultApplicationAPIGroup), Kind: kubernetesAppKind, Name: k8sTestApp, + }}, + } + env := newKubernetesTestEnv(t, bj) + patch := []byte(`{"metadata":{"annotations":{"` + kubernetesReplicasBeforeAnnotation + `":"2"}},"spec":{"replicas":0}}`) + if _, err := env.backup.Interface.Resource(kamajiControlPlaneGVR).Namespace(k8sTestNS).Patch(ctx, k8sTestRelease, types.MergePatchType, patch, metav1.PatchOptions{}); err != nil { + t.Fatal(err) + } + for range 2 { + got := &backupsv1alpha1.BackupJob{} + _ = env.client.Get(ctx, client.ObjectKeyFromObject(bj), got) + if _, err := env.backup.reconcileKubernetes(ctx, got, k8sResolved()); err != nil { + t.Fatal(err) + } + } + got := &backupsv1alpha1.BackupJob{} + _ = env.client.Get(ctx, client.ObjectKeyFromObject(bj), got) + if got.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("a cluster stopped by an unfinished restore was backed up: %+v", got.Status) + } + if err := env.client.Get(ctx, client.ObjectKey{Namespace: k8sTestNS, Name: "bj3-backup"}, &batchv1.Job{}); err == nil { + t.Fatal("the etcd prefix of a stopped cluster was dumped") + } +} + +func TestStopKubernetesClusterWaitsForTerminatingControlPlane(t *testing.T) { + ctx := context.Background() + run := newK8sRestoreRun(t, `{"volumes":false}`, k8sBackupForRestore()) + cp := unstr(schema.GroupVersionKind{Group: "apps", Version: "v1", Kind: "Deployment"}, k8sTestNS, k8sTestRelease, map[string]any{ + "spec": map[string]any{"selector": map[string]any{"matchLabels": map[string]any{"kamaji.clastix.io/name": k8sTestRelease}}}, + "status": map[string]any{"replicas": int64(0)}, + }) + if _, err := run.env.restore.Interface.Resource(deploymentGVR).Namespace(k8sTestNS).Create(ctx, cp, metav1.CreateOptions{}); err != nil { + t.Fatal(err) + } + pod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "apiserver-0", Labels: map[string]string{"kamaji.clastix.io/name": k8sTestRelease}}} + if err := run.env.client.Create(ctx, pod); err != nil { + t.Fatal(err) + } + run.verifyAndCapture(true) + run.step() + if meta.IsStatusConditionTrue(run.get().Status.Conditions, kubernetesCondStopped) { + t.Fatal("cluster reported stopped while a control-plane Pod is still terminating") + } + if err := run.env.client.Delete(ctx, pod); err != nil { + t.Fatal(err) + } + run.step() + if !meta.IsStatusConditionTrue(run.get().Status.Conditions, kubernetesCondStopped) { + t.Fatal("cluster not reported stopped once its control plane was gone") + } +} + +func TestKubernetesJobReachedWriteConservativeBranches(t *testing.T) { + for _, tc := range []struct { + name string + status corev1.ContainerStatus + want bool + }{ + {"running", corev1.ContainerStatus{Name: "etcd", State: corev1.ContainerState{Running: &corev1.ContainerStateRunning{}}}, true}, + {"restarted after writing", corev1.ContainerStatus{Name: "etcd", + State: corev1.ContainerState{Waiting: &corev1.ContainerStateWaiting{Reason: "CrashLoopBackOff"}}, + LastTerminationState: corev1.ContainerState{Terminated: &corev1.ContainerStateTerminated{Message: kubernetesJobWriting}}}, true}, + {"never ran", corev1.ContainerStatus{Name: "etcd", State: corev1.ContainerState{Waiting: &corev1.ContainerStateWaiting{Reason: "ImagePullBackOff"}}}, false}, + {"failed before writing", corev1.ContainerStatus{Name: "etcd", State: corev1.ContainerState{Terminated: &corev1.ContainerStateTerminated{ExitCode: 1}}}, false}, + } { + t.Run(tc.name, func(t *testing.T) { + run := newK8sRestoreRun(t, "", k8sBackupForRestore()) + pod := &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "p", Labels: map[string]string{"job-name": "rj-restore"}}, + Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{tc.status}}, + } + if err := run.env.client.Create(context.Background(), pod); err != nil { + t.Fatal(err) + } + got, err := run.env.restore.kubernetesJobReachedWrite(context.Background(), run.get(), "rj-restore") + if err != nil || got != tc.want { + t.Fatalf("reached write = %v (%v), want %v", got, err, tc.want) + } + }) + } +} + +// TestReconcileKubernetesRestoreFailsWhenClusterAPIIsNotServed covers a Cluster +// API that no longer serves the version the driver lists. Reading that as a +// cluster without workers would let the volumes path delete DataVolumes that +// running workers still mount. +func TestReconcileKubernetesRestoreFailsWhenClusterAPIIsNotServed(t *testing.T) { + for _, tc := range []struct{ resource, kind string }{ + {"machinedeployments", "MachineDeployment"}, + {"machines", "Machine"}, + } { + t.Run(tc.resource, func(t *testing.T) { + run := newK8sRestoreRun(t, "", k8sBackupForRestore()) + run.verifyAndCapture(true) + run.env.restore.Interface.(*dynamicfake.FakeDynamicClient).PrependReactor("list", tc.resource, func(clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, &meta.NoKindMatchError{GroupKind: schema.GroupKind{Group: "cluster.x-k8s.io", Kind: tc.kind}, SearchedVersions: []string{"v1beta1"}} + }) + run.step() + + got := run.get() + if got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed || !strings.Contains(got.Status.Message, "Cluster API is not served") { + t.Fatalf("restore not failed on an unserved Cluster API: %+v", got.Status) + } + if _, err := run.env.restore.Interface.Resource(dataVolumeGVR).Namespace(k8sTestNS).Get(context.Background(), "pvc-1234", metav1.GetOptions{}); err != nil { + t.Fatalf("backed-up DataVolume deleted: %v", err) + } + if n, _ := run.replicas(kamajiControlPlaneGVR, k8sTestRelease); n != 2 { + t.Fatalf("control plane left at %d replicas", n) + } + if n, _ := run.replicas(machineDeploymentGVR, k8sTestRelease+"-md0"); n != 3 { + t.Fatalf("worker pool left at %d replicas", n) + } + if run.suspended(k8sTestRelease) { + t.Fatal("cluster HelmRelease left suspended") + } + }) + } +} + +// TestReconcileKubernetesRestoreRefusesRecreatedCluster covers an application +// deleted after the backup and created again under the same name. Its new CA +// and service-account key come back only through a rebuild, so an in-place +// restore must refuse before it changes anything. Backups without a recorded +// UID are told apart by the datastore schema. +func TestReconcileKubernetesRestoreRefusesRecreatedCluster(t *testing.T) { + for _, tc := range []struct { + name string + edit func(map[string]string) + }{ + {"recorded UID", func(md map[string]string) { md[kubernetesControlPlaneUIDMetaKey] = "uid-of-the-deleted-cluster" }}, + {"schema of a backup without the UID", func(md map[string]string) { + delete(md, kubernetesControlPlaneUIDMetaKey) + md[kubernetesDatastorePrefixMetaKey] = "uid-of-the-deleted-cluster" + }}, + } { + t.Run(tc.name, func(t *testing.T) { + backup := k8sBackupForRestore() + tc.edit(backup.Spec.DriverMetadata) + run := newK8sRestoreRun(t, "", backup) + run.verifyAndCapture(true) + + got := run.get() + if got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed || !strings.Contains(got.Status.Message, "deleted and created again") { + t.Fatalf("restore into a re-created cluster not refused: %+v", got.Status) + } + if kubernetesStateFromRestoreJob(got) != nil { + t.Fatal("restore mode recorded for a refused restore") + } + if run.suspended(k8sTestRelease) { + t.Fatal("cluster HelmRelease suspended by a refused restore") + } + if n, _ := run.replicas(machineDeploymentGVR, k8sTestRelease+"-md0"); n != 3 { + t.Fatalf("worker pool scaled to %d by a refused restore", n) + } + }) + } +} + +// TestFailedKubernetesRestoreResumesWithoutClusterAPI covers Cluster API going +// unserved after the workers were stopped: the pools cannot be read back, but +// the autoscaler and the suspended HelmReleases must still be put back. +func TestFailedKubernetesRestoreResumesWithoutClusterAPI(t *testing.T) { + run := newK8sRestoreRun(t, "", k8sBackupForRestore()) + run.addMachine("w-1") + run.verifyAndCapture(true) + run.step() // workers stopping, the Machine still there + if n, _ := run.replicas(deploymentGVR, k8sTestRelease+"-cluster-autoscaler"); n != 0 { + t.Fatalf("autoscaler not stopped: %d", n) + } + for _, res := range []string{"machinedeployments", "machines"} { + run.env.restore.Interface.(*dynamicfake.FakeDynamicClient).PrependReactor("list", res, func(clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, &meta.NoKindMatchError{GroupKind: schema.GroupKind{Group: "cluster.x-k8s.io", Kind: "MachineDeployment"}, SearchedVersions: []string{"v1beta1"}} + }) + } + run.step() + + if got := run.get(); got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("restore not failed: %+v", got.Status) + } + if n, _ := run.replicas(deploymentGVR, k8sTestRelease+"-cluster-autoscaler"); n != 1 { + t.Fatalf("autoscaler left at %d replicas", n) + } + if run.suspended(k8sTestRelease) { + t.Fatal("cluster HelmRelease left suspended") + } +} + +// TestRenderKubernetesTemplateRefusesUnexpectedSchema pins the allowlist every +// strategy Job is rendered through: the schema becomes the prefix the restore +// deletes in the etcd the whole tenant shares. +func TestRenderKubernetesTemplateRefusesUnexpectedSchema(t *testing.T) { + tmpl := corev1.PodTemplateSpec{Spec: corev1.PodSpec{Containers: []corev1.Container{{Name: "etcd", Args: []string{"{{ .Datastore.Prefix }} {{ .Datastore.SourcePrefix }}"}}}}} + render := func(prefix, source string) error { + _, err := renderKubernetesTemplate(tmpl, k8sTestApp, k8sTestNS, kubernetesModeRestore, "key", kubernetesDatastore{Prefix: prefix, SourcePrefix: source}, nil) + return err + } + for _, ok := range []string{"0f2c1e4a-77b1-4c39-9a3e-6d1f2b9c8e10", "tenant_root_kubernetes_bk1"} { + if err := render(ok, ok); err != nil { + t.Errorf("schema %q refused: %v", ok, err) + } + } + if err := render("tenant_a_kubernetes_c1", ""); err != nil { + t.Errorf("a Job without a source schema refused: %v", err) + } + for _, bad := range []string{"", "a/b", "/", "..", "a..b", "a.b", "Tenant_a", "a b", "a\nb", "a*"} { + if render(bad, "tenant_a_kubernetes_c1") == nil { + t.Errorf("schema %q accepted", bad) + } + if bad != "" && render("tenant_a_kubernetes_c1", bad) == nil { + t.Errorf("source schema %q accepted", bad) + } + } +} + +// TestReconcileKubernetesRestoreRefusesEditedPrefix covers a Backup whose +// recorded prefix was edited: the restore fails in its preflight, before any +// Job runs against the shared etcd or the cluster is stopped. +func TestReconcileKubernetesRestoreRefusesEditedPrefix(t *testing.T) { + backup := k8sBackupForRestore() + backup.Spec.DriverMetadata[kubernetesDatastorePrefixMetaKey] = "tenant_a_kubernetes_c1/.." + run := newK8sRestoreRun(t, "", backup) + run.step() // StartedAt + run.step() // preflight + + got := run.get() + if got.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed || !strings.Contains(got.Status.Message, "refusing") { + t.Fatalf("restore from an edited prefix not refused: %+v", got.Status) + } + jobs := &batchv1.JobList{} + if err := run.env.client.List(context.Background(), jobs); err != nil || len(jobs.Items) != 0 { + t.Fatalf("Jobs created for an edited prefix: %d %v", len(jobs.Items), err) + } + if run.suspended(k8sTestRelease) { + t.Fatal("cluster HelmRelease suspended") + } +} + +// TestReconcileKubernetesBackupRefusesUnexpectedSchema covers a live +// TenantControlPlane reporting a schema outside the allowlist. +func TestReconcileKubernetesBackupRefusesUnexpectedSchema(t *testing.T) { + ctx := context.Background() + bj := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: k8sTestNS, Name: "bj1"}, + Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: corev1.TypedLocalObjectReference{ + APIGroup: new(backupsv1alpha1.DefaultApplicationAPIGroup), Kind: kubernetesAppKind, Name: k8sTestApp, + }}, + } + env := newKubernetesTestEnv(t, bj) + tcp, err := env.backup.Interface.Resource(tenantControlPlaneGVR).Namespace(k8sTestNS).Get(ctx, k8sTestRelease, metav1.GetOptions{}) + if err != nil { + t.Fatal(err) + } + if err := unstructured.SetNestedField(tcp.Object, "tenant_a/kubernetes_c1", "status", "storage", "setup", "schema"); err != nil { + t.Fatal(err) + } + if _, err := env.backup.Interface.Resource(tenantControlPlaneGVR).Namespace(k8sTestNS).Update(ctx, tcp, metav1.UpdateOptions{}); err != nil { + t.Fatal(err) + } + get := func() *backupsv1alpha1.BackupJob { + out := &backupsv1alpha1.BackupJob{} + if err := env.client.Get(ctx, client.ObjectKeyFromObject(bj), out); err != nil { + t.Fatal(err) + } + return out + } + for i := 0; i < 4 && get().Status.Phase != backupsv1alpha1.BackupJobPhaseFailed; i++ { + if _, err := env.backup.reconcileKubernetes(ctx, get(), k8sResolved()); err != nil { + t.Fatal(err) + } + } + if got := get(); got.Status.Phase != backupsv1alpha1.BackupJobPhaseFailed || !strings.Contains(got.Status.Message, "refusing") { + t.Fatalf("backup of an unexpected schema not refused: %+v", got.Status) + } + jobs := &batchv1.JobList{} + if err := env.client.List(ctx, jobs); err != nil || len(jobs.Items) != 0 { + t.Fatalf("dump Job created for an unexpected schema: %d %v", len(jobs.Items), err) + } +} diff --git a/internal/backupcontroller/kubernetesstrategy_restore.go b/internal/backupcontroller/kubernetesstrategy_restore.go new file mode 100644 index 000000000..830a25927 --- /dev/null +++ b/internal/backupcontroller/kubernetesstrategy_restore.go @@ -0,0 +1,1129 @@ +// SPDX-License-Identifier: Apache-2.0 +package backupcontroller + +import ( + "context" + "encoding/json" + "fmt" + "strconv" + "strings" + "time" + + velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1" + batchv1 "k8s.io/api/batch/v1" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + apimeta "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + ctrl "sigs.k8s.io/controller-runtime" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/yaml" + + strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1" + backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" + "github.com/cozystack/cozystack/internal/template" +) + +const ( + // kubernetesCondMode records on the RestoreJob whether it rebuilds a + // deleted cluster or rolls a live one back (Reason), and which cluster + // (Message). It lives in status, which the creator of a RestoreJob cannot + // set, so a RestoreJob cannot claim a live cluster is gone. + kubernetesCondMode = "KubernetesRestoreMode" + kubernetesRestorePhaseLabel = "kubernetes.strategy.backups.cozystack.io/restore-phase" + + // What a restore changes on a live cluster is recorded on the objects it + // changes rather than on the RestoreJob, so the cluster can be put back + // after the RestoreJob is deleted, and a retry after a failed restore + // keeps the counts from before the first one rather than the zeros. + kubernetesReplicasBeforeAnnotation = "kubernetes.strategy.backups.cozystack.io/replicas-before-restore" + kubernetesSuspendedByAnnotation = "kubernetes.strategy.backups.cozystack.io/suspended-by-restore" + kubernetesReplicasUnset = "unset" + + kubernetesRestoreModeRecreate = "recreate" + kubernetesRestoreModeInPlace = "in-place" + + kubernetesPhaseSecrets = "secrets" + kubernetesPhaseVolumes = "volumes" + + machineExcludeDrainAnnotation = "machine.cluster.x-k8s.io/exclude-node-draining" + machineExcludeVolumeDetachAnnotation = "machine.cluster.x-k8s.io/exclude-wait-for-node-volume-detach" + + kubernetesCondVerified = "KubernetesBackupVerified" + kubernetesCondRecreated = "KubernetesClusterRecreated" + kubernetesCondStopped = "KubernetesClusterStopped" + kubernetesCondSecretsRestored = "KubernetesSecretsRestored" + kubernetesCondEtcdRestored = "KubernetesEtcdRestored" + kubernetesCondVolumesRestored = "KubernetesVolumesRestored" + // kubernetesCondEtcdWriteFailed marks a restore whose etcd Job failed + // after it started replacing the prefix. The control plane is then left + // stopped, by a failure and by the RestoreJob's deletion alike. + kubernetesCondEtcdWriteFailed = "KubernetesEtcdWriteFailed" + // kubernetesCondEtcdJobStarted records that a container of the restore + // Job was seen running, which is the only trace left once the Job's + // deadline has deleted its Pods. + kubernetesCondEtcdJobStarted = "KubernetesEtcdJobStarted" + + // The restore Job writes this to its termination message just before it + // deletes the prefix, so a failed Job tells whether the prefix is still + // intact. + kubernetesJobWriting = "writing" +) + +// KubernetesRestoreOptions is the typed shape of RestoreJob.Spec.Options for +// the Kubernetes driver. Both halves of the backup are restored unless one is +// switched off. +type KubernetesRestoreOptions struct { + // Etcd restores the cluster's etcd prefix (its API objects). + Etcd *bool `json:"etcd,omitempty"` + // Volumes restores the DataVolumes backing the cluster's PersistentVolumes. + Volumes *bool `json:"volumes,omitempty"` +} + +func (o KubernetesRestoreOptions) restoreEtcd() bool { return boolDefault(o.Etcd, true) } +func (o KubernetesRestoreOptions) restoreVolumes() bool { return boolDefault(o.Volumes, true) } + +func parseKubernetesRestoreOptions(opts *runtime.RawExtension) (KubernetesRestoreOptions, error) { + var out KubernetesRestoreOptions + if opts == nil || len(opts.Raw) == 0 { + return out, nil + } + if err := json.Unmarshal(opts.Raw, &out); err != nil { + return KubernetesRestoreOptions{}, fmt.Errorf("decode restoreJob.spec.options: %w", err) + } + if !out.restoreEtcd() && !out.restoreVolumes() { + return KubernetesRestoreOptions{}, fmt.Errorf("restoreJob.spec.options disables both etcd and volumes; nothing to restore") + } + return out, nil +} + +type kubernetesRestoreState struct { + Mode string `json:"mode"` + Release string `json:"release"` +} + +func kubernetesAutoscalerName(release string) string { + return release + "-cluster-autoscaler" +} + +func kubernetesBackupDataVolumes(b *backupsv1alpha1.Backup) []string { + raw := b.Spec.DriverMetadata[kubernetesDataVolumesMetaKey] + if raw == "" { + return nil + } + return strings.Split(raw, ",") +} + +func kubernetesBackupHelmReleases(b *backupsv1alpha1.Backup) ([]map[string]any, error) { + var out []map[string]any + if err := json.Unmarshal([]byte(b.Spec.DriverMetadata[kubernetesHelmReleasesMetaKey]), &out); err != nil { + return nil, fmt.Errorf("decode recorded HelmReleases: %w", err) + } + if len(out) == 0 { + return nil, fmt.Errorf("the Backup records no HelmReleases") + } + return out, nil +} + +// kubernetesPreRestoreKey is where the restore Job keeps the prefix it is +// about to replace. The Job writes it only when none is there and deletes it +// once the restore succeeds, so after a failed restore it holds the prefix +// as it was before the first failed attempt, and a retry does not overwrite +// it with a partial one. It is fixed per backup, so the Backup's cleanup +// removes it. +func kubernetesPreRestoreKey(objectKey string) string { + return objectKey + ".pre-restore" +} + +// reconcileKubernetesRestore restores a tenant Kubernetes application in +// place, under its own name and namespace. +// +// When the application is gone, the restore rebuilds it: control-plane PKI +// Secrets first (the etcd Job authenticates with one of them), then the etcd +// prefix, the volumes, and finally the HelmReleases, so the control plane +// comes up on restored state and the existing CA. +// +// When the application is live, the restore stops what the selected halves +// need stopped — the workers for volumes, so the volumes are detached, then +// the control plane for etcd, so nothing writes while the prefix is replaced +// — writes the state back, and returns the cluster to the replica counts it +// had. +// +// Nothing is changed before the backup's artifacts are confirmed to exist. +func (r *RestoreJobReconciler) reconcileKubernetesRestore(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup) (ctrl.Result, error) { + if rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseSucceeded || + rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed { + return ctrl.Result{}, nil + } + if err := validateKubernetesApplicationRef(backup.Spec.ApplicationRef); err != nil { + return r.markRestoreJobFailed(ctx, rj, err.Error()) + } + appName := backup.Spec.ApplicationRef.Name + if t := rj.Spec.TargetApplicationRef; t != nil && ((t.Name != "" && t.Name != appName) || (t.Kind != "" && t.Kind != kubernetesAppKind)) { + return r.markRestoreJobFailed(ctx, rj, "the Kubernetes strategy restores only into the application the backup was taken from") + } + if rj.Namespace != backup.Namespace { + return r.markRestoreJobFailed(ctx, rj, "the Kubernetes strategy restores only into the namespace the backup was taken from") + } + opts, err := parseKubernetesRestoreOptions(rj.Spec.Options) + if err != nil { + return r.markRestoreJobFailed(ctx, rj, err.Error()) + } + + if rj.Status.StartedAt == nil { + base := rj.DeepCopy() + now := metav1.Now() + rj.Status.StartedAt = &now + rj.Status.Phase = backupsv1alpha1.RestoreJobPhaseRunning + if err := r.Status().Patch(ctx, rj, client.MergeFrom(base)); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + } + + strategy := &strategyv1alpha1.Kubernetes{} + if err := r.Get(ctx, client.ObjectKey{Name: backup.Spec.StrategyRef.Name}, strategy); err != nil { + if apierrors.IsNotFound(err) { + return r.requeueRestoreStrategyNotReady(ctx, rj, backup.Spec.StrategyRef.Name) + } + return ctrl.Result{}, err + } + + if !apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondVerified) { + done, msg, err := r.verifyKubernetesBackup(ctx, rj, backup, strategy, opts) + if err != nil || msg != "" || !done { + return r.kubernetesStepResult(ctx, rj, nil, err, msg, done) + } + return r.setKubernetesRestoreCondition(ctx, rj, kubernetesCondVerified, "the backup's artifacts are present") + } + + state := kubernetesStateFromRestoreJob(rj) + if state == nil { + return r.captureKubernetesRestoreState(ctx, rj, backup) + } + + if state.Mode == kubernetesRestoreModeInPlace { + return r.reconcileKubernetesInPlace(ctx, rj, backup, strategy, opts, state) + } + return r.reconcileKubernetesRecreate(ctx, rj, backup, strategy, opts, state) +} + +func kubernetesStateFromRestoreJob(rj *backupsv1alpha1.RestoreJob) *kubernetesRestoreState { + c := apimeta.FindStatusCondition(rj.Status.Conditions, kubernetesCondMode) + if c == nil { + return nil + } + mode := kubernetesRestoreModeInPlace + if c.Reason == "Recreate" { + mode = kubernetesRestoreModeRecreate + } + return &kubernetesRestoreState{Mode: mode, Release: c.Message} +} + +// verifyKubernetesBackup confirms, before anything is stopped or deleted, +// that the Velero backup completed and still exists, and that the etcd dump +// object is in the bucket. The dump is checked by a verify-mode run of the +// strategy Job, the only component holding the bucket credentials. +func (r *RestoreJobReconciler) verifyKubernetesBackup(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, strategy *strategyv1alpha1.Kubernetes, opts KubernetesRestoreOptions) (bool, string, error) { + name := backup.Spec.DriverMetadata[veleroBackupNameMetadataKey] + if name == "" { + return false, "the Backup records no Velero backup", nil + } + vb := &velerov1.Backup{} + if err := r.Get(ctx, client.ObjectKey{Namespace: veleroNamespace, Name: name}, vb); err != nil { + if apierrors.IsNotFound(err) { + return false, fmt.Sprintf("Velero backup %s/%s no longer exists", veleroNamespace, name), nil + } + return false, "", err + } + if vb.Status.Phase != velerov1.BackupPhaseCompleted { + return false, fmt.Sprintf("Velero backup %s/%s is %q, not Completed", veleroNamespace, name, vb.Status.Phase), nil + } + if !opts.restoreEtcd() { + return true, "", nil + } + return r.runKubernetesJob(ctx, rj, backup, strategy, kubernetesModeVerify, rj.Name+"-verify", kubernetesBackupDatastore(backup)) +} + +// captureKubernetesRestoreState decides between rebuilding a deleted cluster +// and restoring a live one. +func (r *RestoreJobReconciler) captureKubernetesRestoreState(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup) (ctrl.Result, error) { + release := kubernetesReleaseName(backup.Spec.ApplicationRef.Name) + reason := "Recreate" + _, err := r.Resource(helmReleaseGVR).Namespace(rj.Namespace).Get(ctx, release, metav1.GetOptions{}) + switch { + case apierrors.IsNotFound(err): + case err != nil: + return ctrl.Result{}, err + default: + reason = "InPlace" + mismatch, err := r.kubernetesClusterMismatch(ctx, rj.Namespace, release, backup) + if err != nil { + return ctrl.Result{}, err + } + if mismatch != "" { + return r.markRestoreJobFailed(ctx, rj, mismatch) + } + } + apimeta.SetStatusCondition(&rj.Status.Conditions, metav1.Condition{ + Type: kubernetesCondMode, + Status: metav1.ConditionTrue, + Reason: reason, + Message: release, + }) + if err := r.Status().Update(ctx, rj); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil +} + +// capiNotServed reports a list of a Cluster API resource that failed because +// the apiserver does not serve that resource at the version the driver asks +// for: a NotFound on a collection, or no RESTMapper match. +func capiNotServed(err error) bool { + return err != nil && (apierrors.IsNotFound(err) || apimeta.IsNoMatchError(err)) +} + +// kubernetesClusterMismatch reports why the live cluster is not the one the +// backup was taken from, or "" when it is or cannot be told apart. A cluster +// deleted and created again under the same name has a new CA, service-account +// key and Talos secrets, which only a rebuild restores: writing the old prefix +// into it would leave every token and node signed by keys it no longer has. +// Backups that predate the recorded UID fall back to the datastore schema, +// which tells the two apart only on a Kamaji that keys it on the UID. +func (r *RestoreJobReconciler) kubernetesClusterMismatch(ctx context.Context, ns, release string, backup *backupsv1alpha1.Backup) (string, error) { + tcp, err := r.Resource(tenantControlPlaneGVR).Namespace(ns).Get(ctx, release, metav1.GetOptions{}) + if apierrors.IsNotFound(err) { + return "", nil + } + if err != nil { + return "", err + } + md := backup.Spec.DriverMetadata + recorded, live := md[kubernetesControlPlaneUIDMetaKey], string(tcp.GetUID()) + if recorded == "" { + recorded = md[kubernetesDatastorePrefixMetaKey] + live, _, _ = unstructured.NestedString(tcp.Object, "status", "storage", "setup", "schema") + } + if recorded == "" || live == "" || recorded == live { + return "", nil + } + return fmt.Sprintf("TenantControlPlane %s/%s is not the cluster this backup was taken from (%s, backup has %s): it was deleted and created again under the same name, so an in-place restore would write the old state over a new CA and service-account key; delete the application and restore again to rebuild it with the backed-up PKI", ns, release, live, recorded), nil +} + +func (r *RestoreJobReconciler) listClusterMachineDeployments(ctx context.Context, ns, release string) ([]unstructured.Unstructured, error) { + list, err := r.Resource(machineDeploymentGVR).Namespace(ns).List(ctx, metav1.ListOptions{}) + if err != nil { + return nil, err + } + var out []unstructured.Unstructured + for _, md := range list.Items { + if cluster, _, _ := unstructured.NestedString(md.Object, "spec", "clusterName"); cluster == release { + out = append(out, md) + } + } + return out, nil +} + +// --------------------------------------------------------------------------- +// Rebuild a deleted cluster +// --------------------------------------------------------------------------- + +// reconcileKubernetesRecreate rebuilds a deleted cluster. The control-plane +// PKI and the volumes come back first, then the cluster's own HelmRelease. +// Kamaji gives the new TenantControlPlane a fresh datastore schema (keyed on +// its UID), so the etcd half can only be written once that schema exists: +// the restore then stops the new control plane like a live one, writes the +// dump into the new prefix, and starts it again. The worker pools are +// recreated last, so their workers join the restored cluster rather than an +// empty one. +func (r *RestoreJobReconciler) reconcileKubernetesRecreate(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, strategy *strategyv1alpha1.Kubernetes, opts KubernetesRestoreOptions, state *kubernetesRestoreState) (ctrl.Result, error) { + releases, err := kubernetesBackupHelmReleases(backup) + if err != nil { + return r.markRestoreJobFailed(ctx, rj, err.Error()) + } + + if !apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondSecretsRestored) { + done, msg, err := r.runKubernetesVeleroRestore(ctx, rj, backup, strategy, kubernetesPhaseSecrets, state.Release) + if err != nil || msg != "" || !done { + return r.kubernetesStepResult(ctx, rj, nil, err, msg, done) + } + return r.setKubernetesRestoreCondition(ctx, rj, kubernetesCondSecretsRestored, "control-plane Secrets restored") + } + if opts.restoreVolumes() && !apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondVolumesRestored) { + done, msg, err := r.runKubernetesVeleroRestore(ctx, rj, backup, strategy, kubernetesPhaseVolumes, state.Release) + if err != nil || msg != "" || !done { + return r.kubernetesStepResult(ctx, rj, nil, err, msg, done) + } + return r.setKubernetesRestoreCondition(ctx, rj, kubernetesCondVolumesRestored, "volumes restored") + } + + if !opts.restoreEtcd() { + if err := r.createKubernetesHelmReleases(ctx, rj.Namespace, releases, true); err != nil { + return r.markRestoreJobFailed(ctx, rj, err.Error()) + } + return r.markKubernetesRestoreSucceeded(ctx, rj, "cluster rebuilt from backup without its etcd state") + } + + if !apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondRecreated) { + if err := r.createKubernetesHelmReleases(ctx, rj.Namespace, releases[:1], false); err != nil { + return r.markRestoreJobFailed(ctx, rj, err.Error()) + } + ready, err := r.tenantControlPlaneSchemaReady(ctx, rj.Namespace, state.Release) + if err != nil { + return ctrl.Result{}, err + } + if ready { + return r.setKubernetesRestoreCondition(ctx, rj, kubernetesCondRecreated, "the control plane is up with a datastore schema") + } + apimeta.SetStatusCondition(&rj.Status.Conditions, metav1.Condition{ + Type: kubernetesCondRecreated, + Status: metav1.ConditionFalse, + Reason: "Waiting", + Message: "waiting for TenantControlPlane " + state.Release + " to report its datastore schema", + }) + cond := apimeta.FindStatusCondition(rj.Status.Conditions, kubernetesCondRecreated) + if time.Since(cond.LastTransitionTime.Time) > restorePreparationDeadline { + return r.markRestoreJobFailed(ctx, rj, fmt.Sprintf("TenantControlPlane %s/%s reported no datastore schema within %s", rj.Namespace, state.Release, restorePreparationDeadline)) + } + if err := r.Status().Update(ctx, rj); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + } + + return r.reconcileKubernetesInPlace(ctx, rj, backup, strategy, KubernetesRestoreOptions{Etcd: new(true), Volumes: new(false)}, state) +} + +func (r *RestoreJobReconciler) tenantControlPlaneSchemaReady(ctx context.Context, ns, release string) (bool, error) { + tcp, err := r.Resource(tenantControlPlaneGVR).Namespace(ns).Get(ctx, release, metav1.GetOptions{}) + if err != nil { + return false, client.IgnoreNotFound(err) + } + schema, _, _ := unstructured.NestedString(tcp.Object, "status", "storage", "setup", "schema") + return schema != "", nil +} + +// createKubernetesHelmReleases recreates the recorded HelmReleases. A worker +// pool's HelmRelease outlives the deletion of its cluster (KubernetesNodes is +// an application of its own), but its MachineDeployment was garbage-collected +// with the CAPI Cluster, and nothing re-applies an unchanged release. One +// that exists is therefore forced through a Helm upgrade, which recreates +// the objects its release is missing. The cluster's own release is created +// while the rebuild polls for its schema, so it passes force=false: forcing +// it on every poll would re-run its chart while Kamaji brings it up. +func (r *RestoreJobReconciler) createKubernetesHelmReleases(ctx context.Context, ns string, releases []map[string]any, force bool) error { + for _, hr := range releases { + obj := &unstructured.Unstructured{Object: hr} + obj.SetNamespace(ns) + _, err := r.Resource(helmReleaseGVR).Namespace(ns).Create(ctx, obj, metav1.CreateOptions{}) + if err == nil { + continue + } + if !apierrors.IsAlreadyExists(err) { + return fmt.Errorf("recreate HelmRelease %s/%s: %w", ns, obj.GetName(), err) + } + if !force { + continue + } + now := time.Now().UTC().Format(time.RFC3339Nano) + patch, merr := json.Marshal(map[string]any{"metadata": map[string]any{"annotations": map[string]string{ + "reconcile.fluxcd.io/requestedAt": now, + "reconcile.fluxcd.io/forceAt": now, + }}}) + if merr != nil { + return merr + } + if _, err := r.Resource(helmReleaseGVR).Namespace(ns).Patch(ctx, obj.GetName(), types.MergePatchType, patch, metav1.PatchOptions{}); err != nil { + return fmt.Errorf("force HelmRelease %s/%s to reconcile: %w", ns, obj.GetName(), err) + } + } + return nil +} + +// --------------------------------------------------------------------------- +// Restore a live cluster +// --------------------------------------------------------------------------- + +func (r *RestoreJobReconciler) reconcileKubernetesInPlace(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, strategy *strategyv1alpha1.Kubernetes, opts KubernetesRestoreOptions, state *kubernetesRestoreState) (ctrl.Result, error) { + if !apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondStopped) { + stopped, waiting, err := r.stopKubernetesCluster(ctx, rj, backup, opts, state.Release) + if capiNotServed(err) { + // Reading this as "no workers" would delete volumes that running + // workers still mount. Workers are listed before anything is + // stopped, so the resume below has only what it already stopped + // to put back. + return r.failKubernetesInPlace(ctx, rj, state, fmt.Sprintf("Cluster API is not served at %s: %v", machineDeploymentGVR.GroupVersion(), err)) + } + if err != nil { + // Retried like any other wait, under the same deadline: a single + // apiserver blip must not abort a restore. + stopped, waiting = false, fmt.Sprintf("a retry after an API error: %v", err) + } + if stopped { + return r.setKubernetesRestoreCondition(ctx, rj, kubernetesCondStopped, "cluster stopped for restore") + } + // The condition's transition time marks when stopping began: it is + // kept while the status stays False, whatever the message says. + apimeta.SetStatusCondition(&rj.Status.Conditions, metav1.Condition{ + Type: kubernetesCondStopped, + Status: metav1.ConditionFalse, + Reason: "Stopping", + Message: "waiting for " + waiting, + }) + cond := apimeta.FindStatusCondition(rj.Status.Conditions, kubernetesCondStopped) + if time.Since(cond.LastTransitionTime.Time) > restorePreparationDeadline { + return r.failKubernetesInPlace(ctx, rj, state, fmt.Sprintf("the cluster did not stop within %s: still waiting for %s", restorePreparationDeadline, waiting)) + } + if err := r.Status().Update(ctx, rj); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + } + + if res, next, err := r.kubernetesRestoreData(ctx, rj, backup, strategy, opts, state); !next { + return res, err + } + + if err := r.resumeKubernetesCluster(ctx, rj.Namespace, state.Release); err != nil { + return ctrl.Result{}, err + } + if state.Mode == kubernetesRestoreModeRecreate { + releases, err := kubernetesBackupHelmReleases(backup) + if err != nil { + return r.markRestoreJobFailed(ctx, rj, err.Error()) + } + if err := r.createKubernetesHelmReleases(ctx, rj.Namespace, releases[1:], true); err != nil { + return r.markRestoreJobFailed(ctx, rj, err.Error()) + } + return r.markKubernetesRestoreSucceeded(ctx, rj, "cluster rebuilt from backup") + } + return r.markKubernetesRestoreSucceeded(ctx, rj, "cluster restored in place") +} + +// stopKubernetesCluster suspends the cluster's HelmReleases so Flux does not +// undo the scale-down, then stops the workers when volumes are restored — +// deleting the data DataVolumes and PVCs the backup holds once nothing mounts +// them — and the control plane when etcd is. Workers go first: CAPI removes a +// Machine's Node through the tenant API. It reports whether all of that has +// settled and, if not, what it is waiting for. +func (r *RestoreJobReconciler) stopKubernetesCluster(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, opts KubernetesRestoreOptions, release string) (bool, string, error) { + ns := rj.Namespace + mds, err := r.listClusterMachineDeployments(ctx, ns, release) + if err != nil { + return false, "", err + } + for _, hr := range kubernetesHelmReleaseNames(mds, release) { + if err := r.suspendHelmReleaseForRestore(ctx, ns, hr); err != nil { + return false, "", err + } + } + + if opts.restoreVolumes() { + if err := r.stopReplicas(ctx, deploymentGVR, ns, kubernetesAutoscalerName(release)); err != nil { + return false, "", err + } + for _, md := range mds { + if err := r.stopReplicas(ctx, machineDeploymentGVR, ns, md.GetName()); err != nil { + return false, "", err + } + } + remaining, err := r.releaseClusterMachines(ctx, ns, release) + if err != nil { + return false, "", err + } + if remaining > 0 { + return false, fmt.Sprintf("%d worker Machine(s) to be deleted", remaining), nil + } + var pending []string + for _, name := range kubernetesBackupDataVolumes(backup) { + dvGone, err := r.deleteDataVolume(ctx, ns, name) + if err != nil { + return false, "", err + } + pvcGone, err := r.deletePVC(ctx, ns, name) + if err != nil { + return false, "", err + } + if !dvGone || !pvcGone { + pending = append(pending, name) + } + } + if len(pending) > 0 { + return false, "data volumes to be deleted: " + strings.Join(pending, ", "), nil + } + } + + if opts.restoreEtcd() { + if err := r.stopReplicas(ctx, kamajiControlPlaneGVR, ns, release); err != nil { + return false, "", err + } + cp, err := r.Resource(deploymentGVR).Namespace(ns).Get(ctx, release, metav1.GetOptions{}) + switch { + case apierrors.IsNotFound(err): + case err != nil: + return false, "", err + default: + if n, _, _ := unstructured.NestedInt64(cp.Object, "status", "replicas"); n > 0 { + return false, fmt.Sprintf("%d control-plane Pod(s) to stop", n), nil + } + // status.replicas leaves out Pods that are still terminating, and + // a terminating kube-apiserver can still write to etcd. + selector, _, _ := unstructured.NestedStringMap(cp.Object, "spec", "selector", "matchLabels") + if len(selector) > 0 { + pods := &corev1.PodList{} + if err := r.List(ctx, pods, client.InNamespace(ns), client.MatchingLabels(selector)); err != nil { + return false, "", err + } + if len(pods.Items) > 0 { + return false, fmt.Sprintf("%d control-plane Pod(s) to terminate", len(pods.Items)), nil + } + } + } + } + return true, "", nil +} + +// releaseClusterMachines lets the cluster's Machines finish deleting without +// draining their Nodes or waiting for volumes to detach: the workers are +// recreated after the restore, and a drain can block on a PodDisruptionBudget. +// It returns how many Machines remain. +func (r *RestoreJobReconciler) releaseClusterMachines(ctx context.Context, ns, release string) (int, error) { + machines, err := r.Resource(machineGVR).Namespace(ns).List(ctx, metav1.ListOptions{LabelSelector: clusterNameLabel + "=" + release}) + if err != nil { + return 0, err + } + patch := []byte(`{"metadata":{"annotations":{"` + machineExcludeDrainAnnotation + `":"","` + machineExcludeVolumeDetachAnnotation + `":""}}}`) + for _, m := range machines.Items { + ann := m.GetAnnotations() + if _, ok := ann[machineExcludeDrainAnnotation]; ok { + if _, ok := ann[machineExcludeVolumeDetachAnnotation]; ok { + continue + } + } + if _, err := r.Resource(machineGVR).Namespace(ns).Patch(ctx, m.GetName(), types.MergePatchType, patch, metav1.PatchOptions{}); err != nil && !apierrors.IsNotFound(err) { + return 0, err + } + } + return len(machines.Items), nil +} + +// suspendHelmReleaseForRestore suspends a HelmRelease and marks it as +// suspended by the restore. One already suspended is left alone and unmarked, +// so resuming the cluster does not resume it. +func (r *RestoreJobReconciler) suspendHelmReleaseForRestore(ctx context.Context, ns, name string) error { + hr, err := r.Resource(helmReleaseGVR).Namespace(ns).Get(ctx, name, metav1.GetOptions{}) + if err != nil { + return client.IgnoreNotFound(err) + } + if suspended, _, _ := unstructured.NestedBool(hr.Object, "spec", "suspend"); suspended { + return nil + } + patch := []byte(`{"metadata":{"annotations":{"` + kubernetesSuspendedByAnnotation + `":"true"}},"spec":{"suspend":true}}`) + _, err = r.Resource(helmReleaseGVR).Namespace(ns).Patch(ctx, name, types.MergePatchType, patch, metav1.PatchOptions{}) + return client.IgnoreNotFound(err) +} + +// stopReplicas scales an object to zero, first recording its replica count +// on it. A count already recorded is kept: it was taken before an earlier +// restore stopped the object. +func (r *RestoreJobReconciler) stopReplicas(ctx context.Context, gvr schema.GroupVersionResource, ns, name string) error { + obj, err := r.Resource(gvr).Namespace(ns).Get(ctx, name, metav1.GetOptions{}) + if err != nil { + return client.IgnoreNotFound(err) + } + annotations := map[string]any{} + if _, recorded := obj.GetAnnotations()[kubernetesReplicasBeforeAnnotation]; !recorded { + before := kubernetesReplicasUnset + if n, found, _ := unstructured.NestedInt64(obj.Object, "spec", "replicas"); found { + before = strconv.FormatInt(n, 10) + } + annotations[kubernetesReplicasBeforeAnnotation] = before + } + patch, err := json.Marshal(map[string]any{ + "metadata": map[string]any{"annotations": annotations}, + "spec": map[string]any{"replicas": 0}, + }) + if err != nil { + return err + } + _, err = r.Resource(gvr).Namespace(ns).Patch(ctx, name, types.MergePatchType, patch, metav1.PatchOptions{}) + return client.IgnoreNotFound(err) +} + +// resumeReplicas puts back the replica count stopReplicas recorded — removing +// the field when it was unset, so its owner (the autoscaler, for a +// MachineDeployment) takes it back — and drops the record. +func (r *RestoreJobReconciler) resumeReplicas(ctx context.Context, gvr schema.GroupVersionResource, ns, name string) error { + obj, err := r.Resource(gvr).Namespace(ns).Get(ctx, name, metav1.GetOptions{}) + if err != nil { + return client.IgnoreNotFound(err) + } + before, recorded := obj.GetAnnotations()[kubernetesReplicasBeforeAnnotation] + if !recorded { + return nil + } + var replicas any + if before != kubernetesReplicasUnset { + n, err := strconv.ParseInt(before, 10, 64) + if err != nil { + return fmt.Errorf("%s %s/%s: bad %s %q", gvr.Resource, ns, name, kubernetesReplicasBeforeAnnotation, before) + } + replicas = n + } + patch, err := json.Marshal(map[string]any{ + "metadata": map[string]any{"annotations": map[string]any{kubernetesReplicasBeforeAnnotation: nil}}, + "spec": map[string]any{"replicas": replicas}, + }) + if err != nil { + return err + } + _, err = r.Resource(gvr).Namespace(ns).Patch(ctx, name, types.MergePatchType, patch, metav1.PatchOptions{}) + return client.IgnoreNotFound(err) +} + +// resumeKubernetesCluster undoes stopKubernetesCluster from the records on +// the objects themselves, so it needs nothing from the RestoreJob and is safe +// to call more than once. +func (r *RestoreJobReconciler) resumeKubernetesCluster(ctx context.Context, ns, release string) error { + var errs []string + mds, err := r.listClusterMachineDeployments(ctx, ns, release) + if err != nil && !capiNotServed(err) { + return err + } + if err := r.resumeReplicas(ctx, kamajiControlPlaneGVR, ns, release); err != nil { + errs = append(errs, err.Error()) + } + for _, md := range mds { + if err := r.resumeReplicas(ctx, machineDeploymentGVR, ns, md.GetName()); err != nil { + errs = append(errs, err.Error()) + } + } + if err := r.resumeReplicas(ctx, deploymentGVR, ns, kubernetesAutoscalerName(release)); err != nil { + errs = append(errs, err.Error()) + } + for _, name := range kubernetesHelmReleaseNames(mds, release) { + hr, err := r.Resource(helmReleaseGVR).Namespace(ns).Get(ctx, name, metav1.GetOptions{}) + if err != nil { + if !apierrors.IsNotFound(err) { + errs = append(errs, err.Error()) + } + continue + } + if _, marked := hr.GetAnnotations()[kubernetesSuspendedByAnnotation]; !marked { + continue + } + patch := []byte(`{"metadata":{"annotations":{"` + kubernetesSuspendedByAnnotation + `":null}},"spec":{"suspend":false}}`) + if _, err := r.Resource(helmReleaseGVR).Namespace(ns).Patch(ctx, name, types.MergePatchType, patch, metav1.PatchOptions{}); err != nil && !apierrors.IsNotFound(err) { + errs = append(errs, err.Error()) + } + } + if len(errs) > 0 { + return fmt.Errorf("resume cluster: %s", strings.Join(errs, "; ")) + } + return nil +} + +// failKubernetesInPlace fails a live-cluster restore. The cluster is resumed +// unless the etcd prefix may have been partly replaced: bringing a control +// plane up on that would serve a mix of old and restored objects, so it is +// left stopped for a retry or a manual repair. +func (r *RestoreJobReconciler) failKubernetesInPlace(ctx context.Context, rj *backupsv1alpha1.RestoreJob, state *kubernetesRestoreState, msg string) (ctrl.Result, error) { + if apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondEtcdWriteFailed) { + return r.markRestoreJobFailed(ctx, rj, msg) + } + if err := r.resumeKubernetesCluster(ctx, rj.Namespace, state.Release); err != nil { + msg = fmt.Sprintf("%s; the cluster could not be fully resumed: %v", msg, err) + } + return r.markRestoreJobFailed(ctx, rj, msg) +} + +// cleanupKubernetesRestore runs when a RestoreJob is deleted. A live cluster +// it stopped is resumed, unless its etcd prefix may be partly replaced, which +// is left for the next restore to repair and reported as a Warning event. +func (r *RestoreJobReconciler) cleanupKubernetesRestore(ctx context.Context, rj *backupsv1alpha1.RestoreJob) { + state := kubernetesStateFromRestoreJob(rj) + if state == nil || state.Release == "" || rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseSucceeded { + return + } + partial := apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondEtcdWriteFailed) + // A restore Job still running may be rewriting the prefix right now; + // one that failed after it started writing left it partial. Either way + // the control plane must not come up on it. The Job is deleted so it + // stops writing; the garbage collector would only get to it later. + job := &batchv1.Job{} + switch err := r.Get(ctx, client.ObjectKey{Namespace: rj.Namespace, Name: jobNameForRestoreJob(rj)}, job); { + case err == nil && jobConditionState(job) != batchv1.JobComplete: + wrote, werr := r.kubernetesJobReachedWrite(ctx, rj, job.Name) + if werr != nil || wrote || jobConditionState(job) != batchv1.JobFailed { + partial = true + } + policy := metav1.DeletePropagationBackground + _ = r.Delete(ctx, job, &client.DeleteOptions{PropagationPolicy: &policy}) + case err != nil && !apierrors.IsNotFound(err): + partial = true + } + if partial { + if r.Recorder != nil { + r.Recorder.Eventf(rj, corev1.EventTypeWarning, "ClusterLeftStopped", + "%s/%s stays stopped: its etcd prefix may be partly restored; run a new restore from a Backup", rj.Namespace, state.Release) + } + return + } + if err := r.resumeKubernetesCluster(ctx, rj.Namespace, state.Release); err != nil && r.Recorder != nil { + r.Recorder.Eventf(rj, corev1.EventTypeWarning, "CleanupFailed", "resume %s/%s: %v", rj.Namespace, state.Release, err) + } +} + +// --------------------------------------------------------------------------- +// Steps shared by both modes +// --------------------------------------------------------------------------- + +// kubernetesRestoreData writes back the selected halves, etcd before volumes. +// next is true when both are done and the caller may move on; otherwise the +// returned result is the reconcile's. +func (r *RestoreJobReconciler) kubernetesRestoreData(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, strategy *strategyv1alpha1.Kubernetes, opts KubernetesRestoreOptions, state *kubernetesRestoreState) (ctrl.Result, bool, error) { + if opts.restoreEtcd() && !apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondEtcdRestored) { + ds, err := resolveKubernetesDatastore(ctx, r.Interface, rj.Namespace, backup.Spec.ApplicationRef.Name) + if err != nil { + res, rerr := r.kubernetesStepResult(ctx, rj, state, nil, fmt.Sprintf("resolve the cluster's datastore: %v", err), false) + return res, false, rerr + } + ds.SourcePrefix = backup.Spec.DriverMetadata[kubernetesDatastorePrefixMetaKey] + done, msg, err := r.runKubernetesJob(ctx, rj, backup, strategy, kubernetesModeRestore, jobNameForRestoreJob(rj), *ds) + if msg != "" && err == nil { + wrote, werr := r.kubernetesJobReachedWrite(ctx, rj, jobNameForRestoreJob(rj)) + if werr != nil { + return ctrl.Result{}, false, werr + } + if !wrote { + res, rerr := r.kubernetesStepResult(ctx, rj, state, nil, "etcd restore failed before it changed the prefix: "+msg, false) + return res, false, rerr + } + apimeta.SetStatusCondition(&rj.Status.Conditions, metav1.Condition{ + Type: kubernetesCondEtcdWriteFailed, + Status: metav1.ConditionTrue, + Reason: "RestoreJobFailed", + Message: msg, + }) + msg = fmt.Sprintf("etcd restore failed: %s. The prefix may be partly restored, so the control plane is left stopped. "+ + "A copy of the prefix from before the first restore of this backup that did not finish is kept at %s in the backup bucket; run a new RestoreJob to retry", + msg, kubernetesPreRestoreKey(backup.Spec.DriverMetadata[kubernetesObjectMetaKey])) + } + if err == nil && msg == "" && !done && !apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondEtcdJobStarted) { + started, serr := r.kubernetesJobStarted(ctx, rj.Namespace, jobNameForRestoreJob(rj)) + if serr != nil { + return ctrl.Result{}, false, serr + } + if started { + res, cerr := r.setKubernetesRestoreCondition(ctx, rj, kubernetesCondEtcdJobStarted, "the etcd restore Job is running") + return res, false, cerr + } + } + if err != nil || msg != "" || !done { + res, rerr := r.kubernetesStepResult(ctx, rj, state, err, msg, done) + return res, false, rerr + } + res, err := r.setKubernetesRestoreCondition(ctx, rj, kubernetesCondEtcdRestored, "etcd prefix restored") + return res, false, err + } + if opts.restoreVolumes() && !apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondVolumesRestored) { + done, msg, err := r.runKubernetesVeleroRestore(ctx, rj, backup, strategy, kubernetesPhaseVolumes, kubernetesReleaseName(backup.Spec.ApplicationRef.Name)) + if err != nil || msg != "" || !done { + res, rerr := r.kubernetesStepResult(ctx, rj, state, err, msg, done) + return res, false, rerr + } + res, err := r.setKubernetesRestoreCondition(ctx, rj, kubernetesCondVolumesRestored, "volumes restored") + return res, false, err + } + return ctrl.Result{}, true, nil +} + +// kubernetesStepResult turns a step's outcome into the reconcile result: an +// API error is retried, a failure message fails the restore (resuming a live +// cluster first where that is safe), and an unfinished step is polled. +func (r *RestoreJobReconciler) kubernetesStepResult(ctx context.Context, rj *backupsv1alpha1.RestoreJob, state *kubernetesRestoreState, err error, failure string, done bool) (ctrl.Result, error) { + switch { + case err != nil: + return ctrl.Result{}, err + case failure != "": + // A rebuild stops its new control plane for the etcd half the same + // way a live restore does, so both put back whatever they stopped. + if state != nil { + return r.failKubernetesInPlace(ctx, rj, state, failure) + } + return r.markRestoreJobFailed(ctx, rj, failure) + case !done: + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil + } + return ctrl.Result{}, nil +} + +// kubernetesJobReachedWrite reports whether any Pod of the restore Job may +// have changed the prefix: one running now, or one that terminated with the +// marker the script writes just before it deletes the prefix. A container +// that never ran (an image it could not pull) wrote nothing. When the Job's +// Pods are gone (its deadline deletes them), the answer is whether any of +// them was ever seen running. +func (r *RestoreJobReconciler) kubernetesJobReachedWrite(ctx context.Context, rj *backupsv1alpha1.RestoreJob, jobName string) (bool, error) { + pods := &corev1.PodList{} + if err := r.List(ctx, pods, client.InNamespace(rj.Namespace), client.MatchingLabels{"job-name": jobName}); err != nil { + return false, err + } + if len(pods.Items) == 0 { + return apimeta.IsStatusConditionTrue(rj.Status.Conditions, kubernetesCondEtcdJobStarted), nil + } + for _, p := range pods.Items { + for _, cs := range p.Status.ContainerStatuses { + switch { + case cs.State.Running != nil: + return true, nil + case cs.State.Terminated != nil: + if strings.TrimSpace(cs.State.Terminated.Message) == kubernetesJobWriting { + return true, nil + } + case cs.LastTerminationState.Terminated != nil: + if strings.TrimSpace(cs.LastTerminationState.Terminated.Message) == kubernetesJobWriting { + return true, nil + } + } + } + } + return false, nil +} + +// kubernetesJobStarted reports whether a container of the restore Job has +// run, which is what kubernetesJobReachedWrite falls back on once the Job's +// Pods are gone. +func (r *RestoreJobReconciler) kubernetesJobStarted(ctx context.Context, ns, jobName string) (bool, error) { + pods := &corev1.PodList{} + if err := r.List(ctx, pods, client.InNamespace(ns), client.MatchingLabels{"job-name": jobName}); err != nil { + return false, err + } + for _, p := range pods.Items { + for _, cs := range p.Status.ContainerStatuses { + if cs.State.Running != nil || cs.State.Terminated != nil || cs.LastTerminationState.Terminated != nil { + return true, nil + } + } + } + return false, nil +} + +// runKubernetesJob runs the strategy Job in the given mode under the given +// name against ds. It returns done once the Job completed, or a failure +// message once it failed. +func (r *RestoreJobReconciler) runKubernetesJob(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, strategy *strategyv1alpha1.Kubernetes, mode, name string, ds kubernetesDatastore) (bool, string, error) { + if ds.Prefix == "" || len(ds.Endpoints) == 0 { + return false, "the Backup records no etcd datastore prefix or endpoints", nil + } + rendered, err := renderKubernetesTemplate(strategy.Spec.Template, backup.Spec.ApplicationRef.Name, rj.Namespace, + mode, backup.Spec.DriverMetadata[kubernetesObjectMetaKey], ds, kubernetesStrategyParameters(backup)) + if err != nil { + return false, fmt.Sprintf("failed to template Kubernetes strategy: %v", err), nil + } + job, err := ensureBackupBatchJob(ctx, r.Client, r.Scheme, rj, rj.Namespace, name, + map[string]string{ + kubernetesLabelMode: mode, + backupsv1alpha1.OwningJobNameLabel: rj.Name, + backupsv1alpha1.OwningJobNamespaceLabel: rj.Namespace, + }, rendered) + if err != nil { + return false, "", err + } + switch jobConditionState(job) { + case batchv1.JobComplete: + return true, "", nil + case batchv1.JobFailed: + msg := jobFailureMessage(job) + if msg == "" { + msg = fmt.Sprintf("%s Job reported Failed", mode) + } + return false, msg, nil + } + return false, "", nil +} + +// runKubernetesVeleroRestore restores one phase of the Velero backup: +// control-plane Secrets, or the cluster's data DataVolumes and PVCs. It +// returns done once the Velero Restore completed. +func (r *RestoreJobReconciler) runKubernetesVeleroRestore(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, strategy *strategyv1alpha1.Kubernetes, phase, release string) (bool, string, error) { + veleroBackup := backup.Spec.DriverMetadata[veleroBackupNameMetadataKey] + if veleroBackup == "" { + return false, "the Backup records no Velero backup", nil + } + labels := map[string]string{ + backupsv1alpha1.OwningJobNameLabel: rj.Name, + backupsv1alpha1.OwningJobNamespaceLabel: rj.Namespace, + kubernetesRestorePhaseLabel: phase, + } + restores := &velerov1.RestoreList{} + if err := r.List(ctx, restores, client.InNamespace(veleroNamespace), client.MatchingLabels(labels)); err != nil { + return false, "", err + } + if len(restores.Items) == 0 { + spec, err := kubernetesVeleroRestoreSpec(strategy, backup, phase, release) + if err != nil { + return false, fmt.Sprintf("failed to build Velero Restore: %v", err), nil + } + spec.BackupName = veleroBackup + cm, err := r.kubernetesResourceModifiers(ctx, rj, phase, labels) + if err != nil { + return false, "", err + } + spec.ResourceModifier = &corev1.TypedLocalObjectReference{APIGroup: new(""), Kind: "ConfigMap", Name: cm.Name} + restore := &velerov1.Restore{ + ObjectMeta: metav1.ObjectMeta{ + GenerateName: fmt.Sprintf("%s.%s-%s-", rj.Namespace, rj.Name, phase), + Namespace: veleroNamespace, + Labels: labels, + }, + Spec: *spec, + } + if err := r.Create(ctx, restore); err != nil { + return false, "", err + } + if r.Recorder != nil { + r.Recorder.Event(rj, corev1.EventTypeNormal, "VeleroRestoreCreated", fmt.Sprintf("Created Velero Restore %s/%s", veleroNamespace, restore.Name)) + } + return false, "", nil + } + restore := &restores.Items[0] + switch restore.Status.Phase { + case velerov1.RestorePhaseCompleted: + return true, "", nil + case velerov1.RestorePhaseFailed, velerov1.RestorePhasePartiallyFailed, velerov1.RestorePhaseFailedValidation: + msg := fmt.Sprintf("Velero Restore %s/%s finished %s", restore.Namespace, restore.Name, restore.Status.Phase) + if restore.Status.FailureReason != "" { + msg += ": " + restore.Status.FailureReason + } + return false, msg, nil + } + return false, "", nil +} + +// kubernetesVeleroRestoreSpec starts from the strategy's restore spec and +// narrows it to one phase. The selectors mirror the ones the shipped backup +// spec uses, so a phase restores exactly its share of the backup. +func kubernetesVeleroRestoreSpec(strategy *strategyv1alpha1.Kubernetes, backup *backupsv1alpha1.Backup, phase, release string) (*velerov1.RestoreSpec, error) { + spec := &velerov1.RestoreSpec{} + if strategy.Spec.Velero.RestoreSpec != nil { + rendered, err := template.Template(strategy.Spec.Velero.RestoreSpec, map[string]any{ + "Application": map[string]any{ + "metadata": map[string]any{"name": backup.Spec.ApplicationRef.Name, "namespace": backup.Namespace}, + "kind": backup.Spec.ApplicationRef.Kind, + }, + "Parameters": kubernetesStrategyParameters(backup), + }) + if err != nil { + return nil, err + } + spec = rendered + } + spec.IncludedNamespaces = []string{backup.Namespace} + spec.LabelSelector = nil + switch phase { + case kubernetesPhaseSecrets: + spec.IncludedResources = []string{"secrets"} + spec.OrLabelSelectors = []*metav1.LabelSelector{ + {MatchLabels: map[string]string{kamajiNameLabel: release}}, + {MatchLabels: map[string]string{helmInstanceLabel: release}}, + } + case kubernetesPhaseVolumes: + spec.IncludedResources = []string{"persistentvolumeclaims", "datavolumes.cdi.kubevirt.io"} + spec.OrLabelSelectors = []*metav1.LabelSelector{ + {MatchLabels: map[string]string{clusterNameLabel: release}}, + } + spec.RestorePVs = new(true) + default: + return nil, fmt.Errorf("unknown restore phase %q", phase) + } + return spec, nil +} + +// kubernetesResourceModifiers creates the ConfigMap of resource modifiers for +// a phase. Restored Secrets drop their ownerReferences, which point at +// objects of the deleted cluster and would let the garbage collector delete +// them again; restored PVCs and DataVolumes let CDI adopt the restored claim. +func (r *RestoreJobReconciler) kubernetesResourceModifiers(ctx context.Context, rj *backupsv1alpha1.RestoreJob, phase string, labels map[string]string) (*corev1.ConfigMap, error) { + var rules []resourceModifierRule + switch phase { + case kubernetesPhaseSecrets: + patch, err := marshalPatchData(map[string]any{"metadata": map[string]any{"ownerReferences": nil}}) + if err != nil { + return nil, err + } + rules = append(rules, resourceModifierRule{ + Conditions: resourceModifierConditions{GroupResource: "secrets", ResourceNameRegex: ".*", Namespaces: []string{rj.Namespace}}, + MergePatches: []mergePatch{{PatchData: patch}}, + }) + case kubernetesPhaseVolumes: + patch, err := marshalPatchData(map[string]any{"metadata": map[string]any{"annotations": map[string]string{cdiAllowClaimAdoption: "true"}}}) + if err != nil { + return nil, err + } + for _, gr := range []string{"persistentvolumeclaims", "datavolumes.cdi.kubevirt.io"} { + rules = append(rules, resourceModifierRule{ + Conditions: resourceModifierConditions{GroupResource: gr, ResourceNameRegex: ".*", Namespaces: []string{rj.Namespace}}, + MergePatches: []mergePatch{{PatchData: patch}}, + }) + } + } + data, err := yaml.Marshal(resourceModifiers{Version: "v1", ResourceModifierRules: rules}) + if err != nil { + return nil, err + } + name := fmt.Sprintf("restore-modifiers-%s-%s-%s", rj.Namespace, rj.Name, phase) + if len(name) > 253 { + name = name[:253] + } + cm := &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: veleroNamespace, Labels: labels}, + Data: map[string]string{"resource-modifier-rules.yaml": string(data)}, + } + if err := r.Create(ctx, cm); err != nil { + if !apierrors.IsAlreadyExists(err) { + return nil, err + } + existing := &corev1.ConfigMap{} + if err := r.Get(ctx, types.NamespacedName{Namespace: veleroNamespace, Name: name}, existing); err != nil { + return nil, err + } + existing.Data = cm.Data + if err := r.Update(ctx, existing); err != nil { + return nil, err + } + return existing, nil + } + return cm, nil +} + +func (r *RestoreJobReconciler) setKubernetesRestoreCondition(ctx context.Context, rj *backupsv1alpha1.RestoreJob, condType, message string) (ctrl.Result, error) { + apimeta.SetStatusCondition(&rj.Status.Conditions, metav1.Condition{ + Type: condType, + Status: metav1.ConditionTrue, + Reason: "Done", + Message: message, + }) + if err := r.Status().Update(ctx, rj); err != nil { + return ctrl.Result{}, err + } + return ctrl.Result{RequeueAfter: kubernetesPollInterval}, nil +} + +func (r *RestoreJobReconciler) markKubernetesRestoreSucceeded(ctx context.Context, rj *backupsv1alpha1.RestoreJob, message string) (ctrl.Result, error) { + now := metav1.Now() + rj.Status.CompletedAt = &now + rj.Status.Phase = backupsv1alpha1.RestoreJobPhaseSucceeded + apimeta.SetStatusCondition(&rj.Status.Conditions, metav1.Condition{ + Type: "Ready", + Status: metav1.ConditionTrue, + Reason: "RestoreCompleted", + Message: message, + }) + return ctrl.Result{}, r.Status().Update(ctx, rj) +} diff --git a/internal/backupcontroller/restorejob_controller.go b/internal/backupcontroller/restorejob_controller.go index 1a61f0800..1a4dd7943 100644 --- a/internal/backupcontroller/restorejob_controller.go +++ b/internal/backupcontroller/restorejob_controller.go @@ -204,6 +204,8 @@ func (r *RestoreJobReconciler) Reconcile(ctx context.Context, req ctrl.Request) return r.reconcileRedisRestore(ctx, restoreJob, backup) case strategyv1alpha1.KafkaStrategyKind: return r.reconcileKafkaRestore(ctx, restoreJob, backup) + case strategyv1alpha1.KubernetesStrategyKind: + return r.reconcileKubernetesRestore(ctx, restoreJob, backup) default: return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("StrategyRef.Kind not supported: %s", backup.Spec.StrategyRef.Kind)) } @@ -382,6 +384,12 @@ func (r *RestoreJobReconciler) cleanupOnDelete(ctx context.Context, restoreJob * case strategyv1alpha1.VeleroStrategyKind: r.cleanupVeleroRestore(ctx, restoreJob) + case strategyv1alpha1.KubernetesStrategyKind: + // Its Velero Restores are labelled like the Velero driver's, so the + // same cleanup reaps them; a live cluster it stopped is resumed. + r.cleanupVeleroRestore(ctx, restoreJob) + r.cleanupKubernetesRestore(ctx, restoreJob) + case strategyv1alpha1.CNPGStrategyKind, strategyv1alpha1.JobStrategyKind, strategyv1alpha1.AltinityStrategyKind, strategyv1alpha1.MariaDBStrategyKind, strategyv1alpha1.MongoDBStrategyKind, strategyv1alpha1.FoundationDBStrategyKind, strategyv1alpha1.EtcdStrategyKind, strategyv1alpha1.RabbitmqStrategyKind, strategyv1alpha1.RedisStrategyKind, strategyv1alpha1.KafkaStrategyKind: // Nothing to clean up: these drivers don't materialise namespaced // artifacts that outlive the RestoreJob. (Etcd: the operator-side diff --git a/packages/system/backupstrategy-controller/Makefile b/packages/system/backupstrategy-controller/Makefile index 94b1931d9..aabc11d17 100644 --- a/packages/system/backupstrategy-controller/Makefile +++ b/packages/system/backupstrategy-controller/Makefile @@ -6,9 +6,9 @@ include ../../../hack/package.mk # PHONY so a stray file/dir named `image` cannot mark the aggregate up to date # and skip both builds. -.PHONY: image image-backupstrategy-controller image-redis-backup image-backup-http-client +.PHONY: image image-backupstrategy-controller image-redis-backup image-backup-http-client image-kubernetes-backup -image: image-backupstrategy-controller image-redis-backup image-backup-http-client +image: image-backupstrategy-controller image-redis-backup image-backup-http-client image-kubernetes-backup image-backupstrategy-controller: docker buildx build -f images/backupstrategy-controller/Dockerfile ../../.. \ @@ -46,5 +46,16 @@ image-backup-http-client: image-redis-backup yq -i '.backupStrategyController.chBackupClientImage = strenv(IMAGE) | .backupStrategyController.rabbitmqBackupClientImage = strenv(IMAGE)' values.yaml rm -f images/backup-http-client.json +# Chained after image-backup-http-client: one `yq -i` writer at a time. +image-kubernetes-backup: image-backup-http-client + docker buildx build -f images/kubernetes-backup/Dockerfile images/kubernetes-backup \ + $(call image-tags,kubernetes-backup,v$(COZYSTACK_VERSION)) \ + $(call cache-args,kubernetes-backup) \ + --metadata-file images/kubernetes-backup.json \ + $(BUILDX_ARGS) + IMAGE="$(REGISTRY)/kubernetes-backup:$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/kubernetes-backup.json -o json -r)" \ + yq -i '.backupStrategyController.kubernetesBackupClientImage = strenv(IMAGE)' values.yaml + rm -f images/kubernetes-backup.json + test: helm unittest . diff --git a/packages/system/backupstrategy-controller/definitions/strategy.backups.cozystack.io_kubernetes.yaml b/packages/system/backupstrategy-controller/definitions/strategy.backups.cozystack.io_kubernetes.yaml new file mode 100644 index 000000000..0b681c5be --- /dev/null +++ b/packages/system/backupstrategy-controller/definitions/strategy.backups.cozystack.io_kubernetes.yaml @@ -0,0 +1,147 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.16.4 + name: kubernetes.strategy.backups.cozystack.io +spec: + group: strategy.backups.cozystack.io + names: + kind: Kubernetes + listKind: KubernetesList + plural: kubernetes + singular: kubernetes + scope: Cluster + versions: + - name: v1alpha1 + schema: + openAPIV3Schema: + description: |- + Kubernetes defines a backup strategy for a tenant Kubernetes application. + A tenant cluster keeps its state in two places the platform can reach: the + cluster's key prefix in the tenant's shared etcd DataStore, and host-side + objects (control-plane PKI Secrets, the application HelmReleases, and the + DataVolumes that back the tenant's PersistentVolumes). The strategy captures + the first with a one-shot Job and the second with Velero. The two are taken + one after another, so a backup is crash-consistent at best. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: KubernetesSpec specifies the desired behavior of a Kubernetes + backup strategy. + properties: + certificateWarningThreshold: + description: |- + CertificateWarningThreshold is how close to expiry a control-plane + certificate may be before a backup records a warning. Defaults to 720h. + type: string + template: + description: |- + Template holds a PodTemplateSpec for the Job that dumps the cluster's + key prefix from its etcd DataStore to object storage, writes it back on + restore, and deletes the object on cleanup. The Job runs in the + application namespace. Helm-like Go templates are supported in every + string field and are rendered against: + - `.Release.Name` and `.Release.Namespace` — the application's name and + namespace; `.Release.Prefix` is `kubernetes-`. + - `.Mode` — `"backup"`, `"restore"` or `"cleanup"`. + - `.ObjectKey` — the object-storage key of this backup's dump. + - `.Datastore.Prefix` — the cluster's key prefix, without slashes. + - `.Datastore.Endpoints` — comma-separated etcd client URLs. + - `.Datastore.CertificateSecret` — the Secret holding the cluster's + etcd client certificate (`ca.crt`, `server.crt`, `server.key`). + - `.Parameters` — the resolved BackupClass strategy parameters. + + Stored schemaless for the same CRD-size reason as redis_types.go. + x-kubernetes-preserve-unknown-fields: true + velero: + description: |- + Velero holds the Velero Backup and Restore specs for the host-side + objects and volumes. Rendered against `.Application` and `.Parameters`, + like the Velero strategy. + x-kubernetes-preserve-unknown-fields: true + required: + - template + - velero + type: object + status: + properties: + conditions: + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/packages/system/backupstrategy-controller/images/kubernetes-backup/Dockerfile b/packages/system/backupstrategy-controller/images/kubernetes-backup/Dockerfile new file mode 100644 index 000000000..b897e51be --- /dev/null +++ b/packages/system/backupstrategy-controller/images/kubernetes-backup/Dockerfile @@ -0,0 +1,14 @@ +# kubernetes-backup runs the Kubernetes strategy Pod: it dumps a tenant +# cluster's key prefix from its etcd DataStore with etcdctl, moves the dump to +# S3 with curl --aws-sigv4, and writes it back with etcdctl on restore. etcdctl +# matches the etcd the platform deploys. Everything is installed at build time +# so the Pods never run `apk add` and work on air-gapped clusters. +FROM quay.io/coreos/etcd:v3.6.11@sha256:6ae247c7666ceec554c51ba1f9bc8dd2212f975370dbd65710c9ca0e36ae1fff AS etcd + +FROM alpine:3.24@sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6 + +RUN apk add --no-cache curl jq ca-certificates coreutils + +COPY --from=etcd /usr/local/bin/etcdctl /usr/local/bin/etcdctl + +USER 65532:65532 diff --git a/packages/system/backupstrategy-controller/templates/backupclass-default.yaml b/packages/system/backupstrategy-controller/templates/backupclass-default.yaml index 3f3b271e3..347c704d1 100644 --- a/packages/system/backupstrategy-controller/templates/backupclass-default.yaml +++ b/packages/system/backupstrategy-controller/templates/backupclass-default.yaml @@ -78,6 +78,13 @@ spec: apiGroup: strategy.backups.cozystack.io kind: Redis name: cozy-default-redis + - application: + apiGroup: apps.cozystack.io + kind: Kubernetes + strategyRef: + apiGroup: strategy.backups.cozystack.io + kind: Kubernetes + name: cozy-default-kubernetes - application: apiGroup: apps.cozystack.io kind: Kafka diff --git a/packages/system/backupstrategy-controller/templates/rbac.yaml b/packages/system/backupstrategy-controller/templates/rbac.yaml index 8d90980a8..735e16308 100644 --- a/packages/system/backupstrategy-controller/templates/rbac.yaml +++ b/packages/system/backupstrategy-controller/templates/rbac.yaml @@ -62,7 +62,25 @@ rules: # through the dynamic client, so no list/watch) - apiGroups: ["cdi.kubevirt.io"] resources: ["datavolumes"] - verbs: ["get", "delete"] + verbs: ["get", "list", "delete"] +# Kubernetes strategy: read where a tenant cluster keeps its etcd prefix, and +# stop and resume its control plane, worker pools and cluster-autoscaler around +# an in-place restore. +- apiGroups: ["kamaji.clastix.io"] + resources: ["tenantcontrolplanes", "datastores"] + verbs: ["get"] +- apiGroups: ["controlplane.cluster.x-k8s.io"] + resources: ["kamajicontrolplanes"] + verbs: ["get", "patch"] +- apiGroups: ["cluster.x-k8s.io"] + resources: ["machinedeployments"] + verbs: ["get", "list", "patch"] +- apiGroups: ["cluster.x-k8s.io"] + resources: ["machines"] + verbs: ["list", "patch"] +- apiGroups: ["apps"] + resources: ["deployments"] + verbs: ["get", "patch"] # HelmReleases: pre-restore suspends HRs; post-restore rename creates new HR # and deletes old. patch additionally lets the DefaultObjectsGate stamp # reconcile.fluxcd.io/forceAt + requestedAt on THIS chart's own release to diff --git a/packages/system/backupstrategy-controller/templates/strategy-kubernetes-default.yaml b/packages/system/backupstrategy-controller/templates/strategy-kubernetes-default.yaml new file mode 100644 index 000000000..3fab192ad --- /dev/null +++ b/packages/system/backupstrategy-controller/templates/strategy-kubernetes-default.yaml @@ -0,0 +1,304 @@ +{{/* + Two template layers, as in strategy-redis-default.yaml: Helm renders this + file, then the strategy controller renders the resulting CR per BackupJob. + Literal "{{ .Foo }}" for the second layer goes through printf. + + Gated on both a Velero BSL and a resolved $bucketName: the Velero half of + the backup goes to the BSL and the etcd dump to the bucket. Also gated on + its CRD being established, for the upgrade of an existing install that + first ships this kind: see strategy-mongodb-default.yaml. +*/}} +{{- $bucketName := include "backupstrategy-controller.bucketName" . -}} +{{- $crdEstablished := include "backupstrategy-controller.crdEstablished" (dict "root" . "crd" "kubernetes.strategy.backups.cozystack.io") -}} +{{- if and .Values.velero.bslEnabled $bucketName $crdEstablished -}} +apiVersion: strategy.backups.cozystack.io/v1alpha1 +kind: Kubernetes +metadata: + name: cozy-default-kubernetes + annotations: + backups.cozystack.io/endpoint: {{ include "backupstrategy-controller.endpoint" . | quote }} +spec: + template: + spec: + restartPolicy: Never + activeDeadlineSeconds: {{ int .Values.backupStrategyController.kubernetesBackupActiveDeadlineSeconds }} + volumes: + {{- /* + The cluster's own etcd client certificate, the one its kube-apiserver + uses. Optional so a missing Secret fails the script with a clear cause + instead of wedging the Pod on FailedMount. + */}} + - name: etcd-client + secret: + secretName: {{ printf "{{ .Datastore.CertificateSecret }}" | quote }} + optional: true + {{- if .Values.backupStorage.endpointCASecretName }} + - name: s3-ca + secret: + secretName: {{ .Values.backupStorage.endpointCASecretName | quote }} + optional: true + {{- end }} + containers: + - name: kubernetes-backup + image: "{{ .Values.backupStrategyController.kubernetesBackupClientImage }}" + imagePullPolicy: IfNotPresent + volumeMounts: + - name: etcd-client + mountPath: /etc/etcd-client + readOnly: true + {{- if .Values.backupStorage.endpointCASecretName }} + - name: s3-ca + mountPath: /etc/s3-ca + readOnly: true + {{- end }} + env: + - name: ETCDCTL_ENDPOINTS + value: {{ printf "{{ .Datastore.Endpoints }}" | quote }} + - name: S3_ENDPOINT + valueFrom: + secretKeyRef: + name: cozy-backups-creds + key: endpoint + optional: true + - name: S3_BUCKET + valueFrom: + secretKeyRef: + name: cozy-backups-creds + key: bucketName + optional: true + - name: S3_REGION + valueFrom: + secretKeyRef: + name: cozy-backups-creds + key: region + optional: true + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: + name: cozy-backups-creds + key: AWS_ACCESS_KEY_ID + optional: true + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: + name: cozy-backups-creds + key: AWS_SECRET_ACCESS_KEY + optional: true + - name: S3_INSECURE_SKIP_VERIFY + value: {{ .Values.backupStorage.insecureSkipTLSVerify | quote }} + command: ["/bin/sh", "-c"] + args: + - | + set -euo pipefail + : "${S3_ENDPOINT:?missing endpoint in cozy-backups-creds}" + : "${S3_BUCKET:?missing bucketName in cozy-backups-creds}" + : "${S3_REGION:?missing region in cozy-backups-creds}" + : "${AWS_ACCESS_KEY_ID:?missing AWS_ACCESS_KEY_ID in cozy-backups-creds}" + : "${AWS_SECRET_ACCESS_KEY:?missing AWS_SECRET_ACCESS_KEY in cozy-backups-creds}" + + MODE="{{ printf "{{ .Mode }}" }}" + KEY="{{ printf "{{ .ObjectKey }}" }}" + SCHEMA="{{ printf "{{ .Datastore.Prefix }}" }}" + SOURCE_SCHEMA="{{ printf "{{ .Datastore.SourcePrefix }}" }}" + + case "${S3_ENDPOINT}" in + http://*|https://*) BASE="${S3_ENDPOINT}" ;; + *) BASE="https://${S3_ENDPOINT}" ;; + esac + TLS="" + case "${BASE}" in + https://*) + if [ -f /etc/s3-ca/ca.crt ]; then TLS="--cacert /etc/s3-ca/ca.crt"; + elif [ -f /etc/s3-ca/tls.crt ]; then TLS="--cacert /etc/s3-ca/tls.crt"; + elif [ "${S3_INSECURE_SKIP_VERIFY:-false}" = "true" ]; then TLS="-k"; + fi + ;; + esac + s3() { + curl -fsS ${TLS} \ + --aws-sigv4 "aws:amz:${S3_REGION}:s3" \ + --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" "$@" + } + OBJ="${BASE}/${S3_BUCKET}/${KEY}" + {{- /* + Where a restore keeps the prefix it replaces. One per backup, so + cleanup removes it with the dump. A restore writes it only when it + is absent and deletes it on success, so after a failed restore it + holds the prefix from before the first failed attempt and a retry + cannot overwrite it with a partial one. + */}} + PRE_KEY="${KEY}.pre-restore" + PRE_OBJ="${BASE}/${S3_BUCKET}/${PRE_KEY}" + + if [ "${MODE}" = "cleanup" ]; then + for o in "${OBJ}" "${PRE_OBJ}"; do + echo "deleting ${o#"${BASE}/"}" + code=$(curl -sS -o /dev/null -w '%{http_code}' ${TLS} \ + --aws-sigv4 "aws:amz:${S3_REGION}:s3" \ + --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \ + -X DELETE "${o}") + case "${code}" in + 2??|404) echo "deleted (HTTP ${code})" ;; + *) echo "delete failed: HTTP ${code}" >&2; exit 1 ;; + esac + done + exit 0 + fi + + if [ "${MODE}" = "verify" ]; then + s3 -I "${OBJ}" > /dev/null + echo "s3://${S3_BUCKET}/${KEY} exists" + exit 0 + fi + + {{- /* + The prefix comes from the TenantControlPlane: Kamaji's schema is the + TenantControlPlane's UID, or _ for one created by + an older Kamaji. Refuse anything else: an empty or slash-bearing + value would turn the restore's `del --prefix` into a wipe of every + cluster in the shared etcd. + */}} + for s in "${SCHEMA}" "${SOURCE_SCHEMA}"; do + case "${s}" in + ""|*[!a-z0-9_-]*) echo "refusing unexpected datastore prefix '${s}'" >&2; exit 1 ;; + esac + done + PREFIX="/${SCHEMA}/" + {{- /* + The prefix the dump was taken under. A rebuilt cluster has a new + TenantControlPlane and so a new schema; its keys are written under + the new prefix. + */}} + SOURCE_PREFIX="/${SOURCE_SCHEMA}/" + for f in ca.crt server.crt server.key; do + [ -f "/etc/etcd-client/${f}" ] || { echo "etcd client certificate is missing /etc/etcd-client/${f}" >&2; exit 1; } + done + export ETCDCTL_CACERT=/etc/etcd-client/ca.crt + export ETCDCTL_CERT=/etc/etcd-client/server.crt + export ETCDCTL_KEY=/etc/etcd-client/server.key + + if [ "${MODE}" = "backup" ]; then + etcdctl get "${PREFIX}" --prefix -w json > /tmp/dump.json + count=$(jq '.kvs // [] | length' /tmp/dump.json) + if [ "${count}" -eq 0 ]; then + echo "etcd prefix ${PREFIX} is empty" >&2 + exit 1 + fi + echo "dumped ${count} keys under ${PREFIX}; uploading s3://${S3_BUCKET}/${KEY}" + s3 -X PUT --upload-file /tmp/dump.json "${OBJ}" + elif [ "${MODE}" = "restore" ]; then + {{- /* + Everything that can fail without touching etcd runs before the + prefix is deleted: the download, the check that every key belongs + to this cluster, and a copy of the prefix as it is now, which is + what an operator restores from if a failure comes after the delete. + */}} + echo "downloading s3://${S3_BUCKET}/${KEY}" + s3 -o /tmp/dump.json "${OBJ}" + jq -e '.kvs | length > 0' /tmp/dump.json >/dev/null || { echo "dump holds no keys" >&2; exit 1; } + jq -e --arg p "${SOURCE_PREFIX}" 'all(.kvs[]; .key | @base64d | startswith($p))' /tmp/dump.json >/dev/null \ + || { echo "dump holds keys outside ${SOURCE_PREFIX}" >&2; exit 1; } + {{- /* + etcdctl put cannot write an empty value, so a dump holding one + must stop here rather than halfway through the write. + */}} + jq -e 'all(.kvs[]; (.value // "") != "")' /tmp/dump.json >/dev/null \ + || { echo "dump holds a key with an empty value, which etcdctl cannot write" >&2; exit 1; } + jq -r '.kvs[] | [.key, .value] | @tsv' /tmp/dump.json > /tmp/kvs.tsv + {{- /* + Only a definite answer decides about the copy: an existing one is + kept, a missing one is taken, and any other reply (a 5xx, a + timeout) stops the restore while the prefix is still intact, + rather than overwriting the copy an earlier failed restore left. + */}} + code=$(curl -sS -o /dev/null -w '%{http_code}' -I ${TLS} \ + --aws-sigv4 "aws:amz:${S3_REGION}:s3" \ + --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \ + "${PRE_OBJ}") || code=000 + case "${code}" in + 200) echo "keeping s3://${S3_BUCKET}/${PRE_KEY} from an earlier failed restore" ;; + 404) + etcdctl get "${PREFIX}" --prefix -w json > /tmp/before.json + echo "saving the current prefix to s3://${S3_BUCKET}/${PRE_KEY}" + s3 -X PUT --upload-file /tmp/before.json "${PRE_OBJ}" + ;; + *) echo "cannot tell whether s3://${S3_BUCKET}/${PRE_KEY} exists: HTTP ${code}" >&2; exit 1 ;; + esac + {{- /* + Delete first so keys created after the backup do not survive the + restore. Events and master leases are bound to etcd leases that + did not survive; written back without one they would never expire, + so they are skipped. They are matched as whole top-level resources + under the prefix, not as a path segment anywhere, which would also + drop every object in a namespace named "events". + */}} + {{- /* + The termination message tells the driver that this Pod got as far + as changing the prefix: only then is the control plane left + stopped after a failure. + */}} + echo writing > /dev/termination-log + etcdctl del "${PREFIX}" --prefix + n=0 + while IFS="$(printf '\t')" read -r k v; do + key=$(printf '%s' "${k}" | base64 -d) + rel="${key#"${SOURCE_PREFIX}"}" + case "${rel}" in events/*|masterleases/*) continue ;; esac + printf '%s' "${v}" | base64 -d > /tmp/value + etcdctl put "${PREFIX}${rel}" < /tmp/value > /dev/null + n=$((n + 1)) + done < /tmp/kvs.tsv + echo "restored ${n} keys under ${PREFIX}" + code=$(curl -sS -o /dev/null -w '%{http_code}' ${TLS} \ + --aws-sigv4 "aws:amz:${S3_REGION}:s3" \ + --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \ + -X DELETE "${PRE_OBJ}") || code=000 + case "${code}" in + 2??|404) echo "dropped s3://${S3_BUCKET}/${PRE_KEY}" ;; + *) echo "WARNING: could not drop s3://${S3_BUCKET}/${PRE_KEY} (HTTP ${code}); a later restore would keep it as the pre-restore copy, so delete it by hand" >&2 ;; + esac + else + echo "unknown mode: ${MODE}" >&2 + exit 1 + fi + resources: + {{- toYaml .Values.backupStrategyController.kubernetesBackupResources | nindent 12 }} + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault + velero: + {{- /* + Secrets (control-plane and Talos PKI, by the Kamaji and Helm instance + labels) and the DataVolumes and PVCs backing the cluster's + PersistentVolumes (the CSI driver stamps the cluster-name label on them; + worker system disks do not carry it). The HelmReleases are recorded by the + driver on the Backup instead: selecting them by application label would + also select worker disks the lineage webhook labels the same way. The + driver sets the restore's resources and selectors per phase. + */}} + spec: + includedNamespaces: + - {{ printf "{{ .Application.metadata.namespace }}" | quote }} + orLabelSelectors: + - matchLabels: + kamaji.clastix.io/name: {{ printf "kubernetes-{{ .Application.metadata.name }}" | quote }} + - matchLabels: + app.kubernetes.io/instance: {{ printf "kubernetes-{{ .Application.metadata.name }}" | quote }} + - matchLabels: + cluster.x-k8s.io/cluster-name: {{ printf "kubernetes-{{ .Application.metadata.name }}" | quote }} + includedResources: + - secrets + - persistentvolumeclaims + - datavolumes.cdi.kubevirt.io + includeClusterResources: false + storageLocation: cozy-default + snapshotVolumes: true + snapshotMoveData: true + ttl: 720h0m0s + itemOperationTimeout: 24h0m0s + restoreSpec: + includeClusterResources: false +{{- end -}} diff --git a/packages/system/backupstrategy-controller/tests/rendering_test.yaml b/packages/system/backupstrategy-controller/tests/rendering_test.yaml index 3db6adb14..941444070 100644 --- a/packages/system/backupstrategy-controller/tests/rendering_test.yaml +++ b/packages/system/backupstrategy-controller/tests/rendering_test.yaml @@ -22,6 +22,7 @@ templates: - templates/strategy-foundationdb-default.yaml - templates/strategy-velero-vminstance-default.yaml - templates/strategy-velero-vmdisk-default.yaml + - templates/strategy-kubernetes-default.yaml release: name: t @@ -88,11 +89,14 @@ tests: - hasDocuments: count: 0 template: templates/strategy-velero-vmdisk-default.yaml + - hasDocuments: + count: 0 + template: templates/strategy-kubernetes-default.yaml - hasDocuments: count: 0 template: templates/velero-bsl.yaml - - it: "cozy-default BackupClass is the Day-1 tenant interface: renders unconditionally with all eleven routes (FoundationDB intentionally unbound)" + - it: "cozy-default BackupClass is the Day-1 tenant interface: renders unconditionally with all twelve routes (FoundationDB intentionally unbound)" asserts: - hasDocuments: count: 1 @@ -103,7 +107,18 @@ tests: template: templates/backupclass-default.yaml - lengthEqual: path: spec.strategies - count: 11 + count: 12 + template: templates/backupclass-default.yaml + - contains: + path: spec.strategies + content: + application: + apiGroup: apps.cozystack.io + kind: Kubernetes + strategyRef: + apiGroup: strategy.backups.cozystack.io + kind: Kubernetes + name: cozy-default-kubernetes template: templates/backupclass-default.yaml - contains: path: spec.strategies @@ -240,6 +255,18 @@ tests: - hasDocuments: count: 1 template: templates/strategy-velero-vmdisk-default.yaml + - hasDocuments: + count: 1 + template: templates/strategy-kubernetes-default.yaml + # The driver renders these per BackupJob; Helm must leave them literal. + - equal: + path: spec.template.spec.volumes[0].secret.secretName + value: "{{ .Datastore.CertificateSecret }}" + template: templates/strategy-kubernetes-default.yaml + - equal: + path: spec.velero.spec.orLabelSelectors[2].matchLabels["cluster.x-k8s.io/cluster-name"] + value: "kubernetes-{{ .Application.metadata.name }}" + template: templates/strategy-kubernetes-default.yaml - hasDocuments: count: 1 template: templates/velero-bsl.yaml @@ -347,6 +374,7 @@ tests: - {hasDocuments: {count: 0}, template: templates/strategy-foundationdb-default.yaml} - {hasDocuments: {count: 0}, template: templates/strategy-velero-vminstance-default.yaml} - {hasDocuments: {count: 0}, template: templates/strategy-velero-vmdisk-default.yaml} + - {hasDocuments: {count: 0}, template: templates/strategy-kubernetes-default.yaml} # The ungated BackupClass keeps rendering, which is what makes the counts # above an assertion about the CRD gate rather than about a chart-wide # render failure. @@ -432,6 +460,7 @@ tests: - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: etcds.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: foundationdbs.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: kafkas.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} + - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: kubernetes.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: mariadbs.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: mongodbs.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: rabbitmqs.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} @@ -449,6 +478,7 @@ tests: - {hasDocuments: {count: 0}, template: templates/strategy-redis-default.yaml} - {hasDocuments: {count: 0}, template: templates/strategy-velero-vminstance-default.yaml} - {hasDocuments: {count: 0}, template: templates/strategy-velero-vmdisk-default.yaml} + - {hasDocuments: {count: 0}, template: templates/strategy-kubernetes-default.yaml} - it: "with every CRD Established and the BucketClaim resolved, every Strategy CR renders against the COSI bucket name" # The positive half of the case above, through the same live lookups and no @@ -471,6 +501,7 @@ tests: - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: etcds.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: foundationdbs.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: kafkas.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} + - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: kubernetes.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: mariadbs.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: mongodbs.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} - {apiVersion: apiextensions.k8s.io/v1, kind: CustomResourceDefinition, metadata: {name: rabbitmqs.strategy.backups.cozystack.io}, status: {conditions: [{type: Established, status: "True"}]}} @@ -489,6 +520,7 @@ tests: - {hasDocuments: {count: 1}, template: templates/strategy-redis-default.yaml} - {hasDocuments: {count: 1}, template: templates/strategy-velero-vminstance-default.yaml} - {hasDocuments: {count: 1}, template: templates/strategy-velero-vmdisk-default.yaml} + - {hasDocuments: {count: 1}, template: templates/strategy-kubernetes-default.yaml} - it: "velero.bslEnabled=false: Velero strategies and BSL drop out while other strategies keep rendering" set: diff --git a/packages/system/backupstrategy-controller/tests/restore_rbac_test.yaml b/packages/system/backupstrategy-controller/tests/restore_rbac_test.yaml index b13b88173..fa3406d58 100644 --- a/packages/system/backupstrategy-controller/tests/restore_rbac_test.yaml +++ b/packages/system/backupstrategy-controller/tests/restore_rbac_test.yaml @@ -13,11 +13,11 @@ release: namespace: cozy-backup-controller tests: - - it: "ClusterRole can read and delete DataVolumes" + - it: "ClusterRole can read, list and delete DataVolumes" asserts: - contains: path: rules content: apiGroups: ["cdi.kubevirt.io"] resources: ["datavolumes"] - verbs: ["get", "delete"] + verbs: ["get", "list", "delete"] diff --git a/packages/system/backupstrategy-controller/values.yaml b/packages/system/backupstrategy-controller/values.yaml index 590d7aca2..121b682d8 100644 --- a/packages/system/backupstrategy-controller/values.yaml +++ b/packages/system/backupstrategy-controller/values.yaml @@ -37,6 +37,24 @@ backupStrategyController: cpu: "1" memory: 1Gi ephemeral-storage: 4Gi + # kubernetesBackupClientImage is rendered into the Kubernetes strategy Pod, + # which dumps a tenant cluster's etcd prefix with etcdctl and moves it to S3 + # with curl. Built from images/kubernetes-backup/; `make image` re-digests it. + kubernetesBackupClientImage: "ghcr.io/cozystack/cozystack/kubernetes-backup:v1.6.0" + # kubernetesBackupActiveDeadlineSeconds bounds one etcd dump/restore Job. + kubernetesBackupActiveDeadlineSeconds: 3600 + # kubernetesBackupResources sizes the Kubernetes strategy Pod. The dump is held + # in ephemeral storage while it moves and jq reads it into memory, so a + # cluster with a large etcd prefix needs more of both. + kubernetesBackupResources: + requests: + cpu: 10m + memory: 64Mi + ephemeral-storage: 256Mi + limits: + cpu: "1" + memory: 1Gi + ephemeral-storage: 2Gi # The Kafka metadata strategy carries no client image here: unlike the Altinity # and RabbitMQ drivers, which need only curl (plus jq for Altinity), its driver # runs the Kafka Admin-API CLI (kafka-topics / kafka-configs), which only the From d48897cf8224661affbf6be7e8e5ce7bee63dba5 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Tue, 10 Nov 2026 10:28:28 +0100 Subject: [PATCH 403/518] feat(virtual-machine): let each disk set its own boot order The boot order of a VM's disks was the order of the disks list, so the only way to boot from installer or rescue media, such as an ISO attached as a CD-ROM, was to move that media to the front of the list. Callers that treat the first disk as the system disk then see the ISO in its place. An optional bootOrder per disk decides boot priority independently of the list. Disks that leave it unset keep their positional order, so existing VMs render unchanged. Duplicate effective boot orders are refused at render time with the names of both disks, since KubeVirt would otherwise deny the VirtualMachine with a less specific error. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- api/apps/v1alpha1/vminstance/types.go | 3 + .../vminstance/zz_generated.deepcopy.go | 9 +- packages/apps/vm-instance/README.md | 1 + packages/apps/vm-instance/templates/vm.yaml | 11 +- .../tests/disk_boot_order_test.yaml | 128 ++++++++++++++++++ packages/apps/vm-instance/values.schema.json | 5 + packages/apps/vm-instance/values.yaml | 8 ++ .../vm-instance-rd/cozyrds/vm-instance.yaml | 2 +- 8 files changed, 164 insertions(+), 3 deletions(-) create mode 100644 packages/apps/vm-instance/tests/disk_boot_order_test.yaml diff --git a/api/apps/v1alpha1/vminstance/types.go b/api/apps/v1alpha1/vminstance/types.go index 95c800aae..e457f4cac 100644 --- a/api/apps/v1alpha1/vminstance/types.go +++ b/api/apps/v1alpha1/vminstance/types.go @@ -74,6 +74,9 @@ type ConfigSpec struct { } type Disk struct { + // Boot priority of the disk, 1 being tried first. Unset falls back to the disk's position in `disks` (the first disk gets 1), so set it to boot from a disk other than the first, such as an attached installer or rescue ISO, without reordering the list. Effective boot orders must be unique across disks: when one disk is given a value that another takes from its position, set both. + // +kubebuilder:validation:Minimum=1 + BootOrder *int `json:"bootOrder,omitempty"` // Disk bus type (e.g. "sata"). Bus string `json:"bus,omitempty"` // Disk name. diff --git a/api/apps/v1alpha1/vminstance/zz_generated.deepcopy.go b/api/apps/v1alpha1/vminstance/zz_generated.deepcopy.go index 060281042..84f7315ac 100644 --- a/api/apps/v1alpha1/vminstance/zz_generated.deepcopy.go +++ b/api/apps/v1alpha1/vminstance/zz_generated.deepcopy.go @@ -61,7 +61,9 @@ func (in *ConfigSpec) DeepCopyInto(out *ConfigSpec) { if in.Disks != nil { in, out := &in.Disks, &out.Disks *out = make([]Disk, len(*in)) - copy(*out, *in) + for i := range *in { + (*in)[i].DeepCopyInto(&(*out)[i]) + } } if in.Networks != nil { in, out := &in.Networks, &out.Networks @@ -100,6 +102,11 @@ func (in *ConfigSpec) DeepCopy() *ConfigSpec { // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *Disk) DeepCopyInto(out *Disk) { *out = *in + if in.BootOrder != nil { + in, out := &in.BootOrder, &out.BootOrder + *out = new(int) + **out = **in + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Disk. diff --git a/packages/apps/vm-instance/README.md b/packages/apps/vm-instance/README.md index 2365ca9c1..f2a0d001d 100644 --- a/packages/apps/vm-instance/README.md +++ b/packages/apps/vm-instance/README.md @@ -48,6 +48,7 @@ virtctl ssh @ | `disks` | List of disks to attach. | `[]object` | `[]` | | `disks[i].name` | Disk name. | `string` | `""` | | `disks[i].bus` | Disk bus type (e.g. "sata"). | `string` | `""` | +| `disks[i].bootOrder` | Boot priority of the disk, 1 being tried first. Unset falls back to the disk's position in `disks` (the first disk gets 1), so set it to boot from a disk other than the first, such as an attached installer or rescue ISO, without reordering the list. Effective boot orders must be unique across disks: when one disk is given a value that another takes from its position, set both. | `*int` | `null` | | `networks` | Networks to attach the VM to. | `[]object` | `[]` | | `networks[i].name` | Network attachment name. | `string` | `""` | | `subnets` | Deprecated: use networks instead. | `[]object` | `[]` | diff --git a/packages/apps/vm-instance/templates/vm.yaml b/packages/apps/vm-instance/templates/vm.yaml index 8153adf8f..0d7845621 100644 --- a/packages/apps/vm-instance/templates/vm.yaml +++ b/packages/apps/vm-instance/templates/vm.yaml @@ -92,7 +92,16 @@ spec: {{- end }} {{- end }} disks: + {{- /* KubeVirt rejects two devices sharing a bootOrder, so a clash is + reported here with the disk names rather than surfacing as a + webhook denial on the VirtualMachine. */}} + {{- $bootOrderOwners := dict }} {{- range $i, $disk := .Values.disks }} + {{- $bootOrder := int ($disk.bootOrder | default (add $i 1)) }} + {{- with get $bootOrderOwners (toString $bootOrder) }} + {{- fail (printf "disks %q and %q both have bootOrder %d; boot orders must be unique" . $disk.name $bootOrder) }} + {{- end }} + {{- $_ := set $bootOrderOwners (toString $bootOrder) $disk.name }} - name: disk-{{ $disk.name }} {{- $dv := lookup "cdi.kubevirt.io/v1beta1" "DataVolume" $.Release.Namespace (printf "vm-disk-%s" $disk.name) }} {{- if $dv }} @@ -104,7 +113,7 @@ spec: {{- else }} {{ $deviceKey }}: {} {{- end }} - bootOrder: {{ add $i 1 }} + bootOrder: {{ $bootOrder }} {{- else }} {{- fail (printf "Specified disk not exists in cluster: %s" .name) }} {{- end }} diff --git a/packages/apps/vm-instance/tests/disk_boot_order_test.yaml b/packages/apps/vm-instance/tests/disk_boot_order_test.yaml new file mode 100644 index 000000000..146cbaf53 --- /dev/null +++ b/packages/apps/vm-instance/tests/disk_boot_order_test.yaml @@ -0,0 +1,128 @@ +suite: vm-instance disk boot order + +# The first entry of `disks` is the one callers treat as the system disk, so +# booting from installer or rescue media must not require moving that media to +# the front of the list. These cases pin that an explicit bootOrder decides +# which device the VM boots from while list order stays as given, and that +# leaving it unset keeps the positional order existing VMs were rendered with. +release: + name: test-vm + namespace: tenant-test +templates: + - templates/vm.yaml +set: + fullnameOverride: test-vm + instanceType: "" + instanceProfile: "" + resources: + cpu: 1 + sockets: 1 + memory: 1Gi + _cluster: + scheduling: + dedicatedNodesForWindowsVMs: "false" +kubernetesProvider: + scheme: + "cdi.kubevirt.io/v1beta1/DataVolume": + gvr: + group: cdi.kubevirt.io + version: v1beta1 + resource: datavolumes + namespaced: true + objects: + - apiVersion: cdi.kubevirt.io/v1beta1 + kind: DataVolume + metadata: + name: vm-disk-system + namespace: tenant-test + annotations: {} + - apiVersion: cdi.kubevirt.io/v1beta1 + kind: DataVolume + metadata: + name: vm-disk-iso + namespace: tenant-test + annotations: + vm-disk.cozystack.io/optical: "true" + - apiVersion: cdi.kubevirt.io/v1beta1 + kind: DataVolume + metadata: + name: vm-disk-data + namespace: tenant-test + annotations: {} +tests: + - it: keeps the positional boot order when no disk sets bootOrder + set: + disks: + - name: system + - name: iso + - name: data + asserts: + - equal: + path: spec.template.spec.domain.devices.disks + value: + - name: disk-system + disk: {} + bootOrder: 1 + - name: disk-iso + cdrom: {} + bootOrder: 2 + - name: disk-data + disk: {} + bootOrder: 3 + + - it: boots from a CD-ROM in second position without reordering the disks + set: + disks: + - name: system + bootOrder: 2 + - name: iso + bus: sata + bootOrder: 1 + - name: data + asserts: + - equal: + path: spec.template.spec.domain.devices.disks + value: + - name: disk-system + disk: {} + bootOrder: 2 + - name: disk-iso + cdrom: + bus: sata + bootOrder: 1 + - name: disk-data + disk: {} + bootOrder: 3 + - equal: + path: spec.template.spec.volumes[0].dataVolume.name + value: vm-disk-system + + - it: refuses an explicit bootOrder that collides with another disk's positional one + set: + disks: + - name: system + - name: iso + bootOrder: 1 + asserts: + - failedTemplate: + errorMessage: 'disks "system" and "iso" both have bootOrder 1; boot orders must be unique' + + - it: refuses two disks given the same explicit bootOrder + set: + disks: + - name: system + bootOrder: 3 + - name: iso + bootOrder: 3 + asserts: + - failedTemplate: + errorMessage: 'disks "system" and "iso" both have bootOrder 3; boot orders must be unique' + + - it: rejects a bootOrder below 1 through the values schema + set: + disks: + - name: system + bootOrder: 0 + asserts: + - failedTemplate: + errorPattern: "minimum" diff --git a/packages/apps/vm-instance/values.schema.json b/packages/apps/vm-instance/values.schema.json index 7c24a9e52..45adaac72 100644 --- a/packages/apps/vm-instance/values.schema.json +++ b/packages/apps/vm-instance/values.schema.json @@ -69,6 +69,11 @@ "name" ], "properties": { + "bootOrder": { + "description": "Boot priority of the disk, 1 being tried first. Unset falls back to the disk's position in `disks` (the first disk gets 1), so set it to boot from a disk other than the first, such as an attached installer or rescue ISO, without reordering the list. Effective boot orders must be unique across disks: when one disk is given a value that another takes from its position, set both.", + "type": "integer", + "minimum": 1 + }, "bus": { "description": "Disk bus type (e.g. \"sata\").", "type": "string" diff --git a/packages/apps/vm-instance/values.yaml b/packages/apps/vm-instance/values.yaml index 88fc6f8dd..2730812a0 100644 --- a/packages/apps/vm-instance/values.yaml +++ b/packages/apps/vm-instance/values.yaml @@ -10,6 +10,8 @@ ## @field {string} name - Disk name. ## @x-cozystack-options {source: vmdisk} ## @field {string} [bus] - Disk bus type (e.g. "sata"). +## @field {*int} [bootOrder] - Boot priority of the disk, 1 being tried first. Unset falls back to the disk's position in `disks` (the first disk gets 1), so set it to boot from a disk other than the first, such as an attached installer or rescue ISO, without reordering the list. Effective boot orders must be unique across disks: when one disk is given a value that another takes from its position, set both. +## @minimum 1 ## @typedef {struct} Network - Network to attach the VM to. ## @field {string} [name] - Network attachment name. @@ -67,6 +69,12 @@ disks: [] ## - name: example-system ## - name: example-data ## bus: sata +## Boot from an attached ISO while keeping the system disk first in the list: +## disks: +## - name: example-system +## bootOrder: 2 +## - name: example-iso +## bootOrder: 1 ## @param {[]Network} networks - Networks to attach the VM to. networks: [] diff --git a/packages/system/vm-instance-rd/cozyrds/vm-instance.yaml b/packages/system/vm-instance-rd/cozyrds/vm-instance.yaml index e91a334a1..dc533ee59 100644 --- a/packages/system/vm-instance-rd/cozyrds/vm-instance.yaml +++ b/packages/system/vm-instance-rd/cozyrds/vm-instance.yaml @@ -12,7 +12,7 @@ spec: singular: vminstance plural: vminstances openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"external":{"description":"Enable external access from outside the cluster.","type":"boolean","default":false},"externalMethod":{"description":"Method to pass through traffic to the VM.","type":"string","default":"PortList","enum":["PortList","WholeIP"]},"externalPorts":{"description":"Ports to forward from outside the cluster.","type":"array","default":[22],"items":{"type":"integer"}},"externalAllowICMP":{"description":"Whether to accept ICMP traffic to the VM in PortList mode (preserves ping and PMTU discovery). No effect in WholeIP mode. Default true so ping behaves as users expect even when port filtering is in effect.","type":"boolean","default":true},"runStrategy":{"description":"Requested running state of the VirtualMachineInstance","type":"string","default":"Always","enum":["Always","Halted","Manual","RerunOnFailure","Once"]},"instanceType":{"description":"Virtual Machine instance type. Ignored when `resources` sizes the VM, which needs whole-number cpu and sockets plus memory; the whole type is dropped then, so any dedicated CPU placement, hugepages, NUMA or realtime settings it carries go with it. While this field is set, a `resources` block that sizes only part of the VM is rejected outright instead of ignored, so complete it or clear it.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"instanceProfile":{"description":"Virtual Machine preferences profile.","type":"string","default":"ubuntu","x-cozystack-options":{"source":"instanceprofile"}},"disks":{"description":"List of disks to attach.","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"bus":{"description":"Disk bus type (e.g. \"sata\").","type":"string"},"name":{"description":"Disk name.","type":"string","x-cozystack-options":{"source":"vmdisk"}}}}},"networks":{"description":"Networks to attach the VM to.","type":"array","default":[],"items":{"type":"object","properties":{"name":{"description":"Network attachment name.","type":"string","x-cozystack-options":{"source":"network"}}}}},"subnets":{"description":"Deprecated: use networks instead.","type":"array","default":[],"items":{"type":"object","properties":{"name":{"description":"Network attachment name.","type":"string","x-cozystack-options":{"source":"network"}}}}},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"The name of the GPU resource to attach.","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"cpuModel":{"description":"Model specifies the CPU model inside the VMI. List of available models https://github.com/libvirt/libvirt/tree/master/src/cpu_map","type":"string","default":""},"firmware":{"description":"Firmware and boot configuration (UEFI/BIOS selection, Secure Boot, persistent EFI NVRAM).","type":"object","default":{},"properties":{"bootloader":{"description":"Bootloader to boot the VM with: \"uefi\" (OVMF) or \"bios\" (SeaBIOS). Empty inherits the instanceProfile default.","type":"string"},"efiPersistent":{"description":"Persist EFI NVRAM (e.g. enrolled Secure Boot keys, such as an updated Microsoft UEFI CA) across reboots. Only applies when bootloader is \"uefi\".","type":"boolean"},"secureBoot":{"description":"Enable UEFI Secure Boot. Only applies when bootloader is \"uefi\".","type":"boolean"}}},"resources":{"description":"Resource configuration for the virtual machine. Set whole-number cpu and sockets together with memory to size the VM directly, which overrides `instanceType`. Alongside an `instanceType`, a block that sizes only part of the VM is rejected; a whole-number cpu or sockets on its own sizes nothing and is ignored.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated, as a positive whole number such as `2`. Millicore or other unit-suffixed, fractional, zero, negative, above 4294967295 and quoted leading-zero values are rejected at render time, because the value becomes the VM's unsigned integer `domain.cpu.cores`.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated, as a quantity of at least `1M` such as `8Gi`. Less than 1M is rejected at render time, because KubeVirt refuses a non-zero amount below 1M and a zero or negative amount sizes nothing.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"sockets":{"description":"Number of CPU sockets (vCPU topology), as a positive whole number such as `1`. Millicore or other unit-suffixed, fractional, zero, negative, above 4294967295 and quoted leading-zero values are rejected at render time, because the value becomes the VM's unsigned integer `domain.cpu.sockets`.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"sshKeys":{"description":"List of SSH public keys for authentication.","type":"array","default":[],"items":{"type":"string"}},"cloudInit":{"description":"Cloud-init user data.","type":"string","default":""},"netplanDhcp":{"description":"Supply cloud-init network-data that requests DHCP on every `en*` and `eth*` interface and keeps the DNS search domains DHCP offers. Needed for a netplan guest (Ubuntu, Debian) to address a secondary network such as a VPC subnet. Leave it off for a guest rendered by anything other than netplan, which would lose DHCP on every interface, and for a guest that configures an interface statically through netplan, which this configuration would shadow. Turning it on for a VM that has neither `cloudInit` nor `sshKeys` adds a cloud-init disk, and the guest's next boot is then treated as a first boot.","type":"boolean","default":false},"cloudInitSeed":{"description":"Seed string to generate SMBIOS UUID for the VM.","type":"string","default":""}}} + {"title":"Chart Values","type":"object","properties":{"external":{"description":"Enable external access from outside the cluster.","type":"boolean","default":false},"externalMethod":{"description":"Method to pass through traffic to the VM.","type":"string","default":"PortList","enum":["PortList","WholeIP"]},"externalPorts":{"description":"Ports to forward from outside the cluster.","type":"array","default":[22],"items":{"type":"integer"}},"externalAllowICMP":{"description":"Whether to accept ICMP traffic to the VM in PortList mode (preserves ping and PMTU discovery). No effect in WholeIP mode. Default true so ping behaves as users expect even when port filtering is in effect.","type":"boolean","default":true},"runStrategy":{"description":"Requested running state of the VirtualMachineInstance","type":"string","default":"Always","enum":["Always","Halted","Manual","RerunOnFailure","Once"]},"instanceType":{"description":"Virtual Machine instance type. Ignored when `resources` sizes the VM, which needs whole-number cpu and sockets plus memory; the whole type is dropped then, so any dedicated CPU placement, hugepages, NUMA or realtime settings it carries go with it. While this field is set, a `resources` block that sizes only part of the VM is rejected outright instead of ignored, so complete it or clear it.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"instanceProfile":{"description":"Virtual Machine preferences profile.","type":"string","default":"ubuntu","x-cozystack-options":{"source":"instanceprofile"}},"disks":{"description":"List of disks to attach.","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"bootOrder":{"description":"Boot priority of the disk, 1 being tried first. Unset falls back to the disk's position in `disks` (the first disk gets 1), so set it to boot from a disk other than the first, such as an attached installer or rescue ISO, without reordering the list. Effective boot orders must be unique across disks: when one disk is given a value that another takes from its position, set both.","type":"integer","minimum":1},"bus":{"description":"Disk bus type (e.g. \"sata\").","type":"string"},"name":{"description":"Disk name.","type":"string","x-cozystack-options":{"source":"vmdisk"}}}}},"networks":{"description":"Networks to attach the VM to.","type":"array","default":[],"items":{"type":"object","properties":{"name":{"description":"Network attachment name.","type":"string","x-cozystack-options":{"source":"network"}}}}},"subnets":{"description":"Deprecated: use networks instead.","type":"array","default":[],"items":{"type":"object","properties":{"name":{"description":"Network attachment name.","type":"string","x-cozystack-options":{"source":"network"}}}}},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"The name of the GPU resource to attach.","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"cpuModel":{"description":"Model specifies the CPU model inside the VMI. List of available models https://github.com/libvirt/libvirt/tree/master/src/cpu_map","type":"string","default":""},"firmware":{"description":"Firmware and boot configuration (UEFI/BIOS selection, Secure Boot, persistent EFI NVRAM).","type":"object","default":{},"properties":{"bootloader":{"description":"Bootloader to boot the VM with: \"uefi\" (OVMF) or \"bios\" (SeaBIOS). Empty inherits the instanceProfile default.","type":"string"},"efiPersistent":{"description":"Persist EFI NVRAM (e.g. enrolled Secure Boot keys, such as an updated Microsoft UEFI CA) across reboots. Only applies when bootloader is \"uefi\".","type":"boolean"},"secureBoot":{"description":"Enable UEFI Secure Boot. Only applies when bootloader is \"uefi\".","type":"boolean"}}},"resources":{"description":"Resource configuration for the virtual machine. Set whole-number cpu and sockets together with memory to size the VM directly, which overrides `instanceType`. Alongside an `instanceType`, a block that sizes only part of the VM is rejected; a whole-number cpu or sockets on its own sizes nothing and is ignored.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated, as a positive whole number such as `2`. Millicore or other unit-suffixed, fractional, zero, negative, above 4294967295 and quoted leading-zero values are rejected at render time, because the value becomes the VM's unsigned integer `domain.cpu.cores`.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated, as a quantity of at least `1M` such as `8Gi`. Less than 1M is rejected at render time, because KubeVirt refuses a non-zero amount below 1M and a zero or negative amount sizes nothing.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"sockets":{"description":"Number of CPU sockets (vCPU topology), as a positive whole number such as `1`. Millicore or other unit-suffixed, fractional, zero, negative, above 4294967295 and quoted leading-zero values are rejected at render time, because the value becomes the VM's unsigned integer `domain.cpu.sockets`.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"sshKeys":{"description":"List of SSH public keys for authentication.","type":"array","default":[],"items":{"type":"string"}},"cloudInit":{"description":"Cloud-init user data.","type":"string","default":""},"netplanDhcp":{"description":"Supply cloud-init network-data that requests DHCP on every `en*` and `eth*` interface and keeps the DNS search domains DHCP offers. Needed for a netplan guest (Ubuntu, Debian) to address a secondary network such as a VPC subnet. Leave it off for a guest rendered by anything other than netplan, which would lose DHCP on every interface, and for a guest that configures an interface statically through netplan, which this configuration would shadow. Turning it on for a VM that has neither `cloudInit` nor `sshKeys` adds a cloud-init disk, and the guest's next boot is then treated as a first boot.","type":"boolean","default":false},"cloudInitSeed":{"description":"Seed string to generate SMBIOS UUID for the VM.","type":"string","default":""}}} release: prefix: vm-instance- labels: From c617a994531d623b9c9c1946ffd2de620477ae13 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 10 Nov 2026 08:09:37 +0000 Subject: [PATCH 404/518] fix(release): withhold or refuse a unit that does only part of its original on the line A clean cherry-pick says nothing about whether the original landed whole: git deletes a file the line never had without a conflict and leaves a file the line lacks untouched. #5088 reached the 1.6 batch that way, carrying only its test while the two deletions that are the fix had no effect. Every M and D path of an original is now judged at its counterpart on the line, followed through renames on main, on the line, or both. A modified path missing before and after, or left unchanged when the line holds neither the original's blob nor a clean git merge-file result, is omitted; a deleted path absent before is omitted. Content merge-file refuses, mode changes and skipped rename detection become diagnostics and never count as omissions; I/O failures stop the command. The same guard runs in four places: the picker, which resets a partial pick and parks the original; a check-time pass over every unit on the open batch at its own first parent, whatever its label, which fails check --cut and verify; the same pass over every unpicked original's clean net merge, which parks a partial one with no record needed; and fold. contained accepts the renamed path, so a fix of a renamed file passes verify and fold. The unit mapping also closes the found() gap: a merged original whose first pick came out empty is on the batch, not missing. The record moves to version 3 with parked entries and partial units, and reports to format 2 with typed parked entries; format 1 reports still read. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 8 +- cmd/backport-batch/assemble.go | 51 +- cmd/backport-batch/body.go | 54 +- cmd/backport-batch/check.go | 29 +- cmd/backport-batch/defer.go | 13 +- cmd/backport-batch/effects.go | 555 ++++++++++++++++++ cmd/backport-batch/effects_test.go | 350 +++++++++++ cmd/backport-batch/fold_test.go | 56 ++ cmd/backport-batch/history.go | 45 +- cmd/backport-batch/main.go | 20 + cmd/backport-batch/park_test.go | 331 +++++++++++ cmd/backport-batch/record.go | 104 +++- cmd/backport-batch/record_test.go | 43 +- cmd/backport-batch/report.go | 129 +++- cmd/backport-batch/report_test.go | 54 +- cmd/backport-batch/shape.go | 46 +- cmd/backport-batch/verify.go | 9 + docs/release/backport-mechanics.md | 2 +- docs/release/backport-tasks/README.md | 2 +- .../backport-tasks/resolve-conflict.md | 2 +- docs/release/backports.md | 10 +- 21 files changed, 1856 insertions(+), 57 deletions(-) create mode 100644 cmd/backport-batch/effects.go create mode 100644 cmd/backport-batch/effects_test.go create mode 100644 cmd/backport-batch/park_test.go diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index b1837f489..6d9157795 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -37,6 +37,8 @@ The pull request body goes to stdout, built from the line's own `.github/PULL_RE `--no-workflow-files` is for a token that may not push a change to `.github/workflows/`, which the scheduled run's App token may not. An original that changes anything there is not picked: the body lists it under "Waiting for a maintainer" with those files, and the `backport-batch` block records it with `"workflows": true`, so `check`, `notify` and a run that continues the batch know it waits. It is never a conflict. A run without the flag, a maintainer's with their own token, picks it like any other original, onto the open batch when it continues one. +A clean pick can still carry only part of its original: git deletes a file the line never had without a word, and leaves alone a file the line lacks. So every path the original modifies or deletes, `M` or `D` in `git diff --name-status --no-renames M^1 M`, is judged at its counterpart on the line: the path itself, or the one a rename `git diff -M` finds between the line's fork point and `M^1` on `main`, between the fork point and the line, or both in turn. A modified path counts as left without effect when its counterpart exists neither before nor after the pick; when the pick leaves an existing file as it was, the file counts as there only when its blob is the original's, binary files included, or when `git merge-file` of the original's preimage, the line's file and the original's postimage merges cleanly back into the line's file. A deleted path counts as left without effect when its counterpart is absent before a pick that changes something, a deletion the line already made being no change of that pick, or still there after it, whatever the pick changed in it; when the original adds that path itself, as when the line made the same rename, it is judged as a modified file against what the original makes of that path. Picks that change nothing carry the original only when the line already has what it does to every file it modifies or deletes, so a change made where its branch merged `main`, which no pick brings, is still missing. What cannot be judged, a mode or type change, content `merge-file` refuses, or a rename git skipped looking for, is shown with the original and never counts against it. An original whose net merge onto the line, `git merge-tree --merge-base=M^1`, leaves something without effect is not tried, and one whose picks do is taken back off the branch: either is parked as partial, listed under "Parked" with those paths, and its release note stays out of the body. A conflict is never called partial. `backport/include` lifts no park. + `--defer N[,N] --reason TEXT` takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off `origin/` without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got `backport/skip` after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the `backport-batch` block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. When it takes out everything the batch carries, there is nothing left to push, and `--push` closes the batch pull request instead, as [the line's record](#the-lines-record-and-reports) describes. `--drop N[,N] --reason TEXT` drops originals from the line: where `--defer` holds one out of this batch, a drop keeps it out of every later batch of the line too, until `--undrop N[,N]` withdraws it. A drop is recorded in the [line's record](#the-lines-record-and-reports), so both need an open batch pull request that carries one, and exit 2 without it; once a batch merges, its drops are carried into the next batch's record. A dropped original the open batch carries is taken off it as `--defer` would, and the rebuild is pushed the same way; any other drop or undrop changes only the record, so the batch stays as it is even when the line moved since. An undropped original is missing until the next run tries it. A drop names the original's merge commit on `main` as well as its number, and a record whose drop does not match what that merge commit merged stops every command with exit 2. An original the line already carries cannot be dropped, and neither can a number that merged into `main` before the line branched. The order is the rules', then the drop: what `backport/skip`, rules 2 to 4 or the line itself decide comes first, with the drop kept on record beside it, and `backport/include` does not lift a drop. The body lists dropped originals under their own heading among the skipped ones, `check` prints a note for each, and `fold` refuses one. @@ -51,13 +53,15 @@ Exits 1 and prints the URL of every pull request the rules take that is neither An original with `backport/skip` that the open batch picked before the label came prints a warning in either mode, naming the batch and `run --defer`, which takes it out; it fails neither. +A unit on the open batch that leaves part of its original without effect, judged as [`run`](#run) judges a pick, at the unit's own first parent and whatever its original's rule, label or drop, prints a warning naming what it leaves out, and fails `--cut` ahead of any skip, drop or stale warning: take it out with `run --defer`. An original that is parked as partial prints as a note, `#N is parked: only part of it applies`, and fails neither. + ### fold -Checks a hand-made resolution of a deferred conflict before it is pushed onto the batch, in the shape the run gives an original. For a merged pull request `--commit` is a merge whose first parent is the head of the open batch pull request and whose message names the original in exactly one `Backported-from:` trailer; its second parent ends a straight line from that head of picks of the pull request's own commits, in their order, each ending in the `(cherry picked from commit …)` line of the commit it came from, none claiming another commit of the same original, a commit of another pull request never among them. For a squash-merged one it is a single commit on that head whose `(cherry picked from commit …)` line names the original's commit on `main`. Either has to change something, and only files the original changes. The [conflict task](../../docs/release/backport-tasks/resolve-conflict.md) gives the commands. fold prints `ok` and exits 0, or prints why not and exits 1. It never commits or pushes; push one checked resolution at a time, since the next one has to sit on the new head. +Checks a hand-made resolution of a deferred conflict before it is pushed onto the batch, in the shape the run gives an original. For a merged pull request `--commit` is a merge whose first parent is the head of the open batch pull request and whose message names the original in exactly one `Backported-from:` trailer; its second parent ends a straight line from that head of picks of the pull request's own commits, in their order, each ending in the `(cherry picked from commit …)` line of the commit it came from, none claiming another commit of the same original, a commit of another pull request never among them. For a squash-merged one it is a single commit on that head whose `(cherry picked from commit …)` line names the original's commit on `main`. Either has to change something, and only files the original changes or the paths a rename on `main` or on the line gave them, and has to leave nothing the original modifies or deletes without effect, judged as [`run`](#run) judges a pick: a resolution that ports what the line has and leaves out what it lacks is refused, since such a port goes to the line as a pull request of its own. The [conflict task](../../docs/release/backport-tasks/resolve-conflict.md) gives the commands. fold prints `ok` and exits 0, or prints why not and exits 1. It never commits or pushes; push one checked resolution at a time, since the next one has to sit on the new head. ### verify -Holds every unit of a batch to the shape fold checks for one, the clean picks as well as resolutions: it walks the batch's first-parent line from `origin/` to `--head`, or to the head of the open batch pull request, and requires each unit to be one original, merged into `main` after the line branched, in that shape, and no original twice. A commit that carries no original fails it, and so does a merge of the line into the batch. It prints one line per unit, and exits 0 when all pass, 1 when one does not, and 2 when it cannot tell. With `--head` it reads git alone, nothing from GitHub. fold runs only where someone resolved a conflict and chose to run it; [`pr-labeler.yaml`](../../.github/workflows/pr-labeler.yaml) runs verify on every push to a batch pull request, from a build of `main`, with the batch head fetched as data. +Holds every unit of a batch to the shape fold checks for one, the clean picks as well as resolutions: it walks the batch's first-parent line from `origin/` to `--head`, or to the head of the open batch pull request, and requires each unit to be one original, merged into `main` after the line branched, in that shape, and no original twice. A commit that carries no original fails it, and so does a merge of the line into the batch, and a unit that leaves part of its original without effect, judged as [`run`](#run) judges a pick. It prints one line per unit, and exits 0 when all pass, 1 when one does not, and 2 when it cannot tell. With `--head` it reads git alone, nothing from GitHub. fold runs only where someone resolved a conflict and chose to run it; [`pr-labeler.yaml`](../../.github/workflows/pr-labeler.yaml) runs verify on every push to a batch pull request, from a build of `main`, with the batch head fetched as data. ### notify diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index 7f36ff3f4..53093b018 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -198,8 +198,14 @@ func (a *app) run(args []string) int { } } for _, v := range b.qualifying() { + on := false + if carried != nil { + if on, err = b.onBranch(carried, v); err != nil { + return a.fail(err) + } + } switch { - case carried != nil && found(carried, v): + case on: r.onBranch = append(r.onBranch, v) continue case held[v.o.number].Reason != "": @@ -210,6 +216,17 @@ func (a *app) run(args []string) int { r.skipped[ruleOnLine] = append(r.skipped[ruleOnLine], v) continue } + if v.partial != nil { + // Judged at the line's head; what the batch picked since may + // supply what it lacked there. + if err := a.partialAtHead(b, v); err != nil { + return a.fail(err) + } + if v.partial != nil { + r.parked = append(r.parked, v) + continue + } + } if *noWorkflows { files, err := workflowFiles(v.commits) if err != nil { @@ -220,11 +237,19 @@ func (a *app) run(args []string) int { continue } } - p, err := pickOriginal(v) + p, err := pickOriginal(v, func(base, head string) ([]string, error) { + omitted, diag, err := b.omittedEffects(v, base, head) + v.addDiag(diag...) + return omitted, err + }) if err != nil { return a.fail(err) } switch p.status { + case "partial": + v.partial = p.files + v.park = &parkedEntry{Original: v.o.number, Merge: v.o.merge, Kind: parkPartial, Files: sortedClean(p.files)} + r.parked = append(r.parked, v) case "picked": r.onBranch = append(r.onBranch, v) case "conflict": @@ -246,10 +271,23 @@ func (a *app) run(args []string) int { if !descends { return a.fail(fmt.Errorf("%.12s does not descend from --onto %.12s", head, *onto)) } + if err := b.markPartial(r, carried.head); err != nil { + return a.fail(err) + } } return a.finish(b, r, branch, date, *push, false) } +// partialAtHead judges v again where the run has got to, when earlier picks +// moved the branch past the line's head. +func (a *app) partialAtHead(b *batch, v *verdict) error { + head, err := resolve("HEAD") + if err != nil || head == b.lineHead { + return err + } + return b.partialAt(v, head) +} + // keptOut lists the originals with backport/skip, or dropped, that the batch // still carries: a run never takes a commit off the batch, --defer does. func keptOut(b *batch, carried *history) ([]*verdict, error) { @@ -261,7 +299,11 @@ func keptOut(b *batch, carried *history) ([]*verdict, error) { if err := b.load(v); err != nil { return nil, err } - if found(carried, v) { + on, err := b.onBranch(carried, v) + if err != nil { + return nil, err + } + if on { out = append(out, v) } } @@ -365,7 +407,8 @@ func (r *report) lineRecord(doc reportDoc) lineRecord { dropped = []droppedEntry{} } slices.SortFunc(dropped, func(a, b droppedEntry) int { return a.Original - b.Original }) - return lineRecord{Version: 2, Line: doc.Line, Built: &built, Report: doc.ID, Closing: r.closing, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: dropped, Stale: slices.Clone(doc.Overlays.Stale)} + return lineRecord{Version: recordVersion, Line: doc.Line, Built: &built, Report: doc.ID, Closing: r.closing, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: dropped, Stale: slices.Clone(doc.Overlays.Stale), + Parked: slices.Clone(doc.Primary.Parked), Partial: slices.Clone(doc.Overlays.Partial)} } // branchDate is the date a batch branch was opened under. diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index da4691450..282f93669 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -238,8 +238,10 @@ type report struct { overrides []*verdict stale []*verdict start time.Time - // shapes says, for an original on the branch, what it brought. - shapes map[*verdict]string + // shapes says, for an original on the branch, what it brought, and + // partialFiles what a partial unit leaves without effect. + shapes map[*verdict]string + partialFiles map[*verdict][]string // missing are originals the policy takes that the run neither tried nor // found anywhere. missing []*verdict @@ -253,6 +255,11 @@ type report struct { closing *closing // dropped are the maintainers' drops the record keeps. dropped []droppedEntry + // parked are what the batch leaves out because it cannot land whole, + // each with its park, and partial the originals on the branch whose unit + // does only part of what they do on main. + parked []*verdict + partial []*verdict } // ruleName is how output names what kept an original out: the policy's @@ -346,6 +353,18 @@ func (r *report) what(details map[int]bool) string { fmt.Fprintf(&b, "- %s %s: %s\n", prLink(p.v.o.number), titleSpan(p.v.o.title), strings.Join(files, ", ")) } } + if len(r.partial) > 0 { + fmt.Fprintf(&b, "\n### Still on this branch: only part of it applies\n\nEach of these is on this branch but leaves files of its original as they were on %s, so it carries only part of what it does on `main`. Take each out with `backport-batch run --defer` before the cut; what applies of it goes as a pull request of its own.\n\n", codeSpan(r.line.branch())) + for _, v := range r.partial { + fmt.Fprintf(&b, "- %s %s: leaves %s without effect\n", prLink(v.o.number), titleSpan(v.o.title), fileSpans(r.partialFiles[v])) + } + } + if len(r.parked) > 0 { + fmt.Fprintf(&b, "\n### Parked\n\nThese cannot land on %s whole, so the batch does not pick them: each would leave part of it without effect there, or builds on a change that never reaches the line. A maintainer ports what applies as a pull request of its own, or drops the original with `backport-batch run --drop`; `backport/include` does not lift a park.\n\n", codeSpan(r.line.branch())) + for _, v := range r.parked { + fmt.Fprintf(&b, "- %s %s: %s\n", prLink(v.o.number), titleSpan(v.o.title), parkReason(*v.park)) + } + } if len(r.overrides) > 0 { b.WriteString("\n### Maintainer overrides\n\n") for _, v := range r.overrides { @@ -384,6 +403,8 @@ func (r *report) what(details map[int]bool) string { // outcome is what became of an original a label overrode. func (r *report) outcome(v *verdict) string { switch { + case slices.Contains(r.partial, v): + return "still on this branch, but only part of it applies; take it out with `backport-batch run --defer`" case slices.Contains(r.stale, v) && v.rule == ruleDropped: return "dropped, but still on this branch; take it out with `backport-batch run --defer`" case slices.Contains(r.stale, v): @@ -398,12 +419,41 @@ func (r *report) outcome(v *verdict) string { return "deferred to the next batch" case slices.ContainsFunc(r.waiting, func(p pick) bool { return p.v == v }): return "waiting for a maintainer: it changes " + codeSpan(workflowsDir) + case slices.Contains(r.parked, v): + return "parked: " + parkReason(*v.park) case v.rule != 0: return "skipped by " + ruleName(v.rule) + ", " + titleSpan(v.reason) } return "not picked" } +// fileSpans lists files as code spans, the first few of them. +func fileSpans(files []string) string { + var out []string + for _, f := range files[:min(len(files), filesListed)] { + out = append(out, codeSpan(f)) + } + if more := len(files) - len(out); more > 0 { + out = append(out, fmt.Sprintf("%d more", more)) + } + return strings.Join(out, ", ") +} + +// parkReason says why a park leaves its original out, for the body. +func parkReason(p parkedEntry) string { + if p.Kind == parkPartial { + return "only part of it applies; it would leave " + fileSpans(p.Files) + " without effect" + } + var needs []string + for _, n := range p.Needs { + needs = append(needs, prLink(n)) + } + for _, c := range p.Commits { + needs = append(needs, codeSpan(c[:12])) + } + return "builds on " + strings.Join(needs, ", ") + ", which never reaches this line" +} + func (r *report) notes() string { var notes []string for _, v := range r.onBranch { diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index ae5395d45..b9983aca3 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -20,6 +20,7 @@ import ( "fmt" "slices" "strconv" + "strings" ) // check says whether every original the policy takes is on the line, on the @@ -163,6 +164,10 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { named := v.o.url() + " " + terminalSafe(v.o.title) e, isDeferred := deferred[v.o.number] switch { + case b.partialUnits[v] != nil && cut: + out[v] = disposition{outcomeUnmerged, fmt.Sprintf("#%d is on batch #%d, but only part of it applies to %s: it leaves %s without effect; take it out with backport-batch run --defer before the cut: %s", v.o.number, b.pr.Number, b.line.branch(), quotedFiles(b.partialUnits[v]), named)} + case b.partialUnits[v] != nil: + out[v] = disposition{outcomeWarning, fmt.Sprintf("#%d is on batch #%d, but only part of it applies to %s: it leaves %s without effect; take it out with backport-batch run --defer: %s", v.o.number, b.pr.Number, b.line.branch(), quotedFiles(b.partialUnits[v]), named)} case slices.Contains(stale, v): out[v] = disposition{outcomeWarning, fmt.Sprintf("#%d is still on batch #%d; take it out with backport-batch run --defer: %s", v.o.number, b.pr.Number, named)} case v.rule == ruleDropped: @@ -173,14 +178,16 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { if cut { out[v] = disposition{outcomeUnmerged, fmt.Sprintf("#%d is only on open batch PR #%d, merge it before the cut: %s", v.o.number, b.pr.Number, named)} } + case isDeferred && e.Reason != "": + out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred on %s to the next batch: %s (%s)", on, named, terminalSafe(e.Reason))} + case v.park != nil && v.park.Kind == parkPartial: + out[v] = disposition{outcomeNote, fmt.Sprintf("#%d is parked: only part of it applies to %s, which would leave %s without effect; port what applies as a pull request of its own: %s", v.o.number, b.line.branch(), quotedFiles(v.park.Files), named)} case isDeferred && e.Workflows && !b.src.open: out[v] = disposition{outcomeMissing, named} case isDeferred && e.Workflows && cut: out[v] = disposition{outcomeWaiting, fmt.Sprintf("#%d changes %s and waits on batch PR %s for a maintainer, bring it over or defer it before the cut: %s", v.o.number, workflowsDir, on, named)} case isDeferred && e.Workflows: out[v] = disposition{outcomeWarning, fmt.Sprintf("waiting on %s for a maintainer, since it changes %s: %s", on, workflowsDir, named)} - case isDeferred && e.Reason != "": - out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred on %s to the next batch: %s (%s)", on, named, terminalSafe(e.Reason))} case isDeferred: out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred conflict on %s: %s", on, named)} default: @@ -190,6 +197,18 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { return out, nil } +// quotedFiles names files for a terminal, each quoted, the first few of them. +func quotedFiles(files []string) string { + var out []string + for _, f := range files[:min(len(files), filesListed)] { + out = append(out, strconv.Quote(f)) + } + if more := len(files) - len(out); more > 0 { + out = append(out, fmt.Sprintf("%d more", more)) + } + return strings.Join(out, ", ") +} + // terminalSafe keeps a title from carrying escape sequences into a terminal. func terminalSafe(s string) string { return capped(oneLine(clean(s, false)), titleLimit) } @@ -289,5 +308,9 @@ func refuseFold(b *batch, n int, commit string) (string, error) { case carried != v: return fmt.Sprintf("%.12s carries #%d, not #%d", commit, carried.o.number, n), nil } - return "", nil + omitted, _, err := b.omittedEffects(v, u.parents[0], u.commit) + if err != nil || len(omitted) == 0 { + return "", err + } + return leftOut(v, b.line, omitted), nil } diff --git a/cmd/backport-batch/defer.go b/cmd/backport-batch/defer.go index 52fa8cf62..8915d4ebd 100644 --- a/cmd/backport-batch/defer.go +++ b/cmd/backport-batch/defer.go @@ -78,7 +78,7 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { switch v := b.find(n); { case v == nil: return a.fail(fmt.Errorf("#%d did not merge into main after %s branched", n, b.line.branch())) - case v.rule != 0 && !slices.Contains(stale, v): + case v.rule != 0 && !slices.Contains(stale, v) && b.partialUnits[v] == nil: return a.fail(fmt.Errorf("#%d is not one the batch takes: %s, %s", n, ruleName(v.rule), terminalSafe(v.reason))) } hold[n] = reason @@ -140,11 +140,17 @@ func (a *app) reclassify(b *batch, picks *history, hold map[int]string, drops [] } for _, v := range b.qualifying() { e := recorded[v.o.number] + on, err := b.onBranch(picks, v) + if err != nil { + return a.fail(err) + } switch { - case found(picks, v): + case on: r.onBranch = append(r.onBranch, v) case hold[v.o.number] != "": r.held = append(r.held, pick{v: v, status: "held", reason: hold[v.o.number]}) + case v.park != nil: + r.parked = append(r.parked, v) case e.Workflows: r.waiting = append(r.waiting, pick{v: v, status: "waiting", files: e.Files, more: e.More}) case len(e.Files) > 0: @@ -155,6 +161,9 @@ func (a *app) reclassify(b *batch, picks *history, hold map[int]string, drops [] r.missing = append(r.missing, v) } } + if err := b.markPartial(r, picks.head); err != nil { + return a.fail(err) + } return a.finish(b, r, b.pr.Head.Ref, date, push, true) } diff --git a/cmd/backport-batch/effects.go b/cmd/backport-batch/effects.go new file mode 100644 index 000000000..590b70b21 --- /dev/null +++ b/cmd/backport-batch/effects.go @@ -0,0 +1,555 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "os" + "path/filepath" + "slices" + "strconv" + "strings" +) + +// A clean pick can still carry only part of an original: git deletes a file +// the line never had without a word, and leaves alone a file the line lacks. +// What follows tells such a unit from a whole one, by what it leaves without +// effect of the files the original modifies or deletes. + +// treeEntry is one record of git ls-tree. +type treeEntry struct{ mode, kind, oid string } + +// regular says whether git can merge the entry as text: a file, not a +// symlink, a submodule or a tree. +func (e treeEntry) regular() bool { + return e.kind == "blob" && (e.mode == "100644" || e.mode == "100755") +} + +// treeEntries reads the entries paths have at treeish, from full ls-tree +// records: a path that is a tree there, or absent, has none. +func treeEntries(treeish string, paths []string) (map[string]treeEntry, error) { + out := map[string]treeEntry{} + want := map[string]bool{} + for _, p := range paths { + want[p] = true + } + const chunk = 500 + for len(paths) > 0 { + n := min(chunk, len(paths)) + args := append([]string{"--literal-pathspecs", "ls-tree", "-r", "-z", "--full-tree", "--end-of-options", treeish, "--"}, paths[:n]...) + raw, err := git(args...) + if err != nil { + return nil, err + } + for _, rec := range splitNUL(raw) { + meta, path, ok := strings.Cut(rec, "\t") + f := strings.Fields(meta) + if !ok || len(f) != 3 || checkSHA(f[2]) != nil { + return nil, fmt.Errorf("git ls-tree %s: unreadable record %q", treeish, terminalSafe(rec)) + } + if want[path] { + out[path] = treeEntry{mode: f[0], kind: f[1], oid: f[2]} + } + } + paths = paths[n:] + } + return out, nil +} + +// renameSet is what git diff -M found renamed between two commits, old path +// to new, and whether it looked at every candidate. +type renameSet struct { + moved map[string]string + complete bool +} + +// renames reads the renames between from and to, once per pair. git says on +// stderr when it skipped inexact detection for too many files; complete is +// then false, and no path may be called absent for want of a rename. +func (b *batch) renames(from, to string) (renameSet, error) { + key := from + ".." + to + if s, ok := b.renameCache[key]; ok { + return s, nil + } + // -l0 lifts git's own limit, which the picker's merge does not share: + // a rename the pick follows has to be one the checks of its shape see. + stdout, stderr, code, err := gitStatus("diff", "--no-ext-diff", "--no-textconv", "-M", "-l"+strconv.Itoa(b.renameLimit), "--name-status", "-z", "--diff-filter=R", "--end-of-options", from, to) + switch { + case err != nil: + return renameSet{}, err + case code != 0: + return renameSet{}, fmt.Errorf("git diff -M %s %s: exit %d: %s", from, to, code, strings.TrimSpace(stderr)) + } + f := splitNUL(stdout) + if len(f)%3 != 0 { + return renameSet{}, fmt.Errorf("unreadable renames between %s and %s", from, to) + } + s := renameSet{moved: map[string]string{}, complete: !strings.Contains(stderr, "detection was skipped")} + for i := 0; i < len(f); i += 3 { + if !strings.HasPrefix(f[i], "R") { + return renameSet{}, fmt.Errorf("unreadable renames between %s and %s: %q", from, to, terminalSafe(f[i])) + } + s.moved[f[i+1]] = f[i+2] + } + if b.renameCache == nil { + b.renameCache = map[string]renameSet{} + } + b.renameCache[key] = s + return s, nil +} + +// effectPaths are the paths v modifies or deletes; what it adds has nothing +// to correspond to. +func effectPaths(v *verdict) []string { + var out []string + for _, c := range v.changes { + if c.status == 'M' || c.status == 'D' { + out = append(out, c.path) + } + } + return out +} + +// correspondence maps a path of the original (at M^1) to the path holding the +// same file on target, through a main-side rename base..M^1, a line-side +// rename base..target, or both composed through the base path. A path with +// no counterpart maps to itself, and one git skipped rename detection for is +// left out and named in unknown: it is not known to be absent. +func (b *batch) correspondence(v *verdict, target string) (map[string]string, []string, error) { + paths := effectPaths(v) + have, err := treeEntries(target, paths) + if err != nil { + return nil, nil, err + } + corr := map[string]string{} + var missing []string + for _, p := range paths { + if _, ok := have[p]; ok { + corr[p] = p + } else { + missing = append(missing, p) + } + } + if len(missing) == 0 { + return corr, nil, nil + } + onMain, err := b.renames(b.base, v.o.merge+"^1") + if err != nil { + return nil, nil, err + } + onLine, err := b.renames(b.base, target) + if err != nil { + return nil, nil, err + } + fromBase := map[string]string{} + for old, now := range onMain.moved { + fromBase[now] = old + } + candidates := map[string]string{} + var names []string + for _, p := range missing { + q := p + if old, ok := fromBase[p]; ok { + q = old + } + r := q + if now, ok := onLine.moved[q]; ok { + r = now + } + if r != p { + candidates[p] = r + names = append(names, r) + } + } + found, err := treeEntries(target, names) + if err != nil { + return nil, nil, err + } + var unknown []string + for _, p := range missing { + if r, ok := candidates[p]; ok { + if _, ok := found[r]; ok { + corr[p] = r + continue + } + } + if !onMain.complete || !onLine.complete { + unknown = append(unknown, p) + continue + } + corr[p] = p + } + return corr, unknown, nil +} + +// omittedEffects lists the paths v modifies or deletes that the change from +// before to after leaves without effect, each judged at its corresponding +// path c(p) on before. A modified path is omitted when c(p) exists on neither +// side, applied when the change touches it, and when the change leaves an +// existing file alone, satisfied only if the line already has what the +// original makes of it. A deleted path is omitted when c(p) is absent before: +// a deletion the line already did is no change this unit makes. diag says +// what could not be judged, which is never counted as omitted. +func (b *batch) omittedEffects(v *verdict, before, after string) ([]string, []string, error) { + if err := b.load(v); err != nil { + return nil, nil, err + } + corr, unknown, err := b.correspondence(v, before) + if err != nil { + return nil, nil, err + } + var diag []string + for _, p := range unknown { + diag = append(diag, "git skipped rename detection, so where "+strconv.Quote(p)+" is on the line is not known") + } + added := map[string]bool{} + for _, c := range v.changes { + if c.status == 'A' { + added[c.path] = true + } + } + // A deleted path whose counterpart the original adds itself is a rename + // the line already made the same way: it is judged as a modified file, + // against what the original makes of that path. + var at, own, moved []string + movedTo := map[string]string{} + for _, c := range v.changes { + cp, ok := corr[c.path] + if !ok { + continue + } + at = append(at, cp) + switch { + case c.status == 'M': + own = append(own, c.path) + case c.status == 'D' && cp != c.path && added[cp]: + own, moved = append(own, c.path), append(moved, cp) + movedTo[c.path] = cp + } + } + bef, err := treeEntries(before, at) + if err != nil { + return nil, nil, err + } + aft, err := treeEntries(after, at) + if err != nil { + return nil, nil, err + } + pre, err := treeEntries(v.o.merge+"^1", own) + if err != nil { + return nil, nil, err + } + post, err := treeEntries(v.o.merge, append(slices.Clone(own), moved...)) + if err != nil { + return nil, nil, err + } + // A change that makes none is judged by its trees, whatever names them: + // a commit and the tree a net merge wrote can be the same change. + same := before == after + if !same { + bt, err := resolveTree(before) + if err != nil { + return nil, nil, err + } + at, err := resolveTree(after) + if err != nil { + return nil, nil, err + } + same = bt == at + } + var omitted []string + // modified judges a path whose counterpart cp should end up as the + // original's postimage p1 of it. + modified := func(path, cp string, p0, p1 treeEntry) error { + be, inBefore := bef[cp] + ae, inAfter := aft[cp] + switch { + case !inBefore && !inAfter: + omitted = append(omitted, path) + return nil + case inBefore != inAfter || be != ae: + return nil + case be.oid == p1.oid && (p0.mode == p1.mode || be.mode == p1.mode): + return nil + case !be.regular() || !p0.regular() || !p1.regular() || p0.mode != p1.mode: + diag = append(diag, "the mode or type of "+strconv.Quote(path)+" changes, which only a person can judge") + return nil + } + applied, unsupported, err := b.alreadyApplied(p0.oid, be.oid, p1.oid) + switch { + case err != nil: + return fmt.Errorf("judging %s of #%d: %w", strconv.Quote(path), v.o.number, err) + case unsupported: + diag = append(diag, "git cannot merge "+strconv.Quote(path)+" as text, so whether the line has its change is not known") + case !applied: + omitted = append(omitted, path) + } + return nil + } + for _, c := range v.changes { + cp, ok := corr[c.path] + if !ok { + continue + } + switch { + case c.status == 'D' && movedTo[c.path] != "": + if err := modified(c.path, cp, pre[c.path], post[cp]); err != nil { + return nil, nil, err + } + case c.status == 'D': + // Absent before: the line already made the deletion, which is + // no change of a unit that makes others, and is all of one + // that makes none. Still there after: the deletion did not + // happen, whatever the unit changed in the file; keeping what + // the line made of it is a port of its own. + _, inBefore := bef[cp] + _, inAfter := aft[cp] + if !inBefore && !same || inAfter { + omitted = append(omitted, c.path) + } + case c.status == 'M': + if err := modified(c.path, cp, pre[c.path], post[c.path]); err != nil { + return nil, nil, err + } + } + } + for i := range diag { + diag[i] = oneLineText(diag[i]) + } + return omitted, diag, nil +} + +// alreadyApplied says whether cur, a file the unit leaves alone, already +// holds the original's change from pre to post: it is post itself, or git +// merge-file of the three merges cleanly back into cur. unsupported is +// content merge-file refuses, such as differing binary files, which says +// nothing either way. +func (b *batch) alreadyApplied(pre, cur, post string) (applied, unsupported bool, err error) { + if cur == post { + return true, false, nil + } + b.mergeFiles++ + blobs, err := readObjects([]string{pre, cur, post}, "blob") + if err != nil { + return false, false, err + } + dir, err := os.MkdirTemp("", "backport-batch-merge-file-*") + if err != nil { + return false, false, err + } + defer os.RemoveAll(dir) + names := map[string]string{} + for _, f := range []struct{ name, oid string }{{"cur", cur}, {"pre", pre}, {"post", post}} { + names[f.name] = filepath.Join(dir, f.name) + if err := os.WriteFile(names[f.name], blobs[f.oid], 0o600); err != nil { + return false, false, err + } + } + stdout, stderr, code, err := gitStatus("merge-file", "-p", names["cur"], names["pre"], names["post"]) + switch { + case err != nil: + return false, false, err + case code == 0: + return stdout == string(blobs[cur]), false, nil + case code > 0 && code < 128: + // The number of conflicts. + return false, false, nil + case strings.Contains(stderr, "Cannot merge binary files"): + return false, true, nil + } + return false, false, fmt.Errorf("git merge-file: exit %d: %s", code, strings.TrimSpace(stderr)) +} + +// unitsOn maps each original the batch head carries a unit of to that unit, +// by its Backported-from trailer or appended -x line: found needs every +// commit of a merged original, and misses one whose first pick came out +// empty because the line already had that change. +func (b *batch) unitsOn(head string) (map[*verdict]string, error) { + if m, ok := b.unitCache[head]; ok { + return m, nil + } + units, err := batchUnits(head, b.lineHead) + if err != nil { + return nil, err + } + m := map[*verdict]string{} + for _, u := range units { + if v := b.unitOriginal(u); v != nil && m[v] == "" { + m[v] = u.commit + } + } + if b.unitCache == nil { + b.unitCache = map[string]map[*verdict]string{} + } + b.unitCache[head] = m + return m, nil +} + +// onBranch says whether the batch head h carries v. +func (b *batch) onBranch(h *history, v *verdict) (bool, error) { + if found(h, v) { + return true, nil + } + m, err := b.unitsOn(h.head) + return m[v] != "", err +} + +// partialStates finds what applies only in part on the line. Every unit on +// the open batch is checked against its own first parent, whatever the rules +// now say of its original: one that leaves an effect out fails the cut until +// it is taken out. Every original the policy takes and nothing carries is +// checked against the tree its clean net merge onto the line writes, so a +// partial one is found with no record and no adder; a conflict is never +// called partial. +func (b *batch) partialStates() error { + b.partialUnits = map[*verdict][]string{} + if b.pr != nil { + var err error + if b.partialUnits, err = b.unitsPartial(b.prHead); err != nil { + return err + } + on, err := b.unitsOn(b.prHead) + if err != nil { + return err + } + for v, commit := range on { + if v.rule == 0 && v.onBatch == "" { + v.onBatch = commit + } + } + } + for _, v := range b.qualifying() { + if v.onBatch != "" { + continue + } + if err := b.partialAt(v, b.lineHead); err != nil { + return err + } + } + return nil +} + +// unitsPartial maps each original whose unit between the line's head and +// head leaves part of it without effect to what it leaves. +func (b *batch) unitsPartial(head string) (map[*verdict][]string, error) { + units, err := batchUnits(head, b.lineHead) + if err != nil { + return nil, err + } + out := map[*verdict][]string{} + for _, u := range units { + v := b.unitOriginal(u) + if v == nil || len(u.parents) == 0 { + continue + } + omitted, diag, err := b.omittedEffects(v, u.parents[0], u.commit) + if err != nil { + return nil, err + } + v.addDiag(diag...) + if len(omitted) > 0 { + out[v] = append(out[v], omitted...) + } + } + return out, nil +} + +// markPartial puts every original on the branch at head whose unit there +// does only part of it among the partial units, and so among the stale +// ones: still on the batch, and to be taken out before the cut. +func (b *batch) markPartial(r *report, head string) error { + units := b.partialUnits + if b.pr == nil || head != b.prHead { + var err error + if units, err = b.unitsPartial(head); err != nil { + return err + } + } + r.partialFiles = map[*verdict][]string{} + for _, v := range b.verdicts { + files := units[v] + if len(files) == 0 { + continue + } + r.partial = append(r.partial, v) + r.partialFiles[v] = files + if !slices.Contains(r.stale, v) { + r.stale = append(r.stale, v) + } + } + return nil +} + +// partialAt judges v's clean net merge onto head, and parks it, or clears +// the park, by what that merge leaves without effect. +func (b *batch) partialAt(v *verdict, head string) error { + tree, err := b.netTree(head, v.o) + if err != nil { + return err + } + v.partial = nil + if v.park != nil && v.park.Kind == parkPartial { + v.park = nil + } + if tree == "" { + return nil + } + omitted, diag, err := b.omittedEffects(v, head, tree) + if err != nil { + return err + } + v.addDiag(diag...) + if len(omitted) > 0 { + v.partial = omitted + v.park = &parkedEntry{Original: v.o.number, Merge: v.o.merge, Kind: parkPartial, Files: sortedClean(omitted)} + } + return nil +} + +// netTree is the tree a clean net merge of o onto head writes, "" for a +// conflict; the line's head reuses what judging the originals merged. +func (b *batch) netTree(head string, o *original) (string, error) { + if head == b.lineHead && b.lineHist != nil { + return b.lineHist.netTree(o) + } + tree, _, err := noopPick(head, o) + return tree, err +} + +// addDiag keeps what could not be judged of v, each line once. +func (v *verdict) addDiag(lines ...string) { + for _, ln := range lines { + if ln != "" && !slices.Contains(v.diag, ln) { + v.diag = append(v.diag, ln) + } + } +} + +func sortedClean(files []string) []string { + out := cleanFiles(files) + slices.Sort(out) + return slices.Compact(out) +} + +// leftOut is the refusal for a unit that leaves paths of v without effect. +func leftOut(v *verdict, l line, omitted []string) string { + quoted := make([]string, len(omitted)) + for i, p := range omitted { + quoted[i] = strconv.Quote(p) + } + return fmt.Sprintf("#%d leaves %s without effect on %s; port only what the line has as a pull request of its own", v.o.number, strings.Join(quoted, ", "), l.branch()) +} diff --git a/cmd/backport-batch/effects_test.go b/cmd/backport-batch/effects_test.go new file mode 100644 index 000000000..082050cd4 --- /dev/null +++ b/cmd/backport-batch/effects_test.go @@ -0,0 +1,350 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "os" + "path/filepath" + "slices" + "strings" + "testing" +) + +const appFile = "one\ntwo\nthree\nfour\nfive\n" + +// renameWorld is the sign-off's rename history: the line keeps app/old.txt, +// main renames it to app/new.txt with #3, a feat, and #5 fixes app/new.txt. +func renameWorld(t *testing.T) (*fixture, string) { + t.Helper() + f := newFixture(t) + f.squash(2, "chore: an app file", "", map[string]string{"app/old.txt": appFile}) + f.lines() + f.pr(3, "feat(app): rename the app file", "", map[string]string{"app/old.txt": "", "app/new.txt": appFile}) + mc5 := f.pr(5, "fix(app): fix the app file", "", map[string]string{"app/new.txt": strings.Replace(appFile, "three", "THREE", 1)}) + return f, mc5 +} + +// emptyBatch pushes the line's head as the batch branch and opens a batch +// pull request on it with body, for a fold to land on. +func emptyBatch(f *fixture, gh *fakeGH, body string) *foldWorld { + f.t.Helper() + f.git("fetch", "-q", "origin") + f.git("push", "-q", "origin", "origin/release-1.1:refs/heads/"+batchBranch1) + f.git("fetch", "-q", "origin") + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, body)} + return &foldWorld{f: f, gh: gh, head: f.git("rev-parse", "origin/"+batchBranch1)} +} + +// A fix of a file main renamed picks onto the file's old name on the line, +// and every check of the batch's shape takes the old name as the same file: +// verify, a hand fold, and a rebuild by --defer. +func TestRenamePickThroughFoldAndVerify(t *testing.T) { + f, mc5 := renameWorld(t) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + head := openBatch(t, f, gh) + if got := f.subjects("origin/release-1.1", head); !slices.Equal(got, []string{"fix(app): fix the app file (#5)", "fix(app): six (#6)"}) { + t.Fatalf("the batch carries %q, want #5 and #6", got) + } + if got := f.git("show", head+":app/old.txt"); !strings.Contains(got, "THREE") { + t.Fatalf("#5 did not land on app/old.txt: %q", got) + } + if code, out, stderr := f.exec(gh, "verify", "--line", "release-1.1", "--head", head); code != 0 { + t.Errorf("verify = %d\n%s\n%s", code, out, stderr) + } + t.Setenv("GIT_COMMITTER_DATE", "2026-10-27T06:00:00Z") + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "6", "--reason", "fails the e2e", "--push"); code != 0 { + t.Fatalf("run --defer 6 --push = %d\n%s", code, stderr) + } + if code, out, stderr := f.exec(gh, "verify", "--line", "release-1.1", "--head", f.remoteHead(batchBranch1)); code != 0 { + t.Errorf("verify after the rebuild = %d\n%s\n%s", code, out, stderr) + } + + f, mc5 = renameWorld(t) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/old.txt": strings.Replace(appFile, "three", "THREE", 1)}) + fold := w.merge(w.head, pick, "fix(app): fix the app file (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 0 || out != "ok\n" { + t.Errorf("fold of #5 onto app/old.txt = %d, %q\n%s", code, out, stderr) + } +} + +// shaped reads the line from the fixture's remote as verify does. +func shaped(t *testing.T, f *fixture) *batch { + t.Helper() + f.git("fetch", "-q", "origin") + b, err := shapeOf(line11) + if err != nil { + t.Fatal(err) + } + return b +} + +// commitOn writes files onto parent and returns the commit, leaving every +// branch where it was. +func commitOn(f *fixture, parent string, files map[string]string) string { + f.t.Helper() + f.git("switch", "-q", "--detach", parent) + f.apply(files) + f.git("commit", "-q", "--allow-empty", "-m", "a unit") + sha := f.git("rev-parse", "HEAD") + f.git("switch", "-q", "main") + return sha +} + +// A path of the original is followed to where the line keeps the same file: +// through a rename on main, on the line, or both, composed through the path +// at the base. Where git skipped rename detection the path is unknown, never +// absent. +func TestCorrespondence(t *testing.T) { + corr := func(t *testing.T, f *fixture, n int, limit ...int) (map[string]string, []string) { + t.Helper() + b := shaped(t, f) + if len(limit) > 0 { + b.renameLimit = limit[0] + } + v := b.find(n) + if err := b.load(v); err != nil { + t.Fatal(err) + } + c, unknown, err := b.correspondence(v, b.lineHead) + if err != nil { + t.Fatal(err) + } + return c, unknown + } + t.Run("renamed on main", func(t *testing.T) { + f, _ := renameWorld(t) + if c, unknown := corr(t, f, 5); c["app/new.txt"] != "app/old.txt" || len(unknown) != 0 { + t.Errorf("correspondence = %v, %v; want app/new.txt on app/old.txt", c, unknown) + } + }) + t.Run("renamed on the line", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: an app file", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.onLine("release-1.1", "fix(app): the line renames it", map[string]string{"app/a.txt": "", "app/line.txt": appFile}) + f.pr(5, "fix(app): fix it", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + if c, unknown := corr(t, f, 5); c["app/a.txt"] != "app/line.txt" || len(unknown) != 0 { + t.Errorf("correspondence = %v, %v; want app/a.txt on app/line.txt", c, unknown) + } + }) + t.Run("renamed on both", func(t *testing.T) { + f, _ := renameWorld(t) + f.onLine("release-1.1", "fix(app): the line renames it its own way", map[string]string{"app/old.txt": "", "app/release.txt": appFile}) + if c, unknown := corr(t, f, 5); c["app/new.txt"] != "app/release.txt" || len(unknown) != 0 { + t.Errorf("correspondence = %v, %v; want app/new.txt on app/release.txt", c, unknown) + } + }) + t.Run("not renamed, absent", func(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(3, "feat(app): x", "", map[string]string{"app/x.txt": appFile}) + f.pr(5, "fix(app): fix x", "", map[string]string{"app/x.txt": strings.Replace(appFile, "two", "TWO", 1)}) + if c, unknown := corr(t, f, 5); c["app/x.txt"] != "app/x.txt" || len(unknown) != 0 { + t.Errorf("correspondence = %v, %v; want app/x.txt on itself", c, unknown) + } + }) + t.Run("rename detection skipped", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: two app files", "", map[string]string{"app/a.txt": appFile, "app/b.txt": appFile + "six\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): the line renames both, changed", map[string]string{"app/a.txt": "", "app/b.txt": "", "app/x.txt": appFile + "x\n", "app/y.txt": appFile + "six\ny\n"}) + f.pr(5, "fix(app): fix a", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + // Past the limit, git says it skipped looking; a configured limit + // of its own does not decide for the tool. + f.git("config", "diff.renameLimit", "1") + if c, unknown := corr(t, f, 5, 0); c["app/a.txt"] != "app/x.txt" || len(unknown) != 0 { + t.Errorf("correspondence under diff.renameLimit=1 = %v, %v; want app/a.txt on app/x.txt", c, unknown) + } + if c, unknown := corr(t, f, 5, 1); !slices.Equal(unknown, []string{"app/a.txt"}) || c["app/a.txt"] != "" { + t.Errorf("correspondence past the limit = %v, %v; want app/a.txt unknown", c, unknown) + } + }) +} + +// What a change leaves without effect: a modified file the line lacks on both +// sides of it, a deleted file the line never had. A mode change that only a +// person can judge is a diagnostic, not an omission. +func TestOmittedEffects(t *testing.T) { + omitted := func(t *testing.T, f *fixture, n int, files map[string]string) ([]string, []string) { + t.Helper() + b := shaped(t, f) + after := commitOn(f, b.lineHead, files) + om, diag, err := b.omittedEffects(b.find(n), b.lineHead, after) + if err != nil { + t.Fatal(err) + } + return om, diag + } + t.Run("a modified file missing before and after", func(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(3, "feat(app): x", "", map[string]string{"app/x.txt": appFile}) + f.pr(5, "fix(app): fix x and five", "", map[string]string{"app/x.txt": "fixed\n", "app/five.txt": "5\n"}) + if om, diag := omitted(t, f, 5, map[string]string{"app/five.txt": "5\n"}); !slices.Equal(om, []string{"app/x.txt"}) || len(diag) != 0 { + t.Errorf("omitted = %q, %q; want app/x.txt", om, diag) + } + }) + t.Run("a deleted file absent before", func(t *testing.T) { + f, _ := policyWorld(t) + if om, _ := omitted(t, f, 5, map[string]string{"chart/tests/values_test.yaml": "suite: values\n"}); !slices.Equal(om, []string{"chart/templates/policy.yaml", "chart/tests/policy_test.yaml"}) { + t.Errorf("omitted = %q; want both policy files", om) + } + }) + t.Run("a mode change", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a script", "", map[string]string{"app/run.sh": "#!/bin/sh\n", "app/b.txt": "old\n"}) + f.lines() + f.git("switch", "-q", "-c", "pr-5") + if err := os.Chmod(filepath.Join(f.dir, "app/run.sh"), 0o755); err != nil { + t.Fatal(err) + } + f.write("app/b.txt", "fixed\n") + f.git("add", "-A") + f.git("commit", "-q", "-m", "run it") + f.git("switch", "-q", "main") + f.git("merge", "-q", "--no-ff", "-m", "fix(app): make the script runnable (#5)", "pr-5") + f.git("push", "-q", "origin", "main") + om, diag := omitted(t, f, 5, map[string]string{"app/b.txt": "fixed\n"}) + if len(om) != 0 || len(diag) != 1 || !strings.Contains(diag[0], `the mode or type of "app/run.sh" changes`) { + t.Errorf("omitted = %q, %q; want a diagnostic for app/run.sh and nothing omitted", om, diag) + } + }) +} + +// fixedWorld is the sign-off's two-file case: the base has a and b as old, +// #5 fixes both, and the line already has a as onLine says, if anything. +func fixedWorld(t *testing.T, a, old, onLine, fixed string) (*fixture, string) { + t.Helper() + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{a: old, "app/b.txt": "old\n"}) + f.lines() + if onLine != "" { + f.onLine("release-1.1", "fix(app): a, on the line", map[string]string{a: onLine}) + } + mc5 := f.pr(5, "fix(app): a and b", "", map[string]string{a: fixed, "app/b.txt": "fixed\n"}) + return f, mc5 +} + +// The line can already have what an original does to a file: then a pick +// that leaves the file alone carries the original whole. The check-time +// pass, the picker, fold and verify all take it, binary files included, +// without asking git merge-file what blob equality settles. A different fix +// on the line, or a deletion of what the line never had, is still left out. +func TestAlreadyAppliedFileIsNotOmitted(t *testing.T) { + const textPost = "one\nfixed\nthree\nfour\nfive\n" + accepted := func(t *testing.T, a, old, onLine, fixed string) { + t.Helper() + f, _ := fixedWorld(t, a, old, onLine, fixed) + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/5 ") || strings.Contains(stderr, "parked") { + t.Errorf("check = %d, want #5 missing and not parked:\n%s\n%s", code, stdout, stderr) + } + gh := &fakeGH{} + head := openBatch(t, f, gh) + if got := f.subjects("origin/release-1.1", head); !slices.Equal(got, []string{"fix(app): a and b (#5)"}) { + t.Errorf("picked %q, want #5", got) + } + if code, out, stderr := f.exec(gh, "verify", "--line", "release-1.1", "--head", head); code != 0 { + t.Errorf("verify = %d\n%s\n%s", code, out, stderr) + } + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || strings.Contains(stdout, "only part of it applies") { + t.Errorf("check --cut = %d, want #5 only on the batch, whole:\n%s\n%s", code, stdout, stderr) + } + f, mc5 := fixedWorld(t, a, old, onLine, fixed) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/b.txt": "fixed\n"}) + fold := w.merge(w.head, pick, "fix(app): a and b (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 0 { + t.Errorf("fold = %d, %q\n%s", code, out, stderr) + } + } + t.Run("the line has the original's file", func(t *testing.T) { + accepted(t, "app/a.txt", "old\n", "fixed\n", "fixed\n") + }) + t.Run("the line has the fix beside lines of its own", func(t *testing.T) { + accepted(t, "app/a.txt", strings.Replace(textPost, "fixed", "old", 1), textPost+"line only\n", textPost) + }) + t.Run("the line has the original's binary file", func(t *testing.T) { + accepted(t, "app/a.bin", "old\x00payload", "fixed\x00payload", "fixed\x00payload") + f, _ := fixedWorld(t, "app/a.bin", "old\x00payload", "fixed\x00payload", "fixed\x00payload") + b := shaped(t, f) + om, diag, err := b.omittedEffects(b.find(5), b.lineHead, commitOn(f, b.lineHead, map[string]string{"app/b.txt": "fixed\n"})) + if err != nil || len(om)+len(diag) != 0 || b.mergeFiles != 0 { + t.Errorf("omitted = %q, %q, %v, merge-file ran %d times; want nothing, and merge-file never asked", om, diag, err, b.mergeFiles) + } + }) + t.Run("an earlier unit of the batch brought the file", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{"app/a.txt": "old\n", "app/b.txt": "old\n"}) + f.lines() + f.pr(4, "fix(app): a", "", map[string]string{"app/a.txt": "fixed\n"}) + f.squash(40, "chore: put a back for now", "", map[string]string{"app/a.txt": "old\n"}) + f.pr(5, "fix(app): a and b", "", map[string]string{"app/a.txt": "fixed\n", "app/b.txt": "fixed\n"}) + gh := &fakeGH{} + head := openBatch(t, f, gh) + if got := f.subjects("origin/release-1.1", head); !slices.Equal(got, []string{"fix(app): a (#4)", "fix(app): a and b (#5)"}) { + t.Errorf("picked %q, want #4 and #5", got) + } + if code, out, stderr := f.exec(gh, "verify", "--line", "release-1.1", "--head", head); code != 0 { + t.Errorf("verify = %d\n%s\n%s", code, out, stderr) + } + }) + t.Run("the line has a different fix", func(t *testing.T) { + f, mc5 := fixedWorld(t, "app/a.txt", "old\n", "other\n", "fixed\n") + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/b.txt": "fixed\n"}) + fold := w.merge(w.head, pick, "fix(app): a and b (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 1 || !strings.Contains(out, `leaves "app/a.txt" without effect`) { + t.Errorf("fold = %d, %q, want it refused for app/a.txt\n%s", code, out, stderr) + } + }) + t.Run("the line has a different binary file", func(t *testing.T) { + f, _ := fixedWorld(t, "app/a.bin", "old\x00payload", "other\x00payload", "fixed\x00payload") + b := shaped(t, f) + om, diag, err := b.omittedEffects(b.find(5), b.lineHead, commitOn(f, b.lineHead, map[string]string{"app/b.txt": "fixed\n"})) + if err != nil || len(om) != 0 || len(diag) != 1 || !strings.Contains(diag[0], `cannot merge "app/a.bin" as text`) { + t.Errorf("omitted = %q, %q, %v; want a diagnostic for app/a.bin, nothing omitted", om, diag, err) + } + }) + t.Run("a deletion of what the line never had", func(t *testing.T) { + f, mc5 := policyWorld(t) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"chart/tests/values_test.yaml": "suite: values\n"}) + fold := w.merge(w.head, pick, "fix(chart): remove the policy a rebase brought back (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 1 || !strings.Contains(out, "without effect") { + t.Errorf("fold = %d, %q, want it refused\n%s", code, out, stderr) + } + }) +} + +// A change that makes none, whatever names its two sides, leaves a deletion +// the line already made as it is: the original is there, not partial. +func TestDeletionAlreadyMadeByNoChange(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): the line drops the policy", map[string]string{"app/policy.yaml": ""}) + f.pr(5, "fix(app): drop the policy", "", map[string]string{"app/policy.yaml": ""}) + b := shaped(t, f) + tree := f.git("rev-parse", b.lineHead+"^{tree}") + for _, after := range []string{b.lineHead, tree} { + if om, _, err := b.omittedEffects(b.find(5), b.lineHead, after); err != nil || len(om) != 0 { + t.Errorf("omitted against %.12s = %q, %v; want nothing", after, om, err) + } + } +} diff --git a/cmd/backport-batch/fold_test.go b/cmd/backport-batch/fold_test.go index 89797d0d5..5c064c2e3 100644 --- a/cmd/backport-batch/fold_test.go +++ b/cmd/backport-batch/fold_test.go @@ -348,3 +348,59 @@ func TestFoldRefusesADroppedOriginal(t *testing.T) { t.Errorf("fold = %d, %q, %s; want a refusal naming the drop", code, stdout, stderr) } } + +// A resolution that ports what the line has of #5 and leaves alone a file +// the line lacks is no fold: what #5 does to that file would be lost, and a +// line-only port is a pull request of its own. +func TestFoldRefusesAnOmittedEffect(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(3, "feat(app): a feature the line never gets", "", map[string]string{"app/feature2.txt": "v1\n"}) + mc5 := f.pr(5, "fix(app): five", "", map[string]string{"app/feature2.txt": "v2\n", "app/five.txt": "5\n"}) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/five.txt": "5\n"}) + fold := w.merge(w.head, pick, "fix(app): five (#5)\n\n"+batchTrailer+": "+mc5) + code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold) + if code != 1 || out != "refused: #5 leaves \"app/feature2.txt\" without effect on release-1.1; port only what the line has as a pull request of its own\n" { + t.Errorf("fold = %d, %q, want it refused for app/feature2.txt\n%s", code, out, stderr) + } +} + +// A resolution that keeps a file the original deletes leaves the deletion +// without effect. +func TestFoldRefusesAKeptDeletion(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: x", "", map[string]string{"app/x.txt": "x\n"}) + f.lines() + mc5 := f.pr(5, "fix(app): five, and x goes", "", map[string]string{"app/x.txt": "", "app/five.txt": "5\n"}) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/five.txt": "5\n"}) + fold := w.merge(w.head, pick, "fix(app): five, and x goes (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 1 || !strings.Contains(out, `leaves "app/x.txt" without effect`) { + t.Errorf("fold that keeps app/x.txt = %d, %q\n%s", code, out, stderr) + } +} + +// A fold that keeps a file the original deletes leaves the deletion without +// effect, whatever it changes in the file: a port that keeps what the line +// made of it, or the file with a line added, is a pull request of its own. +func TestFoldRefusesAChangedKeptDeletion(t *testing.T) { + for name, kept := range map[string]string{ + "the line's own content": "line only\n", + "the file, one line more": "feature\nline only\n# reviewed\n", + } { + t.Run(name, func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a feature", "", map[string]string{"app/x.txt": "feature\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): a line-only setting beside the feature", map[string]string{"app/x.txt": "feature\nline only\n"}) + mc5 := f.pr(5, "fix(app): drop the feature, and five", "", map[string]string{"app/x.txt": "", "app/five.txt": "5\n"}) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/x.txt": kept, "app/five.txt": "5\n"}) + fold := w.merge(w.head, pick, "fix(app): drop the feature, and five (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 1 || !strings.Contains(out, `leaves "app/x.txt" without effect`) { + t.Errorf("fold = %d, %q, want it refused\n%s", code, out, stderr) + } + }) + } +} diff --git a/cmd/backport-batch/history.go b/cmd/backport-batch/history.go index 74f4c66af..a530aa244 100644 --- a/cmd/backport-batch/history.go +++ b/cmd/backport-batch/history.go @@ -205,6 +205,9 @@ type history struct { picks []pickRef batches map[string]string backports map[int]string + // trees caches what a net merge of an original onto head writes, by its + // merge commit: "" for a conflict. + trees map[string]string } func readHistory(head, base string, textUntil time.Time) (*history, error) { @@ -225,7 +228,7 @@ func readHistory(head, base string, textUntil time.Time) (*history, error) { // readPicks reads only the messages of head's commits outside base, which is // all a batch branch is asked about; textUntil is the batch-only day. func readPicks(head, base string, textUntil time.Time) (*history, error) { - h := &history{head: head, reachable: map[string]bool{}, appended: map[string]string{}, batches: map[string]string{}, backports: map[int]string{}} + h := &history{head: head, reachable: map[string]bool{}, appended: map[string]string{}, batches: map[string]string{}, backports: map[int]string{}, trees: map[string]string{}} oids, err := revList("--end-of-options", head, "^"+base) if err != nil { return nil, err @@ -332,34 +335,54 @@ func (h *history) carries(o *original, commits []string) (string, error) { if c, ok := h.backports[o.number]; ok { return `named by "Backport of" in ` + c[:12], nil } - noop, err := noopPick(h.head, o) + tree, err := h.netTree(o) if err != nil { return "", err } - if noop { + headTree, err := resolveTree(h.head) + if err != nil { + return "", err + } + if tree == headTree { return "a pick would change nothing", nil } return "", nil } -// noopPick reports whether cherry-picking the original onto head would leave -// head's tree as it is: the content is there, whatever names it. -func noopPick(head string, o *original) (bool, error) { +// netTree is noopPick's tree for o onto the branch's head, read once. +func (h *history) netTree(o *original) (string, error) { + if tree, ok := h.trees[o.merge]; ok { + return tree, nil + } + tree, _, err := noopPick(h.head, o) + if err == nil { + h.trees[o.merge] = tree + } + return tree, err +} + +// noopPick merges the original onto head as one change and returns the tree +// that writes for a clean merge, "" for a conflict, and whether the tree is +// head's own: the content is there, whatever names it. +func noopPick(head string, o *original) (string, bool, error) { stdout, stderr, code, err := gitStatus("merge-tree", "--write-tree", "--no-messages", "--merge-base="+o.merge+"^1", "--end-of-options", head, o.merge) switch { case err != nil: - return false, err + return "", false, err case code == 1: - return false, nil + return "", false, nil case code != 0: - return false, fmt.Errorf("git merge-tree for #%d: exit %d: %s", o.number, code, strings.TrimSpace(stderr)) + return "", false, fmt.Errorf("git merge-tree for #%d: exit %d: %s", o.number, code, strings.TrimSpace(stderr)) } tree, _, _ := strings.Cut(stdout, "\n") + if err := checkSHA(tree); err != nil { + return "", false, fmt.Errorf("git merge-tree for #%d: %w", o.number, err) + } headTree, err := resolveTree(head) if err != nil { - return false, err + return "", false, err } - return tree == headTree, nil + return tree, tree == headTree, nil } func resolveTree(commit string) (string, error) { diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index c02601240..161df97b9 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -211,6 +211,13 @@ type verdict struct { onBatch string // label is the maintainer override the original carries, if any. label string + // partial lists what its clean net merge onto the line would leave + // without effect, and park why the batch leaves it out, if it does. + partial []string + park *parkedEntry + // diag is what could not be judged about it, shown whatever became of + // it. + diag []string } // batch is everything read about a line before anything is written. @@ -242,6 +249,16 @@ type batch struct { // textUntil is the batch-only day: what a commit's text says counts as // evidence only on commits from before it. textUntil time.Time + // partialUnits are the originals whose unit on the open batch leaves + // what it lists without effect. + partialUnits map[*verdict][]string + // What one command read once and asks again: renames by commit pair, + // units by batch head, and how often git merge-file ran. + renameCache map[string]renameSet + unitCache map[string]map[*verdict]string + mergeFiles int + // renameLimit is the -l rename detection runs with; 0, git's widest. + renameLimit int } // fetch moves to the top of the working tree first: every path the batch @@ -355,6 +372,9 @@ func (a *app) prepare(l line) (*batch, error) { } b.verdicts = append(b.verdicts, v) } + if err := b.partialStates(); err != nil { + return nil, err + } return b, nil } diff --git a/cmd/backport-batch/park_test.go b/cmd/backport-batch/park_test.go new file mode 100644 index 000000000..6c8f51d0a --- /dev/null +++ b/cmd/backport-batch/park_test.go @@ -0,0 +1,331 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "slices" + "strings" + "testing" +) + +var line11, _ = parseLine("release-1.1") + +// handBatch pushes a batch carrying merges in the batch's shape, as a run +// made one before the partial guard, and opens its pull request with a +// record that defers nothing. +func handBatch(f *fixture, gh *fakeGH, merges ...string) string { + f.t.Helper() + f.git("fetch", "-q", "origin") + f.git("switch", "-q", "--detach", "origin/release-1.1") + for _, mc := range merges { + if parents := strings.Fields(f.git("rev-list", "--parents", "-n", "1", mc)); len(parents) == 2 { + f.git("cherry-pick", "-x", mc) + continue + } + base := f.git("rev-parse", "HEAD") + for _, c := range strings.Fields(f.git("rev-list", "--reverse", "--no-merges", mc+"^1.."+mc+"^2")) { + f.git("cherry-pick", "-x", c) + } + merge := f.joined(base, "HEAD", f.git("log", "-1", "--format=%B", mc)+"\n\n"+batchTrailer+": "+mc) + f.git("update-ref", "HEAD", merge) + } + f.git("push", "-q", "origin", "HEAD:refs/heads/"+batchBranch1) + f.git("switch", "-q", "main") + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, "batch\n\n"+deferredBlock(line11, nil, nil, nil)+"\n")} + return f.remoteHead(batchBranch1) +} + +// policyWorld is #5088's shape: #3, a feat the line never gets, adds a +// policy and its test; #5 deletes both again and adds a test of its own. +func policyWorld(t *testing.T) (*fixture, string) { + t.Helper() + f := newFixture(t) + f.lines() + f.pr(3, "feat(chart): a network policy", "", map[string]string{"chart/templates/policy.yaml": "kind: NetworkPolicy\n", "chart/tests/policy_test.yaml": "suite: policy\n"}) + mc5 := f.pr(5, "fix(chart): remove the policy a rebase brought back", "```release-note\nfix(chart): the stray network policy is gone.\n```\n", + map[string]string{"chart/templates/policy.yaml": "", "chart/tests/policy_test.yaml": "", "chart/tests/values_test.yaml": "suite: values\n"}) + return f, mc5 +} + +// A clean pick that would delete files the line never had, and so carry only +// #5's test, is withheld: #5 is parked as partial, with the paths it would +// leave without effect, and its release note stays out of the body. +func TestPartialPickIsWithheld(t *testing.T) { + f, mc5 := policyWorld(t) + f.pr(6, "fix(app): six", "```release-note\nfix(app): six works.\n```\n", map[string]string{"app/six.txt": "6\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(app): six (#6)"}) { + t.Errorf("picked %q, want only #6", got) + } + want := `"parked":[{"original":5,"merge":"` + mc5 + `","kind":"partial","files":["chart/templates/policy.yaml","chart/tests/policy_test.yaml"]}]` + if !strings.Contains(stdout, want) { + t.Errorf("the record does not park #5 as partial (%s):\n%s", want, stdout) + } + if !strings.Contains(stdout, "### Parked") || !strings.Contains(stdout, "`chart/templates/policy.yaml`") { + t.Errorf("the body does not say #5 is parked and why:\n%s", stdout) + } + if strings.Contains(stdout, "stray network policy") || !strings.Contains(stdout, "six works") { + t.Errorf("the release notes are not #6's alone:\n%s", stdout) + } +} + +// A file the line removed itself, which no original added after the line +// branched, still leaves #5's deletion without effect: check finds it with no +// record and parks it, and run picks nothing. +func TestPartialWithoutAnAdder(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: add a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): the line drops the policy", map[string]string{"app/policy.yaml": ""}) + f.pr(5, "fix(app): drop the policy", "", map[string]string{"app/policy.yaml": "", "app/policy_test.txt": "gone\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); len(got) != 0 { + t.Errorf("picked %q, want nothing", got) + } + if !strings.Contains(stdout, "### Parked") { + t.Errorf("the body does not park #5:\n%s", stdout) + } + f.git("switch", "-q", "main") + code, stdout, stderr = f.exec(nil, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "note: #5 is parked: only part of it applies to release-1.1") { + t.Errorf("check = %d, want 0 with a parked-partial note:\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } +} + +// The picker judges what its picks did, not what the original's merge did: +// #5's first commit deletes a file the line no longer has and comes out +// empty, its second adds a test, and the unit left would be the test alone. +func TestPartialByAnEmptyOwnCommit(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: add a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): the line drops the policy", map[string]string{"app/policy.yaml": ""}) + f.prCommits(5, "fix(app): drop the policy and test it", "```release-note\nfix(app): no more policy.\n```\n", + change{files: map[string]string{"app/policy.yaml": ""}}, + change{files: map[string]string{"app/policy_test.txt": "gone\n"}}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); len(got) != 0 { + t.Errorf("picked %q, want #5 withheld", got) + } + if strings.Contains(stdout, "no more policy") || !strings.Contains(stdout, "### Parked") { + t.Errorf("#5's note is in the body, or #5 is not parked:\n%s", stdout) + } +} + +// A batch built before the guard can already carry a partial unit. It fails +// the cut and verify whatever its label says, and --defer takes it off, +// closing a batch it was the last unit of. +func TestExistingPartialUnitFailsTheCut(t *testing.T) { + f, mc5 := policyWorld(t) + gh := &fakeGH{} + head := handBatch(f, gh, mc5) + code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut") + if code != 1 || !strings.Contains(stdout, "#5 is on batch #90, but only part of it applies to release-1.1") { + t.Errorf("check --cut = %d, want 1 naming the partial unit:\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } + code, stdout, stderr = f.exec(gh, "verify", "--line", "release-1.1", "--head", head) + if code != 1 || !strings.Contains(stdout, `#5 leaves "chart/templates/policy.yaml", "chart/tests/policy_test.yaml" without effect on release-1.1`) { + t.Errorf("verify = %d, want 1 naming what #5 leaves out:\n%s\n%s", code, stdout, stderr) + } + gh.labels = map[string][]int{labelSkip: {5}} + code, stdout, stderr = f.exec(gh, "check", "--line", "release-1.1", "--cut") + if code != 1 || !strings.Contains(stdout, "#5 is on batch #90, but only part of it applies to release-1.1") { + t.Errorf("check --cut with backport/skip = %d, want the partial unit to still fail it:\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } + code, _, stderr = f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "only part of it applies", "--push") + if code != 0 || prState(gh.find(90)) != "closed" || f.remoteHead(batchBranch1) != "" { + t.Errorf("run --defer 5 --push = %d, #90 %s, branch at %q; want the batch closed and its branch gone:\n%s", code, prState(gh.find(90)), f.remoteHead(batchBranch1), stderr) + } +} + +// An original whose first pick came out empty, because the line already had +// that change, is on the batch all the same: check does not call it missing, +// a continuation keeps it picked, and so does a rebuild without another +// original. +func TestOnBatchAfterAnEmptyFirstPick(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{"app/a.txt": "old\n", "app/b.txt": "old\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): a, on the line first", map[string]string{"app/a.txt": "fixed\n"}) + f.prCommits(5, "fix(app): a and b", "", + change{files: map[string]string{"app/a.txt": "fixed\n"}}, + change{files: map[string]string{"app/b.txt": "fixed\n"}}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + head := openBatch(t, f, gh) + if got := f.subjects("origin/release-1.1", head); !slices.Equal(got, []string{"fix(app): a and b (#5)", "fix(app): six (#6)"}) { + t.Fatalf("the batch carries %q, want #5 and #6", got) + } + code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1") + if code != 0 { + t.Errorf("check = %d, want 0 with #5 on the batch:\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } + code, stdout, _ = f.exec(gh, "check", "--line", "release-1.1", "--cut") + if code != 1 || !strings.Contains(stdout, "#5 is only on open batch PR #90") { + t.Errorf("check --cut = %d, want #5 only on the batch:\n%s", code, stdout) + } + code, stdout, stderr = f.exec(gh, "run", "--line", "release-1.1", "--continue") + if code != 0 || !strings.Contains(stdout, `"picked":[5,6]`) { + t.Errorf("run --continue = %d, want #5 and #6 picked:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + code, stdout, stderr = f.exec(gh, "run", "--line", "release-1.1", "--defer", "6", "--reason", "fails the e2e") + if code != 0 || !strings.Contains(stdout, `"picked":[5]`) || !strings.Contains(stdout, `"missing":[]`) { + t.Errorf("run --defer 6 = %d, want #5 still picked:\n%s\n%s", code, stdout, stderr) + } +} + +// The picker judges what its picks did where the run has got to: #4 +// deletes the file #5 also deletes, so after #4 #5 would bring only its +// test, which the net merge at the line's head cannot see. +func TestPartialByAnEarlierPick(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.pr(4, "fix(app): drop the policy", "", map[string]string{"app/policy.yaml": "", "app/four.txt": "4\n"}) + f.pr(40, "feat(app): the policy again", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.pr(5, "fix(app): drop the policy and test it", "", map[string]string{"app/policy.yaml": "", "app/policy_test.txt": "gone\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, `"picked":[4]`) || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("run = %d, want #4 picked and #5 parked as partial:\n%s\n%s", code, stdout, stderr) + } +} + +// What an earlier pick brought clears a partial park the line's head +// gave: the line removed the policy, #4 brings it back, and #5's deletion +// then has something to delete. +func TestPartialClearedByAnEarlierPick(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): the line drops the policy", map[string]string{"app/policy.yaml": ""}) + f.pr(3, "feat(app): no policy", "", map[string]string{"app/policy.yaml": ""}) + f.pr(4, "fix(app): the policy, back", "", map[string]string{"app/policy.yaml": "allow: some\n"}) + f.pr(5, "fix(app): drop the policy and test it", "", map[string]string{"app/policy.yaml": "", "app/policy_test.txt": "gone\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, `"picked":[4,5]`) { + t.Errorf("run = %d, want #4 and #5 picked:\n%s\n%s", code, stdout, stderr) + } +} + +// A file the line renamed the way the original does is the original's own +// rename, not a deletion left without effect. +func TestSameRenameOnBothSidesIsWhole(t *testing.T) { + const long = "l01\nl02\nl03\nl04\nl05\nl06\nl07\nl08\nl09\nl10\nl11\nl12\n" + f := newFixture(t) + f.squash(2, "chore: a file", "", map[string]string{"app/p.txt": long}) + f.lines() + f.onLine("release-1.1", "fix(app): the line renames p to q", map[string]string{"app/p.txt": "", "app/q.txt": long}) + f.pr(5, "fix(app): rename p to q, and fix l10", "", map[string]string{"app/p.txt": "", "app/q.txt": strings.Replace(long, "l10", "L10", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, `"picked":[5]`) { + t.Errorf("run = %d, want #5 picked whole:\n%s\n%s", code, stdout, stderr) + } +} + +// The line's own rename stands for the original's only with the original's +// change in it: a fold that leaves the renamed file as it was is refused. +func TestSameRenameStillNeedsItsFix(t *testing.T) { + const long = "l01\nl02\nl03\nl04\nl05\nl06\nl07\nl08\nl09\nl10\nl11\nl12\n" + f := newFixture(t) + f.squash(2, "chore: a file", "", map[string]string{"app/p.txt": long}) + f.lines() + f.onLine("release-1.1", "fix(app): the line renames p to q", map[string]string{"app/p.txt": "", "app/q.txt": long}) + mc5 := f.pr(5, "fix(app): rename p to q, fix l10, and five", "", map[string]string{"app/p.txt": "", "app/q.txt": strings.Replace(long, "l10", "L10", 1), "app/five.txt": "5\n"}) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/five.txt": "5\n"}) + fold := w.merge(w.head, pick, "fix(app): rename p to q, fix l10, and five (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 1 || !strings.Contains(out, `leaves "app/p.txt" without effect`) { + t.Errorf("fold without the fix in q = %d, %q\n%s", code, out, stderr) + } +} + +// A pick that comes out empty is judged too: #5's own commit did half its +// fix, which the line has, and a merge of main into its branch did the +// rest, which no pick carries; #5 is not on the line. +func TestEmptyPicksStillJudged(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: b", "", map[string]string{"app/b.txt": "old\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): b, half of it", map[string]string{"app/b.txt": "middle\n"}) + f.git("switch", "-q", "-c", "pr-5", "main") + f.apply(map[string]string{"app/b.txt": "middle\n"}) + f.git("commit", "-q", "-m", "b, half of it") + f.git("switch", "-q", "main") + f.apply(map[string]string{"app/main.txt": "moved\n"}) + f.git("commit", "-q", "-m", "chore: move main along") + f.git("switch", "-q", "pr-5") + f.git("merge", "-q", "--no-ff", "--no-commit", "main") + f.apply(map[string]string{"app/b.txt": "fixed\n"}) + f.git("commit", "-q", "-m", "Merge branch 'main' into pr-5") + f.git("switch", "-q", "main") + f.git("merge", "-q", "--no-ff", "-m", "fix(app): b (#5)", "pr-5") + f.git("push", "-q", "origin", "main") + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stdout, `"on-line":[5]`) || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("run = %d, want #5 parked as partial, not on the line:\n%s\n%s", code, stdout, stderr) + } +} + +// --defer takes off a unit that does only part of its original whatever +// rule now keeps that original out, as check tells it to. +func TestDeferTakesOffAPartialExcludedUnit(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(3, "feat(chart): a network policy", "", map[string]string{"chart/templates/policy.yaml": "kind: NetworkPolicy\n"}) + mc5 := f.pr(5, "feat(chart): remove the policy, and test values", "", map[string]string{"chart/templates/policy.yaml": "", "chart/tests/values_test.yaml": "suite: values\n"}) + gh := &fakeGH{} + handBatch(f, gh, mc5) + if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "only part of it applies") { + t.Fatalf("check --cut = %d, want the partial unit to fail it:\n%s", code, stdout) + } + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "only part of it applies", "--push"); code != 0 || prState(gh.find(90)) != "closed" { + t.Errorf("run --defer 5 --push = %d, #90 %s; want the unit taken off and the batch closed:\n%s", code, prState(gh.find(90)), stderr) + } +} + +// An empty pick leaves a deletion undone that only a merge of main into the +// original's branch made: the original is not on the line. +func TestEmptyPickLeavesADeletionUndone(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{"app/a.txt": "old feature\n", "app/b.txt": "old\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): b, already", map[string]string{"app/b.txt": "fixed\n"}) + f.git("switch", "-q", "-c", "pr-5", "main") + f.apply(map[string]string{"app/b.txt": "fixed\n"}) + f.git("commit", "-q", "-m", "b") + f.git("switch", "-q", "main") + f.apply(map[string]string{"app/main.txt": "moved\n"}) + f.git("commit", "-q", "-m", "chore: move main along") + f.git("switch", "-q", "pr-5") + f.git("merge", "-q", "--no-ff", "--no-commit", "main") + f.git("rm", "-q", "app/a.txt") + f.git("commit", "-q", "-m", "Merge branch 'main' into pr-5") + f.git("switch", "-q", "main") + f.git("merge", "-q", "--no-ff", "-m", "fix(app): b, and a goes (#5)", "pr-5") + f.git("push", "-q", "origin", "main") + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stdout, `"on-line":[5]`) || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("run = %d, want #5 parked as partial, not on the line:\n%s\n%s", code, stdout, stderr) + } +} diff --git a/cmd/backport-batch/record.go b/cmd/backport-batch/record.go index 4ab43933e..313279670 100644 --- a/cmd/backport-batch/record.go +++ b/cmd/backport-batch/record.go @@ -34,7 +34,8 @@ import ( // carries a record of its own. // lineRecord is the one line of JSON in the block. Version 1, the block -// before it had a version, holds only line and deferred. +// before it had a version, holds only line and deferred; version 3 adds what +// is parked and which units on the batch do only part of their original. type lineRecord struct { Version int `json:"version"` Line string `json:"line"` @@ -44,6 +45,80 @@ type lineRecord struct { Deferred []deferredEntry `json:"deferred"` Dropped []droppedEntry `json:"dropped"` Stale []int `json:"stale"` + Parked []parkedEntry `json:"parked,omitempty"` + Partial []int `json:"partial,omitempty"` +} + +// recordVersion is the version a run writes. +const recordVersion = 3 + +// lineRecordV2 is version 2, which has none of version 3's keys. +type lineRecordV2 struct { + Version int `json:"version"` + Line string `json:"line"` + Built *builtFrom `json:"built,omitempty"` + Report string `json:"report,omitempty"` + Closing *closing `json:"closing"` + Deferred []deferredEntry `json:"deferred"` + Dropped []droppedEntry `json:"dropped"` + Stale []int `json:"stale"` +} + +// The kinds of park: an original that builds on what never reaches the +// line, and one whose clean pick would leave part of it without effect. +const ( + parkDependency = "dependency" + parkPartial = "partial" +) + +// parkedEntry is an original the batch leaves out because it cannot land on +// the line whole: what it needs that never comes, or the paths it would +// leave without effect. announced says notify has named it on the open +// batch, for exactly this entry. +type parkedEntry struct { + Original int `json:"original"` + Merge string `json:"merge"` + Kind string `json:"kind"` + Needs []int `json:"needs,omitempty"` + Commits []string `json:"commits,omitempty"` + Files []string `json:"files,omitempty"` + Announced bool `json:"announced,omitempty"` +} + +// validate holds a parked entry to what a run writes. Its errors read after +// "the block" or "report ". +func (p parkedEntry) validate() error { + sortedUnique := func(n int, less func(i int) bool) bool { + for i := 1; i < n; i++ { + if !less(i) { + return false + } + } + return true + } + switch { + case checkSHA(p.Merge) != nil: + return fmt.Errorf("parks #%d by merge %q", p.Original, p.Merge) + case !sortedUnique(len(p.Needs), func(i int) bool { return p.Needs[i-1] < p.Needs[i] }) || slices.ContainsFunc(p.Needs, func(n int) bool { return n <= 0 || n == p.Original }): + return fmt.Errorf("parks #%d for originals %v, which are not what a run writes", p.Original, p.Needs) + case !sortedUnique(len(p.Commits), func(i int) bool { return p.Commits[i-1] < p.Commits[i] }) || slices.ContainsFunc(p.Commits, func(c string) bool { return checkSHA(c) != nil }): + return fmt.Errorf("parks #%d for commits %q, which are not what a run writes", p.Original, p.Commits) + case !sortedUnique(len(p.Files), func(i int) bool { return p.Files[i-1] < p.Files[i] }) || !slices.Equal(p.Files, cleanFiles(p.Files)): + return fmt.Errorf("parks #%d for files %q, which are not what a run writes", p.Original, p.Files) + } + switch p.Kind { + case parkPartial: + if len(p.Files) == 0 || len(p.Needs)+len(p.Commits) > 0 { + return fmt.Errorf("parks #%d as partial with no files it leaves without effect, or with what it needs", p.Original) + } + case parkDependency: + if len(p.Needs)+len(p.Commits) == 0 || len(p.Files) > 0 { + return fmt.Errorf("parks #%d for a dependency it does not name, or with files", p.Original) + } + default: + return fmt.Errorf("parks #%d as %q, which is no kind of park", p.Original, p.Kind) + } + return nil } // builtFrom is what a classification was made of; batch is the head it @@ -173,9 +248,13 @@ func parseRecord(body string, l line) (*lineRecord, error) { err = decodeStrict(blocks[0], &v1) rec = lineRecord{Version: 1, Line: v1.Line, Deferred: v1.Deferred} case 2: + var v2 lineRecordV2 + err = decodeStrict(blocks[0], &v2) + rec = lineRecordWithout(v2) + case 3: err = decodeStrict(blocks[0], &rec) default: - return nil, fmt.Errorf("the %s block is version %d, which this backport-batch does not read; it reads 1 and 2", deferredFence, version) + return nil, fmt.Errorf("the %s block is version %d, which this backport-batch does not read; it reads 1 to %d", deferredFence, version, recordVersion) } if err != nil { return nil, fmt.Errorf("reading the %s block: %w", deferredFence, err) @@ -244,6 +323,14 @@ func (rec *lineRecord) validate(l line) error { return fmt.Errorf("drops #%d for %q, which is no reason a run writes", d.Original, d.Reason) } } + for _, p := range rec.Parked { + if err := once(p.Original); err != nil { + return err + } + if err := p.validate(); err != nil { + return err + } + } staleSeen := map[int]bool{} for _, n := range rec.Stale { if n <= 0 || staleSeen[n] { @@ -251,6 +338,14 @@ func (rec *lineRecord) validate(l line) error { } staleSeen[n] = true } + partialSeen := map[int]bool{} + for _, n := range rec.Partial { + // A unit that does only part of its original is still on the batch. + if !staleSeen[n] || partialSeen[n] { + return fmt.Errorf("lists #%d as a partial unit more than once, or not as still on the batch", n) + } + partialSeen[n] = true + } if rec.Version < 2 { return nil } @@ -297,6 +392,11 @@ func (b *builtFrom) validate() error { var runURLRE = regexp.MustCompile(`^https://[^/\s]+/[^/\s]+/[^/\s]+/actions/runs/[0-9]+(/attempts/[0-9]+)?$`) +// lineRecordWithout is a version 2 record as the newest version holds it. +func lineRecordWithout(v2 lineRecordV2) lineRecord { + return lineRecord{Version: v2.Version, Line: v2.Line, Built: v2.Built, Report: v2.Report, Closing: v2.Closing, Deferred: v2.Deferred, Dropped: v2.Dropped, Stale: v2.Stale} +} + // recordBlock is the block that holds rec. One line of JSON: encoding/json // escapes newlines and <, > and &, so nothing in a file name or a reason can // end the fence or open HTML. diff --git a/cmd/backport-batch/record_test.go b/cmd/backport-batch/record_test.go index 13ab6cc79..ed09d1cd2 100644 --- a/cmd/backport-batch/record_test.go +++ b/cmd/backport-batch/record_test.go @@ -18,6 +18,7 @@ package main import ( "fmt" + "slices" "strings" "testing" ) @@ -272,7 +273,7 @@ func TestRecordNamesItsInputs(t *testing.T) { want := builtFrom{Main: f.git("rev-parse", "origin/main"), Line: f.git("rev-parse", "origin/release-1.0"), Newer: f.git("rev-parse", "origin/release-1.1"), Policy: f.git("rev-parse", "origin/main"), Batch: f.remoteHead(branch), Tool: f.tool, Run: "https://github.com/cozystack/cozystack/actions/runs/123/attempts/2", At: "2026-10-26T06:00:00Z", Options: runOptions{NoWorkflowFiles: true, Mode: modeNew}} - if rec.Version != 2 || *rec.Built != want || bodyReportOf(t, stdout, l).doc.Built != want || rec.Report != bodyReportOf(t, stdout, l).doc.ID { + if rec.Version != recordVersion || *rec.Built != want || bodyReportOf(t, stdout, l).doc.Built != want || rec.Report != bodyReportOf(t, stdout, l).doc.ID { t.Errorf("record = %+v, built %+v; want built %+v and the report it names", rec, rec.Built, want) } @@ -366,3 +367,43 @@ func TestPRStateNamesOnlyKnownStates(t *testing.T) { } } } + +// v3 is v2 as version 3, with extra after the stale list. +func v3(extra string) string { + return strings.Replace(v2(extra), `"version":2`, `"version":3`, 1) +} + +// Version 3 adds what is parked and which units do only part of their +// original, each held to what a run writes, and version 2 has neither. +func TestRecordVersion3(t *testing.T) { + l, _ := parseLine("release-1.1") + block := func(json string) string { return "text\n\n" + deferredFence + "\n" + json + "\n```\n" } + sha, other := strings.Repeat("a", 40), strings.Repeat("c", 40) + parked := `,"parked":[{"original":10,"merge":"` + sha + `","kind":"partial","files":["a","b"]},{"original":11,"merge":"` + sha + `","kind":"dependency","needs":[3,4],"commits":["` + other + `"],"announced":true}],"partial":[9]` + rec, err := parseRecord(block(v3(parked)), l) + if err != nil || rec.Version != 3 || len(rec.Parked) != 2 || !rec.Parked[1].Announced || !slices.Equal(rec.Partial, []int{9}) { + t.Fatalf("parseRecord of a whole v3 record = %+v, %v", rec, err) + } + for name, json := range map[string]string{ + "version 2 with a park": v2(parked), + "version 4": strings.Replace(v3(""), `"version":3`, `"version":4`, 1), + "a park of a deferred original": v3(strings.Replace(parked, `"original":10`, `"original":5`, 1)), + "a park of a dropped original": v3(strings.Replace(parked, `"original":10`, `"original":8`, 1)), + "a partial park with no files": v3(strings.Replace(parked, `"files":["a","b"]`, `"files":[]`, 1)), + "a partial park that needs": v3(strings.Replace(parked, `"files":["a","b"]`, `"files":["a"],"needs":[3]`, 1)), + "unsorted files": v3(strings.Replace(parked, `"files":["a","b"]`, `"files":["b","a"]`, 1)), + "a file that drives a terminal": v3(strings.Replace(parked, `"files":["a","b"]`, `"files":["a\u001b[2J"]`, 1)), + "a dependency park with no need": v3(strings.Replace(strings.Replace(parked, `"needs":[3,4],`, ``, 1), `"commits":["`+other+`"],`, ``, 1)), + "unsorted needs": v3(strings.Replace(parked, `"needs":[3,4]`, `"needs":[4,3]`, 1)), + "a park that needs itself": v3(strings.Replace(parked, `"needs":[3,4]`, `"needs":[3,11]`, 1)), + "a short commit": v3(strings.Replace(parked, `"commits":["`+other+`"]`, `"commits":["abc"]`, 1)), + "a short merge": v3(strings.Replace(parked, `"merge":"`+sha+`","kind":"partial"`, `"merge":"abc","kind":"partial"`, 1)), + "an unknown kind": v3(strings.Replace(parked, `"kind":"partial"`, `"kind":"later"`, 1)), + "a partial unit not on the batch": v3(strings.Replace(parked, `"partial":[9]`, `"partial":[10]`, 1)), + "a partial unit twice": v3(strings.Replace(parked, `"partial":[9]`, `"partial":[9,9]`, 1)), + } { + if _, err := parseRecord(block(json), l); err == nil { + t.Errorf("%s: parseRecord accepted it", name) + } + } +} diff --git a/cmd/backport-batch/report.go b/cmd/backport-batch/report.go index 46e37c955..fc6d4dda8 100644 --- a/cmd/backport-batch/report.go +++ b/cmd/backport-batch/report.go @@ -23,6 +23,7 @@ import ( "encoding/json" "errors" "fmt" + "maps" "os" "regexp" "runtime/debug" @@ -43,7 +44,9 @@ import ( const ( reportFence = "```backport-batch-report" reportMarker = "` and is one per pull request: a later run edits it in place rather than adding another, and once nothing waits it says so. Only a comment whose author the batch trusts counts as it, so one someone else wrote with the same first line is neither edited nor believed. It mentions `--mention`, an `org/team` or a login, with or without `@`, only while something waits. A failed batch CI is not among what it reports: the run has no trigger on the batch's CI finishing. @@ -116,7 +122,7 @@ An original the policy takes can build on code its line does not have: another p - A file it modifies or deletes whose counterpart, as [`run`](#run) follows renames, the line does not have is named by the first-parent commit on `main` that added it since the line branched: `git --literal-pathspecs log --first-parent --diff-merges=first-parent --no-renames --diff-filter=A -1 ..M^1 -- `. A file nothing added since the fork point is shown with the original, since no other pull request brought it. - A line it removes or changes in a file the line has is given by `git blame --porcelain -w`, at `M^1` named by its full commit, to a commit, which counts through the first-parent commit on `main` that brought it: each commit of `main`'s first-parent line since the fork point brings itself and every commit of the range reachable from its other parents that no earlier one brought. A commit from before the fork point, which the line has, and the original's own count for nothing. The commit counts only while one of its lines, with its edges trimmed and nothing else, is absent from the line's version of the file, so a hand backport under another commit clears it. A file the line already has as the original makes it, the same blob or a clean `git merge-file` back into the line's file, builds on nothing. -Lines a tool wrote are not blamed: a generated file whole, `zz_generated*`, one with a `Code generated … DO NOT EDIT` marker in its first 100 lines, `go.sum` and the package-manager lock files, and in an ApplicationDefinition under `packages/system/*-rd/cozyrds/` the keys `hack/update-crd.sh` writes, found by their indentation. Controller-gen CRDs and `values.schema.json` are blamed like any other file. Each `git log`, `git diff` and `git blame` the detector and the rename detection run has two minutes, and rename detection that runs out of time counts as skipped, and an original has at most 256 files, missing or blamed, 20 000 blamed lines and 8 MiB per file read for it. Evidence it cannot read, past one of those bounds, a binary file, a symlink, a submodule, a commit blame names that did not come through `main`, a path that is not UTF-8, or a rename git skipped looking for, is shown with the original, whatever becomes of it, under its line in the body and in the report's `diag`. Output of git the tool cannot read whole stops it with exit 2, and is never taken for an absent dependency. A dependency carried only by lines an original adds is not seen. `run` prints how long reading the line took on stderr. +Lines a tool wrote are not blamed: a generated file whole, `zz_generated*`, one with a `Code generated … DO NOT EDIT` marker in its first 100 lines, `go.sum` and the package-manager lock files, and in an ApplicationDefinition under `packages/system/*-rd/cozyrds/` the keys `hack/update-crd.sh` writes, found by their indentation. Controller-gen CRDs and `values.schema.json` are blamed like any other file. Each `git log`, `git diff` and `git blame` the detector and the rename detection run has two minutes, and rename detection that runs out of time counts as skipped, and an original has at most 256 files, missing or blamed, 20 000 blamed lines and 8 MiB per file read for it. Evidence it cannot read, past one of those bounds, a binary file, a symlink, a submodule, a commit blame names that did not come through `main`, a path that is not UTF-8, or a rename git skipped looking for, is shown with the original, whatever becomes of it, under its line in the body and in the report's `diag`. Output of git the tool cannot read whole stops it with exit 2, and is never taken for an absent dependency. A dependency carried only by lines an original adds is not seen. `run` prints how long reading the line took on stderr. What follows from a dependency, a park or a wait, is in [`run`](#run) and [`check`](#check). ## The line's record and reports @@ -126,7 +132,7 @@ The record is written only while its pull request is open: a merged or closed ba The record is only as trustworthy as the body it sits in. Anyone with write access can edit any body, the bot's included, and neither the bot's authorship nor anything in the block proves who edited it last. A hand edit that leaves a readable record is believed; one that breaks it stops every command with exit 2, naming the pull request. A reason or a file name in the record counts only in the form a run writes it, one line with no control, zero-width or bidi character, so text someone typed into the body never reaches a terminal, a body or a comment as they typed it. When the newest batch pull request is deleted and no later one exists, nothing shows it was there: the older record governs, and what was recorded since is lost. -The record holds only what later commands act on. Every run that classifies the line also makes a report, one line of JSON in a `backport-batch-report` block: every original merged into `main` since the line branched in exactly one category, picked, missing, deferred with what it was deferred for, or skipped with the rule that skipped it, each drop the record keeps with the merge commit it is bound to, its reason and the batch pull request that recorded it, the commits on `main` that merged no pull request, the labels, and what the classification was made of: the heads of `main`, the line and the newer line, the policy, the batch head it describes, the tool's commit, the Actions run and the flags. Its ID is a hash of all that but the time and the run, so two runs that find the same make the same report, and it names no pull request, so a new batch's report is whole before GitHub numbers the pull request. The body carries the latest report beside the record, and a comment on the open batch pull request keeps each one: `` and the same block, folded away. A report already posted is not posted again, and the tool never edits or deletes such a comment, so a report the next run replaces in the body is still read back by `report`, also after the batch merged. A comment that looks like one but whose author the batch does not trust is not read. When a run stops after writing the body but before posting its comment, the next `run --push` on the open batch posts that report first, byte for byte; once the batch is closed nothing is written to it, and the report stays in its body. A run that picks nothing while no batch is open keeps no report. +The record holds only what later commands act on. Version 3, which this tool writes, adds what is parked and why, with the merge commit each park is bound to, the units on the batch that do only part of their original, the picks that differ from theirs, and what each wait on another original waits on; it reads versions 1 to 3, and a tool that reads only 1 and 2 refuses version 3 with exit 2. Every run that classifies the line also makes a report, one line of JSON in a `backport-batch-report` block: every original merged into `main` since the line branched in exactly one category, picked, missing, deferred with what it was deferred for, or skipped with the rule that skipped it, each drop the record keeps with the merge commit it is bound to, its reason and the batch pull request that recorded it, the commits on `main` that merged no pull request, the labels, and what the classification was made of: the heads of `main`, the line and the newer line, the policy, the batch head it describes, the tool's commit, the Actions run and the flags. Its ID is a hash of all that but the time and the run, so two runs that find the same make the same report, and it names no pull request, so a new batch's report is whole before GitHub numbers the pull request. The body carries the latest report beside the record, and a comment on the open batch pull request keeps each one: `` and the same block, folded away. A report already posted is not posted again, and the tool never edits or deletes such a comment, so a report the next run replaces in the body is still read back by `report`, also after the batch merged. A comment that looks like one but whose author the batch does not trust is not read. When a run stops after writing the body but before posting its comment, the next `run --push` on the open batch posts that report first, byte for byte; once the batch is closed nothing is written to it, and the report stays in its body. A run that picks nothing while no batch is open keeps no report. A `--defer` that leaves the batch carrying nothing closes it, and writes everything while the pull request is still open: the report comment, then the body, whose record keeps the deferrals and says the batch is closing, with its number, branch and the head the rebuild started from. It then reads the pull request and the branch again, closes the pull request while both are at that head, and deletes the branch with `--force-with-lease` on it, printing `closed `. The next batch tries what was deferred again. A close that stops part way is finished by the next `run --push`, before it does anything else: it closes and deletes when the pull request is open at the frozen head, deletes the branch when the pull request is closed and the branch still at that head, and goes on as usual once both are gone. A run that finishes a close stops there, with exit 0, or with exit 2 when it was asked for `--defer`, `--drop` or `--undrop`, which it does not apply. It never takes a head it did not freeze: with the branch moved, it stops with exit 2 and names both heads. `run` without `--push`, `check` and `fold` stop with exit 2 while the pull request is still open; once it is closed they go on, warning about a branch the close left, and `notify` says nothing. A closed pull request is never written to, so its record says it was closing for good, and a closed pull request whose branch is gone counts as closed. diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index 753868f72..35ad92214 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -33,6 +33,8 @@ type pick struct { files []string // failed is the commit whose pick conflicted. failed string + // needs are the originals still to come it waits on. + needs []int // more counts the conflicting files a recorded conflict no longer names. more int reason string @@ -204,6 +206,19 @@ func (a *app) run(args []string) int { return a.fail(err) } } + // cls is what each original became in this run, to an original that + // builds on it; the rules decide the rest. + mode := analysisMode{noWorkflowFiles: *noWorkflows} + cls := map[*verdict]edgeClass{} + classOf := func(f *verdict) edgeClass { + if c, ok := b.ruleClass(f); ok { + return c + } + if c, ok := cls[f]; ok { + return c + } + return edgeMissing + } for _, v := range b.qualifying() { on := false if carried != nil { @@ -214,9 +229,11 @@ func (a *app) run(args []string) int { switch { case on: r.onBranch = append(r.onBranch, v) + cls[v] = edgeSatisfied continue case held[v.o.number].Reason != "": r.held = append(r.held, pick{v: v, status: "held", reason: held[v.o.number].Reason}) + cls[v] = edgeDeferred continue case v.onBatch != "": v.rule, v.reason = ruleOnLine, fmt.Sprintf("on the open batch pull request #%d", b.pr.Number) @@ -231,6 +248,17 @@ func (a *app) run(args []string) int { } if v.partial != nil { r.parked = append(r.parked, v) + cls[v] = edgeNever + continue + } + } + if len(v.deps) > 0 { + kind, err := a.dependsAt(b, r, v, mode, classOf) + if err != nil { + return a.fail(err) + } + if kind != edgeSatisfied { + cls[v] = kind continue } } @@ -241,6 +269,7 @@ func (a *app) run(args []string) int { } if len(files) > 0 { r.waiting = append(r.waiting, pick{v: v, status: "waiting", files: files}) + cls[v] = edgeWorkflow continue } } @@ -257,10 +286,13 @@ func (a *app) run(args []string) int { v.partial = p.files v.park = &parkedEntry{Original: v.o.number, Merge: v.o.merge, Kind: parkPartial, Files: sortedClean(p.files)} r.parked = append(r.parked, v) + cls[v] = edgeNever case "picked": r.onBranch = append(r.onBranch, v) + cls[v] = edgeSatisfied case "conflict": r.deferred = append(r.deferred, p) + cls[v] = edgeDeferred case "empty": v.rule, v.reason = ruleOnLine, "the earlier picks already carry it" r.skipped[ruleOnLine] = append(r.skipped[ruleOnLine], v) @@ -285,6 +317,50 @@ func (a *app) run(args []string) int { return a.finish(b, r, branch, date, *push, false) } +// dependsAt decides, from what v builds on that the branch lacks, whether +// the run leaves it out: waiting for a maintainer in the scheduled mode +// when only tests or docs build on what never comes, parked when anything +// else never comes, or waiting on prerequisites still to come. What is +// there, the line's, the batch's, or what this run picked or found empty, +// counts for nothing; once earlier picks moved the branch, v is judged at +// its head. It returns what v became to an original that builds on it, +// edgeSatisfied when the run goes on to pick it. +func (a *app) dependsAt(b *batch, r *report, v *verdict, mode analysisMode, classOf func(*verdict) edgeClass) (edgeClass, error) { + deps := v.deps + head, err := resolve("HEAD") + if err != nil { + return 0, err + } + if head != b.lineHead { + if deps, err = b.depsAt(v, head); err != nil { + return 0, err + } + } + never, pending := edges(deps, classOf) + primary, err := workflowPrimary(mode, v, never, pending) + switch { + case err != nil: + return 0, err + case primary: + files, err := workflowFiles(v.commits) + if err != nil { + return 0, err + } + v.addDiag(testsOnly(v, never, b.line)) + r.waiting = append(r.waiting, pick{v: v, status: "waiting", files: files}) + return edgeWorkflow, nil + case len(never) > 0: + v.park = dependencyPark(v, never) + r.parked = append(r.parked, v) + return edgeNever, nil + case len(pending) > 0: + class, needs := strongest(pending) + r.needs = append(r.needs, pick{v: v, status: "waiting", needs: numbers(needs)}) + return class, nil + } + return edgeSatisfied, nil +} + // partialAtHead judges v again where the run has got to, when earlier picks // moved the branch past the line's head. func (a *app) partialAtHead(b *batch, v *verdict) error { diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index 02dc2fd90..d64bfb5d5 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -176,6 +176,9 @@ type deferredEntry struct { More int `json:"more,omitempty"` Reason string `json:"reason,omitempty"` Workflows bool `json:"workflows,omitempty"` + // Needs, from version 3, are the originals still to come that it waits + // on, those of the kind that decides the wait. + Needs []int `json:"needs,omitempty"` } // How many conflicting files of one original the body names, in the list and @@ -194,7 +197,7 @@ type deferredRecord struct { // had a version. encoding/json leaves bidi and zero-width characters as they // are, so file names go through clean first, as the reason does. func deferredBlock(l line, conflicts, held, waiting []pick) string { - data, _ := json.Marshal(deferredRecord{Line: l.branch(), Deferred: deferredEntries(conflicts, held, waiting)}) + data, _ := json.Marshal(deferredRecord{Line: l.branch(), Deferred: deferredEntries(conflicts, held, waiting, nil)}) return deferredFence + "\n" + string(data) + "\n```" } @@ -231,6 +234,7 @@ type report struct { deferred []pick // conflicts held []pick // what --defer took out, with the reason waiting []pick // what changes workflows, left to a maintainer + needs []pick // what waits on other originals still to come skipped map[int][]*verdict other int // overrides are the originals a backport/ label kept out or let in, and @@ -381,6 +385,12 @@ func (r *report) what(details map[int]bool) string { fmt.Fprintf(&b, "- %s %s: %s\n%s", prLink(v.o.number), titleSpan(v.o.title), parkReason(*v.park), diagItems(v)) } } + if len(r.needs) > 0 { + fmt.Fprintf(&b, "\n### Waiting on another original\n\nThese build on another original that has not reached %s yet, so the batch does not try them; each names the one it waits on, and the batch tries it again once that one is there.\n\n", codeSpan(r.line.branch())) + for _, p := range r.needs { + fmt.Fprintf(&b, "- %s %s: after %s\n%s", prLink(p.v.o.number), titleSpan(p.v.o.title), prLinks(p.needs), diagItems(p.v)) + } + } if len(r.overrides) > 0 { b.WriteString("\n### Maintainer overrides\n\n") for _, v := range r.overrides { @@ -437,6 +447,9 @@ func (r *report) outcome(v *verdict) string { return "waiting for a maintainer: it changes " + codeSpan(workflowsDir) case slices.Contains(r.parked, v): return "parked: " + parkReason(*v.park) + case slices.ContainsFunc(r.needs, func(p pick) bool { return p.v == v }): + i := slices.IndexFunc(r.needs, func(p pick) bool { return p.v == v }) + return "waiting on " + prLinks(r.needs[i].needs) case v.rule != 0: return "skipped by " + ruleName(v.rule) + ", " + titleSpan(v.reason) } @@ -465,6 +478,15 @@ func fileSpans(files []string) string { return strings.Join(out, ", ") } +// prLinks links to pull requests by number. +func prLinks(numbers []int) string { + var out []string + for _, n := range numbers { + out = append(out, prLink(n)) + } + return strings.Join(out, ", ") +} + // parkReason says why a park leaves its original out, for the body. func parkReason(p parkedEntry) string { if p.Kind == parkPartial { diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index b9983aca3..d8acf4e37 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -51,7 +51,13 @@ func (a *app) check(args []string) int { if err := a.warnLeftBranch(b); err != nil { return a.fail(err) } - disp, err := b.dispositions(*cut) + // The scheduled run's mode, as the open batch saved it; with none open, + // the scheduled run's, which is what the next batch is built in. + mode := analysisMode{noWorkflowFiles: true} + if b.src.open { + mode.noWorkflowFiles = b.src.savedNoWorkflowFiles() + } + disp, err := b.dispositions(mode, *cut) if err != nil { return a.fail(err) } @@ -138,8 +144,11 @@ type disposition struct { // dispositions says what check reports about each original that needs a line // of its own; the others pass in silence. The text is what follows the line's -// name, or the original alone for one that is missing. -func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { +// name, or the original alone for one that is missing. Originals are taken in +// merge order, each by the first of: a unit on the batch that does only part +// of it, a reason a person deferred it for, a park, a wait for a maintainer +// or on other originals, and what the record says of it. +func (b *batch) dispositions(mode analysisMode, cut bool) (map[*verdict]disposition, error) { if c := b.src.closingOpen(); c != nil { return nil, fmt.Errorf("batch pull request #%d is being closed; backport-batch run --line %s --push finishes it", c.PR, b.line.branch()) } @@ -159,10 +168,40 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { on += " (" + prState(c) + ")" } } + if err := b.analyze(); err != nil { + return nil, err + } + if b.pr != nil { + if err := b.keepRecordedPartials(b.prHead); err != nil { + return nil, err + } + } + held := map[int]string{} + for n, e := range deferred { + if e.Reason != "" { + held[n] = e.Reason + } + } + states, err := b.classify(mode, held, deferred, b.src.open) + if err != nil { + return nil, err + } + // waitsOn names what an original waits on, as check says it. + waitsOn := func(s *state) string { + var names []string + for _, f := range s.needs { + names = append(names, fmt.Sprintf("#%d", f.o.number)) + } + return strings.Join(names, " and ") + } out := map[*verdict]disposition{} for _, v := range b.verdicts { named := v.o.url() + " " + terminalSafe(v.o.title) e, isDeferred := deferred[v.o.number] + s := states[v] + if s == nil { + s = &state{kind: stOwn} + } switch { case b.partialUnits[v] != nil && cut: out[v] = disposition{outcomeUnmerged, fmt.Sprintf("#%d is on batch #%d, but only part of it applies to %s: it leaves %s without effect; take it out with backport-batch run --defer before the cut: %s", v.o.number, b.pr.Number, b.line.branch(), quotedFiles(b.partialUnits[v]), named)} @@ -178,17 +217,41 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { if cut { out[v] = disposition{outcomeUnmerged, fmt.Sprintf("#%d is only on open batch PR #%d, merge it before the cut: %s", v.o.number, b.pr.Number, named)} } - case isDeferred && e.Reason != "": + case s.kind == stHeld: out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred on %s to the next batch: %s (%s)", on, named, terminalSafe(e.Reason))} - case v.park != nil && v.park.Kind == parkPartial: + case s.kind == stPartial: out[v] = disposition{outcomeNote, fmt.Sprintf("#%d is parked: only part of it applies to %s, which would leave %s without effect; port what applies as a pull request of its own: %s", v.o.number, b.line.branch(), quotedFiles(v.park.Files), named)} + case s.kind == stWorkflowPrimary && s.class == edgeMissing: + out[v] = disposition{outcomeMissing, named} + case s.kind == stParked: + out[v] = disposition{outcomeNote, fmt.Sprintf("#%d is parked: it builds on %s, which never reaches %s; port what applies as a pull request of its own, or drop it: %s", v.o.number, parkNames(*v.park), b.line.branch(), named)} + case s.kind == stNeeds && s.wait == edgeMissing: + out[v] = disposition{outcomeMissing, fmt.Sprintf("%s (it waits on %s, which is not on the line either)", named, waitsOn(s))} + case s.kind == stNeeds && s.wait == edgeWorkflow && cut: + out[v] = disposition{outcomeWaiting, fmt.Sprintf("#%d waits on %s, which changes %s and waits on batch PR %s for a maintainer; bring it over, or defer both, before the cut: %s", v.o.number, waitsOn(s), workflowsDir, on, named)} + case s.kind == stNeeds && s.wait == edgeWorkflow: + out[v] = disposition{outcomeWarning, fmt.Sprintf("#%d waits on %s, which waits on %s for a maintainer: %s", v.o.number, waitsOn(s), on, named)} + case s.kind == stNeeds && s.wait == edgeDeferred: + what := "a deferred conflict on " + on + if f := s.needs[0]; deferred[f.o.number].Reason != "" { + what = "deferred on " + on + " to the next batch" + } else if states[f] != nil && states[f].kind == stNeeds { + what = "which waits itself" + } + out[v] = disposition{outcomeWarning, fmt.Sprintf("#%d waits on %s, %s: %s", v.o.number, waitsOn(s), what, named)} + case s.kind == stNeeds: + newer := "the newer line" + if b.newer != nil { + newer = b.newer.branch() + } + out[v] = disposition{outcomeNote, fmt.Sprintf("#%d waits on %s, which is not on %s yet: %s", v.o.number, waitsOn(s), newer, named)} case isDeferred && e.Workflows && !b.src.open: out[v] = disposition{outcomeMissing, named} case isDeferred && e.Workflows && cut: out[v] = disposition{outcomeWaiting, fmt.Sprintf("#%d changes %s and waits on batch PR %s for a maintainer, bring it over or defer it before the cut: %s", v.o.number, workflowsDir, on, named)} case isDeferred && e.Workflows: out[v] = disposition{outcomeWarning, fmt.Sprintf("waiting on %s for a maintainer, since it changes %s: %s", on, workflowsDir, named)} - case isDeferred: + case isDeferred && len(e.Needs) == 0: out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred conflict on %s: %s", on, named)} default: out[v] = disposition{outcomeMissing, named} @@ -197,6 +260,18 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { return out, nil } +// parkNames names what a park waits on for a terminal. +func parkNames(p parkedEntry) string { + var out []string + for _, n := range p.Needs { + out = append(out, fmt.Sprintf("#%d", n)) + } + for _, c := range p.Commits { + out = append(out, c[:12]) + } + return strings.Join(out, ", ") +} + // quotedFiles names files for a terminal, each quoted, the first few of them. func quotedFiles(files []string) string { var out []string diff --git a/cmd/backport-batch/defer.go b/cmd/backport-batch/defer.go index 8915d4ebd..de77d5d5c 100644 --- a/cmd/backport-batch/defer.go +++ b/cmd/backport-batch/defer.go @@ -138,22 +138,50 @@ func (a *app) reclassify(b *batch, picks *history, hold map[int]string, drops [] if r.stale, err = keptOut(b, picks); err != nil { return a.fail(err) } + // What the rebuild took off is no longer there for what builds on it. + units, err := b.unitsOn(picks.head) + if err != nil { + return a.fail(err) + } + for _, v := range b.verdicts { + v.batched = units[v] + } + on := map[*verdict]bool{} for _, v := range b.qualifying() { - e := recorded[v.o.number] - on, err := b.onBranch(picks, v) - if err != nil { + if on[v], err = b.onBranch(picks, v); err != nil { return a.fail(err) } + if !on[v] { + v.onBatch = "" + } + } + if err := b.keepRecordedPartials(picks.head); err != nil { + return a.fail(err) + } + states, err := b.classify(analysisMode{noWorkflowFiles: r.opts.NoWorkflowFiles}, hold, recorded, true) + if err != nil { + return a.fail(err) + } + for _, v := range b.qualifying() { + e := recorded[v.o.number] + s := states[v] + if s == nil { + s = &state{kind: stOwn} + } switch { - case on: + case on[v]: r.onBranch = append(r.onBranch, v) case hold[v.o.number] != "": r.held = append(r.held, pick{v: v, status: "held", reason: hold[v.o.number]}) - case v.park != nil: + case s.kind == stPartial || s.kind == stParked: r.parked = append(r.parked, v) + case s.kind == stNeeds: + r.needs = append(r.needs, pick{v: v, status: "waiting", needs: numbers(s.needs)}) + case s.kind == stWorkflowPrimary && !e.Workflows: + r.missing = append(r.missing, v) case e.Workflows: r.waiting = append(r.waiting, pick{v: v, status: "waiting", files: e.Files, more: e.More}) - case len(e.Files) > 0: + case len(e.Files) > 0 && len(e.Needs) == 0: r.deferred = append(r.deferred, pick{v: v, status: "conflict", files: e.Files, more: e.More, failed: e.Commit}) default: // Merged into main since the batch was assembled, or recorded diff --git a/cmd/backport-batch/deps.go b/cmd/backport-batch/deps.go index 33133a1e5..895897709 100644 --- a/cmd/backport-batch/deps.go +++ b/cmd/backport-batch/deps.go @@ -436,6 +436,23 @@ func (b *batch) dependencies(v *verdict, target string) ([]dependency, []string, for _, d := range byMerge { slices.Sort(d.paths) deps = append(deps, *d) + // The rules decide before the line is looked at, so whether the + // line carries a prerequisite they keep out is asked here. + if f := d.v; f != nil && f.rule != 0 && f.rule != ruleOnLine && f.rule != ruleNewer && b.lineHist != nil { + if _, asked := b.lineCarries[f]; !asked { + if err := b.load(f); err != nil { + return nil, nil, err + } + how, err := b.lineHist.carries(f.o, f.commits) + if err != nil { + return nil, nil, err + } + if b.lineCarries == nil { + b.lineCarries = map[*verdict]bool{} + } + b.lineCarries[f] = how != "" + } + } } slices.SortFunc(deps, func(x, y dependency) int { return strings.Compare(x.merge, y.merge) }) return deps, diag, nil @@ -614,25 +631,9 @@ func (b *batch) beforeBase(commit string) (bool, error) { return ok, err } -// depNote is how a dependency is shown with its original. -func (d dependency) note(l line) string { - who := fmt.Sprintf("%.12s, a commit on main that merged no pull request", d.merge) - if d.v != nil { - who = fmt.Sprintf("#%d", d.v.o.number) - } - quoted := make([]string, len(d.paths)) - for i, p := range d.paths { - quoted[i] = shown(p) - } - how := "in " + strings.Join(quoted, ", ") - if d.added { - how = "which added " + strings.Join(quoted, ", ") - } - return oneLineText(fmt.Sprintf("builds on %s, %s, which %s lacks", who, how, l.branch())) -} - // analyze reads what every original the policy takes and nothing carries -// builds on that the line lacks, and keeps it with the original. +// builds on that the line lacks, and keeps it with the original, for the +// parks and waits that follow from it. func (b *batch) analyze() error { defer b.timed(time.Now()) for _, v := range b.qualifying() { @@ -645,9 +646,267 @@ func (b *batch) analyze() error { } v.deps = deps v.addDiag(diag...) - for _, d := range deps { - v.addDiag(d.note(b.line)) - } } return nil } + +// analysisMode is how a classification treats an original that changes a +// workflow: as the scheduled run does, whose token cannot push one, or as a +// maintainer's run. +type analysisMode struct{ noWorkflowFiles bool } + +// edgeClass is what a prerequisite is to an original that builds on it, +// from what never matters to what decides first. +type edgeClass int + +const ( + edgeSatisfied edgeClass = iota + // edgeNewer is held by rule 6 on this line. + edgeNewer + // edgeDeferred is a deferred conflict, or deferred for a reason. + edgeDeferred + // edgeWorkflow waits on the open batch for a maintainer. + edgeWorkflow + // edgeMissing is on neither the line, the batch nor the record. + edgeMissing + // edgeNever never reaches the line. + edgeNever +) + +// edge is a dependency whose prerequisite has not reached the line, with +// what that prerequisite is. +type edge struct { + dep dependency + class edgeClass +} + +// ruleClass is what the rules alone make of a prerequisite, or ok false when +// it depends on what this classification makes of it: one the line or the +// open batch carries is there, whatever its label, drop or rule says, and a +// direct push and an original the rules keep out never come. +func (b *batch) ruleClass(f *verdict) (edgeClass, bool) { + switch { + case f == nil: + return edgeNever, true + case f.rule == ruleOnLine || f.onBatch != "" || f.batched != "" || b.lineCarries[f]: + return edgeSatisfied, true + case f.rule == ruleNewer: + return edgeNewer, true + case f.rule != 0: + return edgeNever, true + } + return 0, false +} + +// edges sorts dependencies by what classOf says of each prerequisite into +// those that never come and those still to come, dropping the ones there. +func edges(deps []dependency, classOf func(*verdict) edgeClass) (never, pending []edge) { + for _, d := range deps { + switch c := classOf(d.v); c { + case edgeSatisfied: + case edgeNever: + never = append(never, edge{d, c}) + default: + pending = append(pending, edge{d, c}) + } + } + return never, pending +} + +// strongest is the class of prerequisite that decides a wait, and the +// prerequisites of that class, in merge order. +func strongest(pending []edge) (edgeClass, []*verdict) { + top := edgeSatisfied + for _, e := range pending { + top = max(top, e.class) + } + var out []*verdict + for _, e := range pending { + if e.class == top && !slices.Contains(out, e.dep.v) { + out = append(out, e.dep.v) + } + } + return top, out +} + +// auxiliary says whether a path holds tests or documentation, which an +// original can change without building on what it needs to work. +func auxiliary(p string) bool { + parts := strings.Split(p, "/") + base := parts[len(parts)-1] + for _, d := range parts[:len(parts)-1] { + if d == "test" || d == "tests" || d == "testdata" { + return true + } + } + switch { + case strings.HasSuffix(base, "_test.go"), strings.Contains(base, ".test."), strings.Contains(base, ".spec."), strings.HasSuffix(base, ".bats"): + return true + case len(parts) > 1 && parts[0] == "docs": + return true + } + return len(parts) == 1 && strings.HasSuffix(base, ".md") +} + +// workflowPrimary says whether v waits for a maintainer rather than parks: +// in the scheduled mode, an original that changes a workflow, all of whose +// missing dependencies that never come are tests or docs, and whose +// workflow lines build on nothing still missing. The workflow is what it is +// for, and a maintainer brings over what applies. Only what is still +// missing counts: never and pending are what is left once what is there was +// taken out. +func workflowPrimary(mode analysisMode, v *verdict, never, pending []edge) (bool, error) { + if !mode.noWorkflowFiles || len(never) == 0 { + return false, nil + } + for _, e := range never { + if slices.ContainsFunc(e.dep.paths, func(p string) bool { return !auxiliary(p) }) { + return false, nil + } + } + for _, e := range append(slices.Clone(never), pending...) { + if slices.ContainsFunc(e.dep.paths, func(p string) bool { return strings.HasPrefix(p, workflowsDir) }) { + return false, nil + } + } + files, err := workflowFiles(v.commits) + return len(files) > 0, err +} + +// testsOnly is the note a workflow original gets that waits for a +// maintainer although its tests build on what never comes. +func testsOnly(v *verdict, never []edge, l line) string { + var who []string + for _, e := range never { + if e.dep.v != nil { + who = append(who, fmt.Sprintf("#%d", e.dep.v.o.number)) + } else { + who = append(who, e.dep.merge[:12]) + } + } + slices.Sort(who) + return oneLineText(fmt.Sprintf("only its tests or docs build on %s, which never reaches %s", strings.Join(slices.Compact(who), ", "), l.branch())) +} + +// dependencyPark is the park of v for prerequisites that never come. +func dependencyPark(v *verdict, never []edge) *parkedEntry { + p := &parkedEntry{Original: v.o.number, Merge: v.o.merge, Kind: parkDependency} + for _, e := range never { + if e.dep.v != nil { + p.Needs = append(p.Needs, e.dep.v.o.number) + } else { + p.Commits = append(p.Commits, e.dep.merge) + } + } + slices.Sort(p.Needs) + p.Needs = slices.Compact(p.Needs) + slices.Sort(p.Commits) + p.Commits = slices.Compact(p.Commits) + return p +} + +// depsAt is what v builds on that target lacks, read once per target: the +// run asks again at its head once earlier picks moved the branch. +func (b *batch) depsAt(v *verdict, target string) ([]dependency, error) { + key := target + " " + v.o.merge + if d, ok := b.depCache[key]; ok { + return d, nil + } + deps, diag, err := b.dependencies(v, target) + if err != nil { + return nil, err + } + v.addDiag(diag...) + if b.depCache == nil { + b.depCache = map[string][]dependency{} + } + b.depCache[key] = deps + return deps, nil +} + +// stateKind is what a classification makes of an original the policy takes +// that nothing carries, in the order that decides it. +type stateKind int + +const ( + // stHeld is deferred for a reason a person gave. + stHeld stateKind = iota + 1 + // stPartial would leave part of it without effect on the line. + stPartial + // stWorkflowPrimary waits for a maintainer, though its tests build on + // what never comes. + stWorkflowPrimary + // stParked builds on what never comes. + stParked + // stNeeds waits on prerequisites still to come. + stNeeds + // stOwn is what the record says of it, or missing. + stOwn +) + +// state is a classification of one original: what it is, what it is to an +// original that builds on it, and for a wait the class that decides it and +// the prerequisites of that class. +type state struct { + kind stateKind + class edgeClass + wait edgeClass + needs []*verdict +} + +// classify decides, in merge order, what becomes of every original the +// policy takes that neither the line nor the open batch carries, as an +// operation that picks nothing sees it: check, and the ones that only +// change the record. held are the reasons people deferred originals for, +// recorded what the record defers, and listed says the record's workflow +// waits still wait on an open batch. Parks it finds go into each verdict. +func (b *batch) classify(mode analysisMode, held map[int]string, recorded map[int]deferredEntry, listed bool) (map[*verdict]*state, error) { + states := map[*verdict]*state{} + classOf := func(f *verdict) edgeClass { + if c, ok := b.ruleClass(f); ok { + return c + } + if s := states[f]; s != nil { + return s.class + } + return edgeMissing + } + for _, v := range b.qualifying() { + if v.onBatch != "" { + continue + } + s := &state{} + states[v] = s + e := recorded[v.o.number] + never, pending := edges(v.deps, classOf) + primary, err := workflowPrimary(mode, v, never, pending) + if err != nil { + return nil, err + } + switch { + case held[v.o.number] != "": + s.kind, s.class = stHeld, edgeDeferred + case v.park != nil && v.park.Kind == parkPartial: + s.kind, s.class = stPartial, edgeNever + case primary: + s.kind, s.class = stWorkflowPrimary, edgeMissing + if listed && e.Workflows { + s.class = edgeWorkflow + } + case len(never) > 0: + v.park = dependencyPark(v, never) + s.kind, s.class = stParked, edgeNever + case len(pending) > 0: + s.kind = stNeeds + s.wait, s.needs = strongest(pending) + s.class = s.wait + case listed && e.Workflows: + s.kind, s.class = stOwn, edgeWorkflow + case !e.Workflows && len(e.Needs) == 0 && (e.Commit != "" || len(e.Files) > 0): + s.kind, s.class = stOwn, edgeDeferred + default: + s.kind, s.class = stOwn, edgeMissing + } + } + return states, nil +} diff --git a/cmd/backport-batch/drop.go b/cmd/backport-batch/drop.go index b374081cf..1af2e5b3a 100644 --- a/cmd/backport-batch/drop.go +++ b/cmd/backport-batch/drop.go @@ -117,6 +117,9 @@ func (a *app) redrop(b *batch, numbers []int, reason string, undrop bool, push b if err := b.load(v); err != nil { return a.fail(err) } + if err := b.rejudged(v); err != nil { + return a.fail(err) + } if undrop { continue } @@ -159,6 +162,38 @@ func (b *batch) bindDrops(merges []*original) (map[int]droppedEntry, error) { }) } +// waitOrder refuses a record whose waits or parks need an original merged +// after the one that needs it: a dependency is always on main first. +func (b *batch) waitOrder(merges []*original) error { + if b.src.rec == nil { + return nil + } + at := map[int]int{} + for i, o := range merges { + at[o.number] = i + } + check := func(n int, needs []int) error { + i, ok := at[n] + for _, f := range needs { + if j, known := at[f]; ok && known && j >= i { + return fmt.Errorf("batch pull request #%d has #%d wait on #%d, which merged after it", b.src.carrier.Number, n, f) + } + } + return nil + } + for _, d := range b.src.rec.Deferred { + if err := check(d.Original, d.Needs); err != nil { + return err + } + } + for _, p := range b.src.rec.Parked { + if err := check(p.Original, p.Needs); err != nil { + return err + } + } + return nil +} + // bindEntries binds drops to main: one whose original is among merges, the // originals main merged since the line branched, has to name that // original's merge exactly; any other has to be a merge of its number on diff --git a/cmd/backport-batch/effects.go b/cmd/backport-batch/effects.go index 6bff85d7f..350f9a961 100644 --- a/cmd/backport-batch/effects.go +++ b/cmd/backport-batch/effects.go @@ -444,6 +444,7 @@ func (b *batch) partialStates() error { return err } for v, commit := range on { + v.batched = commit if v.rule == 0 && v.onBatch == "" { v.onBatch = commit } @@ -511,6 +512,58 @@ func (b *batch) markPartial(r *report, head string) error { return nil } +// keepRecordedPartials judges again, at head, what the batch carries now, +// each original the open record parks as partial that the line's head +// judges whole: the run's picker found it partial after the picks before +// it, which check and an operation that picks nothing do not replay. It +// stays parked while its net merge onto head still leaves something +// without effect, and is the line's again once that went, as when the pick +// that made it partial was taken off. +func (b *batch) keepRecordedPartials(head string) error { + if !b.src.open || b.src.rec == nil { + return nil + } + for _, p := range b.src.rec.Parked { + v := b.find(p.Original) + if p.Kind != parkPartial || v == nil || v.rule != 0 || v.onBatch != "" || v.park != nil || v.o.merge != p.Merge { + continue + } + if err := b.partialAt(v, head); err != nil { + return err + } + } + return nil +} + +// rejudged reads again what partialStates and analyze read for v once its +// rules changed, as a withdrawn drop changes them: it is back where the +// line puts it. +func (b *batch) rejudged(v *verdict) error { + if b.pr != nil { + on, err := b.unitsOn(b.prHead) + if err != nil { + return err + } + v.batched = on[v] + if v.rule == 0 && v.onBatch == "" { + v.onBatch = on[v] + } + } + if v.rule != 0 || v.onBatch != "" { + return nil + } + if err := b.partialAt(v, b.lineHead); err != nil { + return err + } + deps, diag, err := b.dependencies(v, b.lineHead) + if err != nil { + return err + } + v.deps = deps + v.addDiag(diag...) + return nil +} + // partialAt judges v's clean net merge onto head, and parks it, or clears // the park, by what that merge leaves without effect. func (b *batch) partialAt(v *verdict, head string) error { diff --git a/cmd/backport-batch/fold_test.go b/cmd/backport-batch/fold_test.go index 5c064c2e3..f783802b2 100644 --- a/cmd/backport-batch/fold_test.go +++ b/cmd/backport-batch/fold_test.go @@ -381,6 +381,22 @@ func TestFoldRefusesAKeptDeletion(t *testing.T) { } } +// An original parked for what it builds on can still land as a complete +// hand adaptation: every file it changes, changed, in the batch's shape. +func TestFoldAcceptsACompleteAdaptationOfAParkedOriginal(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(3, "feat(app): a loud two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + mc5 := f.pr(5, "fix(app): an excited two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/a.txt": strings.Replace(appFile, "two", "two!", 1)}) + fold := w.merge(w.head, pick, "fix(app): an excited two (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 0 || out != "ok\n" { + t.Errorf("fold of an adaptation of #5 = %d, %q\n%s", code, out, stderr) + } +} + // A fold that keeps a file the original deletes leaves the deletion without // effect, whatever it changes in the file: a port that keeps what the line // made of it, or the file with a line added, is a pull request of its own. diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index 85bd3ee12..23b357dff 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -207,8 +207,11 @@ type verdict struct { // and by load for anything else. commits []string changes []fileChange - // onBatch is the commit of the open batch pull request that picks it. + // onBatch is the commit of the open batch pull request that picks it, + // and batched the unit of the open batch that carries it whatever the + // rules now say of it. onBatch string + batched string // label is the maintainer override the original carries, if any. label string // partial lists what its clean net merge onto the line would leave @@ -268,6 +271,10 @@ type batch struct { ancestors map[string]bool // detecting is how long judging partial units and dependencies took. detecting time.Duration + depCache map[string][]dependency + // lineCarries says, for a prerequisite the rules keep out, whether the + // line carries it all the same. + lineCarries map[*verdict]bool } // fetch moves to the top of the working tree first: every path the batch @@ -374,6 +381,9 @@ func (a *app) prepare(l line) (*batch, error) { if b.drops, err = b.bindDrops(merges); err != nil { return nil, err } + if err := b.waitOrder(merges); err != nil { + return nil, err + } for _, o := range merges { v := &verdict{o: o, label: labels[o.number]} if err := b.judge(v, b.drops); err != nil { diff --git a/cmd/backport-batch/notify.go b/cmd/backport-batch/notify.go index 7191f42b4..34f0d8afb 100644 --- a/cmd/backport-batch/notify.go +++ b/cmd/backport-batch/notify.go @@ -87,20 +87,42 @@ func (a *app) notify(args []string) int { return a.fail(err) } deferred := map[int]deferredEntry{} + var partial []int if src.open { - deferred = src.rec.deferred() + deferred, partial = src.rec.deferred(), src.rec.Partial } + // An original that waits on others is listed once, after what decides + // its wait, where that one is listed. var conflicts, waiting []int + after := map[int][]int{} for n, e := range deferred { - switch { - case e.Workflows: + switch place := placement(deferred, n, map[int]bool{}); { + case place == "workflow": waiting = append(waiting, n) - case e.Reason == "": + case place == "conflict": conflicts = append(conflicts, n) } + if len(e.Needs) > 0 { + after[n] = e.Needs + } } slices.Sort(conflicts) slices.Sort(waiting) + item := func(n int) string { + if needs := after[n]; len(needs) > 0 { + return fmt.Sprintf("\n- %s, after %s", prLink(n), prLinks(needs)) + } + return "\n- " + prLink(n) + } + count := func(ns []int) int { + c := 0 + for _, n := range ns { + if len(after[n]) == 0 { + c++ + } + } + return c + } today := a.now().UTC().Truncate(24 * time.Hour) days := int(today.Sub(opened).Hours() / 24) @@ -110,21 +132,34 @@ func (a *app) notify(args []string) int { fmt.Fprintf(a.stdout, "%s: batch pull request #%d has been open since %s, %d days\n", l.branch(), pr.Number, opened.Format(time.DateOnly), days) text = append(text, fmt.Sprintf("This batch has been open since %s, %d days, through at least one scheduled run. Merge it, or close it and the next run starts a new one; until then each run adds what the rules take to it.", opened.Format(time.DateOnly), days)) } - if len(conflicts) > 0 { - fmt.Fprintf(a.stdout, "%s: %d deferred conflicts on #%d wait for a maintainer\n", l.branch(), len(conflicts), pr.Number) + if len(partial) > 0 { + fmt.Fprintf(a.stdout, "%s: %d units on #%d carry only part of their originals\n", l.branch(), len(partial), pr.Number) var b strings.Builder - fmt.Fprintf(&b, "These conflicts wait for a maintainer, as [resolving a conflict](%sresolve-conflict.md) describes; unresolved, each goes to the next batch:\n", tasksURL) + b.WriteString("These are on the batch but carry only part of their originals on this line; take each out with `backport-batch run --defer` before the cut, and port what applies as a pull request of its own:\n") + for _, n := range partial { + b.WriteString(item(n)) + } + text = append(text, b.String()) + } + if len(conflicts) > 0 { + fmt.Fprintf(a.stdout, "%s: %d deferred conflicts on #%d wait for a maintainer\n", l.branch(), count(conflicts), pr.Number) + var b strings.Builder + fmt.Fprintf(&b, "These conflicts wait for a maintainer, as [resolving a conflict](%sresolve-conflict.md) describes; unresolved, each goes to the next batch, and so does what waits on one:\n", tasksURL) for _, n := range conflicts { - fmt.Fprintf(&b, "\n- %s", prLink(n)) + b.WriteString(item(n)) } text = append(text, b.String()) } if len(waiting) > 0 { - fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s and wait for a maintainer\n", l.branch(), len(waiting), pr.Number, workflowsDir) + if count(waiting) == len(waiting) { + fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s and wait for a maintainer\n", l.branch(), len(waiting), pr.Number, workflowsDir) + } else { + fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s or wait on one that does\n", l.branch(), len(waiting), pr.Number, workflowsDir) + } var b strings.Builder - fmt.Fprintf(&b, "These change `%s`, which the batch's token may not push, and wait for a maintainer to bring them over, as [changes to workflows](%s#changes-to-workflows) describes:\n", workflowsDir, policyURL) + fmt.Fprintf(&b, "These change `%s`, which the batch's token may not push, and wait for a maintainer to bring them over, as [changes to workflows](%s#changes-to-workflows) describes, or wait on one that does:\n", workflowsDir, policyURL) for _, n := range waiting { - fmt.Fprintf(&b, "\n- %s", prLink(n)) + b.WriteString(item(n)) } text = append(text, b.String()) } @@ -134,6 +169,36 @@ func (a *app) notify(args []string) int { return a.comment(pr, *mention, today, text) } +// placement says under which list notify names original n of the record: +// workflow for a wait for a maintainer, conflict for a deferred conflict, "" +// for neither, a deferral for a reason among them. One that waits on others +// goes where what decides its wait goes, followed down the record; one that +// is not in the record is held by rule 6 or missing, and waits on nobody. +func placement(deferred map[int]deferredEntry, n int, seen map[int]bool) string { + e, ok := deferred[n] + switch { + case !ok || seen[n]: + return "" + case e.Workflows: + return "workflow" + case e.Reason != "": + return "" + case len(e.Needs) == 0: + return "conflict" + } + seen[n] = true + place := "" + for _, f := range e.Needs { + switch p := placement(deferred, f, seen); { + case p == "workflow": + return p + case p == "conflict": + place = p + } + } + return place +} + // comment writes what waits into the one notify comment on the batch pull // request: it edits the comment if there is one, and opens it only when // something waits. Only a comment by an author the batch trusts is taken for diff --git a/cmd/backport-batch/park_test.go b/cmd/backport-batch/park_test.go index 6c8f51d0a..94a06ada5 100644 --- a/cmd/backport-batch/park_test.go +++ b/cmd/backport-batch/park_test.go @@ -17,6 +17,7 @@ limitations under the License. package main import ( + "fmt" "slices" "strings" "testing" @@ -198,6 +199,263 @@ func TestOnBatchAfterAnEmptyFirstPick(t *testing.T) { } } +// featWorld is #5071's shape: #3, a feat the line never gets, adds a file, +// and #5 fixes that file; #6 is a plain fix beside it. +func featWorld(t *testing.T) *fixture { + t.Helper() + f := newFixture(t) + f.lines() + f.pr(3, "feat(app): tap", "", map[string]string{"app/tap.go": appFile}) + f.pr(5, "fix(app): fix the tap", "```release-note\nfix(app): the tap works.\n```\n", map[string]string{"app/tap.go": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + return f +} + +// A fix of a file a feat added on main, which the line never gets, is +// parked, not tried, and check takes the park as a note. +func TestParkAFixOfAFeatureTheLineLacks(t *testing.T) { + f := featWorld(t) + gh := &fakeGH{} + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push") + if code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + if strings.Contains(stdout, "### Deferred conflicts") || !strings.Contains(stdout, `"parked":[{"original":5,`) || !strings.Contains(stdout, `"kind":"dependency","needs":[3]`) { + t.Errorf("#5 is not parked on #3:\n%s", stdout) + } + if !strings.Contains(stdout, "### Parked") || !strings.Contains(stdout, "builds on [#3]") { + t.Errorf("the body does not say #5 builds on #3:\n%s", stdout) + } + f.git("switch", "-q", "main") + reopen(f, gh) + code, stdout, stderr = f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "note: #5 is parked: it builds on #3, which never reaches release-1.1") { + t.Errorf("check = %d, want 0 with a note that #5 is parked:\n%s\n%s", code, stdout, stderr) + } +} + +// backport/include lifts no park. +func TestIncludeDoesNotLiftThePark(t *testing.T) { + f := featWorld(t) + code, stdout, stderr := f.exec(&fakeGH{labels: map[string][]int{labelInclude: {5}}}, "run", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, `"parked":[{"original":5,`) || !strings.Contains(stdout, "`backport/include`, parked: builds on") { + t.Errorf("run with backport/include on #5 = %d, want #5 still parked:\n%s\n%s", code, stdout, stderr) + } +} + +// A prerequisite that is itself parked, but adapted onto the batch by hand, +// satisfies what builds on it: #3 is parked on #2, a feat, and #5 changes a +// line #3 wrote. +func TestAdaptedPrerequisiteSatisfiesItsDependent(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{"app/a.txt": appFile, "app/b.txt": appFile}) + f.lines() + f.pr(20, "feat(app): a loud two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + mc3 := f.pr(3, "fix(app): an excited two, a loud three", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/b.txt": strings.Replace(appFile, "three", "THREE", 1)}) + f.pr(5, "fix(app): an excited three", "", map[string]string{"app/b.txt": strings.Replace(appFile, "three", "THREE!", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, `"kind":"dependency","needs":[3]`) { + t.Fatalf("run = %d, want #5 parked as long as #3 is:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + w := emptyBatch(f, &fakeGH{}, "batch\n\n"+deferredBlock(line11, nil, nil, nil)+"\n") + // #3 adapted by hand: every file it changes, changed, in the batch's + // shape. + pick := w.pick(w.head, f.git("rev-parse", mc3+"^2"), map[string]string{"app/a.txt": strings.Replace(appFile, "two", "two!", 1), "app/b.txt": strings.Replace(appFile, "three", "THREE", 1)}) + fold := w.merge(w.head, pick, "fix(app): an excited two, a loud three (#3)\n\n"+batchTrailer+": "+mc3) + f.git("push", "-q", "origin", fold+":refs/heads/"+batchBranch1) + w.gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, w.gh.prs[0].Body)} + code, stdout, stderr := f.exec(w.gh, "check", "--line", "release-1.1") + if code != 1 || strings.Contains(stderr, "#5 is parked") || strings.Contains(stdout+stderr, "waits on") || !strings.Contains(stdout, "/pull/5 ") { + t.Errorf("check = %d, want #5 missing, not parked:\n%s\n%s", code, stdout, stderr) + } + code, stdout, stderr = f.exec(w.gh, "run", "--line", "release-1.1", "--continue") + if code != 0 || !strings.Contains(stdout, `"picked":[3,5]`) { + t.Errorf("run --continue = %d, want #5 picked after #3's adaptation:\n%s\n%s", code, stdout, stderr) + } +} + +// What an earlier unit of the batch brought counts: #3 writes the line #8 +// changes, a feat takes it back and another feat, #6, writes it again, so +// blame names #6, which never comes; once #3 is picked the branch has the +// line all the same. +func TestContentFromAnotherUnitSatisfies(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(3, "fix(app): a loud two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(4, "feat(app): a quiet two", "", map[string]string{"app/a.txt": appFile}) + f.pr(6, "feat(app): a loud two again", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(8, "fix(app): an excited two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(app): a loud two (#3)", "fix(app): an excited two (#8)"}) { + t.Errorf("picked %q, want #3 and #8:\n%s", got, stdout) + } +} + +// auxWorld is #5142's shape: #3, a feat, adds a bats test; #5 fixes a +// workflow and that test, so all it builds on that never comes is a test; +// #6 is a plain fix. +func auxWorld(t *testing.T) *fixture { + t.Helper() + f := newFixture(t) + f.squash(2, "chore: a workflow", "", map[string]string{".github/workflows/release.yaml": "on: push\njobs: {}\n"}) + f.lines() + f.pr(3, "feat(ci): test the changelog", "", map[string]string{"hack/changelog.bats": "@test x {\n true\n}\n"}) + f.pr(5, "fix(ci): run the release on tags", "", map[string]string{".github/workflows/release.yaml": "on: [push, tag]\njobs: {}\n", "hack/changelog.bats": "@test x {\n false\n}\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + return f +} + +// A workflow original whose only missing dependency is a test waits for a +// maintainer in the scheduled run, whose token cannot push workflows, and +// check holds the cut on it. A maintainer's own run parks it. With no batch +// open, or one whose body carries no record, it is missing; a record from +// before the mode was saved says the scheduled mode by listing a wait. +func TestWorkflowModeCarriesToCheck(t *testing.T) { + f := auxWorld(t) + gh := &fakeGH{} + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 || strings.Contains(stdout, `"parked":[{`) || !strings.Contains(stdout, `{"original":5,"files":[".github/workflows/release.yaml"],"workflows":true}`) { + t.Fatalf("scheduled run = %d, want #5 waiting, not parked:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#5 changes .github/workflows/ and waits on batch PR #90") { + t.Errorf("check --cut = %d, want #5 waiting:\n%s\n%s", code, stdout, stderr) + } + v1 := "batch\n\n" + deferredBlock(line11, nil, nil, []pick{{v: &verdict{o: &original{number: 5}}, files: []string{".github/workflows/release.yaml"}}}) + "\n" + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, v1)} + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#5 changes .github/workflows/ and waits on batch PR #90") { + t.Errorf("check --cut with a version 1 record = %d, want #5 waiting:\n%s\n%s", code, stdout, stderr) + } + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, "a batch someone wrote")} + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/5 ") || strings.Contains(stderr, "parked") { + t.Errorf("check under a batch with no record = %d, want #5 missing:\n%s\n%s", code, stdout, stderr) + } + + f = auxWorld(t) + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/5 ") || strings.Contains(stderr, "parked") { + t.Errorf("check with no batch = %d, want #5 missing:\n%s\n%s", code, stdout, stderr) + } + gh = &fakeGH{} + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 0 || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Fatalf("a maintainer's run = %d, want #5 parked:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "note: #5 is parked") { + t.Errorf("check after a maintainer's run = %d, want 0 with #5 parked:\n%s\n%s", code, stdout, stderr) + } +} + +// Evidence found only in a test file still parks an original that changes +// no workflow: #4973's shape. +func TestTestOnlyEvidenceParksOtherOriginals(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: nodes", "", map[string]string{"app/nodes.go": "package app\n", "app/nodes_test.go": "package app\n\nfunc a() {}\n"}) + f.lines() + f.pr(3, "feat(app): proxmox nodes", "", map[string]string{"app/nodes_test.go": "package app\n\nfunc a() { proxmox() }\n"}) + f.pr(5, "fix(app): nodes", "", map[string]string{"app/nodes.go": "package app\n\nvar fixed = true\n", "app/nodes_test.go": "package app\n\nfunc a() { proxmox(); fixed() }\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 || strings.Contains(stdout, "### Deferred conflicts") || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("run = %d, want #5 parked on #3:\n%s\n%s", code, stdout, stderr) + } +} + +// What an original builds on through lines it adds is not seen: #4973's +// $isProxmox. The original is picked; CI on the batch is what catches it. +func TestAddedLinesDependencyIsNotSeen(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: nodes", "", map[string]string{"app/nodes.tpl": "a\nb\n"}) + f.lines() + f.pr(3, "feat(app): define isProxmox", "", map[string]string{"app/proxmox.tpl": "isProxmox\n"}) + f.pr(5, "fix(app): use isProxmox", "", map[string]string{"app/nodes.tpl": "a\nif isProxmox\nb\n"}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stdout, `"parked":[{`) || !strings.Contains(stdout, `"picked":[5]`) { + t.Errorf("run = %d, want #5 picked:\n%s\n%s", code, stdout, stderr) + } +} + +// With no batch open, check reads a closed record whatever its version and +// whatever mode it saved as the scheduled run's: the workflow original it +// left waiting is missing, not parked, while its drops and its other +// deferrals still count. +func TestClosedRecordModeFallback(t *testing.T) { + for _, version := range []int{2, 3} { + for _, saved := range []bool{false, true} { + t.Run(fmt.Sprintf("version %d, noWorkflowFiles %v", version, saved), func(t *testing.T) { + f := auxWorld(t) + mc6 := f.git("log", "--first-parent", "--format=%H", "-1", "--grep", "(#6)$", "main") + mc7 := f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + sha := f.git("rev-parse", "origin/main") + rec := lineRecord{Version: version, Line: "release-1.1", Report: strings.Repeat("b", 64), + Built: &builtFrom{Main: sha, Line: sha, Policy: sha, Batch: sha, At: "2026-10-26T06:00:00Z", Options: runOptions{NoWorkflowFiles: saved, Mode: modeContinue}}, + Deferred: []deferredEntry{{Original: 5, Files: []string{".github/workflows/release.yaml"}, Workflows: true}, {Original: 7, Commit: f.git("rev-parse", mc7+"^2"), Files: []string{"app/seven.txt"}}}, + Dropped: []droppedEntry{{Original: 6, Merge: mc6, Reason: "not for this line", On: 90, Date: "2026-10-26"}}, Stale: []int{}} + pr := f.batchPR(90, "release-1.1", batchBranch1, "batch\n\n"+recordBlock(rec)+"\n") + at := "2026-10-26T12:00:00Z" + pr.State, pr.MergedAt = "closed", &at + code, stdout, stderr := f.exec(&fakeGH{prs: []pullRequest{pr}}, "check", "--line", "release-1.1") + if code != 1 || !strings.Contains(stdout, "/pull/5 ") || strings.Contains(stderr, "#5 is parked") { + t.Errorf("check = %d, want #5 missing and not parked:\n%s\n%s", code, stdout, stderr) + } + if !strings.Contains(stderr, "note: #6 is dropped") || !strings.Contains(stderr, "warning: deferred conflict on #90 (merged): https://github.com/cozystack/cozystack/pull/7 ") { + t.Errorf("check lost the closed record's drop or deferral:\n%s", stderr) + } + }) + } + } +} + +// Withdrawing a drop puts the original back where the rules and the line +// put it: a partial original is parked again, in the record as in check. +func TestUndropRestoresThePark(t *testing.T) { + for name, world := range map[string]func(*testing.T) *fixture{ + "partial": func(t *testing.T) *fixture { + f, _ := policyWorld(t) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + return f + }, + "dependency": featWorld, + } { + t.Run(name, func(t *testing.T) { + f := world(t) + gh := &fakeGH{} + openBatch(t, f, gh) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--drop", "5", "--reason", "not for 1.1", "--push") + reopen(f, gh) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--undrop", "5", "--push") + reopen(f, gh) + if rec := recordOf(t, gh); len(rec.Parked) != 1 || rec.Parked[0].Original != 5 || rec.Parked[0].Kind != name { + t.Errorf("after the undrop the record parks %+v, want #5 parked %s", rec.Parked, name) + } + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "note: #5 is parked") { + t.Errorf("check = %d, want #5 parked:\n%s\n%s", code, stdout, stderr) + } + }) + } +} + +// A prerequisite the open batch carries is there for what builds on it, +// whatever its label now says: check and run agree that #7 is not parked. +func TestPrerequisiteStillOnTheBatchSatisfies(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + gh := &fakeGH{} + openBatch(t, f, gh) + f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + gh.labels = map[string][]int{labelSkip: {5}} + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || strings.Contains(stderr, "#7 is parked") || !strings.Contains(stdout, "/pull/7 ") { + t.Errorf("check = %d, want #7 missing, not parked:\n%s\n%s", code, stdout, stderr) + } + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue"); code != 0 || !strings.Contains(stdout, `"picked":[7]`) { + t.Errorf("run --continue = %d, want #7 picked:\n%s\n%s", code, stdout, stderr) + } +} + // The picker judges what its picks did where the run has got to: #4 // deletes the file #5 also deletes, so after #4 #5 would bring only its // test, which the net merge at the line's head cannot see. @@ -231,6 +489,64 @@ func TestPartialClearedByAnEarlierPick(t *testing.T) { } } +// A reason a person deferred an original for comes before its partial +// park. +func TestReasonComesBeforeThePartialPark(t *testing.T) { + f, _ := policyWorld(t) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "waits for the policy rework", "--push") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "warning: deferred on #90 to the next batch: https://github.com/cozystack/cozystack/pull/5 ") || strings.Contains(stderr, "#5 is parked") { + t.Errorf("check = %d, want #5 deferred for its reason:\n%s\n%s", code, stdout, stderr) + } +} + +// The workflow exception holds only while what never comes is tests or +// docs, and while the workflow lines build on nothing missing: a +// workflow original that also fixes a file a feat added, or whose workflow +// line another original still to come wrote, parks in the scheduled run too. +func TestWorkflowExceptionNeedsTestsOnly(t *testing.T) { + t.Run("a never dependency outside tests", func(t *testing.T) { + f := auxWorld(t) + f.pr(20, "feat(app): tap", "", map[string]string{"app/tap.go": appFile}) + f.pr(7, "fix(ci): test the tap, and fix it", "", map[string]string{".github/workflows/tap.yaml": "on: push\n", "app/tap.go": strings.Replace(appFile, "two", "TWO", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1", "--no-workflow-files"); code != 0 || !strings.Contains(stdout, `"parked":[{"original":7,`) { + t.Errorf("scheduled run = %d, want #7 parked:\n%s\n%s", code, stdout, stderr) + } + }) + t.Run("a workflow line still to come", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a workflow", "", map[string]string{".github/workflows/release.yaml": "on: push\njobs: {}\n", "app/four.txt": "old\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): four, the line's way", map[string]string{"app/four.txt": "line\n"}) + f.pr(3, "feat(ci): test the changelog", "", map[string]string{"hack/changelog.bats": "@test x {\n true\n}\n"}) + f.pr(4, "fix(ci): four", "", map[string]string{"app/four.txt": "four\n", ".github/workflows/release.yaml": "on: [push]\njobs: {}\n"}) + f.pr(5, "fix(ci): run the release on tags", "", map[string]string{".github/workflows/release.yaml": "on: [push, tag]\njobs: {}\n", "hack/changelog.bats": "@test x {\n false\n}\n"}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1", "--no-workflow-files"); code != 0 || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("scheduled run = %d, want #5 parked:\n%s\n%s", code, stdout, stderr) + } + }) +} + +// What a rebuild takes off the batch is no longer there for what builds on +// it: deferring #5 makes #7, which merged after the batch was built, wait +// on it. +func TestRebuildTakesThePrerequisiteAway(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "fails the e2e"); code != 0 || !strings.Contains(stdout, `{"original":7,"files":[],"needs":[5]}`) { + t.Errorf("run --defer 5 = %d, want #7 waiting on #5:\n%s\n%s", code, stdout, stderr) + } +} + // A file the line renamed the way the original does is the original's own // rename, not a deletion left without effect. func TestSameRenameOnBothSidesIsWhole(t *testing.T) { @@ -245,6 +561,54 @@ func TestSameRenameOnBothSidesIsWhole(t *testing.T) { } } +// A partial park only the picker saw, where the net merge at the line's +// head looks whole, holds in check and in the operations that pick nothing +// as the open record keeps it: check does not call the original missing. +func TestPickerParkHoldsInCheck(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.pr(4, "fix(app): drop the policy", "", map[string]string{"app/policy.yaml": "", "app/four.txt": "4\n"}) + f.pr(40, "feat(app): the policy again", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.pr(5, "fix(app): drop the policy and test it", "", map[string]string{"app/policy.yaml": "", "app/policy_test.txt": "gone\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "note: #5 is parked: only part of it applies") { + t.Errorf("check = %d, want #5 parked as the record says:\n%s\n%s", code, stdout, stderr) + } + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "6", "--reason", "fails the e2e"); code != 0 || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("run --defer 6 = %d, want #5 still parked:\n%s\n%s", code, stdout, stderr) + } + // Deferring #4, which made #5 partial, ends the park: #5 applies whole + // on what the batch carries now. + f.git("switch", "-q", "main") + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "4", "--reason", "fails the e2e", "--push") + if code != 0 || strings.Contains(stdout, `"parked":[{`) || !strings.Contains(stdout, `"missing":[5]`) { + t.Errorf("run --defer 4 = %d, want #5 missing, no longer parked:\n%s\n%s", code, stdout, stderr) + } + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || strings.Contains(stderr, "#5 is parked") || !strings.Contains(stdout, "/pull/5 ") { + t.Errorf("check after deferring #4 = %d, want #5 missing:\n%s\n%s", code, stdout, stderr) + } +} + +// A workflow original that waits for a maintainer only as the open record +// lists it: one merged since the batch was built is missing, and so is +// what waits on it. +func TestUnlistedWorkflowWaitIsMissing(t *testing.T) { + f := auxWorld(t) + gh := &fakeGH{} + mustRun(t, f, gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push") + f.git("switch", "-q", "main") + reopen(f, gh) + f.pr(8, "fix(ci): run the new release on tags", "", map[string]string{".github/workflows/new.yaml": "on: tag\n", "hack/changelog.bats": "@test x {\n false\n}\n# eight\n", "app/eight.txt": appFile + "TWO\n"}) + f.pr(9, "fix(app): nine", "", map[string]string{"app/eight.txt": appFile + "TWO!\n"}) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/9 fix(app): nine (it waits on #8, which is not on the line either)") { + t.Errorf("check = %d, want #9 missing, waiting on #8:\n%s\n%s", code, stdout, stderr) + } +} + // The line's own rename stands for the original's only with the original's // change in it: a fold that leaves the renamed file as it was is refused. func TestSameRenameStillNeedsItsFix(t *testing.T) { @@ -262,6 +626,47 @@ func TestSameRenameStillNeedsItsFix(t *testing.T) { } } +// A prerequisite the line carries is there, though the rules would keep it +// out: a feat carried by a hand adaptation does not park what builds on it. +func TestCarriedExcludedPrerequisiteSatisfies(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a", "", map[string]string{"app/a.txt": appFile}) + f.lines() + mc3 := f.pr(3, "feat(app): a loud two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.onLine("release-1.1", "feat(app): a loud two, adapted\n\n(cherry picked from commit "+mc3+")", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "Two", 1)}) + f.pr(5, "fix(app): an excited two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stdout, `"parked":[{`) { + t.Errorf("run = %d, want #5 not parked on the carried #3:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.1"); code != 1 || strings.Contains(stderr, "#5 is parked") { + t.Errorf("check = %d, want #5 missing, not parked:\n%s\n%s", code, stdout, stderr) + } +} + +// With no batch open, a workflow wait a closed record keeps waits on +// nothing: the original is judged as the scheduled run judges it, so one +// that also builds on a feat the line lacks is parked, in check as in run. +func TestClosedWorkflowWaitIsJudgedAsRunJudgesIt(t *testing.T) { + f := auxWorld(t) + f.pr(20, "feat(app): tap", "", map[string]string{"app/tap.go": appFile}) + f.pr(7, "fix(ci): test the tap, and fix it", "", map[string]string{".github/workflows/tap.yaml": "on: push\n", "app/tap.go": strings.Replace(appFile, "two", "TWO", 1)}) + sha := f.git("rev-parse", "origin/main") + rec := lineRecord{Version: 2, Line: "release-1.1", Report: strings.Repeat("b", 64), + Built: &builtFrom{Main: sha, Line: sha, Policy: sha, Batch: sha, At: "2026-10-26T06:00:00Z", Options: runOptions{NoWorkflowFiles: true, Mode: modeContinue}}, + Deferred: []deferredEntry{{Original: 7, Files: []string{".github/workflows/tap.yaml"}, Workflows: true}}, Dropped: []droppedEntry{}, Stale: []int{}} + pr := f.batchPR(90, "release-1.1", batchBranch1, "batch\n\n"+recordBlock(rec)+"\n") + at := "2026-10-26T12:00:00Z" + pr.State, pr.MergedAt = "closed", &at + gh := &fakeGH{prs: []pullRequest{pr}} + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); strings.Contains(stdout, "/pull/7 ") || !strings.Contains(stderr, "note: #7 is parked: it builds on #20") { + t.Errorf("check --cut = %d, want #7 parked:\n%s\n%s", code, stdout, stderr) + } + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files"); code != 0 || !strings.Contains(stdout, `"parked":[{"original":7,`) { + t.Errorf("scheduled run = %d, want #7 parked:\n%s\n%s", code, stdout, stderr) + } +} + // A pick that comes out empty is judged too: #5's own commit did half its // fix, which the line has, and a merge of main into its branch did the // rest, which no pick carries; #5 is not on the line. @@ -305,6 +710,51 @@ func TestDeferTakesOffAPartialExcludedUnit(t *testing.T) { } } +// A stored wait on an original merged after the one that waits cannot be +// what a run wrote. +func TestForwardWaitIsRefused(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + sha := f.git("rev-parse", "origin/main") + rec := lineRecord{Version: recordVersion, Line: "release-1.1", Report: strings.Repeat("b", 64), + Built: &builtFrom{Main: sha, Line: sha, Policy: sha, Batch: sha, At: "2026-10-26T06:00:00Z", Options: runOptions{Mode: modeContinue}}, + Deferred: []deferredEntry{{Original: 5, Files: []string{}, Needs: []int{7}}}, Dropped: []droppedEntry{}, Stale: []int{}} + f.git("push", "-q", "origin", "origin/release-1.1:refs/heads/"+batchBranch1) + gh := &fakeGH{prs: []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, "batch\n\n"+recordBlock(rec)+"\n")}} + if code, _, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 2 || !strings.Contains(stderr, "#5 wait on #7") { + t.Errorf("check = %d, want 2 for a wait on a later original:\n%s", code, stderr) + } +} + +// The workflow exception looks only at what still keeps the original from +// the line: a workflow line its prerequisite wrote, which the open batch +// carries though that prerequisite is labelled backport/skip, does not +// stop it, in check as in run. +func TestWorkflowExceptionIgnoresWhatIsThere(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a workflow", "", map[string]string{".github/workflows/release.yaml": "on: push\njobs: {}\n"}) + f.lines() + f.pr(3, "feat(ci): test the changelog", "", map[string]string{"hack/changelog.bats": "@test x {\n true\n}\n"}) + f.pr(4, "fix(ci): run the release on pushes", "", map[string]string{".github/workflows/release.yaml": "on: [push]\njobs: {}\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.pr(7, "fix(ci): run the release on tags too", "", map[string]string{".github/workflows/release.yaml": "on: [push, tag]\njobs: {}\n", "hack/changelog.bats": "@test x {\n false\n}\n"}) + gh := &fakeGH{} + mustRun(t, f, gh, "run", "--line", "release-1.1", "--push") + f.git("switch", "-q", "main") + reopen(f, gh) + gh.labels = map[string][]int{labelSkip: {4}} + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--no-workflow-files", "--push"); code != 0 || !strings.Contains(stdout, `{"original":7,"files":[".github/workflows/release.yaml"],"workflows":true}`) { + t.Fatalf("scheduled run = %d, want #7 waiting for a maintainer:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#7 changes .github/workflows/ and waits on batch PR #90") { + t.Errorf("check --cut = %d, want #7 waiting, as run has it:\n%s\n%s", code, stdout, stderr) + } +} + // An empty pick leaves a deletion undone that only a merge of main into the // original's branch made: the original is not on the line. func TestEmptyPickLeavesADeletionUndone(t *testing.T) { @@ -329,3 +779,18 @@ func TestEmptyPickLeavesADeletionUndone(t *testing.T) { t.Errorf("run = %d, want #5 parked as partial, not on the line:\n%s\n%s", code, stdout, stderr) } } + +// A file the line already has as the original makes it builds on nothing +// the line lacks: #5 fixes a.txt, which the line fixed the same way, and +// b.txt, so it picks whole and is not parked on the feat that wrote a.txt. +func TestAlreadyAppliedFileBuildsOnNothing(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{"app/a.txt": appFile, "app/b.txt": "old\n"}) + f.lines() + f.pr(3, "feat(app): a loud two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.onLine("release-1.1", "fix(app): two, the line's way", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + f.pr(5, "fix(app): an excited two, and b", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/b.txt": "fixed\n"}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stdout, `"parked":[{`) || !strings.Contains(stdout, `"picked":[5]`) { + t.Errorf("run = %d, want #5 picked, not parked:\n%s\n%s", code, stdout, stderr) + } +} diff --git a/cmd/backport-batch/record.go b/cmd/backport-batch/record.go index 81b7626e3..0077b5302 100644 --- a/cmd/backport-batch/record.go +++ b/cmd/backport-batch/record.go @@ -21,6 +21,7 @@ import ( "errors" "fmt" "io" + "maps" "regexp" "slices" "strings" @@ -258,6 +259,9 @@ func parseRecord(body string, l line) (*lineRecord, error) { default: return nil, fmt.Errorf("the %s block is version %d, which this backport-batch does not read; it reads 1 to %d", deferredFence, version, recordVersion) } + if err == nil && version < 3 && slices.ContainsFunc(rec.Deferred, func(d deferredEntry) bool { return d.Needs != nil }) { + err = fmt.Errorf("version %d has no waits on other originals", version) + } if err != nil { return nil, fmt.Errorf("reading the %s block: %w", deferredFence, err) } @@ -294,7 +298,7 @@ func (rec *lineRecord) validate(l line) error { // A conflict names the commit that failed and its files, a deferral // its reason, and a wait its workflow files. kinds := 0 - for _, is := range []bool{d.Commit != "" || len(d.Files) > 0 && !d.Workflows && d.Reason == "", d.Reason != "", d.Workflows} { + for _, is := range []bool{d.Commit != "" || len(d.Files) > 0 && !d.Workflows && d.Reason == "" && len(d.Needs) == 0, d.Reason != "", d.Workflows, len(d.Needs) > 0} { if is { kinds++ } @@ -312,6 +316,13 @@ func (rec *lineRecord) validate(l line) error { return fmt.Errorf("names files of #%d that are not as a run writes them: %q", d.Original, d.Files) case d.More < 0: return fmt.Errorf("counts %d more files for #%d", d.More, d.Original) + case len(d.Needs) > 0 && (len(d.Files) > 0 || d.More != 0): + return fmt.Errorf("has #%d wait on other originals with files of its own", d.Original) + } + for i, n := range d.Needs { + if n <= 0 || n == d.Original || i > 0 && d.Needs[i-1] >= n { + return fmt.Errorf("has #%d wait on %v, which is not what a run writes", d.Original, d.Needs) + } } } for _, d := range rec.Dropped { @@ -347,6 +358,27 @@ func (rec *lineRecord) validate(l line) error { } changedSeen[n] = true } + if err := rec.waitsHold(); err != nil { + return err + } + // What is on the batch is neither deferred nor parked. + left := map[int]bool{} + for _, d := range rec.Deferred { + left[d.Original] = true + } + for _, p := range rec.Parked { + left[p.Original] = true + if slices.Contains(p.Commits, p.Merge) { + return fmt.Errorf("parks #%d for its own merge", p.Original) + } + } + for _, list := range [][]int{rec.Partial, rec.Changed} { + for _, n := range list { + if left[n] { + return fmt.Errorf("has #%d on the batch and left off it at once", n) + } + } + } partialSeen := map[int]bool{} for _, n := range rec.Partial { // A unit that does only part of its original is still on the batch. @@ -401,6 +433,74 @@ func (b *builtFrom) validate() error { var runURLRE = regexp.MustCompile(`^https://[^/\s]+/[^/\s]+/[^/\s]+/actions/runs/[0-9]+(/attempts/[0-9]+)?$`) +// waitsHold holds what waits and parks need to what the record says of the +// originals they need: a wait needs ones still to come, so none parked or +// dropped, a park ones that never come, so none deferred, and no wait ends +// up waiting on itself. Its errors read after "the block". +func (rec *lineRecord) waitsHold() error { + deferred, never := map[int][]int{}, map[int]bool{} + for _, d := range rec.Deferred { + deferred[d.Original] = d.Needs + } + for _, p := range rec.Parked { + never[p.Original] = true + } + for _, d := range rec.Dropped { + never[d.Original] = true + } + for _, d := range rec.Deferred { + for _, n := range d.Needs { + if never[n] { + return fmt.Errorf("has #%d wait on #%d, which never comes", d.Original, n) + } + } + } + for _, p := range rec.Parked { + for _, n := range p.Needs { + if _, ok := deferred[n]; ok { + return fmt.Errorf("parks #%d for #%d, which is still to come", p.Original, n) + } + } + } + needs := map[int][]int{} + for _, d := range rec.Deferred { + needs[d.Original] = d.Needs + } + for _, p := range rec.Parked { + needs[p.Original] = p.Needs + } + return waitCycle(needs) +} + +// waitCycle refuses waits that lead back to where they started, which no +// run writes: a prerequisite merged before what needs it. +func waitCycle(needs map[int][]int) error { + state := map[int]int{} // 1 on the path, 2 done + var visit func(n int) error + visit = func(n int) error { + switch state[n] { + case 1: + return fmt.Errorf("has #%d wait on itself through what it waits on", n) + case 2: + return nil + } + state[n] = 1 + for _, f := range needs[n] { + if err := visit(f); err != nil { + return err + } + } + state[n] = 2 + return nil + } + for _, n := range slices.Sorted(maps.Keys(needs)) { + if err := visit(n); err != nil { + return err + } + } + return nil +} + // lineRecordWithout is a version 2 record as the newest version holds it. func lineRecordWithout(v2 lineRecordV2) lineRecord { return lineRecord{Version: v2.Version, Line: v2.Line, Built: v2.Built, Report: v2.Report, Closing: v2.Closing, Deferred: v2.Deferred, Dropped: v2.Dropped, Stale: v2.Stale} diff --git a/cmd/backport-batch/record_test.go b/cmd/backport-batch/record_test.go index d50685edb..4bbddc58c 100644 --- a/cmd/backport-batch/record_test.go +++ b/cmd/backport-batch/record_test.go @@ -380,10 +380,32 @@ func TestRecordVersion3(t *testing.T) { block := func(json string) string { return "text\n\n" + deferredFence + "\n" + json + "\n```\n" } sha, other := strings.Repeat("a", 40), strings.Repeat("c", 40) parked := `,"parked":[{"original":10,"merge":"` + sha + `","kind":"partial","files":["a","b"]},{"original":11,"merge":"` + sha + `","kind":"dependency","needs":[3,4],"commits":["` + other + `"],"announced":true}],"partial":[9]` - rec, err := parseRecord(block(v3(parked)), l) - if err != nil || rec.Version != 3 || len(rec.Parked) != 2 || !rec.Parked[1].Announced || !slices.Equal(rec.Partial, []int{9}) { + waits := strings.Replace(v3(parked), `{"original":6,"files":[],"reason":"fails the e2e"}`, `{"original":6,"files":[],"needs":[3,5]}`, 1) + rec, err := parseRecord(block(waits), l) + if err != nil || rec.Version != 3 || len(rec.Parked) != 2 || !rec.Parked[1].Announced || !slices.Equal(rec.Partial, []int{9}) || !slices.Equal(rec.deferred()[6].Needs, []int{3, 5}) { t.Fatalf("parseRecord of a whole v3 record = %+v, %v", rec, err) } + for name, json := range map[string]string{ + "version 1 with a wait": `{"line":"release-1.1","deferred":[{"original":6,"files":[],"needs":[3,5]}]}`, + "a wait with files": strings.Replace(waits, `"files":[],"needs":[3,5]`, `"files":["a"],"needs":[3,5]`, 1), + "a wait that is a conflict": strings.Replace(waits, `"files":[],"needs":[3,5]`, `"commit":"`+sha+`","files":[],"needs":[3,5]`, 1), + "a wait on itself": strings.Replace(waits, `"needs":[3,5]`, `"needs":[3,6]`, 1), + "an unsorted wait": strings.Replace(waits, `"needs":[3,5]`, `"needs":[5,3]`, 1), + "a wait on nothing": strings.Replace(waits, `"needs":[3,5]`, `"needs":[0]`, 1), + "a wait cycle": strings.Replace(strings.Replace(waits, `"needs":[3,5]`, `"needs":[12]`, 1), `"deferred":[`, `"deferred":[{"original":12,"files":[],"needs":[6]},`, 1), + "a wait on a parked one": strings.Replace(waits, `"needs":[3,5]`, `"needs":[10]`, 1), + "a wait on a dropped one": strings.Replace(waits, `"needs":[3,5]`, `"needs":[8]`, 1), + "a park on a deferred one": strings.Replace(waits, `"needs":[3,4]`, `"needs":[3,5]`, 1), + "a partial unit also deferred": strings.Replace(strings.Replace(waits, `"stale":[9]`, `"stale":[9,5]`, 1), `"partial":[9]`, `"partial":[9,5]`, 1), + "a changed pick also parked": v3(parked + `,"changed":[10]`), + "a changed pick also deferred": v3(parked + `,"changed":[5]`), + "a park on its own merge": strings.Replace(waits, `"commits":["`+other+`"]`, `"commits":["`+sha+`"]`, 1), + "a park cycle": strings.Replace(strings.Replace(waits, `"needs":[3,4]`, `"needs":[3,13]`, 1), `"parked":[`, `"parked":[{"original":13,"merge":"`+sha+`","kind":"dependency","needs":[11]},`, 1), + } { + if _, err := parseRecord(block(json), l); err == nil { + t.Errorf("%s: parseRecord accepted it", name) + } + } for name, json := range map[string]string{ "version 2 with a park": v2(parked), "version 4": strings.Replace(v3(""), `"version":3`, `"version":4`, 1), diff --git a/cmd/backport-batch/report.go b/cmd/backport-batch/report.go index de201fd4a..d1ae4a4b5 100644 --- a/cmd/backport-batch/report.go +++ b/cmd/backport-batch/report.go @@ -145,7 +145,7 @@ func numbers(vs []*verdict) []int { // deferredEntries is what the record and the report say about the conflicts, // the deferrals and the waits of a run. -func deferredEntries(conflicts, held, waiting []pick) []deferredEntry { +func deferredEntries(conflicts, held, waiting, needs []pick) []deferredEntry { out := []deferredEntry{} for _, p := range conflicts { files := cleanFiles(p.files[:min(len(p.files), filesKept)]) @@ -158,6 +158,9 @@ func deferredEntries(conflicts, held, waiting []pick) []deferredEntry { files := cleanFiles(p.files[:min(len(p.files), filesKept)]) out = append(out, deferredEntry{Original: p.v.o.number, Files: files, More: len(p.files) - len(files) + p.more, Workflows: true}) } + for _, p := range needs { + out = append(out, deferredEntry{Original: p.v.o.number, Files: []string{}, Needs: slices.Sorted(slices.Values(p.needs))}) + } return out } @@ -188,7 +191,7 @@ func buildReport(b *batch, r *report) (reportDoc, error) { doc.Overrides.Skip = append(doc.Overrides.Skip, v.o.number) } } - doc.Primary = reportPrimary{Picked: numbers(r.onBranch), Missing: numbers(r.missing), Deferred: deferredEntries(r.deferred, r.held, r.waiting), Parked: parkedEntries(r.parked)} + doc.Primary = reportPrimary{Picked: numbers(r.onBranch), Missing: numbers(r.missing), Deferred: deferredEntries(r.deferred, r.held, r.waiting, r.needs), Parked: parkedEntries(r.parked)} for _, l := range doc.Primary.Skipped.lists() { *l.list = []int{} } @@ -415,6 +418,9 @@ func (d *reportDoc) links() error { return fmt.Errorf("parks #%d for #%d, which it has %s", p.Original, n, or(c, "nowhere")) } } + if other[p.Merge] || slices.Contains(p.Commits, p.Merge) { + return fmt.Errorf("parks #%d by a merge it counts as no pull request's, or for its own merge", p.Original) + } for _, c := range p.Commits { if !other[c] { return fmt.Errorf("parks #%d for %.12s, which it does not count among main's commits that merged no pull request", p.Original, c) @@ -480,6 +486,28 @@ func (d *reportDoc) links() error { if e.Workflows && !d.Built.Options.NoWorkflowFiles { return fmt.Errorf("has #%d waiting for its workflow files, but was made bringing them over", e.Original) } + // What a wait waits on is still to come: not on the line, deferred, + // or held by rule 6. + for _, n := range e.Needs { + switch c := where[n]; c { + case "missing", "deferred", "skipped.newer": + default: + return fmt.Errorf("has #%d wait on #%d, which it has %s", e.Original, n, or(c, "nowhere")) + } + } + if d.Format == 1 && len(e.Needs) > 0 { + return fmt.Errorf("has #%d wait on other originals, which format 1 does not", e.Original) + } + } + needs := map[int][]int{} + for _, e := range d.Primary.Deferred { + needs[e.Original] = e.Needs + } + for _, p := range d.Primary.Parked { + needs[p.Original] = p.Needs + } + if err := waitCycle(needs); err != nil { + return err } if d.Built.Options.Mode == modeClose && len(d.Primary.Picked)+len(d.Overlays.Stale) > 0 { return errors.New("closes the batch, but still has originals on it") @@ -734,6 +762,12 @@ func (a *app) printReport(pr *pullRequest, s storedReport, inBody *storedReport, switch { case e.Workflows: text += "waits for a maintainer, changes " + strings.Join(e.Files, ", ") + case len(e.Needs) > 0: + var needs []string + for _, n := range e.Needs { + needs = append(needs, "#"+strconv.Itoa(n)) + } + text += "waits on " + strings.Join(needs, ", ") case e.Reason != "": text += e.Reason default: diff --git a/cmd/backport-batch/report_test.go b/cmd/backport-batch/report_test.go index 1e5048e04..40a9a0673 100644 --- a/cmd/backport-batch/report_test.go +++ b/cmd/backport-batch/report_test.go @@ -633,6 +633,27 @@ func TestReportRefusesWhatItCannotVouchFor(t *testing.T) { d.Primary.Parked = []parkedEntry{{Original: 16, Merge: strings.Repeat("b", 40), Kind: parkDependency, Commits: []string{strings.Repeat("c", 40)}}} }, "a partial unit not on the batch": func(d *reportDoc) { d.Overlays.Partial = []int{5} }, + "a wait on an original it picks": func(d *reportDoc) { + d.Primary.Deferred = []deferredEntry{{Original: 6, Files: []string{}, Needs: []int{5}}} + }, + "a wait on an original it puts nowhere": func(d *reportDoc) { + d.Primary.Deferred = []deferredEntry{{Original: 6, Files: []string{}, Needs: []int{19}}} + }, + "a park on its own merge as a commit": func(d *reportDoc) { + d.Other = append(d.Other, strings.Repeat("b", 40)) + d.Primary.Parked = []parkedEntry{{Original: 16, Merge: strings.Repeat("b", 40), Kind: parkDependency, Commits: []string{strings.Repeat("b", 40)}}} + }, + "a park cycle": func(d *reportDoc) { + d.Primary.Parked = []parkedEntry{{Original: 16, Merge: strings.Repeat("b", 40), Kind: parkDependency, Needs: []int{18}}, {Original: 18, Merge: strings.Repeat("b", 40), Kind: parkDependency, Needs: []int{16}}} + }, + "a wait cycle": func(d *reportDoc) { + d.Primary.Deferred = []deferredEntry{{Original: 6, Files: []string{}, Needs: []int{7}}, {Original: 7, Files: []string{}, Needs: []int{6}}} + }, + "format 1 with a wait": func(d *reportDoc) { + d.Format = 1 + d.Primary.Missing = []int{19} + d.Primary.Deferred = []deferredEntry{{Original: 6, Files: []string{}, Needs: []int{19}}} + }, "format 1 with a park": func(d *reportDoc) { d.Format = 1 d.Primary.Parked = []parkedEntry{{Original: 16, Merge: strings.Repeat("b", 40), Kind: parkPartial, Files: []string{"a"}}} diff --git a/cmd/backport-batch/wait_test.go b/cmd/backport-batch/wait_test.go new file mode 100644 index 000000000..35dd5c1d6 --- /dev/null +++ b/cmd/backport-batch/wait_test.go @@ -0,0 +1,236 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "strings" + "testing" +) + +// chainWorld has #5 conflict on the line and change a shared line, and #7 +// change that line again: #7 builds on #5. +func chainWorld(t *testing.T) (*fixture, string) { + t.Helper() + f := newFixture(t) + f.squash(2, "chore: five and shared", "", map[string]string{"app/five.txt": "old\n", "app/shared.txt": appFile}) + f.lines() + f.onLine("release-1.1", "fix(app): five, the line's way", map[string]string{"app/five.txt": "line\n"}) + mc5 := f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "five\n", "app/shared.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/shared.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + return f, mc5 +} + +// An original that builds on a deferred conflict waits for it: check warns, +// notify lists it after the conflict, and once the conflict lands, the next +// run picks it. +func TestWaitForADeferredDependency(t *testing.T) { + f, mc5 := chainWorld(t) + gh := &fakeGH{} + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push") + if code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + if !strings.Contains(stdout, `{"original":7,"files":[],"needs":[5]}`) || !strings.Contains(stdout, "### Waiting on another original") { + t.Errorf("#7 does not wait on #5:\n%s", stdout) + } + f.git("switch", "-q", "main") + reopen(f, gh) + code, stdout, stderr = f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "warning: #7 waits on #5, a deferred conflict on #90: https://github.com/cozystack/cozystack/pull/7 ") { + t.Errorf("check = %d, want a warning that #7 waits on #5:\n%s\n%s", code, stdout, stderr) + } + code, stdout, _ = f.exec(gh, "notify", "--line", "release-1.1", "--push") + if code != 0 || !strings.Contains(gh.issueComments[len(gh.issueComments)-1].Body, "- [#7](https://github.com/cozystack/cozystack/pull/7), after [#5](https://github.com/cozystack/cozystack/pull/5)") { + t.Errorf("notify = %d, want #7 listed after #5:\n%s\n%v", code, stdout, gh.issueComments) + } + // The batch merges, and #5 lands on the line by hand, resolved: the + // next batch picks #7. + mergeBatch(f, gh) + f.onLine("release-1.1", "fix(app): five (#5)\n\n(cherry picked from commit "+mc5+")", map[string]string{"app/five.txt": "five\n", "app/shared.txt": strings.Replace(appFile, "two", "TWO", 1)}) + code, stdout, stderr = f.exec(gh, "run", "--line", "release-1.1", "--continue") + if code != 0 || !strings.Contains(stdout, `"picked":[7]`) { + t.Errorf("run --continue after #5 landed = %d, want #7 picked:\n%s\n%s", code, stdout, stderr) + } +} + +// Of several prerequisites still to come, the strongest decides: one not on +// the line at all, then one waiting for a maintainer, then a conflict or a +// deferral, then one rule 6 holds; a chain inherits what its end decides. +func TestWaitReduction(t *testing.T) { + t.Run("a conflict and a workflow wait", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/five.txt": "old\n", "app/a.txt": appFile, "app/b.txt": appFile}) + f.lines() + f.onLine("release-1.1", "fix(app): five, the line's way", map[string]string{"app/five.txt": "line\n"}) + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "five\n", "app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(6, "fix(ci): six", "", map[string]string{".github/workflows/six.yaml": "on: push\n", "app/b.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/b.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + gh := &fakeGH{} + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 || !strings.Contains(stdout, `{"original":7,"files":[],"needs":[6]}`) { + t.Fatalf("run = %d, want #7 waiting on #6 alone:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#7 waits on #6, which changes .github/workflows/ and waits on batch PR #90 for a maintainer") { + t.Errorf("check --cut = %d, want #7 to hold the cut:\n%s\n%s", code, stdout, stderr) + } + if code, stdout, _ := f.exec(gh, "notify", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, "2 originals on #90 change .github/workflows/ or wait on one that does") { + t.Errorf("notify = %d, want #7 among the waits:\n%s", code, stdout) + } + }) + t.Run("a conflict and one not on the line", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/five.txt": "old\n", "app/a.txt": appFile, "app/b.txt": appFile}) + f.lines() + f.onLine("release-1.1", "fix(app): five, the line's way", map[string]string{"app/five.txt": "line\n"}) + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "five\n", "app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + f.pr(9, "fix(app): nine", "", map[string]string{"app/b.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(10, "fix(app): ten", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/b.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/10 fix(app): ten (it waits on #9, which is not on the line either)") { + t.Errorf("check = %d, want #10 to fail it, waiting on #9:\n%s\n%s", code, stdout, stderr) + } + }) + t.Run("a prerequisite on the open batch", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + gh := &fakeGH{} + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 0 || !strings.Contains(stdout, `"picked":[5,7]`) { + t.Fatalf("run = %d, want #5 and #7 picked:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#5 is only on open batch PR #90") || strings.Contains(stdout+stderr, "waits on") { + t.Errorf("check --cut = %d, want #5 and #7 only on the batch and no wait:\n%s\n%s", code, stdout, stderr) + } + }) + t.Run("a direct reason", func(t *testing.T) { + f := featWorld(t) + gh := &fakeGH{} + openBatch(t, f, gh) + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "needs the tap", "--push"); code != 0 { + t.Fatalf("run --defer = %d\n%s", code, stderr) + } + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "warning: deferred on #90 to the next batch: https://github.com/cozystack/cozystack/pull/5 ") || strings.Contains(stderr, "parked") { + t.Errorf("check = %d, want #5 deferred for its reason, not parked:\n%s\n%s", code, stdout, stderr) + } + }) + t.Run("a prerequisite rule 6 holds", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + mc7 := f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/seven.txt": "7\n"}) + f.onLine("release-1.1", "fix(app): seven (#7)\n\n(cherry picked from commit "+mc7+")", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/seven.txt": "7\n"}) + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.0"); code != 0 || !strings.Contains(stderr, "note: #7 waits on #5, which is not on release-1.1 yet") { + t.Errorf("check of release-1.0 = %d, want a note that #7 waits on #5:\n%s\n%s", code, stdout, stderr) + } + }) + t.Run("a chain", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(4, "fix(app): four", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(5, "fix(app): five", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!!", 1)}) + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/7 fix(app): seven (it waits on #5, which is not on the line either)") { + t.Errorf("check = %d, want #7 failing it through #5:\n%s\n%s", code, stdout, stderr) + } + }) +} + +// run's report, check and notify give an original one disposition: a +// workflow original whose only missing dependency is a test waits for a +// maintainer, and one deferred for a reason stays deferred, though it +// builds on what never comes; neither is parked. +func TestDispositionOrderIsSharedByRunCheckNotify(t *testing.T) { + f := auxWorld(t) + f.pr(20, "feat(app): tap", "", map[string]string{"app/tap.go": appFile}) + f.pr(7, "fix(app): fix the tap", "", map[string]string{"app/tap.go": strings.Replace(appFile, "two", "TWO", 1)}) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "7", "--reason", "waits for the tap", "--push") + if code != 0 { + t.Fatalf("run --defer = %d\n%s", code, stderr) + } + if strings.Contains(stdout, `"parked":[{`) || !strings.Contains(stdout, `{"original":5,"files":[".github/workflows/release.yaml"],"workflows":true}`) || !strings.Contains(stdout, `{"original":7,"files":[],"reason":"waits for the tap"}`) { + t.Errorf("the report does not have #5 waiting and #7 deferred, unparked:\n%s", stdout) + } + reopen(f, gh) + code, stdout, stderr = f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || strings.Contains(stderr, "parked") || !strings.Contains(stderr, "warning: waiting on #90 for a maintainer, since it changes .github/workflows/: https://github.com/cozystack/cozystack/pull/5 ") || !strings.Contains(stderr, "warning: deferred on #90 to the next batch: https://github.com/cozystack/cozystack/pull/7 ") { + t.Errorf("check = %d, want #5 waiting and #7 deferred:\n%s\n%s", code, stdout, stderr) + } + code, stdout, _ = f.exec(gh, "notify", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, "1 originals on #90 change .github/workflows/") || strings.Contains(stdout, "parked") || strings.Contains(stdout, "conflicts") { + t.Errorf("notify = %d, want #5 waiting and nothing for #7:\n%s", code, stdout) + } +} + +// A never and a pending prerequisite together park. +func TestParkWinsOverWait(t *testing.T) { + f, _ := chainWorld(t) + f.pr(20, "feat(app): tap", "", map[string]string{"app/tap.go": appFile}) + f.pr(8, "fix(app): eight", "", map[string]string{"app/shared.txt": strings.Replace(appFile, "two", "TWO!!", 1), "app/tap.go": strings.Replace(appFile, "two", "TWO", 1)}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, `"kind":"dependency","needs":[20]`) || strings.Contains(stdout, `"original":8,"files":[],"needs"`) { + t.Errorf("run = %d, want #8 parked on #20, not waiting on #7:\n%s\n%s", code, stdout, stderr) + } +} + +// A prerequisite picked earlier in the same batch satisfies what builds on +// it, and so does one whose pick came out empty because an earlier pick +// already carried it: #4280 and #3938. +func TestDependencySatisfiedEarlierInTheBatch(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(3, "fix(kafka): delete hooks", "", map[string]string{"app/hooks.yaml": appFile}) + f.pr(4, "fix(kafka): fix the delete hook", "", map[string]string{"app/hooks.yaml": strings.Replace(appFile, "two", "TWO", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, `"picked":[3,4]`) { + t.Errorf("run = %d, want #3 and #4 picked:\n%s\n%s", code, stdout, stderr) + } + + // #30 adds back the file #2 brought and a feat took away, so its pick + // comes out empty after #2's, and #4 builds on #30. + f = newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(20, "fix(kafka): delete hooks, and two", "", map[string]string{"app/hooks.yaml": appFile, "app/two.txt": "2\n"}) + f.pr(21, "feat(kafka): no delete hooks", "", map[string]string{"app/hooks.yaml": ""}) + f.pr(30, "fix(kafka): delete hooks again", "", map[string]string{"app/hooks.yaml": appFile}) + f.pr(4, "fix(kafka): fix the delete hook", "", map[string]string{"app/hooks.yaml": strings.Replace(appFile, "two", "TWO", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, `"picked":[4,20]`) || strings.Contains(stdout, `"needs"`) { + t.Errorf("run with #30's pick empty = %d, want #4 picked:\n%s\n%s", code, stdout, stderr) + } +} diff --git a/docs/release/backport-mechanics.md b/docs/release/backport-mechanics.md index 1ee035618..3474bffea 100644 --- a/docs/release/backport-mechanics.md +++ b/docs/release/backport-mechanics.md @@ -14,7 +14,7 @@ go build ./cmd/backport-batch ./backport-batch check --line release-X.Y --cut ``` -[`backport-batch check`](../../cmd/backport-batch/README.md) applies the rules to every pull request merged into `main` since the line branched, and looks for each one they take on the line and then on the open batch pull request: by picks whose `-x` lines name its merge commit or every commit of its own, by a pick that would change nothing, and, on commits from before the policy's batch-only day, by what their text says, such as `Backport of #N`. With `--cut` a change found only on the open batch pull request fails the check too, since a cut tags the line and would not carry it: merge the batch pull request first. A conflict the batch deferred, or an original deferred to the next batch, prints as a warning without failing the check, also once that batch merged or closed. An original that waits on the open batch for a maintainer fails the cut, and so does a unit on the open batch that carries only part of its original. A change that reached the line by hand counts like any other, by what reached the line. +[`backport-batch check`](../../cmd/backport-batch/README.md) applies the rules to every pull request merged into `main` since the line branched, and looks for each one they take on the line and then on the open batch pull request: by picks whose `-x` lines name its merge commit or every commit of its own, by a pick that would change nothing, and, on commits from before the policy's batch-only day, by what their text says, such as `Backport of #N`. With `--cut` a change found only on the open batch pull request fails the check too, since a cut tags the line and would not carry it: merge the batch pull request first. A conflict the batch deferred, or an original deferred to the next batch, prints as a warning without failing the check, also once that batch merged or closed. An original that waits on the open batch for a maintainer fails the cut, and so does a unit on the open batch that carries only part of its original. An original parked because it builds on a change the line never gets, or would land only in part, prints as a note; one that waits on another original fails, warns or notes by what it waits on. A change that reached the line by hand counts like any other, by what reached the line. ### Backporting code that touches monotonic counters diff --git a/docs/release/backport-tasks/resolve-conflict.md b/docs/release/backport-tasks/resolve-conflict.md index ed3798d10..885e568d4 100644 --- a/docs/release/backport-tasks/resolve-conflict.md +++ b/docs/release/backport-tasks/resolve-conflict.md @@ -34,7 +34,7 @@ git add git -c core.hooksPath=/dev/null cherry-pick --continue --no-edit ``` -When the original builds on something the line does not have, a helper, a value or another pull request, the resolution would have to bring that along, which this task does not allow: abort with `git cherry-pick --abort`, go back with `git switch --detach "$base"`, and comment on the batch pull request which change the original depends on; it stays deferred until that change is on the line. When that change will never reach the line, drop the original there instead, with `backport-batch run --line --drop N --reason "" --push`, so later batches stop trying it. A commit whose change the line already has stops the pick as empty; skip it with `git cherry-pick --skip`. A resolution that ports only what the line has and leaves a file of the original as it was, one the line does not have for instance, is refused by `fold`: such a port goes to the line as a pull request of its own, as [fixes that exist only on a release line](../backports.md#fixes-that-exist-only-on-a-release-line) describes, and the original is dropped there. +When the original builds on something the line does not have, a helper, a value or another pull request, the resolution would have to bring that along, which this task does not allow: abort with `git cherry-pick --abort`, go back with `git switch --detach "$base"`, and comment on the batch pull request which change the original depends on; it stays deferred until that change is on the line. An original the batch parked because it builds on such a change is the same task, and lands only as a complete adaptation: every file it changes changed, unless the line already has that file as the original makes it. When that change will never reach the line, drop the original there instead, with `backport-batch run --line --drop N --reason "" --push`, so later batches stop trying it. A commit whose change the line already has stops the pick as empty; skip it with `git cherry-pick --skip`. A resolution that ports only what the line has and leaves a file of the original as it was, one the line does not have for instance, is refused by `fold`: such a port goes to the line as a pull request of its own, as [fixes that exist only on a release line](../backports.md#fixes-that-exist-only-on-a-release-line) describes, and the original is dropped there. For a merged pull request, join the picks to the batch head with the merge commit: diff --git a/docs/release/backports.md b/docs/release/backports.md index f76cc3438..71fa467eb 100644 --- a/docs/release/backports.md +++ b/docs/release/backports.md @@ -27,6 +27,8 @@ A pull request the rules take for every line can still be wrong for one of them, The batch also leaves a pull request out of a line on its own when it cannot land there whole. A pick that would leave part of what the pull request does on `main` without effect on the line, a change to a file the line does not have, or the deletion of a file the line never had or already removed, is not made: the pull request is parked, the batch pull request lists it under "Parked" with those files, and `check` prints it as a note without failing. A file the line already has as the pull request makes it counts as there, and so does a file `main` or the line renamed, under its other name. A unit already on the open batch that does only part of its original fails `check --cut` and `backport-batch verify`, whatever its label says, until `backport-batch run --defer` takes it out. What applies of a parked pull request goes to the line as a pull request of its own, as [fixes that exist only on a release line](#fixes-that-exist-only-on-a-release-line) describes. +A pull request is parked too when it builds on a change the line never gets: it changes a file a `feat` added on `main`, say, or a line such a change wrote, and the line has neither. `backport-batch` reads that from git, as [its README](../../cmd/backport-batch/README.md#what-an-original-builds-on) describes, and names the change in the batch pull request. One that builds on a change that is still to come, a deferred conflict, one waiting for a maintainer, one rule 6 holds or one not tried yet, waits for it instead, listed under "Waiting on another original", and the batch tries it again once that change is on the line; one picked earlier in the same batch counts as there. A complete hand adaptation of a parked pull request, every file it changes changed, still lands on the batch through [resolving a conflict](./backport-tasks/resolve-conflict.md). + ### What waits for the next minor `feat`, `chore` and `docs` pull requests, and dependency bumps (scope `deps`), ship with the next minor, not in a patch release; only a maintainer's `backport/include`, below, takes one. If a class of fixes should also wait for a minor, it gets a new rule on this page; a single pull request that the rules get wrong is what the overrides below are for. @@ -103,7 +105,7 @@ Each run also reports where every pull request merged into `main` since the line ### Changes to workflows -The scheduled run's token may not push a commit that changes `.github/workflows/`, so the scheduled run leaves such an original out (`run --no-workflow-files`). The batch pull request lists it under "Waiting for a maintainer", with the workflow files it changes, and so does the batch's comment. It is no conflict. The policy still takes it: `check` warns about it while it waits, and `check --cut` fails on it. When nothing else is picked there is no batch pull request to list it on, and `check` reports it as missing. +The scheduled run's token may not push a commit that changes `.github/workflows/`, so the scheduled run leaves such an original out (`run --no-workflow-files`). That holds also for one whose only missing dependency that never comes is a test or documentation, which a maintainer's run, with the token that may push the workflow, parks instead: what lands of it then is a pull request of its own; `check` reads which of the two the open batch was assembled as from its record, and with no batch open takes the scheduled run's. The batch pull request lists it under "Waiting for a maintainer", with the workflow files it changes, and so does the batch's comment. It is no conflict. The policy still takes it: `check` warns about it while it waits, and `check --cut` fails on it. When nothing else is picked there is no batch pull request to list it on, and `check` reports it as missing. A maintainer brings it over with their own token, one that may push workflow changes (the Workflows permission, or the classic `workflow` scope), by continuing the open batch without the flag: From 885752eb5aeed5e31b3fd6dd2922bca64ee990cd Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 10 Nov 2026 11:11:15 +0000 Subject: [PATCH 408/518] feat(release): announce a parked original once on the open batch A park is a decision a maintainer has to act on: port what applies as a pull request of its own, or drop the original. Listing every park in every notify comment would make that list noise within a week, and a park nobody hears about is a fix that silently never ships. notify now names each park the open batch's record holds that is not yet marked announced, and once its comment is written marks those parks in the record with gh pr edit --body-file. It writes only while the pull request is open and its body still carries the record notify read, so a run that rewrote the body in between loses nothing; a mark that fails exits 2 and the next notify names the park once more. run carries the mark into the next record, a merged or closed carrier's too, only while the park is the same event: merge, kind, prerequisites, commits and files. A park that changed is news and is named again. With no batch open nothing is named (M2); the next batch's first notify does it. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 2 + cmd/backport-batch/assemble.go | 4 +- cmd/backport-batch/body.go | 3 + cmd/backport-batch/fixture_test.go | 16 +++ cmd/backport-batch/github.go | 7 ++ cmd/backport-batch/notify.go | 72 ++++++++++++- cmd/backport-batch/notify_test.go | 160 +++++++++++++++++++++++++++++ cmd/backport-batch/record.go | 33 ++++++ docs/release/backports.md | 2 +- 9 files changed, 295 insertions(+), 4 deletions(-) diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index c0b34bb3c..81afd0ef6 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -73,6 +73,8 @@ Reads the [line's record](#the-lines-record-and-reports) first, as the other com An original the record has wait on others is listed once, as `#M, after #F`, where what decides its wait is listed: after a deferred conflict with the conflicts, after a workflow wait with the waits; one waiting on what is missing or held by rule 6 is not listed. A unit on the batch that carries only part of its original is listed apart, to be taken out before the cut. +Each park the record of the open batch holds is named once. `run` marks a park `announced` when the line's record it found, open or closed, holds the same park announced: the same merge, kind, prerequisites, commits and files; a park that changed is news again. `notify` lists the parks not yet marked, and with `--push`, once the comment is written, marks them in the record with `gh pr edit --body-file`, only while the pull request is open and its body still carries the record `notify` read. A mark that fails after the comment exits 2, and the next `notify` names those parks once more. With no batch open nothing is named, and the next batch's first `notify` names what it parks. + With `--push` it writes the same into one comment on the batch pull request: the conflicts, each to be resolved as the [conflict task](../../docs/release/backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows. The comment starts with `` and is one per pull request: a later run edits it in place rather than adding another, and once nothing waits it says so. Only a comment whose author the batch trusts counts as it, so one someone else wrote with the same first line is neither edited nor believed. It mentions `--mention`, an `org/team` or a login, with or without `@`, only while something waits. A failed batch CI is not among what it reports: the run has no trigger on the batch's CI finishing. diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index 35ad92214..3c025517e 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -401,7 +401,7 @@ func found(h *history, v *verdict) bool { // newReport starts the report of a run with what the rules left out and what // the labels overrode. func newReport(b *batch) *report { - r := &report{line: b.line, newer: b.newer, skipped: map[int][]*verdict{}, other: len(b.other), start: b.start} + r := &report{line: b.line, newer: b.newer, skipped: map[int][]*verdict{}, other: len(b.other), start: b.start, carried: b.src.rec} for _, v := range b.verdicts { if v.rule != 0 { r.skipped[v.rule] = append(r.skipped[v.rule], v) @@ -491,7 +491,7 @@ func (r *report) lineRecord(doc reportDoc) lineRecord { } slices.SortFunc(dropped, func(a, b droppedEntry) int { return a.Original - b.Original }) return lineRecord{Version: recordVersion, Line: doc.Line, Built: &built, Report: doc.ID, Closing: r.closing, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: dropped, Stale: slices.Clone(doc.Overlays.Stale), - Parked: slices.Clone(doc.Primary.Parked), Partial: slices.Clone(doc.Overlays.Partial), Changed: slices.Clone(doc.Overlays.Changed)} + Parked: carryAnnounced(doc.Primary.Parked, r.carried), Partial: slices.Clone(doc.Overlays.Partial), Changed: slices.Clone(doc.Overlays.Changed)} } // branchDate is the date a batch branch was opened under. diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index d64bfb5d5..dbb9d6050 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -264,6 +264,9 @@ type report struct { // does only part of what they do on main. parked []*verdict partial []*verdict + // carried is the line's record this run found, whose announcements the + // parks that did not change keep. + carried *lineRecord } // ruleName is how output names what kept an original out: the policy's diff --git a/cmd/backport-batch/fixture_test.go b/cmd/backport-batch/fixture_test.go index 545e4c6c6..ac1307c46 100644 --- a/cmd/backport-batch/fixture_test.go +++ b/cmd/backport-batch/fixture_test.go @@ -458,6 +458,22 @@ func (g *fakeGH) editPull(number int, title, bodyFile string) error { return nil } +func (g *fakeGH) editBody(number int, bodyFile string) error { + if err := g.write(number, "a body edit"); err != nil { + return err + } + if err := g.inject("editBody"); err != nil { + return err + } + if err := g.readBody(bodyFile); err != nil { + return err + } + if pr := g.find(number); pr != nil { + pr.Body = g.bodies[len(g.bodies)-1] + } + return nil +} + func (g *fakeGH) closePull(number int) error { if err := g.write(number, "a close"); err != nil { return err diff --git a/cmd/backport-batch/github.go b/cmd/backport-batch/github.go index 072673a77..7161ef15b 100644 --- a/cmd/backport-batch/github.go +++ b/cmd/backport-batch/github.go @@ -127,6 +127,8 @@ type github interface { // createPull opens the draft and returns its number and URL. createPull(base, head, title, bodyFile string) (int, string, error) editPull(number int, title, bodyFile string) error + // editBody replaces the body of pull request number and nothing else. + editBody(number int, bodyFile string) error closePull(number int) error comments(number int) ([]issueComment, error) createComment(number int, bodyFile string) error @@ -260,6 +262,11 @@ func (ghCLI) editPull(number int, title, bodyFile string) error { return err } +func (ghCLI) editBody(number int, bodyFile string) error { + _, err := capture("gh", "pr", "edit", strconv.Itoa(number), "--body-file", bodyFile) + return err +} + func (ghCLI) closePull(number int) error { _, err := capture("gh", "pr", "close", strconv.Itoa(number)) return err diff --git a/cmd/backport-batch/notify.go b/cmd/backport-batch/notify.go index 34f0d8afb..fe301a2ff 100644 --- a/cmd/backport-batch/notify.go +++ b/cmd/backport-batch/notify.go @@ -88,8 +88,14 @@ func (a *app) notify(args []string) int { } deferred := map[int]deferredEntry{} var partial []int + var parks []parkedEntry if src.open { deferred, partial = src.rec.deferred(), src.rec.Partial + for _, p := range src.rec.Parked { + if !p.Announced { + parks = append(parks, p) + } + } } // An original that waits on others is listed once, after what decides // its wait, where that one is listed. @@ -163,10 +169,74 @@ func (a *app) notify(args []string) int { } text = append(text, b.String()) } + if len(parks) > 0 { + fmt.Fprintf(a.stdout, "%s: %d parked originals on #%d, named once\n", l.branch(), len(parks), pr.Number) + var b strings.Builder + fmt.Fprintf(&b, "These cannot land on this line whole, so the batch parked them; each is named here once. A maintainer ports what applies as a pull request of its own, as [fixes that exist only on a release line](%s#fixes-that-exist-only-on-a-release-line) describes, or drops the original with `backport-batch run --drop`:\n", policyURL) + for _, p := range parks { + fmt.Fprintf(&b, "\n- %s: %s", prLink(p.Original), parkReason(p)) + } + text = append(text, b.String()) + } if !*push { return 0 } - return a.comment(pr, *mention, today, text) + if code := a.comment(pr, *mention, today, text); code != 0 || len(parks) == 0 { + return code + } + if err := a.announced(pr, src.rec, parks); err != nil { + return a.fail(fmt.Errorf("%w; the next notify names them again", err)) + } + return 0 +} + +// announced marks parks as named in the record of the open batch pull +// request, which the comment just named: only while it is open and its body +// still carries the record notify read, so a run that rewrote the body +// meanwhile loses nothing; the next notify names those parks again. +func (a *app) announced(pr *pullRequest, rec *lineRecord, parks []parkedEntry) error { + marked := *rec + marked.Parked = slices.Clone(rec.Parked) + for i, p := range marked.Parked { + if slices.ContainsFunc(parks, func(q parkedEntry) bool { return q.Original == p.Original && parkEvent(q) == parkEvent(p) }) { + marked.Parked[i].Announced = true + } + } + now, err := a.gh.pull(pr.Number) + if err != nil { + return err + } + if st := prState(&now); st != "open" { + return fmt.Errorf("batch pull request #%d is %s now; nothing is written to it", pr.Number, st) + } + lines, err := fencedLines(now.Body, deferredFence) + if err != nil || len(lines) != 1 { + return fmt.Errorf("batch pull request #%d no longer carries the record notify read", pr.Number) + } + read, err := parseRecord(now.Body, notifyLineOf(rec)) + if err != nil || recordBlock(*read) != recordBlock(*rec) { + return fmt.Errorf("batch pull request #%d no longer carries the record notify read", pr.Number) + } + // GitHub keeps the CRLF line ends a body edited on the web has; the + // record is found, and the body written back, with LF ones. + body := strings.ReplaceAll(now.Body, "\r\n", "\n") + block := deferredFence + "\n" + lines[0] + "\n```" + if strings.Count(body, block) != 1 { + return fmt.Errorf("batch pull request #%d carries the record notify read, but not as one block it can mark", pr.Number) + } + body = strings.Replace(body, block, recordBlock(marked), 1) + name, err := writeTemp("backport-batch-body-*.md", body) + if err != nil { + return err + } + defer os.Remove(name) + return a.gh.editBody(pr.Number, name) +} + +// notifyLineOf is the line a record is for, which it was read against. +func notifyLineOf(rec *lineRecord) line { + l, _ := parseLine(rec.Line) + return l } // placement says under which list notify names original n of the record: diff --git a/cmd/backport-batch/notify_test.go b/cmd/backport-batch/notify_test.go index 4bfb650e9..7f870baee 100644 --- a/cmd/backport-batch/notify_test.go +++ b/cmd/backport-batch/notify_test.go @@ -18,6 +18,7 @@ package main import ( "bytes" + "errors" "slices" "strings" "testing" @@ -334,3 +335,162 @@ func TestNotifyWritesNothingOnABatchClosedMeanwhile(t *testing.T) { }) } } + +// parkedBody is an open batch body whose version 3 record parks p. +func parkedBody(p ...parkedEntry) string { + sha := strings.Repeat("a", 40) + rec := lineRecord{Version: recordVersion, Line: "release-1.1", Report: strings.Repeat("b", 64), + Built: &builtFrom{Main: sha, Line: sha, Policy: sha, Batch: sha, At: "2026-10-26T06:00:00Z", Options: runOptions{Mode: modeContinue}}, + Deferred: []deferredEntry{}, Dropped: []droppedEntry{}, Stale: []int{}, Parked: p} + return "batch\n\n" + recordBlock(rec) + "\n" +} + +// notifyAt runs notify at a time of day, with gh faked. +func notifyAt(gh *fakeGH, at time.Time, args ...string) (int, string, string) { + var stdout, stderr bytes.Buffer + a := &app{gh: gh, stdout: &stdout, stderr: &stderr, now: func() time.Time { return at }} + code := a.main(append([]string{"notify", "--line", "release-1.1"}, args...)) + return code, stdout.String(), stderr.String() +} + +// A park is announced once on the open batch: notify names it in its +// comment and marks it announced in the record; a later notify is silent +// about it. A mark that fails to land repeats the announcement once, the +// next day. With no batch open nothing is announced, and the first notify of +// the next batch does it. +func TestNotifyAnnouncesAParkOnce(t *testing.T) { + park := parkedEntry{Original: 5, Merge: strings.Repeat("c", 40), Kind: parkDependency, Needs: []int{3}} + parked := func(gh *fakeGH) []parkedEntry { + t.Helper() + rec, err := parseRecord(gh.prs[0].Body, notifyLine) + if err != nil { + t.Fatal(err) + } + return rec.Parked + } + gh := &fakeGH{prs: []pullRequest{notifyPR("20261026", parkedBody(park))}} + code, stdout, stderr := notifyRun(gh, "--push") + if code != 0 || !strings.Contains(stdout, "release-1.1: 1 parked originals on #90, named once") || len(gh.issueComments) != 1 || !strings.Contains(gh.issueComments[0].Body, "- [#5](https://github.com/cozystack/cozystack/pull/5): builds on [#3](") { + t.Fatalf("notify = %d, want #5 announced:\n%s\n%s\n%v", code, stdout, stderr, gh.issueComments) + } + if p := parked(gh); len(p) != 1 || !p[0].Announced { + t.Errorf("the record parks %+v, want #5 marked announced", p) + } + if code, stdout, _ = notifyRun(gh, "--push"); code != 0 || strings.Contains(stdout, "parked") || strings.Contains(gh.issueComments[0].Body, "[#5]") { + t.Errorf("notify again = %d, want nothing about #5:\n%s\n%s", code, stdout, gh.issueComments[0].Body) + } + + gh = &fakeGH{prs: []pullRequest{notifyPR("20261026", parkedBody(park))}, fail: map[string]error{"editBody": errors.New("HTTP 502")}} + if code, _, _ := notifyRun(gh, "--push"); code != 2 || len(gh.issueComments) != 1 || parked(gh)[0].Announced { + t.Fatalf("notify with a failing mark = %d, want 2 after the comment, the record unmarked", code) + } + code, _, stderr = notifyAt(gh, time.Date(2026, 10, 27, 0, 5, 0, 0, time.UTC), "--push") + if code != 0 || !strings.Contains(gh.issueComments[0].Body, "[#5](") || !parked(gh)[0].Announced { + t.Errorf("notify the next day = %d, want #5 named once more and marked:\n%s\n%s", code, stderr, gh.issueComments[0].Body) + } + + // Closed, or rewritten by a run, between the comment and the mark: the + // mark is not written, and the next notify names #5 again. + gh = &fakeGH{prs: []pullRequest{notifyPR("20261026", parkedBody(park))}} + gh.after = map[string]func(){"createComment": func() { gh.prs[0].State = "closed" }} + if code, _, _ := notifyRun(gh, "--push"); code != 2 || len(gh.closedWrites) != 0 || parked(gh)[0].Announced { + t.Errorf("notify on a batch closed meanwhile = %d, writes %q; want 2 and no mark", code, gh.closedWrites) + } + gh = &fakeGH{prs: []pullRequest{notifyPR("20261026", parkedBody(park))}} + rewritten := parkedBody(park, parkedEntry{Original: 6, Merge: strings.Repeat("d", 40), Kind: parkPartial, Files: []string{"x"}}) + gh.after = map[string]func(){"createComment": func() { gh.prs[0].Body = rewritten }} + if code, _, _ := notifyRun(gh, "--push"); code != 2 || gh.prs[0].Body != rewritten || len(gh.bodies) != 0 { + t.Errorf("notify on a body rewritten meanwhile = %d; want 2 and the new body left as it is", code) + } + + // A body someone edited on the web comes back with CRLF line ends; the + // mark lands all the same. + gh = &fakeGH{prs: []pullRequest{notifyPR("20261026", strings.ReplaceAll(parkedBody(park), "\n", "\r\n"))}} + if code, _, stderr := notifyRun(gh, "--push"); code != 0 || !parked(gh)[0].Announced { + t.Errorf("notify on a CRLF body = %d, want #5 marked:\n%s", code, stderr) + } + + closed := notifyPR("20261026", parkedBody(park)) + at := "2026-10-26T12:00:00Z" + closed.State, closed.MergedAt = "closed", &at + gh = &fakeGH{prs: []pullRequest{closed}} + if code, stdout, _ := notifyRun(gh, "--push"); code != 0 || stdout != "" || len(gh.issueComments) != 0 || len(gh.bodies) != 0 { + t.Errorf("notify with no batch open = %d, %q, %d comments, %d edits; want nothing", code, stdout, len(gh.issueComments), len(gh.bodies)) + } + next := notifyPR("20261027", parkedBody(park)) + next.Number = 91 + gh.prs = append(gh.prs, next) + if code, _, _ := notifyRun(gh, "--push"); code != 0 || len(gh.issueComments) != 1 || !strings.Contains(gh.issueComments[0].Body, "[#5](") { + t.Errorf("the next batch's notify = %d, want #5 announced on it: %v", code, gh.issueComments) + } +} + +// A park carries its announcement into the next record only while it is +// the same park: the same merge, kind, prerequisites, commits and files. +func TestParkEventIdentity(t *testing.T) { + a, b := strings.Repeat("a", 40), strings.Repeat("b", 40) + was := parkedEntry{Original: 5, Merge: a, Kind: parkDependency, Commits: []string{a}, Announced: true} + carrier := &lineRecord{Parked: []parkedEntry{was}} + for name, tc := range map[string]struct { + now parkedEntry + announced bool + }{ + "the same park": {parkedEntry{Original: 5, Merge: a, Kind: parkDependency, Commits: []string{a}}, true}, + "another unnumbered commit": {parkedEntry{Original: 5, Merge: a, Kind: parkDependency, Commits: []string{b}}, false}, + "another kind": {parkedEntry{Original: 5, Merge: a, Kind: parkPartial, Files: []string{"x"}}, false}, + "another prerequisite": {parkedEntry{Original: 5, Merge: a, Kind: parkDependency, Commits: []string{a}, Needs: []int{3}}, false}, + "another merge": {parkedEntry{Original: 5, Merge: b, Kind: parkDependency, Commits: []string{a}}, false}, + "another original": {parkedEntry{Original: 6, Merge: a, Kind: parkDependency, Commits: []string{a}}, false}, + } { + got := carryAnnounced([]parkedEntry{tc.now}, carrier) + if got[0].Announced != tc.announced { + t.Errorf("%s: announced = %v, want %v", name, got[0].Announced, tc.announced) + } + } + if got := carryAnnounced([]parkedEntry{was}, nil); got[0].Announced { + t.Error("a park with no carrier came out announced") + } + unannounced := was + unannounced.Announced = false + if got := carryAnnounced([]parkedEntry{was}, &lineRecord{Parked: []parkedEntry{unannounced}}); got[0].Announced { + t.Error("a park the carrier holds unannounced came out announced") + } +} + +// A park keeps its announcement through a continuation and into the next +// batch, so notify does not name it again. +func TestRolloverKeepsTheAnnouncement(t *testing.T) { + f := featWorld(t) + gh := &fakeGH{} + openBatch(t, f, gh) + if code, _, stderr := f.exec(gh, "notify", "--line", "release-1.1", "--push"); code != 0 || !strings.Contains(notifyComment(gh), "[#5](") { + t.Fatalf("notify = %d, want #5 announced:\n%s", code, stderr) + } + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push"); code != 0 || !strings.Contains(stdout, `"needs":[3],"announced":true`) { + t.Fatalf("run --continue = %d, want #5's park still announced:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + mergeBatch(f, gh) + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + gh.nextPR = 91 + f.now = f.now.Add(24 * time.Hour) + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push"); code != 0 || !strings.Contains(stdout, `"needs":[3],"announced":true`) { + t.Fatalf("the next batch's run = %d, want #5's park announced from the merged batch:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + gh.issueComments = nil + if code, stdout, _ := f.exec(gh, "notify", "--line", "release-1.1", "--push"); code != 0 || strings.Contains(stdout, "parked") || strings.Contains(notifyComment(gh), "[#5](") { + t.Errorf("notify on the next batch = %d, want nothing about #5:\n%s\n%s", code, stdout, notifyComment(gh)) + } +} + +// notifyComment is the notify comment on the fake, or "". +func notifyComment(gh *fakeGH) string { + for _, c := range slices.Backward(gh.issueComments) { + if strings.HasPrefix(c.Body, notifyMarker) { + return c.Body + } + } + return "" +} diff --git a/cmd/backport-batch/record.go b/cmd/backport-batch/record.go index 0077b5302..37dd7946a 100644 --- a/cmd/backport-batch/record.go +++ b/cmd/backport-batch/record.go @@ -88,6 +88,39 @@ type parkedEntry struct { Announced bool `json:"announced,omitempty"` } +// parkEvent is what makes a park the same park from one record to the +// next: its merge, kind, prerequisites, commits and files. A park whose +// event changed is news, and notify names it again. +func parkEvent(p parkedEntry) string { + data, _ := json.Marshal(struct { + Merge string `json:"merge"` + Kind string `json:"kind"` + Needs []int `json:"needs"` + Commits []string `json:"commits"` + Files []string `json:"files"` + }{p.Merge, p.Kind, p.Needs, p.Commits, p.Files}) + return string(data) +} + +// carryAnnounced marks each park announced that the carrier's record holds +// announced as the same event, a closed carrier's too, so a new batch does +// not name again what an earlier one named. +func carryAnnounced(parked []parkedEntry, carrier *lineRecord) []parkedEntry { + out := slices.Clone(parked) + for i := range out { + out[i].Announced = false + } + if carrier == nil { + return out + } + for i := range out { + out[i].Announced = slices.ContainsFunc(carrier.Parked, func(c parkedEntry) bool { + return c.Announced && c.Original == out[i].Original && parkEvent(c) == parkEvent(out[i]) + }) + } + return out +} + // validate holds a parked entry to what a run writes. Its errors read after // "the block" or "report ". func (p parkedEntry) validate() error { diff --git a/docs/release/backports.md b/docs/release/backports.md index 71fa467eb..ad7424c40 100644 --- a/docs/release/backports.md +++ b/docs/release/backports.md @@ -117,7 +117,7 @@ That picks it onto the batch in the batch's shape, after what the batch carries, ### The scheduled run -[`backport-batch.yaml`](../../.github/workflows/backport-batch.yaml) runs [`hack/backport-batch-scheduled.sh`](../../hack/backport-batch-scheduled.sh) on Monday and Thursday. For each line the batch runs for, newest first, it continues the open batch pull request or opens one, runs `check`, and then `backport-batch notify`, which reports what waits on the batch. A batch open for three days or more, so one the previous run already saw, gets a comment on its pull request saying so and a line in the run's summary. The comment is one per pull request, updated in place on every run. It also lists the deferred conflicts, which a maintainer resolves as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows. A failed batch CI waits for a maintainer too, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the comment does not list it, and it shows on the batch pull request's checks. `hack/backport-batch-scheduled.sh --dry-run` does the same in the current worktree and changes nothing on GitHub. Its `check` sees the line without the batch the dry run just built, so it can exit 1 where the real run's check passes. +[`backport-batch.yaml`](../../.github/workflows/backport-batch.yaml) runs [`hack/backport-batch-scheduled.sh`](../../hack/backport-batch-scheduled.sh) on Monday and Thursday. For each line the batch runs for, newest first, it continues the open batch pull request or opens one, runs `check`, and then `backport-batch notify`, which reports what waits on the batch. A batch open for three days or more, so one the previous run already saw, gets a comment on its pull request saying so and a line in the run's summary. The comment is one per pull request, updated in place on every run. It also lists the deferred conflicts, which a maintainer resolves as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows, and names each parked original once: a park named on one batch is not named again on the next while it stays the same park, and with no batch open nothing is named until the next batch opens. A failed batch CI waits for a maintainer too, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the comment does not list it, and it shows on the batch pull request's checks. `hack/backport-batch-scheduled.sh --dry-run` does the same in the current worktree and changes nothing on GitHub. Its `check` sees the line without the batch the dry run just built, so it can exit 1 where the real run's check passes. The batch runs as the cozystack-ci GitHub App, which the release workflows already use, and needs no secret of its own. The job mints a token from `COZYSTACK_CI_APP_ID` and `COZYSTACK_CI_PRIVATE_KEY` for this repository alone, with Contents and Pull requests write and nothing more, and fails when either secret is missing. Its pushes start CI on the batch pull request, which a push with the job's own `GITHUB_TOKEN` would not. Its commits carry the App's bot, `cozystack-ci[bot]`, whose batch pull request and comment `backport-batch` trusts: `BACKPORT_BATCH_BOT`, which the job sets from the token, names it, and a run without the variable trusts `cozystack-ci[bot]` too. The App's installation has to grant this repository Contents and Pull requests read and write, which an organization admin sees among the organization's installed GitHub Apps. It has no Workflows permission, so a change to `.github/workflows/` waits for a maintainer, as [changes to workflows](#changes-to-workflows) describes. From 47d9ca01f3f936f7ad02571b7a7dbf8fc9e3ec6e Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Tue, 10 Nov 2026 05:37:15 +0700 Subject: [PATCH 409/518] feat(observability-proxy): publish the proxy on its own host Clients outside the cluster, such as kubectl, scripts and third-party tools, need to call the proxy with their Kubernetes bearer token. The kube-apiserver service proxy cannot carry it: kube-apiserver drops the Authorization header once it has authenticated the request. The console route sits behind the dashboard gatekeeper, which with OIDC enabled answers an API client that has no session cookie with 401, and exists only where the dashboard itself is published. A route straight to the proxy Service is the path that keeps the token. It stays off until "observability-proxy" is listed in publishing.exposedServices, and follows the dashboard for TLS and for the Ingress versus Gateway API switch. The edge must not end a request the proxy still serves. The tenant ingress-nginx reads upstream with a 10s timeout, while a log tail may stay silent for the proxy's 5m idle limit, so the Ingress waits longer than that. Cilium leaves Envoy's 15s route timeout in place on an HTTPRoute that sets none, which would cut every tail, so the route disables it: the proxy bounds each request on its own. Assisted-by: LLM Signed-off-by: Maxim Belyy --- .../templates/httproute.yaml | 28 +++ .../templates/ingress.yaml | 52 +++++ .../tests/publishing_test.yaml | 196 ++++++++++++++++++ 3 files changed, 276 insertions(+) create mode 100644 packages/system/observability-proxy/templates/httproute.yaml create mode 100644 packages/system/observability-proxy/templates/ingress.yaml create mode 100644 packages/system/observability-proxy/tests/publishing_test.yaml diff --git a/packages/system/observability-proxy/templates/httproute.yaml b/packages/system/observability-proxy/templates/httproute.yaml new file mode 100644 index 000000000..5d4931031 --- /dev/null +++ b/packages/system/observability-proxy/templates/httproute.yaml @@ -0,0 +1,28 @@ +{{- $cluster := .Values._cluster | default dict }} +{{- $host := index $cluster "root-host" }} +{{- $exposeServices := splitList "," ((index $cluster "expose-services") | default "") }} +{{- $exposeIngress := (index $cluster "expose-ingress") | default "tenant-root" }} +{{- $gatewayEnabled := (index $cluster "gateway-enabled") | default "false" }} +{{- if and (has "observability-proxy" $exposeServices) (eq $gatewayEnabled "true") }} +apiVersion: gateway.networking.k8s.io/v1 +kind: HTTPRoute +metadata: + name: observability-proxy +spec: + parentRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: cozystack + namespace: {{ $exposeIngress }} + hostnames: + - {{ printf "observability-proxy.%s" $host | quote }} + rules: + - backendRefs: + - name: observability-proxy + port: 80 + {{- /* Without a request timeout Cilium leaves Envoy's 15s route default + in place, which would cut every tail. The proxy bounds each + request itself, so the route sets none. */}} + timeouts: + request: 0s +{{- end }} diff --git a/packages/system/observability-proxy/templates/ingress.yaml b/packages/system/observability-proxy/templates/ingress.yaml new file mode 100644 index 000000000..59e79a32b --- /dev/null +++ b/packages/system/observability-proxy/templates/ingress.yaml @@ -0,0 +1,52 @@ +{{- $cluster := .Values._cluster | default dict }} +{{- $solver := (index $cluster "solver") | default "http01" }} +{{- $clusterIssuer := (index $cluster "issuer-name") | default "letsencrypt-prod" }} +{{- $host := index $cluster "root-host" }} +{{- $exposeServices := splitList "," ((index $cluster "expose-services") | default "") }} +{{- $exposeIngress := (index $cluster "expose-ingress") | default "tenant-root" }} +{{- $gatewayEnabled := (index $cluster "gateway-enabled") | default "false" }} +{{- $wildcardSecret := (index $cluster "wildcard-secret-name") | default "" }} +{{- if and (has "observability-proxy" $exposeServices) (ne $gatewayEnabled "true") }} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: observability-proxy + annotations: + {{- if not $wildcardSecret }} + cert-manager.io/cluster-issuer: {{ $clusterIssuer }} + {{- if eq $solver "http01" }} + acme.cert-manager.io/http01-ingress-ingressclassname: {{ $exposeIngress }} + {{- end }} + {{- end }} + {{- /* The proxy refuses a body above 1 MiB itself; nginx's 1m is the + same bound, so an oversized body stops at the edge. */}} + nginx.ingress.kubernetes.io/proxy-body-size: 1m + {{- /* A tail flushes every line; buffered, lines would reach the client + in batches. */}} + nginx.ingress.kubernetes.io/proxy-buffering: "off" + {{- /* The controller default is 10s. The longest silence the proxy + allows is a tail with no new line for --tail-idle-timeout (5m), + after which it ends the response itself. */}} + nginx.ingress.kubernetes.io/proxy-read-timeout: "360" +spec: + ingressClassName: {{ $exposeIngress }} + rules: + - host: observability-proxy.{{ $host }} + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: observability-proxy + port: + name: http + tls: + - hosts: + - observability-proxy.{{ $host }} + {{- /* Wildcard mode: ingress-nginx serves its default certificate, and + the tls section stays to keep the HTTP-to-HTTPS redirect. */}} + {{- if not $wildcardSecret }} + secretName: observability-proxy-tls + {{- end }} +{{- end }} diff --git a/packages/system/observability-proxy/tests/publishing_test.yaml b/packages/system/observability-proxy/tests/publishing_test.yaml new file mode 100644 index 000000000..0efc3ca82 --- /dev/null +++ b/packages/system/observability-proxy/tests/publishing_test.yaml @@ -0,0 +1,196 @@ +suite: observability-proxy public endpoint +# Clients outside the cluster reach the proxy on its own host with a +# Kubernetes bearer token, so the route must pass Authorization and the +# /apis/... path through untouched, and must not end a request before the +# proxy does: a non-tail request may take up to --request-timeout (1m), and +# a tail may stay silent up to --tail-idle-timeout (5m). Whole objects are +# compared rather than single keys, so an annotation or a field that should +# not be there fails the test too. +templates: + - templates/ingress.yaml + - templates/httproute.yaml + +release: + name: observability-proxy + namespace: cozy-observability-proxy + +tests: + - it: publishes nothing without the cluster values + asserts: + - template: templates/ingress.yaml + hasDocuments: + count: 0 + - template: templates/httproute.yaml + hasDocuments: + count: 0 + + - it: publishes nothing while observability-proxy is not in expose-services + set: + _cluster: + root-host: example.org + expose-services: "api,dashboard" + gateway-enabled: "false" + asserts: + - template: templates/ingress.yaml + hasDocuments: + count: 0 + - template: templates/httproute.yaml + hasDocuments: + count: 0 + + - it: publishes nothing in Gateway mode while observability-proxy is not in expose-services + set: + _cluster: + root-host: example.org + expose-services: "api,dashboard" + gateway-enabled: "true" + asserts: + - template: templates/ingress.yaml + hasDocuments: + count: 0 + - template: templates/httproute.yaml + hasDocuments: + count: 0 + + - it: renders an Ingress with a per-host certificate when gateway is off + set: + _cluster: + root-host: example.org + expose-services: "dashboard,observability-proxy" + expose-ingress: tenant-public + gateway-enabled: "false" + asserts: + - template: templates/httproute.yaml + hasDocuments: + count: 0 + - template: templates/ingress.yaml + isKind: + of: Ingress + - template: templates/ingress.yaml + equal: + path: metadata.name + value: observability-proxy + - template: templates/ingress.yaml + equal: + path: metadata.annotations + value: + cert-manager.io/cluster-issuer: letsencrypt-prod + acme.cert-manager.io/http01-ingress-ingressclassname: tenant-public + nginx.ingress.kubernetes.io/proxy-body-size: 1m + nginx.ingress.kubernetes.io/proxy-buffering: "off" + nginx.ingress.kubernetes.io/proxy-read-timeout: "360" + - template: templates/ingress.yaml + equal: + path: spec + value: + ingressClassName: tenant-public + rules: + - host: observability-proxy.example.org + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: observability-proxy + port: + name: http + tls: + - hosts: + - observability-proxy.example.org + secretName: observability-proxy-tls + + - it: leaves the http01 ingress class out under the dns01 solver + set: + _cluster: + root-host: example.org + expose-services: "observability-proxy" + gateway-enabled: "false" + solver: dns01 + issuer-name: letsencrypt-stage + asserts: + - template: templates/ingress.yaml + equal: + path: metadata.annotations + value: + cert-manager.io/cluster-issuer: letsencrypt-stage + nginx.ingress.kubernetes.io/proxy-body-size: 1m + nginx.ingress.kubernetes.io/proxy-buffering: "off" + nginx.ingress.kubernetes.io/proxy-read-timeout: "360" + - template: templates/ingress.yaml + equal: + path: spec.ingressClassName + value: tenant-root + + - it: relies on the wildcard certificate and keeps the TLS section for the redirect + set: + _cluster: + root-host: example.org + expose-services: "observability-proxy" + gateway-enabled: "false" + wildcard-secret-name: wildcard-tls + asserts: + - template: templates/ingress.yaml + equal: + path: metadata.annotations + value: + nginx.ingress.kubernetes.io/proxy-body-size: 1m + nginx.ingress.kubernetes.io/proxy-buffering: "off" + nginx.ingress.kubernetes.io/proxy-read-timeout: "360" + - template: templates/ingress.yaml + equal: + path: spec.tls + value: + - hosts: + - observability-proxy.example.org + + - it: renders an HTTPRoute without a route timeout when gateway is on + set: + _cluster: + root-host: example.org + expose-services: "observability-proxy" + expose-ingress: tenant-public + gateway-enabled: "true" + asserts: + - template: templates/ingress.yaml + hasDocuments: + count: 0 + - template: templates/httproute.yaml + isKind: + of: HTTPRoute + - template: templates/httproute.yaml + isAPIVersion: + of: gateway.networking.k8s.io/v1 + - template: templates/httproute.yaml + equal: + path: metadata.name + value: observability-proxy + - template: templates/httproute.yaml + equal: + path: spec + value: + parentRefs: + - group: gateway.networking.k8s.io + kind: Gateway + name: cozystack + namespace: tenant-public + hostnames: + - observability-proxy.example.org + rules: + - backendRefs: + - name: observability-proxy + port: 80 + timeouts: + request: 0s + + - it: attaches to the tenant-root Gateway when expose-ingress is unset + set: + _cluster: + root-host: example.org + expose-services: "observability-proxy" + gateway-enabled: "true" + asserts: + - template: templates/httproute.yaml + equal: + path: spec.parentRefs[?(@.name == "cozystack")].namespace + value: tenant-root From 77acd23bdda7a3f514a2b090aa2a5d5096c38a86 Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Tue, 10 Nov 2026 05:38:18 +0700 Subject: [PATCH 410/518] feat(platform): let the observability proxy attach to the tenant Gateway With the Gateway API enabled, the publishing Gateway admits routes only from the publishing tenant and the namespaces in gateway.attachedNamespaces. The observability proxy publishes its HTTPRoute from its own namespace, so without an entry the route renders and is never served. Clusters that override gateway.attachedNamespaces replace the whole list and have to add the namespace themselves. Assisted-by: LLM Signed-off-by: Maxim Belyy --- .../tests/observability_proxy_wiring_test.yaml | 10 ++++++++++ packages/core/platform/values.yaml | 1 + 2 files changed, 11 insertions(+) diff --git a/packages/core/platform/tests/observability_proxy_wiring_test.yaml b/packages/core/platform/tests/observability_proxy_wiring_test.yaml index 9e6ec583d..84777ca7f 100644 --- a/packages/core/platform/tests/observability_proxy_wiring_test.yaml +++ b/packages/core/platform/tests/observability_proxy_wiring_test.yaml @@ -5,6 +5,7 @@ suite: observability-proxy package wiring templates: - templates/sources.yaml - templates/bundles/system.yaml + - templates/apps.yaml release: name: cozystack namespace: cozy-system @@ -67,3 +68,12 @@ tests: kind: Package name: cozystack.observability-proxy any: true + + # The proxy's HTTPRoute lives in its own namespace, and the tenant Gateway + # admits routes only from the namespaces listed here. + - it: lets the proxy namespace attach a route to the publishing Gateway + template: templates/apps.yaml + asserts: + - matchRegex: + path: stringData["values.yaml"] + pattern: 'gateway-attached-namespaces: "([^"]*,)?cozy-observability-proxy(,[^"]*)?"' diff --git a/packages/core/platform/values.yaml b/packages/core/platform/values.yaml index f2ce252ed..5ca66ac17 100644 --- a/packages/core/platform/values.yaml +++ b/packages/core/platform/values.yaml @@ -668,6 +668,7 @@ gateway: - cozy-kubevirt-cdi - cozy-monitoring - cozy-linstor-gui + - cozy-observability-proxy - default # Platform monitoring configuration. monitoring: From 773a3a96d3d7a378914c72823857580f3920d8b7 Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Tue, 10 Nov 2026 05:42:46 +0700 Subject: [PATCH 411/518] feat(tenant): allow tenant egress to the observability proxy The observability proxy's Ingress is served by the ingress controller in the publishing tenant, and tenant egress is denied except for the allow-to-* policies. Without one for the proxy namespace, every request through the Ingress and every HTTP-01 challenge for its certificate times out at the controller. The policy matches the whole namespace because cert-manager runs the HTTP-01 solver pods there, next to the proxy. Assisted-by: LLM Signed-off-by: Maxim Belyy --- .../apps/tenant/templates/networkpolicy.yaml | 14 ++++ ...etworkpolicy_observability_proxy_test.yaml | 64 +++++++++++++++++++ 2 files changed, 78 insertions(+) create mode 100644 packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml diff --git a/packages/apps/tenant/templates/networkpolicy.yaml b/packages/apps/tenant/templates/networkpolicy.yaml index 5285b495a..bc6904e0a 100644 --- a/packages/apps/tenant/templates/networkpolicy.yaml +++ b/packages/apps/tenant/templates/networkpolicy.yaml @@ -259,6 +259,20 @@ spec: - ports: - port: "8443" protocol: TCP +--- +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + # The whole namespace, as for linstor-gui: the ingress controller also + # has to reach the ACME HTTP-01 solver pods cert-manager starts there. + name: allow-to-observability-proxy + namespace: {{ include "tenant.name" . }} +spec: + endpointSelector: {} + egress: + - toEndpoints: + - matchLabels: + "k8s:io.kubernetes.pod.namespace": cozy-observability-proxy {{- if .Values.monitoring }} --- apiVersion: cilium.io/v2 diff --git a/packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml b/packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml new file mode 100644 index 000000000..3f3e027fd --- /dev/null +++ b/packages/apps/tenant/tests/networkpolicy_observability_proxy_test.yaml @@ -0,0 +1,64 @@ +suite: tenant egress to the observability proxy +# The observability-proxy Ingress in cozy-observability-proxy is served by +# the ingress controller running in tenant-root. Tenant egress is +# default-deny plus explicit allow-to-* policies, so without this one the +# controller reaches neither the proxy nor the ACME HTTP-01 solver pod that +# cert-manager starts in the same namespace; hence the whole namespace, as +# allow-to-linstor-gui does. +templates: + - templates/networkpolicy.yaml +tests: + - it: lets every pod of the root tenant reach the proxy namespace + release: + name: tenant-root + namespace: tenant-root + asserts: + - documentSelector: + path: metadata.name + value: allow-to-observability-proxy + isKind: + of: CiliumNetworkPolicy + - documentSelector: + path: metadata.name + value: allow-to-observability-proxy + equal: + path: metadata.namespace + value: tenant-root + - documentSelector: + path: metadata.name + value: allow-to-observability-proxy + equal: + path: spec + value: + endpointSelector: {} + egress: + - toEndpoints: + - matchLabels: + "k8s:io.kubernetes.pod.namespace": cozy-observability-proxy + + - it: renders in a nested tenant with every module off + release: + name: tenant-bar + namespace: tenant-foo + set: + monitoring: false + ingress: false + etcd: false + asserts: + - documentSelector: + path: metadata.name + value: allow-to-observability-proxy + equal: + path: metadata.namespace + value: tenant-foo-bar + + - it: renders nothing where the tenant Cilium policies are off + release: + name: tenant-root + namespace: tenant-root + set: + _cluster: + bundle-name: isp-hosted + asserts: + - hasDocuments: + count: 0 From d21d9de641db88f2e9fec6fbbe45d1d8500dc5a4 Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Tue, 10 Nov 2026 05:43:31 +0700 Subject: [PATCH 412/518] fix(observability-proxy): end silent tails before the Gateway does Behind a Cilium Gateway, Envoy resets a stream that carries no data for 300s, the default stream idle timeout, and the proxy's own tail idle limit was the same 5m. A tail that went quiet could then be cut by Envoy rather than ended by the proxy, which the client sees as a reset stream instead of a finished response. The HTTPRoute cannot raise Envoy's limit, so the proxy ends a silent tail a little earlier. Assisted-by: LLM Signed-off-by: Maxim Belyy --- .../system/observability-proxy/templates/deployment.yaml | 4 ++++ .../system/observability-proxy/templates/httproute.yaml | 4 +++- .../system/observability-proxy/templates/ingress.yaml | 5 +++-- .../observability-proxy/tests/deployment_test.yaml | 9 +++++++++ .../observability-proxy/tests/publishing_test.yaml | 2 +- 5 files changed, 20 insertions(+), 4 deletions(-) diff --git a/packages/system/observability-proxy/templates/deployment.yaml b/packages/system/observability-proxy/templates/deployment.yaml index 581c15fde..dd95dfbb6 100644 --- a/packages/system/observability-proxy/templates/deployment.yaml +++ b/packages/system/observability-proxy/templates/deployment.yaml @@ -38,6 +38,10 @@ spec: - --cluster-domain={{ $clusterDomain }} - --root-tenant={{ .Values.observabilityProxy.rootTenant }} - --auth-cache-ttl={{ .Values.observabilityProxy.authCacheTTL }} + {{- /* Below the 300s stream idle timeout of a Cilium Gateway, so a + published tail that stops sending lines is ended by the proxy, + not reset by Envoy. */}} + - --tail-idle-timeout=4m30s ports: - name: http containerPort: 8080 diff --git a/packages/system/observability-proxy/templates/httproute.yaml b/packages/system/observability-proxy/templates/httproute.yaml index 5d4931031..1ddd2f720 100644 --- a/packages/system/observability-proxy/templates/httproute.yaml +++ b/packages/system/observability-proxy/templates/httproute.yaml @@ -22,7 +22,9 @@ spec: port: 80 {{- /* Without a request timeout Cilium leaves Envoy's 15s route default in place, which would cut every tail. The proxy bounds each - request itself, so the route sets none. */}} + request itself, so the route disables it. Envoy's 300s stream + idle timeout still applies; the proxy's tail idle limit stays + below it (see deployment.yaml). */}} timeouts: request: 0s {{- end }} diff --git a/packages/system/observability-proxy/templates/ingress.yaml b/packages/system/observability-proxy/templates/ingress.yaml index 59e79a32b..7c6b1a029 100644 --- a/packages/system/observability-proxy/templates/ingress.yaml +++ b/packages/system/observability-proxy/templates/ingress.yaml @@ -25,8 +25,9 @@ metadata: in batches. */}} nginx.ingress.kubernetes.io/proxy-buffering: "off" {{- /* The controller default is 10s. The longest silence the proxy - allows is a tail with no new line for --tail-idle-timeout (5m), - after which it ends the response itself. */}} + allows is a tail with no new line for --tail-idle-timeout (4m30s), + after which it ends the response itself; before the first byte + the token and access reviews may add up to --request-timeout (1m). */}} nginx.ingress.kubernetes.io/proxy-read-timeout: "360" spec: ingressClassName: {{ $exposeIngress }} diff --git a/packages/system/observability-proxy/tests/deployment_test.yaml b/packages/system/observability-proxy/tests/deployment_test.yaml index ddf9d34b8..978f9e968 100644 --- a/packages/system/observability-proxy/tests/deployment_test.yaml +++ b/packages/system/observability-proxy/tests/deployment_test.yaml @@ -40,6 +40,15 @@ tests: path: spec.template.spec.containers[0].args content: --auth-cache-ttl=10s + # Envoy behind a Cilium Gateway resets a stream after 300s without data + # (envoy.streamIdleTimeoutDurationSeconds), so a silent tail has to be + # ended by the proxy first, or the client sees a reset instead of an end. + - it: ends a silent tail before the Gateway stream idle timeout + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --tail-idle-timeout=4m30s + - it: falls back to cozy.local when the platform passes no cluster domain asserts: - contains: diff --git a/packages/system/observability-proxy/tests/publishing_test.yaml b/packages/system/observability-proxy/tests/publishing_test.yaml index 0efc3ca82..676c38e08 100644 --- a/packages/system/observability-proxy/tests/publishing_test.yaml +++ b/packages/system/observability-proxy/tests/publishing_test.yaml @@ -3,7 +3,7 @@ suite: observability-proxy public endpoint # Kubernetes bearer token, so the route must pass Authorization and the # /apis/... path through untouched, and must not end a request before the # proxy does: a non-tail request may take up to --request-timeout (1m), and -# a tail may stay silent up to --tail-idle-timeout (5m). Whole objects are +# a tail may stay silent up to --tail-idle-timeout (4m30s). Whole objects are # compared rather than single keys, so an annotation or a field that should # not be there fails the test too. templates: From 19e91d3957e0119d78f6c9e536110b3bdb3546db Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Tue, 10 Nov 2026 19:43:21 +0700 Subject: [PATCH 413/518] feat(observability-proxy): include platform exporter series in the namespace root A platform exporter such as the ClickHouse operator's runs in its own cozy-* namespace and reports on objects in tenant namespaces. vmagent labels its series with the exporter's namespace and moves the namespace the exporter reported to exported_namespace, so the namespace root, scoped to namespace="" alone, never returned them. The scope gains a second extra_filters[] value matching exported_namespace="" only on series scraped from a namespace with the platform prefix. A bare exported_namespace condition would let any tenant pod that writes a namespace label inject series into another tenant's namespace root. vmselect ORs the two values, and the label APIs repeat both in match[], where the match[] and extra_filters[] cross join still selects exactly their union. An empty prefix keeps the previous single-value scope. Assisted-by: LLM Signed-off-by: Maxim Belyy --- internal/observabilityproxy/metricsparams.go | 42 +++- .../observabilityproxy/metricsparams_test.go | 24 +- .../observabilityproxy/metricsscope_test.go | 237 ++++++++++++++++++ internal/observabilityproxy/server.go | 2 +- .../observabilityproxy/server_metrics_test.go | 5 +- internal/observabilityproxy/server_test.go | 1 + 6 files changed, 292 insertions(+), 19 deletions(-) create mode 100644 internal/observabilityproxy/metricsscope_test.go diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index 8beb347d0..c300cdc7a 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -22,6 +22,7 @@ import ( "net/http" "net/url" "regexp" + "slices" "strconv" "strings" "time" @@ -46,6 +47,9 @@ type MetricsConfig struct { LongTermStore string ShortTermRetention time.Duration Limits MetricsLimits + // PlatformNamespacePrefix names the namespaces whose exporters may + // attribute a series to another namespace; empty trusts none. + PlatformNamespacePrefix string // UpstreamURL maps a store name to its Prometheus API base URL, without // a trailing slash; nil means the in-cluster vmselect Service. UpstreamURL func(store string) string @@ -85,15 +89,34 @@ func namespaceSelector(ns string) string { return "{namespace=" + strconv.Quote(ns) + "}" } +// namespaceScope selects the series of namespace ns. Besides its own, it +// takes the series an exporter in a platform namespace attributes to ns: +// vmagent sets namespace to the scraped target's and, without honorLabels, +// moves the exporter's own namespace label to exported_namespace. Only +// platform namespaces are trusted with that label, since any tenant pod can +// write it; tenants cannot run pods or scrape objects in them. +func (c MetricsConfig) namespaceScope(ns string) []string { + scope := []string{namespaceSelector(ns)} + if c.PlatformNamespacePrefix != "" { + // MetricsQL anchors =~ at both ends. + platform := strconv.Quote(regexp.QuoteMeta(c.PlatformNamespacePrefix) + ".*") + scope = append(scope, "{namespace=~"+platform+",exported_namespace="+strconv.Quote(ns)+"}") + } + return scope +} + // BuildMetricsParams rebuilds the upstream parameters from an allowlist and -// adds the namespace scope as the only extra_filters[] value. vmselect ORs -// repeated extra_filters, so a client value would widen the scope; it ANDs -// the single one into every selector of the query and every match[], which -// is what lets the query pass through unparsed. -func BuildMetricsParams(endpoint string, in url.Values, namespace string, now time.Time, lim MetricsLimits) (metricsRequest, *HTTPError) { +// sends scope as the only extra_filters[] values. vmselect ORs repeated +// extra_filters, so a client value would widen the scope; it ANDs each into +// every selector of the query and every match[], which is what lets the +// query pass through unparsed. +func BuildMetricsParams(endpoint string, in url.Values, scope []string, now time.Time, lim MetricsLimits) (metricsRequest, *HTTPError) { req := metricsRequest{path: endpoint, params: url.Values{}} + if len(scope) == 0 { + return req, errorf(http.StatusInternalServerError, "no metrics scope") + } out := req.params - out.Set("extra_filters[]", namespaceSelector(namespace)) + out["extra_filters[]"] = slices.Clone(scope) timeout := lim.MaxTimeout if v := in.Get("timeout"); v != "" { d, herr := parseMetricsDuration("timeout", v) @@ -192,8 +215,11 @@ func BuildMetricsParams(endpoint string, in url.Values, namespace string, now ti } // vmselect started with -search.ignoreExtraFiltersAtLabelsAPI drops // extra_filters[] on the label APIs unless a match[] is present - // (getCommonParamsInternal), so the scope is repeated here. - out.Set("match[]", namespaceSelector(namespace)) + // (getCommonParamsInternal), so the scope is repeated here. Where both + // are sent, vmselect ANDs every match[] with every extra_filters[]; + // a cross pair is narrower than one of its halves, so the union is + // still the scope. + out["match[]"] = slices.Clone(scope) } return req, nil } diff --git a/internal/observabilityproxy/metricsparams_test.go b/internal/observabilityproxy/metricsparams_test.go index a4a435567..58551a4f7 100644 --- a/internal/observabilityproxy/metricsparams_test.go +++ b/internal/observabilityproxy/metricsparams_test.go @@ -19,6 +19,7 @@ package observabilityproxy import ( "net/http" "net/url" + "slices" "strconv" "strings" "testing" @@ -29,13 +30,20 @@ func testMetricsLimits() MetricsLimits { return MetricsLimits{MaxRange: 30 * 24 * time.Hour, MaxPoints: 11000, MaxTimeout: 30 * time.Second} } -const wantMetricsScope = `{namespace="tenant-a"}` +// wantMetricsScope is the namespace root's scope under the default +// platform prefix: the namespace's own series, then the series a platform +// exporter attributes to it. +var wantMetricsScope = []string{`{namespace="tenant-a"}`, `{namespace=~"cozy-.*",exported_namespace="tenant-a"}`} + +func testScope() []string { + return MetricsConfig{PlatformNamespacePrefix: "cozy-"}.namespaceScope("tenant-a") +} func unix(t time.Time) string { return strconv.FormatInt(t.Unix(), 10) } func buildMetrics(t *testing.T, endpoint string, in url.Values) metricsRequest { t.Helper() - req, err := BuildMetricsParams(endpoint, in, "tenant-a", testNow, testMetricsLimits()) + req, err := BuildMetricsParams(endpoint, in, testScope(), testNow, testMetricsLimits()) if err != nil { t.Fatalf("BuildMetricsParams(%s, %v): %v", endpoint, in, err) } @@ -43,7 +51,7 @@ func buildMetrics(t *testing.T, endpoint string, in url.Values) metricsRequest { } func metricsStatus(endpoint string, in url.Values) int { - _, err := BuildMetricsParams(endpoint, in, "tenant-a", testNow, testMetricsLimits()) + _, err := BuildMetricsParams(endpoint, in, testScope(), testNow, testMetricsLimits()) if err == nil { return http.StatusOK } @@ -84,8 +92,8 @@ func TestBuildMetricsParamsAllowlist(t *testing.T) { t.Errorf("parameter %q forwarded: %v", k, req.params) } } - if got := req.params["extra_filters[]"]; len(got) != 1 || got[0] != wantMetricsScope { - t.Errorf("extra_filters[] = %q, want exactly [%s]", got, wantMetricsScope) + if got := req.params["extra_filters[]"]; !slices.Equal(got, wantMetricsScope) { + t.Errorf("extra_filters[] = %q, want exactly %q", got, wantMetricsScope) } for _, m := range req.params["match[]"] { if strings.Contains(m, "tenant-b") { @@ -116,8 +124,8 @@ func TestNamespaceSelectorEscapes(t *testing.T) { func TestBuildMetricsParamsLabelEndpointsAlwaysMatch(t *testing.T) { for _, endpoint := range []string{"api/v1/labels", "api/v1/label/__name__/values"} { req := buildMetrics(t, endpoint, url.Values{}) - if got := req.params["match[]"]; len(got) != 1 || got[0] != wantMetricsScope { - t.Errorf("%s without match[] sends %q, want the namespace selector", endpoint, got) + if got := req.params["match[]"]; !slices.Equal(got, wantMetricsScope) { + t.Errorf("%s without match[] sends %q, want the namespace scope %q", endpoint, got, wantMetricsScope) } req = buildMetrics(t, endpoint, url.Values{"match[]": {"up", `{job="x"}`}}) if got := req.params["match[]"]; len(got) != 2 || got[0] != "up" || got[1] != `{job="x"}` { @@ -198,7 +206,7 @@ func TestBuildMetricsParamsCanonicalForms(t *testing.T) { }) want := url.Values{ "query": {"up"}, "start": {"2026-11-04T11:00:00.250Z"}, "end": {"2026-11-04T12:00:00.000Z"}, "step": {"90"}, - "timeout": {"30"}, "extra_filters[]": {wantMetricsScope}, + "timeout": {"30"}, "extra_filters[]": wantMetricsScope, } if req.params.Encode() != want.Encode() { t.Errorf("params = %v\nwant %v", req.params, want) diff --git a/internal/observabilityproxy/metricsscope_test.go b/internal/observabilityproxy/metricsscope_test.go new file mode 100644 index 000000000..ff8d52f41 --- /dev/null +++ b/internal/observabilityproxy/metricsscope_test.go @@ -0,0 +1,237 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package observabilityproxy + +import ( + "net/http" + "net/url" + "regexp" + "slices" + "strconv" + "strings" + "testing" + "time" +) + +func TestNamespaceScope(t *testing.T) { + tests := []struct { + prefix string + want []string + }{ + {"cozy-", []string{`{namespace="tenant-a"}`, `{namespace=~"cozy-.*",exported_namespace="tenant-a"}`}}, + {"", []string{`{namespace="tenant-a"}`}}, + // A prefix is literal: a dot in it must not match any character. + {"sys.", []string{`{namespace="tenant-a"}`, `{namespace=~"sys\\..*",exported_namespace="tenant-a"}`}}, + } + for _, tt := range tests { + if got := (MetricsConfig{PlatformNamespacePrefix: tt.prefix}).namespaceScope("tenant-a"); !slices.Equal(got, tt.want) { + t.Errorf("prefix %q: scope = %q, want %q", tt.prefix, got, tt.want) + } + } +} + +func TestNamespaceScopeEscapesExportedNamespace(t *testing.T) { + for _, ns := range []string{`a"} or {namespace=~".+`, `back\slash`, "new\nline"} { + scope := MetricsConfig{PlatformNamespacePrefix: "cozy-"}.namespaceScope(ns) + filters, ok := parseSelector(scope[1]) + if !ok || len(filters) != 2 || filters[1].label != "exported_namespace" || filters[1].value != ns { + t.Errorf("scope %q does not decode to exported_namespace=%q: %+v", scope[1], ns, filters) + } + } +} + +// TestBuildMetricsParamsScopeOnEveryEndpoint checks that every endpoint of +// the namespace root sends the whole scope and nothing a client adds. +func TestBuildMetricsParamsScopeOnEveryEndpoint(t *testing.T) { + endpoints := map[string]url.Values{ + "api/v1/query": {"query": {"up"}}, + "api/v1/query_range": {"query": {"up"}, "start": {unix(testNow.Add(-5 * time.Minute))}, "end": {unix(testNow)}, "step": {"60"}}, + "api/v1/series": {"match[]": {"up"}}, + "api/v1/labels": {}, + "api/v1/label/pod/values": {}, + } + for _, prefix := range []string{"cozy-", ""} { + scope := MetricsConfig{PlatformNamespacePrefix: prefix}.namespaceScope("tenant-a") + for endpoint, in := range endpoints { + in = cloneValues(in) + in["extra_filters[]"] = []string{`{namespace=~".+"}`} + req, herr := BuildMetricsParams(endpoint, in, scope, testNow, testMetricsLimits()) + if herr != nil { + t.Fatalf("%s: %v", endpoint, herr) + } + if got := req.params["extra_filters[]"]; !slices.Equal(got, scope) { + t.Errorf("prefix %q, %s: extra_filters[] = %q, want %q", prefix, endpoint, got, scope) + } + } + } +} + +// An empty scope would send no extra_filters[] and read every namespace. +func TestBuildMetricsParamsRefusesEmptyScope(t *testing.T) { + if _, herr := BuildMetricsParams("api/v1/query", url.Values{"query": {"up"}}, nil, testNow, testMetricsLimits()); herr == nil || herr.Code != http.StatusInternalServerError { + t.Errorf("empty scope: %v, want 500", herr) + } +} + +// TestBuildMetricsParamsCopiesScope guards against the forwarded parameters +// sharing the caller's slice. +func TestBuildMetricsParamsCopiesScope(t *testing.T) { + scope := testScope() + req, herr := BuildMetricsParams("api/v1/labels", url.Values{}, scope, testNow, testMetricsLimits()) + if herr != nil { + t.Fatal(herr) + } + scope[1] = `{namespace=~".+"}` + if !slices.Equal(req.params["extra_filters[]"], wantMetricsScope) || !slices.Equal(req.params["match[]"], wantMetricsScope) { + t.Errorf("params follow the caller's slice: %v", req.params) + } +} + +// TestVmselectSeesScopedSeries runs the parameters the proxy builds +// through a model of vmselect v1.136's series selection: repeated +// extra_filters[] are ORed (searchutil.GetExtraTagFilters), and every +// match[] or query selector is ANDed with each of them +// (searchutil.JoinTagFilterss, called from getCommonParamsInternal and +// promql eval). Regex filters match the whole value. +func TestVmselectSeesScopedSeries(t *testing.T) { + series := []struct { + labels map[string]string + visible bool + }{ + {map[string]string{"namespace": "tenant-a"}, true}, + {map[string]string{"namespace": "cozy-clickhouse-operator", "exported_namespace": "tenant-a", "chi": "clickhouse-db"}, true}, + // A tenant pod claiming another namespace is not a platform exporter. + {map[string]string{"namespace": "tenant-y", "exported_namespace": "tenant-a"}, false}, + {map[string]string{"namespace": "xcozy-a", "exported_namespace": "tenant-a"}, false}, + {map[string]string{"namespace": "cozy-clickhouse-operator", "exported_namespace": "tenant-b"}, false}, + {map[string]string{"namespace": "cozy-clickhouse-operator"}, false}, + {map[string]string{"namespace": "tenant-b"}, false}, + } + for _, endpoint := range []string{"api/v1/series", "api/v1/labels"} { + in := url.Values{} + if endpoint == "api/v1/series" { + in.Set("match[]", `{__name__=~".*"}`) + } + req := buildMetrics(t, endpoint, in) + for _, s := range series { + if got := vmselectSelects(t, req.params, s.labels); got != s.visible { + t.Errorf("%s: series %v visible = %v, want %v", endpoint, s.labels, got, s.visible) + } + } + } + + // Without a prefix the exporter's series stay out of the namespace. + req, herr := BuildMetricsParams("api/v1/labels", url.Values{}, MetricsConfig{}.namespaceScope("tenant-a"), testNow, testMetricsLimits()) + if herr != nil { + t.Fatal(herr) + } + if vmselectSelects(t, req.params, map[string]string{"namespace": "cozy-clickhouse-operator", "exported_namespace": "tenant-a"}) { + t.Error("exporter series visible with the platform prefix disabled") + } +} + +func cloneValues(in url.Values) url.Values { + out := url.Values{} + for k, v := range in { + out[k] = slices.Clone(v) + } + return out +} + +type labelFilter struct { + label, value string + regex bool +} + +func (f labelFilter) matches(labels map[string]string) bool { + v := labels[f.label] + if f.regex { + return regexp.MustCompile("^(?:" + f.value + ")$").MatchString(v) + } + return v == f.value +} + +// parseSelector reads the selectors the proxy and these tests write: +// {name="value",name=~"regex",...} with values quoted as MetricsQL reads +// them (strconv.Unquote in metricsql's extractStringValue). +func parseSelector(s string) ([]labelFilter, bool) { + body, ok := strings.CutPrefix(s, "{") + if !ok { + return nil, false + } + var out []labelFilter + for body != "}" { + i := strings.IndexAny(body, "=!") + if i <= 0 { + return nil, false + } + f := labelFilter{label: body[:i]} + body = body[i:] + switch { + case strings.HasPrefix(body, "=~"): + f.regex, body = true, body[2:] + case strings.HasPrefix(body, "="): + body = body[1:] + default: + return nil, false + } + lit, err := strconv.QuotedPrefix(body) + if err != nil { + return nil, false + } + if f.value, err = strconv.Unquote(lit); err != nil { + return nil, false + } + out = append(out, f) + body = strings.TrimPrefix(body[len(lit):], ",") + } + return out, true +} + +func vmselectSelects(t *testing.T, params url.Values, labels map[string]string) bool { + t.Helper() + parse := func(sels []string) [][]labelFilter { + var out [][]labelFilter + for _, s := range sels { + f, ok := parseSelector(s) + if !ok { + t.Fatalf("cannot parse selector %q", s) + } + out = append(out, f) + } + return out + } + matches, extra := parse(params["match[]"]), parse(params["extra_filters[]"]) + if len(matches) == 0 { + matches = [][]labelFilter{nil} + } + for _, m := range matches { + for _, e := range extra { + all := true + for _, f := range append(slices.Clone(m), e...) { + if f.label == "__name__" { + continue + } + all = all && f.matches(labels) + } + if all { + return true + } + } + } + return false +} diff --git a/internal/observabilityproxy/server.go b/internal/observabilityproxy/server.go index eb540a271..2e8d8ebf9 100644 --- a/internal/observabilityproxy/server.go +++ b/internal/observabilityproxy/server.go @@ -289,7 +289,7 @@ func (s *Server) serveMetrics(w http.ResponseWriter, r *http.Request, route Rout return herr } now := s.now() - req, herr := BuildMetricsParams(route.Endpoint, r.Form, route.Namespace, now, s.cfg.Metrics.Limits) + req, herr := BuildMetricsParams(route.Endpoint, r.Form, s.cfg.Metrics.namespaceScope(route.Namespace), now, s.cfg.Metrics.Limits) if herr != nil { return herr } diff --git a/internal/observabilityproxy/server_metrics_test.go b/internal/observabilityproxy/server_metrics_test.go index 1ef5e761e..b4cfdeef5 100644 --- a/internal/observabilityproxy/server_metrics_test.go +++ b/internal/observabilityproxy/server_metrics_test.go @@ -20,6 +20,7 @@ import ( "io" "net/http" "net/url" + "slices" "sync" "testing" "time" @@ -75,7 +76,7 @@ func TestServerMetricsRebuildsUpstreamRequest(t *testing.T) { if r.path != "/api/v1/query_range" { t.Errorf("upstream path = %s", r.path) } - if got := r.form["extra_filters[]"]; len(got) != 1 || got[0] != wantMetricsScope { + if got := r.form["extra_filters[]"]; !slices.Equal(got, wantMetricsScope) { t.Errorf("extra_filters[] = %q", got) } if r.form.Has("extra_filters") || r.form.Has("extra_label") || r.form.Has("nocache") { @@ -105,7 +106,7 @@ func TestServerMetricsStoreByRange(t *testing.T) { if want := []string{"shortterm", "longterm", "shortterm"}; len(h.stores) != 3 || h.stores[0] != want[0] || h.stores[1] != want[1] || h.stores[2] != want[2] { t.Fatalf("stores = %v, want %v", h.stores, want) } - if r := h.got[2]; r.path != "/api/v1/label/pod/values" || r.form.Get("match[]") != wantMetricsScope { + if r := h.got[2]; r.path != "/api/v1/label/pod/values" || !slices.Equal(r.form["match[]"], wantMetricsScope) { t.Fatalf("label values upstream = %s %v", r.path, r.form) } } diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index 2670d0e3e..d807ae4f3 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -83,6 +83,7 @@ func newHarness(t *testing.T, cfg Config, handler http.HandlerFunc) *harness { cfg.Metrics.UpstreamURL = cfg.UpstreamURL if cfg.Metrics.ShortTermStore == "" { cfg.Metrics.ShortTermStore, cfg.Metrics.LongTermStore, cfg.Metrics.ShortTermRetention = "shortterm", "longterm", 72*time.Hour + cfg.Metrics.PlatformNamespacePrefix = "cozy-" } if cfg.Metrics.Limits == (MetricsLimits{}) { cfg.Metrics.Limits = testMetricsLimits() From 053026c8dfd942f9cc7120a58e1881114e412f26 Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Tue, 10 Nov 2026 19:43:21 +0700 Subject: [PATCH 414/518] feat(observability-proxy): add the --metrics-platform-namespace-prefix flag The prefix decides which namespaces are trusted to attribute series to another namespace, so it defaults to cozy-, the prefix of every platform namespace, and can be emptied to turn that off. It is limited to namespace name characters: a regex metacharacter or an upper case letter would match namespaces other than the ones named. Assisted-by: LLM Signed-off-by: Maxim Belyy --- cmd/observability-proxy/main.go | 6 ++++++ cmd/observability-proxy/main_test.go | 18 +++++++++++++++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index a3687fe08..c527f5061 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -92,6 +92,7 @@ func parseFlags(args []string) (options, error) { fs.DurationVar(&o.metrics.Limits.MaxRange, "metrics-max-range", 30*24*time.Hour, "longest [start, end] and step a metrics request may use") fs.IntVar(&o.metrics.Limits.MaxPoints, "metrics-max-points", 11000, "most points per series a range query may return") fs.DurationVar(&o.metrics.Limits.MaxTimeout, "metrics-max-timeout", 30*time.Second, "timeout forwarded to vmselect; a longer client timeout is clamped; must be below --request-timeout when that is set") + fs.StringVar(&o.metrics.PlatformNamespacePrefix, "metrics-platform-namespace-prefix", "cozy-", "namespaces with this prefix run platform exporters, whose exported_namespace label adds series to that namespace's scope; empty disables") fs.IntVar(&o.limiter.MaxConcurrentPerUser, "max-concurrent-per-user", 10, "concurrent non-tail requests per user; 0 disables") fs.Float64Var(&o.limiter.RatePerUser, "rate-limit-per-user", 10, "sustained requests per second per user; 0 disables") fs.IntVar(&o.limiter.RateBurst, "rate-limit-burst", 20, "request burst per user") @@ -112,6 +113,11 @@ func parseFlags(args []string) (options, error) { return o, fmt.Errorf("invalid metrics store name %q", store) } } + // Anything but namespace name characters would make the prefix match + // differently from the namespaces it is meant to name. + if p := o.metrics.PlatformNamespacePrefix; p != "" && len(validation.IsDNS1123Label(p+"x")) > 0 { + return o, fmt.Errorf("invalid metrics platform namespace prefix %q", p) + } switch { case o.rootTenant == "": return o, errors.New("--root-tenant must not be empty") diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index bf7ac396b..13e137dd2 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -43,6 +43,10 @@ func TestParseFlagsDefaults(t *testing.T) { if m.RootTenant != o.rootTenant { t.Errorf("metrics root tenant = %q, want --root-tenant", m.RootTenant) } + // The prefix every platform namespace carries. + if m.PlatformNamespacePrefix != "cozy-" { + t.Errorf("platform namespace prefix = %q, want cozy-", m.PlatformNamespacePrefix) + } } func TestParseFlagsOverrides(t *testing.T) { @@ -63,15 +67,22 @@ func TestParseFlagsMetricsOverrides(t *testing.T) { o, err := parseFlags([]string{ "--root-tenant=tenant-x", "--metrics-shortterm-store=fast", "--metrics-longterm-store=slow", "--metrics-shortterm-retention=24h", "--metrics-max-range=168h", "--metrics-max-points=500", "--metrics-max-timeout=10s", + "--metrics-platform-namespace-prefix=sys-", }) if err != nil { t.Fatal(err) } m := o.metrics if m.RootTenant != "tenant-x" || m.ShortTermStore != "fast" || m.LongTermStore != "slow" || m.ShortTermRetention != 24*time.Hour || - m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second { + m.Limits.MaxRange != 168*time.Hour || m.Limits.MaxPoints != 500 || m.Limits.MaxTimeout != 10*time.Second || + m.PlatformNamespacePrefix != "sys-" { t.Errorf("overrides not applied: %+v", m) } + // An empty prefix switches the exporter branch of the scope off. + o, err = parseFlags([]string{"--metrics-platform-namespace-prefix="}) + if err != nil || o.metrics.PlatformNamespacePrefix != "" { + t.Errorf("empty platform namespace prefix: %q, %v", o.metrics.PlatformNamespacePrefix, err) + } } func TestParseFlagsInvalid(t *testing.T) { @@ -101,6 +112,11 @@ func TestParseFlagsInvalid(t *testing.T) { {"--metrics-max-range=0s"}, {"--metrics-max-points=1"}, {"--metrics-max-timeout=0s"}, + // Only namespace name characters: a regex metacharacter or an upper + // case letter would widen or empty the scope. + {"--metrics-platform-namespace-prefix=Cozy-"}, + {"--metrics-platform-namespace-prefix=cozy.*"}, + {"--metrics-platform-namespace-prefix=-"}, {"--unknown"}, // The proxy would cut the request while vmselect is still within // the timeout it was given. From 84d490e1bfc3bf355d8de4194fe1b0be4480b586 Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Tue, 10 Nov 2026 19:43:21 +0700 Subject: [PATCH 415/518] feat(observability-proxy): expose the platform namespace prefix in the chart The value is always passed, also when empty, so that clearing it in the values turns the exporter series off instead of falling back to the binary's default. Assisted-by: LLM Signed-off-by: Maxim Belyy --- .../templates/deployment.yaml | 1 + .../tests/deployment_test.yaml | 16 ++++++++++++++++ packages/system/observability-proxy/values.yaml | 4 ++++ 3 files changed, 21 insertions(+) diff --git a/packages/system/observability-proxy/templates/deployment.yaml b/packages/system/observability-proxy/templates/deployment.yaml index d49de140b..c726c4282 100644 --- a/packages/system/observability-proxy/templates/deployment.yaml +++ b/packages/system/observability-proxy/templates/deployment.yaml @@ -41,6 +41,7 @@ spec: - --metrics-shortterm-store={{ .Values.observabilityProxy.metrics.shorttermStore }} - --metrics-longterm-store={{ .Values.observabilityProxy.metrics.longtermStore }} - --metrics-shortterm-retention={{ .Values.observabilityProxy.metrics.shorttermRetention }} + - --metrics-platform-namespace-prefix={{ .Values.observabilityProxy.metrics.platformNamespacePrefix }} ports: - name: http containerPort: 8080 diff --git a/packages/system/observability-proxy/tests/deployment_test.yaml b/packages/system/observability-proxy/tests/deployment_test.yaml index a5e948920..a4c5e760c 100644 --- a/packages/system/observability-proxy/tests/deployment_test.yaml +++ b/packages/system/observability-proxy/tests/deployment_test.yaml @@ -51,6 +51,9 @@ tests: - contains: path: spec.template.spec.containers[0].args content: --metrics-shortterm-retention=72h + - contains: + path: spec.template.spec.containers[0].args + content: --metrics-platform-namespace-prefix=cozy- - it: follows renamed metrics stores set: @@ -69,6 +72,19 @@ tests: path: spec.template.spec.containers[0].args content: --metrics-shortterm-retention=24h + # An empty value must reach the binary as an explicit empty flag; a + # dropped flag would fall back to the cozy- default. + - it: passes an empty platform namespace prefix through + set: + observabilityProxy.metrics.platformNamespacePrefix: "" + asserts: + - contains: + path: spec.template.spec.containers[0].args + content: --metrics-platform-namespace-prefix= + - notContains: + path: spec.template.spec.containers[0].args + content: --metrics-platform-namespace-prefix=cozy- + - it: falls back to cozy.local when the platform passes no cluster domain asserts: - contains: diff --git a/packages/system/observability-proxy/values.yaml b/packages/system/observability-proxy/values.yaml index 67334a237..3199d30bc 100644 --- a/packages/system/observability-proxy/values.yaml +++ b/packages/system/observability-proxy/values.yaml @@ -14,6 +14,10 @@ observabilityProxy: shorttermStore: shortterm longtermStore: longterm shorttermRetention: 72h + # Exporters in namespaces with this prefix may attribute a series to a + # tenant namespace through exported_namespace, and the series is then + # part of that namespace's metrics. Empty disables it. + platformNamespacePrefix: cozy- resources: requests: cpu: 10m From 59a11c1788fa23db53995e0de52485c7e61b1fec Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Tue, 10 Nov 2026 15:52:21 +0300 Subject: [PATCH 416/518] refactor(postgres): judge an attach record's clock drift in one place The job that attached the plugin and a job waiting on another job's attach each parsed the Cluster's attach record and applied the clock-drift margin on their own, so the two could drift apart on the next change. Both now take the record from one parser that also refuses one too far ahead, and differ only in what they do with a record they cannot use. The margin is now pinned for the waiting job from both sides: a record a minute ahead still holds it for the rollout. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 25 +++++++++---------- .../cnpgstrategy_controller_test.go | 1 + 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index bdc04bb0e..375af221c 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -505,10 +505,6 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp // The Cluster's record covers an attach whose status write was lost. attachedAt := rollout.LastTransitionTime recorded, ok, err := cnpgRecordedAttach(attachment.attachedAt) - if err == nil && ok && time.Until(recorded) > cnpgAttachClockSkewTolerance { - // Every instance would predate it, and the deadline would count from it. - err = fmt.Errorf("%s is in the future", attachment.attachedAt) - } if err != nil { return r.markBackupJobFailed(ctx, j, fmt.Sprintf( "cannot tell when the barman-cloud plugin was attached: the %s annotation of cnpg.io/Cluster %s/%s: %v", @@ -553,15 +549,12 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp rolledOut = true } else if existingBackup == nil && attachment.attachedAt != "" { // Another backup job attached the plugin. Its rollout can still be - // running while the attach is recent, and later than that the - // attaching job has failed it already. The record is that job's: - // one this job cannot read does not fail it, and neither does one - // further ahead than clocks drift, which every instance would - // predate for as long as it lies ahead. + // running while the attach is recent; past the rollout deadline + // this job stops waiting, as the attaching job normally has judged + // the rollout by then unless a wait of its own moved its deadline + // on. The record is that job's: + // one this job cannot use does not fail it. recorded, _, err := cnpgRecordedAttach(attachment.attachedAt) - if err == nil && time.Until(recorded) > cnpgAttachClockSkewTolerance { - err = fmt.Errorf("%s is in the future", attachment.attachedAt) - } if err != nil { logger.Info("ignoring an unusable attach record another backup job left", "cluster", clusterName, "annotation", cnpgPluginAttachedAtAnnotation, "error", err) @@ -924,7 +917,10 @@ func (r *BackupJobReconciler) cnpgPluginRolloutPending(ctx context.Context, name } // cnpgRecordedAttach parses a cnpgPluginAttachedAtAnnotation value; ok is -// false for "", a Cluster that carries no record. +// false for "", a Cluster that carries no record. A record further ahead of +// this controller's clock than cnpgAttachClockSkewTolerance is an error: +// every instance would predate it, and a deadline counted from it would not +// run out for as long as it lies ahead. func cnpgRecordedAttach(raw string) (recorded time.Time, ok bool, err error) { if raw == "" { return time.Time{}, false, nil @@ -933,6 +929,9 @@ func cnpgRecordedAttach(raw string) (recorded time.Time, ok bool, err error) { if err != nil { return time.Time{}, false, err } + if time.Until(recorded) > cnpgAttachClockSkewTolerance { + return time.Time{}, false, fmt.Errorf("%s is in the future", raw) + } return recorded, true, nil } diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 78c9e9966..1d36117fa 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -3227,6 +3227,7 @@ func TestReconcileCNPG_JobWithoutLatchWaitsForAnotherJobsRollout(t *testing.T) { {name: "attach older than the rollout deadline", record: time.Now().Add(-2 * cnpgDefaultBackupDeadline).UTC().Format(time.RFC3339), podCreated: time.Now().Add(-3 * cnpgDefaultBackupDeadline), wantBackup: true}, {name: "unreadable record", record: "yesterday", podCreated: time.Now().Add(-time.Hour), wantBackup: true}, {name: "record further ahead than clocks drift", record: time.Now().Add(time.Hour).UTC().Format(time.RFC3339), podCreated: time.Now().Add(-time.Hour), wantBackup: true}, + {name: "record ahead within clock drift", record: time.Now().Add(time.Minute).UTC().Format(time.RFC3339), podCreated: time.Now().Add(-time.Hour), wantBackup: false}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { From c6f1d6c55938725cf631c2ea92584602af036dfd Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Tue, 10 Nov 2026 14:37:16 +0100 Subject: [PATCH 417/518] fix(vm-instance): keep an external PortList VM valid with no ports The API server rejects a Service that declares no port, so a VM switched to external PortList with an empty port list could not be upgraded: the release failed and was retried forever, while the old Service kept the previously listed ports open. Render a single SCTP placeholder in that case. cozy-proxy only admits TCP and UDP ports into its PortList filter, so the VM keeps its external IP for egress and ICMP while all inbound TCP and UDP traffic is dropped, which is what an empty port list means. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../apps/vm-instance/templates/service.yaml | 7 ++ .../vm-instance/tests/service_ports_test.yaml | 70 +++++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100644 packages/apps/vm-instance/tests/service_ports_test.yaml diff --git a/packages/apps/vm-instance/templates/service.yaml b/packages/apps/vm-instance/templates/service.yaml index cfeffa810..b757ab96d 100644 --- a/packages/apps/vm-instance/templates/service.yaml +++ b/packages/apps/vm-instance/templates/service.yaml @@ -30,6 +30,13 @@ spec: {{- if .Values.external }} {{- if and (eq .Values.externalMethod "WholeIP") (not .Values.externalPorts) }} - port: 65535 + {{- else if not .Values.externalPorts }} + {{- /* A Service needs at least one port. cozy-proxy only admits TCP and + UDP ports into its PortList filter, so an SCTP placeholder keeps the + external IP while every inbound TCP and UDP packet is dropped. */}} + - name: no-ports + port: 65535 + protocol: SCTP {{- else }} {{- range .Values.externalPorts }} - name: port-{{ . }} diff --git a/packages/apps/vm-instance/tests/service_ports_test.yaml b/packages/apps/vm-instance/tests/service_ports_test.yaml new file mode 100644 index 000000000..d7689d957 --- /dev/null +++ b/packages/apps/vm-instance/tests/service_ports_test.yaml @@ -0,0 +1,70 @@ +suite: vm-instance external Service ports + +# The API server rejects a Service without ports, so an external PortList VM +# whose port list was emptied could not be upgraded at all. The placeholder +# must be SCTP: cozy-proxy only admits TCP and UDP ports into its filter, so +# any TCP or UDP placeholder would open that port on the VM. +release: + name: test-vm + namespace: tenant-test +templates: + - templates/service.yaml +set: + fullnameOverride: test-vm + external: true + +tests: + - it: renders an SCTP placeholder for PortList with no ports + set: + externalMethod: PortList + externalPorts: [] + asserts: + - equal: + path: spec.type + value: LoadBalancer + - equal: + path: metadata.annotations["networking.cozystack.io/wholeIP"] + value: "false" + - lengthEqual: + path: spec.ports + count: 1 + - equal: + path: spec.ports[0] + value: + name: no-ports + port: 65535 + protocol: SCTP + + - it: renders the listed ports for PortList + set: + externalMethod: PortList + externalPorts: [22, 443] + asserts: + - lengthEqual: + path: spec.ports + count: 2 + - equal: + path: spec.ports[0] + value: + name: port-22 + port: 22 + targetPort: 22 + - notContains: + path: spec.ports + content: + name: no-ports + port: 65535 + protocol: SCTP + + - it: keeps the WholeIP placeholder when no ports are listed + set: + externalMethod: WholeIP + externalPorts: [] + asserts: + - equal: + path: metadata.annotations["networking.cozystack.io/wholeIP"] + value: "true" + - equal: + path: spec.ports + value: + - port: 65535 From 9cd51cac2e64aacfe812be7ee5be2cab94c55d20 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Mon, 9 Nov 2026 18:38:56 +0300 Subject: [PATCH 418/518] ci(e2e): skip final cleanup on disposable PR runners Destroying runner-local state after a completed suite can push the job past its deadline even though the runner is discarded. Keep cleanup between preparation attempts and for reused hosts instead. Fixes: 186547d835cc ("test(e2e): run the merge-gating lanes on Talos containers") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .github/workflows/e2e-fork.yaml | 15 -------------- .github/workflows/pull-requests.yaml | 22 -------------------- hack/cozyreport.bats | 31 ++++++++++++++++++++++++++++ 3 files changed, 31 insertions(+), 37 deletions(-) diff --git a/.github/workflows/e2e-fork.yaml b/.github/workflows/e2e-fork.yaml index df6ed1268..f988632e9 100644 --- a/.github/workflows/e2e-fork.yaml +++ b/.github/workflows/e2e-fork.yaml @@ -771,21 +771,6 @@ jobs: name: cozyreport path: _out/cozyreport.tgz if-no-files-found: warn - # Both halves, and in this order: the compose project owns the nodes and - # the zpools outlive it. - - name: Tear down sandbox - if: always() - run: | - # Annotated, not fatal: same reasoning as the in-tree lane. The runner - # is discarded after the job, so a dirty host has no next tenant, and - # failing here would post the required "E2E Tests" status as a failure - # indistinguishable from a real one. - cleanup_rc=0 - make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || cleanup_rc=$? - make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true - if [ "$cleanup_rc" -ne 0 ]; then - echo "::error title=sandbox teardown::teardown exited $cleanup_rc; the host may hold a stale compose project or zpool" - fi # Flip the pending "E2E Tests" commit status posted by `resolve` to its # terminal state. Runs on failure too, so a failed publish or e2e turns the diff --git a/.github/workflows/pull-requests.yaml b/.github/workflows/pull-requests.yaml index b8b06fedf..7d3787ff9 100644 --- a/.github/workflows/pull-requests.yaml +++ b/.github/workflows/pull-requests.yaml @@ -1348,28 +1348,6 @@ jobs: Resume from inside: `breakpoint resume`. Otherwise the breakpoint exits 10 minutes after the last SSH session disconnects. - # ▸ Tear down environment (always runs) - # - # Both halves, and in this order: the compose project owns the nodes and - # the zpools outlive it. A leftover pool makes the next run on a reused - # host adopt storage that already carries a dead cluster's volumes. - - name: Tear down sandbox - if: always() - run: | - # Loud, but never the job's verdict. A failed teardown means a dirty - # host, and this runner is discarded after the job, so there is no next - # run to protect -- while failing the job here posts the required - # "E2E Tests" status as a failure that no reader can tell apart from a - # real one. The annotation is what a reused host (a developer machine) - # needs; `delete-cluster-container` returning non-zero is not evidence - # about the product. - cleanup_rc=0 - make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete-cluster-container || cleanup_rc=$? - make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME delete || true - if [ "$cleanup_rc" -ne 0 ]; then - echo "::error title=sandbox teardown::teardown exited $cleanup_rc; the host may hold a stale compose project or zpool" - fi - # Report the required "E2E Tests" gate as a commit status rather than relying # on a job named "E2E Tests". A required *job* that is skipped counts # as PASSED by branch protection — desired for docs-only PRs, but the exact diff --git a/hack/cozyreport.bats b/hack/cozyreport.bats index cc2668071..ee8911080 100644 --- a/hack/cozyreport.bats +++ b/hack/cozyreport.bats @@ -966,6 +966,37 @@ STUB } } +@test "PR E2E jobs clean up between preparation attempts but not after testing" { + tmp="$(mktemp -d)" + mkdir -p "$tmp/bin" + cat >"$tmp/bin/make" <<'STUB' +#!/bin/sh +printf '%s\n' "$*" >>"$CLEANUP_CALLS" +case "$*" in + *prepare-env-container*) + if [ ! -f "$CLEANUP_ATTEMPT" ]; then + touch "$CLEANUP_ATTEMPT" + exit 1 + fi + ;; +esac +STUB + chmod +x "$tmp/bin/make" + for wf in pull-requests e2e-fork; do + workflow="$HACK_DIR/../.github/workflows/$wf.yaml" + yq --unwrapScalar '.jobs.e2e.steps[] | select(.name == "Prepare environment") | .run' "$workflow" >"$tmp/prepare.sh" + CLEANUP_CALLS="$tmp/$wf.calls" CLEANUP_ATTEMPT="$tmp/$wf.attempt" SANDBOX_NAME=test PATH="$tmp/bin:$PATH" bash -eu "$tmp/prepare.sh" + printf '%s\n' 'SANDBOX_NAME=test prepare-env-container' '-C packages/core/testing SANDBOX_NAME=test delete-cluster-container' 'SANDBOX_NAME=test prepare-env-container' >"$tmp/expected" + diff "$tmp/expected" "$tmp/$wf.calls" + scripts="$(yq --unwrapScalar '.jobs.e2e.steps[] | select(.name != "Prepare environment") | .run // ""' "$workflow")" + if printf '%s\n' "$scripts" | grep -Eq 'delete-cluster-container|docker[[:space:]]+(compose.*down|rm)|zpool[[:space:]]+destroy|make.*[[:space:]]delete([[:space:]]|$)'; then + echo "FAIL: $wf destroys runner-local state outside preparation" + false + fi + done + rm -rf "$tmp" +} + @test "the step that runs this collector has a ceiling of its own everywhere" { # Every bound inside the script protects one read. None of them protects the # STEP: the tarball is written on the collector's last line, the upload is the From 5a2582e21cade53235c5c163ab4c3b658217d4fc Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Tue, 10 Nov 2026 23:21:56 +0300 Subject: [PATCH 419/518] test(observability-proxy): preserve JSON seed request bodies Send JSON seeds with their content type so VictoriaLogs does not consume log records while parsing form parameters before ingestion. Fixes: ebb5a9cec4f6 ("test(observability-proxy): cover the namespace logs scope end to end") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- hack/e2e-chainsaw/observability-proxy/lib.sh | 2 +- hack/observability-proxy-seed.bats | 51 ++++++++++++++++++++ 2 files changed, 52 insertions(+), 1 deletion(-) create mode 100644 hack/observability-proxy-seed.bats diff --git a/hack/e2e-chainsaw/observability-proxy/lib.sh b/hack/e2e-chainsaw/observability-proxy/lib.sh index a9a1f28a5..9601cb826 100644 --- a/hack/e2e-chainsaw/observability-proxy/lib.sh +++ b/hack/e2e-chainsaw/observability-proxy/lib.sh @@ -25,7 +25,7 @@ forward() { ingest() { printf '{"kubernetes_namespace_name":"%s","log":"%s"}\n' "$1" "$2" | timeout 15 curl --silent --show-error --fail --data-binary @- \ - --header "AccountID: ${3:-0}" --header "ProjectID: 0" \ + --header 'Content-Type: application/json' --header "AccountID: ${3:-0}" --header "ProjectID: 0" \ "$STORE/insert/jsonline?_stream_fields=kubernetes_namespace_name&_msg_field=log" } diff --git a/hack/observability-proxy-seed.bats b/hack/observability-proxy-seed.bats new file mode 100644 index 000000000..8815b0db5 --- /dev/null +++ b/hack/observability-proxy-seed.bats @@ -0,0 +1,51 @@ +#!/usr/bin/env bats + +load test_helper + +REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)" + +setup() { + strict_setup + mkdir -p "$BATS_TEST_TMPDIR/bin" + export SEED_CAPTURE="$BATS_TEST_TMPDIR/seed" + export TMPDIR="$BATS_TEST_TMPDIR" + export PATH="$BATS_TEST_TMPDIR/bin:$PATH" + cat > "$BATS_TEST_TMPDIR/bin/curl" <<'EOF' +#!/usr/bin/env bash +set -eu +printf '%s\n' "$@" > "$SEED_CAPTURE.args" +cat > "$SEED_CAPTURE.body" +if ! grep --fixed-strings --line-regexp --quiet 'Content-Type: application/json' "$SEED_CAPTURE.args"; then + echo 'seed request is missing its JSON content type' >&2 + exit 22 +fi +EOF + chmod +x "$BATS_TEST_TMPDIR/bin/curl" +} + +assert_seed_request() { + local account=$1 + # shellcheck source=e2e-chainsaw/observability-proxy/lib.sh + . "$REPO_ROOT/hack/e2e-chainsaw/observability-proxy/lib.sh" + if [ "$account" = 0 ]; then + ingest tenant-test-logsa e2e-marker-a + else + ingest tenant-test-logsa e2e-marker-a "$account" + fi + grep --fixed-strings --line-regexp --quiet "AccountID: $account" "$SEED_CAPTURE.args" + grep --fixed-strings --line-regexp --quiet 'ProjectID: 0' "$SEED_CAPTURE.args" + grep --fixed-strings --line-regexp --quiet 'http://localhost:19471/insert/jsonline?_stream_fields=kubernetes_namespace_name&_msg_field=log' "$SEED_CAPTURE.args" + grep --fixed-strings --line-regexp --quiet -- '--data-binary' "$SEED_CAPTURE.args" + grep --fixed-strings --line-regexp --quiet '@-' "$SEED_CAPTURE.args" + printf '%s\n' '{"kubernetes_namespace_name":"tenant-test-logsa","log":"e2e-marker-a"}' > "$SEED_CAPTURE.expected" + cmp "$SEED_CAPTURE.expected" "$SEED_CAPTURE.body" + rm "$body" +} + +@test "observability seeds send JSON without consuming the body as form data" { + assert_seed_request 0 +} + +@test "observability seeds preserve an explicit VictoriaLogs account" { + assert_seed_request 7 +} From 7c9aa2fe103e9a06416adb3aaec0d085e7a5d7f9 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Tue, 10 Nov 2026 23:43:06 +0300 Subject: [PATCH 420/518] fix(seaweedfs): hand storage class names to the data plane as strings The volume and admin storage classes were written unquoted into the values of the data-plane release. A class name made only of digits, which Kubernetes allows, arrived there as a number, and a value with a line break could add keys next to the field, past the schema that is the tenant's input gate. Both are now quoted. lifecycle.storageClass, new in this release, also carries the StorageClass name rules in its schema, so a value that cannot name a class is refused when the application is written instead of leaving the admin claim pending. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../extra/seaweedfs/templates/seaweedfs.yaml | 4 +- .../extra/seaweedfs/tests/lifecycle_test.yaml | 39 +++++++++++++++ .../tests/volume_storage_class_test.yaml | 47 +++++++++++++++++++ packages/extra/seaweedfs/values.schema.json | 4 +- packages/extra/seaweedfs/values.yaml | 2 + .../seaweedfs-rd/cozyrds/seaweedfs.yaml | 2 +- 6 files changed, 94 insertions(+), 4 deletions(-) create mode 100644 packages/extra/seaweedfs/tests/volume_storage_class_test.yaml diff --git a/packages/extra/seaweedfs/templates/seaweedfs.yaml b/packages/extra/seaweedfs/templates/seaweedfs.yaml index 9990a450c..52009724b 100644 --- a/packages/extra/seaweedfs/templates/seaweedfs.yaml +++ b/packages/extra/seaweedfs/templates/seaweedfs.yaml @@ -260,7 +260,7 @@ spec: type: "persistentVolumeClaim" size: "{{ .Values.volume.size }}" {{- with .Values.volume.storageClass }} - storageClass: {{ . }} + storageClass: {{ . | quote }} {{- end }} maxVolumes: 0 {{- if .Values.volume.diskType }} @@ -387,7 +387,7 @@ spec: existingSecret: {{ .Release.Name }}-admin-credentials {{- with .Values.lifecycle.storageClass }} data: - storageClass: {{ . }} + storageClass: {{ . | quote }} {{- end }} worker: enabled: {{ .Values.lifecycle.enabled }} diff --git a/packages/extra/seaweedfs/tests/lifecycle_test.yaml b/packages/extra/seaweedfs/tests/lifecycle_test.yaml index 7af101faf..633514280 100644 --- a/packages/extra/seaweedfs/tests/lifecycle_test.yaml +++ b/packages/extra/seaweedfs/tests/lifecycle_test.yaml @@ -119,6 +119,45 @@ tests: - notExists: path: spec.values.seaweedfs.admin.data + # A StorageClass name may be all digits. Handed on unquoted it reaches the + # data-plane release as a number, and the claim it ends up in is rejected. + - it: hands a numeric storage class name on as a string + template: templates/seaweedfs.yaml + set: + <<: *enabled + lifecycle: + enabled: true + storageClass: "123" + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - equal: + path: spec.values.seaweedfs.admin.data.storageClass + value: "123" + + - it: refuses a storage class that is not a StorageClass name + template: templates/seaweedfs.yaml + set: + <<: *enabled + lifecycle: + enabled: true + storageClass: "replicated\n image:\n repository: example/other" + asserts: + - failedTemplate: + errorPattern: "/lifecycle/storageClass" + + - it: refuses a storage class longer than a StorageClass name can be + template: templates/seaweedfs.yaml + set: + <<: *enabled + lifecycle: + enabled: true + storageClass: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + asserts: + - failedTemplate: + errorPattern: "/lifecycle/storageClass" + - it: monitors the admin and the worker while lifecycle is on template: templates/seaweedfs.yaml set: *enabled diff --git a/packages/extra/seaweedfs/tests/volume_storage_class_test.yaml b/packages/extra/seaweedfs/tests/volume_storage_class_test.yaml new file mode 100644 index 000000000..6543afc98 --- /dev/null +++ b/packages/extra/seaweedfs/tests/volume_storage_class_test.yaml @@ -0,0 +1,47 @@ +suite: seaweedfs volume storage class reaches the data plane as a string + +templates: + - templates/seaweedfs.yaml + +release: + name: seaweedfs + namespace: tenant-root + +set: &platform + _namespace: + host: example.org + ingress: tenant-root + _cluster: + issuer-name: letsencrypt-prod + solver: http01 + +tests: + - it: hands a numeric storage class name on as a string + set: + <<: *platform + volume: + storageClass: "123" + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - equal: + path: spec.values.seaweedfs.volume.dataDirs[0].storageClass + value: "123" + + # The value is written into the values of another release; a line break in + # it must not open new keys there. + - it: keeps a storage class with a line break inside its own field + set: + <<: *platform + volume: + storageClass: "local\n image: example/other" + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - equal: + path: spec.values.seaweedfs.volume.dataDirs[0].storageClass + value: "local\n image: example/other" + - notExists: + path: spec.values.seaweedfs.volume.dataDirs[0].image diff --git a/packages/extra/seaweedfs/values.schema.json b/packages/extra/seaweedfs/values.schema.json index 000755ffe..0df1bef8f 100644 --- a/packages/extra/seaweedfs/values.schema.json +++ b/packages/extra/seaweedfs/values.schema.json @@ -882,7 +882,9 @@ "storageClass": { "description": "StorageClass of the admin server's 1Gi job-state volume. Empty uses the cluster default; `volume.storageClass` is not used, because in MultiZone it can be local to one zone. Prefer a replicated class: on a node-local one, losing the node stops lifecycle enforcement. Set it before enabling lifecycle: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. To move the volume, disable lifecycle, delete the `admin-data-seaweedfs-admin-0` PVC of the instance, change the class and enable lifecycle again.", "type": "string", - "default": "" + "default": "", + "maxLength": 253, + "pattern": "^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)?$" } } } diff --git a/packages/extra/seaweedfs/values.yaml b/packages/extra/seaweedfs/values.yaml index 3e5f37953..fdc58a11e 100644 --- a/packages/extra/seaweedfs/values.yaml +++ b/packages/extra/seaweedfs/values.yaml @@ -196,6 +196,8 @@ s3: ## @typedef {struct} Lifecycle - Enforcement of S3 bucket lifecycle rules. ## @field {bool} [enabled] - Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new. ## @field {string} [storageClass] - StorageClass of the admin server's 1Gi job-state volume. Empty uses the cluster default; `volume.storageClass` is not used, because in MultiZone it can be local to one zone. Prefer a replicated class: on a node-local one, losing the node stops lifecycle enforcement. Set it before enabling lifecycle: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. To move the volume, disable lifecycle, delete the `admin-data-seaweedfs-admin-0` PVC of the instance, change the class and enable lifecycle again. +## @maxLength 253 +## @pattern ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)?$ ## @param {Lifecycle} [lifecycle] - S3 bucket lifecycle configuration. lifecycle: diff --git a/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml b/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml index f2efca2f8..6cafe5c5d 100644 --- a/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml +++ b/packages/system/seaweedfs-rd/cozyrds/seaweedfs.yaml @@ -8,7 +8,7 @@ spec: singular: seaweedfs plural: seaweedfses openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"host":{"description":"The hostname used to access SeaweedFS externally (defaults to 's3' subdomain for the tenant host).","type":"string","default":""},"topology":{"description":"The topology of the SeaweedFS cluster.","type":"string","default":"Simple","enum":["Simple","MultiZone","Client"]},"replicationFactor":{"description":"Replication factor: number of replicas for each volume in the SeaweedFS cluster.","type":"integer","default":2},"db":{"description":"Database configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of database replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""}}},"master":{"description":"Master service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of master replicas.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"volumeSizeLimitMB":{"description":"Size in MB at which a volume stops taking writes, and the divisor that sets how many buckets a volume server can serve. Lower it to fit more buckets on the same disk, at the cost of more volume files for the same stored data; on a large volume it is worth raising instead. Changing it on an instance that already holds data costs something either way: lowering it stops writes to every bucket whose volumes all reach the new value until that bucket can grow, and raising it takes the room new buckets grow into, so run the check in docs/operations/seaweedfs-lowering-the-volume-size-limit.md first. Must be between 1 and 30000. A value close to or above `volume.size` leaves each volume server a single slot rather than failing, since the server floors its slot count at one.","type":"integer","default":400,"maximum":30000,"minimum":1}}},"filer":{"description":"Filer service configuration.","type":"object","default":{},"properties":{"grpcHost":{"description":"The hostname used to expose or access the filer service externally.","type":"string","default":""},"grpcPort":{"description":"The port used to access the filer service externally.","type":"integer","default":443},"replicas":{"description":"Number of filer replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"whitelist":{"description":"A list of IP addresses or CIDR ranges that are allowed to access the filer service.","type":"array","default":[],"items":{"type":"string"}}}},"volume":{"description":"Volume service configuration.","type":"object","default":{},"properties":{"diskType":{"description":"SeaweedFS disk type tag for the default volume servers (e.g., \"hdd\", \"ssd\").","type":"string","default":""},"pools":{"description":"A map of storage pools. Each pool creates a separate Volume StatefulSet with its own disk type.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of volume replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""},"zones":{"description":"A map of zones for MultiZone topology. Each zone can have its own number of replicas and size.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"dataCenter":{"description":"SeaweedFS data center name for this zone. Defaults to the zone name.","type":"string"},"nodeSelector":{"description":"YAML nodeSelector for this zone (default: topology.kubernetes.io/zone: ).","type":"string"},"pools":{"description":"A map of storage pools for this zone. Each pool creates a separate Volume StatefulSet per zone.","type":"object","additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of replicas in the zone.","type":"integer"},"size":{"description":"Zone storage size.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store zone data. Defaults to volume.storageClass.","type":"string"}}}}}},"s3":{"description":"S3 service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of S3 replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"lifecycle":{"description":"S3 bucket lifecycle configuration.","type":"object","default":{},"properties":{"enabled":{"description":"Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new.","type":"boolean","default":false},"storageClass":{"description":"StorageClass of the admin server's 1Gi job-state volume. Empty uses the cluster default; `volume.storageClass` is not used, because in MultiZone it can be local to one zone. Prefer a replicated class: on a node-local one, losing the node stops lifecycle enforcement. Set it before enabling lifecycle: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. To move the volume, disable lifecycle, delete the `admin-data-seaweedfs-admin-0` PVC of the instance, change the class and enable lifecycle again.","type":"string","default":""}}}}} + {"title":"Chart Values","type":"object","properties":{"host":{"description":"The hostname used to access SeaweedFS externally (defaults to 's3' subdomain for the tenant host).","type":"string","default":""},"topology":{"description":"The topology of the SeaweedFS cluster.","type":"string","default":"Simple","enum":["Simple","MultiZone","Client"]},"replicationFactor":{"description":"Replication factor: number of replicas for each volume in the SeaweedFS cluster.","type":"integer","default":2},"db":{"description":"Database configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of database replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""}}},"master":{"description":"Master service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of master replicas.","type":"integer","default":3},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"volumeSizeLimitMB":{"description":"Size in MB at which a volume stops taking writes, and the divisor that sets how many buckets a volume server can serve. Lower it to fit more buckets on the same disk, at the cost of more volume files for the same stored data; on a large volume it is worth raising instead. Changing it on an instance that already holds data costs something either way: lowering it stops writes to every bucket whose volumes all reach the new value until that bucket can grow, and raising it takes the room new buckets grow into, so run the check in docs/operations/seaweedfs-lowering-the-volume-size-limit.md first. Must be between 1 and 30000. A value close to or above `volume.size` leaves each volume server a single slot rather than failing, since the server floors its slot count at one.","type":"integer","default":400,"maximum":30000,"minimum":1}}},"filer":{"description":"Filer service configuration.","type":"object","default":{},"properties":{"grpcHost":{"description":"The hostname used to expose or access the filer service externally.","type":"string","default":""},"grpcPort":{"description":"The port used to access the filer service externally.","type":"integer","default":443},"replicas":{"description":"Number of filer replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"whitelist":{"description":"A list of IP addresses or CIDR ranges that are allowed to access the filer service.","type":"array","default":[],"items":{"type":"string"}}}},"volume":{"description":"Volume service configuration.","type":"object","default":{},"properties":{"diskType":{"description":"SeaweedFS disk type tag for the default volume servers (e.g., \"hdd\", \"ssd\").","type":"string","default":""},"pools":{"description":"A map of storage pools. Each pool creates a separate Volume StatefulSet with its own disk type.","type":"object","default":{},"additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of volume replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size.","default":"10Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":""},"zones":{"description":"A map of zones for MultiZone topology. Each zone can have its own number of replicas and size.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"dataCenter":{"description":"SeaweedFS data center name for this zone. Defaults to the zone name.","type":"string"},"nodeSelector":{"description":"YAML nodeSelector for this zone (default: topology.kubernetes.io/zone: ).","type":"string"},"pools":{"description":"A map of storage pools for this zone. Each pool creates a separate Volume StatefulSet per zone.","type":"object","additionalProperties":{"type":"object","required":["diskType"],"properties":{"diskType":{"description":"SeaweedFS disk type tag (e.g., \"ssd\", \"hdd\", \"nvme\").","type":"string"},"replicas":{"description":"Number of volume replicas. Defaults to volume.replicas (Simple) or zone.replicas/volume.replicas (MultiZone).","type":"integer"},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted. Defaults to volume.resourcesPreset.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume size. Defaults to volume.size (Simple) or zone.size/volume.size (MultiZone).","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"Kubernetes StorageClass for the pool. Defaults to volume.storageClass.","type":"string"}}}},"replicas":{"description":"Number of replicas in the zone.","type":"integer"},"size":{"description":"Zone storage size.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store zone data. Defaults to volume.storageClass.","type":"string"}}}}}},"s3":{"description":"S3 service configuration.","type":"object","default":{},"properties":{"replicas":{"description":"Number of S3 replicas.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"Number of CPU cores allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Amount of memory allocated.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.small","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"lifecycle":{"description":"S3 bucket lifecycle configuration.","type":"object","default":{},"properties":{"enabled":{"description":"Run the SeaweedFS admin server and a worker that applies bucket lifecycle rules, once a day. When disabled, rules are still accepted and stored, but not applied. Ignored with the Client topology, whose rules are enforced by the instance it connects to. Rules stored before enabling are applied on the first pass, minutes after the switch. While on, in a bucket without versioning enabled, an object overwritten between the daily listing and its expiry delete can be deleted although it is new.","type":"boolean","default":false},"storageClass":{"description":"StorageClass of the admin server's 1Gi job-state volume. Empty uses the cluster default; `volume.storageClass` is not used, because in MultiZone it can be local to one zone. Prefer a replicated class: on a node-local one, losing the node stops lifecycle enforcement. Set it before enabling lifecycle: the class is part of the admin StatefulSet's volume claim template, which cannot change, so changing it while lifecycle is on makes every upgrade of the instance fail until it is reverted. To move the volume, disable lifecycle, delete the `admin-data-seaweedfs-admin-0` PVC of the instance, change the class and enable lifecycle again.","type":"string","default":"","maxLength":253,"pattern":"^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)?$"}}}}} release: prefix: "" labels: From 0e4711b17cfb08afd1b40820fb97166cba203ed0 Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Wed, 11 Nov 2026 05:41:51 +0700 Subject: [PATCH 421/518] docs(observability-proxy): bound the exporter trust to platform scrapes The cozy-* condition holds only for series the platform's vmagent scraped. A tenant's nested Kubernetes cluster remote-writes into the same store and picks its own namespace labels, cozy-* among them, so the comment must not promise that tenants can never produce such a series. That writer could already pass the namespace="" branch, and the exporter branch gives it nothing more. Assisted-by: LLM Signed-off-by: Maxim Belyy --- internal/observabilityproxy/metricsparams.go | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/internal/observabilityproxy/metricsparams.go b/internal/observabilityproxy/metricsparams.go index c300cdc7a..b2086e494 100644 --- a/internal/observabilityproxy/metricsparams.go +++ b/internal/observabilityproxy/metricsparams.go @@ -92,9 +92,12 @@ func namespaceSelector(ns string) string { // namespaceScope selects the series of namespace ns. Besides its own, it // takes the series an exporter in a platform namespace attributes to ns: // vmagent sets namespace to the scraped target's and, without honorLabels, -// moves the exporter's own namespace label to exported_namespace. Only -// platform namespaces are trusted with that label, since any tenant pod can -// write it; tenants cannot run pods or scrape objects in them. +// moves the exporter's own namespace label to exported_namespace. Any pod +// can write that label, so it is trusted only where the platform's vmagent +// set namespace to one with the platform prefix, in which tenants cannot run +// pods or scrape objects. Series remote-written from a tenant's nested +// cluster carry whatever namespace that cluster chose, so they are outside +// this guarantee for both values alike. func (c MetricsConfig) namespaceScope(ns string) []string { scope := []string{namespaceSelector(ns)} if c.PlatformNamespacePrefix != "" { From 8351e23d4d653c6460b42b0447bc906e5cb3e7a0 Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Wed, 11 Nov 2026 05:42:02 +0700 Subject: [PATCH 422/518] docs(observability-proxy): give the real reason for the prefix check The prefix is quoted before it reaches the regex, so a metacharacter or an upper case letter cannot widen the scope. Such a prefix matches no namespace at all and silently drops the exporter series, which is what the flag check guards against. Assisted-by: LLM Signed-off-by: Maxim Belyy --- cmd/observability-proxy/main.go | 4 ++-- cmd/observability-proxy/main_test.go | 3 +-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/cmd/observability-proxy/main.go b/cmd/observability-proxy/main.go index c527f5061..e6652ad5f 100644 --- a/cmd/observability-proxy/main.go +++ b/cmd/observability-proxy/main.go @@ -113,8 +113,8 @@ func parseFlags(args []string) (options, error) { return o, fmt.Errorf("invalid metrics store name %q", store) } } - // Anything but namespace name characters would make the prefix match - // differently from the namespaces it is meant to name. + // The prefix is matched literally, so one no namespace name can start + // with would turn the exporter branch off without a word. if p := o.metrics.PlatformNamespacePrefix; p != "" && len(validation.IsDNS1123Label(p+"x")) > 0 { return o, fmt.Errorf("invalid metrics platform namespace prefix %q", p) } diff --git a/cmd/observability-proxy/main_test.go b/cmd/observability-proxy/main_test.go index 13e137dd2..a08f15d9e 100644 --- a/cmd/observability-proxy/main_test.go +++ b/cmd/observability-proxy/main_test.go @@ -112,8 +112,7 @@ func TestParseFlagsInvalid(t *testing.T) { {"--metrics-max-range=0s"}, {"--metrics-max-points=1"}, {"--metrics-max-timeout=0s"}, - // Only namespace name characters: a regex metacharacter or an upper - // case letter would widen or empty the scope. + // No namespace name starts with these, so they would match nothing. {"--metrics-platform-namespace-prefix=Cozy-"}, {"--metrics-platform-namespace-prefix=cozy.*"}, {"--metrics-platform-namespace-prefix=-"}, From e7b5a5a1a9f049d91b4dede0fb66d088261dfe4f Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Tue, 10 Nov 2026 23:43:55 +0300 Subject: [PATCH 423/518] fix(seaweedfs): keep storage class names strings in the volume claims The upstream admin and volume StatefulSets write storageClassName unquoted, so a class name made only of digits, which Kubernetes allows and the parent chart now hands on as a string, turned back into a number there and the claim was rejected. The field is quoted, and left out when no class is named: quoting an empty value would set an empty storageClassName, which disables dynamic provisioning instead of using the cluster default. Upstream 4.52 and master still write both fields unquoted, so this is a chart patch, applied with --forward so make update stops once upstream carries an equivalent change. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- packages/system/seaweedfs/Makefile | 1 + .../templates/admin/admin-statefulset.yaml | 4 +- .../templates/volume/volume-statefulset.yaml | 4 +- .../patches/storage-class-string.patch | 30 +++++++++++++ .../seaweedfs/tests/lifecycle_test.yaml | 26 ++++++++++++ .../tests/volume_storage_class_test.yaml | 42 +++++++++++++++++++ 6 files changed, 105 insertions(+), 2 deletions(-) create mode 100644 packages/system/seaweedfs/patches/storage-class-string.patch create mode 100644 packages/system/seaweedfs/tests/volume_storage_class_test.yaml diff --git a/packages/system/seaweedfs/Makefile b/packages/system/seaweedfs/Makefile index 717ae0d49..54a9c0338 100644 --- a/packages/system/seaweedfs/Makefile +++ b/packages/system/seaweedfs/Makefile @@ -66,6 +66,7 @@ update: # fail here, where BSD patch would otherwise reverse it and remove the # section without an error. patch --forward --no-backup-if-mismatch -p4 < patches/s3-grpc-tls.patch + patch --forward --no-backup-if-mismatch -p4 < patches/storage-class-string.patch #patch --no-backup-if-mismatch -p4 < patches/retention-policy-delete.yaml $(MAKE) update-image diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml index ad112981f..b3d070651 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml @@ -330,7 +330,9 @@ spec: {{- end }} spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: {{ .Values.admin.data.storageClass }} + {{- with .Values.admin.data.storageClass }} + storageClassName: {{ . | quote }} + {{- end }} resources: requests: storage: {{ .Values.admin.data.size }} diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml index 04011d1d6..0702d703e 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml @@ -404,7 +404,9 @@ spec: {{- end }} spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: {{ $dir.storageClass }} + {{- with $dir.storageClass }} + storageClassName: {{ . | quote }} + {{- end }} resources: requests: storage: {{ $dir.size }} diff --git a/packages/system/seaweedfs/patches/storage-class-string.patch b/packages/system/seaweedfs/patches/storage-class-string.patch new file mode 100644 index 000000000..eed67258b --- /dev/null +++ b/packages/system/seaweedfs/patches/storage-class-string.patch @@ -0,0 +1,30 @@ +diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml +index ad112981f..b3d070651 100644 +--- a/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml ++++ b/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml +@@ -330,7 +330,9 @@ spec: + {{- end }} + spec: + accessModes: [ "ReadWriteOnce" ] +- storageClassName: {{ .Values.admin.data.storageClass }} ++ {{- with .Values.admin.data.storageClass }} ++ storageClassName: {{ . | quote }} ++ {{- end }} + resources: + requests: + storage: {{ .Values.admin.data.size }} +diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml +index 04011d1d6..0702d703e 100644 +--- a/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml ++++ b/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml +@@ -404,7 +404,9 @@ spec: + {{- end }} + spec: + accessModes: [ "ReadWriteOnce" ] +- storageClassName: {{ $dir.storageClass }} ++ {{- with $dir.storageClass }} ++ storageClassName: {{ . | quote }} ++ {{- end }} + resources: + requests: + storage: {{ $dir.size }} diff --git a/packages/system/seaweedfs/tests/lifecycle_test.yaml b/packages/system/seaweedfs/tests/lifecycle_test.yaml index 094e23804..7f3cec8ff 100644 --- a/packages/system/seaweedfs/tests/lifecycle_test.yaml +++ b/packages/system/seaweedfs/tests/lifecycle_test.yaml @@ -147,3 +147,29 @@ tests: asserts: - notExists: path: spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution + + # The parent hands the class on as a string, and a class name may be all + # digits; emitted unquoted it turns back into a number the claim rejects. + - it: keeps a numeric admin storage class name a string + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + set: + seaweedfs: + admin: + enabled: true + secret: + existingSecret: seaweedfs-admin-credentials + data: + storageClass: "123" + asserts: + - equal: + path: spec.volumeClaimTemplates[0].spec.storageClassName + value: "123" + + # An empty storageClassName disables dynamic provisioning; with no class + # named, the claim must leave the field out so the cluster default applies. + - it: leaves the admin claim on the cluster default when no class is named + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + set: *enabled + asserts: + - notExists: + path: spec.volumeClaimTemplates[0].spec.storageClassName diff --git a/packages/system/seaweedfs/tests/volume_storage_class_test.yaml b/packages/system/seaweedfs/tests/volume_storage_class_test.yaml new file mode 100644 index 000000000..037dc0969 --- /dev/null +++ b/packages/system/seaweedfs/tests/volume_storage_class_test.yaml @@ -0,0 +1,42 @@ +suite: seaweedfs volume claims keep the storage class a string + +templates: + - charts/seaweedfs/templates/volume/volume-statefulset.yaml + +release: + name: seaweedfs-system + namespace: tenant-root + +tests: + # The parent hands volume.storageClass on as a string, and a class name may + # be all digits; emitted unquoted it turns back into a number the claim + # rejects. + - it: keeps a numeric volume storage class name a string + set: + seaweedfs: + volume: + dataDirs: + - name: data1 + type: persistentVolumeClaim + size: 10Gi + storageClass: "123" + maxVolumes: 0 + asserts: + - equal: + path: spec.volumeClaimTemplates[0].spec.storageClassName + value: "123" + + # An empty storageClassName disables dynamic provisioning; with no class + # named, the claim must leave the field out so the cluster default applies. + - it: leaves the volume claim on the cluster default when no class is named + set: + seaweedfs: + volume: + dataDirs: + - name: data1 + type: persistentVolumeClaim + size: 10Gi + maxVolumes: 0 + asserts: + - notExists: + path: spec.volumeClaimTemplates[0].spec.storageClassName From f83db514b0eb88cefc2923d17aad59daccb4e358 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 10:29:22 +0300 Subject: [PATCH 424/518] fix(seaweedfs): quote every storage class the upstream claims carry The master, filer, volume index, log and all-in-one claims of the upstream chart write storageClassName unquoted, like the data claims did, so a class name made only of digits renders as a number and the claim is rejected. None of them is set by Cozystack today, but they are values of the same chart that a later change can start handing on, and the patch already exists for the data claims. They follow the same rule: quoted, and left out when no class is named so that the cluster default applies. The deprecated filer.enablePVC claim is left alone: it renders only with filer.data unset, and the template fails earlier on an unset filer.data, so it cannot be reached. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../templates/admin/admin-statefulset.yaml | 4 +- .../templates/all-in-one/all-in-one-pvc.yaml | 2 +- .../templates/filer/filer-statefulset.yaml | 8 +- .../templates/master/master-statefulset.yaml | 8 +- .../templates/volume/volume-statefulset.yaml | 8 +- .../patches/storage-class-string.patch | 102 ++++++++++++++- .../tests/storage_class_string_test.yaml | 123 ++++++++++++++++++ 7 files changed, 245 insertions(+), 10 deletions(-) create mode 100644 packages/system/seaweedfs/tests/storage_class_string_test.yaml diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml index b3d070651..1ce6f3cab 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml @@ -346,7 +346,9 @@ spec: {{- end }} spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: {{ .Values.admin.logs.storageClass }} + {{- with .Values.admin.logs.storageClass }} + storageClassName: {{ . | quote }} + {{- end }} resources: requests: storage: {{ .Values.admin.logs.size }} diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/all-in-one/all-in-one-pvc.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/all-in-one/all-in-one-pvc.yaml index 4a402f318..f5329a24e 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/all-in-one/all-in-one-pvc.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/all-in-one/all-in-one-pvc.yaml @@ -19,7 +19,7 @@ spec: accessModes: {{- toYaml (.Values.allInOne.data.accessModes | default (list "ReadWriteOnce")) | nindent 4 }} {{- if .Values.allInOne.data.storageClass }} - storageClassName: {{ .Values.allInOne.data.storageClass }} + storageClassName: {{ .Values.allInOne.data.storageClass | quote }} {{- end }} resources: requests: diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/filer/filer-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/filer/filer-statefulset.yaml index e505faf1c..ced2d597a 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/filer/filer-statefulset.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/filer/filer-statefulset.yaml @@ -451,7 +451,9 @@ spec: {{- end }} spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: {{ .Values.filer.data.storageClass }} + {{- with .Values.filer.data.storageClass }} + storageClassName: {{ . | quote }} + {{- end }} resources: requests: storage: {{ .Values.filer.data.size }} @@ -467,7 +469,9 @@ spec: {{- end }} spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: {{ .Values.filer.logs.storageClass }} + {{- with .Values.filer.logs.storageClass }} + storageClassName: {{ . | quote }} + {{- end }} resources: requests: storage: {{ .Values.filer.logs.size }} diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/master/master-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/master/master-statefulset.yaml index 914f0dfd1..5df83e687 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/master/master-statefulset.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/master/master-statefulset.yaml @@ -336,7 +336,9 @@ spec: {{- end }} spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: {{ .Values.master.data.storageClass }} + {{- with .Values.master.data.storageClass }} + storageClassName: {{ . | quote }} + {{- end }} resources: requests: storage: {{ .Values.master.data.size }} @@ -352,7 +354,9 @@ spec: {{- end }} spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: {{ .Values.master.logs.storageClass }} + {{- with .Values.master.logs.storageClass }} + storageClassName: {{ . | quote }} + {{- end }} resources: requests: storage: {{ .Values.master.logs.size }} diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml index 0702d703e..88b13822d 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml @@ -424,7 +424,9 @@ spec: {{- end }} spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: {{ $volume.idx.storageClass }} + {{- with $volume.idx.storageClass }} + storageClassName: {{ . | quote }} + {{- end }} resources: requests: storage: {{ $volume.idx.size }} @@ -440,7 +442,9 @@ spec: {{- end }} spec: accessModes: [ "ReadWriteOnce" ] - storageClassName: {{ $volume.logs.storageClass }} + {{- with $volume.logs.storageClass }} + storageClassName: {{ . | quote }} + {{- end }} resources: requests: storage: {{ $volume.logs.size }} diff --git a/packages/system/seaweedfs/patches/storage-class-string.patch b/packages/system/seaweedfs/patches/storage-class-string.patch index eed67258b..0c4562105 100644 --- a/packages/system/seaweedfs/patches/storage-class-string.patch +++ b/packages/system/seaweedfs/patches/storage-class-string.patch @@ -1,5 +1,5 @@ diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml -index ad112981f..b3d070651 100644 +index ad112981f..1ce6f3cab 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/admin/admin-statefulset.yaml @@ -330,7 +330,9 @@ spec: @@ -13,8 +13,84 @@ index ad112981f..b3d070651 100644 resources: requests: storage: {{ .Values.admin.data.size }} +@@ -344,7 +346,9 @@ spec: + {{- end }} + spec: + accessModes: [ "ReadWriteOnce" ] +- storageClassName: {{ .Values.admin.logs.storageClass }} ++ {{- with .Values.admin.logs.storageClass }} ++ storageClassName: {{ . | quote }} ++ {{- end }} + resources: + requests: + storage: {{ .Values.admin.logs.size }} +diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/all-in-one/all-in-one-pvc.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/all-in-one/all-in-one-pvc.yaml +index 4a402f318..f5329a24e 100644 +--- a/packages/system/seaweedfs/charts/seaweedfs/templates/all-in-one/all-in-one-pvc.yaml ++++ b/packages/system/seaweedfs/charts/seaweedfs/templates/all-in-one/all-in-one-pvc.yaml +@@ -19,7 +19,7 @@ spec: + accessModes: + {{- toYaml (.Values.allInOne.data.accessModes | default (list "ReadWriteOnce")) | nindent 4 }} + {{- if .Values.allInOne.data.storageClass }} +- storageClassName: {{ .Values.allInOne.data.storageClass }} ++ storageClassName: {{ .Values.allInOne.data.storageClass | quote }} + {{- end }} + resources: + requests: +diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/filer/filer-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/filer/filer-statefulset.yaml +index e505faf1c..ced2d597a 100644 +--- a/packages/system/seaweedfs/charts/seaweedfs/templates/filer/filer-statefulset.yaml ++++ b/packages/system/seaweedfs/charts/seaweedfs/templates/filer/filer-statefulset.yaml +@@ -451,7 +451,9 @@ spec: + {{- end }} + spec: + accessModes: [ "ReadWriteOnce" ] +- storageClassName: {{ .Values.filer.data.storageClass }} ++ {{- with .Values.filer.data.storageClass }} ++ storageClassName: {{ . | quote }} ++ {{- end }} + resources: + requests: + storage: {{ .Values.filer.data.size }} +@@ -467,7 +469,9 @@ spec: + {{- end }} + spec: + accessModes: [ "ReadWriteOnce" ] +- storageClassName: {{ .Values.filer.logs.storageClass }} ++ {{- with .Values.filer.logs.storageClass }} ++ storageClassName: {{ . | quote }} ++ {{- end }} + resources: + requests: + storage: {{ .Values.filer.logs.size }} +diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/master/master-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/master/master-statefulset.yaml +index 914f0dfd1..5df83e687 100644 +--- a/packages/system/seaweedfs/charts/seaweedfs/templates/master/master-statefulset.yaml ++++ b/packages/system/seaweedfs/charts/seaweedfs/templates/master/master-statefulset.yaml +@@ -336,7 +336,9 @@ spec: + {{- end }} + spec: + accessModes: [ "ReadWriteOnce" ] +- storageClassName: {{ .Values.master.data.storageClass }} ++ {{- with .Values.master.data.storageClass }} ++ storageClassName: {{ . | quote }} ++ {{- end }} + resources: + requests: + storage: {{ .Values.master.data.size }} +@@ -352,7 +354,9 @@ spec: + {{- end }} + spec: + accessModes: [ "ReadWriteOnce" ] +- storageClassName: {{ .Values.master.logs.storageClass }} ++ {{- with .Values.master.logs.storageClass }} ++ storageClassName: {{ . | quote }} ++ {{- end }} + resources: + requests: + storage: {{ .Values.master.logs.size }} diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml -index 04011d1d6..0702d703e 100644 +index 04011d1d6..88b13822d 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/volume/volume-statefulset.yaml @@ -404,7 +404,9 @@ spec: @@ -28,3 +104,25 @@ index 04011d1d6..0702d703e 100644 resources: requests: storage: {{ $dir.size }} +@@ -422,7 +424,9 @@ spec: + {{- end }} + spec: + accessModes: [ "ReadWriteOnce" ] +- storageClassName: {{ $volume.idx.storageClass }} ++ {{- with $volume.idx.storageClass }} ++ storageClassName: {{ . | quote }} ++ {{- end }} + resources: + requests: + storage: {{ $volume.idx.size }} +@@ -438,7 +442,9 @@ spec: + {{- end }} + spec: + accessModes: [ "ReadWriteOnce" ] +- storageClassName: {{ $volume.logs.storageClass }} ++ {{- with $volume.logs.storageClass }} ++ storageClassName: {{ . | quote }} ++ {{- end }} + resources: + requests: + storage: {{ $volume.logs.size }} diff --git a/packages/system/seaweedfs/tests/storage_class_string_test.yaml b/packages/system/seaweedfs/tests/storage_class_string_test.yaml new file mode 100644 index 000000000..038cf1fed --- /dev/null +++ b/packages/system/seaweedfs/tests/storage_class_string_test.yaml @@ -0,0 +1,123 @@ +suite: seaweedfs claims keep the storage class a string + +# A StorageClass name may be all digits. Emitted unquoted it renders as a +# number, which the claim rejects; and an empty storageClassName disables +# dynamic provisioning, so with no class named the field has to stay out for +# the cluster default to apply. + +templates: + - charts/seaweedfs/templates/master/master-statefulset.yaml + - charts/seaweedfs/templates/filer/filer-statefulset.yaml + - charts/seaweedfs/templates/volume/volume-statefulset.yaml + - charts/seaweedfs/templates/admin/admin-statefulset.yaml + - charts/seaweedfs/templates/all-in-one/all-in-one-pvc.yaml + # filer-statefulset includes it for its checksum annotation. + - charts/seaweedfs/templates/s3/s3-secret.yaml + +release: + name: seaweedfs-system + namespace: tenant-root + +tests: + - it: keeps numeric master data and logs storage class names strings + template: charts/seaweedfs/templates/master/master-statefulset.yaml + set: + seaweedfs.master.data: {type: persistentVolumeClaim, size: 1Gi, storageClass: "123"} + seaweedfs.master.logs: {type: persistentVolumeClaim, size: 1Gi, storageClass: "456"} + asserts: + - equal: + path: spec.volumeClaimTemplates[?(@.metadata.name == "data-tenant-root")].spec.storageClassName + value: "123" + - equal: + path: spec.volumeClaimTemplates[?(@.metadata.name == "seaweedfs-master-log-volume")].spec.storageClassName + value: "456" + + - it: leaves the master claims on the cluster default when no class is named + template: charts/seaweedfs/templates/master/master-statefulset.yaml + set: + seaweedfs.master.data: {type: persistentVolumeClaim, size: 1Gi} + seaweedfs.master.logs: {type: persistentVolumeClaim, size: 1Gi} + asserts: + - notExists: + path: spec.volumeClaimTemplates[0].spec.storageClassName + - notExists: + path: spec.volumeClaimTemplates[1].spec.storageClassName + + - it: keeps numeric filer data and logs storage class names strings + template: charts/seaweedfs/templates/filer/filer-statefulset.yaml + set: + seaweedfs.filer.data: {type: persistentVolumeClaim, size: 1Gi, storageClass: "123"} + seaweedfs.filer.logs: {type: persistentVolumeClaim, size: 1Gi, storageClass: "456"} + asserts: + - equal: + path: spec.volumeClaimTemplates[0].spec.storageClassName + value: "123" + - equal: + path: spec.volumeClaimTemplates[1].spec.storageClassName + value: "456" + + - it: leaves the filer claims on the cluster default when no class is named + template: charts/seaweedfs/templates/filer/filer-statefulset.yaml + set: + seaweedfs.filer.data: {type: persistentVolumeClaim, size: 1Gi} + seaweedfs.filer.logs: {type: persistentVolumeClaim, size: 1Gi} + asserts: + - notExists: + path: spec.volumeClaimTemplates[0].spec.storageClassName + - notExists: + path: spec.volumeClaimTemplates[1].spec.storageClassName + + - it: keeps numeric volume idx and logs storage class names strings + template: charts/seaweedfs/templates/volume/volume-statefulset.yaml + set: + seaweedfs.volume.dataDirs: [] + seaweedfs.volume.idx: {type: persistentVolumeClaim, size: 1Gi, storageClass: "123"} + seaweedfs.volume.logs: {type: persistentVolumeClaim, size: 1Gi, storageClass: "456"} + asserts: + - equal: + path: spec.volumeClaimTemplates[0].spec.storageClassName + value: "123" + - equal: + path: spec.volumeClaimTemplates[1].spec.storageClassName + value: "456" + + - it: leaves the volume idx and logs claims on the cluster default when no class is named + template: charts/seaweedfs/templates/volume/volume-statefulset.yaml + set: + seaweedfs.volume.dataDirs: [] + seaweedfs.volume.idx: {type: persistentVolumeClaim, size: 1Gi} + seaweedfs.volume.logs: {type: persistentVolumeClaim, size: 1Gi} + asserts: + - notExists: + path: spec.volumeClaimTemplates[0].spec.storageClassName + - notExists: + path: spec.volumeClaimTemplates[1].spec.storageClassName + + - it: keeps a numeric admin logs storage class name a string + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + set: + seaweedfs.admin.enabled: true + seaweedfs.admin.logs: {type: persistentVolumeClaim, size: 1Gi, storageClass: "456"} + asserts: + - equal: + path: spec.volumeClaimTemplates[1].spec.storageClassName + value: "456" + + - it: leaves the admin logs claim on the cluster default when no class is named + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + set: + seaweedfs.admin.enabled: true + seaweedfs.admin.logs: {type: persistentVolumeClaim, size: 1Gi} + asserts: + - notExists: + path: spec.volumeClaimTemplates[1].spec.storageClassName + + - it: keeps a numeric all-in-one storage class name a string + template: charts/seaweedfs/templates/all-in-one/all-in-one-pvc.yaml + set: + seaweedfs.allInOne.enabled: true + seaweedfs.allInOne.data: {type: persistentVolumeClaim, size: 1Gi, storageClass: "123"} + asserts: + - equal: + path: spec.storageClassName + value: "123" From e6eb3345c60f2eaec19f73e4b24a47e55ea2fef0 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 10:29:48 +0300 Subject: [PATCH 425/518] fix(seaweedfs): escape pool and zone storage classes in the release values Pool and zone storage classes were wrapped in quotes by hand, which does not escape a quote or a backslash in the value: a quote closes the string early and the data-plane release values stop parsing, with a YAML error that does not name the field. They are now quoted like the volume and admin classes. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../extra/seaweedfs/templates/seaweedfs.yaml | 6 +- .../tests/volume_storage_class_test.yaml | 55 +++++++++++++++++++ 2 files changed, 58 insertions(+), 3 deletions(-) diff --git a/packages/extra/seaweedfs/templates/seaweedfs.yaml b/packages/extra/seaweedfs/templates/seaweedfs.yaml index 52009724b..9a6e952e5 100644 --- a/packages/extra/seaweedfs/templates/seaweedfs.yaml +++ b/packages/extra/seaweedfs/templates/seaweedfs.yaml @@ -279,7 +279,7 @@ spec: type: "persistentVolumeClaim" size: "{{ $pool.size | default $.Values.volume.size }}" {{- with ($pool.storageClass | default $.Values.volume.storageClass) }} - storageClass: "{{ . }}" + storageClass: {{ . | quote }} {{- end }} maxVolumes: 0 extraArgs: @@ -295,7 +295,7 @@ spec: type: "persistentVolumeClaim" size: "{{ $zone.size | default $.Values.volume.size }}" {{- with ($zone.storageClass | default $.Values.volume.storageClass) }} - storageClass: "{{ . }}" + storageClass: {{ . | quote }} {{- end }} maxVolumes: 0 nodeSelector: | @@ -320,7 +320,7 @@ spec: type: "persistentVolumeClaim" size: "{{ $pool.size | default $zone.size | default $.Values.volume.size }}" {{- with ($pool.storageClass | default $zone.storageClass | default $.Values.volume.storageClass) }} - storageClass: "{{ . }}" + storageClass: {{ . | quote }} {{- end }} maxVolumes: 0 nodeSelector: | diff --git a/packages/extra/seaweedfs/tests/volume_storage_class_test.yaml b/packages/extra/seaweedfs/tests/volume_storage_class_test.yaml index 6543afc98..2474f3358 100644 --- a/packages/extra/seaweedfs/tests/volume_storage_class_test.yaml +++ b/packages/extra/seaweedfs/tests/volume_storage_class_test.yaml @@ -45,3 +45,58 @@ tests: value: "local\n image: example/other" - notExists: path: spec.values.seaweedfs.volume.dataDirs[0].image + + # A pool or zone class is quoted by hand around the value, which a quote + # character inside it closes early, breaking the release values. + - it: keeps a pool storage class with a quote inside its own field + set: + <<: *platform + volume: + pools: + fast: + diskType: ssd + storageClass: 'local"x' + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - equal: + path: spec.values.seaweedfs.volumes.fast.dataDirs[0].storageClass + value: 'local"x' + + - it: keeps a zone storage class with a quote inside its own field + set: + <<: *platform + topology: MultiZone + replicationFactor: 1 + volume: + zones: + a: + storageClass: 'local"x' + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - equal: + path: spec.values.seaweedfs.volumes.a.dataDirs[0].storageClass + value: 'local"x' + + - it: keeps a zone pool storage class with a quote inside its own field + set: + <<: *platform + topology: MultiZone + replicationFactor: 1 + volume: + zones: + a: + pools: + fast: + diskType: ssd + storageClass: 'local"x' + documentSelector: + path: metadata.name + value: seaweedfs-system + asserts: + - equal: + path: spec.values.seaweedfs.volumes.a-fast.dataDirs[0].storageClass + value: 'local"x' From f5114ae04c42cd68b24b6bd8ce22d6ce68463893 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 10:30:11 +0300 Subject: [PATCH 426/518] fix(seaweedfs): keep the metadata database storage class a string db.storageClass went unquoted into the values of the database release and again into its CNPG Cluster. A class name made only of digits, which Kubernetes allows, became a number that the Cluster rejects, and a line break in the value could add keys to the database release values. It is quoted at both steps. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../seaweedfs/templates/seaweedfs-db.yaml | 2 +- .../tests/db_storage_class_test.yaml | 30 +++++++++++++++++++ .../seaweedfs-db/templates/database.yaml | 2 +- .../seaweedfs-db/tests/database_test.yaml | 14 +++++++++ 4 files changed, 46 insertions(+), 2 deletions(-) create mode 100644 packages/extra/seaweedfs/tests/db_storage_class_test.yaml diff --git a/packages/extra/seaweedfs/templates/seaweedfs-db.yaml b/packages/extra/seaweedfs/templates/seaweedfs-db.yaml index c86677c4d..261fcc93e 100644 --- a/packages/extra/seaweedfs/templates/seaweedfs-db.yaml +++ b/packages/extra/seaweedfs/templates/seaweedfs-db.yaml @@ -42,6 +42,6 @@ spec: resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.db.resourcesPreset .Values.db.resources $) | nindent 8 }} size: {{ .Values.db.size }} {{- with .Values.db.storageClass }} - storageClass: {{ . }} + storageClass: {{ . | quote }} {{- end }} {{- end }} diff --git a/packages/extra/seaweedfs/tests/db_storage_class_test.yaml b/packages/extra/seaweedfs/tests/db_storage_class_test.yaml new file mode 100644 index 000000000..da4f6cd2d --- /dev/null +++ b/packages/extra/seaweedfs/tests/db_storage_class_test.yaml @@ -0,0 +1,30 @@ +suite: seaweedfs metadata database storage class reaches its release as a string + +templates: + - templates/seaweedfs-db.yaml + +release: + name: seaweedfs + namespace: tenant-root + +tests: + # A StorageClass name may be all digits; handed on unquoted it reaches the + # database release as a number, and a line break in it would open new keys + # in that release's values. + - it: hands a numeric storage class name on as a string + set: + db: + storageClass: "123" + asserts: + - equal: + path: spec.values.db.storageClass + value: "123" + + - it: keeps a storage class with a line break inside its own field + set: + db: + storageClass: "local\n replicas: 9" + asserts: + - equal: + path: spec.values.db.storageClass + value: "local\n replicas: 9" diff --git a/packages/system/seaweedfs-db/templates/database.yaml b/packages/system/seaweedfs-db/templates/database.yaml index 807bd13af..136a3be83 100644 --- a/packages/system/seaweedfs-db/templates/database.yaml +++ b/packages/system/seaweedfs-db/templates/database.yaml @@ -11,7 +11,7 @@ spec: storage: size: {{ .Values.db.size }} {{- with .Values.db.storageClass }} - storageClass: {{ . }} + storageClass: {{ . | quote }} {{- end }} monitoring: diff --git a/packages/system/seaweedfs-db/tests/database_test.yaml b/packages/system/seaweedfs-db/tests/database_test.yaml index 3eff5a112..2e6244cf1 100644 --- a/packages/system/seaweedfs-db/tests/database_test.yaml +++ b/packages/system/seaweedfs-db/tests/database_test.yaml @@ -60,3 +60,17 @@ tests: - equal: path: spec.inheritedMetadata.labels["policy.cozystack.io/allow-to-apiserver"] value: "true" + + # A StorageClass name may be all digits; emitted unquoted it renders as a + # number, which the Cluster rejects. + - it: keeps a numeric storage class name a string + release: + name: seaweedfs-db + namespace: cozy-seaweedfs + set: + db: + storageClass: "123" + asserts: + - equal: + path: spec.storage.storageClass + value: "123" From f1dd2d17fc864ac5b8a7417fdb3d272b0fbc5c57 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Wed, 11 Nov 2026 09:20:19 +0100 Subject: [PATCH 427/518] fix(backups): give the barman-cloud sidecar an explicit CPU limit The sidecar was left without a CPU limit so that a throttled sidecar would slow WAL archiving down rather than fail it. In a tenant with resourceQuotas that does not hold: the tenant LimitRange gives every container a 250m CPU limit by default, so the sidecar gets it and compresses full backups and WAL segments on a quarter of a core. A 1.6 GB database took 30 minutes to back up. Set an explicit one-core limit in both places that build the sidecar configuration, the backup driver and cozy-lib, the same way the memory request and limit already override the LimitRange defaults. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- internal/backupcontroller/cnpgstrategy_controller.go | 10 +++++++--- .../backupcontroller/cnpgstrategy_controller_test.go | 7 +++++-- .../apps/postgres/tests/barman_resources_test.yaml | 8 ++------ packages/library/cozy-lib/templates/_barman.tpl | 11 +++++++---- 4 files changed, 21 insertions(+), 15 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 19f255c6c..3e2f05c0d 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -2224,9 +2224,12 @@ func buildBarmanPlugin(objectStoreName, serverName string) cnpgtypes.PluginConfi // high-water mark on a 38 MB database is 254 MiB, so 128Mi cannot hold it. A // tenant that leaves resourceQuotas empty has no LimitRange, and there the // sidecar had no requests and no limit at all. 256Mi holds the measured working -// set and 1Gi is four times the measurement. No CPU limit is set, matching the -// entityOperator precedent in packages/apps/kafka: a throttled sidecar stalls -// WAL archiving instead of failing it. +// set and 1Gi is four times the measurement. The CPU limit is explicit for the +// same reason: left unset, that LimitRange gives the sidecar its 250m default, +// so gzip compression of full backups and WAL segments runs on a quarter of a +// core (a 1.6 GB database took 30 minutes to back up). One core keeps the +// sidecar from starving the instance while removing that ceiling; it counts +// against the tenant's limits.cpu budget, one core per instance pod. // // Checksum: since botocore ~1.36 the default (when_supported) attaches a // flexible checksum to every PutObject, which non-AWS S3-compatible backends @@ -2246,6 +2249,7 @@ func barmanSidecarConfiguration() *cnpgtypes.InstanceSidecarConfiguration { corev1.ResourceMemory: resource.MustParse("256Mi"), }, Limits: corev1.ResourceList{ + corev1.ResourceCPU: resource.MustParse("1"), corev1.ResourceMemory: resource.MustParse("1Gi"), }, }, diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 551209fac..bc2827371 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2152,8 +2152,11 @@ func TestApplyClusterPluginBackup_PatchesExistingCluster(t *testing.T) { sc.Resources.Requests.Cpu().MilliValue() != 100 { t.Errorf("ObjectStore leaves Barman subject to undersized tenant resource defaults: %+v", sc.Resources) } - if _, ok := sc.Resources.Limits[corev1.ResourceCPU]; ok { - t.Errorf("ObjectStore caps sidecar CPU, which throttles WAL archiving instead of failing it: %+v", sc.Resources.Limits) + // Without an explicit CPU limit the tenant LimitRange caps the sidecar at + // its 250m default, and every backup and WAL upload compresses on a + // quarter of a core. + if sc.Resources.Limits.Cpu().MilliValue() != 1000 { + t.Errorf("ObjectStore leaves the sidecar CPU limit to the tenant LimitRange default: %+v", sc.Resources.Limits) } // The ObjectStore must be owner-referenced to the Cluster so Kubernetes GC // removes it when the Cluster is deleted (no orphan in the platform flow, diff --git a/packages/apps/postgres/tests/barman_resources_test.yaml b/packages/apps/postgres/tests/barman_resources_test.yaml index 905fde2bb..1903fe8b1 100644 --- a/packages/apps/postgres/tests/barman_resources_test.yaml +++ b/packages/apps/postgres/tests/barman_resources_test.yaml @@ -13,7 +13,7 @@ set: s3CredentialsSecret: name: postgres-backup tests: - - it: sets resources for native backup without inheriting the tenant 128Mi limit + - it: sets resources for native backup without inheriting the tenant 128Mi and 250m limits set: backup.enabled: true asserts: @@ -24,15 +24,11 @@ tests: cpu: 100m memory: 256Mi limits: + cpu: "1" memory: 1Gi documentSelector: path: kind value: ObjectStore - - notExists: - path: spec.instanceSidecarConfiguration.resources.limits.cpu - documentSelector: - path: kind - value: ObjectStore - equal: path: spec.configuration.s3Credentials.accessKeyId.name value: postgres-backup diff --git a/packages/library/cozy-lib/templates/_barman.tpl b/packages/library/cozy-lib/templates/_barman.tpl index c37b9673f..3b187dda6 100644 --- a/packages/library/cozy-lib/templates/_barman.tpl +++ b/packages/library/cozy-lib/templates/_barman.tpl @@ -13,10 +13,12 @@ LimitRange, which is cozy-keycloak and any tenant that leaves resourceQuotas empty, gave the sidecar no requests and no limit at all, so nothing reserved memory for it and nothing bounded it. The same request and limit settle both: 256Mi holds the measured working set and 1Gi is four times the measurement. -No CPU limit is set, matching the entityOperator precedent in -packages/apps/kafka: a throttled sidecar stalls WAL archiving instead of -failing it. Inside a tenant the limit counts against the ResourceQuota -limits.memory budget, 1Gi per instance pod. +The CPU limit is explicit for the same reason: left unset, that LimitRange +gives the sidecar its 250m default, so gzip compression of full backups and +WAL segments runs on a quarter of a core (a 1.6 GB database took 30 minutes +to back up). One core keeps the sidecar from starving the instance while +removing that ceiling. Inside a tenant both limits count against the +ResourceQuota budget, one core and 1Gi per instance pod. Checksum: since botocore ~1.36 (early 2025) the default RequestChecksumCalculation is when_supported, which attaches a flexible checksum @@ -39,6 +41,7 @@ instanceSidecarConfiguration: cpu: 100m memory: 256Mi limits: + cpu: "1" memory: 1Gi env: - name: AWS_REQUEST_CHECKSUM_CALCULATION From fb5ac1826a54f7176504187544f76edab00775c4 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Wed, 11 Nov 2026 12:12:11 +0300 Subject: [PATCH 428/518] test(observability-proxy): prepare seed stubs in both unit runners The POSIX runner does not invoke Bats setup hooks. Initialize the capture stub from the test body so the compatibility lane stays offline. Fixes: 5a2582e21cad ("test(observability-proxy): preserve JSON seed request bodies") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- hack/observability-proxy-seed.bats | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/hack/observability-proxy-seed.bats b/hack/observability-proxy-seed.bats index 8815b0db5..26caf2091 100644 --- a/hack/observability-proxy-seed.bats +++ b/hack/observability-proxy-seed.bats @@ -4,13 +4,14 @@ load test_helper REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)" -setup() { +prepare_seed_capture() { strict_setup - mkdir -p "$BATS_TEST_TMPDIR/bin" - export SEED_CAPTURE="$BATS_TEST_TMPDIR/seed" - export TMPDIR="$BATS_TEST_TMPDIR" - export PATH="$BATS_TEST_TMPDIR/bin:$PATH" - cat > "$BATS_TEST_TMPDIR/bin/curl" <<'EOF' + seed_tmp=$(mktemp -d) + seed_old_path=$PATH + mkdir -p "$seed_tmp/bin" + export SEED_CAPTURE="$seed_tmp/seed" + export PATH="$seed_tmp/bin:$PATH" + cat > "$seed_tmp/bin/curl" <<'EOF' #!/usr/bin/env bash set -eu printf '%s\n' "$@" > "$SEED_CAPTURE.args" @@ -20,11 +21,12 @@ if ! grep --fixed-strings --line-regexp --quiet 'Content-Type: application/json' exit 22 fi EOF - chmod +x "$BATS_TEST_TMPDIR/bin/curl" + chmod +x "$seed_tmp/bin/curl" } assert_seed_request() { local account=$1 + prepare_seed_capture # shellcheck source=e2e-chainsaw/observability-proxy/lib.sh . "$REPO_ROOT/hack/e2e-chainsaw/observability-proxy/lib.sh" if [ "$account" = 0 ]; then @@ -39,7 +41,9 @@ assert_seed_request() { grep --fixed-strings --line-regexp --quiet '@-' "$SEED_CAPTURE.args" printf '%s\n' '{"kubernetes_namespace_name":"tenant-test-logsa","log":"e2e-marker-a"}' > "$SEED_CAPTURE.expected" cmp "$SEED_CAPTURE.expected" "$SEED_CAPTURE.body" - rm "$body" + rm "$body" "$SEED_CAPTURE.args" "$SEED_CAPTURE.body" "$SEED_CAPTURE.expected" "$seed_tmp/bin/curl" + rmdir "$seed_tmp/bin" "$seed_tmp" + export PATH="$seed_old_path" } @test "observability seeds send JSON without consuming the body as form data" { From c26d2773a51d0ca471bd461c6a343a12a42c14a1 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Mon, 9 Nov 2026 14:34:09 +0300 Subject: [PATCH 429/518] test(clickhouse): name what really leaves the round-trip's Secret behind The projected Secret the preconditions delete is left by whichever BackupJob ran in tenant-root before this Test, from this suite or another: Chainsaw does not fix the order of the Tests in a suite, so no particular one can be named. The restore budget is also not spent on a credentials hold: by then the stamp is older than the backup's own two-minute wait. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index e67b2674a..3c41dbdd3 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -393,9 +393,11 @@ spec: # bucket's SeaweedFS and its Ingress controller sit. # # The precondition step deletes tenant-root's projected cozy-backups-creds, -# which the mongodb and rabbitmq round-trips leave there after their -# cozy-default BackupJobs. Nothing reads it between suites, and the projector -# writes it again on the next BackupJob, which is the moment under test. +# which any BackupJob that ran in tenant-root before this Test leaves there: +# the projector runs for each BackupJob whatever its strategy, and Chainsaw +# does not fix the order of the Tests in a suite, so the earlier one may come +# from this suite or another. It writes the Secret again on the next +# BackupJob, which is the moment under test. apiVersion: chainsaw.kyverno.io/v1alpha1 kind: Test metadata: @@ -547,7 +549,8 @@ spec: # the Ingress serving the Test certificate, plus one last 30s # attempt and its 10s pause # 480s BackupJob to Succeeded, including the driver's 2m hold - # 480s RestoreJob to Succeeded, same hold + # 480s RestoreJob to Succeeded; no hold by then, the stamp is + # already older than the backup's own 2m wait # 90s the EXIT trap: Corefile replace plus a 60s Secret delete # 1330s (22m10s) # 25m leaves 2m50s for the SQL and kubectl calls between those waits, so From 8ba0380043c0eaaa4c01dcee8ae205702ed3f754 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 14:42:49 +0300 Subject: [PATCH 430/518] docs(clickhouse): name the clickhouse-backup handlers that reload the config clickhouse-backup reloads its config file in each handler that reaches S3, among them the create_remote, restore_remote and list calls the strategy makes. The status and action-log endpoints the strategy polls read in-memory state and do not reload it, so the mounted file reaches the sidecar on the calls that need credentials rather than on every request. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/credentials_projector.go | 7 ++++--- .../backupcontroller/credentials_projector_test.go | 3 ++- packages/apps/clickhouse/templates/clickhouse.yaml | 12 ++++++------ packages/apps/clickhouse/tests/backup_test.yaml | 7 ++++--- 4 files changed, 16 insertions(+), 13 deletions(-) diff --git a/internal/backupcontroller/credentials_projector.go b/internal/backupcontroller/credentials_projector.go index 576991d86..aaba89b3c 100644 --- a/internal/backupcontroller/credentials_projector.go +++ b/internal/backupcontroller/credentials_projector.go @@ -450,9 +450,10 @@ const credentialsUpdatedAtAnnotation = "backups.cozystack.io/credentials-updated // clickHouseBackupConfigKey holds the s3 section of a clickhouse-backup // config. The ClickHouse chart mounts it as the sidecar's config file // instead of passing the values as environment: clickhouse-backup reloads -// the file on every API call and kubelet rewrites a mounted Secret in -// place, so credentials projected after the pod started, or rotated since, -// reach a running sidecar. +// the file in each API handler that reaches S3, among them the +// create_remote, restore_remote and list calls the strategy makes, and +// kubelet rewrites a mounted Secret in place, so credentials projected +// after the pod started, or rotated since, reach a running sidecar. const clickHouseBackupConfigKey = "clickhouse-backup.yml" // buildClickHouseBackupConfig renders that file. The projected endpoint is diff --git a/internal/backupcontroller/credentials_projector_test.go b/internal/backupcontroller/credentials_projector_test.go index dd3cf131b..74fd936d1 100644 --- a/internal/backupcontroller/credentials_projector_test.go +++ b/internal/backupcontroller/credentials_projector_test.go @@ -673,7 +673,8 @@ func TestProject_HappyPath_HTTPEndpoint(t *testing.T) { // The ClickHouse sidecar reads its S3 settings from this key as a mounted // file, which kubelet refreshes in place: clickhouse-backup reloads its -// config file on every API call, while its environment is fixed at start. +// config file in each API handler that reaches S3, while its environment +// is fixed at start. func TestProject_WritesClickHouseBackupConfig(t *testing.T) { cases := []struct { name string diff --git a/packages/apps/clickhouse/templates/clickhouse.yaml b/packages/apps/clickhouse/templates/clickhouse.yaml index 4934e8db1..5d0c1c6ab 100644 --- a/packages/apps/clickhouse/templates/clickhouse.yaml +++ b/packages/apps/clickhouse/templates/clickhouse.yaml @@ -230,12 +230,12 @@ spec: backupStorage.forcePathStyle included, comes from the clickhouse-backup.yml key the platform projects into cozy-backups-creds, mounted below as the config file. - clickhouse-backup reloads that file on every API call and - kubelet rewrites a mounted Secret in place, so credentials - projected after the Pod started, or rotated since, reach - the running sidecar; environment would be read only at - start and would override the file, so none of those is - set here. + clickhouse-backup reloads that file in each API handler + that reaches S3 and kubelet rewrites a mounted Secret in + place, so credentials projected after the Pod started, or + rotated since, reach the running sidecar; environment + would be read only at start and would override the file, + so none of those is set here. S3_FORCE_PATH_STYLE selects path-style vs vhost-style S3 addressing. On the legacy paths (own Secret via diff --git a/packages/apps/clickhouse/tests/backup_test.yaml b/packages/apps/clickhouse/tests/backup_test.yaml index 7448e02a3..7954d5d3d 100644 --- a/packages/apps/clickhouse/tests/backup_test.yaml +++ b/packages/apps/clickhouse/tests/backup_test.yaml @@ -47,9 +47,10 @@ tests: - equal: path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="S3_PATH")].value value: tenant-test/ch-test - # clickhouse-backup reloads this file on every API call and kubelet - # rewrites the mounted Secret in place, so credentials projected after - # the Pod started, or rotated since, reach the running sidecar. + # clickhouse-backup reloads this file in each API handler that reaches + # S3 and kubelet rewrites the mounted Secret in place, so credentials + # projected after the Pod started, or rotated since, reach the running + # sidecar. - equal: path: spec.templates.podTemplates[0].spec.containers[?(@.name=="clickhouse-backup")].env[?(@.name=="CLICKHOUSE_BACKUP_CONFIG")].value value: /etc/clickhouse-backup-s3/config.yml From 35823c6c9c4d960f6417ad6666f8d2ca8b59b996 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 14:56:26 +0300 Subject: [PATCH 431/518] test(clickhouse): list the system-bucket round-trip in the suite header The header names the backup Tests of the suite and why they are split; it still described two of them after the third was added, so a reader starting from it would miss the only Test that covers the system-bucket path. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml index 3c41dbdd3..e6253c06a 100644 --- a/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/clickhouse/chainsaw-test.yaml @@ -91,7 +91,7 @@ spec: status: replicas: 1 --- -# ClickHouse Altinity backup/restore E2E — the two backup Tests below sit in +# ClickHouse Altinity backup/restore E2E — the three backup Tests below sit in # this suite alongside the plain `clickhouse` Test above. Chainsaw port of the # intended hack/e2e-apps/backup-clickhouse.bats (cozystack#2600), shaped like the # etcd/mariadb/postgres suites: @@ -107,8 +107,15 @@ spec: # driving the validated example scripts # (examples/backups/clickhouse). Also runs in # CI, in tenant-root. +# clickhouse-3-system-bucket-roundtrip +# — backup and in-place restore through the +# shipped cozy-default BackupClass and the +# platform system bucket, for a release +# whose Pod started before its credentials +# were projected. Runs in CI, in tenant-root. # -# Why the split: the two Tests answer different questions at different cost. The +# Why the split: the first two Tests answer different questions at different +# cost, and the third covers the system-bucket path neither of them takes. The # contracts Test needs no object store, so it stays in the default tenant-test # namespace next to the other app suites and is the seconds-long guard on the # chart wiring the Altinity strategy depends on — the sidecar container/port and From 320d0299ad8e4ebb4d34568fdc561473f2808b86 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:03:59 +0300 Subject: [PATCH 432/518] fix(backups): refuse a MariaDB restore from a Backup being deleted Deleting a MariaDB Backup now deletes its dump, and mariadb-operator restores the dump nearest the requested time when the exact one is gone. A restore that starts while the Backup's cleanup runs can lose its dump between the check and the operator's read, and then completes on another run's data without any error. The restore now reads the Backup from the API server before creating the operator Restore and fails when it is being deleted or is gone. The copy the reconcile was handed comes from the cache and can predate the delete; a transient read error keeps the RestoreJob waiting like any other failed dump check. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadb_restore_dump_test.go | 52 +++++++++++++++++++ .../mariadbstrategy_controller.go | 14 +++++ 2 files changed, 66 insertions(+) diff --git a/internal/backupcontroller/mariadb_restore_dump_test.go b/internal/backupcontroller/mariadb_restore_dump_test.go index 6c8202adf..218c91de6 100644 --- a/internal/backupcontroller/mariadb_restore_dump_test.go +++ b/internal/backupcontroller/mariadb_restore_dump_test.go @@ -14,6 +14,7 @@ import ( "k8s.io/client-go/tools/record" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1" "github.com/cozystack/cozystack/internal/backupcontroller/mariadbapp" @@ -184,6 +185,57 @@ func recordedDump(name string) map[string]string { return map[string]string{mariadbDumpFileKey: name, mariadbDumpTimeKey: mdbRestoreTime} } +// A Backup being deleted is deleting its dump, so a restore that starts +// now can find it gone once the operator reads it, and the operator then +// restores the nearest other dump. The Backup is read from the API server, +// since the copy the reconcile was handed can predate the delete. +func TestReconcileMariaDBRestore_RefusesABackupBeingDeleted(t *testing.T) { + creds := s3CredsSecret("tenant", "creds") + present := func() *fakeS3Lister { + return &fakeS3Lister{objects: []s3ObjectInfo{{Key: mdbRestorePrefix + mdbRestoreDump, LastModified: mdbRestoreT0.Add(20 * time.Second)}}} + } + cases := map[string]map[string]string{"recorded dump": recordedDump(mdbRestoreDump), "legacy Backup": nil} + for name, md := range cases { + t.Run(name+", being deleted", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, md, present(), creds) + ctx := context.Background() + live := &backupsv1alpha1.Backup{} + if err := f.c.Get(ctx, client.ObjectKeyFromObject(f.backup), live); err != nil { + t.Fatal(err) + } + live.Finalizers = []string{backupFinalizer} + if err := f.c.Update(ctx, live); err != nil { + t.Fatal(err) + } + if err := f.c.Delete(ctx, live); err != nil { + t.Fatal(err) + } + f.reconcile(t) + f.wantFailed(t, "src-bk", "being deleted") + }) + t.Run(name+", already gone", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, md, present(), creds) + if err := f.c.Delete(context.Background(), f.backup.DeepCopy()); err != nil { + t.Fatal(err) + } + f.reconcile(t) + f.wantFailed(t, "src-bk", "is gone") + }) + } + t.Run("a Backup that cannot be read keeps the RestoreJob waiting", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds) + f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*backupsv1alpha1.Backup); ok { + return errors.New("apiserver unavailable") + } + return c.Get(ctx, key, obj, opts...) + }, + }) + f.wantWaiting(t, f.reconcile(t), "src-bk", "apiserver unavailable") + }) +} + func TestReconcileMariaDBRestore_RecordedDump(t *testing.T) { creds := s3CredsSecret("tenant", "creds") diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 6d6312eb4..e3e00dbec 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -669,6 +669,20 @@ func (e mariadbRestoreRefused) Error() string { return string(e) } // checked to still exist. A Backup that records no dump is resolved by // legacyMariaDBRestoreTargetTime. func (r *RestoreJobReconciler) mariadbRestoreTargetTime(ctx context.Context, rj *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, src *mariadbtypes.Backup) (string, error) { + // Deleting a Backup deletes its dump, and the operator restores the + // nearest other dump once its own is gone, so a restore must not start + // from a Backup being deleted. The copy in hand can predate the delete; + // the cleanup holds the Backup only for RestoreJobs it can already see. + live := &backupsv1alpha1.Backup{} + if err := r.apiReader().Get(ctx, client.ObjectKeyFromObject(backup), live); err != nil { + if apierrors.IsNotFound(err) { + return "", mariadbRestoreRefused(fmt.Sprintf("Backup %s/%s is gone, and its dump may be gone with it", backup.Namespace, backup.Name)) + } + return "", fmt.Errorf("read Backup %s/%s: %w", backup.Namespace, backup.Name, err) + } + if !live.DeletionTimestamp.IsZero() { + return "", mariadbRestoreRefused(fmt.Sprintf("Backup %s/%s is being deleted, and its dump with it", backup.Namespace, backup.Name)) + } // A failed Job is Complete=True too, with reason JobFailed, and a // BackupJob that took Complete=True for success made a Backup of it. if cond := apimeta.FindStatusCondition(src.Status.Conditions, mariadbtypes.ConditionTypeComplete); cond != nil && cond.Reason == mariadbtypes.ConditionReasonJobFailed { From 654c543264bbcba9c939a7c8b436ffbc02340c4d Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:04:53 +0300 Subject: [PATCH 433/518] fix(backups): see a MariaDB restore the cache has not caught up with A MariaDB restore now checks on the API server that its Backup is not being deleted before it starts, so a restore that started first exists on the API server before the delete. The cleanup read RestoreJobs from the informer cache, which can lag behind, and could delete the dump under a restore it had not seen yet. Reading them uncached closes that window from both sides. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/backup_controller.go | 15 +++++++++++++++ .../backupcontroller/mariadbstrategy_cleanup.go | 2 +- .../mariadbstrategy_cleanup_test.go | 12 ++++++++++++ 3 files changed, 28 insertions(+), 1 deletion(-) diff --git a/internal/backupcontroller/backup_controller.go b/internal/backupcontroller/backup_controller.go index 9d755059e..d8c69bf1c 100644 --- a/internal/backupcontroller/backup_controller.go +++ b/internal/backupcontroller/backup_controller.go @@ -43,6 +43,10 @@ type BackupReconciler struct { // cleanups project them into the Backup's namespace, and the MariaDB // cleanup reads them to reach the platform bucket. CredentialsConfig BackupCredentialsConfig + // APIReader is the manager's uncached reader. The MariaDB cleanup lists + // the RestoreJobs of a Backup through it: a restore that started before + // the delete may not be in the cache yet. Wired in SetupWithManager. + APIReader client.Reader // newS3ObjectLister and newS3ObjectRemover override the S3 client the // MariaDB cleanup deletes a dump with; nil means a real minio client. @@ -294,7 +298,18 @@ func (r *BackupReconciler) cleanupCNPGBackup(ctx context.Context, backup *backup // SetupWithManager registers the BackupReconciler with the Manager. func (r *BackupReconciler) SetupWithManager(mgr ctrl.Manager) error { + r.APIReader = mgr.GetAPIReader() return ctrl.NewControllerManagedBy(mgr). For(&backupsv1alpha1.Backup{}). Complete(r) } + +// apiReader returns the uncached reader. Reconcilers built by hand in unit +// tests have none and fall back to their fake Client, which has no cache to +// bypass. +func (r *BackupReconciler) apiReader() client.Reader { + if r.APIReader != nil { + return r.APIReader + } + return r.Client +} diff --git a/internal/backupcontroller/mariadbstrategy_cleanup.go b/internal/backupcontroller/mariadbstrategy_cleanup.go index 94a7d10f4..531b806c9 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup.go @@ -200,7 +200,7 @@ func contestedMariaDBDump(ctx context.Context, lister s3ObjectLister, reader cli // backup and has not finished, or "". func (r *BackupReconciler) activeRestoreOf(ctx context.Context, backup *backupsv1alpha1.Backup) (string, error) { list := &backupsv1alpha1.RestoreJobList{} - if err := r.List(ctx, list, client.InNamespace(backup.Namespace)); err != nil { + if err := r.apiReader().List(ctx, list, client.InNamespace(backup.Namespace)); err != nil { return "", fmt.Errorf("list RestoreJobs: %w", err) } for i := range list.Items { diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index ddca34523..20f2757d4 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -763,6 +763,18 @@ func TestCleanupMariaDBBackup_WaitsForARestoreFromIt(t *testing.T) { } }) } + // A restore checks that its Backup is not being deleted before it + // starts, so one that started first is on the API server before the + // delete, though not necessarily in this controller's cache yet. + t.Run("a restore the cache has not seen yet holds it", func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + r.APIReader = newMariaDBStrategyTestClient(t, restoreJobOf("rj", "bk", backupsv1alpha1.RestoreJobPhaseRunning)) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || !requeues(res) || len(bucket.removed) != 0 { + t.Fatalf("got res=%+v err=%v removed=%v, want the Backup held while the restore runs", res, err, bucket.removed) + } + }) t.Run("finished restores and restores of other Backups do not hold it", func(t *testing.T) { bucket := &fakeS3Bucket{} r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), From 37a74e59394a69700e99b04a697f50b9a78e2625 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:05:51 +0300 Subject: [PATCH 434/518] test(backups): reach the S3 calls the MariaDB cleanup timeout bounds Contesting a recorded dump lists the bucket before the delete, and the timeout test supplied no lister, so a real client failed on DNS before the blocking delete was ever called and the suite stayed green with the timeout removed. The test now lists the dump through the fake bucket, blocks in the listing and in the delete in turn, and requires that the blocking call was entered. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadbstrategy_cleanup_test.go | 75 +++++++++++++------ 1 file changed, 54 insertions(+), 21 deletions(-) diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index 20f2757d4..0ce6b2d01 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -961,36 +961,69 @@ func TestCleanupMariaDBBackup_PlatformEndpointFallsBackToTheConfiguredOne(t *tes } } -// blockingBucket answers only when its context ends, as an unreachable -// endpoint does. -type blockingBucket struct{ fakeS3Bucket } +// blockingBucket lists the dump it holds and answers the call named by +// blockOn only when its context ends, as an unreachable endpoint does. +type blockingBucket struct { + fakeS3Bucket + blockOn string + entered chan struct{} +} -func (b *blockingBucket) RemoveObject(ctx context.Context, _, _ string) error { +func (b *blockingBucket) block(ctx context.Context, call string) bool { + if b.blockOn != call { + return false + } + close(b.entered) <-ctx.Done() - return ctx.Err() + return true +} + +func (b *blockingBucket) ListObjects(ctx context.Context, bucket, prefix string, recursive bool) ([]s3ObjectInfo, error) { + if b.block(ctx, "list") { + return nil, ctx.Err() + } + return b.fakeS3Bucket.ListObjects(ctx, bucket, prefix, recursive) +} + +func (b *blockingBucket) RemoveObject(ctx context.Context, bucket, key string) error { + if b.block(ctx, "remove") { + return ctx.Err() + } + return b.fakeS3Bucket.RemoveObject(ctx, bucket, key) } func TestCleanupMariaDBBackup_BoundsAnUnreachableEndpoint(t *testing.T) { prev := mariadbDumpCleanupTimeout mariadbDumpCleanupTimeout = 50 * time.Millisecond t.Cleanup(func() { mariadbDumpCleanupTimeout = prev }) - bucket := &blockingBucket{} - c := newMariaDBStrategyTestClient(t, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) - r := &BackupReconciler{Client: c, Scheme: c.Scheme(), Recorder: record.NewFakeRecorder(20), - newS3ObjectRemover: func(s3Connection) (s3ObjectRemover, error) { return bucket, nil }} + for _, call := range []string{"list", "remove"} { + t.Run(call, func(t *testing.T) { + bucket := &blockingBucket{blockOn: call, entered: make(chan struct{})} + bucket.objects = []s3ObjectInfo{{Key: cleanupPrefix + cleanupDumpName, LastModified: cleanupCreated.Add(2 * time.Minute)}} + c := newMariaDBStrategyTestClient(t, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + r := &BackupReconciler{Client: c, Scheme: c.Scheme(), Recorder: record.NewFakeRecorder(20), + newS3ObjectLister: func(s3Connection) (s3ObjectLister, error) { return bucket, nil }, + newS3ObjectRemover: func(s3Connection) (s3ObjectRemover, error) { return bucket, nil }} - done := make(chan ctrl.Result, 1) - go func() { - res, _ := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) - done <- res - }() - select { - case res := <-done: - if !requeues(res) { - t.Errorf("res = %+v, want the Backup held and retried", res) - } - case <-time.After(5 * time.Second): - t.Fatal("cleanup did not give up on an unreachable endpoint") + done := make(chan ctrl.Result, 1) + go func() { + res, _ := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + done <- res + }() + select { + case res := <-done: + select { + case <-bucket.entered: + default: + t.Fatalf("cleanup returned without reaching the blocking %s", call) + } + if !requeues(res) { + t.Errorf("res = %+v, want the Backup held and retried", res) + } + case <-time.After(5 * time.Second): + t.Fatalf("cleanup did not give up on an unreachable endpoint blocking %s", call) + } + }) } } From bbeca3453534124975682bda008f0d025aaed65c Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:07:03 +0300 Subject: [PATCH 435/518] fix(backups): retry a failed read of the platform backup credentials The MariaDB cleanup reaches the platform bucket through the platform's current endpoint so that a Backup taken before the endpoint moved is still deleted. A failed read of the platform credentials made it fall back to the endpoint recorded on the operator Backup, which may be the old address: there the dump is not found, a missing dump counts as already deleted, and the operator Backup that pointed at the real dump was deleted with a success log. A read that fails with an API error now holds the Backup and retries. A missing or malformed platform Secret still falls back, since it is a standing state that a retry would not change. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadbstrategy_cleanup.go | 10 +++++-- .../mariadbstrategy_cleanup_test.go | 30 +++++++++++++++++++ 2 files changed, 38 insertions(+), 2 deletions(-) diff --git a/internal/backupcontroller/mariadbstrategy_cleanup.go b/internal/backupcontroller/mariadbstrategy_cleanup.go index 531b806c9..b4617918e 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup.go @@ -220,12 +220,18 @@ func (r *BackupReconciler) activeRestoreOf(ctx context.Context, backup *backupsv // Secret the platform credentials are projected to, is reached as the // platform reaches it now, so a Backup taken before the endpoint moved is // still deleted. Any other bucket, or the platform bucket while the -// platform credentials cannot be read, is reached as op recorded it; a -// delete that fails there is handled by retryMariaDBCleanup like any other. +// platform credentials are missing or malformed, is reached as op recorded +// it; a delete that fails there is handled by retryMariaDBCleanup like any +// other. A failed read of the platform credentials is retried instead: the +// recorded endpoint may be the one the platform moved off, where the dump +// would be found missing and reported deleted. func (r *BackupReconciler) mariadbCleanupConnection(ctx context.Context, op *mariadbtypes.Backup) (s3Connection, error) { if r.CredentialsConfig.IsEnabled() { conn, bucket, err := r.platformS3Connection(ctx) + var projErr *ProjectionError switch { + case errors.As(err, &projErr) && projErr.Reason == ReasonAPIError: + return s3Connection{}, err case err != nil: getLogger(ctx).Info("cannot read the platform backup credentials; using the operator Backup's own", "operatorBackup", op.Name, "error", err) case bucket == op.Spec.Storage.S3.Bucket && op.Spec.Storage.S3.AccessKeyIdSecretKeyRef.Name == r.CredentialsConfig.TargetSecretName: diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index 0ce6b2d01..1eecfeaff 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -869,6 +869,36 @@ func TestCleanupMariaDBBackup_UnreadablePlatformCredentialsFallBackToTheOperator } } +// A missing or malformed platform Secret is a standing state, and the +// operator Backup's own credentials are the best left. A failed read says +// nothing about the Secret: falling back then can list a moved endpoint's +// old address, find no dump there and report it deleted. +func TestCleanupMariaDBBackup_TransientPlatformCredentialsReadIsRetried(t *testing.T) { + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: cleanupPrefix + cleanupDumpName, LastModified: cleanupCreated.Add(2 * time.Minute)}}} + c := interceptor.NewClient(newMariaDBStrategyTestClient(t, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), + platformCredsSecret("bkt", "s3.new.example")).(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, c client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*corev1.Secret); ok && key.Name == "cozy-backups-credentials" { + return errors.New("apiserver unavailable") + } + return c.Get(ctx, key, obj, opts...) + }, + }) + r, rec := cleanupReconcilerWithClient(c, bucket) + r.CredentialsConfig = platformCredsConfig("https://s3.new.example") + + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || !requeues(res) { + t.Fatalf("got res=%+v err=%v, want the Backup held and retried", res, err) + } + if len(bucket.conns) != 0 || len(bucket.removed) != 0 || !operatorBackupExists(t, r.Client) { + t.Errorf("conns=%v removed=%v operatorBackupExists=%v, want the bucket untouched", bucket.conns, bucket.removed, operatorBackupExists(t, r.Client)) + } + if events := drainEvents(rec); !hasEvent(events, "Warning ArtifactCleanupFailed") { + t.Errorf("events = %v, want a Warning ArtifactCleanupFailed", events) + } +} + func TestCleanupMariaDBBackup_UnreadablePlatformCredentialsAndAFailingDeleteHold(t *testing.T) { bucket := &fakeS3Bucket{removeErr: errors.New("dial tcp: no such host")} r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) From 3334152da7d78e755ab7d3cc85ed8d459c0ba86e Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:08:43 +0300 Subject: [PATCH 436/518] fix(backups): report a recorded MariaDB dump that was not found A recorded dump missing from the listing was logged as deleted, with no event. That is right when an earlier attempt or maxRetention removed it, but the listing goes through the platform's current endpoint, and after an endpoint move the dump can still sit on the old storage: the cleanup then deleted the operator Backup that pointed at it and said it had deleted the dump. The cleanup now records a Normal ArtifactAlreadyAbsent event naming the key and the endpoint it listed, so whoever later looks for that dump can tell where it was not found. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadbstrategy_cleanup.go | 36 ++++++++++++------- .../mariadbstrategy_cleanup_test.go | 14 ++++++-- 2 files changed, 35 insertions(+), 15 deletions(-) diff --git a/internal/backupcontroller/mariadbstrategy_cleanup.go b/internal/backupcontroller/mariadbstrategy_cleanup.go index b4617918e..25775185b 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup.go @@ -118,8 +118,10 @@ func (r *BackupReconciler) cleanupMariaDBBackup(ctx context.Context, backup *bac if err != nil { return r.retryMariaDBCleanup(ctx, backup, location, fmt.Errorf("build S3 client: %w", err)) } + absent := false if name != "" { - other, err := contestedMariaDBDump(s3ctx, lister, r.Client, op, name) + other, found, err := contestedMariaDBDump(s3ctx, lister, r.Client, op, name) + absent = !found if err != nil { return r.retryMariaDBCleanup(ctx, backup, location+name, err) } @@ -158,42 +160,52 @@ func (r *BackupReconciler) cleanupMariaDBBackup(ctx context.Context, backup *bac if err := r.Delete(ctx, op, &client.DeleteOptions{PropagationPolicy: &policy}); err != nil && !apierrors.IsNotFound(err) { return ctrl.Result{}, fmt.Errorf("delete k8s.mariadb.com/Backup %s/%s: %w", op.Namespace, op.Name, err) } + if absent { + logger.Info("MariaDB dump already absent; deleted its operator Backup", "backup", backup.Name, "dump", location+name, "endpoint", conn.Endpoint, "operatorBackup", opName) + if r.Recorder != nil { + r.Recorder.Eventf(backup, corev1.EventTypeNormal, "ArtifactAlreadyAbsent", + "%s was not found when listed through %s, so it was deleted earlier or never reached that storage; k8s.mariadb.com/Backup %s is deleted", + location+name, conn.Endpoint, opName) + } + return ctrl.Result{}, nil + } logger.Info("deleted MariaDB dump and its operator Backup", "backup", backup.Name, "dump", location+name, "operatorBackup", opName) return ctrl.Result{}, nil } // contestedMariaDBDump returns the name of another operator Backup on op's // bucket and prefix whose run, finished or not, could have written the dump -// name, or "" when none could. The size probe that recorded name let only -// runs finished by then contest it, and a run can upload its dump and prune -// op's through maxRetention before it completes. A dump no longer in the -// bucket is contested by nobody: deleting it again removes nothing. -func contestedMariaDBDump(ctx context.Context, lister s3ObjectLister, reader client.Reader, op *mariadbtypes.Backup, name string) (string, error) { +// name, or "" when none could, and whether the listing found the dump. The +// size probe that recorded name let only runs finished by then contest it, +// and a run can upload its dump and prune op's through maxRetention before +// it completes. A dump no longer in the bucket is contested by nobody: +// deleting it again removes nothing. +func contestedMariaDBDump(ctx context.Context, lister s3ObjectLister, reader client.Reader, op *mariadbtypes.Backup, name string) (string, bool, error) { ts, ok := parseMariaDBLogicalBackupFile(name) if !ok { - return "", fmt.Errorf("%q is not the name of a mariadb-operator dump", name) + return "", false, fmt.Errorf("%q is not the name of a mariadb-operator dump", name) } s3 := op.Spec.Storage.S3 key := mariadbOperatorS3Prefix(s3.Prefix) + name objects, err := lister.ListObjects(ctx, s3.Bucket, key, false) if err != nil { - return "", fmt.Errorf("list s3://%s/%s: %w", s3.Bucket, key, err) + return "", false, fmt.Errorf("list s3://%s/%s: %w", s3.Bucket, key, err) } i := slices.IndexFunc(objects, func(obj s3ObjectInfo) bool { return obj.Key == key }) if i < 0 { - return "", nil + return "", false, nil } others, err := neighbourMariaDBRuns(ctx, reader, op, true) if err != nil { - return "", fmt.Errorf("list operator Backups: %w", err) + return "", true, fmt.Errorf("list operator Backups: %w", err) } d := mariadbDump{Key: key, Name: name, Timestamp: ts, LastModified: objects[i].LastModified} for _, other := range others { if other.placesOnEitherClock(d) { - return other.name, nil + return other.name, true, nil } } - return "", nil + return "", true, nil } // activeRestoreOf returns the name of a RestoreJob that restores from diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index 1eecfeaff..da9e0902e 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -165,7 +165,7 @@ func assertReleasedUntouched(t *testing.T, r *BackupReconciler, rec *record.Fake } func TestCleanupMariaDBBackup_DeletesRecordedDumpThenOperatorBackup(t *testing.T) { - bucket := &fakeS3Bucket{} + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: cleanupPrefix + cleanupDumpName, LastModified: cleanupCreated.Add(2 * time.Minute)}}} r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) @@ -512,7 +512,7 @@ func TestContestedMariaDBDump_RejectsANameWithoutADumpTime(t *testing.T) { name := "backup.not-a-time.sql" bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: cleanupPrefix + name, LastModified: cleanupCreated}}} c := newMariaDBStrategyTestClient(t, op) - if _, err := contestedMariaDBDump(context.Background(), bucket, c, op, name); err == nil { + if _, _, err := contestedMariaDBDump(context.Background(), bucket, c, op, name); err == nil { t.Fatal("err = nil, want the unreadable dump time reported") } } @@ -534,9 +534,17 @@ func TestCleanupMariaDBBackup_RecordedDumpAlreadyGoneIsNotContested(t *testing.T if operatorBackupExists(t, r.Client) { t.Error("operator Backup kept, want it deleted") } - if events := drainEvents(rec); hasEvent(events, "Warning ArtifactNotDeleted") { + events := drainEvents(rec) + if hasEvent(events, "Warning ArtifactNotDeleted") { t.Errorf("events = %v, want no ArtifactNotDeleted", events) } + // The listing may have reached other storage than the dump went to, as + // after an endpoint move, so the absence is reported with where it was + // looked for rather than logged as a delete. + if joined := strings.Join(events, "\n"); !hasEvent(events, "Normal ArtifactAlreadyAbsent") || + !strings.Contains(joined, "s3://bkt/"+cleanupPrefix+cleanupDumpName) || !strings.Contains(joined, "through s3.example:9000") { + t.Errorf("events = %v, want a Normal ArtifactAlreadyAbsent naming the key and the endpoint listed", events) + } } // A run started after the recorded dump was uploaded cannot have written it. From d57ebbe808e831637062a920ba9f12fd5580a541 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:09:15 +0300 Subject: [PATCH 437/518] docs(backups): state what the MariaDB dump delete does not cover The comment on who deletes a MariaDB Backup listed namespace teardown without saying that teardown can remove the operator Backup first, in which case the dump is left behind with a Warning, so it promised a delete the code does not make. The remover contract claimed a missing key is never an error, which holds only for a backend that answers the delete with 204, the one status minio-go accepts. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/artifact_size.go | 4 +++- internal/backupcontroller/backup_controller.go | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/internal/backupcontroller/artifact_size.go b/internal/backupcontroller/artifact_size.go index 40d827c62..97a67a729 100644 --- a/internal/backupcontroller/artifact_size.go +++ b/internal/backupcontroller/artifact_size.go @@ -39,7 +39,9 @@ type s3ObjectLister interface { } type s3ObjectRemover interface { - // RemoveObject deletes key; a key that does not exist is not an error. + // RemoveObject deletes key. S3 answers 204 for a key that does not + // exist and minio-go takes 204 as success; a backend answering a + // missing key with any other status makes it an error. RemoveObject(ctx context.Context, bucket, key string) error } diff --git a/internal/backupcontroller/backup_controller.go b/internal/backupcontroller/backup_controller.go index d8c69bf1c..a632f851d 100644 --- a/internal/backupcontroller/backup_controller.go +++ b/internal/backupcontroller/backup_controller.go @@ -142,7 +142,9 @@ func (r *BackupReconciler) cleanupOnDelete(ctx context.Context, backup *backupsv // a dump on the platform bucket is deleted here or never. A dump in // a tenant's own bucket is deleted too: the Backup proves it owns // it, and tenants cannot delete Backups, so the delete comes from an - // administrator or from namespace teardown (see cleanupMariaDBBackup). + // administrator or from namespace teardown. Teardown can remove the + // operator Backup first, and the dump is then left behind with a + // Warning, since its storage is read from it (see cleanupMariaDBBackup). return r.cleanupMariaDBBackup(ctx, backup) case strategyv1alpha1.MongoDBStrategyKind: // Split by flow (see cleanupMongoDBBackup). A legacy backup writes to the From 14a842ad168c0fbe3c650a6ff541c4af163ee114 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:15:18 +0300 Subject: [PATCH 438/518] fix(backups): find a MariaDB operator Restore the cache has not seen The MariaDB restore creates its operator Restore once and then follows it, finding it by label in the informer cache. Nothing watches operator Restores, so a reconcile that ran before the new Restore reached the cache saw none: it created a second one, or, now that a restore refuses a Backup being deleted, failed the RestoreJob while its Restore was still running, after which the cleanup deleted the dump that Restore was reading. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadb_restore_dump_test.go | 38 +++++++++++++++++++ .../mariadbstrategy_controller.go | 6 ++- 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/internal/backupcontroller/mariadb_restore_dump_test.go b/internal/backupcontroller/mariadb_restore_dump_test.go index 218c91de6..1919d41c9 100644 --- a/internal/backupcontroller/mariadb_restore_dump_test.go +++ b/internal/backupcontroller/mariadb_restore_dump_test.go @@ -222,6 +222,44 @@ func TestReconcileMariaDBRestore_RefusesABackupBeingDeleted(t *testing.T) { f.wantFailed(t, "src-bk", "is gone") }) } + // The refusal applies only before the operator Restore exists. One + // created by an earlier reconcile but not yet in the cache must be + // found and followed, or the RestoreJob fails under a running restore + // and the cleanup deletes the dump that restore reads. + t.Run("an operator Restore the cache has not seen yet is followed", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds) + ctx := context.Background() + live := &backupsv1alpha1.Backup{} + if err := f.c.Get(ctx, client.ObjectKeyFromObject(f.backup), live); err != nil { + t.Fatal(err) + } + live.Finalizers = []string{backupFinalizer} + if err := f.c.Update(ctx, live); err != nil { + t.Fatal(err) + } + if err := f.c.Delete(ctx, live); err != nil { + t.Fatal(err) + } + if err := f.c.Get(ctx, client.ObjectKeyFromObject(live), live); err != nil || live.DeletionTimestamp.IsZero() { + t.Fatalf("Backup not left being deleted: %v", err) + } + live.ResourceVersion = "" + apiServer := newMariaDBStrategyTestClient(t, live, &mariadbtypes.Restore{ObjectMeta: metav1.ObjectMeta{ + Namespace: "tenant", Name: "rj-abcde", + Labels: map[string]string{ + backupsv1alpha1.OwningJobNameLabel: "rj", + backupsv1alpha1.OwningJobNamespaceLabel: "tenant", + }, + }}) + f.r.APIReader = apiServer + f.reconcile(t) + if f.rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("RestoreJob failed under a running restore: %s", f.rj.Status.Message) + } + if got := f.restores(t); len(got) != 0 { + t.Errorf("created %d operator Restores; want the existing one followed", len(got)) + } + }) t.Run("a Backup that cannot be read keeps the RestoreJob waiting", func(t *testing.T) { f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds) f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index e3e00dbec..b97f7548b 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -756,10 +756,12 @@ func (r *RestoreJobReconciler) awaitMariaDBDumpCheck(ctx context.Context, rj *ba } // findMariaDBRestoreForJob returns the k8s.mariadb.com/Restore labelled -// with the RestoreJob, if a previous reconcile already created one. +// with the RestoreJob, if a previous reconcile already created one. It is +// read uncached: one the cache has not caught up with would be created +// again, or, once its Backup is being deleted, refused while it runs. func (r *RestoreJobReconciler) findMariaDBRestoreForJob(ctx context.Context, rj *backupsv1alpha1.RestoreJob) (*mariadbtypes.Restore, error) { list := &mariadbtypes.RestoreList{} - if err := r.List(ctx, list, + if err := r.apiReader().List(ctx, list, client.InNamespace(rj.Namespace), client.MatchingLabels{ backupsv1alpha1.OwningJobNameLabel: rj.Name, From ee6404a0c88af0850910b9bbb25ad8cc292f30ed Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:28:45 +0300 Subject: [PATCH 439/518] fix(backups): retry a failed lookup of a MariaDB operator Restore The lookup now goes to the API server, which can time out or throttle, and a failed lookup failed the RestoreJob. It says nothing about whether the operator Restore exists, so the RestoreJob could be failed while its Restore was running, and the Backup's cleanup would then stop waiting for it and delete the dump the Restore reads. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadb_restore_dump_test.go | 22 +++++++++++++++++++ .../mariadbstrategy_controller.go | 4 +++- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/internal/backupcontroller/mariadb_restore_dump_test.go b/internal/backupcontroller/mariadb_restore_dump_test.go index 1919d41c9..5ec96e663 100644 --- a/internal/backupcontroller/mariadb_restore_dump_test.go +++ b/internal/backupcontroller/mariadb_restore_dump_test.go @@ -260,6 +260,28 @@ func TestReconcileMariaDBRestore_RefusesABackupBeingDeleted(t *testing.T) { t.Errorf("created %d operator Restores; want the existing one followed", len(got)) } }) + // A failed lookup says nothing about whether the operator Restore + // exists, so the RestoreJob must not be failed under one that runs. + t.Run("an operator Restore lookup that fails is retried", func(t *testing.T) { + f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds) + f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*mariadbtypes.RestoreList); ok { + return errors.New("etcdserver: request timed out") + } + return c.List(ctx, list, opts...) + }, + }) + if _, err := f.r.reconcileMariaDBRestore(context.Background(), f.rj, f.backup); err == nil { + t.Error("err = nil, want the failed lookup returned for a retry") + } + if f.rj.Status.Phase == backupsv1alpha1.RestoreJobPhaseFailed { + t.Fatalf("RestoreJob failed on a lookup error: %s", f.rj.Status.Message) + } + if got := f.restores(t); len(got) != 0 { + t.Errorf("created %d operator Restores without knowing whether one exists", len(got)) + } + }) t.Run("a Backup that cannot be read keeps the RestoreJob waiting", func(t *testing.T) { f := newMariaDBRestoreFixture(t, recordedDump(mdbRestoreDump), present(), creds) f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index b97f7548b..28a24a14e 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -598,7 +598,9 @@ func (r *RestoreJobReconciler) reconcileMariaDBRestore(ctx context.Context, rest mdbRestore, err := r.findMariaDBRestoreForJob(ctx, restoreJob) if err != nil { - return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("failed to ensure k8s.mariadb.com/Restore: %v", err)) + // Retried: failing the RestoreJob here could fail it under an + // operator Restore that is running. + return ctrl.Result{}, fmt.Errorf("find k8s.mariadb.com/Restore of RestoreJob %s/%s: %w", restoreJob.Namespace, restoreJob.Name, err) } // The dump is chosen once, as the operator Restore is created; one that // already exists has made its choice. From 82468082890a9e6a587ae6c94f7ac02622f6610d Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:30:41 +0300 Subject: [PATCH 440/518] fix(backups): hold a MariaDB dump while an operator Restore reads it The cleanup waited for a restore by the phase of its RestoreJob, but the phase is not the restore: a RestoreJob fails on its deadline while the operator Restore it created keeps running, and that Restore still reads the dump. Deleting the dump then made mariadb-operator restore the nearest other one. The cleanup now also holds the Backup while an operator Restore of its operator Backup has not been marked Complete, which mariadb-operator does once the Restore's Job has completed or failed, so what keeps the dump is the restore that reads it rather than the status of the request. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadbstrategy_cleanup.go | 32 +++++-- .../mariadbstrategy_cleanup_test.go | 83 +++++++++++++++++++ 2 files changed, 108 insertions(+), 7 deletions(-) diff --git a/internal/backupcontroller/mariadbstrategy_cleanup.go b/internal/backupcontroller/mariadbstrategy_cleanup.go index 25775185b..17d612955 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup.go @@ -59,15 +59,15 @@ func (r *BackupReconciler) cleanupMariaDBBackup(ctx context.Context, backup *bac // The operator Restore reads the dump through the operator Backup, so // neither may go while a restore from this Backup runs. - restore, err := r.activeRestoreOf(ctx, backup) + restore, err := r.activeRestoreOf(ctx, backup, opName) if err != nil { return ctrl.Result{}, err } if restore != "" { - logger.Debug("holding MariaDB Backup cleanup while a restore from it runs", "backup", backup.Name, "restorejob", restore) + logger.Debug("holding MariaDB Backup cleanup while a restore from it runs", "backup", backup.Name, "restore", restore) if r.Recorder != nil { r.Recorder.Eventf(backup, corev1.EventTypeNormal, "ArtifactCleanupWaiting", - "RestoreJob %s is restoring from this Backup; its dump is deleted once the restore finishes. Annotate the Backup %s=true to release it now and leave the dump", + "%s is restoring from this Backup; its dump is deleted once the restore finishes. Annotate the Backup %s=true to release it now and leave the dump", restore, mariadbSkipArtifactCleanupAnnotation) } return ctrl.Result{RequeueAfter: mariadbCleanupRetryInterval}, nil @@ -208,9 +208,14 @@ func contestedMariaDBDump(ctx context.Context, lister s3ObjectLister, reader cli return "", true, nil } -// activeRestoreOf returns the name of a RestoreJob that restores from -// backup and has not finished, or "". -func (r *BackupReconciler) activeRestoreOf(ctx context.Context, backup *backupsv1alpha1.Backup) (string, error) { +// activeRestoreOf names a restore from backup that has not finished: a +// RestoreJob of it, or an operator Restore reading the dump of opName, the +// operator Backup it was made of; "" when there is none. The operator +// Restore is checked on its own because a RestoreJob can fail, on its +// deadline, while the Restore it created keeps reading the dump. +// mariadb-operator marks a Restore Complete=True once its Job has +// completed or failed. +func (r *BackupReconciler) activeRestoreOf(ctx context.Context, backup *backupsv1alpha1.Backup, opName string) (string, error) { list := &backupsv1alpha1.RestoreJobList{} if err := r.apiReader().List(ctx, list, client.InNamespace(backup.Namespace)); err != nil { return "", fmt.Errorf("list RestoreJobs: %w", err) @@ -221,7 +226,20 @@ func (r *BackupReconciler) activeRestoreOf(ctx context.Context, backup *backupsv continue } if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseSucceeded && rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed { - return rj.Name, nil + return "RestoreJob " + rj.Name, nil + } + } + restores := &mariadbtypes.RestoreList{} + if err := r.apiReader().List(ctx, restores, client.InNamespace(backup.Namespace)); err != nil { + return "", fmt.Errorf("list k8s.mariadb.com/Restores: %w", err) + } + for i := range restores.Items { + restore := &restores.Items[i] + if restore.Spec.BackupRef == nil || restore.Spec.BackupRef.Name != opName { + continue + } + if !apimeta.IsStatusConditionTrue(restore.Status.Conditions, mariadbtypes.ConditionTypeComplete) { + return "k8s.mariadb.com/Restore " + restore.Name, nil } } return "", nil diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index da9e0902e..ec2dea1a8 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -753,6 +753,89 @@ func restoreJobOf(name, backupName string, phase backupsv1alpha1.RestoreJobPhase } } +// operatorRestoreOf is a k8s.mariadb.com/Restore reading the dump of the +// operator Backup backupRef, with the given Complete condition. +func operatorRestoreOf(name, backupRef string, complete *metav1.Condition) *mariadbtypes.Restore { + restore := &mariadbtypes.Restore{ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: name}} + if backupRef != "" { + restore.Spec.BackupRef = &mariadbtypes.BackupReference{Name: backupRef} + } + if complete != nil { + restore.Status.Conditions = []metav1.Condition{*complete} + } + return restore +} + +func restoreComplete(status metav1.ConditionStatus, reason string) *metav1.Condition { + return &metav1.Condition{Type: mariadbtypes.ConditionTypeComplete, Status: status, Reason: reason, LastTransitionTime: metav1.Now()} +} + +// The RestoreJob's phase is not the restore: one failed on its deadline +// leaves its operator Restore running, and that Restore still reads the +// dump. Until the operator marks it Complete, as it does a finished or a +// failed Job, the dump stays. +func TestCleanupMariaDBBackup_WaitsForAnOperatorRestoreOfItsDump(t *testing.T) { + held := map[string]*metav1.Condition{ + "running": restoreComplete(metav1.ConditionFalse, mariadbtypes.ConditionReasonJobRunning), + "suspended": restoreComplete(metav1.ConditionFalse, mariadbtypes.ConditionReasonJobSuspended), + "not yet reconciled": nil, + } + for name, cond := range held { + t.Run(name, func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, rec := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), + restoreJobOf("rj", "bk", backupsv1alpha1.RestoreJobPhaseFailed), operatorRestoreOf("rj-abcde", "op-bk", cond)) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || !requeues(res) || len(bucket.removed) != 0 || !operatorBackupExists(t, r.Client) { + t.Fatalf("got res=%+v err=%v removed=%v, want the Backup held while the operator Restore runs", res, err, bucket.removed) + } + if events := drainEvents(rec); !hasEvent(events, "Normal ArtifactCleanupWaiting") || !strings.Contains(strings.Join(events, "\n"), "k8s.mariadb.com/Restore rj-abcde") { + t.Errorf("events = %v, want a Normal ArtifactCleanupWaiting naming the operator Restore", events) + } + }) + } + released := map[string]*mariadbtypes.Restore{ + "completed": operatorRestoreOf("rj-abcde", "op-bk", restoreComplete(metav1.ConditionTrue, mariadbtypes.ConditionReasonJobComplete)), + "failed": operatorRestoreOf("rj-abcde", "op-bk", restoreComplete(metav1.ConditionTrue, mariadbtypes.ConditionReasonJobFailed)), + "of another operator Backup": operatorRestoreOf("rj-abcde", "op-other", nil), + "from storage, not a Backup": operatorRestoreOf("rj-abcde", "", nil), + } + for name, restore := range released { + t.Run(name, func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), restore) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || requeues(res) || len(bucket.removed) != 1 { + t.Fatalf("got res=%+v err=%v removed=%v, want the dump deleted", res, err, bucket.removed) + } + }) + } + t.Run("one the cache has not seen yet", func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + r.APIReader = newMariaDBStrategyTestClient(t, operatorRestoreOf("rj-abcde", "op-bk", nil)) + res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) + if err != nil || !requeues(res) || len(bucket.removed) != 0 { + t.Fatalf("got res=%+v err=%v removed=%v, want the Backup held", res, err, bucket.removed) + } + }) + t.Run("a failed listing deletes nothing", func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + r.APIReader = interceptor.NewClient(newMariaDBStrategyTestClient(t).(client.WithWatch), interceptor.Funcs{ + List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*mariadbtypes.RestoreList); ok { + return errors.New("etcdserver: request timed out") + } + return c.List(ctx, list, opts...) + }, + }) + if _, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)); err == nil || len(bucket.removed) != 0 { + t.Fatalf("err=%v removed=%v, want the error returned and nothing deleted", err, bucket.removed) + } + }) +} + func TestCleanupMariaDBBackup_WaitsForARestoreFromIt(t *testing.T) { for _, phase := range []backupsv1alpha1.RestoreJobPhase{backupsv1alpha1.RestoreJobPhaseEmpty, backupsv1alpha1.RestoreJobPhasePending, backupsv1alpha1.RestoreJobPhaseRunning} { t.Run(string(phase), func(t *testing.T) { From 9fe06a874b871a9de76318753ea1f96490c941a9 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Wed, 11 Nov 2026 15:35:18 +0300 Subject: [PATCH 441/518] test(release): isolate report deduplication from workflow attempts CI reruns expose GITHUB_RUN_ATTEMPT to a test that expects a URL for the first attempt. Keep its synthetic run identity independent of the surrounding workflow. Fixes: 5a997fb7d435 ("feat(release): post an immutable report of every classification and write record version 2") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- cmd/backport-batch/report_test.go | 1 + 1 file changed, 1 insertion(+) diff --git a/cmd/backport-batch/report_test.go b/cmd/backport-batch/report_test.go index 2f6b8ef1a..b11208a04 100644 --- a/cmd/backport-batch/report_test.go +++ b/cmd/backport-batch/report_test.go @@ -130,6 +130,7 @@ func TestReportDedupAndTrust(t *testing.T) { t.Setenv("GITHUB_SERVER_URL", "https://github.com") t.Setenv("GITHUB_REPOSITORY", "cozystack/cozystack") t.Setenv("GITHUB_RUN_ID", "100") + t.Setenv("GITHUB_RUN_ATTEMPT", "1") // The report the next run makes, read off a run that publishes nothing, // planted by someone the batch does not trust. From 0195a66faeea7a143ccf3aea2ba2f95df67d9178 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:35:33 +0300 Subject: [PATCH 442/518] fix(backups): release a MariaDB Backup when the operator CRDs are gone Listing the operator Restores of a Backup before reading its operator Backup made the cleanup fail on a cluster without the mariadb-operator CRDs, retried for good, so the Backup and its namespace never went away. The operator Backup read that follows already releases the Backup in that case; with no Restore kind no restore can run, so the listing now lets the cleanup reach it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/mariadbstrategy_cleanup.go | 5 +++++ .../backupcontroller/mariadbstrategy_cleanup_test.go | 9 +++++++++ 2 files changed, 14 insertions(+) diff --git a/internal/backupcontroller/mariadbstrategy_cleanup.go b/internal/backupcontroller/mariadbstrategy_cleanup.go index 17d612955..3ff8adfbe 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup.go @@ -231,6 +231,11 @@ func (r *BackupReconciler) activeRestoreOf(ctx context.Context, backup *backupsv } restores := &mariadbtypes.RestoreList{} if err := r.apiReader().List(ctx, restores, client.InNamespace(backup.Namespace)); err != nil { + // Without the operator's CRDs no Restore can run; the operator + // Backup read that follows releases the Backup. + if apimeta.IsNoMatchError(err) { + return "", nil + } return "", fmt.Errorf("list k8s.mariadb.com/Restores: %w", err) } for i := range restores.Items { diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index ec2dea1a8..e9a90029b 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -1148,6 +1148,9 @@ func TestCleanupMariaDBBackup_BoundsAnUnreachableEndpoint(t *testing.T) { } } +// Without the mariadb-operator CRDs every k8s.mariadb.com read fails, the +// listing of operator Restores included: no Restore can run then, so the +// Backup is released rather than held for good. func TestCleanupMariaDBBackup_ReleasesWhenTheOperatorCRDIsGone(t *testing.T) { bucket := &fakeS3Bucket{} c := interceptor.NewClient(newMariaDBStrategyTestClient(t, tenantNamespace()).(client.WithWatch), interceptor.Funcs{ @@ -1157,6 +1160,12 @@ func TestCleanupMariaDBBackup_ReleasesWhenTheOperatorCRDIsGone(t *testing.T) { } return c.Get(ctx, key, obj, opts...) }, + List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*mariadbtypes.RestoreList); ok { + return &apimeta.NoKindMatchError{GroupKind: mariadbtypes.GroupVersion.WithKind("Restore").GroupKind()} + } + return c.List(ctx, list, opts...) + }, }) r, rec := cleanupReconcilerWithClient(c, bucket) res, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)) From f2507b4b84f905e13c0dea1565c64f9cf5699a7e Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:35:33 +0300 Subject: [PATCH 443/518] docs(backups): name what the MariaDB cleanup reads uncached The reader's comment named only RestoreJobs, and the restore path said the cleanup holds a Backup only for RestoreJobs it can see, while the cleanup now lists both RestoreJobs and operator Restores straight from the API server. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/backup_controller.go | 5 +++-- internal/backupcontroller/mariadbstrategy_controller.go | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/internal/backupcontroller/backup_controller.go b/internal/backupcontroller/backup_controller.go index a632f851d..b67c640ab 100644 --- a/internal/backupcontroller/backup_controller.go +++ b/internal/backupcontroller/backup_controller.go @@ -44,8 +44,9 @@ type BackupReconciler struct { // cleanup reads them to reach the platform bucket. CredentialsConfig BackupCredentialsConfig // APIReader is the manager's uncached reader. The MariaDB cleanup lists - // the RestoreJobs of a Backup through it: a restore that started before - // the delete may not be in the cache yet. Wired in SetupWithManager. + // the RestoreJobs and operator Restores of a Backup through it: a + // restore that started before the delete may not be in the cache yet. + // Wired in SetupWithManager. APIReader client.Reader // newS3ObjectLister and newS3ObjectRemover override the S3 client the diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 28a24a14e..edff9d187 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -674,7 +674,8 @@ func (r *RestoreJobReconciler) mariadbRestoreTargetTime(ctx context.Context, rj // Deleting a Backup deletes its dump, and the operator restores the // nearest other dump once its own is gone, so a restore must not start // from a Backup being deleted. The copy in hand can predate the delete; - // the cleanup holds the Backup only for RestoreJobs it can already see. + // the cleanup holds the Backup only for restores that exist when it + // lists them. live := &backupsv1alpha1.Backup{} if err := r.apiReader().Get(ctx, client.ObjectKeyFromObject(backup), live); err != nil { if apierrors.IsNotFound(err) { From a6a704768ce0deb6013a6ab33772d46d0e16da6e Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 15:41:03 +0300 Subject: [PATCH 444/518] test(backups): pin a failed RestoreJob listing in the MariaDB cleanup A RestoreJob listing that fails says nothing about whether a restore reads the dump. Swallowing that error left the whole suite green, while it would let the cleanup delete the dump under a running restore. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../mariadbstrategy_cleanup_test.go | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index e9a90029b..354bf83f3 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -866,6 +866,21 @@ func TestCleanupMariaDBBackup_WaitsForARestoreFromIt(t *testing.T) { t.Fatalf("got res=%+v err=%v removed=%v, want the Backup held while the restore runs", res, err, bucket.removed) } }) + t.Run("a failed listing deletes nothing", func(t *testing.T) { + bucket := &fakeS3Bucket{} + r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + r.APIReader = interceptor.NewClient(newMariaDBStrategyTestClient(t).(client.WithWatch), interceptor.Funcs{ + List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + if _, ok := list.(*backupsv1alpha1.RestoreJobList); ok { + return errors.New("etcdserver: request timed out") + } + return c.List(ctx, list, opts...) + }, + }) + if _, err := r.cleanupMariaDBBackup(context.Background(), cleanupBackup(cleanupDumpName)); err == nil || len(bucket.removed) != 0 || !operatorBackupExists(t, r.Client) { + t.Fatalf("err=%v removed=%v, want the error returned and nothing deleted", err, bucket.removed) + } + }) t.Run("finished restores and restores of other Backups do not hold it", func(t *testing.T) { bucket := &fakeS3Bucket{} r, _ := cleanupReconciler(t, bucket, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace(), From 5b6708c7f94342f9a6582daf7de4793c488f1ae5 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 18:04:35 +0300 Subject: [PATCH 445/518] fix(platform): leave tenant-root's ingress empty when the root has none cozystack-basics wrote _namespace.ingress: tenant-root whatever the root Tenant's spec.ingress said, so applications in tenant-root were told an ingress-nginx runs there when nothing deploys it. SeaweedFS then waited forever on ingress-nginx-system, and the platform backup bucket with it. Every other tenant already derives the key and leaves it empty without ingress. cozystack-basics cannot read the root Tenant, so the platform carries its ingress state the way it carries monitoring: ingress.rootEnabled reaches the Secret as _cluster.ingress-enabled and gates the key. The default keeps the value existing clusters render, and a Secret written before the key existed keeps it too. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- packages/core/platform/templates/apps.yaml | 4 ++ .../apps_ingress_enabled_wiring_test.yaml | 35 ++++++++++ packages/core/platform/values.yaml | 13 ++++ .../templates/cozystack-values-secret.yaml | 6 ++ .../tests/cozystack-values-secret_test.yaml | 66 ++++++++++++++++--- 5 files changed, 116 insertions(+), 8 deletions(-) create mode 100644 packages/core/platform/tests/apps_ingress_enabled_wiring_test.yaml diff --git a/packages/core/platform/templates/apps.yaml b/packages/core/platform/templates/apps.yaml index 4a0f0ae89..db6c7109b 100644 --- a/packages/core/platform/templates/apps.yaml +++ b/packages/core/platform/templates/apps.yaml @@ -261,6 +261,10 @@ stringData: null/omitted in custom values. Missing key -> the true default, matching the chart values.yaml. */}} monitoring-enabled: {{ dig "rootEnabled" true (.Values.monitoring | default dict) | quote }} + {{- /* Whether the root tenant runs its own ingress-nginx + (Tenant/root spec.ingress). `dig`, as for monitoring-enabled + above, so an explicit false survives. */}} + ingress-enabled: {{ dig "rootEnabled" true (.Values.ingress | default dict) | quote }} {{- /* The rate each shared-central tracing tenant's collector holds it to. Only written when set: every release in every tenant reads this Secret, and a key appearing would upgrade all of them. */}} diff --git a/packages/core/platform/tests/apps_ingress_enabled_wiring_test.yaml b/packages/core/platform/tests/apps_ingress_enabled_wiring_test.yaml new file mode 100644 index 000000000..68f6a8168 --- /dev/null +++ b/packages/core/platform/tests/apps_ingress_enabled_wiring_test.yaml @@ -0,0 +1,35 @@ +suite: apps.yaml ingress.rootEnabled → _cluster wiring +templates: + - templates/apps.yaml + +release: + name: cozy-platform + namespace: cozy-system + +# ingress.rootEnabled records whether the root tenant runs its own +# ingress-nginx (Tenant/root spec.ingress). It reaches cozystack-basics through +# _cluster.ingress-enabled in the cozystack-values Secret, where it decides +# whether tenant-root's _namespace.ingress names tenant-root or stays empty. + +tests: + - it: writes ingress-enabled true by default + asserts: + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^\s*ingress-enabled:\s*"true"' + + - it: writes ingress-enabled false when ingress.rootEnabled is disabled + set: + ingress.rootEnabled: false + asserts: + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^\s*ingress-enabled:\s*"false"' + + - it: falls back to true (no panic) when ingress is null + set: + ingress: null + asserts: + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^\s*ingress-enabled:\s*"true"' diff --git a/packages/core/platform/values.yaml b/packages/core/platform/values.yaml index 23ed9a448..f2e35b5e5 100644 --- a/packages/core/platform/values.yaml +++ b/packages/core/platform/values.yaml @@ -669,6 +669,19 @@ gateway: - cozy-monitoring - cozy-linstor-gui - default +# Platform ingress configuration. +ingress: + # Whether the root tenant (tenant-root) runs its own ingress-nginx, i.e. + # Tenant/root spec.ingress=true. It decides whether tenant-root's + # _namespace.ingress names tenant-root or stays empty, which is what + # applications there (SeaweedFS waiting on ingress-nginx-system, the + # ingressClassName of their Ingresses) and every tenant inheriting from the + # root read. Nothing in the charts sets Tenant/root spec.ingress, so keep + # this in sync with it by hand; changing it re-renders cozystack-basics and + # the releases reading the cozystack-values Secret. The default is true only + # to keep the value existing clusters already render. + rootEnabled: true + # Platform monitoring configuration. monitoring: # Whether the root tenant (tenant-root) hosts the platform diff --git a/packages/system/cozystack-basics/templates/cozystack-values-secret.yaml b/packages/system/cozystack-basics/templates/cozystack-values-secret.yaml index 560bc4598..72f24e1f9 100644 --- a/packages/system/cozystack-basics/templates/cozystack-values-secret.yaml +++ b/packages/system/cozystack-basics/templates/cozystack-values-secret.yaml @@ -15,7 +15,13 @@ stringData: _namespace: host: {{ index .Values._cluster "root-host" | quote }} etcd: tenant-root + {{- /* Without ingress on the root Tenant there is no + ingress-nginx-system in tenant-root, so the key stays empty, as + it does for a sub-tenant with no ingress anywhere above it. A + missing key keeps tenant-root for Secrets written before it. */ -}} + {{- if eq (toString (index .Values._cluster "ingress-enabled" | default "true")) "true" }} ingress: tenant-root + {{- end }} {{- /* `gateway` is conditional, the other four `_namespace` keys are unconditional. The `apps/tenant` chart gates tenant-root via `ne (include "tenant.name" .) "tenant-root"`, so the namespace.yaml diff --git a/packages/system/cozystack-basics/tests/cozystack-values-secret_test.yaml b/packages/system/cozystack-basics/tests/cozystack-values-secret_test.yaml index 28a88b0e1..33db1dcd5 100644 --- a/packages/system/cozystack-basics/tests/cozystack-values-secret_test.yaml +++ b/packages/system/cozystack-basics/tests/cozystack-values-secret_test.yaml @@ -19,14 +19,16 @@ set: # never reaches tenant-root — this Secret is the only place that # value gets set for the root tenant. # -# Four of the five `_namespace.*` keys (`etcd`, `ingress`, -# `monitoring`, `seaweedfs`) are always rendered, because their -# dependent platform apps are always installable in tenant-root. -# `gateway` is conditional on `_cluster.gateway-enabled`: when the -# platform-level flag is false (the default), Gateway is not -# installed, and writing `gateway: tenant-root` would point apps at a -# Gateway that doesn't exist. The fixtures below pin both halves of -# that contract. +# Three of the five `_namespace.*` keys (`etcd`, `monitoring`, +# `seaweedfs`) are always rendered, because their dependent platform +# apps are always installable in tenant-root. `gateway` is conditional +# on `_cluster.gateway-enabled`: when the platform-level flag is false +# (the default), Gateway is not installed, and writing +# `gateway: tenant-root` would point apps at a Gateway that doesn't +# exist. `ingress` is conditional on `_cluster.ingress-enabled` for the +# same reason: with ingress off on the root Tenant there is no +# ingress-nginx-system in tenant-root, and seaweedfs would wait on it +# forever. The fixtures below pin both halves of each contract. tests: - it: omits _namespace.gateway when _cluster.gateway-enabled is missing (default) asserts: @@ -52,6 +54,54 @@ tests: path: stringData["values.yaml"] pattern: '(?m)^\s+gateway: tenant-root$' + - it: emits _namespace.ingress=tenant-root when _cluster.ingress-enabled is missing (Secrets written before the key existed) + asserts: + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^\s+ingress: tenant-root$' + + - it: emits _namespace.ingress=tenant-root when _cluster.ingress-enabled is "true" + set: + _cluster: + ingress-enabled: "true" + asserts: + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^\s+ingress: tenant-root$' + + - it: emits _namespace.ingress=tenant-root when _cluster.ingress-enabled is a bool true + set: + _cluster: + ingress-enabled: true + asserts: + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^\s+ingress: tenant-root$' + + - it: omits _namespace.ingress when _cluster.ingress-enabled is "false" + set: + _cluster: + ingress-enabled: "false" + asserts: + - notMatchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^\s+ingress:' + + - it: keeps _namespace.{etcd,monitoring,seaweedfs} when ingress is off + set: + _cluster: + ingress-enabled: "false" + asserts: + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^\s+etcd: tenant-root$' + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^\s+monitoring: tenant-root$' + - matchRegex: + path: stringData["values.yaml"] + pattern: '(?m)^\s+seaweedfs: tenant-root$' + - it: keeps _namespace.{etcd,ingress,monitoring,seaweedfs} unconditional when Gateway is off — regression guard for a refactor that accidentally gates them on the same flag asserts: - matchRegex: From b26001b6f916e60a6939048b2309d84773df6300 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 18:04:36 +0300 Subject: [PATCH 446/518] test(seaweedfs): pin the ingress-nginx dependency to the namespace's ingress seaweedfs-system waits on ingress-nginx-system only when _namespace.ingress names its own namespace. With tenant-root's key now empty when the root Tenant runs no ingress, an empty value has to drop the dependency, and a value inherited from a parent must not add it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../tests/ingress_dependency_test.yaml | 61 +++++++++++++++++++ 1 file changed, 61 insertions(+) create mode 100644 packages/extra/seaweedfs/tests/ingress_dependency_test.yaml diff --git a/packages/extra/seaweedfs/tests/ingress_dependency_test.yaml b/packages/extra/seaweedfs/tests/ingress_dependency_test.yaml new file mode 100644 index 000000000..654918b92 --- /dev/null +++ b/packages/extra/seaweedfs/tests/ingress_dependency_test.yaml @@ -0,0 +1,61 @@ +suite: seaweedfs waits for ingress-nginx only in a namespace that runs it +templates: + - templates/seaweedfs.yaml + +release: + name: seaweedfs + namespace: tenant-root + +set: + topology: Simple + replicationFactor: 2 + _cluster: + issuer-name: letsencrypt-prod + +tests: + - it: depends on ingress-nginx-system when the namespace runs its own ingress + documentSelector: + path: metadata.name + value: seaweedfs-system + set: + _namespace: + host: example.org + ingress: tenant-root + asserts: + - contains: + path: spec.dependsOn + content: + name: ingress-nginx-system + namespace: tenant-root + + - it: does not depend on ingress-nginx-system when the root tenant has no ingress + documentSelector: + path: metadata.name + value: seaweedfs-system + set: + _namespace: + host: example.org + ingress: "" + asserts: + - notContains: + path: spec.dependsOn + content: + name: ingress-nginx-system + namespace: tenant-root + + - it: does not depend on ingress-nginx-system when ingress lives in a parent namespace + release: + namespace: tenant-acme + documentSelector: + path: metadata.name + value: seaweedfs-system + set: + _namespace: + host: acme.example.org + ingress: tenant-root + asserts: + - notContains: + path: spec.dependsOn + content: + name: ingress-nginx-system + namespace: tenant-acme From c61e6a842e2ecd79f76574cf671d2649560e72f2 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 18:20:38 +0300 Subject: [PATCH 447/518] fix(kubernetes): skip the proxied addon Ingress in a tenant without ingress An empty _namespace.ingress means no ingress-nginx serves the tenant, which tenant-root now reports when the root Tenant has ingress off. The Ingress would carry ingressClassName "", which the API rejects, and the release would fail to install where it used to leave an unserved Ingress behind. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- packages/apps/kubernetes/templates/ingress.yaml | 2 +- .../tests/ingress_proxied_gate_test.yaml | 17 +++++++++++++++++ 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/packages/apps/kubernetes/templates/ingress.yaml b/packages/apps/kubernetes/templates/ingress.yaml index 9b35f9683..9b88a7fce 100644 --- a/packages/apps/kubernetes/templates/ingress.yaml +++ b/packages/apps/kubernetes/templates/ingress.yaml @@ -3,7 +3,7 @@ on an etcd-less tenant the chart collapses to the awaiting-etcd beacon, so no addon controller exists and the Service would select nothing. */}} {{- if and .Values.addons.ingressNginx.enabled .Values._namespace.etcd (eq .Values.addons.ingressNginx.exposeMethod "Proxied") }} -{{- if .Values.addons.ingressNginx.hosts }} +{{- if and .Values.addons.ingressNginx.hosts $ingress }} --- apiVersion: networking.k8s.io/v1 kind: Ingress diff --git a/packages/apps/kubernetes/tests/ingress_proxied_gate_test.yaml b/packages/apps/kubernetes/tests/ingress_proxied_gate_test.yaml index 9583b16a3..75e9349a8 100644 --- a/packages/apps/kubernetes/tests/ingress_proxied_gate_test.yaml +++ b/packages/apps/kubernetes/tests/ingress_proxied_gate_test.yaml @@ -79,3 +79,20 @@ tests: asserts: - hasDocuments: count: 0 + + # An empty _namespace.ingress means no ingress-nginx serves this tenant, and + # the API rejects the ingressClassName "" the Ingress would carry. + - it: renders the Service but no Ingress when the tenant has no ingress + set: + _namespace: + ingress: "" + addons: + ingressNginx: + enabled: true + exposeMethod: Proxied + hosts: ["app.example.com"] + asserts: + - hasDocuments: + count: 1 + - isKind: + of: Service From 275c840a8c9576ebc45acd6d7e6bc295192afcc3 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 18:20:38 +0300 Subject: [PATCH 448/518] fix(monitoring): leave Grafana's ingress out in a tenant without ingress grafana-operator creates the Ingress from spec.ingress verbatim, so an empty _namespace.ingress reached the API as ingressClassName "" and was rejected. With no ingress-nginx in the tenant there is nothing for that Ingress to attach to. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../monitoring/templates/grafana/grafana.yaml | 2 +- .../grafana_ingress_empty_class_test.yaml | 40 +++++++++++++++++++ 2 files changed, 41 insertions(+), 1 deletion(-) create mode 100644 packages/system/monitoring/tests/grafana_ingress_empty_class_test.yaml diff --git a/packages/system/monitoring/templates/grafana/grafana.yaml b/packages/system/monitoring/templates/grafana/grafana.yaml index 85517151a..2da6d2fc6 100644 --- a/packages/system/monitoring/templates/grafana/grafana.yaml +++ b/packages/system/monitoring/templates/grafana/grafana.yaml @@ -280,7 +280,7 @@ spec: path: auth.ini {{- end }} {{- end }} - {{- if ne $gatewayEnabled "true" }} + {{- if and (ne $gatewayEnabled "true") $ingress }} ingress: {{- /* Operator wildcard mode: nginx serves the default SSL cert, so skip per-host ACME. */}} {{- if not $wildcardSecret }} diff --git a/packages/system/monitoring/tests/grafana_ingress_empty_class_test.yaml b/packages/system/monitoring/tests/grafana_ingress_empty_class_test.yaml new file mode 100644 index 000000000..9e6b0350c --- /dev/null +++ b/packages/system/monitoring/tests/grafana_ingress_empty_class_test.yaml @@ -0,0 +1,40 @@ +suite: grafana ingress needs an ingress class to name +templates: + - templates/grafana/grafana.yaml + +release: + name: monitoring + namespace: tenant-root + +# An empty _namespace.ingress means no ingress-nginx serves this tenant, and +# the API rejects the ingressClassName "" the operator would put on the +# Ingress it creates from spec.ingress. + +tests: + - it: leaves spec.ingress out when the tenant has no ingress + set: + _cluster: + issuer-name: letsencrypt-prod + gateway-enabled: "false" + _namespace: + host: example.org + ingress: "" + asserts: + - equal: + path: kind + value: Grafana + - notExists: + path: spec.ingress + + - it: keeps spec.ingress when the tenant has an ingress + set: + _cluster: + issuer-name: letsencrypt-prod + gateway-enabled: "false" + _namespace: + host: example.org + ingress: tenant-root + asserts: + - equal: + path: spec.ingress.spec.ingressClassName + value: tenant-root From 67fcadf6891ac783b93e094d8b43ab6866d9a6a1 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Wed, 11 Nov 2026 18:20:38 +0300 Subject: [PATCH 449/518] fix(seaweedfs): disable the S3 Ingress in a tenant without ingress The S3 Ingress kept rendering with no ingress-nginx to serve it. Now that seaweedfs-system no longer waits on ingress-nginx-system there, the release completes and the unserved Ingress would stay behind. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../extra/seaweedfs/templates/seaweedfs.yaml | 1 + .../tests/ingress_dependency_test.yaml | 29 +++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/packages/extra/seaweedfs/templates/seaweedfs.yaml b/packages/extra/seaweedfs/templates/seaweedfs.yaml index 27e5557e0..694ed5c18 100644 --- a/packages/extra/seaweedfs/templates/seaweedfs.yaml +++ b/packages/extra/seaweedfs/templates/seaweedfs.yaml @@ -338,6 +338,7 @@ spec: s3: replicas: {{ .Values.s3.replicas }} ingress: + enabled: {{ not (empty $ingress) }} className: {{ $ingress }} host: {{ .Values.host | default (printf "s3.%s" $host) }} annotations: diff --git a/packages/extra/seaweedfs/tests/ingress_dependency_test.yaml b/packages/extra/seaweedfs/tests/ingress_dependency_test.yaml index 654918b92..55acf8198 100644 --- a/packages/extra/seaweedfs/tests/ingress_dependency_test.yaml +++ b/packages/extra/seaweedfs/tests/ingress_dependency_test.yaml @@ -59,3 +59,32 @@ tests: content: name: ingress-nginx-system namespace: tenant-acme + + - it: disables the S3 Ingress when the namespace has no ingress + documentSelector: + path: metadata.name + value: seaweedfs-system + set: + _namespace: + host: example.org + ingress: "" + asserts: + - equal: + path: spec.values.seaweedfs.s3.ingress.enabled + value: false + + - it: keeps the S3 Ingress when the namespace has an ingress + documentSelector: + path: metadata.name + value: seaweedfs-system + set: + _namespace: + host: example.org + ingress: tenant-root + asserts: + - equal: + path: spec.values.seaweedfs.s3.ingress.enabled + value: true + - equal: + path: spec.values.seaweedfs.s3.ingress.className + value: tenant-root From 6a937653515e8ea7bd2a4e90443538b5662e0d34 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 07:53:34 +0000 Subject: [PATCH 450/518] fix(seaweedfs): bump the COSI driver to v0.4.0 for the lost-grant fix seaweedfs-cosi-driver v0.3.1 grants access by rewriting the legacy /etc/iam/identity.json. Each S3 gateway migrates that file to per-identity files and renames it away whenever it changes, so a grant written while a gateway is migrating is lost: the BucketAccess reports accessGranted and its Secret is written, but every gateway rejects the key with InvalidAccessKeyId and never recovers. Users added to a Bucket in one change lose a key regularly. v0.4.0 also writes the per-identity file on grant. The rest of v0.4.0 does not reach the platform. The readwrite-governance-bypass access policy is opt-in and no BucketAccessClass uses it. The refusal to delete a bucket that still holds locked objects applies only to object-lock buckets, and their classes are Retain, so the sidecar never calls DriverDeleteBucket for them. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag | 2 +- packages/system/seaweedfs/tests/cosi_image_test.yaml | 6 +++--- packages/system/seaweedfs/values.yaml | 5 ++++- 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag b/packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag index 49f252950..bf4084277 100644 --- a/packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag +++ b/packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag @@ -1 +1 @@ -ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1 +ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.4.0 diff --git a/packages/system/seaweedfs/tests/cosi_image_test.yaml b/packages/system/seaweedfs/tests/cosi_image_test.yaml index 7840d200d..d43aa176e 100644 --- a/packages/system/seaweedfs/tests/cosi_image_test.yaml +++ b/packages/system/seaweedfs/tests/cosi_image_test.yaml @@ -7,7 +7,7 @@ suite: seaweedfs COSI driver image pin # BucketAccess and ignores the accessPolicy parameter. The chart ships a # -readonly BucketAccessClass with parameters.accessPolicy: readonly, so on # v0.1.2 a credential meant to be read-only is silently granted read-write. -# The package values.yaml overrides cosi.image to v0.3.1, which honours +# The package values.yaml overrides cosi.image to v0.4.0, which honours # accessPolicy: readonly. # # This test does not set the image itself; it relies on the package values.yaml @@ -25,7 +25,7 @@ release: namespace: cozy-seaweedfs tests: - - it: pins the cosi-driver container to v0.3.1 (read-only access enforcement) + - it: pins the cosi-driver container to v0.4.0 (read-only access enforcement) template: charts/seaweedfs/templates/cosi/cosi-deployment.yaml set: seaweedfs.cosi.enabled: true @@ -35,4 +35,4 @@ tests: value: seaweedfs-cosi-driver - equal: path: spec.template.spec.containers[0].image - value: ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1 + value: ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.4.0 diff --git a/packages/system/seaweedfs/values.yaml b/packages/system/seaweedfs/values.yaml index bce78c47f..8bde04233 100644 --- a/packages/system/seaweedfs/values.yaml +++ b/packages/system/seaweedfs/values.yaml @@ -303,7 +303,10 @@ seaweedfs: # S3 actions for every BucketAccess and ignores the accessPolicy parameter. # v0.3.x honours accessPolicy: readonly, so a -readonly BucketAccessClass # actually issues read-only (Read,List) credentials instead of read-write. - image: "ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.3.1" + # v0.4.0 also writes each grant to the per-identity IAM file: v0.3.1 + # rewrites only the legacy identity.json, which the S3 gateways migrate + # concurrently, so a grant could be lost for good. + image: "ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.4.0" podLabels: policy.cozystack.io/allow-to-apiserver: "true" driverName: "seaweedfs.objectstorage.k8s.io" From 5a95bcdd64e042701bb448f35d33536cfbd1ef9c Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 12 Nov 2026 11:10:43 +0300 Subject: [PATCH 451/518] fix(clickhouse): keep a run's archive name within the old name's length The archive name is part of every object key, a key is capped at 1024 bytes, and the legacy s3PathOverride prefix has no length limit. The full UID made the name 37 characters longer than before, so a prefix that fit the old name pushed the manifest key past the cap and failed the backup. The run's name now drops "clickhouse-" and the "T" and "Z" of the timestamp and spends the room on the first 12 hex digits of the UID, so it is exactly as long as the old name for every application. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- hack/altinity-backup-strategy.bats | 2 +- .../altinitystrategy_controller.go | 22 +++++++++-- .../altinitystrategy_controller_test.go | 39 ++++++++++++++++--- 3 files changed, 53 insertions(+), 10 deletions(-) diff --git a/hack/altinity-backup-strategy.bats b/hack/altinity-backup-strategy.bats index 97ec869cc..070b6bad5 100644 --- a/hack/altinity-backup-strategy.bats +++ b/hack/altinity-backup-strategy.bats @@ -392,7 +392,7 @@ run_script() { remote_rows clickhouse-myapp-20260304T040000Z clickhouse-myapp-20260304T050607Z \ "clickhouse-myapp-20260304T050700Z|broken (can't stat metadata.json)" \ clickhouse-myapp-20260304T050710Z-copy \ - clickhouse-myapp-20260304T050720Z-0f3c9a1e-5b7d-4c2a-9e8f-1a2b3c4d5e6f \ + myapp-20260304050720-0f3c9a1e5b7d \ clickhouse-myapp-20260304T060000Z > "$tmp/state/remote" out=$(run_script "$tmp" 2>&1) grep -qxF 'POST http://chi-clickhouse-myapp-clickhouse-0-0.tenant-x.svc:7171/backup/restore_remote/clickhouse-myapp-20260304T050607Z' "$tmp/state/requests" diff --git a/internal/backupcontroller/altinitystrategy_controller.go b/internal/backupcontroller/altinitystrategy_controller.go index c29162474..ef91dc177 100644 --- a/internal/backupcontroller/altinitystrategy_controller.go +++ b/internal/backupcontroller/altinitystrategy_controller.go @@ -57,6 +57,12 @@ const ( // archive relies on. altinityArchiveTimeLayout = "20060102T150405Z" + // altinityArchiveRunTimeLayout and altinityArchiveUIDLength shape the + // archive name of a run; see altinityArchiveName for why they are this + // short. + altinityArchiveRunTimeLayout = "20060102150405" + altinityArchiveUIDLength = 12 + // Polling cadence for the Altinity Job lifecycle. Mirrors the CNPG / // Velero strategy cadence so behaviour is uniform across drivers. altinityPollInterval = 5 * time.Second @@ -122,11 +128,19 @@ func backupParameters(b *backupsv1alpha1.Backup) map[string]string { // one-second resolution, and two Plans on one schedule start their runs in // the same second; the strategy takes a finished upload of its archive as an // earlier attempt of its own run, so a shared name would let one run report -// the other's archive. The UID goes after the timestamp so names still sort -// chronologically, and the restore of a Backup that records no archive, which -// accepts only clickhouse--, never picks one of these. +// the other's archive. +// +// The name is no longer than altinityArchiveNameBeforeUIDs, because it sits in +// every object key, a key is capped at 1024 bytes, and the legacy +// s3PathOverride prefix has no length limit of its own: the UID takes the room +// that dropping "clickhouse-", "T" and "Z" frees. It goes after the timestamp +// so names still sort chronologically, and the restore of a Backup that +// records no archive, which accepts only clickhouse--, +// never picks one of these. func altinityArchiveName(j *backupsv1alpha1.BackupJob) string { - return altinityArchiveNameBeforeUIDs(j) + "-" + string(j.UID) + uid := strings.ReplaceAll(string(j.UID), "-", "") + uid = uid[:min(len(uid), altinityArchiveUIDLength)] + return fmt.Sprintf("%s-%s-%s", j.Spec.ApplicationRef.Name, j.Status.StartedAt.UTC().Format(altinityArchiveRunTimeLayout), uid) } // altinityArchiveNameBeforeUIDs is the name a controller that did not append diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index 15a450e59..7f2c906aa 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -735,11 +735,36 @@ func TestAltinityArchiveName_FromStartedAtInUTC(t *testing.T) { Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: newAltinityClickHouseRef("ch-test")}, Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt}, } - if got, want := altinityArchiveName(j), "clickhouse-ch-test-20260304T050607Z-0f3c9a1e-5b7d-4c2a-9e8f-1a2b3c4d5e6f"; got != want { + if got, want := altinityArchiveName(j), "ch-test-20260304050607-0f3c9a1e5b7d"; got != want { t.Errorf("altinityArchiveName = %q, want %q", got, want) } } +// TestAltinityArchiveName_NoLongerThanTheNameWithoutUID pins the S3 key +// budget: the archive name is part of every object key, a key may not exceed +// 1024 bytes, and the legacy s3PathOverride prefix is unbounded, so a prefix +// that fit the name without the UID must still fit this one. +func TestAltinityArchiveName_NoLongerThanTheNameWithoutUID(t *testing.T) { + startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC)) + for n := 1; n <= 63; n++ { + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{UID: "0f3c9a1e-5b7d-4c2a-9e8f-1a2b3c4d5e6f"}, + Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: newAltinityClickHouseRef(strings.Repeat("a", n))}, + Status: backupsv1alpha1.BackupJobStatus{StartedAt: &startedAt}, + } + name, legacy := altinityArchiveName(j), altinityArchiveNameBeforeUIDs(j) + if len(name) > len(legacy) { + t.Errorf("application name of %d characters: archive %q is longer than %q", n, name, legacy) + } + // The manifest key of a prefix that left the name without the UID + // exactly at the limit. + prefix := strings.Repeat("p", 1024-len("/"+legacy+"/metadata.json")) + if key := prefix + "/" + name + "/metadata.json"; len(key) > 1024 { + t.Errorf("application name of %d characters: manifest key is %d bytes", n, len(key)) + } + } +} + // TestAltinityArchiveName_DistinctForRunsStartedInTheSameSecond pins that two // BackupJobs of one application started within the same second, as two Plans // on the same schedule are, write different archives. With a shared name, the @@ -760,7 +785,7 @@ func TestAltinityArchiveName_DistinctForRunsStartedInTheSameSecond(t *testing.T) t.Fatalf("two runs started in the same second share archive %q", daily) } for _, name := range []string{daily, weekly} { - if !strings.HasPrefix(name, "clickhouse-ch-test-20260304T050607Z-") { + if !strings.HasPrefix(name, "ch-test-20260304050607-") { t.Errorf("archive %q does not start with the application and start time, so names stop sorting chronologically", name) } } @@ -793,7 +818,7 @@ func TestReconcileAltinity_RecordsTheArchiveTheJobWasGiven(t *testing.T) { } r, _, _ := newAltinityTestEnv(t, app, clientfake.NewClientBuilder().WithObjects(backupJob, strategy)) ctx := context.Background() - const want = "clickhouse-ch-test-20260304T050607Z-test-bj-uid" + const want = "ch-test-20260304050607-testbjuid" if _, err := r.reconcileAltinity(ctx, backupJob, resolved); err != nil { t.Fatalf("reconcileAltinity() creating the Job: %v", err) @@ -844,6 +869,10 @@ func TestReconcileAltinity_RecordsTheArchiveOfAJobNamedBeforeUIDs(t *testing.T) {name: "the name without the UID", archive: legacy, want: legacy}, {name: "a longer name that starts with it", archive: legacy + "-copy", want: ""}, {name: "a longer name that ends with it", archive: "copy-" + legacy, want: ""}, + {name: "a longer name that continues it after a period", archive: legacy + ".copy", want: ""}, + {name: "a longer name that leads into it with a period", archive: "copy." + legacy, want: ""}, + {name: "a longer name that continues it after an underscore", archive: legacy + "_copy", want: ""}, + {name: "a longer name that leads into it with an underscore", archive: "copy_" + legacy, want: ""}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { @@ -977,13 +1006,13 @@ func TestReconcileAltinity_RecordsNoArchiveTheJobWasNotGiven(t *testing.T) { // A Backup recording this run's own archive is the one an earlier reconcile // created before its status write was lost, and is adopted. func TestReconcileAltinity_BackupNameAlreadyTaken(t *testing.T) { - const thisRun = "clickhouse-ch-test-20260305T050607Z-test-bj-uid" + const thisRun = "ch-test-20260305050607-testbjuid" cases := []struct { name string existing string // archive the existing Backup records; empty for none wantSucceeded bool }{ - {name: "earlier run's Backup records another archive", existing: "clickhouse-ch-test-20260304T050607Z-earlier-bj-uid"}, + {name: "earlier run's Backup records another archive", existing: "ch-test-20260304050607-earlierbjuid"}, {name: "earlier run's Backup records no archive", existing: ""}, {name: "this run's Backup from a reconcile whose status write was lost", existing: thisRun, wantSucceeded: true}, } From c63e83466acb6129adba9f414d8d62165e34c0cc Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 14:29:06 +0500 Subject: [PATCH 452/518] ci(e2e): raise the E2E job limit from 215 to 265 minutes A green run of the full Chainsaw suite no longer fits in 215 minutes: the last four nightly runs were cancelled at the limit, and pull requests that select the full suite end the same way while every suite that got to run passed. All lanes that run the suite keep one figure, so a change that passes on a pull request is not cut off on nightly or on a tag. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- .github/workflows/e2e-fork.yaml | 2 +- .github/workflows/e2e-tag.yaml | 4 ++-- .github/workflows/nightly.yaml | 4 ++-- .github/workflows/pull-requests.yaml | 6 +++--- hack/cozyreport.sh | 2 +- hack/cozytest.sh | 2 +- hack/e2e-chainsaw/.chainsaw.yaml | 4 ++-- hack/e2e-chainsaw/computeplane/chainsaw-test.yaml | 2 +- hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml | 3 ++- 9 files changed, 15 insertions(+), 14 deletions(-) diff --git a/.github/workflows/e2e-fork.yaml b/.github/workflows/e2e-fork.yaml index f988632e9..65b78230d 100644 --- a/.github/workflows/e2e-fork.yaml +++ b/.github/workflows/e2e-fork.yaml @@ -543,7 +543,7 @@ jobs: # Covers a green full suite. It does not cover a tenant-Kubernetes suite # running into its node-join deadline, so a red there can end as a # cancelled job. Held to one figure across every lane that runs the suite. - timeout-minutes: 215 + timeout-minutes: 265 permissions: contents: read packages: read diff --git a/.github/workflows/e2e-tag.yaml b/.github/workflows/e2e-tag.yaml index f390077d0..d9bba8559 100644 --- a/.github/workflows/e2e-tag.yaml +++ b/.github/workflows/e2e-tag.yaml @@ -137,7 +137,7 @@ jobs: # Covers a green full suite. It does not cover a tenant-Kubernetes suite # running into its node-join deadline, so a red there can end as a # cancelled job. Held to one figure across every lane that runs the suite. - timeout-minutes: 215 + timeout-minutes: 265 permissions: contents: read checks: write # Breakpoint action updates its dedicated "Breakpoint Open" Check Run. @@ -292,7 +292,7 @@ jobs: # collect-report packages cozyreport plus per-failed-test crust-gather # snapshots captured before Chainsaw cleanup. # - # A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz` + # A ceiling of its own, not the job's 265 minutes. `Upload cozyreport.tgz` # is the next step and the collector writes its tarball at the very end of # its run, so a read that never returns spends what is left of the job # budget, the job is cancelled, and the upload never runs -- the artifact diff --git a/.github/workflows/nightly.yaml b/.github/workflows/nightly.yaml index 495ccec38..57cce6790 100644 --- a/.github/workflows/nightly.yaml +++ b/.github/workflows/nightly.yaml @@ -250,7 +250,7 @@ jobs: # Covers a green full suite. It does not cover a tenant-Kubernetes suite # running into its node-join deadline, so a red there can end as a # cancelled job. Held to one figure across every lane that runs the suite. - timeout-minutes: 215 + timeout-minutes: 265 permissions: contents: read packages: read @@ -385,7 +385,7 @@ jobs: path: _out/chainsaw-report.xml if-no-files-found: ignore - # A ceiling of its own, not the job's 215 minutes. `Upload cozyreport.tgz` + # A ceiling of its own, not the job's 265 minutes. `Upload cozyreport.tgz` # is the next step and the collector writes its tarball at the very end of # its run, so a read that never returns spends what is left of the job # budget, the job is cancelled, and the upload never runs -- the artifact diff --git a/.github/workflows/pull-requests.yaml b/.github/workflows/pull-requests.yaml index 7d3787ff9..24ee473ef 100644 --- a/.github/workflows/pull-requests.yaml +++ b/.github/workflows/pull-requests.yaml @@ -953,7 +953,7 @@ jobs: # Covers a green full suite. It does not cover a tenant-Kubernetes suite # running into its node-join deadline, so a red there can end as a # cancelled job. Held to one figure across every lane that runs the suite. - timeout-minutes: 215 + timeout-minutes: 265 permissions: contents: read packages: read @@ -1224,7 +1224,7 @@ jobs: # ▸ Collect debug information (always runs) # - # A ceiling of its own, not the job's 215 minutes. `Upload + # A ceiling of its own, not the job's 265 minutes. `Upload # cozyreport.tgz` is the next step and the collector writes its tarball # at the very end of its run, so a read that never returns spends what # is left of the job budget, the job is cancelled, and the upload never @@ -1235,7 +1235,7 @@ jobs: # 30 minutes is measured against the step: it takes 3-42s across sampled # runs, on failed e2e jobs as well as green ones. It RESERVES nothing -- # `timeout-minutes` caps a step, it does not carve budget out of the job, - # so a suite that has already burned 210 of the 215 leaves this step five + # so a suite that has already burned 260 of the 265 leaves this step five # minutes and the upload is cancelled with everything else. What the cap # buys is that a hung read cannot be the reason. # `continue-on-error` because the ceiling must not change what the job diff --git a/hack/cozyreport.sh b/hack/cozyreport.sh index 0f20b1261..0e6c12b15 100755 --- a/hack/cozyreport.sh +++ b/hack/cozyreport.sh @@ -121,7 +121,7 @@ # the number; do not expect to find it written here. # # TWO ceilings sit outside it, and they fail differently. The job is capped at -# `timeout-minutes: 215`, and reaching that cancels every step still to come -- +# `timeout-minutes: 265`, and reaching that cancels every step still to come -- # the artifact upload among them. The step running this script is capped at 30 of # its own, and reaching THAT ends only this step: the uploads after it still run, # and `continue-on-error` on that step keeps the overrun out of the job's result, diff --git a/hack/cozytest.sh b/hack/cozytest.sh index 02a40f383..257bdcd08 100755 --- a/hack/cozytest.sh +++ b/hack/cozytest.sh @@ -208,7 +208,7 @@ _cozy_on_exit() { # capture would read as a complete one listing fewer crash-loops than # actually occurred. This handler's whole worst case is ~1350s (330 + 390 # + 630, each leg including its 30s kill grace) against a job capped at - # 215 minutes with no per-step timeout — + # 265 minutes with no per-step timeout — # deliberate, for the same reason recorded in hack/e2e-chainsaw/.chainsaw.yaml: # a bounded, honest wait beats a kill that collects nothing. if [ "$_prev_rc" -ne 0 ]; then diff --git a/hack/e2e-chainsaw/.chainsaw.yaml b/hack/e2e-chainsaw/.chainsaw.yaml index 87aa95619..6748a91c6 100644 --- a/hack/e2e-chainsaw/.chainsaw.yaml +++ b/hack/e2e-chainsaw/.chainsaw.yaml @@ -66,14 +66,14 @@ spec: # script: it SIGKILLs the shell alone, orphaned children hold the pipe open, # and the op overruns (#4268 measured 606s under a 300s ceiling), so the # nominal ~5m was a figure a busy catch could exceed, not a cap. Tests run - # sequentially, and the E2E job is capped at 215 timeout-minutes. A cascade + # sequentially, and the E2E job is capped at 265 timeout-minutes. A cascade # reddening many suites can therefore spend a large share of the job in # diagnostics. That is deliberate — the old ceiling # bought its speed by collecting nothing at all — and it is bounded: both legs # only approach their ceilings on a genuinely degraded cluster, a systemic # failure usually reds the install gate and skips the E2E step entirely, and # the report-upload steps are `if: always()`. That last point does NOT extend - # to a job timeout: a 215-minute cap ends the job `cancelled`, and the ~4m48s + # to a job timeout: a 265-minute cap ends the job `cancelled`, and the ~4m48s # grace window is consumed by the earlier `Collect report` step, so both # cozyreport.tgz and the job log are lost (measured on runs 29557003429 and # 29568562337). If this ever does start truncating runs, bound the number of diff --git a/hack/e2e-chainsaw/computeplane/chainsaw-test.yaml b/hack/e2e-chainsaw/computeplane/chainsaw-test.yaml index 2a02dc06e..4e0c86535 100644 --- a/hack/e2e-chainsaw/computeplane/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/computeplane/chainsaw-test.yaml @@ -35,7 +35,7 @@ # and the teardown. This is the third suite in a full run that brings up a # tenant cluster, next to kubernetes-latest and kubernetes-previous. A failure # that stalls worker join everywhere reddens all three, and their red paths -# together can exceed the E2E job's 215-minute cap, which cancels the job and +# together can exceed the E2E job's 265-minute cap, which cancels the job and # loses its log and report. The cap is sized for the red path of the # kubernetes pair only; this suite is not counted in it, and that is an # accepted risk. diff --git a/hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml b/hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml index 75ee5a2fa..695425364 100644 --- a/hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml +++ b/hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml @@ -80,7 +80,8 @@ spec: # wait, all on the passing path. Raising the ceiling to cover them is not # available -- the guard in hack/run-kubernetes-serial-console_test.bats # holds COZY_OP_CEILING equal to BOTH suites' `timeout:`, and two suites - # at 79m plus their 22m teardowns exceed the job's own 215-minute cap. + # at 79m plus their 22m teardowns take 202m, which with the install and + # the rest of a full run does not fit the job's own 265-minute cap. # What is available is a measurement: on the green run of this branch # (33724179053) the whole Test took 569.72s against this 4020s op, with # kubernetes-previous, which carries no OIDC fold, at 470.19s. The From 90f0b043a601a82b8e8a1c544c476225f7073918 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 10:23:59 +0500 Subject: [PATCH 453/518] fix(kubernetes)!: remove the talos block the chart no longer uses The kubernetes chart declared a talos block (version, schematicID, imageFactoryURL, installerRepository, registryMirrors). Workers are KubernetesNodes pools and take every one of these from their own talos values, so an operator who set them on the cluster got no error and no effect. The only reader left was a copy of the Talos to Kubernetes support-matrix check, and the pool chart already runs that check itself, against the Talos release each pool actually boots, including one an osImage override selects. The cluster's copy compared the Kubernetes version against a Talos release no node runs. The block leaves the API schema, hence the "!". Stored specs that still carry it keep rendering: the values schema does not close the object and the API does not validate spec against it. A helm-unittest case pins that. Readers outside the chart moved with it. The e2e install read talos.schematicID from this chart's values.yaml in a best-effort block that only warns, so removing the field would have stopped the image cache being seeded without failing anything; it now reads the pool chart, as the proxmox template script already does. The e2e no longer writes a talos block into the Kubernetes CR, the spec test checks the pool chart for the key it emits, the release checklist diffs the pool chart's pins as well, and the catalog guard drops the two tests that compared or read the parent's block. The null-normalization Go tests used talos keys of the real Kubernetes schema as their fixtures; they use oidc.mode, the same required-with-default shape under an optional root section, and addons.cilium.valuesOverride for the free-form case. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- api/apps/v1alpha1/kubernetes/types.go | 21 -------- .../kubernetes/zz_generated.deepcopy.go | 17 ------ docs/release/changelogs.md | 2 +- hack/check-worker-image-catalog-defaults.bats | 27 ---------- hack/e2e-chainsaw/_lib/run-kubernetes.sh | 8 +-- hack/e2e-install-cozystack.bats | 6 +-- hack/e2e-talos-image-cache.yaml | 4 +- hack/run-kubernetes-talos-spec_test.bats | 8 +-- packages/apps/kubernetes/README.md | 14 +---- .../apps/kubernetes/templates/cluster.yaml | 26 --------- .../tests/stored_talos_values_test.yaml | 41 ++++++++++++++ .../tests/talos_k8s_support_matrix_test.yaml | 53 ------------------- packages/apps/kubernetes/values.schema.json | 40 -------------- packages/apps/kubernetes/values.yaml | 19 ------- .../kubernetes-rd/cozyrds/kubernetes.yaml | 4 +- .../rest_nullstrip_admission_test.go | 4 +- .../apps/application/rest_nullstrip_test.go | 31 +++++------ 17 files changed, 76 insertions(+), 249 deletions(-) create mode 100644 packages/apps/kubernetes/tests/stored_talos_values_test.yaml delete mode 100644 packages/apps/kubernetes/tests/talos_k8s_support_matrix_test.yaml diff --git a/api/apps/v1alpha1/kubernetes/types.go b/api/apps/v1alpha1/kubernetes/types.go index df989c5c5..fc6f19746 100644 --- a/api/apps/v1alpha1/kubernetes/types.go +++ b/api/apps/v1alpha1/kubernetes/types.go @@ -41,9 +41,6 @@ type ConfigSpec struct { // Kubernetes control-plane configuration. // +kubebuilder:default:={} ControlPlane ControlPlane `json:"controlPlane"` - // Talos worker image configuration. - // +kubebuilder:default:={} - Talos Talos `json:"talos"` // OIDC authentication and per-user RBAC for the tenant kube-apiserver. See https://cozystack.io/docs/v1.6/kubernetes/oidc-authentication/ for the operator guide. // +kubebuilder:default:={} Oidc OIDC `json:"oidc"` @@ -349,24 +346,6 @@ type Scheduler struct { ResourcesPreset ResourcesPreset `json:"resourcesPreset"` } -type Talos struct { - // Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. - // +kubebuilder:default:="https://factory.talos.dev" - ImageFactoryURL string `json:"imageFactoryURL"` - // OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. - // +kubebuilder:default:="factory.talos.dev/installer" - InstallerRepository string `json:"installerRepository"` - // Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. - // +kubebuilder:default:={} - RegistryMirrors k8sRuntime.RawExtension `json:"registryMirrors"` - // Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. - // +kubebuilder:default:="ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" - SchematicID string `json:"schematicID"` - // Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`. - // +kubebuilder:default:="v1.13.6" - Version string `json:"version"` -} - type VeleroAddon struct { // Enable Velero. // +kubebuilder:default:=false diff --git a/api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go b/api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go index 6b033290f..69129b193 100644 --- a/api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go +++ b/api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go @@ -154,7 +154,6 @@ func (in *ConfigSpec) DeepCopyInto(out *ConfigSpec) { in.Proxmox.DeepCopyInto(&out.Proxmox) in.Addons.DeepCopyInto(&out.Addons) in.ControlPlane.DeepCopyInto(&out.ControlPlane) - in.Talos.DeepCopyInto(&out.Talos) in.Oidc.DeepCopyInto(&out.Oidc) } @@ -570,22 +569,6 @@ func (in *Scheduler) DeepCopy() *Scheduler { return out } -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *Talos) DeepCopyInto(out *Talos) { - *out = *in - in.RegistryMirrors.DeepCopyInto(&out.RegistryMirrors) -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Talos. -func (in *Talos) DeepCopy() *Talos { - if in == nil { - return nil - } - out := new(Talos) - in.DeepCopyInto(out) - return out -} - // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *VeleroAddon) DeepCopyInto(out *VeleroAddon) { *out = *in diff --git a/docs/release/changelogs.md b/docs/release/changelogs.md index f26ac56c5..e5d7db401 100644 --- a/docs/release/changelogs.md +++ b/docs/release/changelogs.md @@ -60,7 +60,7 @@ git diff --name-only $PREV..$NEW -- 'packages/*/charts/*/Chart.yaml' 'packages/* done # B. Out-of-Chart.yaml pins (Talos, tenant K8s, vendored images) -for path in packages/core/talos/images/talos/profiles/installer.yaml packages/core/installer/values.yaml packages/apps/kubernetes/values.yaml; do +for path in packages/core/talos/images/talos/profiles/installer.yaml packages/core/installer/values.yaml packages/apps/kubernetes/values.yaml packages/apps/kubernetes-nodes/values.yaml; do git diff $PREV..$NEW -- "$path" | head -40 done git diff --name-only $PREV..$NEW -- 'images/*/Dockerfile' diff --git a/hack/check-worker-image-catalog-defaults.bats b/hack/check-worker-image-catalog-defaults.bats index 261def920..ed695444e 100644 --- a/hack/check-worker-image-catalog-defaults.bats +++ b/hack/check-worker-image-catalog-defaults.bats @@ -148,30 +148,3 @@ NODES_SCHEMA="$REPO_ROOT/packages/apps/kubernetes-nodes/values.schema.json" esac done } - -# ----------------------------------------------------------------------------- -# Third guard, and the premise the first one rests on. The catalog is keyed off -# the kubernetes-nodes chart's `talos` defaults, but the parent kubernetes chart -# carries its own copy of the same block since the Phase 2 split, and its own -# values.yaml says to keep the two in sync. Nothing enforced that: the pool -# chart's `make update` copies files/versions.yaml and images/kubectl.tag from -# the parent and not the talos block, so a one-sided bump leaves the parent's -# support-matrix guard reasoning about one Talos release while every worker boots -# another — and the catalog check above would go on passing, because it only ever -# reads one of the two files. - -PARENT_VALUES="$REPO_ROOT/packages/apps/kubernetes/values.yaml" - -@test "the parent kubernetes chart and kubernetes-nodes agree on their Talos defaults" { - for key in version schematicID imageFactoryURL installerRepository registryMirrors; do - parent=$(yq ".talos.${key}" "$PARENT_VALUES") - pool=$(yq ".talos.${key}" "$APP_VALUES") - if [ "$parent" != "$pool" ]; then - echo "talos.${key} differs between the two charts:" >&2 - echo " packages/apps/kubernetes/values.yaml: $parent" >&2 - echo " packages/apps/kubernetes-nodes/values.yaml: $pool" >&2 - echo "both are declared 'keep in sync' and a worker pool follows the second while the parent's support-matrix guard reads the first" >&2 - return 1 - fi - done -} diff --git a/hack/e2e-chainsaw/_lib/run-kubernetes.sh b/hack/e2e-chainsaw/_lib/run-kubernetes.sh index 14e247e13..9aaf9f1f6 100644 --- a/hack/e2e-chainsaw/_lib/run-kubernetes.sh +++ b/hack/e2e-chainsaw/_lib/run-kubernetes.sh @@ -18,7 +18,7 @@ cozy_e2e_storage_class() { esac } -# talos_spec_block: emit the tenant Kubernetes CR `spec.talos` block with the Talos +# talos_spec_block: emit the tenant KubernetesNodes CR `spec.talos` block with the Talos # OS image cache (imageFactoryURL) when the in-sandbox cache is up. Prints nothing # when it falls back to the public default, so the chart default applies. # Indented for insertion directly under `spec:` in the heredoc below. @@ -5702,8 +5702,9 @@ YAML # Point worker DataVolume imports at the in-sandbox Talos OS image cache when it # is up, falling back to the public factory otherwise. Worker image pulls use - # the chart's public-registry default. Emitted right under spec: as - # `talos: { imageFactoryURL: ... }`, or nothing when the default applies. + # the chart's public-registry default. Emitted right under spec: of the + # KubernetesNodes pool below as `talos: { imageFactoryURL: ... }`, or nothing + # when the default applies. The Kubernetes CR takes no worker image field. local talos_block talos_block=$(talos_spec_block) @@ -5714,7 +5715,6 @@ metadata: name: "${test_name}" namespace: tenant-test spec: -${talos_block} ${oidc_block} addons: certManager: diff --git a/hack/e2e-install-cozystack.bats b/hack/e2e-install-cozystack.bats index 47315ac85..645a66d87 100644 --- a/hack/e2e-install-cozystack.bats +++ b/hack/e2e-install-cozystack.bats @@ -49,10 +49,10 @@ # Best-effort: never fail the suite on the band-aid. Remove once tenant workers # no longer bulk-pull the OS image from the public internet in CI. local sid ver - sid=$(yq '.talos.schematicID' packages/apps/kubernetes/values.yaml 2>/dev/null) - ver=$(yq '.talos.version' packages/apps/kubernetes/values.yaml 2>/dev/null) + sid=$(yq '.talos.schematicID' packages/apps/kubernetes-nodes/values.yaml 2>/dev/null) + ver=$(yq '.talos.version' packages/apps/kubernetes-nodes/values.yaml 2>/dev/null) if [ -z "$sid" ] || [ "$sid" = "null" ] || [ -z "$ver" ] || [ "$ver" = "null" ]; then - echo "WARNING: could not read talos.schematicID/version from values.yaml — skipping mirror (fallback to public factory)" + echo "WARNING: could not read talos.schematicID/version from packages/apps/kubernetes-nodes/values.yaml — skipping mirror (fallback to public factory)" return 0 fi # The CiliumClusterwideNetworkPolicy in the manifest is intentionally NOT applied diff --git a/hack/e2e-talos-image-cache.yaml b/hack/e2e-talos-image-cache.yaml index cac252d69..fe499d1bd 100644 --- a/hack/e2e-talos-image-cache.yaml +++ b/hack/e2e-talos-image-cache.yaml @@ -14,7 +14,7 @@ # This Deployment fetches the image ONCE at e2e-install time with an aggressive # curl retry loop (decoupling the one flaky external fetch from the many # per-worker imports), then serves it locally over plain HTTP with range support. -# Tenant `Kubernetes` CRs point `spec.talos.imageFactoryURL` at this Service +# Tenant `KubernetesNodes` CRs point `spec.talos.imageFactoryURL` at this Service # (see hack/e2e-chainsaw/_lib/run-kubernetes.sh) so every worker imports from the # warm local copy instead of the public internet. Tenant namespaces run under a # default-deny Cilium egress installed by the tenant chart, whose allow rules cover @@ -28,7 +28,7 @@ # mirror can only help, never make CI worse. # # The __SCHEMATIC_ID__ / __TALOS_VERSION__ placeholders are substituted at apply -# time by hack/e2e-install-cozystack.bats from packages/apps/kubernetes/values.yaml. +# time by hack/e2e-install-cozystack.bats from packages/apps/kubernetes-nodes/values.yaml. # # CI band-aid, not a product artifact. The durable fix is the overridable # `talos.imageFactoryURL` chart value this manifest exercises; a self-hosted diff --git a/hack/run-kubernetes-talos-spec_test.bats b/hack/run-kubernetes-talos-spec_test.bats index 9eabc119d..5901ee4c6 100644 --- a/hack/run-kubernetes-talos-spec_test.bats +++ b/hack/run-kubernetes-talos-spec_test.bats @@ -2,7 +2,7 @@ # ----------------------------------------------------------------------------- # Unit tests for talos_spec_block in hack/e2e-chainsaw/_lib/run-kubernetes.sh # -# It is the composition point for the tenant Kubernetes CR's `spec.talos`: the +# It is the composition point for the tenant KubernetesNodes CR's `spec.talos`: the # Talos OS image cache contributes imageFactoryURL when available. The result is # spliced into an unquoted heredoc directly under `spec:`, so an indentation slip # does not fail this function -- it produces a CR the API rejects, and every @@ -32,7 +32,7 @@ load test_helper # Splice a talos_spec_block result under `spec:` of a minimal CR and print the # document, so yq can be asked what the tenant apiserver would see. spec_doc() { - printf 'apiVersion: apps.cozystack.io/v1alpha1\nkind: Kubernetes\nmetadata:\n name: t\nspec:\n%s\n host: ""\n' "$1" + printf 'apiVersion: apps.cozystack.io/v1alpha1\nkind: KubernetesNodes\nmetadata:\n name: t\nspec:\n%s\n cluster: t\n' "$1" } @test "no Talos image cache renders nothing, so the chart default applies" { @@ -59,6 +59,6 @@ spec_doc() { # talos_spec_block writes imageFactoryURL by hand. If the chart ever renames # it, the CR would be rejected only at e2e time, and this file is the cheapest # place to notice. - yq -e '.talos | has("imageFactoryURL")' packages/apps/kubernetes/values.yaml >/dev/null \ - || { echo "packages/apps/kubernetes/values.yaml has no talos.imageFactoryURL; talos_spec_block would emit a key the chart rejects" >&2; exit 1; } + yq -e '.talos | has("imageFactoryURL")' packages/apps/kubernetes-nodes/values.yaml >/dev/null \ + || { echo "packages/apps/kubernetes-nodes/values.yaml has no talos.imageFactoryURL; talos_spec_block would emit a key the pool chart ignores" >&2; exit 1; } } diff --git a/packages/apps/kubernetes/README.md b/packages/apps/kubernetes/README.md index 2ac5cfc45..bd4d83a64 100644 --- a/packages/apps/kubernetes/README.md +++ b/packages/apps/kubernetes/README.md @@ -87,7 +87,7 @@ See the reference for components utilized in this service: - **`ephemeralStorage` renamed to `diskSize`** (v1.4): The `nodeGroups[name].ephemeralStorage` field has been renamed to `nodeGroups[name].diskSize`. Existing clusters are migrated transparently by platform migration 41 during the pre-upgrade hook — no manual action is required. Newly written values should use `diskSize`. Existing VMs are rolling-updated via CAPI when the new values are applied. With the Talos worker rollover in this release the field now sizes the **single system disk** (Talos OS image streamed from `factory.talos.dev`, kubelet state, containerd image cache, local-path PVCs) — the pre-Talos `disk-kubelet` PVC layout was removed. State on that single disk persists across same-VM reboots (virt-launcher restart, guest reboot, node failure); VM replacement by CAPI (e.g. nodeGroup field change, MachineHealthCheck remediation) provisions a fresh disk. -- **Air-gapped tenant workers (Phase 1 Talos rollover)**: tenant worker bootstrap moves from Ubuntu containerDisk + kubeadm to a Talos OS disk image fetched over HTTP by CDI and a `.../installer/...` installer reference. Both sources are now overridable — set `talos.imageFactoryURL` (default `https://factory.talos.dev`) to a self-hosted Image Factory, a caching mirror, or an internal HTTP file server, and `talos.installerRepository` (default `factory.talos.dev/installer`) to a mirrored registry. The Helm-rendered `*-patch-containerd` Secret that previously plumbed the platform-wide `registries.mirrors` config to tenant workers (via the kubeadm template's containerd certs.d mount) has no consumer in the Talos machineconfig, so the Secret is removed together with the platform `registries` value that fed it. A Package that still sets `registries` keeps installing, the key is ignored. Registry mirrors for tenant workers, **in-guest container-image pulls** (pods pulling from registries inside the tenant cluster) included, are configured per worker pool through `talos.registryMirrors` of the `KubernetesNodes` resource. +- **Air-gapped tenant workers (Phase 1 Talos rollover)**: tenant worker bootstrap moves from Ubuntu containerDisk + kubeadm to a Talos OS disk image fetched over HTTP by CDI and a `.../installer/...` installer reference. Both sources are now overridable on each worker pool, which is a KubernetesNodes resource: set its `talos.imageFactoryURL` (default `https://factory.talos.dev`) to a self-hosted Image Factory, a caching mirror, or an internal HTTP file server, and its `talos.installerRepository` (default `factory.talos.dev/installer`) to a mirrored registry. The `Kubernetes` resource itself has no `talos` field. The Helm-rendered `*-patch-containerd` Secret that previously plumbed the platform-wide `registries.mirrors` config to tenant workers (via the kubeadm template's containerd certs.d mount) has no consumer in the Talos machineconfig, so the Secret is removed together with the platform `registries` value that fed it. A Package that still sets `registries` keeps installing, the key is ignored. Registry mirrors for tenant workers, **in-guest container-image pulls** (pods pulling from registries inside the tenant cluster) included, are configured per worker pool through `talos.registryMirrors` of the `KubernetesNodes` resource. - **Worker MachineHealthCheck remediation is now enabled by default** (Phase 1 Talos rollover): the worker MHC `maxUnhealthy` moved from a hard-coded `0` (remediation effectively **off**) to a configurable `nodeHealthCheck.maxUnhealthy` defaulting to `"50%"`. CAPI now auto-remediates (deletes and replaces) unhealthy worker Machines while at least 50% of a group stays healthy. `"50%"` deliberately leaves headroom for transient unhealthy nodes during the kubeadm→Talos rollover and slow first boots from `factory.talos.dev`; set `nodeHealthCheck.maxUnhealthy: "0%"` to keep the previous remediation-off behaviour until your fleet is stable on Talos workers. @@ -238,18 +238,6 @@ A storage shared between tenants cannot be scoped: `Datastore.Allocate` on it is | `controlPlane.kubelet.preferredAddressTypes` | Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one. | `[]string` | `[]` | -### Talos Worker Image - -| Name | Description | Type | Value | -| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------ | -| `talos` | Talos worker image configuration. | `object` | `{}` | -| `talos.version` | Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`. | `string` | `v1.13.6` | -| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` | -| `talos.imageFactoryURL` | Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. | `string` | `https://factory.talos.dev` | -| `talos.installerRepository` | OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` | -| `talos.registryMirrors` | Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. | `object` | `{}` | - - ### OIDC | Name | Description | Type | Value | diff --git a/packages/apps/kubernetes/templates/cluster.yaml b/packages/apps/kubernetes/templates/cluster.yaml index b27582694..a5f285846 100644 --- a/packages/apps/kubernetes/templates/cluster.yaml +++ b/packages/apps/kubernetes/templates/cluster.yaml @@ -1,29 +1,3 @@ -{{- /* - Talos -> Kubernetes support matrix. Each Talos minor release supports a - bounded window of Kubernetes minor versions; running a kubelet outside - that window produces a silently broken Talos+kubelet combination that - HelmRelease cannot detect. Fail the chart render instead, so the operator - sees the mismatch in Flux/dashboards before workers roll over. - - Upstream declares the window per Talos *minor* release, and patch releases - never move it, so the lookup key is the minor derived from talos.version. - Keying on the full patch version would silently disable this guard on every - patch bump — exactly the mismatch it exists to catch. - - The matrix only fires for Talos minors it knows about; operators who - override talos.version to something outside this list take responsibility - for compatibility themselves (extend the dict or upstream a PR). - Source: https://docs.siderolabs.com/talos/v1.13/getting-started/support-matrix -*/}} -{{- $talosK8sSupportMatrix := dict - "v1.13" (list "v1.31" "v1.32" "v1.33" "v1.34" "v1.35" "v1.36") -}} -{{- $talosMinor := regexFind "^v[0-9]+\\.[0-9]+" .Values.talos.version }} -{{- with index $talosK8sSupportMatrix $talosMinor }} -{{- if not (has $.Values.version .) }} -{{- fail (printf "Kubernetes %s is not supported by Talos %s. Supported versions: %v. Bump tenant 'version' or set talos.version explicitly." $.Values.version $.Values.talos.version .) }} -{{- end }} -{{- end }} {{- $etcd := .Values._namespace.etcd }} {{- /* Substrate selects the infrastructure half of the cluster. The control plane is Kamaji either way, so this changes exactly one object — the diff --git a/packages/apps/kubernetes/tests/stored_talos_values_test.yaml b/packages/apps/kubernetes/tests/stored_talos_values_test.yaml new file mode 100644 index 000000000..f95d05d85 --- /dev/null +++ b/packages/apps/kubernetes/tests/stored_talos_values_test.yaml @@ -0,0 +1,41 @@ +suite: stored Kubernetes specs that still carry a talos block + +# The chart declares no `talos` key: worker pools are KubernetesNodes releases +# and read their own `talos`, and the Talos to Kubernetes support matrix is +# checked there. A Kubernetes spec stored before that still carries the block, +# and both the values schema (no additionalProperties: false at the root) and +# the aggregated API accept it, so the chart has to keep rendering it as if it +# were absent. + +templates: + - templates/cluster.yaml + +values: + - values/common.yaml + +release: + name: test-k8s + namespace: tenant-test + +tests: + - it: renders the same control plane for a stored spec carrying every talos key the chart used to declare + set: + talos: + version: "v1.13.6" + schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" + imageFactoryURL: "https://factory.example.com" + installerRepository: "registry.example.com/installer" + registryMirrors: + ghcr.io: + endpoints: + - https://registry.example.com + version: "v1.31" + asserts: + - hasDocuments: + count: 4 + - isKind: + of: Cluster + documentIndex: 0 + - isKind: + of: KamajiControlPlane + documentIndex: 1 diff --git a/packages/apps/kubernetes/tests/talos_k8s_support_matrix_test.yaml b/packages/apps/kubernetes/tests/talos_k8s_support_matrix_test.yaml deleted file mode 100644 index 176813dd8..000000000 --- a/packages/apps/kubernetes/tests/talos_k8s_support_matrix_test.yaml +++ /dev/null @@ -1,53 +0,0 @@ -suite: Talos to Kubernetes support-matrix guard tests - -# Each Talos minor release supports a bounded window of Kubernetes minor -# versions; running a kubelet outside that window produces a silently -# broken Talos+kubelet combination that HelmRelease cannot detect. The -# chart fails render with a clear message instead, so the operator sees -# the mismatch in Flux/dashboards before workers roll over. Source of -# truth is the $talosK8sSupportMatrix dict at the top of cluster.yaml. -# -# The dict is keyed by Talos minor, and the template derives that minor from -# talos.version, so every patch release of a supported minor stays guarded. -# -# Failure-path coverage is left as a follow-up: values.schema.json -# enforces Version enum (v1.31..v1.35), which is strictly tighter than -# the Talos v1.13 matrix entry (v1.31..v1.36), so the only way to -# trigger the fail() guard via helm-unittest is to feed a Version the -# schema rejects before the template ever runs. Schema-rejection happens -# before template execution, so the chart fail() message is unreachable -# from the test harness until either the matrix gains an exclusion the -# schema enum still allows, or helm-unittest gains a schema-skip toggle. - -templates: - - templates/cluster.yaml - -values: - - values/common.yaml - -tests: - - it: accepts a Kubernetes version inside the Talos v1.13 support window - release: - name: test-k8s - namespace: tenant-test - set: - talos: - version: "v1.13.6" - schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" - version: "v1.31" - asserts: - - hasDocuments: - count: 4 - - - it: keeps rendering for an older patch release of the same Talos minor - release: - name: test-k8s - namespace: tenant-test - set: - talos: - version: "v1.13.0" - schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" - version: "v1.31" - asserts: - - hasDocuments: - count: 4 diff --git a/packages/apps/kubernetes/values.schema.json b/packages/apps/kubernetes/values.schema.json index 3885bccf6..425d0703f 100644 --- a/packages/apps/kubernetes/values.schema.json +++ b/packages/apps/kubernetes/values.schema.json @@ -953,46 +953,6 @@ } } }, - "talos": { - "description": "Talos worker image configuration.", - "type": "object", - "default": {}, - "required": [ - "imageFactoryURL", - "installerRepository", - "registryMirrors", - "schematicID", - "version" - ], - "properties": { - "imageFactoryURL": { - "description": "Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash.", - "type": "string", - "default": "https://factory.talos.dev" - }, - "installerRepository": { - "description": "OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `\u003cinstallerRepository\u003e/\u003cschematicID\u003e:\u003cversion\u003e`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.", - "type": "string", - "default": "factory.talos.dev/installer" - }, - "registryMirrors": { - "description": "Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap.", - "type": "object", - "default": {}, - "x-kubernetes-preserve-unknown-fields": true - }, - "schematicID": { - "description": "Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.", - "type": "string", - "default": "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" - }, - "version": { - "description": "Talos release used for worker OS image and installer. Must satisfy the chart's Talos\u003c-\u003eKubernetes support matrix against the chosen `version`.", - "type": "string", - "default": "v1.13.6" - } - } - }, "oidc": { "description": "OIDC authentication and per-user RBAC for the tenant kube-apiserver. See https://cozystack.io/docs/v1.6/kubernetes/oidc-authentication/ for the operator guide.", "type": "object", diff --git a/packages/apps/kubernetes/values.yaml b/packages/apps/kubernetes/values.yaml index 007b54a53..d6708e4fc 100644 --- a/packages/apps/kubernetes/values.yaml +++ b/packages/apps/kubernetes/values.yaml @@ -342,25 +342,6 @@ controlPlane: kubelet: preferredAddressTypes: [] -## -## @section Talos Worker Image -## - -## @typedef {struct} Talos - Talos worker image configuration. -## @field {string} version - Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`. -## @field {string} schematicID - Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. -## @field {string} imageFactoryURL - Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. -## @field {string} installerRepository - OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. -## @field {object} registryMirrors - Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. - -## @param {Talos} talos - Talos worker image configuration. -talos: - version: "v1.13.6" - schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" - imageFactoryURL: "https://factory.talos.dev" - installerRepository: "factory.talos.dev/installer" - registryMirrors: {} - ## ## @section OIDC ## diff --git a/packages/system/kubernetes-rd/cozyrds/kubernetes.yaml b/packages/system/kubernetes-rd/cozyrds/kubernetes.yaml index 3be6aa3b9..be692143d 100644 --- a/packages/system/kubernetes-rd/cozyrds/kubernetes.yaml +++ b/packages/system/kubernetes-rd/cozyrds/kubernetes.yaml @@ -32,7 +32,7 @@ spec: singular: kubernetes plural: kuberneteses openAPISchema: |- - {"title":"Chart Values","type":"object","x-cozystack-name":{"type":"string","description":"Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes--`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.","maxLength":32},"properties":{"storageClass":{"description":"Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"version":{"description":"Kubernetes major.minor version to deploy","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"host":{"description":"External hostname for Kubernetes cluster. Defaults to `.` if empty.","type":"string","default":""},"substrate":{"description":"Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["ipv4Config"],"properties":{"allowedNodes":{"description":"Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster.","type":"array","default":[],"items":{"type":"string"}},"ccm":{"description":"Proxmox cloud-controller-manager settings.","type":"object","default":{},"required":["credentialsSecretName"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks.","type":"string","default":""}}},"csi":{"description":"Proxmox CSI driver settings.","type":"object","default":{},"required":["credentialsSecretName","enabled"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.","type":"string","default":""},"enabled":{"description":"Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.","type":"boolean","default":true},"storageClasses":{"description":"StorageClasses to create in the tenant.","type":"array","default":[],"items":{"type":"object","required":["name","storage"],"properties":{"fstype":{"description":"Filesystem the driver formats volumes with: ext4 or xfs.","type":"string"},"name":{"description":"StorageClass name inside the tenant cluster.","type":"string"},"reclaimPolicy":{"description":"Delete or Retain.","type":"string"},"ssd":{"description":"Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.","type":"boolean"},"storage":{"description":"Proxmox storage id the volumes are created on (as it appears in `pvesm status`).","type":"string"}}}}}},"dnsServers":{"description":"Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.","type":"array","default":[],"items":{"type":"string"}},"insecure":{"description":"Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts.","type":"boolean","default":false},"ipv4Config":{"description":"Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional.","type":"object","default":{},"required":["gateway","prefix"],"properties":{"addresses":{"description":"Ranges or CIDRs, e.g. `[\"10.0.0.120-10.0.0.170\"]`. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address.","type":"array","default":[],"items":{"type":"string"}},"gateway":{"description":"Default gateway for the workers.","type":"string","default":""},"prefix":{"description":"Netmask prefix length.","type":"integer","default":24}}}}},"addons":{"description":"Cluster addons configuration.","type":"object","default":{},"required":["certManager","cilium","coredns","gatewayAPI","gpuOperator","hami","ingressNginx","monitoringAgents","ouroboros","velero","verticalPodAutoscaler"],"properties":{"certManager":{"description":"Cert-manager addon.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable cert-manager.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"cilium":{"description":"Cilium CNI plugin. Always installed as the tenant cluster's CNI; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true},"coredns":{"description":"CoreDNS addon. Always installed as the tenant cluster's DNS; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true},"gatewayAPI":{"description":"Gateway API addon.","type":"object","default":{},"required":["enabled"],"properties":{"enabled":{"description":"Enable Gateway API.","type":"boolean","default":false}}},"gpuOperator":{"description":"NVIDIA GPU Operator.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable GPU Operator.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"hami":{"description":"HAMi GPU virtualization middleware.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable HAMi (requires GPU Operator).","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"ingressNginx":{"description":"Ingress-NGINX controller.","type":"object","default":{},"required":["enabled","exposeMethod","valuesOverride"],"properties":{"enabled":{"description":"Enable the controller (requires nodes labeled `ingress-nginx`).","type":"boolean","default":false},"exposeMethod":{"description":"Method to expose the controller. Allowed values: `Proxied`, `LoadBalancer`.","type":"string","default":"Proxied","enum":["Proxied","LoadBalancer"]},"hosts":{"description":"Domains routed to this tenant cluster when `exposeMethod` is `Proxied`.","type":"array","default":[],"items":{"type":"string"}},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"monitoringAgents":{"description":"Monitoring agents.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable monitoring agents.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"ouroboros":{"description":"Hairpin-NAT fix for ingress-nginx with PROXY-protocol.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable ouroboros. Requires addons.ingressNginx.enabled (chart-render fail otherwise). Only useful when PROXY-protocol is wired on the tenant ingress-nginx via valuesOverride.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides. Operator-key wins over cozystack defaults.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"velero":{"description":"Velero backup/restore addon.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable Velero.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"verticalPodAutoscaler":{"description":"Vertical Pod Autoscaler. Installed and removed together with addons.monitoringAgents.enabled; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true}}},"controlPlane":{"description":"Kubernetes control-plane configuration.","type":"object","default":{},"required":["apiServer","controllerManager","konnectivity","replicas","scheduler"],"properties":{"apiServer":{"description":"API Server configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"admissionControllers":{"description":"Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it.","type":"array","default":[],"items":{"type":"string","enum":["AlwaysAdmit","AlwaysDeny","AlwaysPullImages","CertificateApproval","CertificateSigning","CertificateSubjectRestriction","DefaultIngressClass","DefaultStorageClass","DefaultTolerationSeconds","DenyEscalatingExec","DenyExecOnPrivileged","DenyServiceExternalIPs","EventRateLimit","ExtendedResourceToleration","ImagePolicyWebhook","LimitPodHardAntiAffinityTopology","LimitRanger","MutatingAdmissionWebhook","NamespaceAutoProvision","NamespaceExists","NamespaceLifecycle","NodeRestriction","OwnerReferencesPermissionEnforcement","PersistentVolumeClaimResize","PersistentVolumeLabel","PodNodeSelector","PodSecurity","PodSecurityPolicy","PodTolerationRestriction","Priority","ResourceQuota","RuntimeClass","SecurityContextDeny","ServiceAccount","StorageObjectInUseProtection","TaintNodesByCondition","ValidatingAdmissionWebhook"]}},"extraArgs":{"description":"Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior).","type":"array","default":[],"items":{"type":"string"}},"extraVolumeMounts":{"description":"Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default.","type":"array","default":[],"items":{"type":"object","x-kubernetes-preserve-unknown-fields":true}},"extraVolumes":{"description":"Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default.","type":"array","default":[],"items":{"type":"object","x-kubernetes-preserve-unknown-fields":true}},"resources":{"description":"CPU and memory resources for API Server.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"c1.medium","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"controllerManager":{"description":"Controller Manager configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Controller Manager.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"konnectivity":{"description":"Konnectivity configuration.","type":"object","default":{},"required":["server"],"properties":{"server":{"description":"Konnectivity Server configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Konnectivity.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}},"kubelet":{"description":"Kubelet connection settings.","type":"object","default":{},"properties":{"preferredAddressTypes":{"description":"Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one.","type":"array","default":[],"items":{"type":"string","enum":["Hostname","InternalIP","ExternalIP","InternalDNS","ExternalDNS"]}}}},"replicas":{"description":"Number of control-plane replicas.","type":"integer","default":2},"scheduler":{"description":"Scheduler configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Scheduler.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}},"talos":{"description":"Talos worker image configuration.","type":"object","default":{},"required":["imageFactoryURL","installerRepository","registryMirrors","schematicID","version"],"properties":{"imageFactoryURL":{"description":"Base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash.","type":"string","default":"https://factory.talos.dev"},"installerRepository":{"description":"OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:`. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.","type":"string","default":"factory.talos.dev/installer"},"registryMirrors":{"description":"Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true},"schematicID":{"description":"Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here.","type":"string","default":"ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"},"version":{"description":"Talos release used for worker OS image and installer. Must satisfy the chart's Talos<->Kubernetes support matrix against the chosen `version`.","type":"string","default":"v1.13.6"}}},"oidc":{"description":"OIDC authentication and per-user RBAC for the tenant kube-apiserver. See https://cozystack.io/docs/v1.6/kubernetes/oidc-authentication/ for the operator guide.","type":"object","default":{},"required":["mode"],"properties":{"customConfig":{"description":"Tenant-supplied AuthenticationConfiguration; consumed only when `mode: CustomConfig`.","type":"object","default":{},"properties":{"config":{"description":"Inline AuthenticationConfiguration YAML (`apiserver.config.k8s.io/v1beta1`). The chart writes the value verbatim into a Secret mounted at `/etc/kubernetes/authentication-config/config.yaml` on the kube-apiserver. Mutually exclusive with `secretRef.name`.","type":"string","default":""},"secretRef":{"description":"Reference to an existing Secret in the tenant namespace carrying the `AuthenticationConfiguration`.","type":"object","default":{},"properties":{"name":{"description":"Name of an existing Secret in the tenant (release) namespace whose `config.yaml` key holds an `apiserver.config.k8s.io/v1beta1` `AuthenticationConfiguration`. Mutually exclusive with `customConfig.config`.","type":"string","default":""}}}}},"mode":{"description":"Identity mode. `None`: no OIDC, only the static admin kubeconfig works. `System`: trust the platform Keycloak realm (`cozy` by default) via a per-cluster public client with audience binding; zero-config default. `CustomConfig`: trust a tenant-supplied issuer directly (BYO); `cozy` is not in the path.","type":"string","default":"None","enum":["None","System","CustomConfig"]},"users":{"description":"Users granted access to the tenant cluster; each entry produces one ClusterRoleBinding inside the tenant cluster. Works for both `System` and `CustomConfig` modes.","type":"array","default":[],"items":{"type":"object","required":["email","role"],"properties":{"email":{"description":"Email address matched against the `email` claim from the issuer. Used verbatim as the `User:` subject in the ClusterRoleBinding inside the tenant cluster. The `email` claim is a built-in OIDC scope requested explicitly by the chart-generated `kubectl oidc-login` kubeconfig; every conformant OIDC provider (cozy realm included) emits it.","type":"string"},"role":{"description":"Role to bind: `admin` maps to `ClusterRole/cluster-admin`, `view` maps to `ClusterRole/view`.","type":"string","enum":["admin","view"]}}}}}}}} + {"title":"Chart Values","type":"object","x-cozystack-name":{"type":"string","description":"Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes--`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.","maxLength":32},"properties":{"storageClass":{"description":"Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"version":{"description":"Kubernetes major.minor version to deploy","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"host":{"description":"External hostname for Kubernetes cluster. Defaults to `.` if empty.","type":"string","default":""},"substrate":{"description":"Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["ipv4Config"],"properties":{"allowedNodes":{"description":"Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster.","type":"array","default":[],"items":{"type":"string"}},"ccm":{"description":"Proxmox cloud-controller-manager settings.","type":"object","default":{},"required":["credentialsSecretName"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks.","type":"string","default":""}}},"csi":{"description":"Proxmox CSI driver settings.","type":"object","default":{},"required":["credentialsSecretName","enabled"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.","type":"string","default":""},"enabled":{"description":"Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.","type":"boolean","default":true},"storageClasses":{"description":"StorageClasses to create in the tenant.","type":"array","default":[],"items":{"type":"object","required":["name","storage"],"properties":{"fstype":{"description":"Filesystem the driver formats volumes with: ext4 or xfs.","type":"string"},"name":{"description":"StorageClass name inside the tenant cluster.","type":"string"},"reclaimPolicy":{"description":"Delete or Retain.","type":"string"},"ssd":{"description":"Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.","type":"boolean"},"storage":{"description":"Proxmox storage id the volumes are created on (as it appears in `pvesm status`).","type":"string"}}}}}},"dnsServers":{"description":"Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.","type":"array","default":[],"items":{"type":"string"}},"insecure":{"description":"Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts.","type":"boolean","default":false},"ipv4Config":{"description":"Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional.","type":"object","default":{},"required":["gateway","prefix"],"properties":{"addresses":{"description":"Ranges or CIDRs, e.g. `[\"10.0.0.120-10.0.0.170\"]`. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address.","type":"array","default":[],"items":{"type":"string"}},"gateway":{"description":"Default gateway for the workers.","type":"string","default":""},"prefix":{"description":"Netmask prefix length.","type":"integer","default":24}}}}},"addons":{"description":"Cluster addons configuration.","type":"object","default":{},"required":["certManager","cilium","coredns","gatewayAPI","gpuOperator","hami","ingressNginx","monitoringAgents","ouroboros","velero","verticalPodAutoscaler"],"properties":{"certManager":{"description":"Cert-manager addon.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable cert-manager.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"cilium":{"description":"Cilium CNI plugin. Always installed as the tenant cluster's CNI; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true},"coredns":{"description":"CoreDNS addon. Always installed as the tenant cluster's DNS; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true},"gatewayAPI":{"description":"Gateway API addon.","type":"object","default":{},"required":["enabled"],"properties":{"enabled":{"description":"Enable Gateway API.","type":"boolean","default":false}}},"gpuOperator":{"description":"NVIDIA GPU Operator.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable GPU Operator.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"hami":{"description":"HAMi GPU virtualization middleware.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable HAMi (requires GPU Operator).","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"ingressNginx":{"description":"Ingress-NGINX controller.","type":"object","default":{},"required":["enabled","exposeMethod","valuesOverride"],"properties":{"enabled":{"description":"Enable the controller (requires nodes labeled `ingress-nginx`).","type":"boolean","default":false},"exposeMethod":{"description":"Method to expose the controller. Allowed values: `Proxied`, `LoadBalancer`.","type":"string","default":"Proxied","enum":["Proxied","LoadBalancer"]},"hosts":{"description":"Domains routed to this tenant cluster when `exposeMethod` is `Proxied`.","type":"array","default":[],"items":{"type":"string"}},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"monitoringAgents":{"description":"Monitoring agents.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable monitoring agents.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"ouroboros":{"description":"Hairpin-NAT fix for ingress-nginx with PROXY-protocol.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable ouroboros. Requires addons.ingressNginx.enabled (chart-render fail otherwise). Only useful when PROXY-protocol is wired on the tenant ingress-nginx via valuesOverride.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides. Operator-key wins over cozystack defaults.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"velero":{"description":"Velero backup/restore addon.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable Velero.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"verticalPodAutoscaler":{"description":"Vertical Pod Autoscaler. Installed and removed together with addons.monitoringAgents.enabled; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true}}},"controlPlane":{"description":"Kubernetes control-plane configuration.","type":"object","default":{},"required":["apiServer","controllerManager","konnectivity","replicas","scheduler"],"properties":{"apiServer":{"description":"API Server configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"admissionControllers":{"description":"Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it.","type":"array","default":[],"items":{"type":"string","enum":["AlwaysAdmit","AlwaysDeny","AlwaysPullImages","CertificateApproval","CertificateSigning","CertificateSubjectRestriction","DefaultIngressClass","DefaultStorageClass","DefaultTolerationSeconds","DenyEscalatingExec","DenyExecOnPrivileged","DenyServiceExternalIPs","EventRateLimit","ExtendedResourceToleration","ImagePolicyWebhook","LimitPodHardAntiAffinityTopology","LimitRanger","MutatingAdmissionWebhook","NamespaceAutoProvision","NamespaceExists","NamespaceLifecycle","NodeRestriction","OwnerReferencesPermissionEnforcement","PersistentVolumeClaimResize","PersistentVolumeLabel","PodNodeSelector","PodSecurity","PodSecurityPolicy","PodTolerationRestriction","Priority","ResourceQuota","RuntimeClass","SecurityContextDeny","ServiceAccount","StorageObjectInUseProtection","TaintNodesByCondition","ValidatingAdmissionWebhook"]}},"extraArgs":{"description":"Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior).","type":"array","default":[],"items":{"type":"string"}},"extraVolumeMounts":{"description":"Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default.","type":"array","default":[],"items":{"type":"object","x-kubernetes-preserve-unknown-fields":true}},"extraVolumes":{"description":"Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default.","type":"array","default":[],"items":{"type":"object","x-kubernetes-preserve-unknown-fields":true}},"resources":{"description":"CPU and memory resources for API Server.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"c1.medium","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"controllerManager":{"description":"Controller Manager configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Controller Manager.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"konnectivity":{"description":"Konnectivity configuration.","type":"object","default":{},"required":["server"],"properties":{"server":{"description":"Konnectivity Server configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Konnectivity.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}},"kubelet":{"description":"Kubelet connection settings.","type":"object","default":{},"properties":{"preferredAddressTypes":{"description":"Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one.","type":"array","default":[],"items":{"type":"string","enum":["Hostname","InternalIP","ExternalIP","InternalDNS","ExternalDNS"]}}}},"replicas":{"description":"Number of control-plane replicas.","type":"integer","default":2},"scheduler":{"description":"Scheduler configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Scheduler.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}},"oidc":{"description":"OIDC authentication and per-user RBAC for the tenant kube-apiserver. See https://cozystack.io/docs/v1.6/kubernetes/oidc-authentication/ for the operator guide.","type":"object","default":{},"required":["mode"],"properties":{"customConfig":{"description":"Tenant-supplied AuthenticationConfiguration; consumed only when `mode: CustomConfig`.","type":"object","default":{},"properties":{"config":{"description":"Inline AuthenticationConfiguration YAML (`apiserver.config.k8s.io/v1beta1`). The chart writes the value verbatim into a Secret mounted at `/etc/kubernetes/authentication-config/config.yaml` on the kube-apiserver. Mutually exclusive with `secretRef.name`.","type":"string","default":""},"secretRef":{"description":"Reference to an existing Secret in the tenant namespace carrying the `AuthenticationConfiguration`.","type":"object","default":{},"properties":{"name":{"description":"Name of an existing Secret in the tenant (release) namespace whose `config.yaml` key holds an `apiserver.config.k8s.io/v1beta1` `AuthenticationConfiguration`. Mutually exclusive with `customConfig.config`.","type":"string","default":""}}}}},"mode":{"description":"Identity mode. `None`: no OIDC, only the static admin kubeconfig works. `System`: trust the platform Keycloak realm (`cozy` by default) via a per-cluster public client with audience binding; zero-config default. `CustomConfig`: trust a tenant-supplied issuer directly (BYO); `cozy` is not in the path.","type":"string","default":"None","enum":["None","System","CustomConfig"]},"users":{"description":"Users granted access to the tenant cluster; each entry produces one ClusterRoleBinding inside the tenant cluster. Works for both `System` and `CustomConfig` modes.","type":"array","default":[],"items":{"type":"object","required":["email","role"],"properties":{"email":{"description":"Email address matched against the `email` claim from the issuer. Used verbatim as the `User:` subject in the ClusterRoleBinding inside the tenant cluster. The `email` claim is a built-in OIDC scope requested explicitly by the chart-generated `kubectl oidc-login` kubeconfig; every conformant OIDC provider (cozy realm included) emits it.","type":"string"},"role":{"description":"Role to bind: `admin` maps to `ClusterRole/cluster-admin`, `view` maps to `ClusterRole/view`.","type":"string","enum":["admin","view"]}}}}}}}} release: prefix: kubernetes- labels: @@ -50,7 +50,7 @@ spec: tags: - compute icon: PHN2ZyB3aWR0aD0iMTQ0IiBoZWlnaHQ9IjE0NCIgdmlld0JveD0iMCAwIDE0NCAxNDQiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CjxyZWN0IHdpZHRoPSIxNDQiIGhlaWdodD0iMTQ0IiByeD0iMjQiIGZpbGw9InVybCgjcGFpbnQwX2xpbmVhcl82ODFfMjg0NSkiLz4KPHBhdGggZD0iTTcxLjk5NjggMTlDNzAuMzAzOSAxOS4wMDAyIDY4LjkzMTIgMjAuNTMyMiA2OC45MzE0IDIyLjQyMjFDNjguOTMxNCAyMi40NTExIDY4LjkzNzMgMjIuNDc4OCA2OC45Mzc5IDIyLjUwNzZDNjguOTM1NCAyMi43NjQ0IDY4LjkyMzEgMjMuMDczNyA2OC45MzE0IDIzLjI5NzNDNjguOTcxNyAyNC4zODczIDY5LjIwODIgMjUuMjIxNiA2OS4zNTA2IDI2LjIyNThDNjkuNjA4NCAyOC4zNzUyIDY5LjgyNDUgMzAuMTU2OSA2OS42OTEyIDMxLjgxM0M2OS41NjE1IDMyLjQzNzUgNjkuMTAzNyAzMy4wMDg2IDY4LjY5NTYgMzMuNDA1Nkw2OC42MjM1IDM0LjcwODZDNjYuNzgzOSAzNC44NjE3IDY0LjkzMTkgMzUuMTQyMSA2My4wODIxIDM1LjU2NDFDNTUuMTIyNiAzNy4zNzk4IDQ4LjI2OTUgNDEuNDk5MSA0My4wNTIgNDcuMDYwOUM0Mi43MTM0IDQ2LjgyODggNDIuMTIxMSA0Ni40MDE5IDQxLjk0NSA0Ni4yNzEyQzQxLjM5NzcgNDYuMzQ1NCA0MC44NDQ1IDQ2LjUxNTEgNDAuMTI0MSA0Ni4wOTM1QzM4Ljc1MjIgNDUuMTY1NyAzNy41MDI4IDQzLjg4NTEgMzUuOTkxIDQyLjM0MjRDMzUuMjk4MiA0MS42MDQ0IDM0Ljc5NjYgNDAuOTAxOCAzMy45NzM1IDQwLjE5MDRDMzMuNzg2NiA0MC4wMjg5IDMzLjUwMTQgMzkuODEwNCAzMy4yOTIzIDM5LjY0NDJDMzIuNjQ4OSAzOS4xMjg4IDMxLjg5IDM4Ljg2IDMxLjE1NyAzOC44MzQ4QzMwLjIxNDcgMzguODAyNCAyOS4zMDc1IDM5LjE3MjUgMjguNzEzOCAzOS45MjA2QzI3LjY1ODQgNDEuMjUwNiAyNy45OTYzIDQzLjI4MzMgMjkuNDY3MSA0NC40NjE0QzI5LjQ4MiA0NC40NzM0IDI5LjQ5NzkgNDQuNDgyNyAyOS41MTI5IDQ0LjQ5NDNDMjkuNzE1IDQ0LjY1ODkgMjkuOTYyNSA0NC44Njk4IDMwLjE0ODMgNDUuMDA3NkMzMS4wMjE3IDQ1LjY1NTUgMzEuODE5NSA0NS45ODcyIDMyLjY4OTcgNDYuNTAxNUMzNC41MjMxIDQ3LjYzOTEgMzYuMDQzIDQ4LjU4MjMgMzcuMjQ4NiA0OS43MTk2QzM3LjcxOTQgNTAuMjIzNyAzNy44MDE2IDUxLjExMjIgMzcuODY0MyA1MS40OTY0TDM4Ljg0NjggNTIuMzc4MkMzMy41ODcyIDYwLjMzMDggMzEuMTUzIDcwLjE1MzkgMzIuNTkxNSA4MC4xNjI3TDMxLjMwNzcgODAuNTM3OEMzMC45NjkzIDgwLjk3NjggMzAuNDkxMiA4MS42Njc2IDI5Ljk5MTEgODEuODczOEMyOC40MTM4IDgyLjM3MjkgMjYuNjM4NyA4Mi41NTYyIDI0LjQ5NTYgODIuNzgxOUMyMy40ODk0IDgyLjg2NiAyMi42MjEzIDgyLjgxNTggMjEuNTU0NiA4My4wMTg4QzIxLjMxOTggODMuMDYzNSAyMC45OTI3IDgzLjE0OTEgMjAuNzM1OCA4My4yMDk3QzIwLjcyNjkgODMuMjExNiAyMC43MTg2IDgzLjIxNDIgMjAuNzA5NiA4My4yMTYyQzIwLjY5NTYgODMuMjE5NSAyMC42NzcyIDgzLjIyNjMgMjAuNjYzOCA4My4yMjk0QzE4Ljg1NyA4My42NjggMTcuNjk2MyA4NS4zMzY1IDE4LjA2OTkgODYuOTgwNUMxOC40NDM3IDg4LjYyNDggMjAuMjA4NiA4OS42MjQ4IDIyLjAyNjIgODkuMjMxMkMyMi4wMzkzIDg5LjIyODIgMjIuMDU4NCA4OS4yMjc3IDIyLjA3MiA4OS4yMjQ2QzIyLjA5MjYgODkuMjE5OSAyMi4xMTA2IDg5LjIwOTkgMjIuMTMxIDg5LjIwNDlDMjIuMzg0NCA4OS4xNDkgMjIuNzAxOSA4OS4wODY4IDIyLjkyMzYgODkuMDI3MkMyMy45NzIzIDg4Ljc0NTEgMjQuNzMxOCA4OC4zMzA2IDI1LjY3NDYgODcuOTY3N0MyNy43MDI5IDg3LjIzNjggMjkuMzgyOCA4Ni42MjYyIDMxLjAxOTUgODYuMzg4M0MzMS43MDMgODYuMzM0NSAzMi40MjMyIDg2LjgxMiAzMi43ODE0IDg3LjAxMzRMMzQuMTE3NyA4Ni43ODMxQzM3LjE5MjYgOTYuMzYxMyA0My42MzY2IDEwNC4xMDMgNTEuNzk2MyAxMDguOTYxTDUxLjIzOTYgMTEwLjMwM0M1MS40NDAzIDExMC44MjQgNTEuNjYxNiAxMTEuNTMgNTEuNTEyMSAxMTIuMDQ1QzUwLjkxNzEgMTEzLjU5NSA0OS44OTggMTE1LjIzMSA0OC43Mzc0IDExNy4wNTVDNDguMTc1NSAxMTcuODk4IDQ3LjYwMDQgMTE4LjU1MiA0Ny4wOTM0IDExOS41MTZDNDYuOTcyIDExOS43NDcgNDYuODE3NSAxMjAuMTAyIDQ2LjcwMDQgMTIwLjM0NkM0NS45MTI1IDEyMi4wMzkgNDYuNDkwNCAxMjMuOTkgNDguMDAzOCAxMjQuNzIyQzQ5LjUyNjggMTI1LjQ1OSA1MS40MTcxIDEyNC42ODIgNTIuMjM1MiAxMjIuOTg1QzUyLjIzNjQgMTIyLjk4MiA1Mi4yNDA2IDEyMi45OCA1Mi4yNDE3IDEyMi45NzhDNTIuMjQyNiAxMjIuOTc2IDUyLjI0MDkgMTIyLjk3MyA1Mi4yNDE3IDEyMi45NzFDNTIuMzU4MiAxMjIuNzMxIDUyLjUyMzMgMTIyLjQxNSA1Mi42MjE2IDEyMi4xODhDNTMuMDU2IDEyMS4xODkgNTMuMjAwNSAxMjAuMzMyIDUzLjUwNTkgMTE5LjM2NUM1NC4zMTcgMTE3LjMxOCA1NC43NjI2IDExNS4xNyA1NS44NzkxIDExMy44MzJDNTYuMTg0OSAxMTMuNDY2IDU2LjY4MzMgMTEzLjMyNSA1Ny4yMDAxIDExMy4xODZMNTcuODk0NCAxMTEuOTIyQzY1LjAwOCAxMTQuNjY1IDcyLjk3MDUgMTE1LjQwMiA4MC45MjQ1IDExMy41ODdDODIuNzM5MSAxMTMuMTczIDg0LjQ5MDggMTEyLjYzNyA4Ni4xODQzIDExMS45OTRDODYuMzc5NCAxMTIuMzQyIDg2Ljc0MiAxMTMuMDExIDg2LjgzOTMgMTEzLjE3OUM4Ny4zNjQ0IDExMy4zNTEgODcuOTM3NyAxMTMuNDM5IDg4LjQwNDcgMTE0LjEzM0M4OS4yNDAxIDExNS41NjcgODkuODExNCAxMTcuMjYzIDkwLjUwNzMgMTE5LjMxMkM5MC44MTI4IDEyMC4yNzkgOTAuOTYzOCAxMjEuMTM2IDkxLjM5ODEgMTIyLjEzNkM5MS40OTcxIDEyMi4zNjMgOTEuNjYxNCAxMjIuNjg0IDkxLjc3OCAxMjIuOTI1QzkyLjU5NDQgMTI0LjYyOCA5NC40OTA3IDEyNS40MDcgOTYuMDE1OSAxMjQuNjY5Qzk3LjUyOTIgMTIzLjkzNyA5OC4xMDc3IDEyMS45ODYgOTcuMzE5NCAxMjAuMjkzQzk3LjIwMjMgMTIwLjA0OSA5Ny4wNDEyIDExOS42OTUgOTYuOTE5OCAxMTkuNDY0Qzk2LjQxMjcgMTE4LjQ5OSA5NS44Mzc3IDExNy44NTIgOTUuMjc1OCAxMTcuMDA5Qzk0LjExNTIgMTE1LjE4NSA5My4xNTI2IDExMy42NyA5Mi41NTc1IDExMi4xMkM5Mi4zMDg3IDExMS4zMiA5Mi41OTk1IDExMC44MjMgOTIuNzkzMyAxMTAuMzAzQzkyLjY3NzIgMTEwLjE3IDkyLjQyODggMTA5LjQxNCA5Mi4yODI0IDEwOS4wNTlDMTAwLjc2MiAxMDQuMDI5IDEwNy4wMTcgOTUuOTk4NSAxMDkuOTU1IDg2LjcyMzlDMTEwLjM1MSA4Ni43ODY1IDExMS4wNDEgODYuOTA5MSAxMTEuMjY1IDg2Ljk1NDJDMTExLjcyNiA4Ni42NDg3IDExMi4xNDkgODYuMjUwMSAxMTIuOTgxIDg2LjMxNTlDMTE0LjYxNyA4Ni41NTM3IDExNi4yOTcgODcuMTY0NSAxMTguMzI2IDg3Ljg5NTNDMTE5LjI2OCA4OC4yNTgxIDEyMC4wMjggODguNjc5MyAxMjEuMDc3IDg4Ljk2MTRDMTIxLjI5OCA4OS4wMjEgMTIxLjYxNiA4OS4wNzY2IDEyMS44NjkgODkuMTMyNUMxMjEuODg5IDg5LjEzNzUgMTIxLjkwOCA4OS4xNDc1IDEyMS45MjggODkuMTUyMkMxMjEuOTQyIDg5LjE1NTMgMTIxLjk2MSA4OS4xNTU4IDEyMS45NzQgODkuMTU4OEMxMjMuNzkyIDg5LjU1MiAxMjUuNTU3IDg4LjU1MjYgMTI1LjkzIDg2LjkwODFDMTI2LjMwMyA4NS4yNjQxIDEyNS4xNDMgODMuNTk1MiAxMjMuMzM2IDgzLjE1N0MxMjMuMDc0IDgzLjA5NyAxMjIuNzAxIDgyLjk5NSAxMjIuNDQ2IDgyLjk0NjVDMTIxLjM3OSA4Mi43NDM1IDEyMC41MTEgODIuNzkzNSAxMTkuNTA1IDgyLjcwOTVDMTE3LjM2MSA4Mi40ODM5IDExNS41ODYgODIuMzAwNCAxMTQuMDA5IDgxLjgwMTRDMTEzLjM2NiA4MS41NTA3IDExMi45MDggODAuNzgxOSAxMTIuNjg2IDgwLjQ2NTVMMTExLjQ0OCA4MC4xMDM1QzExMi4wOSA3NS40MzggMTExLjkxNyA3MC41ODI1IDExMC44MDYgNjUuNzI0M0MxMDkuNjg1IDYwLjgyMDggMTA3LjcwNCA1Ni4zMzYxIDEwNS4wNjIgNTIuMzg0OEMxMDUuMzc5IDUyLjA5NDggMTA1Ljk3OSA1MS41NjEyIDEwNi4xNDkgNTEuNDA0M0MxMDYuMTk5IDUwLjg1MTcgMTA2LjE1NiA1MC4yNzIyIDEwNi43MjUgNDkuNjYwM0MxMDcuOTMxIDQ4LjUyMyAxMDkuNDUxIDQ3LjU3OTkgMTExLjI4NCA0Ni40NDIzQzExMi4xNTQgNDUuOTI3OSAxMTIuOTU5IDQ1LjU5NjQgMTEzLjgzMiA0NC45NDg0QzExNC4wMyA0NC44MDE5IDExNC4yOTkgNDQuNTY5OSAxMTQuNTA3IDQ0LjQwMjJDMTE1Ljk3NyA0My4yMjM3IDExNi4zMTYgNDEuMTkxMSAxMTUuMjYgMzkuODYxNEMxMTQuMjA0IDM4LjUzMTcgMTEyLjE1OSAzOC40MDY1IDExMC42ODggMzkuNTg1QzExMC40NzkgMzkuNzUxNiAxMTAuMTk1IDM5Ljk2ODggMTEwLjAwNyA0MC4xMzEyQzEwOS4xODQgNDAuODQyNiAxMDguNjc2IDQxLjU0NTIgMTA3Ljk4MyA0Mi4yODMyQzEwNi40NzEgNDMuODI1OSAxMDUuMjIyIDQ1LjExMyAxMDMuODUgNDYuMDQwOUMxMDMuMjU1IDQ2LjM4ODUgMTAyLjM4NSA0Ni4yNjgyIDEwMS45OSA0Ni4yNDQ5TDEwMC44MjQgNDcuMDgwNkM5NC4xNzUzIDQwLjA3NjMgODUuMTIzNSAzNS41OTgyIDc1LjM3NjYgMzQuNzI4M0M3NS4zNDk0IDM0LjMxNzkgNzUuMzEzNyAzMy41NzYxIDc1LjMwNDYgMzMuMzUyOUM3NC45MDU2IDMyLjk2OTMgNzQuNDIzNSAzMi42NDE4IDc0LjMwMjQgMzEuODEzQzc0LjE2OTEgMzAuMTU2OSA3NC4zOTE3IDI4LjM3NTIgNzQuNjQ5NiAyNi4yMjU4Qzc0Ljc5MTkgMjUuMjIxNiA3NS4wMjg0IDI0LjM4NzMgNzUuMDY4OCAyMy4yOTczQzc1LjA3OCAyMy4wNDk1IDc1LjA2MzIgMjIuNjkgNzUuMDYyMiAyMi40MjIxQzc1LjA2MiAyMC41MzIyIDczLjY4OTggMTguOTk5OCA3MS45OTY4IDE5Wk02OC4xNTg1IDQyLjg4ODZMNjcuMjQ4IDU5LjA0NDdMNjcuMTgyNSA1OS4wNzc2QzY3LjEyMTQgNjAuNTIyOSA2NS45Mzc1IDYxLjY3NyA2NC40ODM5IDYxLjY3N0M2My44ODg0IDYxLjY3NyA2My4zMzg4IDYxLjQ4NDkgNjIuODkyMiA2MS4xNTcxTDYyLjg2NiA2MS4xNzAzTDQ5LjY4MDcgNTEuNzc5NEM1My43MzMxIDQ3Ljc3NTkgNTguOTE2NCA0NC44MTcyIDY0Ljg5IDQzLjQ1NDZDNjUuOTgxMiA0My4yMDU2IDY3LjA3MTkgNDMuMDIwOSA2OC4xNTg1IDQyLjg4ODZaTTc1Ljg0MTcgNDIuODg4NkM4Mi44MTU5IDQzLjc1MDQgODkuMjY1NyA0Ni45MjMyIDk0LjIwODEgNTEuNzg2TDgxLjEwOCA2MS4xMTc2TDgxLjA2MjEgNjEuMDk3OUM3OS44OTk0IDYxLjk1MTIgNzguMjYxMSA2MS43Mzk0IDc3LjM1NDggNjAuNTk3OEM3Ni45ODM1IDYwLjEzMDEgNzYuNzg4NyA1OS41ODAxIDc2Ljc2NTMgNTkuMDI0OUw3Ni43NTIyIDU5LjAxODRMNzUuODQxNyA0Mi44ODg2Wk00NC44OTkxIDU3LjgxNEw1Ni45MzgyIDY4LjYzM0w1Ni45MjUxIDY4LjY5ODhDNTguMDExNyA2OS42NDc5IDU4LjE3MiA3MS4yOTQ5IDU3LjI2NTcgNzIuNDM2OEM1Ni44OTQ0IDcyLjkwNDUgNTYuMzk3NSA3My4yMTgyIDU1Ljg2MzkgNzMuMzY0N0w1NS44NTA4IDczLjQxNzNMNDAuNDE4OCA3Ny44OTIzQzM5LjYzMzQgNzAuNjc2NSA0MS4zMjYxIDYzLjY2MjEgNDQuODk5MSA1Ny44MTRaTTk5LjAwOTQgNTcuODIwNkMxMDAuNzk4IDYwLjczMzYgMTAyLjE1MyA2My45ODcxIDEwMi45NTkgNjcuNTE0M0MxMDMuNzU2IDcwLjk5OTEgMTAzLjk1NiA3NC40Nzc4IDEwMy42MjcgNzcuODM5N0w4OC4xMTY2IDczLjM1MTVMODguMTAzNSA3My4yODU3Qzg2LjcxNDUgNzIuOTA0MyA4NS44NjA5IDcxLjQ4NDggODYuMTg0MyA3MC4wNjExQzg2LjMxNjggNjkuNDc3OCA4Ni42MjQ5IDY4Ljk4NDQgODcuMDQyMyA2OC42MTk4TDg3LjAzNTggNjguNTg2OUw5OS4wMDk0IDU3LjgyMDZaTTY5LjUyNzQgNjkuNDY4OEg3NC40NTk2TDc3LjUyNTEgNzMuMzE4Nkw3Ni40MjQ3IDc4LjEyMjZMNzEuOTk2OCA4MC4yNjE0TDY3LjU1NTggNzguMTE2MUw2Ni40NTU0IDczLjMxMkw2OS41Mjc0IDY5LjQ2ODhaTTg1LjMzOTMgODIuNjQzN0M4NS41NDg5IDgyLjYzMzEgODUuNzU3NiA4Mi42NTIgODUuOTYxNiA4Mi42ODk4TDg1Ljk4NzggODIuNjU2OUwxMDEuOTUgODUuMzY4MkM5OS42MTQyIDkxLjk2MjQgOTUuMTQ0IDk3LjY3NSA4OS4xNzExIDEwMS40OThMODIuOTc0NyA4Ni40NjA2TDgyLjk5NDQgODYuNDM0M0M4Mi40MjUyIDg1LjEwNTUgODIuOTk0OCA4My41NDcyIDg0LjMwNDQgODIuOTEzNUM4NC42Mzk3IDgyLjc1MTMgODQuOTkgODIuNjYxNCA4NS4zMzkzIDgyLjY0MzdaTTU4LjUyOTggODIuNzA5NUM1OS43NDggODIuNzI2NyA2MC44NDA2IDgzLjU3NjEgNjEuMTIzNyA4NC44MjJDNjEuMjU2MiA4NS40MDUyIDYxLjE5MTcgODUuOTgzMSA2MC45NzMgODYuNDkzNUw2MS4wMTg5IDg2LjU1MjdMNTQuODg4IDEwMS40MzlDNDkuMTU1OSA5Ny43NDMyIDQ0LjU5MDQgOTIuMjA5OSA0Mi4xNDgxIDg1LjQyMDhMNTcuOTczMSA4Mi43MjI3TDU3Ljk5OTMgODIuNzU1NkM1OC4xNzYzIDgyLjcyMjkgNTguMzU1OCA4Mi43MDcxIDU4LjUyOTggODIuNzA5NVpNNzEuODk4NiA4OS4yMzEyQzcyLjMyMjkgODkuMjE1NSA3Mi43NTM0IDg5LjMwMyA3My4xNjI3IDg5LjUwMUM3My42OTkyIDg5Ljc2MDYgNzQuMTEzNiA5MC4xNjkyIDc0LjM3NDUgOTAuNjU5Mkg3NC40MzM0TDgyLjIzNDYgMTA0LjgyMUM4MS4yMjIxIDEwNS4xNjIgODAuMTgxMyAxMDUuNDU0IDc5LjExNjcgMTA1LjY5N0M3My4xNTA1IDEwNy4wNTggNjcuMjAzMiAxMDYuNjQ1IDYxLjgxOCAxMDQuODAyTDY5LjU5OTUgOTAuNjY1OEg2OS42MTI2QzcwLjA3OTUgODkuNzg4OCA3MC45NjUgODkuMjY1NiA3MS44OTg2IDg5LjIzMTJaIiBmaWxsPSJ3aGl0ZSIgc3Ryb2tlPSJ3aGl0ZSIgc3Ryb2tlLXdpZHRoPSIwLjI1Ii8+CjxkZWZzPgo8bGluZWFyR3JhZGllbnQgaWQ9InBhaW50MF9saW5lYXJfNjgxXzI4NDUiIHgxPSIxMCIgeTE9IjE1LjUiIHgyPSIxNDQiIHkyPSIxMzEuNSIgZ3JhZGllbnRVbml0cz0idXNlclNwYWNlT25Vc2UiPgo8c3RvcCBzdG9wLWNvbG9yPSIjNEQ4N0ZGIi8+CjxzdG9wIG9mZnNldD0iMSIgc3RvcC1jb2xvcj0iIzA1NDdEMCIvPgo8L2xpbmVhckdyYWRpZW50Pgo8L2RlZnM+Cjwvc3ZnPgo= - keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "storageClass"], ["spec", "version"], ["spec", "host"], ["spec", "substrate"], ["spec", "proxmox"], ["spec", "proxmox", "allowedNodes"], ["spec", "proxmox", "dnsServers"], ["spec", "proxmox", "insecure"], ["spec", "proxmox", "ccm"], ["spec", "proxmox", "ccm", "credentialsSecretName"], ["spec", "proxmox", "csi"], ["spec", "proxmox", "csi", "enabled"], ["spec", "proxmox", "csi", "credentialsSecretName"], ["spec", "proxmox", "csi", "storageClasses"], ["spec", "proxmox", "ipv4Config"], ["spec", "proxmox", "ipv4Config", "addresses"], ["spec", "proxmox", "ipv4Config", "gateway"], ["spec", "proxmox", "ipv4Config", "prefix"], ["spec", "addons"], ["spec", "addons", "certManager"], ["spec", "addons", "certManager", "enabled"], ["spec", "addons", "certManager", "valuesOverride"], ["spec", "addons", "cilium"], ["spec", "addons", "cilium", "valuesOverride"], ["spec", "addons", "gatewayAPI"], ["spec", "addons", "gatewayAPI", "enabled"], ["spec", "addons", "ingressNginx"], ["spec", "addons", "ingressNginx", "enabled"], ["spec", "addons", "ingressNginx", "exposeMethod"], ["spec", "addons", "ingressNginx", "hosts"], ["spec", "addons", "ingressNginx", "valuesOverride"], ["spec", "addons", "gpuOperator"], ["spec", "addons", "gpuOperator", "enabled"], ["spec", "addons", "gpuOperator", "valuesOverride"], ["spec", "addons", "hami"], ["spec", "addons", "hami", "enabled"], ["spec", "addons", "hami", "valuesOverride"], ["spec", "addons", "monitoringAgents"], ["spec", "addons", "monitoringAgents", "enabled"], ["spec", "addons", "monitoringAgents", "valuesOverride"], ["spec", "addons", "verticalPodAutoscaler"], ["spec", "addons", "verticalPodAutoscaler", "valuesOverride"], ["spec", "addons", "velero"], ["spec", "addons", "velero", "enabled"], ["spec", "addons", "velero", "valuesOverride"], ["spec", "addons", "coredns"], ["spec", "addons", "coredns", "valuesOverride"], ["spec", "addons", "ouroboros"], ["spec", "addons", "ouroboros", "enabled"], ["spec", "addons", "ouroboros", "valuesOverride"], ["spec", "controlPlane"], ["spec", "controlPlane", "replicas"], ["spec", "controlPlane", "apiServer"], ["spec", "controlPlane", "apiServer", "resources"], ["spec", "controlPlane", "apiServer", "resourcesPreset"], ["spec", "controlPlane", "apiServer", "extraArgs"], ["spec", "controlPlane", "apiServer", "admissionControllers"], ["spec", "controlPlane", "apiServer", "extraVolumes"], ["spec", "controlPlane", "apiServer", "extraVolumeMounts"], ["spec", "controlPlane", "controllerManager"], ["spec", "controlPlane", "controllerManager", "resources"], ["spec", "controlPlane", "controllerManager", "resourcesPreset"], ["spec", "controlPlane", "scheduler"], ["spec", "controlPlane", "scheduler", "resources"], ["spec", "controlPlane", "scheduler", "resourcesPreset"], ["spec", "controlPlane", "konnectivity"], ["spec", "controlPlane", "konnectivity", "server"], ["spec", "controlPlane", "konnectivity", "server", "resources"], ["spec", "controlPlane", "konnectivity", "server", "resourcesPreset"], ["spec", "controlPlane", "kubelet"], ["spec", "controlPlane", "kubelet", "preferredAddressTypes"], ["spec", "talos"], ["spec", "talos", "version"], ["spec", "talos", "schematicID"], ["spec", "talos", "imageFactoryURL"], ["spec", "talos", "installerRepository"], ["spec", "talos", "registryMirrors"], ["spec", "oidc"], ["spec", "oidc", "mode"], ["spec", "oidc", "customConfig"], ["spec", "oidc", "customConfig", "config"], ["spec", "oidc", "customConfig", "secretRef"], ["spec", "oidc", "customConfig", "secretRef", "name"], ["spec", "oidc", "users"]] + keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "storageClass"], ["spec", "version"], ["spec", "host"], ["spec", "substrate"], ["spec", "proxmox"], ["spec", "proxmox", "allowedNodes"], ["spec", "proxmox", "dnsServers"], ["spec", "proxmox", "insecure"], ["spec", "proxmox", "ccm"], ["spec", "proxmox", "ccm", "credentialsSecretName"], ["spec", "proxmox", "csi"], ["spec", "proxmox", "csi", "enabled"], ["spec", "proxmox", "csi", "credentialsSecretName"], ["spec", "proxmox", "csi", "storageClasses"], ["spec", "proxmox", "ipv4Config"], ["spec", "proxmox", "ipv4Config", "addresses"], ["spec", "proxmox", "ipv4Config", "gateway"], ["spec", "proxmox", "ipv4Config", "prefix"], ["spec", "addons"], ["spec", "addons", "certManager"], ["spec", "addons", "certManager", "enabled"], ["spec", "addons", "certManager", "valuesOverride"], ["spec", "addons", "cilium"], ["spec", "addons", "cilium", "valuesOverride"], ["spec", "addons", "gatewayAPI"], ["spec", "addons", "gatewayAPI", "enabled"], ["spec", "addons", "ingressNginx"], ["spec", "addons", "ingressNginx", "enabled"], ["spec", "addons", "ingressNginx", "exposeMethod"], ["spec", "addons", "ingressNginx", "hosts"], ["spec", "addons", "ingressNginx", "valuesOverride"], ["spec", "addons", "gpuOperator"], ["spec", "addons", "gpuOperator", "enabled"], ["spec", "addons", "gpuOperator", "valuesOverride"], ["spec", "addons", "hami"], ["spec", "addons", "hami", "enabled"], ["spec", "addons", "hami", "valuesOverride"], ["spec", "addons", "monitoringAgents"], ["spec", "addons", "monitoringAgents", "enabled"], ["spec", "addons", "monitoringAgents", "valuesOverride"], ["spec", "addons", "verticalPodAutoscaler"], ["spec", "addons", "verticalPodAutoscaler", "valuesOverride"], ["spec", "addons", "velero"], ["spec", "addons", "velero", "enabled"], ["spec", "addons", "velero", "valuesOverride"], ["spec", "addons", "coredns"], ["spec", "addons", "coredns", "valuesOverride"], ["spec", "addons", "ouroboros"], ["spec", "addons", "ouroboros", "enabled"], ["spec", "addons", "ouroboros", "valuesOverride"], ["spec", "controlPlane"], ["spec", "controlPlane", "replicas"], ["spec", "controlPlane", "apiServer"], ["spec", "controlPlane", "apiServer", "resources"], ["spec", "controlPlane", "apiServer", "resourcesPreset"], ["spec", "controlPlane", "apiServer", "extraArgs"], ["spec", "controlPlane", "apiServer", "admissionControllers"], ["spec", "controlPlane", "apiServer", "extraVolumes"], ["spec", "controlPlane", "apiServer", "extraVolumeMounts"], ["spec", "controlPlane", "controllerManager"], ["spec", "controlPlane", "controllerManager", "resources"], ["spec", "controlPlane", "controllerManager", "resourcesPreset"], ["spec", "controlPlane", "scheduler"], ["spec", "controlPlane", "scheduler", "resources"], ["spec", "controlPlane", "scheduler", "resourcesPreset"], ["spec", "controlPlane", "konnectivity"], ["spec", "controlPlane", "konnectivity", "server"], ["spec", "controlPlane", "konnectivity", "server", "resources"], ["spec", "controlPlane", "konnectivity", "server", "resourcesPreset"], ["spec", "controlPlane", "kubelet"], ["spec", "controlPlane", "kubelet", "preferredAddressTypes"], ["spec", "oidc"], ["spec", "oidc", "mode"], ["spec", "oidc", "customConfig"], ["spec", "oidc", "customConfig", "config"], ["spec", "oidc", "customConfig", "secretRef"], ["spec", "oidc", "customConfig", "secretRef", "name"], ["spec", "oidc", "users"]] secrets: exclude: [] include: diff --git a/pkg/registry/apps/application/rest_nullstrip_admission_test.go b/pkg/registry/apps/application/rest_nullstrip_admission_test.go index 46b150fa1..9fdd252cd 100644 --- a/pkg/registry/apps/application/rest_nullstrip_admission_test.go +++ b/pkg/registry/apps/application/rest_nullstrip_admission_test.go @@ -45,14 +45,14 @@ var nullChartCases = []nullChartCase{ {name: "nullable schema default", schema: `{"type":"object","required":["name"],"properties":{"name":{"type":"string","nullable":true,"default":"example"}}}`, values: `{"name":"example"}`, input: `{"name":null}`}, {name: "item schema defaults", schema: `{"type":"object","properties":{"entries":{"type":"object","additionalProperties":{"type":"object","required":["name"],"properties":{"name":{"type":"string","default":"example"}}}},"items":{"type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"type":"string","default":"example"}}}}}}`, values: `{"entries":{},"items":[{"name":"chart-default"}]}`, input: `{"entries":{"mine":{"name":null}},"items":[{"name":null}]}`}, - {name: "nested defaults", chart: "apps/kubernetes", rd: "kubernetes", input: `{"talos":{"registryMirrors":null,"version":null},"controlPlane":{"replicas":null,"apiServer":{"resourcesPreset":null}},"nodeGroups":{"legacy":{"instanceType":null}}}`, want: `{"talos":{},"controlPlane":{"apiServer":{}},"nodeGroups":{"legacy":{"instanceType":null}}}`, paths: []string{"controlPlane.apiServer.resourcesPreset", "controlPlane.replicas", "talos.registryMirrors", "talos.version"}}, + {name: "nested defaults", chart: "apps/kubernetes", rd: "kubernetes", input: `{"oidc":{"mode":null},"addons":{"cilium":{"valuesOverride":null}},"controlPlane":{"replicas":null,"apiServer":{"resourcesPreset":null}},"nodeGroups":{"legacy":{"instanceType":null}}}`, want: `{"oidc":{},"addons":{"cilium":{}},"controlPlane":{"apiServer":{}},"nodeGroups":{"legacy":{"instanceType":null}}}`, paths: []string{"addons.cilium.valuesOverride", "controlPlane.apiServer.resourcesPreset", "controlPlane.replicas", "oidc.mode"}}, {name: "mariadb users", chart: "apps/mariadb", rd: "mariadb", input: `{"users":{"alice":{"password":null,"maxUserConnections":null},"blank":null}}`}, {name: "rabbitmq vhosts", chart: "apps/rabbitmq", rd: "rabbitmq", input: `{"vhosts":{"vh1":{"roles":null},"blank":null}}`}, {name: "seaweedfs pools", chart: "extra/seaweedfs", rd: "seaweedfs", input: `{"volume":{"pools":{"ssd":{"diskType":null},"blank":null}}}`}, {name: "vm source http", chart: "apps/vm-disk", rd: "vm-disk", input: `{"source":{"http":{"url":null}}}`}, {name: "vm source disk", chart: "apps/vm-disk", rd: "vm-disk", input: `{"source":{"disk":{"name":null}}}`}, {name: "vm source image", chart: "apps/vm-disk", rd: "vm-disk", input: `{"source":{"image":{"name":null}}}`}, - {name: "optional section", chart: "apps/kubernetes", rd: "kubernetes", input: `{"talos":null,"storageClass":null}`}, + {name: "optional section", chart: "apps/kubernetes", rd: "kubernetes", input: `{"oidc":null,"storageClass":null}`}, } func chartSpecSchema(t *testing.T, rd string) *structuralschema.Structural { diff --git a/pkg/registry/apps/application/rest_nullstrip_test.go b/pkg/registry/apps/application/rest_nullstrip_test.go index 7d62844a4..ce56eb0bb 100644 --- a/pkg/registry/apps/application/rest_nullstrip_test.go +++ b/pkg/registry/apps/application/rest_nullstrip_test.go @@ -64,15 +64,16 @@ func normalize(t *testing.T, s *structuralschema.Structural, spec string) ([]str func TestNormalizeSpecNulls_RequiredFields(t *testing.T) { s := kubernetesSpecSchema(t) - paths, got := normalize(t, s, `{"host":"k8s","talos":{"registryMirrors":null,"version":null,"imageFactoryURL":"https://factory.example.invalid"}}`) + paths, got := normalize(t, s, `{"host":"k8s","addons":{"cilium":{"valuesOverride":null}},"oidc":{"mode":null}}`) - want := []string{"talos.registryMirrors", "talos.version"} + want := []string{"addons.cilium.valuesOverride", "oidc.mode"} if !reflect.DeepEqual(paths, want) { t.Errorf("stripped paths = %v, want %v", paths, want) } wantSpec := map[string]any{ - "host": "k8s", - "talos": map[string]any{"imageFactoryURL": "https://factory.example.invalid"}, + "host": "k8s", + "addons": map[string]any{"cilium": map[string]any{}}, + "oidc": map[string]any{}, } if !reflect.DeepEqual(got, wantSpec) { t.Errorf("normalized spec = %v, want %v", got, wantSpec) @@ -82,12 +83,12 @@ func TestNormalizeSpecNulls_RequiredFields(t *testing.T) { func TestNormalizeSpecNulls_OptionalFieldsUntouched(t *testing.T) { s := kubernetesSpecSchema(t) - paths, got := normalize(t, s, `{"talos":null,"storageClass":null,"nodeGroups":null,"version":null}`) + paths, got := normalize(t, s, `{"oidc":null,"storageClass":null,"nodeGroups":null,"version":null}`) if len(paths) != 0 { t.Errorf("stripped paths = %v, want none: the Kubernetes schema requires none of these at the root", paths) } - for _, key := range []string{"talos", "storageClass", "nodeGroups", "version"} { + for _, key := range []string{"oidc", "storageClass", "nodeGroups", "version"} { v, ok := got[key] if !ok || v != nil { t.Errorf("%s = %v (present=%v), want a preserved null", key, v, ok) @@ -96,20 +97,20 @@ func TestNormalizeSpecNulls_OptionalFieldsUntouched(t *testing.T) { } // TestNormalizeSpecNulls_FreeFormSubtreeUntouched pins the boundary: inside -// registryMirrors the keys are registry hostnames the user invents, and the +// addons.cilium.valuesOverride the keys are Helm values the user invents, and the // schema declares nothing there. A null in that position may be the payload, so // it is left exactly as written rather than guessed at. func TestNormalizeSpecNulls_FreeFormSubtreeUntouched(t *testing.T) { s := kubernetesSpecSchema(t) - paths, got := normalize(t, s, `{"talos":{"registryMirrors":{"ghcr.io":null}}}`) + paths, got := normalize(t, s, `{"addons":{"cilium":{"valuesOverride":{"ghcr.io":null}}}}`) if len(paths) != 0 { t.Errorf("stripped paths = %v, want none", paths) } - mirrors := got["talos"].(map[string]any)["registryMirrors"].(map[string]any) - if _, ok := mirrors["ghcr.io"]; !ok { - t.Errorf("free-form key ghcr.io was dropped: %v", mirrors) + overrides := got["addons"].(map[string]any)["cilium"].(map[string]any)["valuesOverride"].(map[string]any) + if _, ok := overrides["ghcr.io"]; !ok { + t.Errorf("free-form key ghcr.io was dropped: %v", overrides) } } @@ -143,12 +144,12 @@ func TestNormalizeSpecNulls_NoDefaultOrUserCollection(t *testing.T) { } func TestNormalizeSpecNulls_PreservesOtherValuesExactly(t *testing.T) { - app := &appsv1alpha1.Application{Spec: &apiextv1.JSON{Raw: []byte(`{"talos":{"registryMirrors":null},"large":9007199254740993,"negative":-9007199254740993,"text":"null","bool":false,"empty":[],"object":{}}`)}} + app := &appsv1alpha1.Application{Spec: &apiextv1.JSON{Raw: []byte(`{"addons":{"cilium":{"valuesOverride":null}},"large":9007199254740993,"negative":-9007199254740993,"text":"null","bool":false,"empty":[],"object":{}}`)}} r := &REST{specSchema: kubernetesSpecSchema(t)} if _, err := r.normalizeSpecNulls(app); err != nil { t.Fatal(err) } - if string(app.Spec.Raw) != `{"bool":false,"empty":[],"large":9007199254740993,"negative":-9007199254740993,"object":{},"talos":{},"text":"null"}` { + if string(app.Spec.Raw) != `{"addons":{"cilium":{}},"bool":false,"empty":[],"large":9007199254740993,"negative":-9007199254740993,"object":{},"text":"null"}` { t.Fatalf("unrelated values changed: %s", app.Spec.Raw) } } @@ -182,8 +183,8 @@ func TestNormalizeSpecNulls_NoSchemaOrNoNulls(t *testing.T) { } func TestEmptyFieldsWarning(t *testing.T) { - short := emptyFieldsWarning([]string{"talos.version", "talos.registryMirrors"}) - if !strings.Contains(short, "talos.version, talos.registryMirrors") || + short := emptyFieldsWarning([]string{"oidc.mode", "addons.cilium.valuesOverride"}) + if !strings.Contains(short, "oidc.mode, addons.cilium.valuesOverride") || !strings.Contains(short, "the chart default applies") { t.Errorf("unexpected warning: %q", short) } From 6b0a564bc48c07d369259f5a459c530d8b47df4f Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 16:57:46 +0500 Subject: [PATCH 454/518] test(kubernetes-nodes): cover the Talos support-matrix guard by render The pool chart checks every worker's Talos release against the Kubernetes version it joins, once for talos.version and once for the release osImage resolves to, and that is now the only such check. Its coverage was two grep tests that the call sites exist; nothing rendered a refusal, and the pairings it has to let through were pinned only in the parent chart, against a copy of the table. The pool's own suite pins the in-window pairings, through talos.version at both ends of the schema's version range and through osImage.factory.version. The refusals need a version the schema does not offer and the versions map does not hold, so they render a copy of the chart with --skip-schema-validation and one extra versions.yaml row, the way the postgres preload allowlist test reaches its backstop. They fail if the second call is fed talos.version instead of the resolved release, if the table is keyed on the full release instead of the minor, or if either call is dropped. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- hack/check-worker-image-catalog-defaults.bats | 21 +++-- hack/kubernetes-nodes-talos-matrix.bats | 81 +++++++++++++++++++ .../tests/talos_k8s_support_matrix_test.yaml | 62 ++++++++++++++ 3 files changed, 152 insertions(+), 12 deletions(-) create mode 100644 hack/kubernetes-nodes-talos-matrix.bats create mode 100644 packages/apps/kubernetes-nodes/tests/talos_k8s_support_matrix_test.yaml diff --git a/hack/check-worker-image-catalog-defaults.bats b/hack/check-worker-image-catalog-defaults.bats index ed695444e..eaf942112 100644 --- a/hack/check-worker-image-catalog-defaults.bats +++ b/hack/check-worker-image-catalog-defaults.bats @@ -78,19 +78,16 @@ CATALOG_VALUES="$REPO_ROOT/packages/system/kubernetes-worker-image/values.yaml" # ----------------------------------------------------------------------------- # Second family of default-drift guards on the same chart: the Talos <-> Kubernetes -# support matrix. The render guard keyed only on `talos.version`, so a pool could -# reach a different Talos release through `osImage.builtin.version` / -# `osImage.factory.version` and boot a kubelet outside its support window with no -# render failure. The table now lives once in templates/_helpers.tpl and both -# paths consult it. +# support matrix. A pool reaches a Talos release through `talos.version` or through +# `osImage.builtin.version` / `osImage.factory.version`, and the table in +# templates/_helpers.tpl has to be consulted for whichever one the pool boots. # -# The symmetry is structural, not observable from a render: the matrix ships one -# row (`v1.13`) whose Kubernetes list is a superset of `values.schema.json`'s -# `version` enum, so no schema-valid pairing can make either call site fail today. -# A helm-unittest case would have to put an out-of-enum `version` in, and -# helm-unittest validates the schema, so it cannot. These two guards cover what a -# render cannot: that both call sites exist, and that the table still means -# something for the defaults the chart ships. +# The matrix ships one row (`v1.13`) whose Kubernetes list is a superset of +# `values.schema.json`'s `version` enum, so no schema-valid pairing makes either +# call site fail today. hack/kubernetes-nodes-talos-matrix.bats renders the +# refusals with the schema skipped; these two guards cover what it does not: that +# both call sites stay wired, and that the table still means something for the +# defaults the chart ships. NODES_TEMPLATE="$REPO_ROOT/packages/apps/kubernetes-nodes/templates/nodegroup.yaml" NODES_SCHEMA="$REPO_ROOT/packages/apps/kubernetes-nodes/values.schema.json" diff --git a/hack/kubernetes-nodes-talos-matrix.bats b/hack/kubernetes-nodes-talos-matrix.bats new file mode 100644 index 000000000..33c2d0af6 --- /dev/null +++ b/hack/kubernetes-nodes-talos-matrix.bats @@ -0,0 +1,81 @@ +#!/usr/bin/env bats +# Unit test: the refusals of the Talos to Kubernetes support-matrix guard in +# packages/apps/kubernetes-nodes (kubernetes-nodes.assertTalosSupportsKubernetes, +# called from templates/nodegroup.yaml for talos.version and again for the +# release osImage resolves to). +# +# On the shipped chart a refusal is unreachable: the values schema offers only +# Kubernetes versions the v1.13 row lists, and files/versions.yaml, which another +# template fails on first, holds the same set. The guard is there for the day a +# Talos row stops listing a version the chart still offers. So this renders a +# copy of the chart with --skip-schema-validation and one extra versions.yaml row +# for a version the matrix does not list. The pairings the guard lets through +# are in the chart's own tests/talos_k8s_support_matrix_test.yaml. + +load test_helper + +REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)" +CHART="$REPO_ROOT/packages/apps/kubernetes-nodes" + +# render_pool renders the pool chart copy with the given --set flags and +# prints helm's output, errors included. The copy lives in the test's own tmpdir. +render_pool() { + copy="$BATS_TEST_TMPDIR/chart" + if [ ! -d "$copy" ]; then + cp -RL "$CHART" "$copy" + printf '"v1.30": "v1.30.14"\n' >> "$copy/files/versions.yaml" + fi + helm template kubernetes-nodes-myk8s-md0 "$copy" --namespace tenant-test \ + --skip-schema-validation \ + --set cluster=myk8s \ + --set _cluster.cluster-domain=cozy.local \ + --set minReplicas=1 --set maxReplicas=3 \ + --set instanceType= --set resources.cpu=2 --set resources.memory=4Gi \ + --set diskSize=20Gi --set storageClass=replicated \ + "$@" 2>&1 +} + +@test "an in-window pairing renders on the same copy, so the refusals below are the guard's" { + out=$(render_pool --set version=v1.31) || { echo "$out" >&2; return 1; } + printf '%s\n' "$out" | grep -q '^kind: KubevirtMachineTemplate$' +} + +@test "a Kubernetes version outside the row of talos.version is refused" { + if out=$(render_pool --set version=v1.30); then + echo "render succeeded with Kubernetes v1.30 on the default Talos" >&2 + return 1 + fi + printf '%s\n' "$out" | grep -qF "Kubernetes v1.30 is not supported by Talos v1.13.6. Supported versions: [v1.31 v1.32 v1.33 v1.34 v1.35 v1.36]. Bump 'version' or set talos.version explicitly." \ + || { echo "$out" >&2; return 1; } +} + +@test "an older patch release of a listed Talos minor is checked against the same row" { + if out=$(render_pool --set version=v1.30 --set talos.version=v1.13.0); then + echo "render succeeded with Kubernetes v1.30 on Talos v1.13.0: the guard is not keyed on the minor" >&2 + return 1 + fi + printf '%s\n' "$out" | grep -qF 'Kubernetes v1.30 is not supported by Talos v1.13.0.' \ + || { echo "$out" >&2; return 1; } +} + +# talos.version names a minor the matrix does not list, so the first call lets the +# pool through and only the call fed the osImage release can refuse it. +@test "a Kubernetes version outside the row of osImage.factory.version is refused" { + if out=$(render_pool --set version=v1.30 --set talos.version=v1.12.0 --set osImage.factory.version=v1.13.0); then + echo "render succeeded with Kubernetes v1.30 on Talos v1.13.0 reached through osImage.factory" >&2 + return 1 + fi + printf '%s\n' "$out" | grep -qF 'Kubernetes v1.30 is not supported by Talos v1.13.0.' \ + && printf '%s\n' "$out" | grep -qF 'nodeGroup md0 reaches this Talos release through its osImage override' \ + || { echo "$out" >&2; return 1; } +} + +@test "a Kubernetes version outside the row of osImage.builtin.version is refused" { + if out=$(render_pool --set version=v1.30 --set talos.version=v1.12.0 --set osImage.builtin.version=v1.13.0); then + echo "render succeeded with Kubernetes v1.30 on Talos v1.13.0 reached through osImage.builtin" >&2 + return 1 + fi + printf '%s\n' "$out" | grep -qF 'Kubernetes v1.30 is not supported by Talos v1.13.0.' \ + && printf '%s\n' "$out" | grep -qF 'nodeGroup md0 reaches this Talos release through its osImage override' \ + || { echo "$out" >&2; return 1; } +} diff --git a/packages/apps/kubernetes-nodes/tests/talos_k8s_support_matrix_test.yaml b/packages/apps/kubernetes-nodes/tests/talos_k8s_support_matrix_test.yaml new file mode 100644 index 000000000..ea1aa5ce3 --- /dev/null +++ b/packages/apps/kubernetes-nodes/tests/talos_k8s_support_matrix_test.yaml @@ -0,0 +1,62 @@ +suite: Talos to Kubernetes support-matrix guard (assertTalosSupportsKubernetes) + +# The guard is keyed by Talos minor and consulted twice in nodegroup.yaml: once +# for talos.version and once for the release osImage resolves to. These cases +# pin the pairings it has to let through: each one fails if the matrix row loses +# the Kubernetes version under test. +# +# The refusals are in hack/kubernetes-nodes-talos-matrix.bats. The values schema +# offers only Kubernetes versions the v1.13 row lists, so no schema-valid input +# reaches a refusal, and helm-unittest always validates the schema; the bats file +# renders with --skip-schema-validation instead. + +templates: + - templates/nodegroup.yaml + +release: + name: kubernetes-nodes-myk8s-md0 + namespace: tenant-test + +set: + cluster: myk8s + _cluster: + cluster-domain: cozy.local + minReplicas: 1 + maxReplicas: 3 + instanceType: "" + diskSize: 20Gi + storageClass: replicated + roles: [ingress-nginx] + resources: + cpu: "2" + memory: 4Gi + +tests: + - it: renders the oldest Kubernetes version the schema offers against the default Talos + set: + version: "v1.31" + asserts: + - notFailedTemplate: {} + - hasDocuments: + count: 4 + + - it: renders the newest Kubernetes version the schema offers against the default Talos + set: + version: "v1.35" + asserts: + - notFailedTemplate: {} + + - it: renders an in-window pairing reached through osImage.factory.version + set: + version: "v1.31" + talos: + version: "v1.12.0" + osImage: + factory: + version: "v1.13.0" + asserts: + - notFailedTemplate: {} + - equal: + path: spec.template.spec.virtualMachineTemplate.spec.dataVolumeTemplates[0].spec.source.http.url + value: https://factory.talos.dev/image/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515/v1.13.0/openstack-amd64.raw.xz + documentSelector: {path: kind, value: KubevirtMachineTemplate} From 1d220136461ba8e7211ad0e8fd158736beca4f6c Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 12 Nov 2026 11:10:21 +0300 Subject: [PATCH 455/518] fix(postgres): hold a backup until the plugin sees its new destination The plugin-barman-cloud sidecar reads the ObjectStore through its own cache, kept for ten seconds (DefaultTTLSeconds in plugin-barman-cloud v0.13.0) and invalidated only by the sidecar's own writes, and WAL archiving keeps it warm. The pass that changed the destination created the cnpg.io/Backup a moment later, so the base backup could be taken against the previous destination while the artifact names the new one, and a restore from it finds nothing. On a live cluster with the shipped plugin, a Backup created two to four seconds after switching the destination landed in the old bucket in two of three tries; with a fifteen second wait it landed in the new one every time. The apply that changes the ObjectStore configuration now stamps the time on the ObjectStore itself, so the record cannot be lost apart from the change, and the job creates its Backup only once that change is older than the cache can be. A change is judged on the configuration a server-side dry run of the same apply returns, so values the apiserver defaults or another field manager owns do not read as a change on every pass. That wait leaves a job with the destination written and no Backup yet, which the queue, looking only for Backups in flight, did not see: a second job with another destination wrote its own, the first took that as a change and stamped again, and the two restarted each other's wait forever. Every apply of the ObjectStore therefore also names the job that made it, which only a job without a Backup of its own and let through by the queue does, and the queue holds every other job while that one runs and has no Backup yet. Once it has one, that Backup holds the destination for as long as CloudNativePG runs it, as before, so a job stuck after its Backup finished does not hold every later backup of the application. This also covers a job waiting for the rollout of the plugin it attached, during which a second job used to switch the destination as well, including when its own write left the configuration as a restore had re-rendered it. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpgstrategy_controller.go | 170 ++++- .../cnpgstrategy_controller_test.go | 585 +++++++++++++++++- 2 files changed, 726 insertions(+), 29 deletions(-) diff --git a/internal/backupcontroller/cnpgstrategy_controller.go b/internal/backupcontroller/cnpgstrategy_controller.go index 375af221c..e43deff33 100644 --- a/internal/backupcontroller/cnpgstrategy_controller.go +++ b/internal/backupcontroller/cnpgstrategy_controller.go @@ -15,6 +15,7 @@ import ( "time" corev1 "k8s.io/api/core/v1" + apiequality "k8s.io/apimachinery/pkg/api/equality" apierrors "k8s.io/apimachinery/pkg/api/errors" apimeta "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/api/resource" @@ -233,9 +234,30 @@ const ( // cannot be lost to a failed BackupJob status write after the attach. cnpgPluginAttachedAtAnnotation = "backups.cozystack.io/barman-plugin-attached-at" - // How far ahead of this controller's clock an attach record may be - // before it is taken as corrupt rather than as written by a leader whose - // clock ran ahead. + // Annotation on the ObjectStore carrying, as RFC 3339, when a backup job + // last changed its configuration. It is written by the apply that makes + // the change, so it cannot be lost apart from it. + cnpgDestinationChangedAtAnnotation = "backups.cozystack.io/destination-changed-at" + + // Annotation on the ObjectStore naming the BackupJob whose apply last + // wrote it. Written by that apply, it holds the destination for that job + // while it runs and has no cnpg.io/Backup of its own yet; from then on + // the queue sees that Backup instead. + cnpgDestinationWrittenByAnnotation = "backups.cozystack.io/destination-written-by" + + // How long after an ObjectStore configuration change a Backup is held. + // The plugin-barman-cloud sidecar reads the ObjectStore through its own + // cache, kept for DefaultTTLSeconds = 10 (plugin-barman-cloud v0.13.0, + // internal/cnpgi/instance/internal/client/client.go) and invalidated + // only by the sidecar's own writes, so a Backup taken sooner can use the + // previous configuration and land in the previous bucket. The margin + // covers the cache's whole-second expiry and the record's whole-second + // timestamp. + cnpgObjectStoreCacheSettle = 15 * time.Second + + // How far ahead of this controller's clock a recorded time, of a plugin + // attach or of a destination change, may be before it is taken as + // corrupt rather than as written by a leader whose clock ran ahead. cnpgAttachClockSkewTolerance = 5 * time.Minute cnpgPodRoleLabel = "cnpg.io/podRole" @@ -366,7 +388,9 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp // The ObjectStore is the Cluster's WAL archive destination, shared by // every backup of the app, so it is written only before this job's - // Backup exists and only while no other job's Backup is in flight. + // Backup exists, and only while no other job's Backup is in flight and + // no other running job without a Backup yet holds the destination it + // last wrote. writeDestination := existingBackup == nil if writeDestination { // The gate in Reconcile sees only a job that has not started, and @@ -386,6 +410,11 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp if err != nil { return ctrl.Result{}, err } + if holder == "" { + if holder, err = r.destinationClaimHolder(ctx, j, clusterName); err != nil { + return ctrl.Result{}, err + } + } if holder != "" { apimeta.RemoveStatusCondition(&j.Status.Conditions, backupCondDestinationApplied) apimeta.SetStatusCondition(&j.Status.Conditions, cnpgDestinationWriteHeld("WaitingForOtherBackup")) @@ -403,7 +432,7 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } attemptedAt := metav1.Now() - attachment, err := r.applyClusterPluginBackup(ctx, j.Namespace, clusterName, rendered, serverName, existingBackup != nil || awaitingRollout, writeDestination) + attachment, err := r.applyClusterPluginBackup(ctx, j.Namespace, clusterName, rendered, serverName, existingBackup != nil || awaitingRollout, writeDestination, j.Name) if err != nil { if apierrors.IsNotFound(err) { // HelmRelease has not yet rendered the Cluster (fresh app, or @@ -504,7 +533,7 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp } // The Cluster's record covers an attach whose status write was lost. attachedAt := rollout.LastTransitionTime - recorded, ok, err := cnpgRecordedAttach(attachment.attachedAt) + recorded, ok, err := cnpgRecordedTime(attachment.attachedAt) if err != nil { return r.markBackupJobFailed(ctx, j, fmt.Sprintf( "cannot tell when the barman-cloud plugin was attached: the %s annotation of cnpg.io/Cluster %s/%s: %v", @@ -548,13 +577,11 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp apimeta.RemoveStatusCondition(&j.Status.Conditions, "Ready") rolledOut = true } else if existingBackup == nil && attachment.attachedAt != "" { - // Another backup job attached the plugin. Its rollout can still be - // running while the attach is recent; past the rollout deadline - // this job stops waiting, as the attaching job normally has judged - // the rollout by then unless a wait of its own moved its deadline - // on. The record is that job's: - // one this job cannot use does not fail it. - recorded, _, err := cnpgRecordedAttach(attachment.attachedAt) + // Another backup job attached the plugin, and its rollout can still + // be running while the attach is recent; past the rollout deadline + // this job stops waiting. The record is that job's: one this job + // cannot use does not fail it. + recorded, _, err := cnpgRecordedTime(attachment.attachedAt) if err != nil { logger.Info("ignoring an unusable attach record another backup job left", "cluster", clusterName, "annotation", cnpgPluginAttachedAtAnnotation, "error", err) @@ -577,6 +604,12 @@ func (r *BackupJobReconciler) reconcileCNPG(ctx context.Context, j *backupsv1alp "cluster", clusterName, "live", attachment.flavor, "application", specFlavor) } + if changedAt := attachment.destinationChangedAt; !changedAt.IsZero() && time.Since(changedAt) < cnpgObjectStoreCacheSettle { + return r.waitCNPGBackupJob(ctx, j, "WaitingForDestinationToSettle", fmt.Sprintf( + "the WAL archive destination of cnpg.io/Cluster %s/%s changed at %s; the barman-cloud plugin can read the previous one for up to %s", + j.Namespace, clusterName, changedAt.UTC().Format(time.RFC3339), cnpgObjectStoreCacheSettle)) + } + written := cnpgDestinationOf(rendered) written.ServerName = serverName cnpgBackup, err := r.ensureCNPGBackup(ctx, j, clusterName, existingBackup, written) @@ -780,7 +813,7 @@ var cnpgClusterGVR = schema.GroupVersionResource{ // // With write unset nothing is written: only what the live Cluster carries is // reported. -func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, namespace, clusterName string, t *strategyv1alpha1.CNPGTemplate, serverName string, attach, write bool) (cnpgPluginAttachment, error) { +func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, namespace, clusterName string, t *strategyv1alpha1.CNPGTemplate, serverName string, attach, write bool, jobName string) (cnpgPluginAttachment, error) { // Live read: the cached Cluster can predate a restore re-render, and the // serverName, the flavor and the UID below must come from the current // object. @@ -836,9 +869,21 @@ func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, name RetentionPolicy: t.BarmanObjectStore.RetentionPolicy, InstanceSidecarConfiguration: barmanSidecarConfiguration(), } + changedAt, err := r.objectStoreChangeRecord(ctx, objStore) + if err != nil { + return cnpgPluginAttachment{}, err + } + // Only a job without a Backup of its own writes, and the queue let it + // through, so it holds the destination from this write while it runs + // and has not created its Backup. + objStore.Annotations = map[string]string{cnpgDestinationWrittenByAnnotation: jobName} + if !changedAt.IsZero() { + objStore.Annotations[cnpgDestinationChangedAtAnnotation] = changedAt.UTC().Format(time.RFC3339) + } if err := r.Patch(ctx, objStore, client.Apply, client.FieldOwner(cnpgFieldManager), client.ForceOwnership); err != nil { return cnpgPluginAttachment{}, fmt.Errorf("apply ObjectStore %s/%s: %w", namespace, objStoreName, err) } + attachment.destinationChangedAt = changedAt patch := newCNPGClusterPatch(namespace, clusterName) patch.Spec.Plugins = []cnpgtypes.PluginConfiguration{buildBarmanPlugin(objStoreName, serverName)} @@ -857,6 +902,83 @@ func (r *BackupJobReconciler) applyClusterPluginBackup(ctx context.Context, name return attachment, nil } +// objectStoreChangeRecord returns the cnpgDestinationChangedAtAnnotation the +// apply of desired should carry: the time of this write when it changes the +// configuration, otherwise the record the live ObjectStore already carries, +// re-sent so the apply does not delete it, and zero when there is none. +// "Changes" is judged on the configuration a server-side dry run of the same +// apply returns, not on desired, so values the apiserver defaults or another +// field manager owns do not read as a change on every pass. Creating the +// ObjectStore stamps no time: the plugin cannot hold a stale copy of an +// object that did not exist, short of one deleted and recreated by hand +// within the cache's lifetime. +func (r *BackupJobReconciler) objectStoreChangeRecord(ctx context.Context, desired *cnpgtypes.ObjectStore) (time.Time, error) { + live := &cnpgtypes.ObjectStore{} + if err := r.apiReader().Get(ctx, client.ObjectKeyFromObject(desired), live); err != nil { + if apierrors.IsNotFound(err) { + return time.Time{}, nil + } + return time.Time{}, fmt.Errorf("read ObjectStore %s/%s: %w", desired.Namespace, desired.Name, err) + } + dry := desired.DeepCopy() + if err := r.Patch(ctx, dry, client.Apply, client.FieldOwner(cnpgFieldManager), client.ForceOwnership, client.DryRunAll); err != nil { + return time.Time{}, fmt.Errorf("dry-run apply ObjectStore %s/%s: %w", desired.Namespace, desired.Name, err) + } + if !apiequality.Semantic.DeepEqual(live.Spec.Configuration, dry.Spec.Configuration) { + return time.Now(), nil + } + recorded, _, err := cnpgRecordedTime(live.Annotations[cnpgDestinationChangedAtAnnotation]) + if err != nil { + // Nothing says when the cache expires; this write's time is the + // latest it can be. + return time.Now(), nil + } + return recorded, nil +} + +// destinationClaimHolder returns the BackupJob other than j that the +// ObjectStore of clusterName names as the last to write it, while that job +// is still running and has no cnpg.io/Backup of its own, or "" when there is +// none. Such a job wrote the destination before otherCNPGBackupInFlight can +// see it: it can still be waiting for the plugin cache, or for the rollout of +// the plugin it attached. Once it has a Backup, that Backup holds the +// destination while CloudNativePG runs it, and the claim stops holding, so a +// job stuck after its Backup finished does not hold every later one. +func (r *BackupJobReconciler) destinationClaimHolder(ctx context.Context, j *backupsv1alpha1.BackupJob, clusterName string) (string, error) { + store := &cnpgtypes.ObjectStore{} + if err := r.apiReader().Get(ctx, client.ObjectKey{Namespace: j.Namespace, Name: clusterName}, store); err != nil { + if apierrors.IsNotFound(err) { + return "", nil + } + return "", fmt.Errorf("read ObjectStore %s/%s: %w", j.Namespace, clusterName, err) + } + holder := store.Annotations[cnpgDestinationWrittenByAnnotation] + if holder == "" || holder == j.Name { + return "", nil + } + holderJob := &backupsv1alpha1.BackupJob{} + if err := r.Get(ctx, client.ObjectKey{Namespace: j.Namespace, Name: holder}, holderJob); err != nil { + if apierrors.IsNotFound(err) { + return "", nil + } + return "", err + } + if holderJob.Status.Phase == backupsv1alpha1.BackupJobPhaseSucceeded || holderJob.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + return "", nil + } + holderBackups := &cnpgtypes.BackupList{} + if err := r.apiReader().List(ctx, holderBackups, client.InNamespace(j.Namespace), client.MatchingLabels{ + backupsv1alpha1.OwningJobNameLabel: holder, + backupsv1alpha1.OwningJobNamespaceLabel: j.Namespace, + }); err != nil { + return "", fmt.Errorf("list cnpg.io/Backups of BackupJob %s/%s: %w", j.Namespace, holder, err) + } + if len(holderBackups.Items) > 0 { + return "", nil + } + return holder, nil +} + // cnpgPluginAttachment is what applyClusterPluginBackup read off the live // Cluster before applying the plugin. type cnpgPluginAttachment struct { @@ -869,8 +991,11 @@ type cnpgPluginAttachment struct { // attachedAt is the cnpgPluginAttachedAtAnnotation the Cluster carried, // "" when it carries no record: the plugin was attached by the chart, or // before the record existed. - attachedAt string - clusterStatus cnpgtypes.ClusterStatus + attachedAt string + // destinationChangedAt is the cnpgDestinationChangedAtAnnotation this + // pass applied to the ObjectStore, zero when it wrote none. + destinationChangedAt time.Time + clusterStatus cnpgtypes.ClusterStatus } // cnpgPluginRolloutPending returns why the rollout that attaching the @@ -916,12 +1041,13 @@ func (r *BackupJobReconciler) cnpgPluginRolloutPending(ctx context.Context, name return "", nil } -// cnpgRecordedAttach parses a cnpgPluginAttachedAtAnnotation value; ok is -// false for "", a Cluster that carries no record. A record further ahead of -// this controller's clock than cnpgAttachClockSkewTolerance is an error: -// every instance would predate it, and a deadline counted from it would not -// run out for as long as it lies ahead. -func cnpgRecordedAttach(raw string) (recorded time.Time, ok bool, err error) { +// cnpgRecordedTime parses a recorded time annotation, the plugin attach +// record on the Cluster or the destination change record on the ObjectStore; +// ok is false for "", an object that carries no record. A record further +// ahead of this controller's clock than cnpgAttachClockSkewTolerance is an +// error: every instance would predate it, and a deadline counted from it +// would not run out for as long as it lies ahead. +func cnpgRecordedTime(raw string) (recorded time.Time, ok bool, err error) { if raw == "" { return time.Time{}, false, nil } diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 1d36117fa..bbfc8b0d7 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -1103,7 +1103,7 @@ func TestApplyClusterPluginBackup_NotFoundOnMissingCluster(t *testing.T) { }, } - _, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-missing", tmpl, "postgres-missing", true, true) + _, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-missing", tmpl, "postgres-missing", true, true, "bj") if err == nil { t.Fatalf("expected NotFound error, got nil") } @@ -1743,7 +1743,7 @@ func TestApplyClusterPluginBackup_ReadsTheLiveClusterNotTheCache(t *testing.T) { BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://bucket/x/"}, } - got, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "strategy-name", true, true) + got, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "strategy-name", true, true, "bj") if err != nil { t.Fatalf("unexpected error: %v", err) } @@ -1953,7 +1953,7 @@ func TestApplyClusterPluginBackup_PreservesLiveServerName(t *testing.T) { BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://bucket/x/"}, } - got, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "strategy-name", true, true) + got, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "strategy-name", true, true, "bj") if err != nil { t.Fatalf("unexpected error: %v", err) } @@ -2122,7 +2122,7 @@ func TestApplyClusterPluginBackup_PatchesExistingCluster(t *testing.T) { }, } - if _, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "tenant-app", true, true); err != nil { + if _, err := r.applyClusterPluginBackup(context.Background(), "tenant", "postgres-app", tmpl, "tenant-app", true, true, "bj"); err != nil { t.Fatalf("unexpected error: %v", err) } @@ -2736,6 +2736,17 @@ func TestReconcileCNPG_WaitsForAnotherRunningBackupOfTheApp(t *testing.T) { if got := f.liveDestination(); got != "s3://bucket/" { t.Errorf("destination = %q, want the job's own s3://bucket/", got) } + if ready != nil && ready.Reason == "WaitingForOtherBackup" { + t.Errorf("Ready = %+v, the job was held", ready) + } + f.settleDestination() + _, backups = f.pass() + ours = 0 + for _, b := range backups { + if b.Labels[backupsv1alpha1.OwningJobNameLabel] == "bj" { + ours++ + } + } if ours != 1 { t.Errorf("got %d cnpg.io/Backups for the job, want 1", ours) } @@ -2819,6 +2830,8 @@ func TestReconcileCNPG_ObjectStoreApplyRecovers(t *testing.T) { } failing = false + j, _ = f.pass() + f.settleDestination() j, backups := f.pass() if len(backups) != 1 { t.Fatalf("got %d cnpg.io/Backups after the write recovered, want 1", len(backups)) @@ -3121,6 +3134,562 @@ func TestReconcileCNPG_AttachRecordOlderThanTheLatchKeepsTheLatch(t *testing.T) } } +func (f *cnpgRolloutFixture) liveObjectStore() *cnpgtypes.ObjectStore { + f.t.Helper() + store := &cnpgtypes.ObjectStore{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: rolloutCluster}, store); err != nil { + f.t.Fatalf("get ObjectStore: %v", err) + } + return store +} + +// setDestinationStamp rewrites the destination-change record on the live +// ObjectStore, as a write that long ago, or a corrupt one, would have left it. +func (f *cnpgRolloutFixture) setDestinationStamp(value string) { + f.t.Helper() + store := f.liveObjectStore() + if store.Annotations == nil { + store.Annotations = map[string]string{} + } + store.Annotations[cnpgDestinationChangedAtAnnotation] = value + if err := f.c.Update(context.Background(), store); err != nil { + f.t.Fatalf("stamp ObjectStore: %v", err) + } +} + +// settleDestination ages the destination-change record, when the ObjectStore +// carries one, as if the plugin's cache had expired since that write. +func (f *cnpgRolloutFixture) settleDestination() { + f.t.Helper() + if _, ok := f.liveObjectStore().Annotations[cnpgDestinationChangedAtAnnotation]; ok { + f.setDestinationStamp(time.Now().Add(-time.Minute).UTC().Format(time.RFC3339)) + } +} + +// addJobWithDestination adds BackupJob name of the same application, resolved +// to a strategy of its own that writes destination, and returns a function +// running one pass of it. +func (f *cnpgRolloutFixture) addJobWithDestination(name, destination string) func() *backupsv1alpha1.BackupJob { + f.t.Helper() + ctx := context.Background() + strategy := &strategyv1alpha1.CNPG{ + ObjectMeta: metav1.ObjectMeta{Name: "cnpg-strategy-" + name}, + Spec: strategyv1alpha1.CNPGSpec{Template: strategyv1alpha1.CNPGTemplate{ + BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: destination}, + }}, + } + if err := f.c.Create(ctx, strategy); err != nil { + f.t.Fatalf("seed strategy: %v", err) + } + job := newRolloutBackupJob(time.Now().Add(-time.Hour)) + job.Name = name + if err := f.c.Create(ctx, job); err != nil { + f.t.Fatalf("seed BackupJob %s: %v", name, err) + } + group := strategyv1alpha1.GroupVersion.Group + resolved := &ResolvedBackupConfig{StrategyRef: corev1.TypedLocalObjectReference{APIGroup: &group, Kind: strategyv1alpha1.CNPGStrategyKind, Name: strategy.Name}} + return func() *backupsv1alpha1.BackupJob { + f.t.Helper() + key := client.ObjectKey{Namespace: rolloutNS, Name: name} + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(ctx, key, j); err != nil { + f.t.Fatalf("get BackupJob %s: %v", name, err) + } + if _, err := f.r.reconcileCNPG(ctx, j, resolved); err != nil { + f.t.Fatalf("reconcileCNPG %s: %v", name, err) + } + if err := f.c.Get(ctx, key, j); err != nil { + f.t.Fatalf("get BackupJob %s after reconcile: %v", name, err) + } + return j + } +} + +// Two jobs of the application with different destinations, neither with a +// Backup yet, would otherwise take turns rewriting the ObjectStore, each +// write restarting the other's wait for the plugin cache so that neither +// ever creates one. The job whose write came first holds the destination, +// by its claim until it has a Backup and by that Backup afterwards, and the +// other waits for it. +func TestReconcileCNPG_JobsWithDifferentDestinationsTakeTurns(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://third/") + pass2 := f.addJobWithDestination("bj2", "s3://other/") + + var j2 *backupsv1alpha1.BackupJob + for range 5 { + f.pass() + f.settleDestination() + j2 = pass2() + f.settleDestination() + } + backups := &cnpgtypes.BackupList{} + if err := f.c.List(context.Background(), backups, client.InNamespace(rolloutNS)); err != nil { + t.Fatalf("list cnpg.io/Backups: %v", err) + } + if len(backups.Items) != 1 || backups.Items[0].Labels[backupsv1alpha1.OwningJobNameLabel] != "bj" { + t.Fatalf("cnpg.io/Backups = %+v, want exactly one, of bj, whose write came first", backups.Items) + } + if got := f.liveDestination(); got != "s3://bucket/" { + t.Errorf("destination = %q, want bj's s3://bucket/ under its running Backup", got) + } + if ready := apimeta.FindStatusCondition(j2.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForOtherBackup" || !strings.Contains(ready.Message, "BackupJob bj ") { + t.Errorf("bj2 Ready = %+v, want WaitingForOtherBackup naming bj", ready) + } +} + +// The claim names the job whose write last reached the ObjectStore, whether +// it changed the configuration, created the object or left it as it was; the +// change record is re-sent as it is by a write that changes nothing. +func TestReconcileCNPG_ClaimNamesTheJobThatWroteTheDestination(t *testing.T) { + t.Run("changed", func(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + f.pass() + if got := f.liveObjectStore().Annotations[cnpgDestinationWrittenByAnnotation]; got != "bj" { + t.Errorf("%s = %q, want bj", cnpgDestinationWrittenByAnnotation, got) + } + }) + t.Run("created", func(t *testing.T) { + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)), + newRolloutInstancePod(time.Now().Add(-time.Hour))) + f.pass() + store := f.liveObjectStore() + if got := store.Annotations[cnpgDestinationWrittenByAnnotation]; got != "bj" { + t.Errorf("%s = %q, want bj", cnpgDestinationWrittenByAnnotation, got) + } + if _, ok := store.Annotations[cnpgDestinationChangedAtAnnotation]; ok { + t.Errorf("creating the ObjectStore stamped %s, which would hold the Backup for nothing", cnpgDestinationChangedAtAnnotation) + } + }) + for name, recorded := range map[string]map[string]string{ + "unchanged": {cnpgDestinationWrittenByAnnotation: "bj-old"}, + "unchanged with a stamp": {cnpgDestinationWrittenByAnnotation: "bj-old", cnpgDestinationChangedAtAnnotation: "2026-01-02T03:04:05Z"}, + } { + t.Run(name, func(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://bucket/") + store := f.liveObjectStore() + store.Annotations = recorded + if err := f.c.Update(context.Background(), store); err != nil { + t.Fatalf("seed claim: %v", err) + } + var sent []map[string]string + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + po := &client.PatchOptions{} + po.ApplyOptions(opts) + if _, ok := obj.(*cnpgtypes.ObjectStore); ok && !slices.Contains(po.DryRun, metav1.DryRunAll) { + sent = append(sent, obj.GetAnnotations()) + } + return cl.Patch(ctx, obj, patch, opts...) + }, + }) + f.pass() + if len(sent) != 1 { + t.Fatalf("got %d ObjectStore writes, want 1", len(sent)) + } + if got := sent[0][cnpgDestinationWrittenByAnnotation]; got != "bj" { + t.Errorf("the write sent %s = %q, want bj, the job that wrote", cnpgDestinationWrittenByAnnotation, got) + } + if got, want := sent[0][cnpgDestinationChangedAtAnnotation], recorded[cnpgDestinationChangedAtAnnotation]; got != want { + t.Errorf("the write sent %s = %q, want the recorded %q", cnpgDestinationChangedAtAnnotation, got, want) + } + }) + } +} + +// A claim stops holding the destination once its job has finished or is +// gone, so the next job writes its own. +func TestReconcileCNPG_ClaimOfAFinishedJobDoesNotHold(t *testing.T) { + for name, phase := range map[string]backupsv1alpha1.BackupJobPhase{ + "succeeded": backupsv1alpha1.BackupJobPhaseSucceeded, + "failed": backupsv1alpha1.BackupJobPhaseFailed, + "gone": "", + } { + t.Run(name, func(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + if name != "gone" { + f.seedOtherBackupJob("bj-other", phase) + } + store := f.liveObjectStore() + store.Annotations = map[string]string{cnpgDestinationWrittenByAnnotation: "bj-other"} + if err := f.c.Update(context.Background(), store); err != nil { + t.Fatalf("seed claim: %v", err) + } + j, _ := f.pass() + if got := f.liveDestination(); got != "s3://bucket/" { + t.Errorf("destination = %q, want bj's s3://bucket/", got) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForOtherBackup" { + t.Errorf("Ready = %+v, held by a %s job", ready, name) + } + }) + } +} + +// A job that attaches the plugin writes a configuration the ObjectStore may +// already carry, for instance after a restore re-rendered the Cluster +// without the plugin. Its write still claims the destination, so a job with +// another one waits through the rollout instead of switching it. +func TestReconcileCNPG_UnchangedWriteClaimsTheDestinationForTheRollout(t *testing.T) { + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), + newRolloutCluster(cnpgClusterHealthyPhase), newRolloutInstancePod(time.Now().Add(-time.Hour))) + f.seedOtherBackupJob("bj-old", backupsv1alpha1.BackupJobPhaseSucceeded) + store := &cnpgtypes.ObjectStore{ + ObjectMeta: metav1.ObjectMeta{Namespace: rolloutNS, Name: rolloutCluster, + Annotations: map[string]string{cnpgDestinationWrittenByAnnotation: "bj-old"}}, + Spec: cnpgtypes.ObjectStoreSpec{Configuration: cnpgtypes.BarmanObjectStoreConfiguration{DestinationPath: "s3://bucket/"}}, + } + if err := f.c.Create(context.Background(), store); err != nil { + t.Fatalf("seed ObjectStore: %v", err) + } + pass2 := f.addJobWithDestination("bj2", "s3://other/") + + f.pass() // latch + if j, _ := f.pass(); apimeta.FindStatusCondition(j.Status.Conditions, "Ready").Reason != "WaitingForPluginRollout" { + t.Fatalf("Ready = %+v, want bj waiting for the rollout of the plugin it attached", apimeta.FindStatusCondition(j.Status.Conditions, "Ready")) + } + j2 := pass2() + if got := f.liveDestination(); got != "s3://bucket/" { + t.Errorf("destination = %q, switched during bj's rollout", got) + } + if ready := apimeta.FindStatusCondition(j2.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForOtherBackup" || !strings.Contains(ready.Message, "BackupJob bj ") { + t.Errorf("bj2 Ready = %+v, want WaitingForOtherBackup naming bj", ready) + } +} + +// The claim covers only the window before its job has a Backup the queue +// can see. Once it has one, the Backup in flight holds the destination, for +// as long as CloudNativePG runs it, and a Backup that finished does not, even +// if its job is stuck before it finishes, so that job cannot hold every later +// backup of the application for as long as it lives. +func TestReconcileCNPG_ClaimOfAJobWithItsOwnBackupDoesNotHold(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + f.seedOtherBackupJob("bj-other", backupsv1alpha1.BackupJobPhaseRunning) + f.seedCNPGBackup("bj-other-cnpg", "bj-other", rolloutCluster, cnpgBackupPhaseComplete) + store := f.liveObjectStore() + store.Annotations = map[string]string{cnpgDestinationWrittenByAnnotation: "bj-other"} + if err := f.c.Update(context.Background(), store); err != nil { + t.Fatalf("seed claim: %v", err) + } + j, _ := f.pass() + if got := f.liveDestination(); got != "s3://bucket/" { + t.Errorf("destination = %q, held by a job whose Backup already finished", got) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready != nil && ready.Reason == "WaitingForOtherBackup" { + t.Errorf("Ready = %+v, held by a job whose Backup already finished", ready) + } +} + +// A claim whose job cannot be read cannot be taken as released: writing over +// it would let two jobs flip the destination again. The error goes back to +// the controller for a retry. +func TestReconcileCNPG_ClaimHolderLookupErrorIsReturned(t *testing.T) { + unavailable := apierrors.NewServiceUnavailable("apiserver unavailable") + for name, setup := range map[string]func(f *cnpgRolloutFixture){ + "holder job read fails": func(f *cnpgRolloutFixture) { + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*backupsv1alpha1.BackupJob); ok && key.Name == "bj-other" { + return unavailable + } + return cl.Get(ctx, key, obj, opts...) + }, + }) + }, + "holder Backups list fails": func(f *cnpgRolloutFixture) { + holders := labels.Set{backupsv1alpha1.OwningJobNameLabel: "bj-other", backupsv1alpha1.OwningJobNamespaceLabel: rolloutNS} + f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + lo := &client.ListOptions{} + lo.ApplyOptions(opts) + if _, ok := list.(*cnpgtypes.BackupList); ok && lo.LabelSelector != nil && lo.LabelSelector.Matches(holders) && !lo.LabelSelector.Matches(labels.Set{backupsv1alpha1.OwningJobNameLabel: "bj", backupsv1alpha1.OwningJobNamespaceLabel: rolloutNS}) { + return unavailable + } + return cl.List(ctx, list, opts...) + }, + }) + }, + } { + t.Run(name, func(t *testing.T) { + assertClaimHolderLookupErrorReturned(t, setup) + }) + } +} + +func assertClaimHolderLookupErrorReturned(t *testing.T, setup func(f *cnpgRolloutFixture)) { + t.Helper() + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + f.seedOtherBackupJob("bj-other", backupsv1alpha1.BackupJobPhaseRunning) + store := f.liveObjectStore() + store.Annotations = map[string]string{cnpgDestinationWrittenByAnnotation: "bj-other"} + if err := f.c.Update(context.Background(), store); err != nil { + t.Fatalf("seed claim: %v", err) + } + setup(f) + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(context.Background(), client.ObjectKey{Namespace: rolloutNS, Name: "bj"}, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + if _, err := f.r.reconcileCNPG(context.Background(), j, f.resolved); err == nil { + t.Fatalf("reconcileCNPG swallowed the claim holder lookup error") + } + if got := f.liveDestination(); got != "s3://other/" { + t.Errorf("destination written although the claim holder could not be read: %q", got) + } +} + +// A running job that holds the claim keeps the destination while it has no +// Backup yet, for instance while the plugin it attached rolls the instances. +func TestReconcileCNPG_ClaimOfARunningJobHolds(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + f.seedOtherBackupJob("bj-other", backupsv1alpha1.BackupJobPhaseRunning) + store := f.liveObjectStore() + store.Annotations = map[string]string{cnpgDestinationWrittenByAnnotation: "bj-other"} + if err := f.c.Update(context.Background(), store); err != nil { + t.Fatalf("seed claim: %v", err) + } + j, backups, res := f.passResult() + if got := f.liveDestination(); got != "s3://other/" { + t.Errorf("destination = %q, rewritten under bj-other's claim", got) + } + if len(backups) != 0 { + t.Errorf("a cnpg.io/Backup was created under another job's claim: %+v", backups) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForOtherBackup" || !strings.Contains(ready.Message, "bj-other") { + t.Errorf("Ready = %+v, want WaitingForOtherBackup naming bj-other", ready) + } + if res.RequeueAfter == 0 { + t.Errorf("a held job is not requeued: %+v", res) + } +} + +// The plugin sidecar reads the ObjectStore through a cache that nothing this +// controller writes invalidates, so a Backup created right after the +// destination changed can be taken against the previous one. The job holds +// the Backup until that cache has expired, and creates it afterwards. +func TestReconcileCNPG_BackupWaitsForThePluginCacheAfterADestinationChange(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + + j, backups, res := f.passResult() + if got := f.liveDestination(); got != "s3://bucket/" { + t.Fatalf("destination = %q, want the job's s3://bucket/", got) + } + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created in the pass that changed the destination: %+v", backups) + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForDestinationToSettle" { + t.Errorf("Ready = %+v, want WaitingForDestinationToSettle", ready) + } + if res.RequeueAfter == 0 { + t.Errorf("a waiting job is not requeued: %+v", res) + } + stamp, err := time.Parse(time.RFC3339, f.liveObjectStore().Annotations[cnpgDestinationChangedAtAnnotation]) + if err != nil || time.Since(stamp) > time.Minute { + t.Fatalf("%s = %v (%v), want the time of this write", cnpgDestinationChangedAtAnnotation, stamp, err) + } + + if _, backups = f.pass(); len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created while the plugin cache could still hold the old destination: %+v", backups) + } + + f.setDestinationStamp(time.Now().Add(-time.Minute).UTC().Format(time.RFC3339)) + if j, backups = f.pass(); len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups once the cache expired, want 1; Ready = %+v", len(backups), apimeta.FindStatusCondition(j.Status.Conditions, "Ready")) + } +} + +// A write that leaves the configuration as it was cannot leave a stale +// cache behind: the Backup is created at once, and the record of the last +// change is re-sent as it is, since an apply that left it out would delete it. +func TestReconcileCNPG_UnchangedDestinationDoesNotWait(t *testing.T) { + const stamp = "2026-01-02T03:04:05Z" + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://bucket/") + f.setDestinationStamp(stamp) + + if j, backups := f.pass(); len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups for an unchanged destination, want 1; Ready = %+v", len(backups), apimeta.FindStatusCondition(j.Status.Conditions, "Ready")) + } + if got := f.liveObjectStore().Annotations[cnpgDestinationChangedAtAnnotation]; got != stamp { + t.Errorf("%s = %q, want the recorded %q", cnpgDestinationChangedAtAnnotation, got, stamp) + } +} + +// The apiserver answers an apply with the merged object: fields it defaults, +// or that another field manager owns, are there although the job never sets +// them. Judged against what the job sends, every pass would read as a change +// and the Backup would never be created; judged against the dry run of the +// same apply, an unchanged configuration stays unchanged. +func TestReconcileCNPG_FieldsTheJobDoesNotSetAreNotAChange(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://bucket/") + store := f.liveObjectStore() + store.Spec.Configuration.Wal = &cnpgtypes.WalBackupConfiguration{Compression: "gzip"} + if err := f.c.Update(context.Background(), store); err != nil { + t.Fatalf("seed ObjectStore: %v", err) + } + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + po := &client.PatchOptions{} + po.ApplyOptions(opts) + if os, ok := obj.(*cnpgtypes.ObjectStore); ok && slices.Contains(po.DryRun, metav1.DryRunAll) { + os.Spec.Configuration.Wal = &cnpgtypes.WalBackupConfiguration{Compression: "gzip"} + return nil + } + return cl.Patch(ctx, obj, patch, opts...) + }, + }) + + if j, backups := f.pass(); len(backups) != 1 { + t.Fatalf("got %d cnpg.io/Backups for a configuration only the server filled in, want 1; Ready = %+v", len(backups), apimeta.FindStatusCondition(j.Status.Conditions, "Ready")) + } + if _, ok := f.liveObjectStore().Annotations[cnpgDestinationChangedAtAnnotation]; ok { + t.Errorf("%s was set by a write that changed nothing", cnpgDestinationChangedAtAnnotation) + } +} + +// Without the live ObjectStore, or the dry run of the apply, the job cannot +// tell whether its write changes the destination, and taking the write as +// unchanged would create the Backup at once against a cache that may still +// hold the previous one. Either failure is retried like a failed write. +func TestReconcileCNPG_UnreadableObjectStoreIsRetried(t *testing.T) { + unavailable := apierrors.NewServiceUnavailable("apiserver unavailable") + for _, tc := range []struct { + name string + setup func(f *cnpgRolloutFixture) + }{ + {name: "live read fails", setup: func(f *cnpgRolloutFixture) { + f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*cnpgtypes.ObjectStore); ok { + return unavailable + } + return cl.Get(ctx, key, obj, opts...) + }, + }) + }}, + {name: "claim read fails", setup: func(f *cnpgRolloutFixture) { + reads := 0 + f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*cnpgtypes.ObjectStore); ok { + if reads++; reads == 1 { + return unavailable + } + } + return cl.Get(ctx, key, obj, opts...) + }, + }) + }}, + {name: "live read at the write fails", setup: func(f *cnpgRolloutFixture) { + reads := 0 + f.r.APIReader = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Get: func(ctx context.Context, cl client.WithWatch, key client.ObjectKey, obj client.Object, opts ...client.GetOption) error { + if _, ok := obj.(*cnpgtypes.ObjectStore); ok { + if reads++; reads > 1 { + return unavailable + } + } + return cl.Get(ctx, key, obj, opts...) + }, + }) + }}, + {name: "dry run fails", setup: func(f *cnpgRolloutFixture) { + f.r.Client = interceptor.NewClient(f.c.(client.WithWatch), interceptor.Funcs{ + Patch: func(ctx context.Context, cl client.WithWatch, obj client.Object, patch client.Patch, opts ...client.PatchOption) error { + po := &client.PatchOptions{} + po.ApplyOptions(opts) + if _, ok := obj.(*cnpgtypes.ObjectStore); ok && slices.Contains(po.DryRun, metav1.DryRunAll) { + return unavailable + } + return cl.Patch(ctx, obj, patch, opts...) + }, + }) + }}, + } { + t.Run(tc.name, func(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://other/") + tc.setup(f) + assertObjectStoreReadRetried(t, f) + }) + } +} + +// assertObjectStoreReadRetried runs one pass and checks it is retried, either +// by returning the error to the controller or by waiting with +// ClusterPluginApplyRetrying, without a Backup, a destination write or a +// failed job. +func assertObjectStoreReadRetried(t *testing.T, f *cnpgRolloutFixture) { + t.Helper() + ctx := context.Background() + key := client.ObjectKey{Namespace: rolloutNS, Name: "bj"} + j := &backupsv1alpha1.BackupJob{} + if err := f.c.Get(ctx, key, j); err != nil { + t.Fatalf("get BackupJob: %v", err) + } + res, reconcileErr := f.r.reconcileCNPG(ctx, j, f.resolved) + if err := f.c.Get(ctx, key, j); err != nil { + t.Fatalf("get BackupJob after reconcile: %v", err) + } + backups := &cnpgtypes.BackupList{} + if err := f.c.List(ctx, backups, client.InNamespace(rolloutNS)); err != nil { + t.Fatalf("list cnpg.io/Backups: %v", err) + } + if len(backups.Items) != 0 { + t.Fatalf("a cnpg.io/Backup was created although the ObjectStore could not be read: %+v", backups.Items) + } + if got := f.liveDestination(); got != "s3://other/" { + t.Errorf("destination = %q, written without knowing whether it changed", got) + } + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("the job failed on a transient read: %s", j.Status.Message) + } + if reconcileErr != nil { + return + } + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "ClusterPluginApplyRetrying" { + t.Errorf("Ready = %+v, want ClusterPluginApplyRetrying", ready) + } + if res.RequeueAfter == 0 { + t.Errorf("a retried read is not requeued: %+v", res) + } +} + +// A change record that cannot be read, or that lies further ahead than +// controller clocks drift, says nothing about when the cache expires, so the +// write replaces it with its own time and the job waits once. It claims the +// destination for that wait, or another job would write its own meanwhile. +func TestReconcileCNPG_UnusableDestinationRecordIsReplaced(t *testing.T) { + for name, value := range map[string]string{ + "does not parse": "yesterday", + "far ahead": time.Now().Add(time.Hour).UTC().Format(time.RFC3339), + } { + t.Run(name, func(t *testing.T) { + f := newAttachedObjectStoreFixture(t, time.Now().Add(-time.Hour), "s3://bucket/") + f.seedOtherBackupJob("bj-old", backupsv1alpha1.BackupJobPhaseSucceeded) + store := f.liveObjectStore() + store.Annotations = map[string]string{cnpgDestinationWrittenByAnnotation: "bj-old"} + if err := f.c.Update(context.Background(), store); err != nil { + t.Fatalf("seed claim: %v", err) + } + f.setDestinationStamp(value) + + j, backups := f.pass() + if len(backups) != 0 { + t.Fatalf("a cnpg.io/Backup was created with an unusable change record: %+v", backups) + } + if j.Status.Phase == backupsv1alpha1.BackupJobPhaseFailed { + t.Fatalf("the job failed: %s", j.Status.Message) + } + stamp, err := time.Parse(time.RFC3339, f.liveObjectStore().Annotations[cnpgDestinationChangedAtAnnotation]) + if err != nil || time.Since(stamp) > time.Minute || time.Until(stamp) > time.Minute { + t.Fatalf("%s = %v (%v), want it replaced with the time of this write", cnpgDestinationChangedAtAnnotation, stamp, err) + } + if got := f.liveObjectStore().Annotations[cnpgDestinationWrittenByAnnotation]; got != "bj" { + t.Errorf("%s = %q, want bj, which waits on the replaced record", cnpgDestinationWrittenByAnnotation, got) + } + }) + } +} + // A later write to a Cluster that already carries the plugin re-sends the // attach record as it is: moving it would make the instances that rolled // with the plugin look older than the attach. @@ -5913,9 +6482,11 @@ func TestReconcileCNPG_ArtifactNamesTheDestinationItsBackupWasCreatedWith(t *tes f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") f.setStrategyDestination("s3://bucket/", "https://s3.one", "creds-one") + f.pass() + f.settleDestination() j, backups := f.pass() if len(backups) != 1 { - t.Fatalf("got %d cnpg.io/Backups after the first pass, want 1; Ready = %+v", len(backups), apimeta.FindStatusCondition(j.Status.Conditions, "Ready")) + t.Fatalf("got %d cnpg.io/Backups once the destination settled, want 1; Ready = %+v", len(backups), apimeta.FindStatusCondition(j.Status.Conditions, "Ready")) } if got := f.liveDestination(); got != "s3://bucket/" { t.Fatalf("destination = %q, want s3://bucket/", got) @@ -6021,7 +6592,7 @@ func TestReconcileCNPG_ArtifactRefusedForAnUnusableDestinationRecord(t *testing. "mistyped endpoint": `{"destinationPath":"s3://bucket/","endpointURL":7}`, } { t.Run(name, func(t *testing.T) { - f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://bucket/") if _, backups := f.pass(); len(backups) != 1 { t.Fatalf("got %d cnpg.io/Backups after the first pass, want 1", len(backups)) } @@ -6175,7 +6746,7 @@ func TestReconcileCNPG_BackupListErrorIsReturned(t *testing.T) { // start: a second lookup before the create could only fail, after the write, // and fail the job with it. func TestReconcileCNPG_LooksUpItsOwnBackupOncePerPass(t *testing.T) { - f := newAttachedObjectStoreFixture(t, time.Now(), "s3://other/") + f := newAttachedObjectStoreFixture(t, time.Now(), "s3://bucket/") f.failingBackupLists(func(own bool, n int) bool { return own && n > 1 }) j, backups := f.pass() From fb8a7ef278803fbc3c80b0fade9434c44d0f7b19 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 17:21:15 +0500 Subject: [PATCH 456/518] docs(kubernetes): describe the cluster as it is and drop the history The README told users two things the chart does not do: that it deploys a dedicated etcd cluster (the control plane uses the tenant's etcd, which the Tenant deploys), and that worker disks match the volume's block size through blockSize.matchVolume, a setting that exists nowhere in the tree. It also described worker nodes as KubeVirt VMs managed by this application, while they are KubernetesNodes pools on KubeVirt or Proxmox, and offered "patch versions" where the user picks a minor. The Breaking Changes section was a release log: entries from several releases, most of them about fields that live on KubernetesNodes now, and each one written as a change rather than as the current behaviour. What a user has to do on upgrade belongs on the per-release upgrade page of the website, so the README points there and keeps only the storageClass immutability note, rewritten as current behaviour. The field descriptions are what the dashboard, the API schema and a rejected name show the user, and several were written against how the application used to work. The name limit was explained by a default md0 pool that no longer exists, while the 32-character cap leaves room for any three-character pool name. The host default left out the kubernetes- prefix the release name carries. A substrate mismatch with a pool was said to leave Machines unreconciled, while the pool refuses to render. The OIDC type carried a roadmap phase in its title, and extraArgs promised "no change to current behavior". The name-cap test looked for "md0" in the rejection message as its sign that the chart's explanation arrives; it looks for the release-name pattern the new explanation gives. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- api/apps/v1alpha1/kubernetes/types.go | 6 +- packages/apps/kubernetes/README.md | 147 ++++++++---------- packages/apps/kubernetes/values.schema.json | 8 +- packages/apps/kubernetes/values.yaml | 10 +- .../kubernetes-rd/cozyrds/kubernetes.yaml | 2 +- .../apps/application/rest_name_schema_test.go | 2 +- 6 files changed, 78 insertions(+), 97 deletions(-) diff --git a/api/apps/v1alpha1/kubernetes/types.go b/api/apps/v1alpha1/kubernetes/types.go index fc6f19746..8500d70ee 100644 --- a/api/apps/v1alpha1/kubernetes/types.go +++ b/api/apps/v1alpha1/kubernetes/types.go @@ -25,10 +25,10 @@ type ConfigSpec struct { // Kubernetes major.minor version to deploy // +kubebuilder:default:="v1.35" Version Version `json:"version"` - // External hostname for Kubernetes cluster. Defaults to `.` if empty. + // External hostname for Kubernetes cluster. Defaults to `kubernetes-.` if empty. // +kubebuilder:default:="" Host string `json:"host"` - // Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one. + // Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one. // +kubebuilder:default:="kubevirt" // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="substrate is immutable" Substrate string `json:"substrate"` @@ -50,7 +50,7 @@ type APIServer struct { // Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it. // +kubebuilder:default:={} AdmissionControllers []AdmissionController `json:"admissionControllers,omitempty"` - // Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior). + // Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default. // +kubebuilder:default:={} ExtraArgs []string `json:"extraArgs,omitempty"` // Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default. diff --git a/packages/apps/kubernetes/README.md b/packages/apps/kubernetes/README.md index bd4d83a64..f7d1062b1 100644 --- a/packages/apps/kubernetes/README.md +++ b/packages/apps/kubernetes/README.md @@ -8,11 +8,9 @@ Cozystack deploys and manages Kubernetes-as-a-service as standalone applications In Cozystack, such clusters are named tenant Kubernetes clusters, while the base Cozystack cluster is called a management or root cluster. Tenant clusters are fully separated from the management cluster and are intended for deploying tenant-specific or customer-developed applications. -Within a tenant cluster, users can take advantage of LoadBalancer services and easily provision physical volumes as needed. -The control-plane operates within containers, while the worker nodes are deployed as virtual machines, all seamlessly managed by the application. +Within a tenant cluster, users can take advantage of LoadBalancer services and easily provision physical volumes as needed. The control plane runs in containers in the management cluster, and the worker nodes are virtual machines, grouped into pools that are separate `KubernetesNodes` resources. -Kubernetes version in tenant clusters is independent of Kubernetes in the management cluster. -Users can select the supported patch versions from 1.31 to 1.35. +Kubernetes version in tenant clusters is independent of Kubernetes in the management cluster. Users choose the Kubernetes minor version, from 1.31 to 1.35, and the chart deploys the patch release it pins for that minor. ## Why Use a Managed Kubernetes Cluster? @@ -58,14 +56,9 @@ Deploying it involves the following components: Each control plane pod includes essential components like `kube-apiserver`, `controller-manager`, and `scheduler`, allowing for efficient multi-tenancy and resource utilization. -- **Etcd Cluster**: A dedicated etcd cluster is deployed using the cozystack [etcd-operator](https://github.com/cozystack/etcd-operator) (`etcd-operator.cozystack.io/v1alpha2`). - It provides reliable and scalable key-value storage for the Kubernetes control plane. +- **Etcd**: the control plane keeps its state in the tenant's etcd, which this application does not deploy. It is the etcd the Tenant runs when `etcd` is enabled on it or on one of its parent tenants, built with the cozystack [etcd-operator](https://github.com/cozystack/etcd-operator), and every tenant Kubernetes cluster that uses it shares it. Until that etcd exists, the application creates only a `-awaiting-etcd` ConfigMap that says so, and provisions the cluster once the etcd is there. -- **Worker Nodes**: Virtual Machines are provisioned to serve as worker nodes using KubeVirt. - These nodes are configured to join the tenant Kubernetes cluster, enabling the deployment and management of workloads. - Worker node disks automatically detect and match the underlying volume's block size - (`blockSize.matchVolume`) to ensure compatibility with storage backends that use - non-512-byte sectors, such as LINSTOR DRBD with 4Kn drives. +- **Worker Nodes**: worker nodes are virtual machines running Talos Linux that join the tenant cluster. They come in pools, and each pool is a separate `KubernetesNodes` resource that names this cluster and sets its own size, disk, Talos image and scaling bounds; this application creates no worker nodes itself. With `substrate: kubevirt` the virtual machines run on KubeVirt in the management cluster, and with `substrate: proxmox` on an external Proxmox VE cluster. - **Cluster API**: Cozystack is using the [Kubernetes Cluster API](https://cluster-api.sigs.k8s.io/) to provision the components of a cluster. @@ -83,23 +76,11 @@ See the reference for components utilized in this service: - [github.com/kubernetes-sigs/cluster-api-provider-kubevirt](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt) - [github.com/kubevirt/csi-driver](https://github.com/kubevirt/csi-driver) -## Breaking Changes +The top-level `storageClass` is the default StorageClass inside the tenant cluster and cannot be changed after the cluster is created; see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. Each worker pool sets the StorageClass of its own disks on its `KubernetesNodes` resource. -- **`ephemeralStorage` renamed to `diskSize`** (v1.4): The `nodeGroups[name].ephemeralStorage` field has been renamed to `nodeGroups[name].diskSize`. Existing clusters are migrated transparently by platform migration 41 during the pre-upgrade hook — no manual action is required. Newly written values should use `diskSize`. Existing VMs are rolling-updated via CAPI when the new values are applied. With the Talos worker rollover in this release the field now sizes the **single system disk** (Talos OS image streamed from `factory.talos.dev`, kubelet state, containerd image cache, local-path PVCs) — the pre-Talos `disk-kubelet` PVC layout was removed. State on that single disk persists across same-VM reboots (virt-launcher restart, guest reboot, node failure); VM replacement by CAPI (e.g. nodeGroup field change, MachineHealthCheck remediation) provisions a fresh disk. +## Upgrading -- **Air-gapped tenant workers (Phase 1 Talos rollover)**: tenant worker bootstrap moves from Ubuntu containerDisk + kubeadm to a Talos OS disk image fetched over HTTP by CDI and a `.../installer/...` installer reference. Both sources are now overridable on each worker pool, which is a KubernetesNodes resource: set its `talos.imageFactoryURL` (default `https://factory.talos.dev`) to a self-hosted Image Factory, a caching mirror, or an internal HTTP file server, and its `talos.installerRepository` (default `factory.talos.dev/installer`) to a mirrored registry. The `Kubernetes` resource itself has no `talos` field. The Helm-rendered `*-patch-containerd` Secret that previously plumbed the platform-wide `registries.mirrors` config to tenant workers (via the kubeadm template's containerd certs.d mount) has no consumer in the Talos machineconfig, so the Secret is removed together with the platform `registries` value that fed it. A Package that still sets `registries` keeps installing, the key is ignored. Registry mirrors for tenant workers, **in-guest container-image pulls** (pods pulling from registries inside the tenant cluster) included, are configured per worker pool through `talos.registryMirrors` of the `KubernetesNodes` resource. - -- **Worker MachineHealthCheck remediation is now enabled by default** (Phase 1 Talos rollover): the worker MHC `maxUnhealthy` moved from a hard-coded `0` (remediation effectively **off**) to a configurable `nodeHealthCheck.maxUnhealthy` defaulting to `"50%"`. CAPI now auto-remediates (deletes and replaces) unhealthy worker Machines while at least 50% of a group stays healthy. `"50%"` deliberately leaves headroom for transient unhealthy nodes during the kubeadm→Talos rollover and slow first boots from `factory.talos.dev`; set `nodeHealthCheck.maxUnhealthy: "0%"` to keep the previous remediation-off behaviour until your fleet is stable on Talos workers. - -- **GitOps-managed tenant `Kubernetes` CRs must bump `spec.version` in Git** (Phase 1 Talos rollover): platform migration 46 patches live `kuberneteses.apps.cozystack.io` objects from `v1.30` to `v1.31` (v1.30 left the Talos↔Kubernetes support matrix). When the tenant `Kubernetes` CR is reconciled from Git (Flux/Argo), the next source reconcile re-applies `version: v1.30` over migration 46's patch, which then trips the chart's `_versions.tpl` guard and the HelmRelease fails. Update `spec.version` to `v1.31` (or newer) in your Git source before/with the platform upgrade. Tenants managed via the API or dashboard need no action — migration 46 handles them. - -- **Remote-accessible LINSTOR StorageClasses are auto-propagated to tenants** (v1.5): infra-cluster LINSTOR StorageClasses whose `linstor.csi.linbit.com/allowRemoteVolumeAccess` is not `"false"` are created inside each tenant under the same name (provisioned by `csi.kubevirt.io`). **Upgrade caveat:** delete any *manually created* tenant StorageClass whose name collides with a propagated infra class (e.g. a hand-made `replicated`) before upgrading, or the propagated class cannot be created and the tenant CSI release will not converge. Infra classes that must stay node-local need `allowRemoteVolumeAccess: "false"` set explicitly — an absent annotation is treated as remote-accessible. Propagation is evaluated at HelmRelease render time, so classes added or removed on the infra cluster after a tenant exists propagate only on that tenant's next reconcile. - -- **Worker pools split into `KubernetesNodes` (Phase 2)**: `spec.nodeGroups`, `spec.nodeHealthCheck` and `spec.maxNodeProvisionTime` are removed from the `Kubernetes` CR — worker pools are now separate `KubernetesNodes` resources (one HelmRelease per pool; see the `kubernetes-nodes` chart), on which `spec.cluster` and `spec.storageClass` are immutable (`self == oldSelf`). Existing pools are adopted automatically on upgrade by platform migration 56 with no worker-VM churn. Consequences: (1) the `Kubernetes` CR's `WorkloadsReady` condition is now **control-plane-only** — worker health is reported on each `KubernetesNodes` resource, so a dashboard or `kubectl wait` watching the parent no longer reflects worker readiness; (2) enabling `addons.ingressNginx` now requires a pool carrying `roles: [ingress-nginx]` (the implicit default `md0` is gone — without such a pool the controller DaemonSet schedules nothing); (3) do **not** edit `Kubernetes` CRs during a platform upgrade — a mid-window parent render can transiently fail Helm ownership validation while migration 56 re-annotates worker objects, and self-heals once the new chart artifact lands; (4) when deleting a cluster, delete its `KubernetesNodes` pools **before** the parent `Kubernetes` resource, so each pool's pre-delete unpin hook can run while the tenant apiserver and the CAPI CRDs still exist — deleting the parent first can leave the pool release reconciling CAPI objects against a cluster that is being torn down (in the pre-split monolith the pools were part of the parent release and died with it; they are now separate resources with their own lifecycle); (5) the removed fields (`nodeGroups`, `nodeHealthCheck`, `maxNodeProvisionTime`) are still **accepted and stored** on an already-upgraded `Kubernetes` CR — the adoption migration deliberately leaves them in place rather than rewriting parent values mid-upgrade — but they have **no effect**: editing them (for example scaling via `spec.nodeGroups..minReplicas`) does nothing, and the API returns an admission warning pointing you at the `KubernetesNodes` resources. A cluster name is also capped at 32 characters so every pool's `KubernetesNodes` release name fits the 53-character Helm limit. ComputePlane clusters (the `computeplane` module) are affected the same way: their worker pools are rendered as separate per-pool `KubernetesNodes` releases too (materialised directly by the module rather than adopted), so the same pool-name length limit applies to their `nodeGroups` keys. - -- **`controlPlane.apiServer.extraArgs` refuses flags the tenant control plane owns** (Kamaji 26.9.4-edge): the tenant control plane now sets a list of kube-apiserver flags itself and drops an `extraArgs` entry naming one, where the previous build let the entry win. The entry stops taking effect once the new build reconciles the control plane, whatever the chart renders. Two of those flags have a field the control plane renders them from, and the chart moves an entry written as `--flag=value` into it while the field is empty: `--enable-admission-plugins` into `controlPlane.apiServer.admissionControllers`, `--kubelet-preferred-address-types` into `controlPlane.kubelet.preferredAddressTypes`. The KamajiControlPlane CRD accepts only the plugin names it lists, so a plugin outside that list that the apiserver leaves off by default can no longer be enabled, and a moved name outside it fails when the KamajiControlPlane is applied, with an error that names it, rather than at render. The chart refuses to render any other entry naming one of the flags, so a tenant carrying one stops reconciling until the entry is removed; the description of `controlPlane.apiServer.extraArgs` lists the flags. The same Kamaji update passes every `extraArgs` entry to the apiserver, so repeats of a flag no longer collapse to the last one: naming the kube feature gates both bare and `kube:`-prefixed across entries, or giving two `--emulated-version` entries that name kube, now stops the apiserver from starting. The chart refuses neither, in any `spec.oidc.mode`: both reach the apiserver as written. Keep one spelling for kube gate terms and one kube emulated version. - -> The top-level `storageClass` field is annotated as immutable in the chart schema — see [storageClass immutability](https://cozystack.io/docs/v1/operations/storage-class-immutability/) for the contract and which consumers enforce it. The per-pool worker `storageClass` now lives on the `KubernetesNodes` resource (see the `kubernetes-nodes` chart). +Changes that need action when Cozystack is upgraded are described in the upgrade notes of each release, for example [Upgrading from v1.6 to v1.7](https://cozystack.io/docs/v1.7/operations/cluster/upgrade-to-v1.7/). ## Proxmox substrate: credentials, privileges and tenant scope @@ -133,36 +114,36 @@ A storage shared between tenants cannot be scoped: `Datastore.Allocate` on it is ### Application-specific Parameters -| Name | Description | Type | Value | -| --------- | ---------------------------------------------------------------------------------------------- | -------- | ------- | -| `version` | Kubernetes major.minor version to deploy | `string` | `v1.35` | -| `host` | External hostname for Kubernetes cluster. Defaults to `.` if empty. | `string` | `""` | +| Name | Description | Type | Value | +| --------- | --------------------------------------------------------------------------------------------------------- | -------- | ------- | +| `version` | Kubernetes major.minor version to deploy | `string` | `v1.35` | +| `host` | External hostname for Kubernetes cluster. Defaults to `kubernetes-.` if empty. | `string` | `""` | ### Substrate -| Name | Description | Type | Value | -| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------- | -| `substrate` | Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one. | `string` | `kubevirt` | -| `proxmox` | Proxmox substrate settings. | `object` | `{}` | -| `proxmox.allowedNodes` | Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster. | `[]string` | `[]` | -| `proxmox.dnsServers` | Nameservers written into each worker's network config. Required by the ProxmoxCluster schema. | `[]string` | `[]` | -| `proxmox.insecure` | Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts. | `bool` | `false` | -| `proxmox.ipv4Config` | Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional. | `object` | `{}` | -| `proxmox.ipv4Config.addresses` | Ranges or CIDRs, e.g. `["10.0.0.120-10.0.0.170"]`. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address. | `[]string` | `[]` | -| `proxmox.ipv4Config.gateway` | Default gateway for the workers. | `string` | `""` | -| `proxmox.ipv4Config.prefix` | Netmask prefix length. | `int` | `24` | -| `proxmox.ccm` | Proxmox cloud-controller-manager settings. | `object` | `{}` | -| `proxmox.ccm.credentialsSecretName` | Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks. | `string` | `""` | -| `proxmox.csi` | Proxmox CSI driver settings. | `object` | `{}` | -| `proxmox.csi.enabled` | Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all. | `bool` | `true` | -| `proxmox.csi.credentialsSecretName` | Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs. | `string` | `""` | -| `proxmox.csi.storageClasses` | StorageClasses to create in the tenant. | `[]object` | `[]` | -| `proxmox.csi.storageClasses[i].name` | StorageClass name inside the tenant cluster. | `string` | `""` | -| `proxmox.csi.storageClasses[i].storage` | Proxmox storage id the volumes are created on (as it appears in `pvesm status`). | `string` | `""` | -| `proxmox.csi.storageClasses[i].fstype` | Filesystem the driver formats volumes with: ext4 or xfs. | `string` | `""` | -| `proxmox.csi.storageClasses[i].ssd` | Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies. | `bool` | `false` | -| `proxmox.csi.storageClasses[i].reclaimPolicy` | Delete or Retain. | `string` | `""` | +| Name | Description | Type | Value | +| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ---------- | +| `substrate` | Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one. | `string` | `kubevirt` | +| `proxmox` | Proxmox substrate settings. | `object` | `{}` | +| `proxmox.allowedNodes` | Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster. | `[]string` | `[]` | +| `proxmox.dnsServers` | Nameservers written into each worker's network config. Required by the ProxmoxCluster schema. | `[]string` | `[]` | +| `proxmox.insecure` | Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts. | `bool` | `false` | +| `proxmox.ipv4Config` | Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional. | `object` | `{}` | +| `proxmox.ipv4Config.addresses` | Ranges or CIDRs, e.g. `["10.0.0.120-10.0.0.170"]`. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address. | `[]string` | `[]` | +| `proxmox.ipv4Config.gateway` | Default gateway for the workers. | `string` | `""` | +| `proxmox.ipv4Config.prefix` | Netmask prefix length. | `int` | `24` | +| `proxmox.ccm` | Proxmox cloud-controller-manager settings. | `object` | `{}` | +| `proxmox.ccm.credentialsSecretName` | Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks. | `string` | `""` | +| `proxmox.csi` | Proxmox CSI driver settings. | `object` | `{}` | +| `proxmox.csi.enabled` | Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all. | `bool` | `true` | +| `proxmox.csi.credentialsSecretName` | Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs. | `string` | `""` | +| `proxmox.csi.storageClasses` | StorageClasses to create in the tenant. | `[]object` | `[]` | +| `proxmox.csi.storageClasses[i].name` | StorageClass name inside the tenant cluster. | `string` | `""` | +| `proxmox.csi.storageClasses[i].storage` | Proxmox storage id the volumes are created on (as it appears in `pvesm status`). | `string` | `""` | +| `proxmox.csi.storageClasses[i].fstype` | Filesystem the driver formats volumes with: ext4 or xfs. | `string` | `""` | +| `proxmox.csi.storageClasses[i].ssd` | Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies. | `bool` | `false` | +| `proxmox.csi.storageClasses[i].reclaimPolicy` | Delete or Retain. | `string` | `""` | ### Cluster Addons @@ -205,37 +186,37 @@ A storage shared between tenants cannot be scoped: `Datastore.Allocate` on it is ### Kubernetes Control Plane Configuration -| Name | Description | Type | Value | -| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ----------- | -| `controlPlane` | Kubernetes control-plane configuration. | `object` | `{}` | -| `controlPlane.replicas` | Number of control-plane replicas. | `int` | `2` | -| `controlPlane.apiServer` | API Server configuration. | `object` | `{}` | -| `controlPlane.apiServer.resources` | CPU and memory resources for API Server. | `object` | `{}` | -| `controlPlane.apiServer.resources.cpu` | CPU available. | `quantity` | `""` | -| `controlPlane.apiServer.resources.memory` | Memory (RAM) available. | `quantity` | `""` | -| `controlPlane.apiServer.resourcesPreset` | Preset if `resources` omitted. | `string` | `c1.medium` | -| `controlPlane.apiServer.extraArgs` | Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior). | `[]string` | `[]` | -| `controlPlane.apiServer.admissionControllers` | Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it. | `[]string` | `[]` | -| `controlPlane.apiServer.extraVolumes` | Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default. | `[]object` | `[]` | -| `controlPlane.apiServer.extraVolumeMounts` | Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default. | `[]object` | `[]` | -| `controlPlane.controllerManager` | Controller Manager configuration. | `object` | `{}` | -| `controlPlane.controllerManager.resources` | CPU and memory resources for Controller Manager. | `object` | `{}` | -| `controlPlane.controllerManager.resources.cpu` | CPU available. | `quantity` | `""` | -| `controlPlane.controllerManager.resources.memory` | Memory (RAM) available. | `quantity` | `""` | -| `controlPlane.controllerManager.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` | -| `controlPlane.scheduler` | Scheduler configuration. | `object` | `{}` | -| `controlPlane.scheduler.resources` | CPU and memory resources for Scheduler. | `object` | `{}` | -| `controlPlane.scheduler.resources.cpu` | CPU available. | `quantity` | `""` | -| `controlPlane.scheduler.resources.memory` | Memory (RAM) available. | `quantity` | `""` | -| `controlPlane.scheduler.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` | -| `controlPlane.konnectivity` | Konnectivity configuration. | `object` | `{}` | -| `controlPlane.konnectivity.server` | Konnectivity Server configuration. | `object` | `{}` | -| `controlPlane.konnectivity.server.resources` | CPU and memory resources for Konnectivity. | `object` | `{}` | -| `controlPlane.konnectivity.server.resources.cpu` | CPU available. | `quantity` | `""` | -| `controlPlane.konnectivity.server.resources.memory` | Memory (RAM) available. | `quantity` | `""` | -| `controlPlane.konnectivity.server.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` | -| `controlPlane.kubelet` | Kubelet connection settings. | `object` | `{}` | -| `controlPlane.kubelet.preferredAddressTypes` | Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one. | `[]string` | `[]` | +| Name | Description | Type | Value | +| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- | +| `controlPlane` | Kubernetes control-plane configuration. | `object` | `{}` | +| `controlPlane.replicas` | Number of control-plane replicas. | `int` | `2` | +| `controlPlane.apiServer` | API Server configuration. | `object` | `{}` | +| `controlPlane.apiServer.resources` | CPU and memory resources for API Server. | `object` | `{}` | +| `controlPlane.apiServer.resources.cpu` | CPU available. | `quantity` | `""` | +| `controlPlane.apiServer.resources.memory` | Memory (RAM) available. | `quantity` | `""` | +| `controlPlane.apiServer.resourcesPreset` | Preset if `resources` omitted. | `string` | `c1.medium` | +| `controlPlane.apiServer.extraArgs` | Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default. | `[]string` | `[]` | +| `controlPlane.apiServer.admissionControllers` | Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it. | `[]string` | `[]` | +| `controlPlane.apiServer.extraVolumes` | Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default. | `[]object` | `[]` | +| `controlPlane.apiServer.extraVolumeMounts` | Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default. | `[]object` | `[]` | +| `controlPlane.controllerManager` | Controller Manager configuration. | `object` | `{}` | +| `controlPlane.controllerManager.resources` | CPU and memory resources for Controller Manager. | `object` | `{}` | +| `controlPlane.controllerManager.resources.cpu` | CPU available. | `quantity` | `""` | +| `controlPlane.controllerManager.resources.memory` | Memory (RAM) available. | `quantity` | `""` | +| `controlPlane.controllerManager.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` | +| `controlPlane.scheduler` | Scheduler configuration. | `object` | `{}` | +| `controlPlane.scheduler.resources` | CPU and memory resources for Scheduler. | `object` | `{}` | +| `controlPlane.scheduler.resources.cpu` | CPU available. | `quantity` | `""` | +| `controlPlane.scheduler.resources.memory` | Memory (RAM) available. | `quantity` | `""` | +| `controlPlane.scheduler.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` | +| `controlPlane.konnectivity` | Konnectivity configuration. | `object` | `{}` | +| `controlPlane.konnectivity.server` | Konnectivity Server configuration. | `object` | `{}` | +| `controlPlane.konnectivity.server.resources` | CPU and memory resources for Konnectivity. | `object` | `{}` | +| `controlPlane.konnectivity.server.resources.cpu` | CPU available. | `quantity` | `""` | +| `controlPlane.konnectivity.server.resources.memory` | Memory (RAM) available. | `quantity` | `""` | +| `controlPlane.konnectivity.server.resourcesPreset` | Preset if `resources` omitted. | `string` | `t1.micro` | +| `controlPlane.kubelet` | Kubelet connection settings. | `object` | `{}` | +| `controlPlane.kubelet.preferredAddressTypes` | Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one. | `[]string` | `[]` | ### OIDC diff --git a/packages/apps/kubernetes/values.schema.json b/packages/apps/kubernetes/values.schema.json index 425d0703f..23af2636d 100644 --- a/packages/apps/kubernetes/values.schema.json +++ b/packages/apps/kubernetes/values.schema.json @@ -3,7 +3,7 @@ "type": "object", "x-cozystack-name": { "type": "string", - "description": "Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes-\u003cname\u003e-\u003cpool\u003e`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.", + "description": "Cluster name. Each worker pool is a KubernetesNodes release named `kubernetes-nodes-\u003cname\u003e-\u003cpool\u003e`, which has to fit the 53-character Helm release name limit, so a cluster name of at most 32 characters leaves at least three characters for the pool name.", "maxLength": 32 }, "properties": { @@ -34,12 +34,12 @@ ] }, "host": { - "description": "External hostname for Kubernetes cluster. Defaults to `\u003ccluster-name\u003e.\u003ctenant-host\u003e` if empty.", + "description": "External hostname for Kubernetes cluster. Defaults to `kubernetes-\u003ccluster-name\u003e.\u003ctenant-host\u003e` if empty.", "type": "string", "default": "" }, "substrate": { - "description": "Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.", + "description": "Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one.", "type": "string", "default": "kubevirt", "x-kubernetes-validations": [ @@ -506,7 +506,7 @@ } }, "extraArgs": { - "description": "Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior).", + "description": "Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default.", "type": "array", "default": [], "items": { diff --git a/packages/apps/kubernetes/values.yaml b/packages/apps/kubernetes/values.yaml index d6708e4fc..2ee0d1580 100644 --- a/packages/apps/kubernetes/values.yaml +++ b/packages/apps/kubernetes/values.yaml @@ -1,4 +1,4 @@ -## @name {string} - Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes--`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit. +## @name {string} - Cluster name. Each worker pool is a KubernetesNodes release named `kubernetes-nodes--`, which has to fit the 53-character Helm release name limit, so a cluster name of at most 32 characters leaves at least three characters for the pool name. ## @maxLength 32 ## @@ -79,13 +79,13 @@ storageClass: replicated version: "v1.35" -## @param {string} host - External hostname for Kubernetes cluster. Defaults to `.` if empty. +## @param {string} host - External hostname for Kubernetes cluster. Defaults to `kubernetes-.` if empty. host: "" ## ## @section Substrate -## @param {string} substrate="kubevirt" - Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one. +## @param {string} substrate="kubevirt" - Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one. ## @immutable substrate: kubevirt @@ -288,7 +288,7 @@ addons: ## @typedef {struct} APIServer - API Server configuration. ## @field {Resources} resources - CPU and memory resources for API Server. ## @field {ResourcesPreset} resourcesPreset="c1.medium" - Preset if `resources` omitted. -## @field {[]string} extraArgs - Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior). +## @field {[]string} extraArgs - Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default. ## @field {[]AdmissionController} admissionControllers - Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it. ## @field {[]object} extraVolumes - Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default. ## @field {[]object} extraVolumeMounts - Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default. @@ -366,7 +366,7 @@ controlPlane: ## @field {string} [config] - Inline AuthenticationConfiguration YAML (`apiserver.config.k8s.io/v1beta1`). The chart writes the value verbatim into a Secret mounted at `/etc/kubernetes/authentication-config/config.yaml` on the kube-apiserver. Mutually exclusive with `secretRef.name`. ## @field {OIDCSecretRef} [secretRef] - Reference to an existing Secret in the tenant namespace carrying the `AuthenticationConfiguration`. -## @typedef {struct} OIDC - OIDC selector for the tenant cluster (Phase 1 — per-cluster audience binding; no per-tenant Keycloak realm). +## @typedef {struct} OIDC - OIDC selector for the tenant cluster. There is no per-tenant Keycloak realm: `System` mode uses the platform realm with a public client and an audience of its own per cluster. ## @field {OIDCMode} mode - Identity mode. `None`: no OIDC, only the static admin kubeconfig works. `System`: trust the platform Keycloak realm (`cozy` by default) via a per-cluster public client with audience binding; zero-config default. `CustomConfig`: trust a tenant-supplied issuer directly (BYO); `cozy` is not in the path. ## @field {OIDCCustomConfig} [customConfig] - Tenant-supplied AuthenticationConfiguration; consumed only when `mode: CustomConfig`. ## @field {[]OIDCUser} [users] - Users granted access to the tenant cluster; each entry produces one ClusterRoleBinding inside the tenant cluster. Works for both `System` and `CustomConfig` modes. diff --git a/packages/system/kubernetes-rd/cozyrds/kubernetes.yaml b/packages/system/kubernetes-rd/cozyrds/kubernetes.yaml index be692143d..52644e331 100644 --- a/packages/system/kubernetes-rd/cozyrds/kubernetes.yaml +++ b/packages/system/kubernetes-rd/cozyrds/kubernetes.yaml @@ -32,7 +32,7 @@ spec: singular: kubernetes plural: kuberneteses openAPISchema: |- - {"title":"Chart Values","type":"object","x-cozystack-name":{"type":"string","description":"Cluster name. Every worker pool is a separate KubernetesNodes release named `kubernetes-nodes--`, so the name must leave room for the default `md0` pool inside the 53-character Helm release name limit.","maxLength":32},"properties":{"storageClass":{"description":"Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"version":{"description":"Kubernetes major.minor version to deploy","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"host":{"description":"External hostname for Kubernetes cluster. Defaults to `.` if empty.","type":"string","default":""},"substrate":{"description":"Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Must match the `substrate` of every KubernetesNodes pool attached to this cluster: the pools reference this cluster's infrastructure object by kind, and a mismatch leaves their Machines unreconciled. Switching an existing cluster is not supported; create a new one.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["ipv4Config"],"properties":{"allowedNodes":{"description":"Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster.","type":"array","default":[],"items":{"type":"string"}},"ccm":{"description":"Proxmox cloud-controller-manager settings.","type":"object","default":{},"required":["credentialsSecretName"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks.","type":"string","default":""}}},"csi":{"description":"Proxmox CSI driver settings.","type":"object","default":{},"required":["credentialsSecretName","enabled"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.","type":"string","default":""},"enabled":{"description":"Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.","type":"boolean","default":true},"storageClasses":{"description":"StorageClasses to create in the tenant.","type":"array","default":[],"items":{"type":"object","required":["name","storage"],"properties":{"fstype":{"description":"Filesystem the driver formats volumes with: ext4 or xfs.","type":"string"},"name":{"description":"StorageClass name inside the tenant cluster.","type":"string"},"reclaimPolicy":{"description":"Delete or Retain.","type":"string"},"ssd":{"description":"Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.","type":"boolean"},"storage":{"description":"Proxmox storage id the volumes are created on (as it appears in `pvesm status`).","type":"string"}}}}}},"dnsServers":{"description":"Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.","type":"array","default":[],"items":{"type":"string"}},"insecure":{"description":"Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts.","type":"boolean","default":false},"ipv4Config":{"description":"Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional.","type":"object","default":{},"required":["gateway","prefix"],"properties":{"addresses":{"description":"Ranges or CIDRs, e.g. `[\"10.0.0.120-10.0.0.170\"]`. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address.","type":"array","default":[],"items":{"type":"string"}},"gateway":{"description":"Default gateway for the workers.","type":"string","default":""},"prefix":{"description":"Netmask prefix length.","type":"integer","default":24}}}}},"addons":{"description":"Cluster addons configuration.","type":"object","default":{},"required":["certManager","cilium","coredns","gatewayAPI","gpuOperator","hami","ingressNginx","monitoringAgents","ouroboros","velero","verticalPodAutoscaler"],"properties":{"certManager":{"description":"Cert-manager addon.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable cert-manager.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"cilium":{"description":"Cilium CNI plugin. Always installed as the tenant cluster's CNI; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true},"coredns":{"description":"CoreDNS addon. Always installed as the tenant cluster's DNS; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true},"gatewayAPI":{"description":"Gateway API addon.","type":"object","default":{},"required":["enabled"],"properties":{"enabled":{"description":"Enable Gateway API.","type":"boolean","default":false}}},"gpuOperator":{"description":"NVIDIA GPU Operator.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable GPU Operator.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"hami":{"description":"HAMi GPU virtualization middleware.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable HAMi (requires GPU Operator).","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"ingressNginx":{"description":"Ingress-NGINX controller.","type":"object","default":{},"required":["enabled","exposeMethod","valuesOverride"],"properties":{"enabled":{"description":"Enable the controller (requires nodes labeled `ingress-nginx`).","type":"boolean","default":false},"exposeMethod":{"description":"Method to expose the controller. Allowed values: `Proxied`, `LoadBalancer`.","type":"string","default":"Proxied","enum":["Proxied","LoadBalancer"]},"hosts":{"description":"Domains routed to this tenant cluster when `exposeMethod` is `Proxied`.","type":"array","default":[],"items":{"type":"string"}},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"monitoringAgents":{"description":"Monitoring agents.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable monitoring agents.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"ouroboros":{"description":"Hairpin-NAT fix for ingress-nginx with PROXY-protocol.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable ouroboros. Requires addons.ingressNginx.enabled (chart-render fail otherwise). Only useful when PROXY-protocol is wired on the tenant ingress-nginx via valuesOverride.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides. Operator-key wins over cozystack defaults.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"velero":{"description":"Velero backup/restore addon.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable Velero.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"verticalPodAutoscaler":{"description":"Vertical Pod Autoscaler. Installed and removed together with addons.monitoringAgents.enabled; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true}}},"controlPlane":{"description":"Kubernetes control-plane configuration.","type":"object","default":{},"required":["apiServer","controllerManager","konnectivity","replicas","scheduler"],"properties":{"apiServer":{"description":"API Server configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"admissionControllers":{"description":"Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it.","type":"array","default":[],"items":{"type":"string","enum":["AlwaysAdmit","AlwaysDeny","AlwaysPullImages","CertificateApproval","CertificateSigning","CertificateSubjectRestriction","DefaultIngressClass","DefaultStorageClass","DefaultTolerationSeconds","DenyEscalatingExec","DenyExecOnPrivileged","DenyServiceExternalIPs","EventRateLimit","ExtendedResourceToleration","ImagePolicyWebhook","LimitPodHardAntiAffinityTopology","LimitRanger","MutatingAdmissionWebhook","NamespaceAutoProvision","NamespaceExists","NamespaceLifecycle","NodeRestriction","OwnerReferencesPermissionEnforcement","PersistentVolumeClaimResize","PersistentVolumeLabel","PodNodeSelector","PodSecurity","PodSecurityPolicy","PodTolerationRestriction","Priority","ResourceQuota","RuntimeClass","SecurityContextDeny","ServiceAccount","StorageObjectInUseProtection","TaintNodesByCondition","ValidatingAdmissionWebhook"]}},"extraArgs":{"description":"Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default (no change to current behavior).","type":"array","default":[],"items":{"type":"string"}},"extraVolumeMounts":{"description":"Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default.","type":"array","default":[],"items":{"type":"object","x-kubernetes-preserve-unknown-fields":true}},"extraVolumes":{"description":"Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default.","type":"array","default":[],"items":{"type":"object","x-kubernetes-preserve-unknown-fields":true}},"resources":{"description":"CPU and memory resources for API Server.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"c1.medium","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"controllerManager":{"description":"Controller Manager configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Controller Manager.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"konnectivity":{"description":"Konnectivity configuration.","type":"object","default":{},"required":["server"],"properties":{"server":{"description":"Konnectivity Server configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Konnectivity.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}},"kubelet":{"description":"Kubelet connection settings.","type":"object","default":{},"properties":{"preferredAddressTypes":{"description":"Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one.","type":"array","default":[],"items":{"type":"string","enum":["Hostname","InternalIP","ExternalIP","InternalDNS","ExternalDNS"]}}}},"replicas":{"description":"Number of control-plane replicas.","type":"integer","default":2},"scheduler":{"description":"Scheduler configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Scheduler.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}},"oidc":{"description":"OIDC authentication and per-user RBAC for the tenant kube-apiserver. See https://cozystack.io/docs/v1.6/kubernetes/oidc-authentication/ for the operator guide.","type":"object","default":{},"required":["mode"],"properties":{"customConfig":{"description":"Tenant-supplied AuthenticationConfiguration; consumed only when `mode: CustomConfig`.","type":"object","default":{},"properties":{"config":{"description":"Inline AuthenticationConfiguration YAML (`apiserver.config.k8s.io/v1beta1`). The chart writes the value verbatim into a Secret mounted at `/etc/kubernetes/authentication-config/config.yaml` on the kube-apiserver. Mutually exclusive with `secretRef.name`.","type":"string","default":""},"secretRef":{"description":"Reference to an existing Secret in the tenant namespace carrying the `AuthenticationConfiguration`.","type":"object","default":{},"properties":{"name":{"description":"Name of an existing Secret in the tenant (release) namespace whose `config.yaml` key holds an `apiserver.config.k8s.io/v1beta1` `AuthenticationConfiguration`. Mutually exclusive with `customConfig.config`.","type":"string","default":""}}}}},"mode":{"description":"Identity mode. `None`: no OIDC, only the static admin kubeconfig works. `System`: trust the platform Keycloak realm (`cozy` by default) via a per-cluster public client with audience binding; zero-config default. `CustomConfig`: trust a tenant-supplied issuer directly (BYO); `cozy` is not in the path.","type":"string","default":"None","enum":["None","System","CustomConfig"]},"users":{"description":"Users granted access to the tenant cluster; each entry produces one ClusterRoleBinding inside the tenant cluster. Works for both `System` and `CustomConfig` modes.","type":"array","default":[],"items":{"type":"object","required":["email","role"],"properties":{"email":{"description":"Email address matched against the `email` claim from the issuer. Used verbatim as the `User:` subject in the ClusterRoleBinding inside the tenant cluster. The `email` claim is a built-in OIDC scope requested explicitly by the chart-generated `kubectl oidc-login` kubeconfig; every conformant OIDC provider (cozy realm included) emits it.","type":"string"},"role":{"description":"Role to bind: `admin` maps to `ClusterRole/cluster-admin`, `view` maps to `ClusterRole/view`.","type":"string","enum":["admin","view"]}}}}}}}} + {"title":"Chart Values","type":"object","x-cozystack-name":{"type":"string","description":"Cluster name. Each worker pool is a KubernetesNodes release named `kubernetes-nodes--`, which has to fit the 53-character Helm release name limit, so a cluster name of at most 32 characters leaves at least three characters for the pool name.","maxLength":32},"properties":{"storageClass":{"description":"Default StorageClass inside the tenant cluster. Remote-accessible LINSTOR classes are auto-propagated to the tenant under the same name.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"version":{"description":"Kubernetes major.minor version to deploy","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"host":{"description":"External hostname for Kubernetes cluster. Defaults to `kubernetes-.` if empty.","type":"string","default":""},"substrate":{"description":"Which infrastructure provider backs this cluster's worker VMs. `kubevirt` runs them inside this cluster; `proxmox` runs them on an external Proxmox VE cluster through capmox. The control plane is Kamaji either way — this selects the infrastructure half only. Every KubernetesNodes pool of this cluster must set the same `substrate`: a pool set to the other one refuses to render once this cluster exists. Switching an existing cluster is not supported; create a new one.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["ipv4Config"],"properties":{"allowedNodes":{"description":"Proxmox nodes capmox may place VMs on. Empty means every node in the Proxmox cluster.","type":"array","default":[],"items":{"type":"string"}},"ccm":{"description":"Proxmox cloud-controller-manager settings.","type":"object","default":{},"required":["credentialsSecretName"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the controller's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. May name the same Secret as `csi.credentialsSecretName`; it is a separate knob so the two can hold separate tokens, since this one only reads VM inventory while the driver attaches and detaches disks.","type":"string","default":""}}},"csi":{"description":"Proxmox CSI driver settings.","type":"object","default":{},"required":["credentialsSecretName","enabled"],"properties":{"credentialsSecretName":{"description":"Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. The parent chart composes them into the controller's config file here, in the management cluster; nothing reaches the tenant, and no token is written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.","type":"string","default":""},"enabled":{"description":"Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.","type":"boolean","default":true},"storageClasses":{"description":"StorageClasses to create in the tenant.","type":"array","default":[],"items":{"type":"object","required":["name","storage"],"properties":{"fstype":{"description":"Filesystem the driver formats volumes with: ext4 or xfs.","type":"string"},"name":{"description":"StorageClass name inside the tenant cluster.","type":"string"},"reclaimPolicy":{"description":"Delete or Retain.","type":"string"},"ssd":{"description":"Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.","type":"boolean"},"storage":{"description":"Proxmox storage id the volumes are created on (as it appears in `pvesm status`).","type":"string"}}}}}},"dnsServers":{"description":"Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.","type":"array","default":[],"items":{"type":"string"}},"insecure":{"description":"Skip verification of the Proxmox API server certificate in the CCM and the CSI controller, both of which run in the management cluster. Default false: the certificate is verified unless an operator turns that off. A stock Proxmox VE install serves a self-signed one, so set true for it, or install a certificate the management cluster trusts.","type":"boolean","default":false},"ipv4Config":{"description":"Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional.","type":"object","default":{},"required":["gateway","prefix"],"properties":{"addresses":{"description":"Ranges or CIDRs, e.g. `[\"10.0.0.120-10.0.0.170\"]`. capmox turns these into an InClusterIPPool, so they must not overlap any DHCP range on the same L2 or two workers will answer to one address.","type":"array","default":[],"items":{"type":"string"}},"gateway":{"description":"Default gateway for the workers.","type":"string","default":""},"prefix":{"description":"Netmask prefix length.","type":"integer","default":24}}}}},"addons":{"description":"Cluster addons configuration.","type":"object","default":{},"required":["certManager","cilium","coredns","gatewayAPI","gpuOperator","hami","ingressNginx","monitoringAgents","ouroboros","velero","verticalPodAutoscaler"],"properties":{"certManager":{"description":"Cert-manager addon.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable cert-manager.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"cilium":{"description":"Cilium CNI plugin. Always installed as the tenant cluster's CNI; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true},"coredns":{"description":"CoreDNS addon. Always installed as the tenant cluster's DNS; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true},"gatewayAPI":{"description":"Gateway API addon.","type":"object","default":{},"required":["enabled"],"properties":{"enabled":{"description":"Enable Gateway API.","type":"boolean","default":false}}},"gpuOperator":{"description":"NVIDIA GPU Operator.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable GPU Operator.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"hami":{"description":"HAMi GPU virtualization middleware.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable HAMi (requires GPU Operator).","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"ingressNginx":{"description":"Ingress-NGINX controller.","type":"object","default":{},"required":["enabled","exposeMethod","valuesOverride"],"properties":{"enabled":{"description":"Enable the controller (requires nodes labeled `ingress-nginx`).","type":"boolean","default":false},"exposeMethod":{"description":"Method to expose the controller. Allowed values: `Proxied`, `LoadBalancer`.","type":"string","default":"Proxied","enum":["Proxied","LoadBalancer"]},"hosts":{"description":"Domains routed to this tenant cluster when `exposeMethod` is `Proxied`.","type":"array","default":[],"items":{"type":"string"}},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"monitoringAgents":{"description":"Monitoring agents.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable monitoring agents.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"ouroboros":{"description":"Hairpin-NAT fix for ingress-nginx with PROXY-protocol.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable ouroboros. Requires addons.ingressNginx.enabled (chart-render fail otherwise). Only useful when PROXY-protocol is wired on the tenant ingress-nginx via valuesOverride.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides. Operator-key wins over cozystack defaults.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"velero":{"description":"Velero backup/restore addon.","type":"object","default":{},"required":["enabled","valuesOverride"],"properties":{"enabled":{"description":"Enable Velero.","type":"boolean","default":false},"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}}},"verticalPodAutoscaler":{"description":"Vertical Pod Autoscaler. Installed and removed together with addons.monitoringAgents.enabled; this section only overrides its Helm values.","type":"object","default":{},"required":["valuesOverride"],"properties":{"valuesOverride":{"description":"Custom Helm values overrides.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true}},"x-cozystack-no-enable-switch":true}}},"controlPlane":{"description":"Kubernetes control-plane configuration.","type":"object","default":{},"required":["apiServer","controllerManager","konnectivity","replicas","scheduler"],"properties":{"apiServer":{"description":"API Server configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"admissionControllers":{"description":"Admission plugins the tenant kube-apiserver enables, passed through to KamajiControlPlane `spec.admissionControllers`, which the control plane renders as `--enable-admission-plugins`. A non-empty list replaces the control plane's default set instead of adding to it. The kube-apiserver enables these plugins in addition to its own default ones, so leaving a plugin out does not turn it off; to turn one off, name it in `--disable-admission-plugins` in `controlPlane.apiServer.extraArgs` and keep it out of this list, since the apiserver refuses to start with a plugin in both, and the control plane's default set names several. Empty by default, which keeps that default set, taken from the TenantControlPlane CRD: CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. `NodeRestriction` is not in it.","type":"array","default":[],"items":{"type":"string","enum":["AlwaysAdmit","AlwaysDeny","AlwaysPullImages","CertificateApproval","CertificateSigning","CertificateSubjectRestriction","DefaultIngressClass","DefaultStorageClass","DefaultTolerationSeconds","DenyEscalatingExec","DenyExecOnPrivileged","DenyServiceExternalIPs","EventRateLimit","ExtendedResourceToleration","ImagePolicyWebhook","LimitPodHardAntiAffinityTopology","LimitRanger","MutatingAdmissionWebhook","NamespaceAutoProvision","NamespaceExists","NamespaceLifecycle","NodeRestriction","OwnerReferencesPermissionEnforcement","PersistentVolumeClaimResize","PersistentVolumeLabel","PodNodeSelector","PodSecurity","PodSecurityPolicy","PodTolerationRestriction","Priority","ResourceQuota","RuntimeClass","SecurityContextDeny","ServiceAccount","StorageObjectInUseProtection","TaintNodesByCondition","ValidatingAdmissionWebhook"]}},"extraArgs":{"description":"Extra command-line flags appended to the tenant kube-apiserver, passed through to KamajiControlPlane `spec.apiServer.extraArgs`. For OIDC use `spec.oidc.mode` — this passthrough is the escape hatch for unrelated apiserver flags. The tenant control plane sets each of the following flags itself and drops an entry naming one before the apiserver sees it. The chart moves an `--enable-admission-plugins=` entry into `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into `controlPlane.kubelet.preferredAddressTypes` while that field is empty, and refuses any other entry naming one of them: `--advertise-address`, `--client-ca-file`, `--enable-admission-plugins`, `--service-cluster-ip-range`, `--kubelet-client-certificate`, `--kubelet-client-key`, `--kubelet-preferred-address-types`, `--proxy-client-cert-file`, `--proxy-client-key-file`, `--requestheader-allowed-names`, `--requestheader-client-ca-file`, `--secure-port`, `--service-account-key-file`, `--service-account-signing-key-file`, `--tls-cert-file`, `--tls-private-key-file`, `--etcd-servers`, `--etcd-cafile`, `--etcd-certfile`, `--etcd-keyfile`, `--etcd-prefix`, `--etcd-compaction-interval`, `--egress-selector-config-file`. When `spec.oidc.mode` is not `None` the chart also renders OIDC flags here, reads what you have written on them, and refuses some combinations outright; the Aggregated API servers section of the Kubernetes OIDC guide on cozystack.io carries which flags, which combinations, and why. Do NOT add legacy `--oidc-*` flags here when `spec.oidc.mode` is not `None`; the chart injects `--authentication-config` and the apiserver refuses to boot with both. Empty by default.","type":"array","default":[],"items":{"type":"string"}},"extraVolumeMounts":{"description":"Extra volume mounts added to the tenant kube-apiserver container, passed through to KamajiControlPlane `spec.apiServer.extraVolumeMounts`. Each `name` must reference a volume declared in `extraVolumes`; the chart-managed talos secret volumes cannot be mounted. Each item is a core/v1 VolumeMount. Empty by default.","type":"array","default":[],"items":{"type":"object","x-kubernetes-preserve-unknown-fields":true}},"extraVolumes":{"description":"Extra volumes added to the control-plane Deployment, passed through to KamajiControlPlane `spec.deployment.extraVolumes`. Use to mount a ConfigMap or Secret holding an AuthenticationConfiguration file referenced by `extraArgs`. Each item is a core/v1 Volume, but because the control-plane pod runs on the management cluster only `configMap` and `secret` sources are allowed (host-reaching sources like `hostPath`/`csi` and token-projection via `projected` are rejected); each volume must have a unique, non-empty name and exactly one source. The names `talos-ca` and `talos-tls-cert` are reserved by the chart. Empty by default.","type":"array","default":[],"items":{"type":"object","x-kubernetes-preserve-unknown-fields":true}},"resources":{"description":"CPU and memory resources for API Server.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"c1.medium","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"controllerManager":{"description":"Controller Manager configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Controller Manager.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}},"konnectivity":{"description":"Konnectivity configuration.","type":"object","default":{},"required":["server"],"properties":{"server":{"description":"Konnectivity Server configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Konnectivity.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}},"kubelet":{"description":"Kubelet connection settings.","type":"object","default":{},"properties":{"preferredAddressTypes":{"description":"Node address types the tenant kube-apiserver tries, in order, when it connects to a kubelet, passed through to KamajiControlPlane `spec.kubelet.preferredAddressTypes`, which the control plane renders as `--kubelet-preferred-address-types`. The KamajiControlPlane CRD rejects a repeated name. Empty by default, which renders `InternalIP`, `ExternalIP`, or the list of a `--kubelet-preferred-address-types=` entry in `controlPlane.apiServer.extraArgs` when there is one.","type":"array","default":[],"items":{"type":"string","enum":["Hostname","InternalIP","ExternalIP","InternalDNS","ExternalDNS"]}}}},"replicas":{"description":"Number of control-plane replicas.","type":"integer","default":2},"scheduler":{"description":"Scheduler configuration.","type":"object","default":{},"required":["resources","resourcesPreset"],"properties":{"resources":{"description":"CPU and memory resources for Scheduler.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Preset if `resources` omitted.","type":"string","default":"t1.micro","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}},"oidc":{"description":"OIDC authentication and per-user RBAC for the tenant kube-apiserver. See https://cozystack.io/docs/v1.6/kubernetes/oidc-authentication/ for the operator guide.","type":"object","default":{},"required":["mode"],"properties":{"customConfig":{"description":"Tenant-supplied AuthenticationConfiguration; consumed only when `mode: CustomConfig`.","type":"object","default":{},"properties":{"config":{"description":"Inline AuthenticationConfiguration YAML (`apiserver.config.k8s.io/v1beta1`). The chart writes the value verbatim into a Secret mounted at `/etc/kubernetes/authentication-config/config.yaml` on the kube-apiserver. Mutually exclusive with `secretRef.name`.","type":"string","default":""},"secretRef":{"description":"Reference to an existing Secret in the tenant namespace carrying the `AuthenticationConfiguration`.","type":"object","default":{},"properties":{"name":{"description":"Name of an existing Secret in the tenant (release) namespace whose `config.yaml` key holds an `apiserver.config.k8s.io/v1beta1` `AuthenticationConfiguration`. Mutually exclusive with `customConfig.config`.","type":"string","default":""}}}}},"mode":{"description":"Identity mode. `None`: no OIDC, only the static admin kubeconfig works. `System`: trust the platform Keycloak realm (`cozy` by default) via a per-cluster public client with audience binding; zero-config default. `CustomConfig`: trust a tenant-supplied issuer directly (BYO); `cozy` is not in the path.","type":"string","default":"None","enum":["None","System","CustomConfig"]},"users":{"description":"Users granted access to the tenant cluster; each entry produces one ClusterRoleBinding inside the tenant cluster. Works for both `System` and `CustomConfig` modes.","type":"array","default":[],"items":{"type":"object","required":["email","role"],"properties":{"email":{"description":"Email address matched against the `email` claim from the issuer. Used verbatim as the `User:` subject in the ClusterRoleBinding inside the tenant cluster. The `email` claim is a built-in OIDC scope requested explicitly by the chart-generated `kubectl oidc-login` kubeconfig; every conformant OIDC provider (cozy realm included) emits it.","type":"string"},"role":{"description":"Role to bind: `admin` maps to `ClusterRole/cluster-admin`, `view` maps to `ClusterRole/view`.","type":"string","enum":["admin","view"]}}}}}}}} release: prefix: kubernetes- labels: diff --git a/pkg/registry/apps/application/rest_name_schema_test.go b/pkg/registry/apps/application/rest_name_schema_test.go index a4bc67ec2..27f0795fe 100644 --- a/pkg/registry/apps/application/rest_name_schema_test.go +++ b/pkg/registry/apps/application/rest_name_schema_test.go @@ -36,7 +36,7 @@ func TestNameCapComesFromTheShippedSchema(t *testing.T) { cap int explains string }{ - {"Kubernetes", "kubernetes-", "../../../../packages/system/kubernetes-rd/cozyrds/kubernetes.yaml", 32, "md0"}, + {"Kubernetes", "kubernetes-", "../../../../packages/system/kubernetes-rd/cozyrds/kubernetes.yaml", 32, "kubernetes-nodes--"}, {"Kafka", "kafka-", "../../../../packages/system/kafka-rd/cozyrds/kafka.yaml", 44, "KRaft"}, } { t.Run(tc.kind, func(t *testing.T) { From b9222c76e53c44cf19182e9f3efbfbd97f5e51c2 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 17:21:15 +0500 Subject: [PATCH 457/518] refactor(kubernetes): drop unused helpers and correct stale comments kubernetes.fullname, its fullnameOverride read, kubernetes.positiveDuration and the Bitnami resources.preset in _resources.tpl have no caller: the first is a helm-create leftover, the duration check moved with the worker pools to the kubernetes-nodes chart, which carries its own copy because a named template does not cross releases, and resource presets come from cozy-lib. A reader who finds them assumes they are in use. The template and test comments described the pre-split chart: phases of a rollout, worker objects at document indices this chart no longer renders, cross-references to files that live in the other chart, and a claim that the cluster CIDRs are kept in one place while the pool chart and the CoreDNS release carry copies. Only Helm comments and test comments change; the chart renders the same bytes for its test values, and the pinned bootstrap-token Job hash is unchanged. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- .../apps/kubernetes/templates/_helpers.tpl | 41 ---------------- .../apps/kubernetes/templates/_resources.tpl | 49 ------------------- .../apps/kubernetes/templates/cluster.yaml | 23 +++++---- .../templates/helmreleases/ingress-nginx.yaml | 5 +- .../templates/oidc-authn-config.yaml | 4 +- .../kubernetes/templates/oidc-keycloak.yaml | 2 +- .../talos/bootstrap-token-tenant-job.yaml | 19 ++++--- .../tests/control_plane_only_test.yaml | 6 +-- packages/apps/kubernetes/tests/oidc_test.yaml | 7 ++- .../apps/kubernetes/tests/values/common.yaml | 4 +- .../tests/worker_adoption_guard_test.yaml | 2 +- 11 files changed, 36 insertions(+), 126 deletions(-) delete mode 100644 packages/apps/kubernetes/templates/_resources.tpl diff --git a/packages/apps/kubernetes/templates/_helpers.tpl b/packages/apps/kubernetes/templates/_helpers.tpl index 300b12732..e27f5ccdb 100644 --- a/packages/apps/kubernetes/templates/_helpers.tpl +++ b/packages/apps/kubernetes/templates/_helpers.tpl @@ -5,24 +5,6 @@ Expand the name of the chart. {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} {{- end }} -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "kubernetes.fullname" -}} -{{- if .Values.fullnameOverride }} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} -{{- else }} -{{- $name := default .Chart.Name .Values.nameOverride }} -{{- if contains $name .Release.Name }} -{{- .Release.Name | trunc 63 | trimSuffix "-" }} -{{- else }} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} -{{- end }} -{{- end }} -{{- end }} - {{/* Create chart name and version as used by the chart label. */}} @@ -185,29 +167,6 @@ never boots. {{- printf "has(claims.groups) && dyn(claims.groups).exists(g, g in [\"%s-view\", \"%s-use\", \"%s-admin\", \"%s-super-admin\"])" $ns $ns $ns $ns -}} {{- end }} -{{- /* -Validates and returns a duration destined for a consumer that does not reject -a bad value: the cluster-autoscaler parses its annotation with -time.ParseDuration and silently falls back to its built-in default on a value -it cannot parse, and the CAPI webhook does not validate unhealthyConditions -timeouts at all, so a zero or negative one applies and remediates a Machine the -moment its condition flips. The values schema types these fields as strings and -stops there, which lets "30" and "-5m" reach the template. The accepted shape -is narrower than Go's duration grammar on purpose, because that grammar also -admits a value large enough to overflow the parser -- the silent fallback again --- and a fraction small enough to round to zero. The first segment must be -positive; later segments may be zero, so the canonical rendering of a duration -("20m0s", "1h0m0s" -- what metav1.Duration serializes to) round-trips. Every -string this admits parses to a positive duration. -*/ -}} -{{- define "kubernetes.positiveDuration" -}} -{{- $value := toString .value -}} -{{- if not (regexMatch "^[1-9][0-9]{0,4}[smh]([0-9]{1,5}[smh]){0,2}$" $value) -}} -{{- fail (printf "%s must be a whole number of s, m or h (e.g. 30m, 1h30m), got %q" .field $value) -}} -{{- end -}} -{{- $value -}} -{{- end -}} - {{/* The two kube-apiserver flags the tenant control plane owns that have a KamajiControlPlane field it renders them from, keyed to that field. diff --git a/packages/apps/kubernetes/templates/_resources.tpl b/packages/apps/kubernetes/templates/_resources.tpl deleted file mode 100644 index 6539c99a1..000000000 --- a/packages/apps/kubernetes/templates/_resources.tpl +++ /dev/null @@ -1,49 +0,0 @@ -{{/* -Copyright Broadcom, Inc. All Rights Reserved. -SPDX-License-Identifier: APACHE-2.0 -*/}} - -{{/* vim: set filetype=mustache: */}} - -{{/* -Return a resource request/limit object based on a given preset. -These presets are for basic testing and not meant to be used in production -{{ include "resources.preset" (dict "type" "nano") -}} -*/}} -{{- define "resources.preset" -}} -{{- $presets := dict - "nano" (dict - "requests" (dict "cpu" "100m" "memory" "128Mi" "ephemeral-storage" "50Mi") - "limits" (dict "memory" "128Mi" "ephemeral-storage" "2Gi") - ) - "micro" (dict - "requests" (dict "cpu" "250m" "memory" "256Mi" "ephemeral-storage" "50Mi") - "limits" (dict "memory" "256Mi" "ephemeral-storage" "2Gi") - ) - "small" (dict - "requests" (dict "cpu" "500m" "memory" "512Mi" "ephemeral-storage" "50Mi") - "limits" (dict "memory" "512Mi" "ephemeral-storage" "2Gi") - ) - "medium" (dict - "requests" (dict "cpu" "500m" "memory" "1Gi" "ephemeral-storage" "50Mi") - "limits" (dict "memory" "1Gi" "ephemeral-storage" "2Gi") - ) - "large" (dict - "requests" (dict "cpu" "1" "memory" "2Gi" "ephemeral-storage" "50Mi") - "limits" (dict "memory" "2Gi" "ephemeral-storage" "2Gi") - ) - "xlarge" (dict - "requests" (dict "cpu" "2" "memory" "4Gi" "ephemeral-storage" "50Mi") - "limits" (dict "memory" "4Gi" "ephemeral-storage" "2Gi") - ) - "2xlarge" (dict - "requests" (dict "cpu" "4" "memory" "8Gi" "ephemeral-storage" "50Mi") - "limits" (dict "memory" "8Gi" "ephemeral-storage" "2Gi") - ) - }} -{{- if hasKey $presets .type -}} -{{- index $presets .type | toYaml -}} -{{- else -}} -{{- printf "ERROR: Preset key '%s' invalid. Allowed values are %s" .type (join "," (keys $presets)) | fail -}} -{{- end -}} -{{- end -}} diff --git a/packages/apps/kubernetes/templates/cluster.yaml b/packages/apps/kubernetes/templates/cluster.yaml index a5f285846..e7c9118ef 100644 --- a/packages/apps/kubernetes/templates/cluster.yaml +++ b/packages/apps/kubernetes/templates/cluster.yaml @@ -40,20 +40,23 @@ */}} {{- /* Worker node pools (KubevirtMachineTemplate, MachineDeployment, MachineHealthCheck, worker WorkloadMonitor) and the Talos - reconcile Job that produces each pool's TalosConfigTemplate now - live in the separate `kubernetes-nodes` chart (one HelmRelease - per pool). This chart renders the control plane only. The - reconcile Job in the kubernetes-nodes release still re-patches - this KamajiControlPlane's apiserver-Service ClusterIP-SAN and - owns each TalosConfigTemplate via an ownerReference to this KCP, - so garbage collection on control-plane uninstall is unchanged. */}} + reconcile Job that produces each pool's TalosConfigTemplate are + rendered by the `kubernetes-nodes` chart (one HelmRelease per + pool). This chart renders the control plane only. The reconcile + Job in the kubernetes-nodes release re-patches this + KamajiControlPlane's apiserver-Service ClusterIP-SAN and owns + each TalosConfigTemplate via an ownerReference to this KCP, so + uninstalling the control plane garbage-collects them. */}} {{- /* Cluster pod/service CIDRs MUST match between the CAPI Cluster spec.clusterNetwork below and the Talos worker machineconfig's cluster.network — Talos surfaces overlaps against the host pod CIDR as a diagnostic at boot, and divergence between CAPI and Talos would - manifest as silently broken kube-proxy / Cilium routing. Keep them - in one place here. + manifest as silently broken kube-proxy / Cilium routing. The worker + side is a copy of these two literals in + packages/apps/kubernetes-nodes/templates/_talosconfigtemplate.tpl, and + templates/helmreleases/coredns.yaml pins the DNS Service IP inside the + service CIDR, so a change here is a change in all three places. */}} {{- $podCIDR := "10.243.0.0/16" }} {{- $serviceCIDR := "10.95.0.0/16" }} @@ -78,7 +81,7 @@ data: to reconcile - this HelmRelease will pick up the DataStore on its next 5m reconcile loop and provision the cluster. {{- else }} -{{- /* Phase 2 worker-pool adoption guard. Worker pools now live in per-pool +{{- /* Worker-pool adoption guard. Worker pools live in per-pool kubernetes-nodes releases; the platform migration adopts each pre-split pool's live objects onto its child release (re-annotating meta.helm.sh/release-name, pinning helm.sh/resource-policy=keep) as a diff --git a/packages/apps/kubernetes/templates/helmreleases/ingress-nginx.yaml b/packages/apps/kubernetes/templates/helmreleases/ingress-nginx.yaml index 746b80583..52c178d8f 100644 --- a/packages/apps/kubernetes/templates/helmreleases/ingress-nginx.yaml +++ b/packages/apps/kubernetes/templates/helmreleases/ingress-nginx.yaml @@ -23,9 +23,8 @@ ingress-nginx: {{- if and .Values.addons.ingressNginx.enabled .Values._namespace.etcd }} {{- /* The ingress-nginx controller DaemonSet pins itself to nodes labelled node-role.kubernetes.io/ingress-nginx (see cozystack.defaultIngressValues - nodeSelector). Since Phase 2 moved worker pools into the separate - kubernetes-nodes chart, that label is produced by a KubernetesNodes pool - with roles: [ingress-nginx]. If no such pool exists the DaemonSet simply + nodeSelector). That label is produced by a KubernetesNodes pool with + roles: [ingress-nginx]. If no such pool exists the DaemonSet simply schedules nothing (degraded, not failed) — the requirement is documented in NOTES.txt and the release note rather than enforced at render time, so an existing cluster is never flipped from Ready to Failed on upgrade. */}} diff --git a/packages/apps/kubernetes/templates/oidc-authn-config.yaml b/packages/apps/kubernetes/templates/oidc-authn-config.yaml index 5a36be06a..8fe2e93b2 100644 --- a/packages/apps/kubernetes/templates/oidc-authn-config.yaml +++ b/packages/apps/kubernetes/templates/oidc-authn-config.yaml @@ -6,8 +6,8 @@ /etc/kubernetes/authentication-config/config.yaml and consumed via --authentication-config=. Compared to the legacy `--oidc-*` flags, the file form supports multiple issuers (the `jwt` block is a list) - and inline private-CA PEM, so future BYO and Phase-2 issuers extend - the same Secret instead of fighting the chart shape. + and inline private-CA PEM, so further issuers extend the same Secret + instead of fighting the chart shape. System mode: Single JWT entry built from `_cluster.root-host` (the cozy realm diff --git a/packages/apps/kubernetes/templates/oidc-keycloak.yaml b/packages/apps/kubernetes/templates/oidc-keycloak.yaml index a8687faf1..99ddcf695 100644 --- a/packages/apps/kubernetes/templates/oidc-keycloak.yaml +++ b/packages/apps/kubernetes/templates/oidc-keycloak.yaml @@ -1,5 +1,5 @@ {{- /* - Phase 1 — System mode: render one per-cluster Keycloak client and a + System mode: render one per-cluster Keycloak client and a per-cluster audience client scope inside the platform `cozy` realm. The clientId is namespace-prefixed so it is globally unique across the diff --git a/packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml b/packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml index 642ac4691..9e8805003 100644 --- a/packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml +++ b/packages/apps/kubernetes/templates/talos/bootstrap-token-tenant-job.yaml @@ -8,10 +8,9 @@ kube-system, which the management cluster can only reach via the Kamaji-issued admin kubeconfig stored as -admin-kubeconfig. - Why a main-phase resource and not a Helm post-install/post-upgrade hook - (the earlier shape): the hook is unreliable on the very failure it was - meant to cover. This release carries - release.cozystack.io/helm-install-disable-wait: "true" (cozystack-api + Why a main-phase resource and not a Helm post-install/post-upgrade hook: + the hook is unreliable on the very failure it was meant to cover. This + release carries release.cozystack.io/helm-install-disable-wait: "true" (cozystack-api sets it on both Install.DisableWait and Upgrade.DisableWait), so there is NO main-resource wait phase in front of the hooks — a post-install hook is not gated on MachineDeployment Ready and does fire on a fresh @@ -25,9 +24,9 @@ -admin-kubeconfig (materialised by Kamaji after the KamajiControlPlane is applied) is not yet available. - As an ordinary main-phase resource (mirroring talos-reconcile-job.yaml) - the Job is applied on every install and every reconcile regardless of - hook phase. It lands together with the KamajiControlPlane and + As an ordinary main-phase resource (like talos-reconcile-job.yaml in the + kubernetes-nodes chart) the Job is applied on every install and every + reconcile regardless of hook phase. It lands together with the KamajiControlPlane and talos-secrets; a wait-for-kubeconfig init container holds the Pod until Kamaji publishes the admin kubeconfig Secret (the Secret volume is optional so kubelet does not FailedMount in the meantime), then the @@ -40,9 +39,9 @@ upgrade that lands while the Job is live (or inside ttlSecondsAfterFinished) with `spec.template: field is immutable`, and RetryOnFailure would loop on it. The Job therefore carries a - content-hash name suffix over its whole rendered spec (mirrors - talos-reconcile-job.yaml and the KubevirtMachineTemplate idiom in - cluster.yaml): an identical render keeps the name and Helm no-ops it; + content-hash name suffix over its whole rendered spec (the idiom of + talos-reconcile-job.yaml and the KubevirtMachineTemplate name in the + kubernetes-nodes chart): an identical render keeps the name and Helm no-ops it; any change that flows into the spec — notably a new images/kubectl.tag — yields a new name so a fresh Job runs and Helm prunes the old one. The hash also folds in the looked-up bootstrap token — both halves, diff --git a/packages/apps/kubernetes/tests/control_plane_only_test.yaml b/packages/apps/kubernetes/tests/control_plane_only_test.yaml index c72c333a4..e89b6e795 100644 --- a/packages/apps/kubernetes/tests/control_plane_only_test.yaml +++ b/packages/apps/kubernetes/tests/control_plane_only_test.yaml @@ -1,8 +1,8 @@ -suite: control-plane-only cluster.yaml render (worker pools moved to kubernetes-nodes) +suite: control-plane-only cluster.yaml render (worker pools are kubernetes-nodes releases) -# Phase 2 moved the worker node pools (KubevirtMachineTemplate, MachineDeployment, +# The worker node pools (KubevirtMachineTemplate, MachineDeployment, # MachineHealthCheck, worker WorkloadMonitor) and the per-pool talos-reconcile Job -# into the separate kubernetes-nodes chart. cluster.yaml now renders the control +# are rendered by the kubernetes-nodes chart. cluster.yaml renders the control # plane only, in source order: Cluster (0), KamajiControlPlane (1), the # control-plane WorkloadMonitor (2), KubevirtCluster (3). This suite pins that # surface so a regression that re-introduces pool objects into the parent chart diff --git a/packages/apps/kubernetes/tests/oidc_test.yaml b/packages/apps/kubernetes/tests/oidc_test.yaml index 8511e392c..55a34aba9 100644 --- a/packages/apps/kubernetes/tests/oidc_test.yaml +++ b/packages/apps/kubernetes/tests/oidc_test.yaml @@ -1,12 +1,11 @@ -suite: tenant kube-apiserver OIDC (Phase 1) +suite: tenant kube-apiserver OIDC values: - values/common.yaml # In cluster.yaml the KamajiControlPlane is document index 1 -# (0: Cluster, 1: KamajiControlPlane, 2: WorkloadMonitor, 3: KubevirtCluster, ...). -# Documents at higher indices are KubevirtMachineTemplate, MachineDeployment, -# MachineHealthCheck, WorkloadMonitor per node group. +# (0: Cluster, 1: KamajiControlPlane, 2: WorkloadMonitor, 3: KubevirtCluster); +# tests/control_plane_only_test.yaml pins that there are no others. tests: ############################################### diff --git a/packages/apps/kubernetes/tests/values/common.yaml b/packages/apps/kubernetes/tests/values/common.yaml index 6377414e1..919116caf 100644 --- a/packages/apps/kubernetes/tests/values/common.yaml +++ b/packages/apps/kubernetes/tests/values/common.yaml @@ -19,5 +19,5 @@ controlPlane: server: resources: {} resourcesPreset: "micro" -# Worker pools are managed via KubernetesNodes resources since the Phase 2 split; -# the kubernetes chart renders the control plane only. +# Worker pools are KubernetesNodes resources; the kubernetes chart renders the +# control plane only. diff --git a/packages/apps/kubernetes/tests/worker_adoption_guard_test.yaml b/packages/apps/kubernetes/tests/worker_adoption_guard_test.yaml index b95d6f414..e7bb3996a 100644 --- a/packages/apps/kubernetes/tests/worker_adoption_guard_test.yaml +++ b/packages/apps/kubernetes/tests/worker_adoption_guard_test.yaml @@ -1,4 +1,4 @@ -suite: Phase 2 worker-pool adoption guard +suite: worker-pool adoption guard # The guard in templates/cluster.yaml refuses to render the control-plane-only # chart when a live worker MachineDeployment for this cluster is NOT owned by From a97b1ee924ea952bc3d52e7e1e1216bf523bd401 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 17:21:15 +0500 Subject: [PATCH 458/518] docs(agents): keep app READMEs and field descriptions about the present App READMEs and values.yaml descriptions drifted into release logs: a Breaking Changes section that collected entries from several releases, descriptions that explained a limit by a default that no longer exists, a roadmap phase in an API type title, and fields documented by what they replaced. The text reaches users through the dashboard, the API schema and the website, where history only misleads, and nothing told an author where the history goes instead. The rule names the place for a change users must act on, the website's per-version upgrade page announced by a "!" title, and asks a change that touches a README or a description to fix what is wrong there. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- docs/agents/overview.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/agents/overview.md b/docs/agents/overview.md index 9ab67c41d..4ebf430c8 100644 --- a/docs/agents/overview.md +++ b/docs/agents/overview.md @@ -158,6 +158,8 @@ When a document could go in either `docs/` or the website, the test is whether a Documenting a feature's current limitations is useful and belongs with the feature. Documenting the *development process* around it does not: no "What is NOT in Phase N", no "out of scope for this PR", no "this PR" phrasing, no action plans or review negotiation. A merged document describes what the code does now, and states a limitation as a limitation rather than as a roadmap position. Scope, phasing and follow-ups belong in the pull request description or an issue. +An app's README and the field descriptions in its `values.yaml` describe what the app does now, for the person using it: no history ("moves from", "previously", "used to", rollout phase names) and no account of how another chart behaved before. A change users must act on is announced by a `!` in the pull request title and described on the website's per-version upgrade page (`content/en/docs/next/operations/cluster/upgrade-to-vX.Y.md`), so an app README carries no history or breaking-changes section. A description says what the field does, what it accepts and what it refuses, not what it replaced, and does not tell the reader to keep it in sync with another chart; when two charts have to agree, a test says so. A change that touches a README or a description fixes what is wrong in that place while it is there. + A document that names an unmerged pull request or an unreleased dependency needs a tracking issue, because nothing re-checks that claim later and it silently becomes false once the thing ships. ### Domain references for specific tasks From 8a45b3dd94d7a250228cc7d5b4b64bd27540c23f Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 16:57:31 +0500 Subject: [PATCH 459/518] docs(kubernetes-nodes): describe the worker pool as it works today The pool's field descriptions, README and NOTES still read as if the pool lived inside the kubernetes chart: five fields told the user to keep them in sync with parent fields that do not exist, NOTES warned about colliding with a parent-managed md0 pool although the parent renders no pools and md0 is the normal name of a native pool, and the installer image was attributed to an upgrade Job that never runs one. A user of the API, the dashboard form or the published README reads these descriptions to decide what a field does, so they now say what the chart renders, which fields have no effect on a Proxmox pool, and how the reservations and the support matrix actually behave. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- api/apps/v1alpha1/kubernetesnodes/types.go | 46 +++--- packages/apps/kubernetes-nodes/README.md | 131 +++++++++--------- .../apps/kubernetes-nodes/templates/NOTES.txt | 15 +- .../apps/kubernetes-nodes/values.schema.json | 46 +++--- packages/apps/kubernetes-nodes/values.yaml | 50 +++---- .../cozyrds/kubernetes-nodes.yaml | 2 +- 6 files changed, 145 insertions(+), 145 deletions(-) diff --git a/api/apps/v1alpha1/kubernetesnodes/types.go b/api/apps/v1alpha1/kubernetesnodes/types.go index 59bbe59c6..a32011516 100644 --- a/api/apps/v1alpha1/kubernetesnodes/types.go +++ b/api/apps/v1alpha1/kubernetesnodes/types.go @@ -18,11 +18,11 @@ type Config struct { } type ConfigSpec struct { - // Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool name (the part after `-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict. + // Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool's objects are named `kubernetes--`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over. // +kubebuilder:default:="" // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="cluster is immutable" Cluster string `json:"cluster"` - // StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. + // StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`. // +kubebuilder:default:="replicated" // +kubebuilder:validation:XValidation:rule="self == oldSelf",message="storageClass is immutable" StorageClass string `json:"storageClass"` @@ -32,10 +32,10 @@ type ConfigSpec struct { // Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling. // +kubebuilder:default:=10 MaxReplicas int `json:"maxReplicas"` - // Virtual machine instance type. + // KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them. // +kubebuilder:default:="u1.medium" InstanceType string `json:"instanceType"` - // System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. + // System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size. // +kubebuilder:default:="20Gi" DiskSize resource.Quantity `json:"diskSize"` // List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller. @@ -44,16 +44,16 @@ type ConfigSpec struct { // Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `"010"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `"010"` reserves what `"10"` does. // +kubebuilder:default:={} Resources Resources `json:"resources,omitempty"` - // CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. + // CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. PodCpuLimit resource.Quantity `json:"podCpuLimit,omitempty"` - // CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. + // CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. PodCpuRequest resource.Quantity `json:"podCpuRequest,omitempty"` // List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). // +kubebuilder:default:={} Gpus []GPU `json:"gpus,omitempty"` - // Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. + // Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. KernelModules []KernelModule `json:"kernelModules,omitempty"` - // Kubelet resource reservations for this pool. + // Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`. // +kubebuilder:default:={} Kubelet Kubelet `json:"kubelet,omitempty"` // Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. @@ -62,25 +62,25 @@ type ConfigSpec struct { // Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At "50%" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two. // +kubebuilder:default:="50%" MaxUnhealthy string `json:"maxUnhealthy"` - // Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning. + // Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (the Talos image import from the Image Factory, or a storage class slow to provision or clone the worker disk). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning. // +kubebuilder:default:="20m" NodeStartupTimeout string `json:"nodeStartupTimeout"` - // How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart. + // How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. // +kubebuilder:default:="5m" ReadyUnknownTimeout string `json:"readyUnknownTimeout"` - // How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart. + // How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. // +kubebuilder:default:="10m" ReadyFalseTimeout string `json:"readyFalseTimeout"` - // How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. Keep in sync with the parent kubernetes chart. + // How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. // +kubebuilder:default:="30m" MaxNodeProvisionTime string `json:"maxNodeProvisionTime"` - // Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`. + // Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool. The render fails when it is ahead. Its pairing with the Talos release the pool resolves (`talos.version`, or the `osImage` override) is checked against the chart's Talos<->Kubernetes support matrix: for a Talos minor the matrix lists, a Kubernetes minor outside that minor's list is refused; a Talos minor the matrix does not list is not checked. // +kubebuilder:default:="v1.35" Version Version `json:"version"` // Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool. // +kubebuilder:default:={} OsImage WorkerImage `json:"osImage,omitempty"` - // Talos worker image configuration. Keep in sync with the parent cluster's `talos`. + // Talos worker image configuration for this pool. // +kubebuilder:default:={} Talos Talos `json:"talos"` // Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment's template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down. @@ -111,13 +111,13 @@ type Kubelet struct { // Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). // +kubebuilder:default:="10%" EvictionSoftMemory string `json:"evictionSoftMemory,omitempty"` - // CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. + // CPU reserved for kubelet and container runtime. When empty, 5% of the worker's CPU, clamped to 50m..500m. KubeReservedCpu string `json:"kubeReservedCpu,omitempty"` - // Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. + // Memory reserved for kubelet and container runtime. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi. KubeReservedMemory string `json:"kubeReservedMemory,omitempty"` - // CPU reserved for host OS. Auto-computed from instanceType if empty. + // CPU reserved for host OS. When empty, 5% of the worker's CPU, clamped to 50m..500m. SystemReservedCpu string `json:"systemReservedCpu,omitempty"` - // Memory reserved for host OS. Auto-computed from instanceType if empty. + // Memory reserved for host OS. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi. SystemReservedMemory string `json:"systemReservedMemory,omitempty"` } @@ -160,13 +160,13 @@ type Resources struct { } type Talos struct { - // Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. + // Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. On `substrate: proxmox` it has no effect on the workers either: the ConfigMap is still rendered, but nothing references it. // +kubebuilder:default:="" ImageFactoryCA string `json:"imageFactoryCA,omitempty"` - // Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. + // Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. On `substrate: proxmox` the workers clone a VM template that already carries the OS, so this field has no effect there beyond that format check. // +kubebuilder:default:="https://factory.talos.dev" ImageFactoryURL string `json:"imageFactoryURL"` - // OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. + // OCI repository prefix for the Talos installer image, written into the worker machine config as `machine.install.image`. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. // +kubebuilder:default:="factory.talos.dev/installer" InstallerRepository string `json:"installerRepository"` // Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. @@ -175,7 +175,7 @@ type Talos struct { // Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID. // +kubebuilder:default:="ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" SchematicID string `json:"schematicID"` - // Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead. + // Default Talos release for this pool's worker OS image and for the installer image written into the worker machine config. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix lists, for each Talos minor the chart knows, the Kubernetes minors it supports and refuses a pool `version` outside that list; a Talos minor it does not list passes unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead. // +kubebuilder:default:="v1.13.6" Version string `json:"version"` } @@ -195,7 +195,7 @@ type WorkerImageBuiltin struct { } type WorkerImageFactory struct { - // Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry. + // Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. This redirects only the worker boot-disk import; the installer image in the worker machine config is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry. ImageFactoryURL string `json:"imageFactoryURL,omitempty"` // Talos schematic ID. Defaults to `talos.schematicID`. SchematicID string `json:"schematicID,omitempty"` diff --git a/packages/apps/kubernetes-nodes/README.md b/packages/apps/kubernetes-nodes/README.md index 900a19a7a..ed200b41f 100644 --- a/packages/apps/kubernetes-nodes/README.md +++ b/packages/apps/kubernetes-nodes/README.md @@ -1,92 +1,93 @@ # Kubernetes Worker Node Pool -`kubernetes-nodes` manages a single worker node pool for a managed Kubernetes -cluster. It is the worker half of the Kubernetes-app split (see -[cozystack/community#8](https://github.com/cozystack/community/pull/8)): the -control plane lives in the `kubernetes` chart, and each pool is a separate -`kubernetes-nodes` release attached to a parent cluster by name. +`kubernetes-nodes` manages one worker node pool of a managed Kubernetes cluster. The control plane and the cluster addons come from the `kubernetes` application; every pool is a separate `KubernetesNodes` resource that attaches to its cluster by name. -A `KubernetesNodes` CR named `-` (e.g. `myk8s-md0`) provisions a -`MachineDeployment` + `KubevirtMachineTemplate` + `MachineHealthCheck` wired to -the parent cluster `kubernetes-` in the same namespace, plus a -`talos-reconcile` Job that produces the pool's `TalosConfigTemplate`. +A `KubernetesNodes` resource named `-` (for example `myk8s-md0`) sets `cluster: ` and renders, in the namespace of the `kubernetes-` cluster: + +- a `MachineDeployment` that the cluster autoscaler scales between `minReplicas` and `maxReplicas`; +- the machine template its workers are created from: a `KubevirtMachineTemplate` on `substrate: kubevirt` (the default), or a `ProxmoxMachineTemplate` on `substrate: proxmox`; +- a `MachineHealthCheck` that replaces workers which do not become Ready or stop being Ready; +- a `WorkloadMonitor` for the pool's workers; +- a `talos-reconcile` Job that creates the pool's `TalosConfigTemplate`, the Talos machine config every worker boots with. + +The pool's Kubernetes minor (`version`) may lag the cluster's during an upgrade but may not lead it: upgrade the `Kubernetes` resource first, then each pool. Delete the pools of a cluster before the cluster itself. ## Parameters ### Linkage -| Name | Description | Type | Value | -| --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ----- | -| `cluster` | Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool name (the part after `-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict. | `string` | `""` | +| Name | Description | Type | Value | +| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- | ----- | +| `cluster` | Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool's objects are named `kubernetes--`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over. | `string` | `""` | ### Common Parameters -| Name | Description | Type | Value | -| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ | -| `storageClass` | StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. | `string` | `replicated` | +| Name | Description | Type | Value | +| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ | +| `storageClass` | StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`. | `string` | `replicated` | ### Pool Sizing -| Name | Description | Type | Value | -| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- | -| `minReplicas` | Minimum number of replicas in this pool. Used as the cluster-autoscaler floor. | `int` | `0` | -| `maxReplicas` | Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling. | `int` | `10` | -| `instanceType` | Virtual machine instance type. | `string` | `u1.medium` | -| `diskSize` | System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. | `quantity` | `20Gi` | -| `roles` | List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller. | `[]string` | `[]` | -| `resources` | Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `"010"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `"010"` reserves what `"10"` does. | `object` | `{}` | -| `resources.cpu` | Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores. | `quantity` | `""` | -| `resources.memory` | Memory (RAM) available. | `quantity` | `""` | -| `podCpuLimit` | CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. | `quantity` | `""` | -| `podCpuRequest` | CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. | `quantity` | `""` | -| `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` | -| `gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` | -| `kernelModules` | Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. | `[]object` | `[]` | -| `kernelModules[i].name` | Module name as `modprobe` takes it, e.g. `nvidia_uvm`. | `string` | `""` | -| `kernelModules[i].parameters` | Module parameters, each as a bare `key=value` string. | `[]string` | `[]` | -| `kubelet` | Kubelet resource reservations for this pool. | `object` | `{}` | -| `kubelet.systemReservedMemory` | Memory reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` | -| `kubelet.kubeReservedMemory` | Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` | -| `kubelet.systemReservedCpu` | CPU reserved for host OS. Auto-computed from instanceType if empty. | `string` | `""` | -| `kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. | `string` | `""` | -| `kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` | -| `kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` | -| `logSerialConsole` | Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. | `bool` | `false` | -| `maxUnhealthy` | Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At "50%" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two. | `string` | `50%` | -| `nodeStartupTimeout` | Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning. | `string` | `20m` | -| `readyUnknownTimeout` | How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart. | `string` | `5m` | -| `readyFalseTimeout` | How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart. | `string` | `10m` | -| `maxNodeProvisionTime` | How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. Keep in sync with the parent kubernetes chart. | `string` | `30m` | +| Name | Description | Type | Value | +| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ----------- | +| `minReplicas` | Minimum number of replicas in this pool. Used as the cluster-autoscaler floor. | `int` | `0` | +| `maxReplicas` | Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling. | `int` | `10` | +| `instanceType` | KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them. | `string` | `u1.medium` | +| `diskSize` | System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size. | `quantity` | `20Gi` | +| `roles` | List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller. | `[]string` | `[]` | +| `resources` | Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `"010"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `"010"` reserves what `"10"` does. | `object` | `{}` | +| `resources.cpu` | Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores. | `quantity` | `""` | +| `resources.memory` | Memory (RAM) available. | `quantity` | `""` | +| `podCpuLimit` | CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. | `quantity` | `""` | +| `podCpuRequest` | CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. | `quantity` | `""` | +| `gpus` | List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). | `[]object` | `[]` | +| `gpus[i].name` | Name of GPU, such as "nvidia.com/AD102GL_L40S". | `string` | `""` | +| `kernelModules` | Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. | `[]object` | `[]` | +| `kernelModules[i].name` | Module name as `modprobe` takes it, e.g. `nvidia_uvm`. | `string` | `""` | +| `kernelModules[i].parameters` | Module parameters, each as a bare `key=value` string. | `[]string` | `[]` | +| `kubelet` | Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`. | `object` | `{}` | +| `kubelet.systemReservedMemory` | Memory reserved for host OS. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi. | `string` | `""` | +| `kubelet.kubeReservedMemory` | Memory reserved for kubelet and container runtime. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi. | `string` | `""` | +| `kubelet.systemReservedCpu` | CPU reserved for host OS. When empty, 5% of the worker's CPU, clamped to 50m..500m. | `string` | `""` | +| `kubelet.kubeReservedCpu` | CPU reserved for kubelet and container runtime. When empty, 5% of the worker's CPU, clamped to 50m..500m. | `string` | `""` | +| `kubelet.evictionHardMemory` | Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). | `string` | `7%` | +| `kubelet.evictionSoftMemory` | Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). | `string` | `10%` | +| `logSerialConsole` | Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. | `bool` | `false` | +| `maxUnhealthy` | Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At "50%" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two. | `string` | `50%` | +| `nodeStartupTimeout` | Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (the Talos image import from the Image Factory, or a storage class slow to provision or clone the worker disk). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning. | `string` | `20m` | +| `readyUnknownTimeout` | How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. | `string` | `5m` | +| `readyFalseTimeout` | How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. | `string` | `10m` | +| `maxNodeProvisionTime` | How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. | `string` | `30m` | ### Kubernetes Version -| Name | Description | Type | Value | -| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | -| `version` | Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`. | `string` | `v1.35` | +| Name | Description | Type | Value | +| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------- | +| `version` | Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool. The render fails when it is ahead. Its pairing with the Talos release the pool resolves (`talos.version`, or the `osImage` override) is checked against the chart's Talos<->Kubernetes support matrix: for a Talos minor the matrix lists, a Kubernetes minor outside that minor's list is refused; a Talos minor the matrix does not list is not checked. | `string` | `v1.35` | ### Talos Worker Image -| Name | Description | Type | Value | -| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------------------------------------------------ | -| `osImage` | Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool. | `object` | `{}` | -| `osImage.builtin` | Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled. | `*object` | `null` | -| `osImage.builtin.schematicID` | Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`. | `string` | `""` | -| `osImage.builtin.version` | Talos version of the golden to clone. Defaults to `talos.version`. | `string` | `""` | -| `osImage.factory` | Import from an Image Factory / mirror (default). | `*object` | `null` | -| `osImage.factory.version` | Talos version. Defaults to `talos.version`. | `string` | `""` | -| `osImage.factory.schematicID` | Talos schematic ID. Defaults to `talos.schematicID`. | `string` | `""` | -| `osImage.factory.imageFactoryURL` | Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry. | `string` | `""` | -| `talos` | Talos worker image configuration. Keep in sync with the parent cluster's `talos`. | `object` | `{}` | -| `talos.version` | Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead. | `string` | `v1.13.6` | -| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` | -| `talos.imageFactoryURL` | Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. | `string` | `https://factory.talos.dev` | -| `talos.installerRepository` | OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` | -| `talos.registryMirrors` | Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. | `object` | `{}` | -| `talos.imageFactoryCA` | Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. | `string` | `""` | +| Name | Description | Type | Value | +| --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------------------------------------------------ | +| `osImage` | Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool. | `object` | `{}` | +| `osImage.builtin` | Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled. | `*object` | `null` | +| `osImage.builtin.schematicID` | Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`. | `string` | `""` | +| `osImage.builtin.version` | Talos version of the golden to clone. Defaults to `talos.version`. | `string` | `""` | +| `osImage.factory` | Import from an Image Factory / mirror (default). | `*object` | `null` | +| `osImage.factory.version` | Talos version. Defaults to `talos.version`. | `string` | `""` | +| `osImage.factory.schematicID` | Talos schematic ID. Defaults to `talos.schematicID`. | `string` | `""` | +| `osImage.factory.imageFactoryURL` | Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. This redirects only the worker boot-disk import; the installer image in the worker machine config is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry. | `string` | `""` | +| `talos` | Talos worker image configuration for this pool. | `object` | `{}` | +| `talos.version` | Default Talos release for this pool's worker OS image and for the installer image written into the worker machine config. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix lists, for each Talos minor the chart knows, the Kubernetes minors it supports and refuses a pool `version` outside that list; a Talos minor it does not list passes unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead. | `string` | `v1.13.6` | +| `talos.schematicID` | Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID. | `string` | `ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515` | +| `talos.imageFactoryURL` | Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. On `substrate: proxmox` the workers clone a VM template that already carries the OS, so this field has no effect there beyond that format check. | `string` | `https://factory.talos.dev` | +| `talos.installerRepository` | OCI repository prefix for the Talos installer image, written into the worker machine config as `machine.install.image`. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. | `string` | `factory.talos.dev/installer` | +| `talos.registryMirrors` | Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. | `object` | `{}` | +| `talos.imageFactoryCA` | Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. On `substrate: proxmox` it has no effect on the workers either: the ConfigMap is still rendered, but nothing references it. | `string` | `""` | ### Substrate diff --git a/packages/apps/kubernetes-nodes/templates/NOTES.txt b/packages/apps/kubernetes-nodes/templates/NOTES.txt index 197b45006..b82912e60 100644 --- a/packages/apps/kubernetes-nodes/templates/NOTES.txt +++ b/packages/apps/kubernetes-nodes/templates/NOTES.txt @@ -7,12 +7,11 @@ This pool is a separate release from the control plane. Three operational notes: cluster that is being torn down. Delete the KubernetesNodes pools first, then the parent Kubernetes cluster. -2. Version skew. Keep this pool's `version` and `talos.*` in sync with the parent - Kubernetes cluster. A pool on a Kubernetes/Talos combination the parent does - not run can produce workers the control plane will not admit. +2. Version skew. This pool's `version` may lag the parent cluster's Kubernetes + minor during an upgrade but may not lead it: a worker kubelet ahead of the + apiserver is refused at render time. Upgrade the parent Kubernetes CR first, + then each pool. -3. Pool name. The pool name (the part after `-` in the KubernetesNodes - name) must not collide with a nodeGroup still managed by the parent kubernetes - chart — most importantly the implicit default `md0`. A collision makes this - release fail on a Helm ownership conflict against the parent-owned objects. - The chart guards this at render time with a clear error. +3. Pool name. The pool's objects are named `kubernetes--`. When a + MachineDeployment of that name already exists and belongs to another Helm + release, this release fails at render time rather than taking it over. diff --git a/packages/apps/kubernetes-nodes/values.schema.json b/packages/apps/kubernetes-nodes/values.schema.json index 0e99096d5..ce9a5286d 100644 --- a/packages/apps/kubernetes-nodes/values.schema.json +++ b/packages/apps/kubernetes-nodes/values.schema.json @@ -3,7 +3,7 @@ "type": "object", "properties": { "cluster": { - "description": "Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-\u003ccluster\u003e`; the KubernetesNodes CR must be named `\u003ccluster\u003e-\u003cpool\u003e`. The pool name (the part after `\u003ccluster\u003e-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.", + "description": "Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-\u003ccluster\u003e`; the KubernetesNodes CR must be named `\u003ccluster\u003e-\u003cpool\u003e`. The pool's objects are named `kubernetes-\u003ccluster\u003e-\u003cpool\u003e`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over.", "type": "string", "default": "", "x-kubernetes-validations": [ @@ -14,7 +14,7 @@ ] }, "storageClass": { - "description": "StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.", + "description": "StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.", "type": "string", "default": "replicated", "x-kubernetes-validations": [ @@ -38,7 +38,7 @@ "default": 10 }, "instanceType": { - "description": "Virtual machine instance type.", + "description": "KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them.", "type": "string", "default": "u1.medium", "x-cozystack-options": { @@ -46,7 +46,7 @@ } }, "diskSize": { - "description": "System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.", + "description": "System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size.", "default": "20Gi", "pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$", "anyOf": [ @@ -101,7 +101,7 @@ } }, "podCpuLimit": { - "description": "CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.", + "description": "CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content.", "pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$", "anyOf": [ { @@ -114,7 +114,7 @@ "x-kubernetes-int-or-string": true }, "podCpuRequest": { - "description": "CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.", + "description": "CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content.", "pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$", "anyOf": [ { @@ -147,7 +147,7 @@ } }, "kernelModules": { - "description": "Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.", + "description": "Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.", "type": "array", "items": { "type": "object", @@ -170,7 +170,7 @@ } }, "kubelet": { - "description": "Kubelet resource reservations for this pool.", + "description": "Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`.", "type": "object", "default": {}, "properties": { @@ -185,19 +185,19 @@ "default": "10%" }, "kubeReservedCpu": { - "description": "CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.", + "description": "CPU reserved for kubelet and container runtime. When empty, 5% of the worker's CPU, clamped to 50m..500m.", "type": "string" }, "kubeReservedMemory": { - "description": "Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.", + "description": "Memory reserved for kubelet and container runtime. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi.", "type": "string" }, "systemReservedCpu": { - "description": "CPU reserved for host OS. Auto-computed from instanceType if empty.", + "description": "CPU reserved for host OS. When empty, 5% of the worker's CPU, clamped to 50m..500m.", "type": "string" }, "systemReservedMemory": { - "description": "Memory reserved for host OS. Auto-computed from instanceType if empty.", + "description": "Memory reserved for host OS. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi.", "type": "string" } } @@ -213,27 +213,27 @@ "default": "50%" }, "nodeStartupTimeout": { - "description": "Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.", + "description": "Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (the Talos image import from the Image Factory, or a storage class slow to provision or clone the worker disk). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.", "type": "string", "default": "20m" }, "readyUnknownTimeout": { - "description": "How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.", + "description": "How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips.", "type": "string", "default": "5m" }, "readyFalseTimeout": { - "description": "How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.", + "description": "How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips.", "type": "string", "default": "10m" }, "maxNodeProvisionTime": { - "description": "How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted. Keep in sync with the parent kubernetes chart.", + "description": "How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted.", "type": "string", "default": "30m" }, "version": { - "description": "Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos\u003c-\u003eKubernetes support matrix against `talos.version`.", + "description": "Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool. The render fails when it is ahead. Its pairing with the Talos release the pool resolves (`talos.version`, or the `osImage` override) is checked against the chart's Talos\u003c-\u003eKubernetes support matrix: for a Talos minor the matrix lists, a Kubernetes minor outside that minor's list is refused; a Talos minor the matrix does not list is not checked.", "type": "string", "default": "v1.35", "enum": [ @@ -268,7 +268,7 @@ "type": "object", "properties": { "imageFactoryURL": { - "description": "Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.", + "description": "Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. This redirects only the worker boot-disk import; the installer image in the worker machine config is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.", "type": "string" }, "schematicID": { @@ -284,7 +284,7 @@ } }, "talos": { - "description": "Talos worker image configuration. Keep in sync with the parent cluster's `talos`.", + "description": "Talos worker image configuration for this pool.", "type": "object", "default": {}, "required": [ @@ -296,17 +296,17 @@ ], "properties": { "imageFactoryCA": { - "description": "Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.", + "description": "Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. On `substrate: proxmox` it has no effect on the workers either: the ConfigMap is still rendered, but nothing references it.", "type": "string", "default": "" }, "imageFactoryURL": { - "description": "Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.", + "description": "Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. On `substrate: proxmox` the workers clone a VM template that already carries the OS, so this field has no effect there beyond that format check.", "type": "string", "default": "https://factory.talos.dev" }, "installerRepository": { - "description": "OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `\u003cinstallerRepository\u003e/\u003cschematicID\u003e:\u003cversion\u003e` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.", + "description": "OCI repository prefix for the Talos installer image, written into the worker machine config as `machine.install.image`. Resolved as `\u003cinstallerRepository\u003e/\u003cschematicID\u003e:\u003cversion\u003e` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.", "type": "string", "default": "factory.talos.dev/installer" }, @@ -322,7 +322,7 @@ "default": "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" }, "version": { - "description": "Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos\u003c-\u003eKubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.", + "description": "Default Talos release for this pool's worker OS image and for the installer image written into the worker machine config. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos\u003c-\u003eKubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix lists, for each Talos minor the chart knows, the Kubernetes minors it supports and refuses a pool `version` outside that list; a Talos minor it does not list passes unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.", "type": "string", "default": "v1.13.6" } diff --git a/packages/apps/kubernetes-nodes/values.yaml b/packages/apps/kubernetes-nodes/values.yaml index c930b0433..99c77e376 100644 --- a/packages/apps/kubernetes-nodes/values.yaml +++ b/packages/apps/kubernetes-nodes/values.yaml @@ -2,7 +2,7 @@ ## @section Linkage ## -## @param {string} cluster - Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool name (the part after `-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict. +## @param {string} cluster - Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool's objects are named `kubernetes--`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over. ## @immutable cluster: "" @@ -22,7 +22,7 @@ cluster: "" ## @section Common Parameters ## -## @param {string} storageClass - StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. +## @param {string} storageClass - StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`. ## @x-cozystack-options {source: storageclass} ## @immutable storageClass: replicated @@ -44,10 +44,10 @@ storageClass: replicated ## @field {[]string} [parameters] - Module parameters, each as a bare `key=value` string. ## @typedef {struct} Kubelet - Kubelet resource reservations for worker nodes. -## @field {string} [systemReservedMemory] - Memory reserved for host OS. Auto-computed from instanceType if empty. -## @field {string} [kubeReservedMemory] - Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty. -## @field {string} [systemReservedCpu] - CPU reserved for host OS. Auto-computed from instanceType if empty. -## @field {string} [kubeReservedCpu] - CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty. +## @field {string} [systemReservedMemory] - Memory reserved for host OS. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi. +## @field {string} [kubeReservedMemory] - Memory reserved for kubelet and container runtime. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi. +## @field {string} [systemReservedCpu] - CPU reserved for host OS. When empty, 5% of the worker's CPU, clamped to 50m..500m. +## @field {string} [kubeReservedCpu] - CPU reserved for kubelet and container runtime. When empty, 5% of the worker's CPU, clamped to 50m..500m. ## @field {string} [evictionHardMemory]="7%" - Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%). ## @field {string} [evictionSoftMemory]="10%" - Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%). @@ -57,11 +57,11 @@ minReplicas: 0 ## @param {int} maxReplicas=10 - Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling. maxReplicas: 10 -## @param {string} instanceType="u1.medium" - Virtual machine instance type. +## @param {string} instanceType="u1.medium" - KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them. ## @x-cozystack-options {source: instancetype} instanceType: u1.medium -## @param {quantity} diskSize="20Gi" - System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. +## @param {quantity} diskSize="20Gi" - System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size. diskSize: 20Gi ## @param {[]string} roles - List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller. @@ -70,16 +70,16 @@ roles: [] ## @param {Resources} [resources] - Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `"010"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `"010"` reserves what `"10"` does. resources: {} -## @param {quantity} [podCpuLimit] - CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. +## @param {quantity} [podCpuLimit] - CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. -## @param {quantity} [podCpuRequest] - CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart. +## @param {quantity} [podCpuRequest] - CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. ## @param {[]GPU} gpus - List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM). gpus: [] -## @param {[]KernelModule} [kernelModules] - Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls. +## @param {[]KernelModule} [kernelModules] - Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. -## @param {Kubelet} [kubelet] - Kubelet resource reservations for this pool. +## @param {Kubelet} [kubelet] - Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`. kubelet: {} ## @param {bool} logSerialConsole=false - Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it. @@ -88,16 +88,16 @@ logSerialConsole: false ## @param {string} maxUnhealthy="50%" - Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer ("0", "1", ...) or a percentage ("0%", "50%"); bare numeric strings are rejected, so the safer default is a percentage. Drop to "0%" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At "50%" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two. maxUnhealthy: "50%" -## @param {string} nodeStartupTimeout="20m" - Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning. +## @param {string} nodeStartupTimeout="20m" - Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (the Talos image import from the Image Factory, or a storage class slow to provision or clone the worker disk). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning. nodeStartupTimeout: "20m" -## @param {string} readyUnknownTimeout="5m" - How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart. +## @param {string} readyUnknownTimeout="5m" - How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. readyUnknownTimeout: "5m" -## @param {string} readyFalseTimeout="10m" - How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart. +## @param {string} readyFalseTimeout="10m" - How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. readyFalseTimeout: "10m" -## @param {string} maxNodeProvisionTime="30m" - How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. Keep in sync with the parent kubernetes chart. +## @param {string} maxNodeProvisionTime="30m" - How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours ("30m", "1h30m"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to ("20m0s", "1h0m0s") is accepted. maxNodeProvisionTime: "30m" ## @@ -111,23 +111,23 @@ maxNodeProvisionTime: "30m" ## @value v1.32 ## @value v1.31 -## @param {Version} version - Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`. +## @param {Version} version - Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool. The render fails when it is ahead. Its pairing with the Talos release the pool resolves (`talos.version`, or the `osImage` override) is checked against the chart's Talos<->Kubernetes support matrix: for a Talos minor the matrix lists, a Kubernetes minor outside that minor's list is refused; a Talos minor the matrix does not list is not checked. version: "v1.35" ## ## @section Talos Worker Image ## -## @typedef {struct} WorkerImageBuiltin - Boot this pool's workers by CDI clone of a golden image from the opt-in worker image catalog (packages/system/kubernetes-worker-image) in cozy-public. The catalog package must be enabled (bundles.enabledPackages) and hold a matching (schematicID, version) entry, otherwise the render fails. A clone is a storage-layer CSI copy, so N workers no longer each stream the OS image over HTTP. Pair this with a StorageClass whose volumes are reachable from any node -- in practice a replicated/DRBD class. The copy runs between two volumes that a node-pinned class requires to be co-located, and nothing here arranges that: the golden in cozy-public has no consumer, so it binds as soon as it imports, on whichever node CDI's importer landed on, while a worker disk binds where its VM is scheduled. The platform also ships `cloneStrategyOverride: csi-clone`, which rules out the host-assisted `copy` strategy that would cross nodes. What a node-pinned class (`allowRemoteVolumeAccess: false`, WaitForFirstConsumer) actually does here has not been measured, so treat `builtin` on one as unsupported rather than slow. +## @typedef {struct} WorkerImageBuiltin - Boot this pool's workers by CDI clone of a golden image from the opt-in worker image catalog (packages/system/kubernetes-worker-image) in cozy-public. The catalog package must be enabled (bundles.enabledPackages) and hold a matching (schematicID, version) entry, otherwise the render fails. A clone is a storage-layer CSI copy, so the workers do not each stream the OS image over HTTP. Pair this with a StorageClass whose volumes are reachable from any node -- in practice a replicated/DRBD class. The copy runs between two volumes that a node-pinned class requires to be co-located, and nothing here arranges that: the golden in cozy-public has no consumer, so it binds as soon as it imports, on whichever node CDI's importer landed on, while a worker disk binds where its VM is scheduled. The platform also ships `cloneStrategyOverride: csi-clone`, which rules out the host-assisted `copy` strategy that would cross nodes. What a node-pinned class (`allowRemoteVolumeAccess: false`, WaitForFirstConsumer) actually does here has not been measured, so treat `builtin` on one as unsupported rather than slow. ## @field {string} [schematicID] - Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`. ## @field {string} [version] - Talos version of the golden to clone. Defaults to `talos.version`. ## @typedef {struct} WorkerImageFactory - Boot this pool's workers by importing the Talos OS image over HTTP from an Image Factory (public, self-hosted, or caching mirror). This is the bring-your-own path and the default when `osImage` is omitted. ## @field {string} [version] - Talos version. Defaults to `talos.version`. ## @field {string} [schematicID] - Talos schematic ID. Defaults to `talos.schematicID`. -## @field {string} [imageFactoryURL] - Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry. +## @field {string} [imageFactoryURL] - Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. This redirects only the worker boot-disk import; the installer image in the worker machine config is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry. -## @typedef {struct} WorkerImage - Worker OS image source for this pool. Set exactly one of `builtin` or `factory`; when the whole field is omitted the workers import over HTTP using the pool's `talos.*` defaults (equivalent to an empty `factory`). The disk-system source is derived entirely from this field, so a pool with `osImage` omitted renders identically to a chart without this feature — adopting the feature never rolls existing workers. Editing this field on an existing pool changes the rendered source and therefore re-images (rolls) that pool, the same as changing any other image reference. +## @typedef {struct} WorkerImage - Worker OS image source for this pool. Set exactly one of `builtin` or `factory`; when the whole field is omitted the workers import over HTTP using the pool's `talos.*` defaults (equivalent to an empty `factory`). The disk-system source is derived entirely from this field. Editing this field on an existing pool changes the rendered source and therefore re-images (rolls) that pool, the same as changing any other image reference. ## @field {*WorkerImageBuiltin} [builtin] - Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled. ## @field {*WorkerImageFactory} [factory] - Import from an Image Factory / mirror (default). @@ -135,14 +135,14 @@ version: "v1.35" osImage: {} ## @typedef {struct} Talos - Talos worker image configuration. -## @field {string} version - Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead. +## @field {string} version - Default Talos release for this pool's worker OS image and for the installer image written into the worker machine config. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix lists, for each Talos minor the chart knows, the Kubernetes minors it supports and refuses a pool `version` outside that list; a Talos minor it does not list passes unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead. ## @field {string} schematicID - Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID. -## @field {string} imageFactoryURL - Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. -## @field {string} installerRepository - OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. +## @field {string} imageFactoryURL - Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. On `substrate: proxmox` the workers clone a VM template that already carries the OS, so this field has no effect there beyond that format check. +## @field {string} installerRepository - OCI repository prefix for the Talos installer image, written into the worker machine config as `machine.install.image`. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash. ## @field {object} registryMirrors - Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. -## @field {string} [imageFactoryCA] - Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. +## @field {string} [imageFactoryCA] - Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. On `substrate: proxmox` it has no effect on the workers either: the ConfigMap is still rendered, but nothing references it. -## @param {Talos} talos - Talos worker image configuration. Keep in sync with the parent cluster's `talos`. +## @param {Talos} talos - Talos worker image configuration for this pool. talos: version: "v1.13.6" schematicID: "ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515" diff --git a/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml b/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml index dffa00956..c76a7c602 100644 --- a/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml +++ b/packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml @@ -21,7 +21,7 @@ spec: singular: kubernetesnodes plural: kubernetesnodeses openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool name (the part after `-`) must not collide with a nodeGroup still managed by the parent kubernetes chart (e.g. the default `md0`), or the render fails on ownership conflict.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"Virtual machine instance type.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `\"010\"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `\"010\"` reserves what `\"10\"` does.","type":"object","default":{},"properties":{"cpu":{"description":"Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"podCpuLimit":{"description":"CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"podCpuRequest":{"description":"CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content. Keep in sync with the parent kubernetes chart.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"kernelModules":{"description":"Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later. The automatic set is itself such a change, on the upgrade that introduces this field, for every existing pool that holds an `nvidia.com/*` GPU and leaves the field unset: that pool rolls.","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Module name as `modprobe` takes it, e.g. `nvidia_uvm`.","type":"string"},"parameters":{"description":"Module parameters, each as a bare `key=value` string.","type":"array","items":{"type":"string"}}}}},"kubelet":{"description":"Kubelet resource reservations for this pool.","type":"object","default":{},"properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. Auto-computed from instanceType if empty.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. Auto-computed from instanceType if empty.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. Auto-computed from instanceType if empty.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.","type":"boolean","default":false},"maxUnhealthy":{"description":"Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer (\"0\", \"1\", ...) or a percentage (\"0%\", \"50%\"); bare numeric strings are rejected, so the safer default is a percentage. Drop to \"0%\" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At \"50%\" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.","type":"string","default":"50%"},"nodeStartupTimeout":{"description":"Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (Talos image fetch from factory.talos.dev or a busy storage class on the kubevirt-csi PVC populator). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.","type":"string","default":"20m"},"readyUnknownTimeout":{"description":"How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"5m"},"readyFalseTimeout":{"description":"How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips. Keep in sync with the parent kubernetes chart.","type":"string","default":"10m"},"maxNodeProvisionTime":{"description":"How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted. Keep in sync with the parent kubernetes chart.","type":"string","default":"30m"},"version":{"description":"Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool -- and must satisfy the Talos<->Kubernetes support matrix against `talos.version`.","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"osImage":{"description":"Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool.","type":"object","default":{},"properties":{"builtin":{"description":"Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled.","type":"object","properties":{"schematicID":{"description":"Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version of the golden to clone. Defaults to `talos.version`.","type":"string"}}},"factory":{"description":"Import from an Image Factory / mirror (default).","type":"object","properties":{"imageFactoryURL":{"description":"Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. NOTE: this redirects only the worker boot-disk import; the in-guest upgrade installer image is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.","type":"string"},"schematicID":{"description":"Talos schematic ID. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version. Defaults to `talos.version`.","type":"string"}}}}},"talos":{"description":"Talos worker image configuration. Keep in sync with the parent cluster's `talos`.","type":"object","default":{},"required":["imageFactoryURL","installerRepository","registryMirrors","schematicID","version"],"properties":{"imageFactoryCA":{"description":"Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there.","type":"string","default":""},"imageFactoryURL":{"description":"Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal.","type":"string","default":"https://factory.talos.dev"},"installerRepository":{"description":"OCI repository prefix for the Talos installer image used by the in-guest `talos-reconcile` upgrade Job. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.","type":"string","default":"factory.talos.dev/installer"},"registryMirrors":{"description":"Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true},"schematicID":{"description":"Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID.","type":"string","default":"ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"},"version":{"description":"Default Talos release for this pool's worker OS image and in-guest installer. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix rejects a pairing it lists as unsupported; it lists the Talos minors this chart ships and passes any other minor unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.","type":"string","default":"v1.13.6"}}},"substrate":{"description":"Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment's template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["network"],"properties":{"dnsServers":{"description":"Nameservers written into the worker machineconfig. On kubevirt the workers use the management cluster's CoreDNS, which they reach over the pod network; an off-cluster Proxmox worker cannot, so it needs reachable resolvers of its own. Required when `substrate` is `proxmox`. Keep in sync with the parent kubernetes chart's `proxmox.dnsServers`.","type":"array","default":[],"items":{"type":"string"}},"full":{"description":"Full clone rather than linked. Default is a linked clone: it costs kilobytes at creation instead of the whole disk, because a ZFS-backed linked clone writes only its own increment (measured: 8K against 10.2G for the same worker full-cloned onto a `sparse 0` pool). The cost is that the clone holds the template's base snapshot, so the template cannot be rotated while any linked clone still references it — which is what the consolidation CronJob resolves 24h after creation. Set true for a pool that must be independent of the template from the first second.","type":"boolean","default":false},"network":{"description":"NIC configuration.","type":"object","default":{},"required":["bridge"],"properties":{"bridge":{"description":"Proxmox bridge the VM NIC attaches to (e.g. `vmbr0`). Required when `substrate` is `proxmox`.","type":"string","default":""},"mtu":{"description":"NIC MTU. Omitted when unset.","type":"integer"},"vlan":{"description":"L2 VLAN tag. Omitted when unset.","type":"integer"}}},"pool":{"description":"Proxmox resource pool the worker VMs are added to. Set the same pool on every node pool of a tenant cluster and grant the tenant's tokens their privileges on `/pool/`: that is the one ACL path that follows every VM capmox creates, replacements included. The pool must exist on the hypervisor; capmox does not create it. Empty adds the workers to no pool.","type":"string","default":""},"storage":{"description":"Proxmox storage for the cloned disk; needs `full: true`. A target storage is a full-clone parameter — a linked clone always lives on the template's storage, and Proxmox refuses the pair — so the render refuses it too. Empty keeps the template's storage.","type":"string","default":""},"templateTags":{"description":"Tags identifying the Talos VM template to clone. capmox matches a template when its tag set is EQUAL to this list, not when it contains it, so every tag on the template must be listed here — a missing one yields `found 0 VM templates with tags ...` while the tags are plainly present. The template has to carry this pool's `talos.schematicID` at `talos.version` in the nocloud flavour, with the Proxmox guest agent enabled for the `qemu-guest-agent` extension of the default schematic; `hack/proxmox-talos-template.sh` builds it from these values and prints the tags to list here. Do not build it from the platform release's `nocloud-amd64.raw.xz`: that is the bare-metal node image, whose `zfs` extension keeps a VM from finishing its boot sequence and reboots it at the Talos boot timeout. Required when `substrate` is `proxmox`.","type":"array","default":[],"items":{"type":"string"}}}}}} + {"title":"Chart Values","type":"object","properties":{"cluster":{"description":"Name of the parent Kubernetes cluster (the `Kubernetes` CR) in the same namespace this node pool attaches to. Required. The pool's CAPI objects are wired to the parent cluster `kubernetes-`; the KubernetesNodes CR must be named `-`. The pool's objects are named `kubernetes--`; when a MachineDeployment of that name already exists and belongs to another Helm release, the render fails instead of taking it over.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"cluster is immutable"}]},"storageClass":{"description":"StorageClass for the worker node system disk. When empty, the cluster default applies. Worker VMs live-migrate, so their disks need ReadWriteMany — the RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — and linstor-csi rejects RWX volumes that are not on a DRBD-backed StorageClass, so prefer a replicated/DRBD class. On an `osImage.builtin` pool this field also decides whether the pool renders at all: CDI takes the storage-layer clone path only when the worker disk and the golden share a StorageClass, and falls back to a host-assisted copy over the pod network across classes, so the chart refuses a pool whose class differs from its golden's. Leaving it empty resolves to the cluster's default StorageClass, and the same comparison is made against that. On a `builtin` pool the resolved class is then written onto the worker disk rather than left for the cluster to fill in at bind time, so a pool that has already cloned keeps binding on the class its disks are on even after an admin moves the default annotation; on the HTTP path an empty class is still left to the cluster. Has no effect on `substrate: proxmox`, where the worker disk is cloned from the VM template onto the template's storage or `proxmox.storage`.","type":"string","default":"replicated","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"minReplicas":{"description":"Minimum number of replicas in this pool. Used as the cluster-autoscaler floor.","type":"integer","default":0},"maxReplicas":{"description":"Maximum number of replicas in this pool. Used as the cluster-autoscaler ceiling.","type":"integer","default":10},"instanceType":{"description":"KubeVirt instance type that sizes each worker VM when `resources` is not set. Ignored when `resources.cpu` and `resources.memory` are both set, and therefore on `substrate: proxmox`, which requires them.","type":"string","default":"u1.medium","x-cozystack-options":{"source":"instancetype"}},"diskSize":{"description":"System disk size for the worker VM. Carries the Talos OS image, kubelet state, containerd image cache, and any local-path PVCs. How the OS image reaches the disk follows this pool's `osImage`: the default and `osImage.factory` stream the Image Factory raw artifact in over HTTP, while `osImage.builtin` makes the disk a storage-layer CDI clone of a shared golden image, with no HTTP fetch. A `builtin` pool must keep diskSize at or above the golden's storage size (6Gi in the shipped worker image catalog), because a CDI clone target cannot be smaller than its source -- the chart rejects a smaller value at render time rather than letting the clone stall. Sizing a `builtin` pool also has to allow for transient headroom the finished disk does not show: CDI populates a larger target by cloning the golden at the source size and then growing it, so a worker being provisioned occupies a temporary clone PVC at the golden's size on top of this diskSize, and both are multiplied by the replica count of a replicated StorageClass. The golden itself is a separate one-time cost for the whole cluster, not a per-worker one. So the peak is the golden, plus `diskSize + golden storage` for each worker cloning at that moment. A pool whose storage backend cannot spare that peak does not fail: the clone waits for space, the VM never boots, and the reason is visible only in the storage layer's own events. Has no effect on `substrate: proxmox`, where the worker disk is a clone of the VM template's disk and keeps its size.","default":"20Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"roles":{"description":"List of node roles. Each role `r` labels nodes with `node-role.kubernetes.io/`. Use `[ingress-nginx]` for a pool that hosts the tenant ingress-nginx controller.","type":"array","default":[],"items":{"type":"string"}},"resources":{"description":"Explicit CPU and memory for each worker node, as an alternative to `instanceType` sizing. Optional: when omitted, the node is sized by `instanceType`. When both `cpu` and `memory` are set, they take precedence and `instanceType` is ignored (the instancetype is omitted from the VM, since KubeVirt cannot override an instancetype's CPU/memory). Set both `cpu` and `memory` together or neither; setting only one is rejected at render time. On the kubevirt substrate `cpu` is written into the VM as spelled and decoded as YAML, so a leading zero is read as octal, as in any Kubernetes manifest: `\"010\"` is eight cores, and the autoscaler capacity counts eight. The kubelet reservations still read the value as a decimal, so `\"010\"` reserves what `\"10\"` does.","type":"object","default":{},"properties":{"cpu":{"description":"Number of vCPU cores for each worker VM, as a whole number such as `2`. On the `kubevirt` substrate the value becomes the VM's integer `domain.cpu.cores`, so millicore, fractional and negative quantities, and counts above 4294967295, are rejected at render time; an unquoted `0` is treated as unset. On the `proxmox` substrate a millicore or fractional quantity is rounded up to whole cores.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"podCpuLimit":{"description":"CPU limit for the worker VM's virt-launcher `compute` container, which is the ceiling the guest vCPU threads, the QEMU emulator thread and the IO threads share. Distinct from `resources.cpu`, which sizes the guest itself. Cannot be combined with `instanceType` sizing: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.limits.cpu`, so the chart rejects that combination at render time. Sizing the pool by `resources` (both `cpu` and `memory`) is what drops the instancetype from the VM, which is where the limit is usable. When unset, KubeVirt derives the limit from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so a worker sized at N vCPUs runs under a ceiling of N with the emulation overhead taken out of the same N. Setting a value declares the limit on the VM directly, which is both what lands it on the container and what stops KubeVirt deriving one. It moves the CPU request with it: KubeVirt copies a declared CPU limit into the CPU request when the VM declares no request of its own, so a pool that sets only this field asks the scheduler for its whole ceiling instead of the fraction of the vCPU count it asked for before, and a ceiling the nodes cannot spare leaves the workers Pending on `Insufficient cpu`. Pair it with `podCpuRequest` to move the ceiling without moving the request. Zero means no ceiling at all rather than the tightest one, since the kubelet turns a zero CPU limit into no CFS quota while the declared key still stops KubeVirt deriving one — but it is only accepted together with `podCpuRequest: 0`, because KubeVirt keeps deriving a request from the vCPU count when the VM declares none and a request above a declared limit of zero is rejected when the worker's Pod is created. A negative value is refused at render time. Changing it rolls the pool, because the worker VM template is named by a hash of its content.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"podCpuRequest":{"description":"CPU request for the worker VM's virt-launcher `compute` container, which is what the scheduler reserves for a worker on its host and what weights it against neighbours on a contended one. Distinct from `resources.cpu`, which sizes the guest itself. It exists to be paired with `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the VM declares no request, so raising the ceiling alone raises what every worker in the pool asks the scheduler for, and this field is what holds the request still while the ceiling moves. Cannot be combined with `instanceType` sizing, for the same reason as `podCpuLimit`: KubeVirt refuses a VM that references an instancetype and also sets `domain.resources.requests.cpu`, so the chart rejects that combination at render time, as it does a value above `podCpuLimit`. Usable without a limit, where it only changes the scheduler's weighting: KubeVirt still derives the ceiling from the guest vCPU count in any namespace carrying a ResourceQuota that limits CPU, so keep the request at or below that count. A request above that derived ceiling is refused only when the worker's Pod is created, and deliberately not at render time: the ceiling exists only where such a ResourceQuota does, so refusing here would take away a configuration that works in a namespace without one. Zero is accepted on its own and reserves nothing, but zero beside a `podCpuLimit` is refused, because KubeVirt cannot tell an explicit zero request from an absent one and replaces both with the limit — which is the outcome this field exists to avoid, arrived at silently. A negative value is refused as well. When unset, KubeVirt computes the request from the guest vCPU count and the cluster CPU allocation ratio. Changing it rolls the pool, because the worker VM template is named by a hash of its content.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"gpus":{"description":"List of GPUs to attach (NVIDIA driver requires at least 4 GiB RAM).","type":"array","default":[],"items":{"type":"object","required":["name"],"properties":{"name":{"description":"Name of GPU, such as \"nvidia.com/AD102GL_L40S\".","type":"string","x-cozystack-options":{"source":"gpu"}}}}},"kernelModules":{"description":"Kernel modules loaded on every worker in this pool, emitted as Talos `machine.kernel.modules`. A Talos system extension installs a module but does not load it, so an extension-provided driver needs its modules declared here. Leave unset to let the chart decide: a pool holding at least one `nvidia.com/*` GPU gets `nvidia` (with `NVreg_NvLinkDisable=1`, which the gpu-operator driver container applies on non-Talos workers; on Talos that container has to be turned off or it clashes with the system extension, and nothing then mounts the ConfigMap carrying the parameter), `nvidia_uvm`, `nvidia_drm`, `nvidia_modeset` (that order — Talos loads the list in sequence and the last three depend on the first), and any other pool gets nothing. Set a non-empty list to replace the chart's choice entirely, or `[]` to opt out and emit no modules even on a GPU pool. The field carries no default, and unset is not the same value as `[]`: a generated parameter table that shows `[]` in its Default column is printing what it prints for any array with no default, not the effective value — leaving the field out selects the automatic set, writing `[]` disables it. The module still has to be in the image, and which extensions the image carries is the pool's schematic: a GPU pool points `osImage.factory.schematicID` (or `osImage.builtin.schematicID`) at a schematic with the NVIDIA extensions, and on Blackwell (GB202) that must be the open-kernel-modules extension. The automatic set keys on the `nvidia.com/` resource prefix alone and nothing cross-checks it against that schematic; on Talos v1.13 a declared module the image does not ship leaves its controller retrying rather than failing the boot, so the symptom is a missing driver, not a dead node. This value is part of the worker machine config, whose content hash names the `TalosConfigTemplate`, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.","type":"array","items":{"type":"object","required":["name"],"properties":{"name":{"description":"Module name as `modprobe` takes it, e.g. `nvidia_uvm`.","type":"string"},"parameters":{"description":"Module parameters, each as a bare `key=value` string.","type":"array","items":{"type":"string"}}}}},"kubelet":{"description":"Kubelet resource reservations for this pool. An empty reservation is computed from the worker's size: `resources` when set, otherwise the `instanceType`.","type":"object","default":{},"properties":{"evictionHardMemory":{"description":"Hard eviction threshold for memory (absolute like 200Mi or percentage like 7%).","type":"string","default":"7%"},"evictionSoftMemory":{"description":"Soft eviction threshold for memory (absolute like 1Gi or percentage like 10%).","type":"string","default":"10%"},"kubeReservedCpu":{"description":"CPU reserved for kubelet and container runtime. When empty, 5% of the worker's CPU, clamped to 50m..500m.","type":"string"},"kubeReservedMemory":{"description":"Memory reserved for kubelet and container runtime. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi.","type":"string"},"systemReservedCpu":{"description":"CPU reserved for host OS. When empty, 5% of the worker's CPU, clamped to 50m..500m.","type":"string"},"systemReservedMemory":{"description":"Memory reserved for host OS. When empty, 5% of the worker's memory, clamped to 256Mi..1Gi.","type":"string"}}},"logSerialConsole":{"description":"Stream each worker VM's guest serial console into a `guest-console-log` container beside virt-launcher, readable with `kubectl logs`. The only view of a worker that stalls during boot before Talos apid answers, when neither a Node nor a certificate request exists to diagnose from. Two costs the field name does not show. Changing the value rolls the pool, because it is part of the worker VM template whose name is a hash of its content -- including the stuck VM whose console was wanted, so it buys visibility into the next boot rather than the current one. And it overrides the platform's cluster-wide `disableSerialConsoleLog`, set because that container has been seen holding virt-launcher in `PodInitializing` (kubevirt/kubevirt#15989), which would take the pool down rather than explain it -- so check that virt-launcher Pods reach Running after enabling it.","type":"boolean","default":false},"maxUnhealthy":{"description":"Maximum number of unhealthy nodes tolerated in this pool before remediation is paused. The MachineHealthCheck admission webhook accepts either a bare integer (\"0\", \"1\", ...) or a percentage (\"0%\", \"50%\"); bare numeric strings are rejected, so the safer default is a percentage. Drop to \"0%\" once the pool is stable. Watch how a percentage behaves in a small group: the MachineHealthCheck controller scales it against the number of machines the check covers and rounds down, then remediates only while the unhealthy count is at or below that result. At \"50%\" a group of one therefore never remediates, because half of one rounds down to zero, and a group of two remediates one unhealthy machine but not two.","type":"string","default":"50%"},"nodeStartupTimeout":{"description":"Maximum time a Machine is allowed to spend reaching the Ready condition before it is remediated. Raise for slow first boots (the Talos image import from the Image Factory, or a storage class slow to provision or clone the worker disk). The MachineHealthCheck measures it from the later of the Machine's creation and its InfrastructureReady transition, so the clock restarts when the infrastructure reports ready: a worker gets this long for the disk image import and the VM to come up, and then this long again for the guest to boot and its kubelet to register with the tenant apiserver. Either leg stretches on a host under load, and a Machine remediated here is deleted before it ever joins, so its replacement starts the same sequence from the beginning.","type":"string","default":"20m"},"readyUnknownTimeout":{"description":"How long a worker Node may report `Ready=Unknown` before the MachineHealthCheck remediates the Machine, which for a KubeVirt worker means deleting the VM. `Ready=Unknown` is what the node lifecycle controller writes once it stops hearing from a kubelet, on a grace period of its own, so the wall clock from a guest going quiet to a deleted worker is this value plus that grace period. A guest starved of CPU stops answering long before it is dead, so a value of a few tens of seconds deletes workers that would have come back on their own. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips.","type":"string","default":"5m"},"readyFalseTimeout":{"description":"How long a worker Node may report `Ready=False` before the MachineHealthCheck remediates the Machine. A Node that has registered but has no working CNI yet reports `Ready=False`, so this value is the whole budget a first boot gets for the CNI rollout to reach that node, and a worker deleted mid-rollout takes the rollout with it while its replacement starts over. Raising it delays replacing a worker that is genuinely dead, but not rescheduling its workload: Pods leave an unreachable Node on the node lifecycle controller's taint-based eviction and the toleration that carries, not on this timer. Validated at render time like maxNodeProvisionTime: the CAPI webhook does not check these timeouts, and a zero or negative value would remediate the moment the condition flips.","type":"string","default":"10m"},"maxNodeProvisionTime":{"description":"How long the cluster-autoscaler waits for a newly created worker to register as a Node before it treats that machine as long-unregistered. Rendered onto this pool's MachineDeployment as the `cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime` annotation, which overrides the autoscaler's built-in default for this pool only. Two things change when the timer fires, and both make a slow join worse rather than better: the machine stops counting towards the pool's upcoming capacity, so the autoscaler asks for a replacement for capacity that is already on its way, and the machine becomes eligible for removal. Size it above the slowest join a healthy worker can take, counting the Talos image import, the guest boot and the CNI rollout. Accepted as a whole number of seconds, minutes or hours (\"30m\", \"1h30m\"), which is narrower than Go's duration syntax on purpose: the autoscaler ignores a value it cannot parse and falls back to its own default without failing, and the narrower shape cannot express a value that overflows the parser or a fraction it rounds to zero. The canonical form a duration serializes back to (\"20m0s\", \"1h0m0s\") is accepted.","type":"string","default":"30m"},"version":{"description":"Kubernetes major.minor version the pool joins. Must not be ahead of the parent cluster's minor (workers may not lead the apiserver); it may lag behind during a rolling upgrade -- bump the parent Kubernetes CR first, then each pool. The render fails when it is ahead. Its pairing with the Talos release the pool resolves (`talos.version`, or the `osImage` override) is checked against the chart's Talos<->Kubernetes support matrix: for a Talos minor the matrix lists, a Kubernetes minor outside that minor's list is refused; a Talos minor the matrix does not list is not checked.","type":"string","default":"v1.35","enum":["v1.35","v1.34","v1.33","v1.32","v1.31"]},"osImage":{"description":"Worker OS image source for this pool. Default (omitted): import over HTTP from the `talos.*` Image Factory. Set `osImage.builtin` to clone a golden from the opt-in worker image catalog instead. Editing this on an existing pool re-images that pool.","type":"object","default":{},"properties":{"builtin":{"description":"Clone a golden image from the worker image catalog. Requires a StorageClass whose volumes are reachable from any node, in practice a replicated/DRBD class: the clone is a storage-layer copy, and on a node-pinned class its two ends land on unrelated nodes -- the golden binds where CDI imported it, a worker disk where its VM is scheduled.","type":"object","properties":{"schematicID":{"description":"Talos schematic ID of the golden to clone. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version of the golden to clone. Defaults to `talos.version`.","type":"string"}}},"factory":{"description":"Import from an Image Factory / mirror (default).","type":"object","properties":{"imageFactoryURL":{"description":"Base URL of the Image Factory serving the boot raw image. Defaults to `talos.imageFactoryURL`. No trailing slash. This redirects only the worker boot-disk import; the installer image in the worker machine config is always sourced from `talos.installerRepository`, so an air-gapped pool must also have `talos.installerRepository` pointed at a reachable registry.","type":"string"},"schematicID":{"description":"Talos schematic ID. Defaults to `talos.schematicID`.","type":"string"},"version":{"description":"Talos version. Defaults to `talos.version`.","type":"string"}}}}},"talos":{"description":"Talos worker image configuration for this pool.","type":"object","default":{},"required":["imageFactoryURL","installerRepository","registryMirrors","schematicID","version"],"properties":{"imageFactoryCA":{"description":"Complete PEM certificate block(s) CDI must trust when importing this pool's worker OS disk image over HTTPS — the no-osImage default and `osImage.factory` path. Empty by default, so the importer uses only its system trust store, which is what the public factory needs. Set this when the factory in use (`imageFactoryURL` or `osImage.factory.imageFactoryURL`) is a self-hosted factory or mirror served by an internal CA, which the importer otherwise rejects with `x509: certificate signed by unknown authority`. The chart renders it into a ConfigMap in the release namespace — CDI requires certConfigMap to sit alongside the DataVolume — and references that from the worker DataVolume's `source.http.certConfigMap`; CDI appends it to the system pool rather than replacing it, so a publicly-trusted endpoint verifies either way. A pool on `osImage.builtin` clones a golden image and builds no HTTP source, so this field has no effect there. On `substrate: proxmox` it has no effect on the workers either: the ConfigMap is still rendered, but nothing references it.","type":"string","default":""},"imageFactoryURL":{"description":"Default base URL of the Talos Image Factory that serves the worker OS disk image (the `openstack-amd64.raw.xz` raw artifact streamed in by CDI over HTTP). Applies only when this pool imports over HTTP — the default and `osImage.factory`, which overrides it via `osImage.factory.imageFactoryURL`. A pool on `osImage.builtin` clones a golden image from the worker image catalog and does not use this field at all. Defaults to the public factory. Point at a self-hosted Image Factory, a caching mirror, or an internal HTTP file server for air-gapped, rate-limited, or flaky-egress environments. No trailing slash. It is interpolated into the worker DataVolume source URL, so on those HTTP paths it must be a plain `http(s)` URL and the render refuses anything else; a pool on `osImage.builtin` builds no such URL and is not held to it, so blanking this field there is legal. On `substrate: proxmox` the workers clone a VM template that already carries the OS, so this field has no effect there beyond that format check.","type":"string","default":"https://factory.talos.dev"},"installerRepository":{"description":"OCI repository prefix for the Talos installer image, written into the worker machine config as `machine.install.image`. Resolved as `/:` against whatever this pool's `osImage` pins, falling back to `talos.schematicID` and `talos.version` — so the installer matches the OS the pool actually booted. The prefix itself has no `osImage.*` override, so an air-gapped pool needs it pointed at a reachable registry as well. Defaults to the public factory's installer path. Override for air-gapped or mirrored registries. No trailing slash.","type":"string","default":"factory.talos.dev/installer"},"registryMirrors":{"description":"Talos `machine.registries.mirrors` passthrough for worker nodes: a map of upstream registry host to `{ endpoints: [ ... ] }`. Empty by default, so workers pull container images (the Talos `kubelet` image included) directly from the upstream registry. Point a host such as `ghcr.io` at an in-cluster pull-through mirror for air-gapped, rate-limited, or flaky-egress environments so a worker's boot does not depend on live public egress. Talos still falls back to the upstream registry unless a host also sets `skipFallback: true`, so a mirror alone does not enforce air-gap. This value is part of the worker machine config, whose content hash names the TalosConfigTemplate, so changing it rolls every existing worker in the pool rather than only affecting nodes created later.","type":"object","default":{},"x-kubernetes-preserve-unknown-fields":true},"schematicID":{"description":"Talos image-factory schematic ID. Defaults to the cozystack-tested vanilla schematic. Operators using custom schematics (system extensions, kernel args) override here. The pool can pin its own via `osImage.builtin.schematicID` or `osImage.factory.schematicID`; when `osImage` omits it, this value applies. Whichever of the two applies is interpolated into an image URL and a DataVolume name, so it must be a plain lowercase alphanumeric identifier -- the render refuses anything else. A custom schematic on `osImage.builtin` also needs its own catalog entry, since the golden is named from the schematic ID.","type":"string","default":"ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"},"version":{"description":"Default Talos release for this pool's worker OS image and for the installer image written into the worker machine config. The pool can pin its own release via `osImage.builtin.version` or `osImage.factory.version`; when `osImage` omits it, this value applies. Both ways in are checked against the chart's Talos<->Kubernetes support matrix, against whichever release the pool actually resolves to, so an override cannot slip past a check `talos.version` would have failed. The matrix lists, for each Talos minor the chart knows, the Kubernetes minors it supports and refuses a pool `version` outside that list; a Talos minor it does not list passes unchecked, so pointing either field at a Talos release the chart does not know about is the operator's call, not a checked one. Whichever release applies is interpolated into an image URL and a DataVolume name, so it must be a full `vMAJOR.MINOR.PATCH` release, optionally with a pre-release suffix -- the render refuses anything else, a minor-only `v1.13` included. The values schema types this as a bare string, so a malformed one is accepted by the CR and fails the pool's HelmRelease instead.","type":"string","default":"v1.13.6"}}},"substrate":{"description":"Which infrastructure provider backs this pool's worker VMs. `kubevirt` runs them inside this cluster on KubeVirt, sized by `instanceType` and booted from a Talos disk image CDI streams from the image factory. `proxmox` runs them on an external Proxmox VE cluster through capmox, sized by `resources`, and cloned from a Talos VM template that already exists on the hypervisor. The two substrates take different Talos platform images and different sizing inputs, so switching an existing pool is refused: the chart reads the live MachineDeployment's template kind and fails the render rather than rolling every worker to the other hypervisor and deleting the template the draining MachineSet still references. Create a new pool on the other substrate and scale this one down.","type":"string","default":"kubevirt","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"substrate is immutable"}]},"proxmox":{"description":"Proxmox substrate settings.","type":"object","default":{},"required":["network"],"properties":{"dnsServers":{"description":"Nameservers written into the worker machineconfig. On kubevirt the workers use the management cluster's CoreDNS, which they reach over the pod network; an off-cluster Proxmox worker cannot, so it needs reachable resolvers of its own. Required when `substrate` is `proxmox`. Keep in sync with the parent kubernetes chart's `proxmox.dnsServers`.","type":"array","default":[],"items":{"type":"string"}},"full":{"description":"Full clone rather than linked. Default is a linked clone: it costs kilobytes at creation instead of the whole disk, because a ZFS-backed linked clone writes only its own increment (measured: 8K against 10.2G for the same worker full-cloned onto a `sparse 0` pool). The cost is that the clone holds the template's base snapshot, so the template cannot be rotated while any linked clone still references it — which is what the consolidation CronJob resolves 24h after creation. Set true for a pool that must be independent of the template from the first second.","type":"boolean","default":false},"network":{"description":"NIC configuration.","type":"object","default":{},"required":["bridge"],"properties":{"bridge":{"description":"Proxmox bridge the VM NIC attaches to (e.g. `vmbr0`). Required when `substrate` is `proxmox`.","type":"string","default":""},"mtu":{"description":"NIC MTU. Omitted when unset.","type":"integer"},"vlan":{"description":"L2 VLAN tag. Omitted when unset.","type":"integer"}}},"pool":{"description":"Proxmox resource pool the worker VMs are added to. Set the same pool on every node pool of a tenant cluster and grant the tenant's tokens their privileges on `/pool/`: that is the one ACL path that follows every VM capmox creates, replacements included. The pool must exist on the hypervisor; capmox does not create it. Empty adds the workers to no pool.","type":"string","default":""},"storage":{"description":"Proxmox storage for the cloned disk; needs `full: true`. A target storage is a full-clone parameter — a linked clone always lives on the template's storage, and Proxmox refuses the pair — so the render refuses it too. Empty keeps the template's storage.","type":"string","default":""},"templateTags":{"description":"Tags identifying the Talos VM template to clone. capmox matches a template when its tag set is EQUAL to this list, not when it contains it, so every tag on the template must be listed here — a missing one yields `found 0 VM templates with tags ...` while the tags are plainly present. The template has to carry this pool's `talos.schematicID` at `talos.version` in the nocloud flavour, with the Proxmox guest agent enabled for the `qemu-guest-agent` extension of the default schematic; `hack/proxmox-talos-template.sh` builds it from these values and prints the tags to list here. Do not build it from the platform release's `nocloud-amd64.raw.xz`: that is the bare-metal node image, whose `zfs` extension keeps a VM from finishing its boot sequence and reboots it at the Talos boot timeout. Required when `substrate` is `proxmox`.","type":"array","default":[],"items":{"type":"string"}}}}}} release: labels: sharding.fluxcd.io/key: tenants From 3c519c7d6f7206b277ab586ba2b5d7139ffbf7fc Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 16:57:46 +0500 Subject: [PATCH 460/518] chore(kubernetes-nodes): name a remedy that works in the foreign-pool error When a MachineDeployment named after the pool already belongs to another Helm release, the render failed with advice to remove the pool from the parent Kubernetes CR's nodeGroups. The parent renders no worker pool any more, so that edit cannot clear the conflict. The cases that do reach this guard are a pool migration 56 left with the cluster's release and two pools whose - split builds the same object name; the message now says so and points at renaming the pool or removing the other one. The foreign-owner branch had no test, so a case pins it, including the message, against a MachineDeployment owned by the cluster's release. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- .../kubernetes-nodes/templates/_helpers.tpl | 12 ++++----- .../tests/clustername_drift_test.yaml | 27 ++++++++++++++++++- 2 files changed, 32 insertions(+), 7 deletions(-) diff --git a/packages/apps/kubernetes-nodes/templates/_helpers.tpl b/packages/apps/kubernetes-nodes/templates/_helpers.tpl index 08a74b7ce..a727a1043 100644 --- a/packages/apps/kubernetes-nodes/templates/_helpers.tpl +++ b/packages/apps/kubernetes-nodes/templates/_helpers.tpl @@ -208,13 +208,13 @@ clusters in one namespace from colliding on a pool named e.g. `md0`. {{/* Fail early with a clear message if this pool's MachineDeployment already exists -under a different Helm release — i.e. the pool name collides with a nodeGroup -still managed by the parent kubernetes chart (most likely the default `md0`). +under a different Helm release: a worker pool migration 56 left with the parent +cluster's release, or another pool whose - split reconstructs +the same object name (cluster `a` with pool `b-c`, cluster `a-b` with pool `c`). Without this guard the collision surfaces as a cryptic Helm "invalid ownership metadata" error at install time. Inert under `helm template`/unittest (lookup -returns nil with no cluster) and a no-op during the Phase 2b adoption -migration, which re-annotates the MachineDeployment onto this release before -it reconciles. +returns nil with no cluster) and a no-op for a pool migration 56 adopted, which +re-annotates the MachineDeployment onto this release before it reconciles. */}} {{- define "kubernetes-nodes.assertNoForeignPool" -}} {{- $clusterName := include "kubernetes-nodes.clusterName" . -}} @@ -224,7 +224,7 @@ it reconciles. {{- if $existing -}} {{- $owner := dig "annotations" "meta.helm.sh/release-name" "" $existing.metadata -}} {{- if and $owner (ne $owner .Release.Name) -}} -{{- fail (printf "kubernetes-nodes: MachineDeployment %q in namespace %q is already managed by release %q, not this pool release %q — the pool name collides with a nodeGroup still managed by the parent kubernetes chart. Rename the pool or remove it from the parent Kubernetes CR's nodeGroups first." $mdName .Release.Namespace $owner .Release.Name) -}} +{{- fail (printf "kubernetes-nodes: MachineDeployment %q in namespace %q is already managed by release %q, not this pool release %q — two pools cannot share the object name kubernetes--: give this pool a name whose object name is free, or remove the existing pool from release %q first." $mdName .Release.Namespace $owner .Release.Name $owner) -}} {{- end -}} {{- /* clusterName-drift guard. The release name kubernetes-nodes-- does not encode where ends and begins: for one release diff --git a/packages/apps/kubernetes-nodes/tests/clustername_drift_test.yaml b/packages/apps/kubernetes-nodes/tests/clustername_drift_test.yaml index 57c4f79ff..8d0b56cf6 100644 --- a/packages/apps/kubernetes-nodes/tests/clustername_drift_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/clustername_drift_test.yaml @@ -1,7 +1,7 @@ suite: worker pool clusterName-drift guard (assertNoForeignPool) # assertNoForeignPool reconstructs the pool MachineDeployment name -# kubernetes-- and looks it up. Beyond the pre-existing +# kubernetes-- and looks it up. Beyond the # foreign-owner branch, it refuses the render when a live MD of that name is # owned by this release but its spec.clusterName disagrees with the clusterName # this pool renders: that is the fingerprint of a post-create .Values.cluster @@ -126,3 +126,28 @@ tests: value: MachineDeployment exists: path: metadata.name + + # The other branch of the same guard: a live MachineDeployment of this name + # that another release owns, here the cluster's own release, which is what a + # pool migration 56 did not adopt still carries. The message has to point at + # a remedy the operator can act on; the cluster's Kubernetes CR renders no + # worker pool, so editing it cannot clear the conflict. + - it: refuses to render over a MachineDeployment another release owns + kubernetesProvider: + scheme: *scheme + objects: + - *kcp + - *schemeMachineSet + - *schemeKmt + - kind: MachineDeployment + apiVersion: cluster.x-k8s.io/v1beta1 + metadata: + name: kubernetes-myk8s-md0 + namespace: tenant-test + annotations: + meta.helm.sh/release-name: kubernetes-myk8s + spec: + clusterName: kubernetes-myk8s + asserts: + - failedTemplate: + errorPattern: 'already managed by release "kubernetes-myk8s", not this pool release "kubernetes-nodes-myk8s-md0" — two pools cannot share the object name kubernetes--: give this pool a name whose object name is free, or remove the existing pool from release "kubernetes-myk8s" first\.' From b5e71bf8f7fdd8419a53a9092b4ee6b0039e735a Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 16:58:03 +0500 Subject: [PATCH 461/518] chore(kubernetes-nodes): drop split-era history from template and test comments Comments in the pool chart still explain it against the monolithic kubernetes chart, a "Phase 2b" migration, a render-parity gate and a tests/render-parity.sh that no longer exist, an Ubuntu+kubeadm worker, and a maxUnhealthy default of "0" that is "50%". The Makefile's update target claimed to copy the Talos support matrix, which is a literal in _helpers.tpl; it copies the Kubernetes version map. A reader of these comments takes them as the description of what the code relies on, so they now describe the code as it is. Only Helm comments are touched: the YAML comments inside the hashed machine config and machine template text stay as they are, since editing them would rename the TalosConfigTemplate or KubevirtMachineTemplate and roll every worker. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- packages/apps/kubernetes-nodes/Makefile | 17 +++++++------ .../kubernetes-nodes/templates/_helpers.tpl | 16 ++++++------- .../templates/_talosconfigtemplate.tpl | 2 +- .../kubernetes-nodes/templates/nodegroup.yaml | 24 +++++++++---------- .../tests/kubelet_reservation_test.yaml | 9 ++++--- .../tests/mhc_rendering_test.yaml | 2 -- .../tests/mhc_timeouts_test.yaml | 5 ++-- .../tests/pod_cpu_limit_test.yaml | 6 ++--- .../tests/pod_cpu_request_test.yaml | 6 ++--- .../tests/render_snapshot_test.yaml | 17 ++++++------- .../tests/talos_worker_config_test.yaml | 1 - 11 files changed, 44 insertions(+), 61 deletions(-) diff --git a/packages/apps/kubernetes-nodes/Makefile b/packages/apps/kubernetes-nodes/Makefile index 701d81eaa..9c3dcb3de 100644 --- a/packages/apps/kubernetes-nodes/Makefile +++ b/packages/apps/kubernetes-nodes/Makefile @@ -9,16 +9,15 @@ generate: cozyvalues-gen -m 'kubernetesnodes' -v values.yaml -s values.schema.json -r README.md -g ../../../api/apps/v1alpha1/kubernetesnodes/types.go ../../../hack/update-crd.sh -# Keep the worker-image inputs (Talos<->Kubernetes support matrix and the -# kubectl.tag used by the talos-reconcile Job) in lockstep with the parent -# kubernetes chart. They are copied, not symlinked, so `helm package` bundles -# them; run this after the parent chart's `make update`. +# Copy the Kubernetes version map (files/versions.yaml, minor -> patch release) +# and the kubectl.tag used by the talos-reconcile Job from the kubernetes +# chart, which owns both. They are copied, not symlinked, so `helm package` +# bundles them; run this after either changes there. # -# _talosconfigtemplate.tpl is not copied from anywhere: the parent chart no -# longer carries worker node pools, so this chart owns the worker machine -# config outright. It stays here because the TalosConfigTemplate it renders is -# content-hash named and both consumers in this chart - nodegroup.yaml, which -# writes bootstrap.configRef, and talos-reconcile-job.yaml, which creates the +# _talosconfigtemplate.tpl is not copied from anywhere: this chart owns the +# worker machine config. The TalosConfigTemplate it renders is content-hash +# named and both consumers in this chart - nodegroup.yaml, which writes +# bootstrap.configRef, and talos-reconcile-job.yaml, which creates the # object - have to hash the same bytes. update: cp ../kubernetes/files/versions.yaml files/versions.yaml diff --git a/packages/apps/kubernetes-nodes/templates/_helpers.tpl b/packages/apps/kubernetes-nodes/templates/_helpers.tpl index a727a1043..5e317f8e6 100644 --- a/packages/apps/kubernetes-nodes/templates/_helpers.tpl +++ b/packages/apps/kubernetes-nodes/templates/_helpers.tpl @@ -165,10 +165,9 @@ The pool attaches to the parent Kubernetes CR named .Values.cluster, whose HelmRelease (and therefore CAPI Cluster / KamajiControlPlane / KubevirtCluster and every worker object it owns) is `kubernetes-`. This chart does NOT use its own .Release.Name for CAPI wiring — the pool lives in a separate -HelmRelease from the control plane, so every reference that the monolithic -chart made through $.Release.Name is reconstructed here as kubernetes- -instead. Linkage is by name convention (mirrors vm-instance -> vm-disk), not -ownerReference or lookup-gated render. +HelmRelease from the control plane, so every reference to the cluster's objects +is built from kubernetes-. Linkage is by name convention (mirrors +vm-instance -> vm-disk), not ownerReference or lookup-gated render. */}} {{- define "kubernetes-nodes.clusterName" -}} {{- if not .Values.cluster -}} @@ -194,9 +193,9 @@ The node-group name for this pool, derived from the release name. A KubernetesNodes CR is named - and gets the release prefix `kubernetes-nodes-`, so the release name is `kubernetes-nodes--`. The group name is the suffix. Enforcing the `-` segment keeps -every rendered object named `kubernetes--` — byte-identical to -what the monolithic chart rendered for the same group — and prevents two -clusters in one namespace from colliding on a pool named e.g. `md0`. +every rendered object named `kubernetes--` — the names worker +objects adopted by migration 56 already carry — and prevents two clusters in +one namespace from colliding on a pool named e.g. `md0`. */}} {{- define "kubernetes-nodes.groupName" -}} {{- $prefix := printf "kubernetes-nodes-%s-" .Values.cluster -}} @@ -253,8 +252,7 @@ control plane is supported (up to n-3 upstream) and is the normal state during a rolling upgrade — the parent Kubernetes CR is bumped first, then each pool. A symmetric equality check would flip every pool of a cluster to render-failure the moment the parent is bumped, blocking all pool operations until each pool is -hand-edited, so the check is directional. The split removed the single `version` -that used to feed both control plane and workers. Looks up the parent +hand-edited, so the check is directional. Looks up the parent KamajiControlPlane (named like the reconstructed clusterName) and compares its spec.version minor against .Values.version. Skipped when the lookup is empty (helm template / unittest, or the parent not yet present) so it validates only diff --git a/packages/apps/kubernetes-nodes/templates/_talosconfigtemplate.tpl b/packages/apps/kubernetes-nodes/templates/_talosconfigtemplate.tpl index 0061d4d26..1f69cd704 100644 --- a/packages/apps/kubernetes-nodes/templates/_talosconfigtemplate.tpl +++ b/packages/apps/kubernetes-nodes/templates/_talosconfigtemplate.tpl @@ -69,7 +69,7 @@ {{- /* Management cluster DNS domain. Tenant Kubernetes workers run as KubeVirt VMs inside the management cluster's pod network and need to resolve management-cluster service short names ("..svc") - the same way an Ubuntu+kubeadm worker did via cloud-init DHCP. The + the same way a pod of the management cluster does. The kubevirt-csi NFS mount path is the load-bearing case: kubelet on the worker host calls `mount.nfs linstor-csi-nfs.cozy-linstor.svc` (no domain suffix), and without searchDomains in /etc/resolv.conf diff --git a/packages/apps/kubernetes-nodes/templates/nodegroup.yaml b/packages/apps/kubernetes-nodes/templates/nodegroup.yaml index 18ec0e16b..6f9d03987 100644 --- a/packages/apps/kubernetes-nodes/templates/nodegroup.yaml +++ b/packages/apps/kubernetes-nodes/templates/nodegroup.yaml @@ -1,17 +1,16 @@ {{- /* Worker node pool for a managed Kubernetes cluster. - This chart is the second half of the Kubernetes-app split (cozystack/community#8): - the control plane (Cluster / KamajiControlPlane / KubevirtCluster / PKI) lives in - the `kubernetes` chart, and each worker pool is a separate `kubernetes-nodes` - release. The pool objects (KubevirtMachineTemplate, MachineDeployment, - MachineHealthCheck, worker WorkloadMonitor) and the talos-reconcile Job that - produces the pool's TalosConfigTemplate are rendered here, wired to the parent - cluster `kubernetes-` by name convention (see _helpers.tpl). Every - reference the monolithic chart made through $.Release.Name is reconstructed here - as the parent clusterName so the objects are byte-identical to what the parent - rendered for the same group — the KubevirtMachineTemplate is content-hash-named, - so any drift would roll every live worker VM. + The control plane (Cluster / KamajiControlPlane / KubevirtCluster / PKI) lives + in the `kubernetes` chart, and each worker pool is a separate + `kubernetes-nodes` release. The pool objects (machine template, + MachineDeployment, MachineHealthCheck, worker WorkloadMonitor) and the + talos-reconcile Job that produces the pool's TalosConfigTemplate are rendered + here, wired to the parent cluster `kubernetes-` by name convention + (see _helpers.tpl). Every reference to the cluster is built from that + clusterName. Pools adopted by migration 56 keep running on objects of exactly + these names and this content: the machine template is content-hash-named, so + any drift in its text would roll every live worker VM. Talos -> Kubernetes support matrix, against the pool default. The pool can also reach a different Talos release through osImage.builtin.version / @@ -1066,8 +1065,7 @@ spec: {{- /* MHC spec.maxUnhealthy is an IntOrString. CAPI admission accepts a bare integer (0, 1, ...) or a percentage string ("0%", "50%"), but rejects a quoted plain integer ("0"). Quote only when the operator - supplied a percentage; otherwise render the value as a bare int so - the admission webhook is happy with the chart-default "0". */}} + supplied a percentage; otherwise render the value as a bare int. */}} {{- $maxUnhealthy := toString $.Values.maxUnhealthy }} {{- $muSource := "maxUnhealthy" }} {{- if hasSuffix "%" $maxUnhealthy }} diff --git a/packages/apps/kubernetes-nodes/tests/kubelet_reservation_test.yaml b/packages/apps/kubernetes-nodes/tests/kubelet_reservation_test.yaml index 57ae0b476..1974c97b2 100644 --- a/packages/apps/kubernetes-nodes/tests/kubelet_reservation_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/kubelet_reservation_test.yaml @@ -1,10 +1,9 @@ suite: kubelet reservation validation -# The kubelet-reservation branch (quantity-format regexes, positive-value and -# eviction-threshold invariants) moved verbatim from the parent chart into -# nodegroup.yaml. These failure-path guards reject malformed input at render -# time before a broken machineconfig reaches a worker. Ported from the parent -# chart's kubelet_reservation_test. +# The kubelet-reservation branch of nodegroup.yaml (quantity-format regexes, +# positive-value and eviction-threshold invariants). These failure-path guards +# reject malformed input at render time before a broken machineconfig reaches +# a worker. templates: - templates/nodegroup.yaml diff --git a/packages/apps/kubernetes-nodes/tests/mhc_rendering_test.yaml b/packages/apps/kubernetes-nodes/tests/mhc_rendering_test.yaml index 92603df9b..50be9041f 100644 --- a/packages/apps/kubernetes-nodes/tests/mhc_rendering_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/mhc_rendering_test.yaml @@ -4,8 +4,6 @@ suite: MachineHealthCheck maxUnhealthy int-vs-percent rendering # but rejects a bare numeric STRING ("0"): a percentage must be a quoted string # and a count must be an unquoted int. nodegroup.yaml renders it as # maxUnhealthy: {{ if hasSuffix "%" ... }}{{ . | quote }}{{ else }}{{ . | int }}{{ end }} -# Ported from the parent chart's mhc_rendering_test after the pool moved to -# kubernetes-nodes. templates: - templates/nodegroup.yaml diff --git a/packages/apps/kubernetes-nodes/tests/mhc_timeouts_test.yaml b/packages/apps/kubernetes-nodes/tests/mhc_timeouts_test.yaml index ecea4ec2e..31fa9e3f2 100644 --- a/packages/apps/kubernetes-nodes/tests/mhc_timeouts_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/mhc_timeouts_test.yaml @@ -16,9 +16,8 @@ suite: MachineHealthCheck timeouts # # `nodeStartupTimeout` bounds the phase before any of that: it is the window a # Machine has to produce a Node at all, and the unhealthyConditions entries are -# evaluated only once one exists. Its default is pinned here because the parity -# gate compares this chart against its counterpart, so a default moved in both -# at once passes there and only fails on this assertion. +# evaluated only once one exists. Its default is pinned here because a moved +# default changes every pool that does not set the field. templates: - templates/nodegroup.yaml diff --git a/packages/apps/kubernetes-nodes/tests/pod_cpu_limit_test.yaml b/packages/apps/kubernetes-nodes/tests/pod_cpu_limit_test.yaml index d7fdb5bd9..c172ac15d 100644 --- a/packages/apps/kubernetes-nodes/tests/pod_cpu_limit_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/pod_cpu_limit_test.yaml @@ -1,7 +1,6 @@ suite: worker pool pod CPU limit -# Mirror of the parent kubernetes chart's suite, on the pool-shaped values of -# this chart. `podCpuLimit` is the only knob that sets +# `podCpuLimit` is the only knob that sets # `domain.resources.limits.cpu` on a worker VM. KubeVirt otherwise derives that # limit from the guest vCPU count whenever the namespace carries a ResourceQuota # that limits CPU, and it stops deriving as soon as the VMI declares the limit @@ -79,8 +78,7 @@ tests: asserts: # Absent, not empty: the KubevirtMachineTemplate is content-hash named, so # a `cpu:` key rendered for a pool that asked for nothing would rename the - # template and roll every live worker VM of the pool on upgrade. The - # render-parity gate compares those bytes against the parent chart. + # template and roll every live worker VM of the pool on upgrade. - notExists: path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.limits.cpu - equal: diff --git a/packages/apps/kubernetes-nodes/tests/pod_cpu_request_test.yaml b/packages/apps/kubernetes-nodes/tests/pod_cpu_request_test.yaml index af5d71041..9f6bf801d 100644 --- a/packages/apps/kubernetes-nodes/tests/pod_cpu_request_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/pod_cpu_request_test.yaml @@ -1,7 +1,6 @@ suite: worker pool pod CPU request -# Mirror of the parent kubernetes chart's suite, on the pool-shaped values of -# this chart. `podCpuRequest` is the only knob that sets +# `podCpuRequest` is the only knob that sets # `domain.resources.requests.cpu` on a worker VM, and it is the other half of # `podCpuLimit`: KubeVirt copies a declared CPU limit into the request when the # VM declares none, so without this field the ceiling and the scheduler @@ -100,8 +99,7 @@ tests: asserts: # Absent, not empty: the KubevirtMachineTemplate is content-hash named, so # a `cpu:` key rendered for a pool that asked for nothing would rename the - # template and roll every live worker VM of the pool on upgrade. The - # render-parity gate compares those bytes against the parent chart. + # template and roll every live worker VM of the pool on upgrade. - notExists: path: spec.template.spec.virtualMachineTemplate.spec.template.spec.domain.resources.requests.cpu - equal: diff --git a/packages/apps/kubernetes-nodes/tests/render_snapshot_test.yaml b/packages/apps/kubernetes-nodes/tests/render_snapshot_test.yaml index 4e51d0f93..5ac60b0f3 100644 --- a/packages/apps/kubernetes-nodes/tests/render_snapshot_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/render_snapshot_test.yaml @@ -2,12 +2,9 @@ suite: worker node pool golden render snapshot # Freezes the full rendered pool (KubevirtMachineTemplate, MachineDeployment, # MachineHealthCheck, worker WorkloadMonitor) for a representative node group. -# In PR-2a this render was proven byte-identical to the monolithic kubernetes -# chart's output via tests/render-parity.sh; PR-2b removed the pool from the -# parent, so cross-chart parity can no longer run. This snapshot takes over as -# the drift guard: any change to the rendered pool objects (which would rename -# the content-hashed KubevirtMachineTemplate and churn live worker VMs on -# upgrade) fails here until the snapshot is regenerated deliberately with +# It is the drift guard for those objects: any change to them (which would +# rename the content-hashed KubevirtMachineTemplate and churn live worker VMs +# on upgrade) fails here until the snapshot is regenerated deliberately with # `helm unittest -u`. # # Coverage note: every case sets instanceType: "" with explicit resources, @@ -44,10 +41,10 @@ tests: asserts: - matchSnapshot: {} - # GPU and kubelet-override branches live on in nodegroup.yaml after the split - # and are content-hash inputs to the KubevirtMachineTemplate; pin them too so - # a change to either branch that would rename the KMT (and roll live GPU or - # kubelet-tuned worker VMs) fails here. Value sets mirror the PR-2a parity gate. + # The GPU and kubelet-override branches of nodegroup.yaml are content-hash + # inputs to the KubevirtMachineTemplate; pin them too so a change to either + # branch that would rename the KMT (and roll live GPU or kubelet-tuned worker + # VMs) fails here. - it: matches the golden render of a GPU pool set: roles: [] diff --git a/packages/apps/kubernetes-nodes/tests/talos_worker_config_test.yaml b/packages/apps/kubernetes-nodes/tests/talos_worker_config_test.yaml index 10bff88f8..bcc387dd6 100644 --- a/packages/apps/kubernetes-nodes/tests/talos_worker_config_test.yaml +++ b/packages/apps/kubernetes-nodes/tests/talos_worker_config_test.yaml @@ -7,7 +7,6 @@ suite: Talos worker machineconfig (GPU labels + kubelet reservations) # - GPU pools must inject nodeLabels gpu=on so HAMi's device plugin # (nodeSelector gpu=on) schedules and advertises nvidia.com/gpu; # - kubelet reservation overrides must reach the machineconfig kubelet config. -# Ported from the parent chart's gpu_node_labels_test / kubelet_reservation_test. templates: - templates/talos-reconcile-job.yaml From 25721617a62c7f51fa4488b99e4822b81467fd51 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 16:58:10 +0500 Subject: [PATCH 462/518] refactor(kubernetes-nodes): remove fallbacks and ignore rules with no effect The worker disk's storageClass fell back from the per-group class to .Values.storageClass, but kubernetes-nodes.group builds the per-group class from .Values.storageClass, so the fallback can only ever return the value it falls back from. It was the two-level class of the chart the pool once lived in, and its comment still described two levels. $muSource held one constant string for two fail messages, and the .helmignore rules for /hack and /images/*/* match nothing in this chart. Rendering the chart for the snapshot values, a GPU pool, a kubelet override, the factory and CA paths, an empty and a foreign storageClass, a kernelModules pool and three Proxmox variants gives byte-identical output before and after, the packaged file list is unchanged, and the snapshot and TalosConfigTemplate hash tests pass without re-pinning, so no machine template or worker is touched. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- packages/apps/kubernetes-nodes/.helmignore | 2 -- .../kubernetes-nodes/templates/nodegroup.yaml | 27 +++++++++---------- 2 files changed, 12 insertions(+), 17 deletions(-) diff --git a/packages/apps/kubernetes-nodes/.helmignore b/packages/apps/kubernetes-nodes/.helmignore index cdada6673..1ea0ae84f 100644 --- a/packages/apps/kubernetes-nodes/.helmignore +++ b/packages/apps/kubernetes-nodes/.helmignore @@ -1,5 +1,3 @@ .helmignore /logos /Makefile -/hack -/images/*/* diff --git a/packages/apps/kubernetes-nodes/templates/nodegroup.yaml b/packages/apps/kubernetes-nodes/templates/nodegroup.yaml index 6f9d03987..7ebf42033 100644 --- a/packages/apps/kubernetes-nodes/templates/nodegroup.yaml +++ b/packages/apps/kubernetes-nodes/templates/nodegroup.yaml @@ -345,7 +345,7 @@ spec: fails the whole release of a pool whose disks are all on the right class and which changed nothing. Only a pool with no disk yet resolves to today's default, which is the class its first clone will actually get. */}} - {{- $cloneWorkerSC = .group.storageClass | default $.Values.storageClass | toString }} + {{- $cloneWorkerSC = .group.storageClass | toString }} {{- if and (not $cloneWorkerSC) $poolDiskName }} {{- $poolPVC := lookup "v1" "PersistentVolumeClaim" $.Release.Namespace $poolDiskName }} {{- if $poolPVC }} @@ -526,12 +526,10 @@ spec: Worker VMs live-migrate, so their disks need ReadWriteMany. The RWX access mode is supplied by the chosen StorageClass's CDI StorageProfile — it is not set on the DataVolume here. linstor-csi - rejects RWX volumes that are not on a DRBD-backed StorageClass, so - an empty nodeGroup storageClass falls back to the application-level - storageClass (a replicated/DRBD class) instead of the cluster - default, which may be a non-DRBD class that the CSI driver would - reject. Both empty still omits the field so the cluster default - applies. + rejects RWX volumes that are not on a DRBD-backed StorageClass, + which is why the pool's storageClass defaults to a replicated/DRBD + class rather than to the cluster default. An empty storageClass + omits the field so the cluster default applies. A builtin pool instead renders the class the guard above resolved for it. Omitting the field there would let the disk bind on whatever the @@ -540,12 +538,12 @@ spec: after the pool's earlier disks were bound turns the next clone into a host-assisted copy over the pod network, the transfer the guard exists to refuse, while the guard reports no drift because it compared the - class those earlier disks sit on. With both levels empty and no cluster - to resolve against the field is still omitted, so the HTTP path renders - exactly as before and the KubevirtMachineTemplate hash named after it - does not move. + class those earlier disks sit on. With the class empty and no cluster + to resolve against the field is still omitted, so the HTTP path does + not render one and the KubevirtMachineTemplate hash named after it + does not depend on the cluster default. */}} - {{- with $cloneWorkerSC | default (.group.storageClass | default $.Values.storageClass) }} + {{- with $cloneWorkerSC | default .group.storageClass }} storageClassName: {{ . | quote }} {{- end }} template: @@ -1067,16 +1065,15 @@ spec: rejects a quoted plain integer ("0"). Quote only when the operator supplied a percentage; otherwise render the value as a bare int. */}} {{- $maxUnhealthy := toString $.Values.maxUnhealthy }} - {{- $muSource := "maxUnhealthy" }} {{- if hasSuffix "%" $maxUnhealthy }} {{- if not (regexMatch "^[0-9]+%$" $maxUnhealthy) }} - {{- fail (printf "%s must be a bare integer (e.g. 0, 1) or an integer percentage (e.g. 50%%), got %q" $muSource $maxUnhealthy) }} + {{- fail (printf "maxUnhealthy must be a bare integer (e.g. 0, 1) or an integer percentage (e.g. 50%%), got %q" $maxUnhealthy) }} {{- end }} maxUnhealthy: {{ $maxUnhealthy | quote }} {{- else if regexMatch "^[0-9]+$" $maxUnhealthy }} maxUnhealthy: {{ $maxUnhealthy | int }} {{- else }} - {{- fail (printf "%s must be a bare integer (e.g. 0, 1) or an integer percentage (e.g. 50%%), got %q" $muSource $maxUnhealthy) }} + {{- fail (printf "maxUnhealthy must be a bare integer (e.g. 0, 1) or an integer percentage (e.g. 50%%), got %q" $maxUnhealthy) }} {{- end }} nodeStartupTimeout: {{ $.Values.nodeStartupTimeout }} selector: From e56e66ae81a239e2961a7655fb941b01c7788f75 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 16:58:16 +0500 Subject: [PATCH 463/518] test(kubernetes-nodes): fail when the pool and cluster CIDRs disagree The tenant pod and service CIDRs are literals in two charts: the kubernetes chart writes them into the CAPI Cluster, the pool chart into every worker's Talos machine config. The parent's comment says they must match and keeps them "in one place", but since the pools moved out the second copy has had nothing tying it to the first, and a mismatch renders cleanly on both sides while workers route over a range the control plane does not use. The test renders both charts and compares the lists, reading the worker side from the TalosConfigTemplate the reconcile Job's heredoc actually emits rather than grepping the Job text. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- ...check-kubernetes-nodes-cidr-agreement.bats | 104 ++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 hack/check-kubernetes-nodes-cidr-agreement.bats diff --git a/hack/check-kubernetes-nodes-cidr-agreement.bats b/hack/check-kubernetes-nodes-cidr-agreement.bats new file mode 100644 index 000000000..b337296de --- /dev/null +++ b/hack/check-kubernetes-nodes-cidr-agreement.bats @@ -0,0 +1,104 @@ +#!/usr/bin/env bats +# ----------------------------------------------------------------------------- +# The tenant cluster's pod and service CIDRs are written by two charts: +# +# * packages/apps/kubernetes renders them into the CAPI Cluster's +# spec.clusterNetwork, which the control plane is built from; +# * packages/apps/kubernetes-nodes renders them into every worker's Talos +# machine config (cluster.network.podSubnets / serviceSubnets), inside the +# TalosConfigTemplate the talos-reconcile Job applies. +# +# Each chart carries its own literal, and neither render fails when the two +# disagree: the workers boot, join, and route pod and Service traffic over +# ranges the control plane does not use. This guard renders both charts and +# compares the lists, so a range moved on one side only fails here. +# +# The worker side is read the way the Job produces it: the heredoc is cut out +# of the Job's command, run through sh, and the TalosConfigTemplate it prints is +# parsed, then the machine config inside it. A text grep of the Job would also +# match a comment. +# +# Requires: helm, yq (mikefarah v4+). Runs under both `bats` and the in-repo +# cozytest.sh runner, so TMP is provisioned inline per test and cleaned without +# an EXIT trap. +# ----------------------------------------------------------------------------- + +load test_helper + +REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)" + +# $1: output directory. Writes pods and services, one CIDR per line. +cluster_cidrs() { + helm template kubernetes-myk8s "$REPO_ROOT/packages/apps/kubernetes" \ + --namespace tenant-test \ + -f "$REPO_ROOT/packages/apps/kubernetes/tests/values-ci.yaml" \ + > "$1/cluster.yaml" 2> "$1/cluster.err" \ + || { cat "$1/cluster.err" >&2; return 1; } + yq 'select(.kind == "Cluster" and .apiVersion == "cluster.x-k8s.io/v1beta1") | .spec.clusterNetwork.pods.cidrBlocks[]' \ + "$1/cluster.yaml" > "$1/cluster.pods" + yq 'select(.kind == "Cluster" and .apiVersion == "cluster.x-k8s.io/v1beta1") | .spec.clusterNetwork.services.cidrBlocks[]' \ + "$1/cluster.yaml" > "$1/cluster.services" +} + +# $1: output directory. Writes pool.pods and pool.services, one CIDR per line. +pool_cidrs() { + helm template kubernetes-nodes-myk8s-md0 "$REPO_ROOT/packages/apps/kubernetes-nodes" \ + --namespace tenant-test \ + --set cluster=myk8s \ + --set _cluster.cluster-domain=cozy.local \ + --set instanceType="" \ + --set resources.cpu=2 --set resources.memory=4Gi \ + --show-only templates/talos-reconcile-job.yaml \ + > "$1/pool.yaml" 2> "$1/pool.err" \ + || { cat "$1/pool.err" >&2; return 1; } + yq 'select(.kind == "Job") | .spec.template.spec.containers[0].command[2]' "$1/pool.yaml" > "$1/cmd.sh" + awk ' + /(^|="?\$\()cat < "$1/heredoc.sh" + sh "$1/heredoc.sh" > "$1/tct.yaml" 2> "$1/heredoc.err" \ + || { cat "$1/heredoc.err" >&2; return 1; } + yq 'select(.kind == "TalosConfigTemplate") | .spec.template.spec.data' "$1/tct.yaml" > "$1/machineconfig.yaml" + yq '.cluster.network.podSubnets[]' "$1/machineconfig.yaml" > "$1/pool.pods" + yq '.cluster.network.serviceSubnets[]' "$1/machineconfig.yaml" > "$1/pool.services" +} + +# $1: directory, $2: pods or services, $3: what to call it in a message. +assert_agree() { + cluster=$(cat "$1/cluster.$2") + pool=$(cat "$1/pool.$2") + # An empty read means a field moved or a render came out short; comparing + # "" with "" would pass on nothing. + if [ -z "$cluster" ] || [ -z "$pool" ]; then + echo "FAIL: could not read the $3 CIDRs from both renders." + echo " kubernetes Cluster.spec.clusterNetwork: '${cluster:-}'" + echo " kubernetes-nodes machine config cluster.network: '${pool:-}'" + return 1 + fi + if [ "$cluster" != "$pool" ]; then + echo "FAIL: the tenant $3 CIDRs differ between the two charts." + echo " packages/apps/kubernetes (CAPI Cluster): $cluster" + echo " packages/apps/kubernetes-nodes (worker machine config): $pool" + echo "Workers would route $3 traffic over a range the control plane does not" + echo "use. Change both charts together. Moving the worker side also renames" + echo "the content-hashed TalosConfigTemplate and rolls every worker." + return 1 + fi +} + +@test "kubernetes and kubernetes-nodes render the same pod CIDRs" { + TMP=$(mktemp -d) + cluster_cidrs "$TMP" + pool_cidrs "$TMP" + assert_agree "$TMP" pods pod + rm -rf "$TMP" +} + +@test "kubernetes and kubernetes-nodes render the same service CIDRs" { + TMP=$(mktemp -d) + cluster_cidrs "$TMP" + pool_cidrs "$TMP" + assert_agree "$TMP" services service + rm -rf "$TMP" +} From 3b5ede1589e93bbd8177f539b623639e7bca4151 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 16:51:16 +0300 Subject: [PATCH 464/518] docs(agents): add task-wide development principles Keep causal claims evidence-based and cluster targets explicit when working across management and tenant APIs. Put decisions before detail so readers can find the answer without losing supporting evidence. Validation: markdownlint (MD013 excluded) and git diff --check pass. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- AGENTS.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index f35097317..6f26a69a5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,10 +51,22 @@ This file provides structured guidance for AI coding assistants and agents worki - PR titles: a Conventional Commits header (`type(scope): description`, types from [`contributing.md`](./docs/agents/contributing.md)) auto-applies `kind/*` and `area/*` via `.github/workflows/pr-labeler.yaml`. Append `!` (or add a `BREAKING CHANGE:` footer) to apply `kind/breaking-change` **Important rules:** + - **Match scope, then read first.** Load a process file only when the task genuinely falls in its scope, and load it before starting work — not midway through. Tasks outside a file's scope should not pull it into context. - **Follow it exactly.** Do not assume you already know the process, and do not skip steps (`changelog.md` especially). Where the documentation and your instinct disagree, the documentation wins. - **[`overview.md`](./docs/agents/overview.md) is the standing reference** for structure and conventions, useful even when no route above points at it. +## Working Principles + +These hold for every task, whichever document the routes above point at. + +- **A causal claim needs a mechanism.** A failure that appeared after a commit, a bump or a rerun was not thereby caused by it, and matching symptoms prove nothing either: unrelated faults often look identical from outside. Back a root cause with a log line, a reproducer, a bisect, or a ruled-out alternative; without one, name at least two hypotheses and what would tell them apart. Before tying a problem to a change, check whether it predates the change (an older run, a condition's `lastTransitionTime`, `git blame`). Close a hypothesis when evidence rules it out. Missing evidence means inconclusive, not coincidence: record that limit, stop treating the hypothesis as a cause, and revisit it only with new evidence or a concrete discriminating check. +- **Verify, do not recall.** Versions, CLI flags, API fields and defaults drift faster than a model's training data. Check the upstream source, the documentation at the pinned version, or `--help` before writing code or advice that depends on them. When a problem is not solved after two attempts, stop and research it instead of trying a third variation. +- **Long flag names.** New and changed commands in docs, scripts, CI and PR text spell options out (`--namespace`, `--output`, `--filename`, `--message`), so that a reader does not need to know what `-f` means to this particular tool. An option with no long form is the only exception. +- **Name the cluster.** Management and tenant clusters are different targets. Every `kubectl` call, including reads, carries `--context`; cluster-facing Helm commands carry `--kube-context`. Use the human-approved target and the task's kubeconfig where one was supplied, not the ambient current context. Before running wrappers or cluster-facing tests, verify that they propagate the chosen target. Tests that need no cluster must not inherit live cluster credentials. +- **Respect the reader's time.** Lead with the answer or decision; prefer direct wording over formalities, filler and repeated conclusions. Put necessary detail after a short summary or in a linked document. +- **Public text.** Everything that reaches GitHub (commits, PR and issue bodies, review comments, code comments) is in English. It names no private infrastructure (cluster, customer or internal namespace names) and none of an agent's private tooling (custom skills, slash commands). Nothing is published on someone's behalf without their approval, unless they have delegated it. + ## Project Overview **Cozystack** is a Kubernetes-based platform for building cloud infrastructure with managed services (databases, VMs, K8s clusters), multi-tenancy, and GitOps delivery. @@ -62,6 +74,7 @@ This file provides structured guidance for AI coding assistants and agents worki ## Quick Reference ### Code Structure + - `packages/core/` - Core platform charts (installer, platform) - `packages/system/` - System components (CSI, CNI, operators) - `packages/apps/` - User-facing applications in catalog @@ -70,6 +83,7 @@ This file provides structured guidance for AI coding assistants and agents worki - `api/` - Kubernetes CRDs ### Conventions + - **Helm Charts**: Umbrella pattern, vendored upstream charts in `charts/` - **Go Code**: Controller-runtime patterns, kubebuilder style - **Git Commits**: Conventional Commits (`type(scope): description`) with `--signoff`; an agent-assisted commit adds one `Assisted-by: LLM` trailer @@ -77,6 +91,7 @@ This file provides structured guidance for AI coding assistants and agents worki - **Prose**: One continuous line per paragraph — see [Prose Formatting](#prose-formatting) ### What NOT to Do + - ❌ Edit `/vendor/`, `zz_generated.*.go`, upstream charts directly - ❌ Modify `go.mod`/`go.sum` manually (use `go get`) - ❌ Force push to main/master From 1e7ba63c57c5237c67e284fecbbf021cb0bbae2a Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 16:52:11 +0300 Subject: [PATCH 465/518] docs(agents): define testing and authoring conventions Require tests for behavior changes without inventing failures for prose or unchanged upstream updates. Keep new image build filenames explicit and preserve the existing lookup policy. Correct the assertion guidance: Bats can fail on a final negation, while cozytest appends a successful return. Separate numeric validation from text coercion and account for null parent maps in Helm values. Validation: markdownlint (MD013 excluded) and git diff --check pass. Fixes: 5137d98e7119 ("test(bats): refuse a negated assertion across every suite") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- AGENTS.md | 2 +- docs/agents/e2e-testing.md | 6 +++--- docs/agents/overview.md | 41 ++++++++++++++++++++++++++++++++++++-- 3 files changed, 43 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6f26a69a5..3e74ff133 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,7 +36,7 @@ This file provides structured guidance for AI coding assistants and agents worki - **E2E tests and E2E CI** (e.g., "write/fix an e2e test", "stabilize a flaky test", "add a bats test", "change the e2e workflow", "why did e2e fail") - Read: [`e2e-testing.md`](./docs/agents/e2e-testing.md) - - Action: Read the entire file and follow the conventions exactly — no new retries on deterministic steps, event-driven backstops before every `kubectl wait`, no test-level `EXIT`/`RETURN` traps (a self-contained one inside a Chainsaw `script` step or an explicit subshell is fine — see the doc), no assertion beginning with `!` (errexit exempts an inverted status, so it cannot fail — write `if cmd; then echo "FAIL: …"; false; fi`), fail-fast on HelmRelease readiness + - Action: Read the entire file and follow the conventions exactly — no new retries on deterministic steps, event-driven backstops before every `kubectl wait`, no test-level `EXIT`/`RETURN` traps (a self-contained one inside a Chainsaw `script` step or an explicit subshell is fine — see the doc), no assertion beginning with `!` (errexit exempts an inverted status, so it can be silently ignored — write `if cmd; then echo "FAIL: …"; false; fi`), fail-fast on HelmRelease readiness - **General questions about contributing** - Read: [`contributing.md`](./docs/agents/contributing.md) diff --git a/docs/agents/e2e-testing.md b/docs/agents/e2e-testing.md index 9ad31d967..243acdecc 100644 --- a/docs/agents/e2e-testing.md +++ b/docs/agents/e2e-testing.md @@ -181,9 +181,9 @@ Consequences to keep in mind when touching either workflow: ### 11. Never negate a BATS assertion with `!` -An assertion spelled `! cmd` inside an `@test` body cannot fail, whichever runner executes it. POSIX exempts a command whose return value is inverted with `!` from `set -e`, so the failing status is dropped where it is produced. `hack/cozytest.sh` then appends `return 0` to every test function, so it never resurfaces as the test's own status. The `bats` binary looks stricter and mostly is not: its ERR trap carries the identical exemption, and `bash -c 'set -eE; trap "echo TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap, and exits zero. The one shape `bats` does catch is a negated assertion standing **last** in the body, where the inverted status is what the function returns — put any statement below it and `bats` passes the test too. This is the same family as the ban on test-level `EXIT`/`RETURN` traps, and unlike that one it is not settled by preferring the stricter runner, because there is no stricter runner to prefer. +An assertion spelled `! cmd` inside an `@test` body can silently pass. POSIX exempts a command whose return value is inverted with `!` from `set -e`, so a failing inverted status does not stop the test body. `hack/cozytest.sh` then appends `return 0` to every test function, so it never resurfaces as the test's own status. The `bats` binary looks stricter and mostly is not: its ERR trap carries the identical exemption, and `bash -c 'set -eE; trap "echo TRAP" ERR; ! true; echo REACHED'` prints `REACHED`, runs no trap, and exits zero. The one shape `bats` does catch is a negated assertion standing **last** in the body, where the inverted status is what the function returns — put any statement below it and `bats` passes the test too. This is the same family as the ban on test-level `EXIT`/`RETURN` traps, and unlike that one it is not settled by preferring the stricter runner, because there is no stricter runner to prefer. -The exemption is also wider than the "negated pipeline" phrasing found in older comments: a bare `! true`, a negated `[ … ]`, one inside a `for` body, and one following `;`, `&&` or `||` all pass. Write the absence so that it fails — `if cmd; then echo "FAIL: "; false; fi` — and let the `false` carry the verdict. `hack/bats-no-negated-assert.bats` enforces this across every `hack/**/*.bats`, subdirectories included. Inverted **conditions** (`if !`, `elif !`, `while !`, `until !`) are untouched, because `set -e` is not what decides a condition. `[ ! -f x ]` is untouched for a different reason and stays a normal, failing assertion: the `!` there is an operator of `test`, not negation of a command, so nothing exempts its status. The scan is lexical, with the limits that implies: a statement assembled at runtime is invisible to it, and the form is counted inside a heredoc or a fixture string where no command exists — which reports a file holding nothing rather than passing one that does, and is why that guard's own fixtures build the token from a separate `printf` argument. +The exemption applies to any inverted command, not just pipelines: `! true`, negated `[ … ]`, negation inside a `for` body, and negation following `;`, `&&` or `||` do not trigger errexit. A later successful statement can hide their failure. Write the absence so that it fails — `if cmd; then echo "FAIL: "; false; fi` — and let the `false` carry the verdict. `hack/bats-no-negated-assert.bats` enforces this across every `hack/**/*.bats`, subdirectories included. Inverted **conditions** (`if !`, `elif !`, `while !`, `until !`) are untouched, because `set -e` is not what decides a condition. `[ ! -f x ]` is untouched for a different reason and stays a normal, failing assertion: the `!` there is an operator of `test`, not negation of a command, so nothing exempts its status. The scan is lexical, with the limits that implies: a statement assembled at runtime is invisible to it, and the form is counted inside a heredoc or a fixture string where no command exists — which reports a file holding nothing rather than passing one that does, and is why that guard's own fixtures build the token from a separate `printf` argument. ## Chainsaw v0.2.15 gotchas @@ -209,7 +209,7 @@ The suite is pinned to Chainsaw **v0.2.15** (the latest release as of May 2026); 7. Failure path attaches scoped diagnostics via a `catch:` block, never a silent pass. The node-join deadline is no exception, and no longer has one: with the soft-red marker gone it fails its suite like any other deadline and its diagnostics run from the same `catch:`. 8. If it touches HelmRelease install/upgrade behavior, add the `status.history` guard: fail the run on a `failed` or `uninstalled` Snapshot of a release whose install strategy is `RetryOnFailure`, fail it on a parent whose strategy is anything else, and report the footprint of every other release without failing. What is reusable today is the predicate and the jsonpath in `hack/e2e-chainsaw/_lib/remediation-guard.sh`; the policy that weighs the footprint against `.spec.install.strategy.name`, judges an empty history against the release's own readiness, and keeps each read's exit status apart from its output lives in `cozy_guard_helmrelease` in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` — read it before rebuilding any of that for another suite. If the suite has a failure path that exits before the guard, give that path `cozy_report_helmrelease_remediation` as well, or the runs the guard exists for are the runs it does not see. 9. A test that creates a tenant `Kubernetes` cluster waits for its release to reach `Ready` **before the assertions it ends on**, with a `--timeout` the guard can read as positive and no longer than a day. Reads earlier in the run are expected and are not what this bounds. The mechanism, how to size the budget and what the guard does with it are under "Do not mask cleanup or teardown failures". One thing to carry from there rather than look up: the naming is part of the rule, not a detail of the check, so an end-assertion added under any name outside the `cozy_assert_*` / `cozy_switch_and_assert_*` convention is invisible to it and passes before the wait without reddening anything — as long as one recognised call site is left behind it. Rename every recognised call and the guard reds instead, on finding none at all, which is the one shape of this mistake it does catch. -10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and the assertion cannot fail; use `if cmd; then echo "FAIL: …"; false; fi`. +10. No BATS assertion begins with `!` — an inverted status is exempt from `set -e` and can be silently ignored; use `if cmd; then echo "FAIL: …"; false; fi`. ## The container lane: what the srv nodes are, and what that costs diff --git a/docs/agents/overview.md b/docs/agents/overview.md index 9ab67c41d..74580a370 100644 --- a/docs/agents/overview.md +++ b/docs/agents/overview.md @@ -16,7 +16,7 @@ The platform exposes infrastructure services via the Kubernetes API with ready-m ## Code Layout -``` +```text . ├── packages/ # Main directory for cozystack packages │ ├── core/ # Core platform logic charts (installer, platform) @@ -50,7 +50,7 @@ The platform exposes infrastructure services via the Kubernetes API with ready-m Every package is a Helm chart following the umbrella chart pattern: -``` +```text packages/// ├── Chart.yaml # Chart definition and parameter docs ├── Makefile # Development workflow targets @@ -110,13 +110,35 @@ A chart also declares the limits on its own resource name there, with `## @name - **E2E tests:** Kyverno Chainsaw suites in `hack/e2e-chainsaw/` (one directory per app), run with `chainsaw test`. Cluster bootstrap (`hack/e2e-install-cozystack.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live in [`e2e-testing.md`](./e2e-testing.md). - **Go tests:** standard `go test`, with Ginkgo/Gomega for controllers. +### Test first, failure path first + +These rules apply to changes with testable behavior. Documentation-only edits and unmodified upstream updates need the relevant checks, not an artificial failing test. First-party adaptations made during a dependency bump remain covered by these rules. + +1. Write the test before changing the implementation and watch it fail for the intended reason, not on a typo or a missing fixture. Then make it green, then refactor under it. +2. Cover relevant failure paths before the happy path: empty or null values, invalid input, a number where a string is expected, a missing resource, a timeout or an API error. Assert the resulting message, exit status and post-failure state required by the operation's contract. +3. Fix regressions in the implementation. Do not skip, weaken or delete a test, or silence a linter, merely to get a green run. When the intended contract changes, update or replace the obsolete assertions and retain coverage of applicable failure cases. +4. A stub needs a failing test for the intended behavior during development. Finish that behavior and make the test pass before merge; do not make the test accept an unintended not-implemented result just to get green CI. +5. A bug found along the way that is out of the PR's scope gets an issue and a test pinning the current broken behaviour, marked `FIXME(#N)`. The PR that fixes the bug flips the test to pin the fixed behaviour. + +Do not negate a BATS assertion with `!`: it can silently pass. Bats catches a failing negation only when it is the last command in the test body; `hack/cozytest.sh` appends a successful return and misses it there too. See [§11 of `e2e-testing.md`](./e2e-testing.md#11-never-negate-a-bats-assertion-with-). + +### Linters + +Fix every linter finding before pushing, the cosmetic ones included. When a rule genuinely conflicts with the project, change the linter configuration and state the reason there, instead of suppressing the finding inline. + +Markdown follows these markdownlint rules: every code fence names a language (`text` when nothing else fits), table separators have spaces around the pipes (`| --- |`), and there is a blank line before and after every list and after every heading. Line length (MD013) is the exception: prose stays one line per paragraph, as [Prose Formatting](../../AGENTS.md#prose-formatting) explains. + ## Conventions ### Helm Charts + - Follow **umbrella chart** pattern for system components - Include upstream charts in `charts/` directory (vendored, not referenced) - Override configuration in root `values.yaml` - Use `values.schema.json` for input validation and dashboard UI rendering +- An unquoted numeric value (`name: 123`) reaches a template as a number: `len` on it aborts the render with `len of type float64` (`int64` for a value given with `--set`) before any `fail` message is reached, and `if not $v` treats `0` as unset. For fields validated as text that permit numeric inputs, reject missing or null values with `kindIs "invalid"`, then coerce with `printf "%v"` and check emptiness with `eq $v ""`. Coercing null first produces the non-empty string ``. +- For integer fields, keep string inputs on their lexical validation path. Validate numeric inputs for type, integrality and bounds before conversion, then format the validated integer in decimal. Generic `printf "%v"` can emit exponent notation for a float (`4200000000` becomes `4.2e+09`), so it is not integer normalization. +- For nested values, validate or normalize each parent map before reading its children; checking only the leaf cannot prevent a nil-pointer error from a missing or null parent. **No `lookup` that copies another object into the render.** A `lookup` reads cluster state at render time, and nothing re-renders the HelmRelease when that state changes, so whatever it copied into the output (a bucket name, an endpoint, a CA, a list of StorageClasses) stays stale until something unrelated upgrades the release. Lookups of this kind already in the charts have caused failures, and several have since been moved to values, so they are not precedent: do not add one of this kind unless a maintainer has authorized that specific one. @@ -124,7 +146,18 @@ A `lookup` is fine when a repeated render gives the same output and carries noth To use a fact about the cluster instead, read it from values: `_cluster` and `_namespace` in the `cozystack-values` Secret, or `valuesFrom` on a specific Secret, whose content change is a values change and so upgrades the release. Ask whether an API is served with `.Capabilities.APIVersions.Has`, since a `lookup` on an unserved group aborts the whole render. Otherwise check at run time in a hook or Job, or let a controller own the object. +### Container images + +For a new container image built from source maintained in this repository, name the build definition `Containerfile` and pass it to `docker buildx build` with `--file` in the package `Makefile`. Keep existing `Dockerfile`s and vendored build definitions unchanged. + +### Examples and fixtures + +Illustrative addresses and names in docs, examples and test fixtures come from the reserved documentation ranges, never from a routable value that merely looks realistic: IPv4 from RFC 5737 (`192.0.2.0/24`, `198.51.100.0/24`, `203.0.113.0/24`), IPv6 from RFC 3849 (`2001:db8::/32`), domains from RFC 2606 and RFC 6761 (`example.com`, `.example`, `.test`, `.invalid`), ASNs from RFC 5398 (`64496`–`64511`, `65536`–`65551`), MAC addresses from RFC 7042 (`00:00:5E:00:53:00`–`00:00:5E:00:53:FF`). Where no reserved range exists, use an obviously synthetic placeholder such as ``. + +When a test depends on a protocol-defined address's behavior, use that value, such as `127.0.0.1` or `localhost` for a local test server. Keep network tests isolated; this exception does not permit arbitrary live endpoints. + ### Go Code + - Follow standard **Go conventions** and idioms - Use **controller-runtime** patterns for Kubernetes controllers - Prefer **kubebuilder** for API definitions and controllers @@ -170,6 +203,7 @@ Before working on these areas, read the relevant doc first: ## Things Agents Should Not Do ### Never Edit These + - Do not modify files in `/vendor/` (Go dependencies) - Do not edit generated files: `zz_generated.*.go` - Do not change `go.mod`/`go.sum` manually (use `go get`) @@ -177,13 +211,16 @@ Before working on these areas, read the relevant doc first: - Do not modify image digests in `values.yaml` (generated by build) ### Version Control + - Do not commit built artifacts from `_out` - Do not commit test artifacts or temporary files ### Git Operations + - Do not force push to main/master - Do not update git config - Do not perform destructive operations without explicit request ### Core Components + - Do not modify `packages/core/platform/` without understanding migration impact From 7475c5af5d72d807422f7de9a5ea66d1a698e9fd Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 16:52:47 +0300 Subject: [PATCH 466/518] docs(agents): define contribution and review requirements Keep commit history self-contained and issue links visible on the PR. Distinguish commit-reference trailers from issue-closing references, and reserve closing keywords for complete fixes. Include review bodies, conversation and commit comments alongside inline threads without imposing the same review depth on every change. Verify feedback against current code before acting on it. Validation: markdownlint passes with MD013 excluded, and MD041 excluded for the PR template, whose existing first heading is level two. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .github/PULL_REQUEST_TEMPLATE.md | 1 + docs/agents/contributing.md | 112 +++++++++++++------------------ 2 files changed, 47 insertions(+), 66 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 068b125b4..cf463c5e8 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -13,6 +13,7 @@ ## What this PR does + ### Screenshots diff --git a/docs/agents/contributing.md b/docs/agents/contributing.md index 96af91026..7db92100a 100644 --- a/docs/agents/contributing.md +++ b/docs/agents/contributing.md @@ -42,14 +42,14 @@ After every change, verify the relevant artifacts before opening the PR: - **Helm templates changed:** `make show` in the package directory renders cleanly. - **Go code changed:** `make generate && make manifests` at the repo root, then commit the regenerated files. - **`values.yaml` annotations changed:** `make generate` in the package; review the `values.schema.json` diff. -- **Migration scripts:** confirm what data actually exists first with `kubectl get -A -o jsonpath='{...}'` on a dev cluster before writing the logic. +- **Migration scripts:** when the task supplies a kubeconfig, export its path as `KUBECONFIG`. Set `KUBE_CONTEXT` to the human-approved dev-cluster context and confirm the existing data with `kubectl --context "${KUBE_CONTEXT:?}" get --all-namespaces --output=jsonpath='{...}'` before writing the logic. ## Commit Format Follow [Conventional Commits](https://www.conventionalcommits.org/) with `--signoff`: ```bash -git commit --signoff -m "type(scope): brief description" +git commit --signoff --message "type(scope): brief description" ``` **Types:** `feat`, `fix`, `security`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore` @@ -67,12 +67,23 @@ Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREA The subject is imperative and specific; `fix`, `wip` and `update` are not subjects. The body, when there is one, explains why the change is made and only that: the diff already shows what changed and how. Write for a reader who has only `git log`, so no review-iteration or planning vocabulary (`address review comments`, `pass 2`, `batch 1`, `as discussed`) and no bare issue or ticket number in place of the explanation. **Examples:** + ```bash -git commit --signoff -m "feat(dashboard): add config hash annotations to restart pods on config changes" -git commit --signoff -m "fix(postgres): update operator to version 1.2.3" -git commit --signoff -m "docs(contributing): add installation guide" +git commit --signoff --message "feat(dashboard): add config hash annotations to restart pods on config changes" +git commit --signoff --message "fix(postgres): update operator to version 1.2.3" +git commit --signoff --message "docs(contributing): add installation guide" ``` +### One Commit, One Logical Change + +A commit is the unit a backport cherry-picks and a revert undoes, so it carries exactly one logical change: one fix, one feature, one refactor, together with the regenerated files and tests that belong to it. When the content of a commit changes during review, rewrite that commit (`git commit --amend`, or a `fixup` in `git rebase --interactive`) rather than stacking a correction on top; the branch should read as if each change had been right the first time. If another commit in the series names the rewritten commit in a `Fixes:` trailer, update that reference to its new hash and subject. + +### References to Commits and Issues + +A commit that fixes a defect introduced by an earlier commit carries a kernel-style `Fixes:` trailer with at least 12 characters of the hash and the full subject, on one unbroken line, which `git log --max-count=1 --abbrev=12 --format='Fixes: %h ("%s")' ` prints. `Fixes: #N` is not used, so that the `Fixes:` trailer always points at a commit. + +Use `Closes: #N` only when the change fully resolves the issue. For partial work, use a non-closing reference such as `Refs: #N`; a closing commit trailer can close the issue when it reaches the default branch even if the PR body uses a non-closing link. For an issue in another repository, qualify the reference as `owner/repository#N`, including in a closing trailer (`Closes: owner/repository#N`). Reference trailers go after the body and before `Assisted-by:` and `Signed-off-by:`. + ### AI Agent Attribution When an AI agent authors or materially assists with a commit, add exactly one trailer line next to the `Signed-off-by:` trailer that `--signoff` produces: @@ -104,13 +115,13 @@ Editing the title does not re-run it. The `edited` trigger was dropped in #3211 **Type → `kind/*`:** -| type | label | -| --------- | ------------------ | -| feat | kind/feature | -| fix | kind/bug | -| docs | kind/documentation | -| chore | kind/cleanup | -| refactor | kind/cleanup | +| type | label | +| --- | --- | +| feat | kind/feature | +| fix | kind/bug | +| docs | kind/documentation | +| chore | kind/cleanup | +| refactor | kind/cleanup | | security, style, perf, test, build, ci, revert | (no kind label) | **Scope → `area/*`** (full mapping in `.github/workflows/pr-labeler.yaml`): @@ -150,13 +161,17 @@ If the branch has extra commits, clean it up: git fetch upstream git checkout -b my-feature upstream/main git cherry-pick -git push -f origin my-feature +git push --force origin my-feature ``` ## Pull Request Body Fill in the template at [`.github/PULL_REQUEST_TEMPLATE.md`](../../.github/PULL_REQUEST_TEMPLATE.md). It includes the required `release-note` block and the [Downstream Repositories](#downstream-repositories) checklist. +Link the issues the PR addresses in its body. Use `Closes #N` when it fully resolves an issue, or `Closes owner/repository#N` for another repository; link partial work without a closing keyword. A closing keyword in a commit can close the issue on merge into the default branch, but does not create the PR's linked-issue entry. Put the reference in the PR body even when the commit already carries it. + +GitHub interprets closing keywords in a PR description only when it targets the default branch. For a release-branch PR, use a non-closing reference and have a maintainer with write access link the issue manually and close it separately when appropriate. See [GitHub's issue-linking rules](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/linking-a-pull-request-to-an-issue). + Create the PR with `gh pr create --title "type(scope): brief description" --body-file `. `--body` and `--body-file` replace the body wholesale, so the template is not applied and its checklists are silently dropped. Start the body file from the template instead: @@ -167,6 +182,14 @@ cp .github/PULL_REQUEST_TEMPLATE.md /tmp/pr-body.md gh pr create --draft --title "type(scope): brief description" --body-file /tmp/pr-body.md ``` +### What Goes in the Commit, What Goes in the PR Body + +The commit message is the permanent record: a reader of `git log` or `git blame` years later has it and nothing else. It carries what broke or was missing and for whom, the exact error string verbatim when there is one, so that `git log --grep` finds it, why this design and not the obvious alternative, why it is safe where the diff does not show that, and one line on how it was tested, including the failure path when the change adds one. + +The PR body is read once, next to those commits, so it does not repeat them. Its first screen says what is broken or missing, where, and what that costs. After that come one sentence per commit and only what no single commit can carry: results that need the whole series, the environment they came from, decisions the reviewer has to make across commits. Questions to reviewers and open doubts belong in the PR body and never in a commit, where they go stale the day it merges. A fact that fits both goes in the commit. + +Size a commit body against accepted commits touching the same files, not against a fixed line cap, which cuts the answers before the narrative. After shortening any of these texts, check that every must-keep fact survived, that every number still sits next to the version or setup it was measured on, that nothing new was added, and that a claim about a set ("every caller", "never happens") is backed by a grep or rewritten as a mechanism. + ## Downstream Repositories Cozystack is upstream for repositories that are **not** kept in sync with it automatically. A change here can break them silently, because nothing in CI compares the two sides. @@ -292,60 +315,17 @@ The release process keeps a **separate** list of repositories, in [`changelog.md - A bug-fix PR should include a behavioural regression test, not just a single field assertion — prove the bug can no longer recur. - Keep the PR description aligned with the template: `## What this PR does` plus the `release-note` block. +- Before posting a claim about a set ("all callers", "the only user", "never happens"), grep for the counterexample, or state the mechanism instead. +- Disagreement carries a mechanism: a code path, a number, a spec clause, a failing test. Time spent and urgency are not arguments. +- Answer the point directly and explain why it is accepted or declined. Read brief feedback literally rather than inferring tone, and avoid unnecessary notifications. Follow the [prose guidance](../../AGENTS.md#working-principles). -## Fetching Unresolved Review Comments +## Collecting Review Feedback -Cozystack uses GitHub review threads with resolution status. Only unresolved threads are actionable — resolved threads are already handled. +Read the feedback from these sources, including existing replies: -The REST endpoint `/pulls/{pr}/reviews` returns review summaries, not individual review comments. Use the GraphQL API to access `reviewThreads` with `isResolved` status: +- Unresolved inline review threads. +- Review submission bodies, even when the review also has inline comments. +- General PR conversation comments. +- Comments posted directly on the PR's commits. -```bash -gh api graphql -F owner=cozystack -F repo=cozystack -F pr= -f query=' -query($owner: String!, $repo: String!, $pr: Int!) { - repository(owner: $owner, name: $repo) { - pullRequest(number: $pr) { - reviewThreads(first: 100) { - nodes { - isResolved - comments(first: 100) { - nodes { - id - path - line - author { login } - bodyText - url - createdAt - } - } - } - } - } - } -}' --jq '.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved == false) | .comments.nodes[]' -``` - -Compact one-line variant: - -```bash -gh api graphql -F owner=cozystack -F repo=cozystack -F pr= -f query=' -query($owner: String!, $repo: String!, $pr: Int!) { - repository(owner: $owner, name: $repo) { - pullRequest(number: $pr) { - reviewThreads(first: 100) { - nodes { - isResolved - comments(first: 100) { - nodes { - path - line - author { login } - bodyText - } - } - } - } - } - } -}' --jq '.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved == false) | .comments.nodes[] | "\(.path):\(.line // "N/A") - \(.author.login): \(.bodyText[:150])"' -``` +Verify each human or automated finding against the current code before deciding whether it needs a change. Do not post a duplicate answer to a point already addressed. From 39d134b3ef7f69cddf1994ca4b967492c3a3d970 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 16:53:37 +0300 Subject: [PATCH 467/518] docs(agents): clarify E2E execution targets Fork workflow edits are exercised only after they reach the default branch. State that boundary so a rerun is not mistaken for testing an unmerged workflow change. Point install checks at the shared readiness helper rather than duplicating its implementation in the instructions. Validation: markdownlint (MD013 excluded) and git diff --check pass. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- docs/agents/e2e-testing.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/agents/e2e-testing.md b/docs/agents/e2e-testing.md index 243acdecc..d0616a5e1 100644 --- a/docs/agents/e2e-testing.md +++ b/docs/agents/e2e-testing.md @@ -81,8 +81,7 @@ Trap-based cleanup was the single biggest source of false failures in the BATS s The install test fails if **any** HelmRelease is not Ready. A toothless gate (backgrounded `kubectl wait` discarding child exit codes + a bare `echo` instead of `exit 1`) once shipped a permanently-failing platform HR through green CI for weeks. -- Use a single `kubectl wait hr --all -A`, then an outcome-based re-list (covers HRs created after the snapshot), and `exit 1` on any non-Ready HR. -- Dump the full Ready-condition message per non-Ready HR so the real error is in the test output. See `hack/e2e-install-cozystack.bats`. +Use `cozy_wait_all_helmreleases_ready` from `hack/e2e-wait-helmreleases.sh`, as `hack/e2e-install-cozystack.bats` does, and preserve its failure status and diagnostics. ### 6. Assert no HelmRelease failed an action it retries in place — via `status.history` @@ -160,6 +159,7 @@ A fork's `pull_request` run carries no registry credentials, so it cannot push t Consequences to keep in mind when touching either workflow: +- **A `workflow_run` workflow always runs the default branch's copy of itself.** An edit to `e2e-fork.yaml` therefore has no effect on any fork PR, the one carrying the edit included, until it is merged to `main`. To see it work, merge first and then push to a fork PR: re-running an existing `e2e-fork` run replays the commit and workflow file it was first triggered with. - **Fork builds export, they do not push (`OCI_EXPORT_DIR`).** The fork's unprivileged build sets `OCI_EXPORT_DIR` (see `hack/common-envs.mk`) so every image is written to a per-image OCI archive artifact instead of being pushed; the privileged `e2e-fork.yaml` later pushes those archives to the registry **by digest**. Any build recipe that shells out to push directly — e.g. a `skopeo copy … docker://…` that bypasses the `image-tags`/`oci-output` macros — must honour `OCI_EXPORT_DIR` too, or a fork build dies with an anonymous-push `denied` (#3257). This is covered by `hack/common-envs_test.bats`. - **"E2E Tests" is a commit status — not a job, and not a check-run.** The required merge-gate context named `E2E Tests` is a GitHub **commit status**, posted by the `plan`/`e2e-report` pair for same-repo PRs and by `e2e-fork.yaml`'s `resolve`/`report` pair for forks — **not** the `e2e` job itself. Each lane opens the context with `pending` and concludes it when the suite is done, and the opener is not optional: a commit status never expires, so a head SHA that already went green would otherwise keep reading green from the moment a re-run, a reopen or a `full-e2e` label starts a new run until that run reports, leaving the pull request mergeable across a suite that may be about to fail. While the gate was a job, a fresh non-success check run appeared automatically and closed that window. The two lanes close it at different speeds, and the fork lane does not close it entirely. Same-repo, the opener is `plan`'s first step, so the window lasts seconds. On a fork the opener is `resolve`, which cannot run until the unprivileged build has finished, so re-running an already-green fork head leaves the old green readable for the length of that build. The gap is structural rather than an oversight: a fork's `pull_request` token is read-only and cannot write a status at all, which is the same property the rest of this design relies on. It is bounded by the build and it is the one case where "the opener closes the window" is not literally true. Two reasons it is a status: - *Not a job:* a required *job* that is skipped (as the `e2e` job is for forks) counts as passed by branch protection — exactly how fork PRs used to merge with e2e never run (#3257). A status is `success` only when the report step posts it; a never-posted required status stays "Expected" and blocks the merge. If you rename the `e2e` job, do **not** name it `E2E Tests` and do not wire branch protection to a job name — the status must stay the gate. From b3ca798bc438f0854d624b19d11fb1bab67a73d7 Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 12 Nov 2026 17:01:29 +0300 Subject: [PATCH 468/518] fix(backups): read the runs that may own a MariaDB dump uncached Before deleting a dump the cleanup checks every other operator Backup on the same prefix that could have written it, and it read them from the informer cache. A run whose operator Backup had not reached the cache yet but had already uploaded its dump was missed, and its dump deleted as this Backup's. Both the recorded and the legacy paths now list the runs from the API server, as the restore checks already do. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../backupcontroller/backup_controller.go | 7 ++--- .../mariadbstrategy_cleanup.go | 4 +-- .../mariadbstrategy_cleanup_test.go | 27 +++++++++++++++++++ 3 files changed, 33 insertions(+), 5 deletions(-) diff --git a/internal/backupcontroller/backup_controller.go b/internal/backupcontroller/backup_controller.go index b67c640ab..5e6292635 100644 --- a/internal/backupcontroller/backup_controller.go +++ b/internal/backupcontroller/backup_controller.go @@ -44,9 +44,10 @@ type BackupReconciler struct { // cleanup reads them to reach the platform bucket. CredentialsConfig BackupCredentialsConfig // APIReader is the manager's uncached reader. The MariaDB cleanup lists - // the RestoreJobs and operator Restores of a Backup through it: a - // restore that started before the delete may not be in the cache yet. - // Wired in SetupWithManager. + // the RestoreJobs and operator Restores of a Backup, and the operator + // Backups that may own its dump, through it: a restore that started + // before the delete, or a run that has already uploaded its dump, may + // not be in the cache yet. Wired in SetupWithManager. APIReader client.Reader // newS3ObjectLister and newS3ObjectRemover override the S3 client the diff --git a/internal/backupcontroller/mariadbstrategy_cleanup.go b/internal/backupcontroller/mariadbstrategy_cleanup.go index 3ff8adfbe..912bb0038 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup.go @@ -120,7 +120,7 @@ func (r *BackupReconciler) cleanupMariaDBBackup(ctx context.Context, backup *bac } absent := false if name != "" { - other, found, err := contestedMariaDBDump(s3ctx, lister, r.Client, op, name) + other, found, err := contestedMariaDBDump(s3ctx, lister, r.apiReader(), op, name) absent = !found if err != nil { return r.retryMariaDBCleanup(ctx, backup, location+name, err) @@ -130,7 +130,7 @@ func (r *BackupReconciler) cleanupMariaDBBackup(ctx context.Context, backup *bac fmt.Sprintf("the recorded dump also falls in the lifetime of k8s.mariadb.com/Backup %s, which may have written it; the operator Backup is kept", other)), nil } } else { - dump, err := findMariaDBDumpWith(s3ctx, lister, r.Client, op, backup.Spec.TakenAt.Time, true) + dump, err := findMariaDBDumpWith(s3ctx, lister, r.apiReader(), op, backup.Spec.TakenAt.Time, true) if errors.Is(err, errMariaDBDumpNotFound) || errors.Is(err, errMariaDBDumpAmbiguous) { return r.releaseMariaDBCleanup(ctx, backup, location, fmt.Sprintf("the Backup records no dump and none is proven to be the one k8s.mariadb.com/Backup %s wrote (%v); the operator Backup is kept", opName, err)), nil diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index 354bf83f3..1d2f10cbe 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -547,6 +547,33 @@ func TestCleanupMariaDBBackup_RecordedDumpAlreadyGoneIsNotContested(t *testing.T } } +// The runs that may own a dump are read from the API server: a run whose +// operator Backup has not reached the cache yet can already have uploaded +// its dump, and missing it would delete that dump as this Backup's. +func TestCleanupMariaDBBackup_ContestsWithRunsTheCacheHasNotSeen(t *testing.T) { + started := cleanupCreated.Add(time.Minute) + theirs := dumpKey(cleanupPrefix, started.Add(30*time.Second), "gzip") + unseen := func() *mariadbtypes.Backup { + b := cleanupOperatorBackup(t) + b.Name = "op-unseen" + b.CreationTimestamp = metav1.NewTime(started) + b.Status.Conditions = nil + return b + } + cases := map[string]string{"recorded dump": strings.TrimPrefix(theirs, cleanupPrefix), "legacy Backup": ""} + for name, recorded := range cases { + t.Run(name, func(t *testing.T) { + bucket := &fakeS3Bucket{objects: []s3ObjectInfo{{Key: theirs, LastModified: started.Add(time.Minute)}}} + backup := cleanupBackup(recorded) + r, rec := cleanupReconciler(t, bucket, backup, cleanupOperatorBackup(t), s3CredsSecret("tenant", "creds"), tenantNamespace()) + r.APIReader = newMariaDBStrategyTestClient(t, cleanupOperatorBackup(t), unseen()) + + res, err := r.cleanupMariaDBBackup(context.Background(), backup) + assertReleasedUntouched(t, r, rec, bucket, res, err) + }) + } +} + // A run started after the recorded dump was uploaded cannot have written it. func TestCleanupMariaDBBackup_DeletesItsRecordedDumpDespiteALaterUnfinishedRun(t *testing.T) { key := cleanupPrefix + cleanupDumpName From 5c6fc10a235c8ff51939fb6b0282fcee13f82b6e Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 12 Nov 2026 17:02:40 +0300 Subject: [PATCH 469/518] test(postgres): pin three guards no test caught when changed A job whose own Backup exists skips the wait for another job's plugin rollout, and the size probe keeps the strategy's Secret key names while it takes the Secret from the record, adding a credentials block only when the strategy has none. Each of these could be changed with the suite still green: dropping the own-Backup check, or forcing the missing-credentials branch either way. Cases for a job with a running Backup next to a recent foreign attach, a strategy without credentials and a strategy with custom key names now fail on each change. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- .../cnpg_artifact_size_test.go | 62 +++++++++++++++++++ .../cnpgstrategy_controller_test.go | 17 +++++ 2 files changed, 79 insertions(+) diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index 29209c5f1..bdf9c5d7b 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -514,3 +514,65 @@ func TestCreateCNPGBackupArtifact_SizeProbeSkipsARecordWithoutCredentials(t *tes t.Errorf("Artifact = %#v, want no size", got.Status.Artifact) } } + +// probeRecordedBackup runs createCNPGBackupArtifact for a backup recorded as +// written with credentials Secret recordedSecret, against rendered, and +// returns the size it measured and the connection it measured through. +func probeRecordedBackup(t *testing.T, rendered *strategyv1alpha1.CNPGTemplate, recordedSecret *corev1.Secret) (int64, s3Connection) { + t.Helper() + apiGroup := backupsv1alpha1.DefaultApplicationAPIGroup + j := &backupsv1alpha1.BackupJob{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "bj-probe"}, + Spec: backupsv1alpha1.BackupJobSpec{ + ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: &apiGroup, Kind: postgresAppKind, Name: "pg"}, + }, + } + cnpgBk := &cnpgtypes.Backup{ObjectMeta: metav1.ObjectMeta{Name: "cnpg-bk"}, Status: cnpgtypes.BackupStatus{BackupID: "20261021T100000"}} + secretName := recordedSecret.Name + written := cnpgDestination{DestinationPath: "s3://bkt/tenant/pg/", EndpointURL: "https://s3.example:8443", S3SecretRef: &secretName} + l := &fakeS3Lister{objects: []s3ObjectInfo{{Key: "tenant/pg/srv/base/20261021T100000/data.tar.gz", Size: 42}}} + r := reconcilerWithLister(t, l) + r.Client = newCNPGStrategyTestClient(t, recordedSecret) + + got, err := r.createCNPGBackupArtifact(context.Background(), j, &ResolvedBackupConfig{}, cnpgBk, "postgres-pg", "srv", "postgresql", written, rendered, newPostgresApp("pg", "tenant")) + if err != nil { + t.Fatalf("createCNPGBackupArtifact: %v", err) + } + if got.Status.Artifact == nil { + t.Fatalf("no artifact") + } + return got.Status.Artifact.SizeBytes, l.conn +} + +// A strategy that has since dropped its credentials still measures a backup +// recorded with a Secret, through that Secret and its default keys. +func TestCreateCNPGBackupArtifact_SizeProbeOfARecordedSecretUnderAStrategyWithoutCredentials(t *testing.T) { + store := newSizedBarmanObjectStore() + store.S3Credentials = nil + size, conn := probeRecordedBackup(t, &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *store}, s3CredsSecret("tenant", "creds-recorded")) + if size != 42 { + t.Errorf("SizeBytes = %d, want 42 measured through the recorded Secret", size) + } + if conn.AccessKey != "AKID" { + t.Errorf("probed with access key %q, want the recorded Secret's AKID", conn.AccessKey) + } +} + +// The record names the Secret, not its keys: the strategy's key names stay, +// so a strategy with its own key names still reads the recorded Secret. +func TestCreateCNPGBackupArtifact_SizeProbeKeepsTheStrategysSecretKeys(t *testing.T) { + store := newSizedBarmanObjectStore() + store.S3Credentials.AccessKeyIDKey = "id" + store.S3Credentials.SecretAccessKeyKey = "key" + secret := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Namespace: "tenant", Name: "creds-recorded"}, + Data: map[string][]byte{"id": []byte("CUSTOMID"), "key": []byte("custom-secret")}, + } + size, conn := probeRecordedBackup(t, &strategyv1alpha1.CNPGTemplate{BarmanObjectStore: *store}, secret) + if size != 42 { + t.Errorf("SizeBytes = %d, want 42 measured with the strategy's key names", size) + } + if conn.AccessKey != "CUSTOMID" { + t.Errorf("probed with access key %q, want CUSTOMID read under the strategy's key name", conn.AccessKey) + } +} diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index bbfc8b0d7..5cd990f18 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -3824,6 +3824,23 @@ func TestReconcileCNPG_JobWithoutLatchWaitsForAnotherJobsRollout(t *testing.T) { } } +// A job whose own Backup already exists is past every wait before it: it +// tracks that Backup even while another job's recent attach is still rolling +// the instances, since waiting would only hold the result of a Backup that +// runs regardless. +func TestReconcileCNPG_JobWithItsOwnBackupDoesNotWaitForAnotherJobsRollout(t *testing.T) { + cluster := newRolloutCluster(cnpgClusterHealthyPhase, buildBarmanPlugin(rolloutCluster, rolloutCluster)) + cluster.Annotations = map[string]string{cnpgPluginAttachedAtAnnotation: time.Now().Add(-5 * time.Minute).UTC().Format(time.RFC3339)} + f := newCNPGRolloutFixture(t, newRolloutBackupJob(time.Now().Add(-time.Hour)), cluster, + newRolloutInstancePod(time.Now().Add(-10*time.Minute).Truncate(time.Second))) + f.seedCNPGBackup("bj-cnpg", "bj", rolloutCluster, "running") + + j, _ := f.pass() + if ready := apimeta.FindStatusCondition(j.Status.Conditions, "Ready"); ready == nil || ready.Reason != "WaitingForCNPGBackup" { + t.Errorf("Ready = %+v, want WaitingForCNPGBackup tracking the job's own Backup", ready) + } +} + // A plugin attached by someone else after the latch, another backup job or // the chart re-rendering a restored Cluster outside the platform flow, // started the rollout before this job's write, which only finds it there: From 583f18437d8f78dbf65fca767c779fe25bb7a53e Mon Sep 17 00:00:00 2001 From: Alexey Artamonov Date: Thu, 12 Nov 2026 17:21:51 +0300 Subject: [PATCH 470/518] test(clickhouse): give the credentials-wait BackupJob a UID The test pins that a backup held for late bucket credentials keeps its archive name, and a run's archive name now carries part of the BackupJob's UID, so the fixture needs one for the name it expects to be the name the controller mints. Assisted-by: LLM Signed-off-by: Alexey Artamonov --- internal/backupcontroller/altinitystrategy_controller_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/internal/backupcontroller/altinitystrategy_controller_test.go b/internal/backupcontroller/altinitystrategy_controller_test.go index 49da21fa0..a9022ceb6 100644 --- a/internal/backupcontroller/altinitystrategy_controller_test.go +++ b/internal/backupcontroller/altinitystrategy_controller_test.go @@ -1640,7 +1640,7 @@ func TestReconcileAltinityRestore_UnreadableHoldStateDoesNotStartTheJob(t *testi func TestReconcileAltinity_CredentialsWaitKeepsTheArchiveName(t *testing.T) { startedAt := metav1.NewTime(time.Date(2026, 3, 4, 5, 6, 7, 0, time.UTC)) backupJob := &backupsv1alpha1.BackupJob{ - ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test"}, + ObjectMeta: metav1.ObjectMeta{Name: "test-bj", Namespace: "tenant-test", UID: "0f3c9a1e-5b7d-4c2a-9e8f-1a2b3c4d5e6f"}, Spec: backupsv1alpha1.BackupJobSpec{ ApplicationRef: newAltinityClickHouseRef("ch-test"), BackupClassName: "clickhouse-backup", @@ -1659,7 +1659,7 @@ func TestReconcileAltinity_CredentialsWaitKeepsTheArchiveName(t *testing.T) { }, } ctx := context.Background() - const want = "clickhouse-ch-test-20260304T050607Z" + const want = "ch-test-20260304050607-0f3c9a1e5b7d" reconcile := func(step string) { t.Helper() j := &backupsv1alpha1.BackupJob{} From 6297ebd70865dfd8a5f084521715a7720c7b8971 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 02:55:31 +0300 Subject: [PATCH 471/518] fix(linstor): run LINSTOR init containers on the platform server image piraeus-operator fills in its own default LINSTOR server image for every container this package leaves unset, and two of those are init containers: run-migration in the controller pod and setup-lvm-configuration in the satellite pods. Only the main containers were pinned to the platform build. run-migration upgrades the LINSTOR database in place. When the operator default is newer than the platform build, it migrates the schema ahead of the controller, the controller then fails to read the migrated database and crash-loops, and the migration cannot be rolled back. This stayed hidden while the operator default was older than the version built here; an operator update that moves the default past it exposes it during the window between the operator and linstor releases reconciling. Pin both init containers to the same image as the main containers, so the database moves only together with the controller. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .../system/linstor/templates/cluster.yaml | 6 +++ .../linstor/templates/satellites-cozy.yaml | 3 ++ .../linstor/tests/server-image_test.yaml | 40 +++++++++++++++++++ 3 files changed, 49 insertions(+) create mode 100644 packages/system/linstor/tests/server-image_test.yaml diff --git a/packages/system/linstor/templates/cluster.yaml b/packages/system/linstor/templates/cluster.yaml index ba5154375..c648c0a8c 100644 --- a/packages/system/linstor/templates/cluster.yaml +++ b/packages/system/linstor/templates/cluster.yaml @@ -36,6 +36,12 @@ spec: enabled: true podTemplate: spec: + {{- /* run-migration upgrades the database in place: left on the + operator's default image it migrates ahead of the controller, + irreversibly. */}} + initContainers: + - name: run-migration + image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }} containers: - name: linstor-controller image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }} diff --git a/packages/system/linstor/templates/satellites-cozy.yaml b/packages/system/linstor/templates/satellites-cozy.yaml index 7897189f4..bb6240128 100644 --- a/packages/system/linstor/templates/satellites-cozy.yaml +++ b/packages/system/linstor/templates/satellites-cozy.yaml @@ -11,6 +11,9 @@ spec: spec: # host-network is recommended by Piraeus while it is not default in the upstream hostNetwork: true + initContainers: + - name: setup-lvm-configuration + image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }} containers: - name: linstor-satellite image: {{ .Values.piraeusServer.image.repository }}:{{ .Values.piraeusServer.image.tag }} diff --git a/packages/system/linstor/tests/server-image_test.yaml b/packages/system/linstor/tests/server-image_test.yaml new file mode 100644 index 000000000..d63d19ae4 --- /dev/null +++ b/packages/system/linstor/tests/server-image_test.yaml @@ -0,0 +1,40 @@ +suite: every LINSTOR server container runs the platform image + +# piraeus-operator ships its own default for the LINSTOR server image and +# applies it to every container this package leaves unset. The init +# containers matter most: run-migration upgrades the LINSTOR database in +# place, so an operator default newer than the platform build migrates the +# schema ahead of the controller, which then refuses to start, and the +# migration cannot be undone. Pinning them to the same image as the main +# containers makes the database move only together with the controller. + +templates: + - templates/cluster.yaml + - templates/satellites-cozy.yaml + +set: + piraeusServer: + image: + repository: registry.example/piraeus-server + tag: v0.0.0-test + +tests: + - it: the controller migration runs the platform image + template: templates/cluster.yaml + asserts: + - equal: + path: spec.controller.podTemplate.spec.initContainers[?(@.name=="run-migration")].image + value: registry.example/piraeus-server:v0.0.0-test + - equal: + path: spec.controller.podTemplate.spec.containers[?(@.name=="linstor-controller")].image + value: registry.example/piraeus-server:v0.0.0-test + + - it: the satellite LVM setup runs the platform image + template: templates/satellites-cozy.yaml + asserts: + - equal: + path: spec.podTemplate.spec.initContainers[?(@.name=="setup-lvm-configuration")].image + value: registry.example/piraeus-server:v0.0.0-test + - equal: + path: spec.podTemplate.spec.containers[?(@.name=="linstor-satellite")].image + value: registry.example/piraeus-server:v0.0.0-test From acfae30136fa9dfdd1afeab5dead9401f74d92fe Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:20:34 +0300 Subject: [PATCH 472/518] fix(linstor): update piraeus-operator to v2.12.1 The nfs-server DaemonSet runs piraeus-csi-nfs-server v1.10.3, which boots systemd as PID 1 in a privileged container with the host /dev mounted. Its getty units hang up the host's getty@tty1 about a second after the pod starts; systemd restarts the host getty until it hits the start limit and gives up, so login on the local or out-of-band console stops working on nodes that run the NFS server. Seen on bare-metal Ubuntu 24.04 nodes with Cozystack 1.6.5 and 1.7.0. Upstream fixed the image by disabling the services the NFS server does not need (piraeusdatastore/linstor-csi#417, reported as piraeusdatastore/piraeus-operator#961); v1.10.6 includes the fix. v2.12.1 is the current stable release. Its image set is LINSTOR 1.35.3 and LINSTOR CSI v1.13.2, which this repository builds itself; both builds follow in the next commits. The CSI release line is what carries IO QoS limits (qos.linbit.com/* StorageClass parameters, added in v1.11.3), which operators have asked for. Behaviour that changes with this range: - The CSIDriver moves to fsGroupPolicy: File, so fsGroup is applied to ReadWriteMany filesystem volumes too. - The operator manages Secrets only in its own namespace, and the CSI controller loses its cluster-wide read on Secrets; nothing in this repository references snapshotter Secrets. - The csi-health-monitor sidecar is removed. - Volume group snapshots are enabled only when the groupsnapshot.storage.k8s.io/v1 API is served; the snapshot controller here still ships v1alpha1, so they stay off. The csi-provisioner arguments are unchanged in this range, so the restated strictTopology list stays complete. The chart is vendored the way `make update` does it, which also drops the stale CRD copy from the vendored chart; piraeus-operator-crds installs the CRDs. Declare the local chart dependency so Helm can validate the umbrella chart without changing its rendered resources. This commit is not deployable on its own. The operator now hands the CSI pods their client certificate through LS_USER_CERTIFICATE_FILE and LS_USER_KEY_FILE, which the linstor-wait-until in the current linstor-csi image does not read, so the CSI init containers retry forever until the linstor-csi update in the next commit. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .../linstor/tests/csi-topology_test.yaml | 2 +- .../piraeus-operator-crds/templates/crds.yaml | 218 +- packages/system/piraeus-operator/Chart.yaml | 4 + .../charts/piraeus/Chart.yaml | 4 +- .../charts/piraeus/templates/config.yaml | 42 +- .../charts/piraeus/templates/crds.yaml | 2173 ----------------- .../charts/piraeus/templates/deployment.yaml | 18 +- .../piraeus/templates/extra-objects.yaml | 15 + .../piraeus/templates/metrics-service.yaml | 1 + .../charts/piraeus/templates/rbac.yaml | 47 +- .../piraeus/templates/rolebindings.yaml | 16 + .../piraeus/templates/serviceaccount.yaml | 1 + .../validating-webhook-configuration.yaml | 2 + .../piraeus/templates/webhook-service.yaml | 1 + .../charts/piraeus/values.yaml | 43 + 15 files changed, 372 insertions(+), 2215 deletions(-) delete mode 100644 packages/system/piraeus-operator/charts/piraeus/templates/crds.yaml create mode 100644 packages/system/piraeus-operator/charts/piraeus/templates/extra-objects.yaml diff --git a/packages/system/linstor/tests/csi-topology_test.yaml b/packages/system/linstor/tests/csi-topology_test.yaml index fbde97a2d..6dd9bb3ea 100644 --- a/packages/system/linstor/tests/csi-topology_test.yaml +++ b/packages/system/linstor/tests/csi-topology_test.yaml @@ -34,7 +34,7 @@ tests: # Until there is, the re-check on a bump is a human obligation, stated in # templates/cluster.yaml beside the list. # - # The list was last verified complete against piraeus-operator v2.10.2: + # The list was last verified complete against piraeus-operator v2.12.1: # eleven upstream flags plus --strict-topology. - it: the restated argument list is complete template: templates/cluster.yaml diff --git a/packages/system/piraeus-operator-crds/templates/crds.yaml b/packages/system/piraeus-operator-crds/templates/crds.yaml index 5bd9212c6..d190bc2b9 100644 --- a/packages/system/piraeus-operator-crds/templates/crds.yaml +++ b/packages/system/piraeus-operator-crds/templates/crds.yaml @@ -151,13 +151,17 @@ spec: *SecretName using the issuer configured here. properties: group: - description: Group of the resource being referred to. + description: |- + Group of the issuer being referred to. + Defaults to 'cert-manager.io'. type: string kind: - description: Kind of the resource being referred to. + description: |- + Kind of the issuer being referred to. + Defaults to 'Issuer'. type: string name: - description: Name of the resource being referred to. + description: Name of the issuer being referred to. type: string required: - name @@ -201,6 +205,11 @@ spec: type: object x-kubernetes-map-type: atomic x-kubernetes-preserve-unknown-fields: true + replicas: + description: Number of desired pods. Defaults to 1. + format: int32 + minimum: 0 + type: integer type: object csiController: description: CSIController controls the deployment of the CSI Controller @@ -320,13 +329,17 @@ spec: issuer configured here. properties: group: - description: Group of the resource being referred to. + description: |- + Group of the issuer being referred to. + Defaults to 'cert-manager.io'. type: string kind: - description: Kind of the resource being referred to. + description: |- + Kind of the issuer being referred to. + Defaults to 'Issuer'. type: string name: - description: Name of the resource being referred to. + description: Name of the issuer being referred to. type: string required: - name @@ -345,6 +358,18 @@ spec: * Store credentials for accessing remotes for backups. See https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-encrypt_commands for more information. type: string + maxConcurrentEvacuations: + description: |- + MaxConcurrentEvacuations limits how many Satellites are evacuated at the same time when using the + "Evacuate" deletion policy. + + This applies both to Satellites removed because their node no longer matches the cluster, and to nodes + removed through ClusterAPI. While the limit is reached, additional Satellites wait for a free slot before + their volumes are moved and, for ClusterAPI managed nodes, before the node is allowed to drain. + A value of 0 (the default) places no limit on the number of concurrent evacuations. + format: int32 + minimum: 0 + type: integer nfsServer: description: NFSServer controls the deployment of the LINSTOR CSI NFS Server DaemonSet. @@ -568,9 +593,10 @@ spec: operator: description: |- Operator represents a key's relationship to the value. - Valid operators are Exists and Equal. Defaults to Equal. + Valid operators are Exists, Equal, Lt, and Gt. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category. + Lt and Gt perform numeric comparisons (requires feature gate TaintTolerationComparisonOperators). type: string tolerationSeconds: description: |- @@ -1054,13 +1080,17 @@ spec: issuer configured here. properties: group: - description: Group of the resource being referred to. + description: |- + Group of the issuer being referred to. + Defaults to 'cert-manager.io'. type: string kind: - description: Kind of the resource being referred to. + description: |- + Kind of the issuer being referred to. + Defaults to 'Issuer'. type: string name: - description: Name of the resource being referred to. + description: Name of the issuer being referred to. type: string required: - name @@ -1321,6 +1351,15 @@ spec: x-kubernetes-list-map-keys: - name x-kubernetes-list-type: map + resourceNameSuffixSeparator: + description: |- + ResourceNameSuffixSeparator is inserted between the base resource name and the Satellite name. + + Defaults to ".". + enum: + - . + - '-' + type: string storagePools: description: StoragePools is a list of storage pools to configure on the node. @@ -1347,18 +1386,66 @@ spec: lvmPool: description: Configures a LVM Volume Group as storage pool. properties: - volumeGroup: + externalLocking: + description: |- + ExternalLocking skips LINSTOR's internal locking for the shared VG, relying on an external + lock manager such as lvmlockd on the hosts instead. + type: boolean + physicalVolumeCreateArguments: + description: |- + PhysicalVolumeCreateArguments are arguments to pass to "pvcreate". + This has no effect on an existing VG, it only applies when the VG initially gets created. + items: + type: string + type: array + sharedSpace: + description: |- + SharedSpace marks the VG as shared between multiple nodes, identified by the given name. + All storage pools backed by the same physical storage must use the same shared space name. + A shared VG must be set up manually on the hosts, it cannot be created from source devices. type: string + volumeGroup: + description: VolumeGroup is the name of the Volume Group + (VG) to use. + type: string + volumeGroupCreateArguments: + description: |- + VolumeGroupCreateArguments are arguments to pass to "vgcreate". + This has no effect on an existing VG, it only applies when the VG initially gets created. + items: + type: string + type: array type: object lvmThinPool: description: Configures a LVM Thin Pool as storage pool. properties: + logicalVolumeCreateArguments: + description: |- + LogicalVolumeCreateArguments are arguments to pass to "lvcreate". + This has no effect on an existing thinpool, it only applies when the thinpool initially gets created. + items: + type: string + type: array + physicalVolumeCreateArguments: + description: |- + PhysicalVolumeCreateArguments are arguments to pass to "pvcreate". + This has no effect on an existing VG, it only applies when the VG initially gets created. + items: + type: string + type: array thinPool: description: ThinPool is the name of the thinpool LV (without VG prefix). type: string volumeGroup: type: string + volumeGroupCreateArguments: + description: |- + VolumeGroupCreateArguments are arguments to pass to "vgcreate". + This has no effect on an existing VG, it only applies when the VG initially gets created. + items: + type: string + type: array type: object name: description: Name of the storage pool in linstor. @@ -1447,6 +1534,16 @@ spec: zPool: description: ZPool is the name of the ZFS zpool. type: string + zpoolCreateArguments: + description: |- + ZpoolCreateArguments are arguments to pass to "zpool create". + This has no effect on an existing zpool, it only applies when the zpool initially gets created. + + If not set, "-o failmode=continue" is automatically added to ensure a failed zpool does not stop replication + to and from other nodes in the cluster. + items: + type: string + type: array type: object zfsThinPool: description: Configures a ZFS system based storage pool, allocating @@ -1455,6 +1552,16 @@ spec: zPool: description: ZPool is the name of the ZFS zpool. type: string + zpoolCreateArguments: + description: |- + ZpoolCreateArguments are arguments to pass to "zpool create". + This has no effect on an existing zpool, it only applies when the zpool initially gets created. + + If not set, "-o failmode=continue" is automatically added to ensure a failed zpool does not stop replication + to and from other nodes in the cluster. + items: + type: string + type: array type: object required: - name @@ -1762,13 +1869,17 @@ spec: issuer configured here. properties: group: - description: Group of the resource being referred to. + description: |- + Group of the issuer being referred to. + Defaults to 'cert-manager.io'. type: string kind: - description: Kind of the resource being referred to. + description: |- + Kind of the issuer being referred to. + Defaults to 'Issuer'. type: string name: - description: Name of the resource being referred to. + description: Name of the issuer being referred to. type: string required: - name @@ -1925,6 +2036,15 @@ spec: repository: description: Repository used to pull workload images. type: string + resourceNameSuffixSeparator: + description: |- + ResourceNameSuffixSeparator is inserted between the base resource name and the Satellite name. + + Defaults to ".". + enum: + - . + - '-' + type: string storagePools: description: StoragePools is a list of storage pools to configure on the node. @@ -1951,18 +2071,66 @@ spec: lvmPool: description: Configures a LVM Volume Group as storage pool. properties: - volumeGroup: + externalLocking: + description: |- + ExternalLocking skips LINSTOR's internal locking for the shared VG, relying on an external + lock manager such as lvmlockd on the hosts instead. + type: boolean + physicalVolumeCreateArguments: + description: |- + PhysicalVolumeCreateArguments are arguments to pass to "pvcreate". + This has no effect on an existing VG, it only applies when the VG initially gets created. + items: + type: string + type: array + sharedSpace: + description: |- + SharedSpace marks the VG as shared between multiple nodes, identified by the given name. + All storage pools backed by the same physical storage must use the same shared space name. + A shared VG must be set up manually on the hosts, it cannot be created from source devices. type: string + volumeGroup: + description: VolumeGroup is the name of the Volume Group + (VG) to use. + type: string + volumeGroupCreateArguments: + description: |- + VolumeGroupCreateArguments are arguments to pass to "vgcreate". + This has no effect on an existing VG, it only applies when the VG initially gets created. + items: + type: string + type: array type: object lvmThinPool: description: Configures a LVM Thin Pool as storage pool. properties: + logicalVolumeCreateArguments: + description: |- + LogicalVolumeCreateArguments are arguments to pass to "lvcreate". + This has no effect on an existing thinpool, it only applies when the thinpool initially gets created. + items: + type: string + type: array + physicalVolumeCreateArguments: + description: |- + PhysicalVolumeCreateArguments are arguments to pass to "pvcreate". + This has no effect on an existing VG, it only applies when the VG initially gets created. + items: + type: string + type: array thinPool: description: ThinPool is the name of the thinpool LV (without VG prefix). type: string volumeGroup: type: string + volumeGroupCreateArguments: + description: |- + VolumeGroupCreateArguments are arguments to pass to "vgcreate". + This has no effect on an existing VG, it only applies when the VG initially gets created. + items: + type: string + type: array type: object name: description: Name of the storage pool in linstor. @@ -2051,6 +2219,16 @@ spec: zPool: description: ZPool is the name of the ZFS zpool. type: string + zpoolCreateArguments: + description: |- + ZpoolCreateArguments are arguments to pass to "zpool create". + This has no effect on an existing zpool, it only applies when the zpool initially gets created. + + If not set, "-o failmode=continue" is automatically added to ensure a failed zpool does not stop replication + to and from other nodes in the cluster. + items: + type: string + type: array type: object zfsThinPool: description: Configures a ZFS system based storage pool, allocating @@ -2059,6 +2237,16 @@ spec: zPool: description: ZPool is the name of the ZFS zpool. type: string + zpoolCreateArguments: + description: |- + ZpoolCreateArguments are arguments to pass to "zpool create". + This has no effect on an existing zpool, it only applies when the zpool initially gets created. + + If not set, "-o failmode=continue" is automatically added to ensure a failed zpool does not stop replication + to and from other nodes in the cluster. + items: + type: string + type: array type: object required: - name diff --git a/packages/system/piraeus-operator/Chart.yaml b/packages/system/piraeus-operator/Chart.yaml index 40b3fce28..7a1925e25 100644 --- a/packages/system/piraeus-operator/Chart.yaml +++ b/packages/system/piraeus-operator/Chart.yaml @@ -1,3 +1,7 @@ apiVersion: v2 name: cozy-piraeus-operator version: 0.0.0 # Placeholder, the actual version will be automatically set during the build process +dependencies: + - name: piraeus + version: "*" + repository: file://charts/piraeus diff --git a/packages/system/piraeus-operator/charts/piraeus/Chart.yaml b/packages/system/piraeus-operator/charts/piraeus/Chart.yaml index 777722f3f..1565e6a27 100644 --- a/packages/system/piraeus-operator/charts/piraeus/Chart.yaml +++ b/packages/system/piraeus-operator/charts/piraeus/Chart.yaml @@ -3,8 +3,8 @@ name: piraeus description: | The Piraeus Operator manages software defined storage clusters using LINSTOR in Kubernetes. type: application -version: 2.10.2 -appVersion: "v2.10.2" +version: 2.12.1 +appVersion: "v2.12.1" maintainers: - name: Piraeus Datastore url: https://piraeus.io diff --git a/packages/system/piraeus-operator/charts/piraeus/templates/config.yaml b/packages/system/piraeus-operator/charts/piraeus/templates/config.yaml index 2e6963f0a..4788ffba0 100644 --- a/packages/system/piraeus-operator/charts/piraeus/templates/config.yaml +++ b/packages/system/piraeus-operator/charts/piraeus/templates/config.yaml @@ -3,6 +3,7 @@ apiVersion: v1 kind: ConfigMap metadata: name: {{ include "piraeus-operator.fullname" . }}-image-config + namespace: {{ .Release.Namespace }} labels: {{- include "piraeus-operator.labels" . | nindent 4 }} data: @@ -17,38 +18,38 @@ data: # quay.io/piraeusdatastore/piraeus-server:v1.24.2 components: linstor-controller: - tag: v1.32.3 + tag: v1.35.3 image: piraeus-server linstor-satellite: - tag: v1.32.3 + tag: v1.35.3 image: piraeus-server linstor-csi: - tag: v1.10.3 + tag: v1.13.2 image: piraeus-csi nfs-server: - tag: v1.10.3 + tag: v1.13.2 image: piraeus-csi-nfs-server drbd-reactor: - tag: v1.10.0 + tag: v1.12.0 image: drbd-reactor ha-controller: - tag: v1.3.1 + tag: v1.3.4 image: piraeus-ha-controller drbd-shutdown-guard: - tag: v1.1.1 + tag: v1.1.2 image: drbd-shutdown-guard ktls-utils: - tag: v1.2.1 + tag: v1.5.0 image: ktls-utils linstor-affinity-controller: - tag: v1.3.0 + tag: v1.5.0 image: linstor-affinity-controller drbd-module-loader: - tag: v9.2.16 + tag: v9.3.4 # The special "match" attribute is used to select an image based on the node's reported OS. # The operator will first check the k8s node's ".status.nodeInfo.osImage" field, and compare it against the list # here. If one matches, that specific image name will be used instead of the fallback image. - image: drbd9-noble # Fallback image: chose a recent kernel, which can hopefully compile whatever config is actually in use + image: drbd9-resolute # Fallback image: chose a recent kernel, which can hopefully compile whatever config is actually in use match: - osImage: Red Hat Enterprise Linux 8\. image: drbd9-almalinux8 @@ -84,6 +85,10 @@ data: image: drbd9-jammy - osImage: Ubuntu 24\.04 image: drbd9-noble + - osImage: Ubuntu 26\.04 + image: drbd9-resolute + - osImage: Debian GNU/Linux 13 + image: drbd9-trixie - osImage: Debian GNU/Linux 12 image: drbd9-bookworm - osImage: Debian GNU/Linux 11 @@ -93,25 +98,22 @@ data: base: registry.k8s.io/sig-storage components: csi-attacher: - tag: v4.10.0 + tag: v4.13.0 image: csi-attacher csi-livenessprobe: - tag: v2.17.0 + tag: v2.20.0 image: livenessprobe csi-provisioner: - tag: v6.1.0 + tag: v6.4.0 image: csi-provisioner csi-snapshotter: - tag: v8.4.0 + tag: v8.7.0 image: csi-snapshotter csi-resizer: - tag: v2.0.0 + tag: v2.3.0 image: csi-resizer - csi-external-health-monitor-controller: - tag: v0.16.0 - image: csi-external-health-monitor-controller csi-node-driver-registrar: - tag: v2.15.0 + tag: v2.18.0 image: csi-node-driver-registrar {{- range $idx, $value := .Values.imageConfigOverride }} {{ add $idx 1 }}_helm_override.yaml: | diff --git a/packages/system/piraeus-operator/charts/piraeus/templates/crds.yaml b/packages/system/piraeus-operator/charts/piraeus/templates/crds.yaml deleted file mode 100644 index 5bd9212c6..000000000 --- a/packages/system/piraeus-operator/charts/piraeus/templates/crds.yaml +++ /dev/null @@ -1,2173 +0,0 @@ - {{ if .Values.installCRDs }} ---- -apiVersion: apiextensions.k8s.io/v1 -kind: CustomResourceDefinition -metadata: - annotations: - controller-gen.kubebuilder.io/version: v0.18.0 - name: linstorclusters.piraeus.io -spec: - group: piraeus.io - names: - kind: LinstorCluster - listKind: LinstorClusterList - plural: linstorclusters - singular: linstorcluster - scope: Cluster - versions: - - additionalPrinterColumns: - - description: If the LINSTOR Cluster is available - jsonPath: .status.conditions[?(@.type=='Available')].status - name: Available - type: string - - description: If the LINSTOR Cluster is fully configured - jsonPath: .status.conditions[?(@.type=='Configured')].status - name: Configured - type: string - - description: The version of the LINSTOR Cluster - jsonPath: .status.version - name: Version - priority: 10 - type: string - - description: The number of running/expected Satellites - jsonPath: .status.satellites - name: Satellites - type: string - - description: The used capacity in all storage pools - jsonPath: .status.capacity - name: Used Capacity - type: string - - description: The number of volumes in the cluster - jsonPath: .status.numberOfVolumes - name: Volumes - type: integer - - description: The number of snapshots in the cluster - jsonPath: .status.numberOfSnapshots - name: Snapshots - priority: 10 - type: integer - - jsonPath: .metadata.creationTimestamp - name: Age - type: date - name: v1 - schema: - openAPIV3Schema: - description: LinstorCluster is the Schema for the linstorclusters API - properties: - apiVersion: - description: |- - APIVersion defines the versioned schema of this representation of an object. - Servers should convert recognized schemas to the latest internal value, and - may reject unrecognized values. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - type: string - kind: - description: |- - Kind is a string value representing the REST resource this object represents. - Servers may infer this from the endpoint the client submits requests to. - Cannot be updated. - In CamelCase. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - type: string - metadata: - type: object - spec: - description: LinstorClusterSpec defines the desired state of LinstorCluster - properties: - affinityController: - description: AffinityController controls the deployment of the Affinity - Controller Deployment. - properties: - enabled: - default: true - description: Enable the component. - type: boolean - podTemplate: - description: |- - Template to apply to Pods of the component. - - The template is applied as a patch to the default deployment, so it can be "sparse", not listing any - containers or volumes that should remain unchanged. - See https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates - type: object - x-kubernetes-map-type: atomic - x-kubernetes-preserve-unknown-fields: true - replicas: - description: Number of desired pods. Defaults to 1. - format: int32 - minimum: 0 - type: integer - type: object - apiTLS: - description: |- - ApiTLS secures the LINSTOR API. - - This configures the TLS key and certificate used to secure the LINSTOR API. - nullable: true - properties: - affinityControllerSecretName: - description: |- - AffinityControllerSecretName references a secret holding the TLS key and certificate used by the Affinity - Controller to monitor volume state. Defaults to "linstor-affinity-controller-tls". - type: string - apiSecretName: - description: |- - ApiSecretName references a secret holding the TLS key and certificate used to protect the API. - Defaults to "linstor-api-tls". - type: string - caReference: - description: |- - CAReference configures the CA certificate to use when validating TLS certificates. - If not set, the TLS secret is expected to contain a "ca.crt" containing the CA certificate. - properties: - key: - default: ca.crt - description: |- - Key to select in the resource. - Defaults to ca.crt if not specified. - type: string - kind: - default: Secret - description: Kind of the resource containing the CA Certificate, - either a ConfigMap or Secret. - enum: - - ConfigMap - - Secret - type: string - name: - description: Name of the resource containing the CA Certificate. - type: string - optional: - description: Optional specifies whether the resource and its - key must exist. - type: boolean - required: - - name - type: object - certManager: - description: |- - CertManager references a cert-manager Issuer or ClusterIssuer. - If set, cert-manager.io/Certificate resources will be created, provisioning the secrets referenced in - *SecretName using the issuer configured here. - properties: - group: - description: Group of the resource being referred to. - type: string - kind: - description: Kind of the resource being referred to. - type: string - name: - description: Name of the resource being referred to. - type: string - required: - - name - type: object - clientSecretName: - description: |- - ClientSecretName references a secret holding the TLS key and certificate used by the operator to configure - the cluster. Defaults to "linstor-client-tls". - type: string - csiControllerSecretName: - description: |- - CsiControllerSecretName references a secret holding the TLS key and certificate used by the CSI Controller - to provision volumes. Defaults to "linstor-csi-controller-tls". - type: string - csiNodeSecretName: - description: |- - CsiNodeSecretName references a secret holding the TLS key and certificate used by the CSI Nodes to query - the volume state. Defaults to "linstor-csi-node-tls". - type: string - nfsServerSecretName: - description: |- - NFSServerSecretName references a secret holding the TLS key and certificate used by the NFS Server to query - the cluster state. Defaults to "linstor-csi-nfs-server-tls". - type: string - type: object - controller: - description: Controller controls the deployment of the LINSTOR Controller - Deployment. - properties: - enabled: - default: true - description: Enable the component. - type: boolean - podTemplate: - description: |- - Template to apply to Pods of the component. - - The template is applied as a patch to the default deployment, so it can be "sparse", not listing any - containers or volumes that should remain unchanged. - See https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates - type: object - x-kubernetes-map-type: atomic - x-kubernetes-preserve-unknown-fields: true - type: object - csiController: - description: CSIController controls the deployment of the CSI Controller - Deployment. - properties: - enabled: - default: true - description: Enable the component. - type: boolean - podTemplate: - description: |- - Template to apply to Pods of the component. - - The template is applied as a patch to the default deployment, so it can be "sparse", not listing any - containers or volumes that should remain unchanged. - See https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates - type: object - x-kubernetes-map-type: atomic - x-kubernetes-preserve-unknown-fields: true - replicas: - description: Number of desired pods. Defaults to 1. - format: int32 - minimum: 0 - type: integer - type: object - csiNode: - description: CSINode controls the deployment of the CSI Node DaemonSet. - properties: - enabled: - default: true - description: Enable the component. - type: boolean - podTemplate: - description: |- - Template to apply to Pods of the component. - - The template is applied as a patch to the default deployment, so it can be "sparse", not listing any - containers or volumes that should remain unchanged. - See https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates - type: object - x-kubernetes-map-type: atomic - x-kubernetes-preserve-unknown-fields: true - type: object - externalController: - description: |- - ExternalController references an external controller. - When set, the Operator will skip deploying a LINSTOR Controller and instead use the external cluster - to register satellites. - properties: - url: - description: URL of the external controller. - minLength: 3 - type: string - required: - - url - type: object - highAvailabilityController: - description: HighAvailabilityController controls the deployment of - the High Availability Controller DaemonSet. - properties: - enabled: - default: true - description: Enable the component. - type: boolean - podTemplate: - description: |- - Template to apply to Pods of the component. - - The template is applied as a patch to the default deployment, so it can be "sparse", not listing any - containers or volumes that should remain unchanged. - See https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates - type: object - x-kubernetes-map-type: atomic - x-kubernetes-preserve-unknown-fields: true - type: object - internalTLS: - description: |- - InternalTLS secures the connection between LINSTOR Controller and Satellite. - - This configures the client certificate used when the Controller connects to a Satellite. This only has an effect - when the Satellite is configured to for secure connections using `LinstorSatellite.spec.internalTLS`. - nullable: true - properties: - caReference: - description: |- - CAReference configures the CA certificate to use when validating TLS certificates. - If not set, the TLS secret is expected to contain a "ca.crt" containing the CA certificate. - properties: - key: - default: ca.crt - description: |- - Key to select in the resource. - Defaults to ca.crt if not specified. - type: string - kind: - default: Secret - description: Kind of the resource containing the CA Certificate, - either a ConfigMap or Secret. - enum: - - ConfigMap - - Secret - type: string - name: - description: Name of the resource containing the CA Certificate. - type: string - optional: - description: Optional specifies whether the resource and its - key must exist. - type: boolean - required: - - name - type: object - certManager: - description: |- - CertManager references a cert-manager Issuer or ClusterIssuer. - If set, a Certificate resource will be created, provisioning the secret references in SecretName using the - issuer configured here. - properties: - group: - description: Group of the resource being referred to. - type: string - kind: - description: Kind of the resource being referred to. - type: string - name: - description: Name of the resource being referred to. - type: string - required: - - name - type: object - secretName: - description: SecretName references a secret holding the TLS key - and certificates. - type: string - type: object - linstorPassphraseSecret: - description: |- - LinstorPassphraseSecret used to configure the LINSTOR master passphrase. - - The referenced secret must contain a single key "MASTER_PASSPHRASE". The master passphrase is used to - * Derive encryption keys for volumes using the LUKS layer. - * Store credentials for accessing remotes for backups. - See https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-encrypt_commands for more information. - type: string - nfsServer: - description: NFSServer controls the deployment of the LINSTOR CSI - NFS Server DaemonSet. - properties: - enabled: - default: true - description: Enable the component. - type: boolean - podTemplate: - description: |- - Template to apply to Pods of the component. - - The template is applied as a patch to the default deployment, so it can be "sparse", not listing any - containers or volumes that should remain unchanged. - See https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates - type: object - x-kubernetes-map-type: atomic - x-kubernetes-preserve-unknown-fields: true - type: object - nodeAffinity: - description: |- - NodeAffinity selects the nodes on which LINSTOR Satellites will be deployed. - See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ - properties: - nodeSelectorTerms: - description: Required. A list of node selector terms. The terms - are ORed. - items: - description: |- - A null or empty node selector term matches no objects. The requirements of - them are ANDed. - The TopologySelectorTerm type implements a subset of the NodeSelectorTerm. - properties: - matchExpressions: - description: A list of node selector requirements by node's - labels. - items: - description: |- - A node selector requirement is a selector that contains values, a key, and an operator - that relates the key and values. - properties: - key: - description: The label key that the selector applies - to. - type: string - operator: - description: |- - Represents a key's relationship to a set of values. - Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. - type: string - values: - description: |- - An array of string values. If the operator is In or NotIn, - the values array must be non-empty. If the operator is Exists or DoesNotExist, - the values array must be empty. If the operator is Gt or Lt, the values - array must have a single element, which will be interpreted as an integer. - This array is replaced during a strategic merge patch. - items: - type: string - type: array - x-kubernetes-list-type: atomic - required: - - key - - operator - type: object - type: array - x-kubernetes-list-type: atomic - matchFields: - description: A list of node selector requirements by node's - fields. - items: - description: |- - A node selector requirement is a selector that contains values, a key, and an operator - that relates the key and values. - properties: - key: - description: The label key that the selector applies - to. - type: string - operator: - description: |- - Represents a key's relationship to a set of values. - Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. - type: string - values: - description: |- - An array of string values. If the operator is In or NotIn, - the values array must be non-empty. If the operator is Exists or DoesNotExist, - the values array must be empty. If the operator is Gt or Lt, the values - array must have a single element, which will be interpreted as an integer. - This array is replaced during a strategic merge patch. - items: - type: string - type: array - x-kubernetes-list-type: atomic - required: - - key - - operator - type: object - type: array - x-kubernetes-list-type: atomic - type: object - x-kubernetes-map-type: atomic - type: array - x-kubernetes-list-type: atomic - required: - - nodeSelectorTerms - type: object - x-kubernetes-map-type: atomic - nodeSelector: - additionalProperties: - type: string - description: |- - NodeSelector selects the nodes on which LINSTOR Satellites will be deployed. - See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ - type: object - patches: - description: |- - Patches is a list of kustomize patches to apply. - - See https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/patches/ for how to create patches. - items: - description: Patch represent either a Strategic Merge Patch or a - JSON patch and its targets. - properties: - options: - description: Options is a list of options for the patch - properties: - allowKindChange: - description: AllowKindChange allows kind changes to the - resource. - type: boolean - allowNameChange: - description: AllowNameChange allows name changes to the - resource. - type: boolean - type: object - patch: - description: Patch is the content of a patch. - minLength: 1 - type: string - target: - description: Target points to the resources that the patch is - applied to - properties: - annotationSelector: - description: |- - AnnotationSelector is a string that follows the label selection expression - https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#api - It matches against the resource annotations. - type: string - group: - type: string - kind: - type: string - labelSelector: - description: |- - LabelSelector is a string that follows the label selection expression - https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#api - It matches against the resource labels. - type: string - name: - description: Name of the resource. - type: string - namespace: - description: Namespace the resource belongs to, if it can - belong to a namespace. - type: string - version: - type: string - type: object - required: - - patch - type: object - type: array - properties: - description: |- - Properties to apply on the cluster level. - - Use to create default settings for DRBD that should apply to all resources or to configure some other cluster - wide default. - items: - properties: - name: - description: Name of the property to set. - minLength: 1 - type: string - value: - description: Value to set the property to. - type: string - required: - - name - type: object - type: array - x-kubernetes-list-map-keys: - - name - x-kubernetes-list-type: map - repository: - description: Repository used to pull workload images. - type: string - tolerations: - description: |- - Tolerations selects the nodes on which LINSTOR Satellites will be deployed. - - The default tolerations for DaemonSets are automatically added. - items: - description: |- - The pod this Toleration is attached to tolerates any taint that matches - the triple using the matching operator . - properties: - effect: - description: |- - Effect indicates the taint effect to match. Empty means match all taint effects. - When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. - type: string - key: - description: |- - Key is the taint key that the toleration applies to. Empty means match all taint keys. - If the key is empty, operator must be Exists; this combination means to match all values and all keys. - type: string - operator: - description: |- - Operator represents a key's relationship to the value. - Valid operators are Exists and Equal. Defaults to Equal. - Exists is equivalent to wildcard for value, so that a pod can - tolerate all taints of a particular category. - type: string - tolerationSeconds: - description: |- - TolerationSeconds represents the period of time the toleration (which must be - of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, - it is not set, which means tolerate the taint forever (do not evict). Zero and - negative values will be treated as 0 (evict immediately) by the system. - format: int64 - type: integer - value: - description: |- - Value is the taint value the toleration matches to. - If the operator is Exists, the value should be empty, otherwise just a regular string. - type: string - type: object - type: array - x-kubernetes-list-type: atomic - type: object - status: - description: LinstorClusterStatus defines the observed state of LinstorCluster - properties: - availableCapacityBytes: - description: The number of bytes in total in all storage pools in - the LINSTOR Cluster. - format: int64 - type: integer - capacity: - description: Capacity mirrors the information from TotalCapacityBytes - and FreeCapacityBytes in a human-readable string - type: string - conditions: - description: Current LINSTOR Cluster state - items: - description: Condition contains details for one aspect of the current - state of this API Resource. - properties: - lastTransitionTime: - description: |- - lastTransitionTime is the last time the condition transitioned from one status to another. - This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. - format: date-time - type: string - message: - description: |- - message is a human readable message indicating details about the transition. - This may be an empty string. - maxLength: 32768 - type: string - observedGeneration: - description: |- - observedGeneration represents the .metadata.generation that the condition was set based upon. - For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date - with respect to the current state of the instance. - format: int64 - minimum: 0 - type: integer - reason: - description: |- - reason contains a programmatic identifier indicating the reason for the condition's last transition. - Producers of specific condition types may define expected values and meanings for this field, - and whether the values are considered a guaranteed API. - The value should be a CamelCase string. - This field may not be empty. - maxLength: 1024 - minLength: 1 - pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ - type: string - status: - description: status of the condition, one of True, False, Unknown. - enum: - - "True" - - "False" - - Unknown - type: string - type: - description: type of condition in CamelCase or in foo.example.com/CamelCase. - maxLength: 316 - pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ - type: string - required: - - lastTransitionTime - - message - - reason - - status - - type - type: object - type: array - x-kubernetes-list-map-keys: - - type - x-kubernetes-list-type: map - freeCapacityBytes: - description: The number of bytes free in all storage pools in the - LINSTOR Cluster. - format: int64 - type: integer - numberOfSnapshots: - description: The number of snapshots in the LINSTOR Cluster. - format: int32 - type: integer - numberOfVolumes: - description: The number of volumes in the LINSTOR Cluster. - format: int32 - type: integer - runningSatellites: - description: The number of LINSTOR Satellites currently running. - format: int32 - type: integer - satellites: - description: Satellites mirrors the information from ScheduledSatellites - and RunningSatellites in a human-readable string - type: string - scheduledSatellites: - description: The number of LINSTOR Satellites that are expected to - run. - format: int32 - type: integer - version: - description: The Version of the LINSTOR Cluster. - type: string - type: object - type: object - served: true - storage: true - subresources: - status: {} ---- -apiVersion: apiextensions.k8s.io/v1 -kind: CustomResourceDefinition -metadata: - annotations: - controller-gen.kubebuilder.io/version: v0.18.0 - name: linstornodeconnections.piraeus.io -spec: - group: piraeus.io - names: - kind: LinstorNodeConnection - listKind: LinstorNodeConnectionList - plural: linstornodeconnections - singular: linstornodeconnection - scope: Cluster - versions: - - additionalPrinterColumns: - - description: If the LINSTOR Node Connection is fully configured - jsonPath: .status.conditions[?(@.type=='Configured')].status - name: Configured - type: string - - jsonPath: .metadata.creationTimestamp - name: Age - type: date - name: v1 - schema: - openAPIV3Schema: - description: LinstorNodeConnection is the Schema for the linstornodeconnections - API - properties: - apiVersion: - description: |- - APIVersion defines the versioned schema of this representation of an object. - Servers should convert recognized schemas to the latest internal value, and - may reject unrecognized values. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - type: string - kind: - description: |- - Kind is a string value representing the REST resource this object represents. - Servers may infer this from the endpoint the client submits requests to. - Cannot be updated. - In CamelCase. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - type: string - metadata: - type: object - spec: - description: LinstorNodeConnectionSpec defines the desired state of LinstorNodeConnection - properties: - paths: - description: Paths configure the network path used when connecting - two nodes. - items: - properties: - interface: - description: Interface to use on both nodes. - type: string - name: - description: Name of the path. - type: string - required: - - interface - - name - type: object - type: array - x-kubernetes-list-map-keys: - - name - x-kubernetes-list-type: map - properties: - description: |- - Properties to apply for the node connection. - - Use to create default settings for DRBD that should apply to all resources connections between a set of - cluster nodes. - items: - properties: - name: - description: Name of the property to set. - minLength: 1 - type: string - value: - description: Value to set the property to. - type: string - required: - - name - type: object - type: array - x-kubernetes-list-map-keys: - - name - x-kubernetes-list-type: map - selector: - description: |- - Selector selects which pair of Satellites the connection should apply to. - If not given, the connection will be applied to all connections. - items: - description: SelectorTerm matches pairs of nodes by checking that - the nodes match all specified requirements. - properties: - matchLabels: - description: MatchLabels is a list of match expressions that - the node pairs must meet. - items: - properties: - key: - description: Key is the name of a node label. - minLength: 1 - type: string - op: - default: Exists - description: |- - Op to apply to the label. - Exists (default) checks for the presence of the label on both nodes in the pair. - DoesNotExist checks that the label is not present on either node in the pair. - In checks for the presence of the label value given by Values on both nodes in the pair. - NotIn checks that both nodes in the pair do not have any of the label values given by Values. - Same checks that the label value is equal in the node pair. - NotSame checks that the label value is not equal in the node pair. - enum: - - Exists - - DoesNotExist - - In - - NotIn - - Same - - NotSame - type: string - values: - description: Values to match on, using the provided Op. - items: - type: string - type: array - required: - - key - type: object - type: array - required: - - matchLabels - type: object - type: array - type: object - status: - description: LinstorNodeConnectionStatus defines the observed state of - LinstorNodeConnection - properties: - conditions: - description: Current LINSTOR Node Connection state - items: - description: Condition contains details for one aspect of the current - state of this API Resource. - properties: - lastTransitionTime: - description: |- - lastTransitionTime is the last time the condition transitioned from one status to another. - This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. - format: date-time - type: string - message: - description: |- - message is a human readable message indicating details about the transition. - This may be an empty string. - maxLength: 32768 - type: string - observedGeneration: - description: |- - observedGeneration represents the .metadata.generation that the condition was set based upon. - For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date - with respect to the current state of the instance. - format: int64 - minimum: 0 - type: integer - reason: - description: |- - reason contains a programmatic identifier indicating the reason for the condition's last transition. - Producers of specific condition types may define expected values and meanings for this field, - and whether the values are considered a guaranteed API. - The value should be a CamelCase string. - This field may not be empty. - maxLength: 1024 - minLength: 1 - pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ - type: string - status: - description: status of the condition, one of True, False, Unknown. - enum: - - "True" - - "False" - - Unknown - type: string - type: - description: type of condition in CamelCase or in foo.example.com/CamelCase. - maxLength: 316 - pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ - type: string - required: - - lastTransitionTime - - message - - reason - - status - - type - type: object - type: array - x-kubernetes-list-map-keys: - - type - x-kubernetes-list-type: map - type: object - type: object - served: true - storage: true - subresources: - status: {} ---- -apiVersion: apiextensions.k8s.io/v1 -kind: CustomResourceDefinition -metadata: - annotations: - controller-gen.kubebuilder.io/version: v0.18.0 - name: linstorsatelliteconfigurations.piraeus.io -spec: - group: piraeus.io - names: - kind: LinstorSatelliteConfiguration - listKind: LinstorSatelliteConfigurationList - plural: linstorsatelliteconfigurations - singular: linstorsatelliteconfiguration - scope: Cluster - versions: - - additionalPrinterColumns: - - description: The node selector used - jsonPath: .spec.nodeSelector - name: Selector - type: string - - description: If the Configuration was applied - jsonPath: .status.conditions[?(@.type=='Applied')].status - name: Applied - type: string - - description: Number of Satellites this Configuration has been applied to - jsonPath: .status.matched - name: Matched - type: integer - - description: Satellites this Configuration has been applied to - jsonPath: .status.appliedTo - name: Satellites - priority: 10 - type: string - - jsonPath: .metadata.creationTimestamp - name: Age - type: date - name: v1 - schema: - openAPIV3Schema: - description: LinstorSatelliteConfiguration is the Schema for the linstorsatelliteconfigurations - API - properties: - apiVersion: - description: |- - APIVersion defines the versioned schema of this representation of an object. - Servers should convert recognized schemas to the latest internal value, and - may reject unrecognized values. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - type: string - kind: - description: |- - Kind is a string value representing the REST resource this object represents. - Servers may infer this from the endpoint the client submits requests to. - Cannot be updated. - In CamelCase. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - type: string - metadata: - type: object - spec: - description: |- - LinstorSatelliteConfigurationSpec defines a partial, desired state of a LinstorSatelliteSpec. - - All the LinstorSatelliteConfiguration resources with matching NodeSelector will - be merged into a single LinstorSatelliteSpec. - properties: - deletionPolicy: - description: |- - DeletionPolicy configures the way LinstorSatellite resources are deleted. - - A LinstorSatellite may be deleted because: - * It no longer matches the affinity and node selector of the LinstorCluster resource. - * The node it references has been removed from Kubernetes. - * It was manually deleted outside the Operator. - - A LinstorSatellite may store the last copy of a volume, in which case it is not desirable to unconditionally remove - the satellite from the cluster. For this reason, the following deletion policies exist: - - * DeletionPolicyEvacuate will start evacuation of the LINSTOR Satellite and wait until it completes before removing the LinstorSatellite object, comparable to the "linstor node evacuate" command. - * DeletionPolicyRetain will retain the LINSTOR Satellite, keeping it registered in LINSTOR, but removing associated Kubernetes resources. - * DeletionPolicyDelete will remove the LINSTOR Satellite from the LINSTOR Cluster without prior eviction, comparable to the "linstor node lost" command. - enum: - - Evacuate - - Retain - - Delete - type: string - evacuationStrategy: - description: EvacuationStrategy configures the evacuation of volumes - from a Satellite when DeletionPolicy "Evacuate" is used. - nullable: true - properties: - attachedVolumeReattachTimeout: - default: 5m - description: |- - AttachedVolumeReattachTimeout configures how long evacuation waits for attached volumes to reattach on - different nodes. Setting this to 0 disable this evacuation step. - type: string - unattachedVolumeAttachTimeout: - default: 5m - description: |- - UnattachedVolumeAttachTimeout configures how long evacuation waits for unattached volumes to attach on - different nodes. Setting this to 0 disable this evacuation step. - type: string - type: object - internalTLS: - description: |- - InternalTLS configures secure communication for the LINSTOR Satellite. - - If set, the control traffic between LINSTOR Controller and Satellite will be encrypted using mTLS. - nullable: true - properties: - caReference: - description: |- - CAReference configures the CA certificate to use when validating TLS certificates. - If not set, the TLS secret is expected to contain a "ca.crt" containing the CA certificate. - properties: - key: - default: ca.crt - description: |- - Key to select in the resource. - Defaults to ca.crt if not specified. - type: string - kind: - default: Secret - description: Kind of the resource containing the CA Certificate, - either a ConfigMap or Secret. - enum: - - ConfigMap - - Secret - type: string - name: - description: Name of the resource containing the CA Certificate. - type: string - optional: - description: Optional specifies whether the resource and its - key must exist. - type: boolean - required: - - name - type: object - certManager: - description: |- - CertManager references a cert-manager Issuer or ClusterIssuer. - If set, a Certificate resource will be created, provisioning the secret references in SecretName using the - issuer configured here. - properties: - group: - description: Group of the resource being referred to. - type: string - kind: - description: Kind of the resource being referred to. - type: string - name: - description: Name of the resource being referred to. - type: string - required: - - name - type: object - secretName: - description: SecretName references a secret holding the TLS key - and certificates. - type: string - tlsHandshakeDaemon: - description: |- - TLSHandshakeDaemon enables tlshd for establishing TLS sessions for use by DRBD. - - If enabled, adds a new sidecar to the LINSTOR Satellite that runs the tlshd handshake daemon. - The daemon uses the TLS certificate and key to establish secure connections on behalf of DRBD. - type: boolean - type: object - ipFamilies: - description: |- - IPFamilies configures the IP Family (IPv4 or IPv6) to use to connect to the LINSTOR Satellite. - - If set, the control traffic between LINSTOR Controller and Satellite will use only the given IP Family. - If not set, the Operator will configure all families found in the Satellites Pods' Status. - items: - description: IPFamily represents the IP Family (IPv4 or IPv6). - enum: - - IPv4 - - IPv6 - type: string - type: array - nodeAffinity: - description: |- - NodeAffinity selects which LinstorSatellite resources this spec should be applied to. - See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ - properties: - nodeSelectorTerms: - description: Required. A list of node selector terms. The terms - are ORed. - items: - description: |- - A null or empty node selector term matches no objects. The requirements of - them are ANDed. - The TopologySelectorTerm type implements a subset of the NodeSelectorTerm. - properties: - matchExpressions: - description: A list of node selector requirements by node's - labels. - items: - description: |- - A node selector requirement is a selector that contains values, a key, and an operator - that relates the key and values. - properties: - key: - description: The label key that the selector applies - to. - type: string - operator: - description: |- - Represents a key's relationship to a set of values. - Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. - type: string - values: - description: |- - An array of string values. If the operator is In or NotIn, - the values array must be non-empty. If the operator is Exists or DoesNotExist, - the values array must be empty. If the operator is Gt or Lt, the values - array must have a single element, which will be interpreted as an integer. - This array is replaced during a strategic merge patch. - items: - type: string - type: array - x-kubernetes-list-type: atomic - required: - - key - - operator - type: object - type: array - x-kubernetes-list-type: atomic - matchFields: - description: A list of node selector requirements by node's - fields. - items: - description: |- - A node selector requirement is a selector that contains values, a key, and an operator - that relates the key and values. - properties: - key: - description: The label key that the selector applies - to. - type: string - operator: - description: |- - Represents a key's relationship to a set of values. - Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. - type: string - values: - description: |- - An array of string values. If the operator is In or NotIn, - the values array must be non-empty. If the operator is Exists or DoesNotExist, - the values array must be empty. If the operator is Gt or Lt, the values - array must have a single element, which will be interpreted as an integer. - This array is replaced during a strategic merge patch. - items: - type: string - type: array - x-kubernetes-list-type: atomic - required: - - key - - operator - type: object - type: array - x-kubernetes-list-type: atomic - type: object - x-kubernetes-map-type: atomic - type: array - x-kubernetes-list-type: atomic - required: - - nodeSelectorTerms - type: object - x-kubernetes-map-type: atomic - nodeSelector: - additionalProperties: - type: string - description: |- - NodeSelector selects which LinstorSatellite resources this spec should be applied to. - See https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ - type: object - patches: - description: |- - Patches is a list of kustomize patches to apply. - - See https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/patches/ for how to create patches. - items: - description: Patch represent either a Strategic Merge Patch or a - JSON patch and its targets. - properties: - options: - description: Options is a list of options for the patch - properties: - allowKindChange: - description: AllowKindChange allows kind changes to the - resource. - type: boolean - allowNameChange: - description: AllowNameChange allows name changes to the - resource. - type: boolean - type: object - patch: - description: Patch is the content of a patch. - minLength: 1 - type: string - target: - description: Target points to the resources that the patch is - applied to - properties: - annotationSelector: - description: |- - AnnotationSelector is a string that follows the label selection expression - https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#api - It matches against the resource annotations. - type: string - group: - type: string - kind: - type: string - labelSelector: - description: |- - LabelSelector is a string that follows the label selection expression - https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#api - It matches against the resource labels. - type: string - name: - description: Name of the resource. - type: string - namespace: - description: Namespace the resource belongs to, if it can - belong to a namespace. - type: string - version: - type: string - type: object - required: - - patch - type: object - type: array - podTemplate: - description: |- - Template to apply to Satellite Pods. - - The template is applied as a patch to the default resource, so it can be "sparse", not listing any - containers or volumes that should remain unchanged. - See https://kubernetes.io/docs/concepts/workloads/pods/#pod-templates - type: object - x-kubernetes-map-type: atomic - x-kubernetes-preserve-unknown-fields: true - properties: - description: Properties is a list of properties to set on the node. - items: - properties: - expandFrom: - description: |- - ExpandFrom can reference multiple resource fields at once. - It either sets the property to an aggregate value based on matched resource fields, or expands to multiple - properties. - properties: - delimiter: - description: Delimiter used to join multiple key and value - pairs together. - type: string - nameTemplate: - description: |- - NameTemplate defines how the property key is expanded. - If set, the template is appended to the defined property name, creating multiple properties instead of one - aggregate. - * $1 is replaced with the matched key. - * $2 is replaced with the matched value. - type: string - nodeFieldRef: - description: Select a field of the node. Supports `metadata.name`, - `metadata.labels['']`, `metadata.annotations['']`. - minLength: 1 - type: string - valueTemplate: - description: |- - ValueTemplate defines how the property value is expanded. - * $1 is replaced with the matched key. - * $2 is replaced with the matched value. - type: string - required: - - nodeFieldRef - type: object - name: - description: Name of the property to set. - minLength: 1 - type: string - optional: - description: Optional values are only set if they have a non-empty - value - type: boolean - value: - description: Value to set the property to. - type: string - valueFrom: - description: ValueFrom sets the value from an existing resource. - properties: - nodeFieldRef: - description: Select a field of the node. Supports `metadata.name`, - `metadata.labels['']`, `metadata.annotations['']`. - minLength: 1 - type: string - required: - - nodeFieldRef - type: object - required: - - name - type: object - type: array - x-kubernetes-list-map-keys: - - name - x-kubernetes-list-type: map - storagePools: - description: StoragePools is a list of storage pools to configure - on the node. - items: - properties: - filePool: - description: Configures a file system based storage pool, allocating - a regular file per volume. - properties: - directory: - description: Directory is the path to the host directory - used to store volume data. - type: string - type: object - fileThinPool: - description: Configures a file system based storage pool, allocating - a sparse file per volume. - properties: - directory: - description: Directory is the path to the host directory - used to store volume data. - type: string - type: object - lvmPool: - description: Configures a LVM Volume Group as storage pool. - properties: - volumeGroup: - type: string - type: object - lvmThinPool: - description: Configures a LVM Thin Pool as storage pool. - properties: - thinPool: - description: ThinPool is the name of the thinpool LV (without - VG prefix). - type: string - volumeGroup: - type: string - type: object - name: - description: Name of the storage pool in linstor. - minLength: 3 - type: string - properties: - description: Properties to set on the storage pool. - items: - properties: - expandFrom: - description: |- - ExpandFrom can reference multiple resource fields at once. - It either sets the property to an aggregate value based on matched resource fields, or expands to multiple - properties. - properties: - delimiter: - description: Delimiter used to join multiple key and - value pairs together. - type: string - nameTemplate: - description: |- - NameTemplate defines how the property key is expanded. - If set, the template is appended to the defined property name, creating multiple properties instead of one - aggregate. - * $1 is replaced with the matched key. - * $2 is replaced with the matched value. - type: string - nodeFieldRef: - description: Select a field of the node. Supports - `metadata.name`, `metadata.labels['']`, `metadata.annotations['']`. - minLength: 1 - type: string - valueTemplate: - description: |- - ValueTemplate defines how the property value is expanded. - * $1 is replaced with the matched key. - * $2 is replaced with the matched value. - type: string - required: - - nodeFieldRef - type: object - name: - description: Name of the property to set. - minLength: 1 - type: string - optional: - description: Optional values are only set if they have - a non-empty value - type: boolean - value: - description: Value to set the property to. - type: string - valueFrom: - description: ValueFrom sets the value from an existing - resource. - properties: - nodeFieldRef: - description: Select a field of the node. Supports - `metadata.name`, `metadata.labels['']`, `metadata.annotations['']`. - minLength: 1 - type: string - required: - - nodeFieldRef - type: object - required: - - name - type: object - type: array - x-kubernetes-list-map-keys: - - name - x-kubernetes-list-type: map - source: - properties: - hostDevices: - description: HostDevices is a list of device paths used - to configure the given pool. - items: - type: string - minItems: 1 - type: array - type: object - zfsPool: - description: Configures a ZFS system based storage pool, allocating - zvols from the given zpool. - properties: - zPool: - description: ZPool is the name of the ZFS zpool. - type: string - type: object - zfsThinPool: - description: Configures a ZFS system based storage pool, allocating - sparse zvols from the given zpool. - properties: - zPool: - description: ZPool is the name of the ZFS zpool. - type: string - type: object - required: - - name - type: object - type: array - type: object - status: - description: LinstorSatelliteConfigurationStatus defines the observed - state of LinstorSatelliteConfiguration - properties: - appliedTo: - description: AppliedTo lists the LinstorSatellite resource this configuration - was applied to - items: - type: string - type: array - conditions: - description: Current LINSTOR Satellite Config state - items: - description: Condition contains details for one aspect of the current - state of this API Resource. - properties: - lastTransitionTime: - description: |- - lastTransitionTime is the last time the condition transitioned from one status to another. - This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. - format: date-time - type: string - message: - description: |- - message is a human readable message indicating details about the transition. - This may be an empty string. - maxLength: 32768 - type: string - observedGeneration: - description: |- - observedGeneration represents the .metadata.generation that the condition was set based upon. - For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date - with respect to the current state of the instance. - format: int64 - minimum: 0 - type: integer - reason: - description: |- - reason contains a programmatic identifier indicating the reason for the condition's last transition. - Producers of specific condition types may define expected values and meanings for this field, - and whether the values are considered a guaranteed API. - The value should be a CamelCase string. - This field may not be empty. - maxLength: 1024 - minLength: 1 - pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ - type: string - status: - description: status of the condition, one of True, False, Unknown. - enum: - - "True" - - "False" - - Unknown - type: string - type: - description: type of condition in CamelCase or in foo.example.com/CamelCase. - maxLength: 316 - pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ - type: string - required: - - lastTransitionTime - - message - - reason - - status - - type - type: object - type: array - x-kubernetes-list-map-keys: - - type - x-kubernetes-list-type: map - matched: - description: Number of configured LinstorSatellite resource. - format: int64 - type: integer - type: object - type: object - served: true - storage: true - subresources: - status: {} ---- -apiVersion: apiextensions.k8s.io/v1 -kind: CustomResourceDefinition -metadata: - annotations: - controller-gen.kubebuilder.io/version: v0.18.0 - name: linstorsatellites.piraeus.io -spec: - group: piraeus.io - names: - kind: LinstorSatellite - listKind: LinstorSatelliteList - plural: linstorsatellites - singular: linstorsatellite - scope: Cluster - versions: - - additionalPrinterColumns: - - description: If the LINSTOR Satellite is connected - jsonPath: .status.conditions[?(@.type=='Available')].status - name: Connected - type: string - - description: If the LINSTOR Satellite is fully configured - jsonPath: .status.conditions[?(@.type=='Configured')].status - name: Configured - type: string - - description: The Satellite Configurations applied to this Satellite - jsonPath: .metadata.annotations.piraeus\.io/applied-configurations - name: Applied Configurations - priority: 10 - type: string - - description: The deletion policy of the Satellite - jsonPath: .spec.deletionPolicy - name: Deletion Policy - type: string - - description: The used capacity on the node - jsonPath: .status.capacity - name: Used Capacity - type: string - - description: The number of volumes on the node - jsonPath: .status.numberOfVolumes - name: Volumes - type: integer - - description: The number of snapshots on the node - jsonPath: .status.numberOfSnapshots - name: Snapshots - priority: 10 - type: integer - - description: The storage providers supported by the node - jsonPath: .status.storageProviders - name: Storage Providers - priority: 10 - type: string - - description: The device layers supported by the node - jsonPath: .status.deviceLayers - name: Device Layers - priority: 10 - type: string - - jsonPath: .metadata.creationTimestamp - name: Age - type: date - name: v1 - schema: - openAPIV3Schema: - description: LinstorSatellite is the Schema for the linstorsatellites API - properties: - apiVersion: - description: |- - APIVersion defines the versioned schema of this representation of an object. - Servers should convert recognized schemas to the latest internal value, and - may reject unrecognized values. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources - type: string - kind: - description: |- - Kind is a string value representing the REST resource this object represents. - Servers may infer this from the endpoint the client submits requests to. - Cannot be updated. - In CamelCase. - More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds - type: string - metadata: - type: object - spec: - description: LinstorSatelliteSpec defines the desired state of LinstorSatellite - properties: - clusterRef: - description: ClusterRef references the LinstorCluster used to create - this LinstorSatellite. - properties: - caReference: - description: |- - CAReference configures the CA certificate to use when validating TLS certificates. - If not set, the TLS secret is expected to contain a "ca.crt" containing the CA certificate. - properties: - key: - default: ca.crt - description: |- - Key to select in the resource. - Defaults to ca.crt if not specified. - type: string - kind: - default: Secret - description: Kind of the resource containing the CA Certificate, - either a ConfigMap or Secret. - enum: - - ConfigMap - - Secret - type: string - name: - description: Name of the resource containing the CA Certificate. - type: string - optional: - description: Optional specifies whether the resource and its - key must exist. - type: boolean - required: - - name - type: object - clientSecretName: - description: ClientSecretName references the secret used by the - operator to validate the https endpoint. - type: string - externalController: - description: |- - ExternalController references an external controller. - When set, the Operator uses the external cluster to register satellites. - properties: - url: - description: URL of the external controller. - minLength: 3 - type: string - required: - - url - type: object - name: - description: Name of the LinstorCluster resource controlling this - satellite. - type: string - type: object - deletionPolicy: - default: Retain - description: |- - DeletionPolicy configures the way LinstorSatellite resources are deleted. - - A LinstorSatellite may be deleted because: - * It no longer matches the affinity and node selector of the LinstorCluster resource. - * The node it references has been removed from Kubernetes. - * It was manually deleted outside the Operator. - - A LinstorSatellite may store the last copy of a volume, in which case it is not desirable to unconditionally remove - the satellite from the cluster. For this reason, the following deletion policies exist: - - * DeletionPolicyEvacuate will start evacuation of the LINSTOR Satellite and wait until it completes before removing the LinstorSatellite object, comparable to the "linstor node evacuate" command. - * DeletionPolicyRetain will retain the LINSTOR Satellite, keeping it registered in LINSTOR, but removing associated Kubernetes resources. - * DeletionPolicyDelete will remove the LINSTOR Satellite from the LINSTOR Cluster without prior eviction, comparable to the "linstor node lost" command. - enum: - - Evacuate - - Retain - - Delete - type: string - evacuationStrategy: - description: EvacuationStrategy configures the evacuation of volumes - from a Satellite when DeletionPolicy "Evacuate" is used. - properties: - attachedVolumeReattachTimeout: - default: 5m - description: |- - AttachedVolumeReattachTimeout configures how long evacuation waits for attached volumes to reattach on - different nodes. Setting this to 0 disable this evacuation step. - type: string - unattachedVolumeAttachTimeout: - default: 5m - description: |- - UnattachedVolumeAttachTimeout configures how long evacuation waits for unattached volumes to attach on - different nodes. Setting this to 0 disable this evacuation step. - type: string - type: object - internalTLS: - description: |- - InternalTLS configures secure communication for the LINSTOR Satellite. - - If set, the control traffic between LINSTOR Controller and Satellite will be encrypted using mTLS. - The Controller will use the client key from `LinstorCluster.spec.internalTLS` when connecting. - nullable: true - properties: - caReference: - description: |- - CAReference configures the CA certificate to use when validating TLS certificates. - If not set, the TLS secret is expected to contain a "ca.crt" containing the CA certificate. - properties: - key: - default: ca.crt - description: |- - Key to select in the resource. - Defaults to ca.crt if not specified. - type: string - kind: - default: Secret - description: Kind of the resource containing the CA Certificate, - either a ConfigMap or Secret. - enum: - - ConfigMap - - Secret - type: string - name: - description: Name of the resource containing the CA Certificate. - type: string - optional: - description: Optional specifies whether the resource and its - key must exist. - type: boolean - required: - - name - type: object - certManager: - description: |- - CertManager references a cert-manager Issuer or ClusterIssuer. - If set, a Certificate resource will be created, provisioning the secret references in SecretName using the - issuer configured here. - properties: - group: - description: Group of the resource being referred to. - type: string - kind: - description: Kind of the resource being referred to. - type: string - name: - description: Name of the resource being referred to. - type: string - required: - - name - type: object - secretName: - description: SecretName references a secret holding the TLS key - and certificates. - type: string - tlsHandshakeDaemon: - description: |- - TLSHandshakeDaemon enables tlshd for establishing TLS sessions for use by DRBD. - - If enabled, adds a new sidecar to the LINSTOR Satellite that runs the tlshd handshake daemon. - The daemon uses the TLS certificate and key to establish secure connections on behalf of DRBD. - type: boolean - type: object - ipFamilies: - description: |- - IPFamilies configures the IP Family (IPv4 or IPv6) to use to connect to the LINSTOR Satellite. - - If set, the control traffic between LINSTOR Controller and Satellite will use only the given IP Family. - If not set, the Operator will configure all families found in the Satellites Pods' Status. - items: - description: IPFamily represents the IP Family (IPv4 or IPv6). - enum: - - IPv4 - - IPv6 - type: string - type: array - patches: - description: |- - Patches is a list of kustomize patches to apply. - - See https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/patches/ for how to create patches. - items: - description: Patch represent either a Strategic Merge Patch or a - JSON patch and its targets. - properties: - options: - description: Options is a list of options for the patch - properties: - allowKindChange: - description: AllowKindChange allows kind changes to the - resource. - type: boolean - allowNameChange: - description: AllowNameChange allows name changes to the - resource. - type: boolean - type: object - patch: - description: Patch is the content of a patch. - minLength: 1 - type: string - target: - description: Target points to the resources that the patch is - applied to - properties: - annotationSelector: - description: |- - AnnotationSelector is a string that follows the label selection expression - https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#api - It matches against the resource annotations. - type: string - group: - type: string - kind: - type: string - labelSelector: - description: |- - LabelSelector is a string that follows the label selection expression - https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#api - It matches against the resource labels. - type: string - name: - description: Name of the resource. - type: string - namespace: - description: Namespace the resource belongs to, if it can - belong to a namespace. - type: string - version: - type: string - type: object - required: - - patch - type: object - type: array - properties: - description: Properties is a list of properties to set on the node. - items: - properties: - expandFrom: - description: |- - ExpandFrom can reference multiple resource fields at once. - It either sets the property to an aggregate value based on matched resource fields, or expands to multiple - properties. - properties: - delimiter: - description: Delimiter used to join multiple key and value - pairs together. - type: string - nameTemplate: - description: |- - NameTemplate defines how the property key is expanded. - If set, the template is appended to the defined property name, creating multiple properties instead of one - aggregate. - * $1 is replaced with the matched key. - * $2 is replaced with the matched value. - type: string - nodeFieldRef: - description: Select a field of the node. Supports `metadata.name`, - `metadata.labels['']`, `metadata.annotations['']`. - minLength: 1 - type: string - valueTemplate: - description: |- - ValueTemplate defines how the property value is expanded. - * $1 is replaced with the matched key. - * $2 is replaced with the matched value. - type: string - required: - - nodeFieldRef - type: object - name: - description: Name of the property to set. - minLength: 1 - type: string - optional: - description: Optional values are only set if they have a non-empty - value - type: boolean - value: - description: Value to set the property to. - type: string - valueFrom: - description: ValueFrom sets the value from an existing resource. - properties: - nodeFieldRef: - description: Select a field of the node. Supports `metadata.name`, - `metadata.labels['']`, `metadata.annotations['']`. - minLength: 1 - type: string - required: - - nodeFieldRef - type: object - required: - - name - type: object - type: array - x-kubernetes-list-map-keys: - - name - x-kubernetes-list-type: map - repository: - description: Repository used to pull workload images. - type: string - storagePools: - description: StoragePools is a list of storage pools to configure - on the node. - items: - properties: - filePool: - description: Configures a file system based storage pool, allocating - a regular file per volume. - properties: - directory: - description: Directory is the path to the host directory - used to store volume data. - type: string - type: object - fileThinPool: - description: Configures a file system based storage pool, allocating - a sparse file per volume. - properties: - directory: - description: Directory is the path to the host directory - used to store volume data. - type: string - type: object - lvmPool: - description: Configures a LVM Volume Group as storage pool. - properties: - volumeGroup: - type: string - type: object - lvmThinPool: - description: Configures a LVM Thin Pool as storage pool. - properties: - thinPool: - description: ThinPool is the name of the thinpool LV (without - VG prefix). - type: string - volumeGroup: - type: string - type: object - name: - description: Name of the storage pool in linstor. - minLength: 3 - type: string - properties: - description: Properties to set on the storage pool. - items: - properties: - expandFrom: - description: |- - ExpandFrom can reference multiple resource fields at once. - It either sets the property to an aggregate value based on matched resource fields, or expands to multiple - properties. - properties: - delimiter: - description: Delimiter used to join multiple key and - value pairs together. - type: string - nameTemplate: - description: |- - NameTemplate defines how the property key is expanded. - If set, the template is appended to the defined property name, creating multiple properties instead of one - aggregate. - * $1 is replaced with the matched key. - * $2 is replaced with the matched value. - type: string - nodeFieldRef: - description: Select a field of the node. Supports - `metadata.name`, `metadata.labels['']`, `metadata.annotations['']`. - minLength: 1 - type: string - valueTemplate: - description: |- - ValueTemplate defines how the property value is expanded. - * $1 is replaced with the matched key. - * $2 is replaced with the matched value. - type: string - required: - - nodeFieldRef - type: object - name: - description: Name of the property to set. - minLength: 1 - type: string - optional: - description: Optional values are only set if they have - a non-empty value - type: boolean - value: - description: Value to set the property to. - type: string - valueFrom: - description: ValueFrom sets the value from an existing - resource. - properties: - nodeFieldRef: - description: Select a field of the node. Supports - `metadata.name`, `metadata.labels['']`, `metadata.annotations['']`. - minLength: 1 - type: string - required: - - nodeFieldRef - type: object - required: - - name - type: object - type: array - x-kubernetes-list-map-keys: - - name - x-kubernetes-list-type: map - source: - properties: - hostDevices: - description: HostDevices is a list of device paths used - to configure the given pool. - items: - type: string - minItems: 1 - type: array - type: object - zfsPool: - description: Configures a ZFS system based storage pool, allocating - zvols from the given zpool. - properties: - zPool: - description: ZPool is the name of the ZFS zpool. - type: string - type: object - zfsThinPool: - description: Configures a ZFS system based storage pool, allocating - sparse zvols from the given zpool. - properties: - zPool: - description: ZPool is the name of the ZFS zpool. - type: string - type: object - required: - - name - type: object - type: array - required: - - clusterRef - type: object - status: - description: LinstorSatelliteStatus defines the observed state of LinstorSatellite - properties: - availableCapacityBytes: - description: The number of bytes in total in all storage pools on - this Satellite. - format: int64 - type: integer - capacity: - description: Capacity mirrors the information from TotalCapacityBytes - and FreeCapacityBytes in a human-readable string. - type: string - conditions: - description: Current LINSTOR Satellite state - items: - description: Condition contains details for one aspect of the current - state of this API Resource. - properties: - lastTransitionTime: - description: |- - lastTransitionTime is the last time the condition transitioned from one status to another. - This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. - format: date-time - type: string - message: - description: |- - message is a human readable message indicating details about the transition. - This may be an empty string. - maxLength: 32768 - type: string - observedGeneration: - description: |- - observedGeneration represents the .metadata.generation that the condition was set based upon. - For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date - with respect to the current state of the instance. - format: int64 - minimum: 0 - type: integer - reason: - description: |- - reason contains a programmatic identifier indicating the reason for the condition's last transition. - Producers of specific condition types may define expected values and meanings for this field, - and whether the values are considered a guaranteed API. - The value should be a CamelCase string. - This field may not be empty. - maxLength: 1024 - minLength: 1 - pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ - type: string - status: - description: status of the condition, one of True, False, Unknown. - enum: - - "True" - - "False" - - Unknown - type: string - type: - description: type of condition in CamelCase or in foo.example.com/CamelCase. - maxLength: 316 - pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ - type: string - required: - - lastTransitionTime - - message - - reason - - status - - type - type: object - type: array - x-kubernetes-list-map-keys: - - type - x-kubernetes-list-type: map - deviceLayers: - description: DeviceLayers lists the device layers (LUKS, CACHE, etc...) - this Satellite supports. - items: - type: string - type: array - freeCapacityBytes: - description: The number of bytes free in all storage pools on this - Satellite. - format: int64 - type: integer - numberOfSnapshots: - description: The number of snapshots on this Satellite. - format: int32 - type: integer - numberOfVolumes: - description: The number of volumes on this Satellite. - format: int32 - type: integer - storageProviders: - description: StorageProviders lists the storage providers (LVM, ZFS, - etc...) this Satellite supports. - items: - type: string - type: array - type: object - type: object - served: true - storage: true - subresources: - status: {} -{{ end }} diff --git a/packages/system/piraeus-operator/charts/piraeus/templates/deployment.yaml b/packages/system/piraeus-operator/charts/piraeus/templates/deployment.yaml index c67501972..0c3209e19 100644 --- a/packages/system/piraeus-operator/charts/piraeus/templates/deployment.yaml +++ b/packages/system/piraeus-operator/charts/piraeus/templates/deployment.yaml @@ -3,6 +3,7 @@ apiVersion: apps/v1 kind: Deployment metadata: name: {{ include "piraeus-operator.fullname" . }}-controller-manager + namespace: {{ .Release.Namespace }} labels: {{- include "piraeus-operator.labels" . | nindent 4 }} spec: @@ -70,6 +71,9 @@ spec: - mountPath: /tmp/k8s-webhook-server/serving-certs name: cert readOnly: true + {{- with .Values.operator.extraVolumeMounts }} + {{- toYaml . | nindent 8 }} + {{- end }} {{- if .Values.kubeRbacProxy.enabled }} - args: - --secure-listen-address=0.0.0.0:8443 @@ -91,8 +95,11 @@ spec: securityContext: {{- toYaml .Values.kubeRbacProxy.securityContext | nindent 12}} volumeMounts: - - mountPath: /etc/tls - name: cert + - mountPath: /etc/tls + name: cert + {{- with .Values.kubeRbacProxy.extraVolumeMounts }} + {{- toYaml . | nindent 8 }} + {{- end }} {{- end }} securityContext: runAsNonRoot: true @@ -100,6 +107,10 @@ spec: {{ toYaml . | nindent 8 }} {{- end }} serviceAccountName: {{ include "piraeus-operator.serviceAccountName" . }} + {{- with .Values.imagePullSecrets }} + imagePullSecrets: + {{- toYaml . | nindent 8 }} + {{- end }} terminationGracePeriodSeconds: 10 priorityClassName: {{ .Values.priorityClassName | default "system-cluster-critical" }} nodeSelector: @@ -113,3 +124,6 @@ spec: secret: defaultMode: 420 secretName: {{ include "piraeus-operator.certifcateName" . }} + {{- with .Values.extraVolumes }} + {{- toYaml . | nindent 6 }} + {{- end }} diff --git a/packages/system/piraeus-operator/charts/piraeus/templates/extra-objects.yaml b/packages/system/piraeus-operator/charts/piraeus/templates/extra-objects.yaml new file mode 100644 index 000000000..b0ec6fad2 --- /dev/null +++ b/packages/system/piraeus-operator/charts/piraeus/templates/extra-objects.yaml @@ -0,0 +1,15 @@ +{{- /* Normalize extraObjects to a list, easier to loop over */ -}} +{{- $extraObjects := .Values.extraManifests | default (list) -}} + +{{- if kindIs "map" $extraObjects -}} + {{- $extraObjects = values $extraObjects -}} +{{- end -}} + +{{- range $extraObjects }} +--- + {{- if kindIs "map" . }} + {{- tpl (toYaml .) $ | nindent 0 }} + {{- else if kindIs "string" . }} + {{- tpl . $ | nindent 0 }} + {{- end }} +{{- end }} diff --git a/packages/system/piraeus-operator/charts/piraeus/templates/metrics-service.yaml b/packages/system/piraeus-operator/charts/piraeus/templates/metrics-service.yaml index d1a24e3ca..49dee196a 100644 --- a/packages/system/piraeus-operator/charts/piraeus/templates/metrics-service.yaml +++ b/packages/system/piraeus-operator/charts/piraeus/templates/metrics-service.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Service metadata: name: {{ include "piraeus-operator.fullname" . }}-controller-manager-metrics-service + namespace: {{ .Release.Namespace }} labels: {{- include "piraeus-operator.labels" . | nindent 4 }} spec: diff --git a/packages/system/piraeus-operator/charts/piraeus/templates/rbac.yaml b/packages/system/piraeus-operator/charts/piraeus/templates/rbac.yaml index c532e02d4..56b7d019b 100644 --- a/packages/system/piraeus-operator/charts/piraeus/templates/rbac.yaml +++ b/packages/system/piraeus-operator/charts/piraeus/templates/rbac.yaml @@ -15,7 +15,6 @@ rules: - events - persistentvolumes - pods - - secrets - serviceaccounts - services verbs: @@ -59,6 +58,13 @@ rules: - pods/eviction verbs: - create +- apiGroups: + - "" + resources: + - pods/exec + verbs: + - create + - get - apiGroups: - apiextensions.k8s.io resources: @@ -71,6 +77,12 @@ rules: - patch - update - watch +- apiGroups: + - apiextensions.k8s.io + resources: + - customresourcedefinitions/status + verbs: + - patch - apiGroups: - apps resources: @@ -235,7 +247,6 @@ rules: - snapshot.storage.k8s.io resources: - volumesnapshotclasses - - volumesnapshots verbs: - get - list @@ -257,6 +268,15 @@ rules: verbs: - patch - update +- apiGroups: + - snapshot.storage.k8s.io + resources: + - volumesnapshots + verbs: + - get + - list + - update + - watch - apiGroups: - storage.k8s.io resources: @@ -283,6 +303,7 @@ rules: - storage.k8s.io resources: - storageclasses + - volumeattributesclasses verbs: - get - list @@ -306,8 +327,30 @@ rules: --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role +metadata: + name: {{ include "piraeus-operator.fullname" . }}-controller-manager + namespace: {{ .Release.Namespace }} + labels: + {{- include "piraeus-operator.labels" . | nindent 4 }} +rules: +- apiGroups: + - "" + resources: + - secrets + verbs: + - create + - delete + - get + - list + - patch + - update + - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role metadata: name: {{ include "piraeus-operator.fullname" . }}-leader-election + namespace: {{ .Release.Namespace }} labels: {{- include "piraeus-operator.labels" . | nindent 4 }} rules: diff --git a/packages/system/piraeus-operator/charts/piraeus/templates/rolebindings.yaml b/packages/system/piraeus-operator/charts/piraeus/templates/rolebindings.yaml index 97e53b952..fc5212cc4 100644 --- a/packages/system/piraeus-operator/charts/piraeus/templates/rolebindings.yaml +++ b/packages/system/piraeus-operator/charts/piraeus/templates/rolebindings.yaml @@ -17,6 +17,22 @@ subjects: namespace: {{ .Release.Namespace }} --- apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "piraeus-operator.fullname" . }}-controller-manager + namespace: {{ .Release.Namespace }} + labels: + {{- include "piraeus-operator.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "piraeus-operator.fullname" . }}-controller-manager +subjects: +- kind: ServiceAccount + name: {{ include "piraeus-operator.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +--- +apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: {{ include "piraeus-operator.fullname" . }}-controller-manager diff --git a/packages/system/piraeus-operator/charts/piraeus/templates/serviceaccount.yaml b/packages/system/piraeus-operator/charts/piraeus/templates/serviceaccount.yaml index 89315de95..c4997dd24 100644 --- a/packages/system/piraeus-operator/charts/piraeus/templates/serviceaccount.yaml +++ b/packages/system/piraeus-operator/charts/piraeus/templates/serviceaccount.yaml @@ -4,6 +4,7 @@ apiVersion: v1 kind: ServiceAccount metadata: name: {{ include "piraeus-operator.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} labels: {{- include "piraeus-operator.labels" . | nindent 4 }} {{ end }} diff --git a/packages/system/piraeus-operator/charts/piraeus/templates/validating-webhook-configuration.yaml b/packages/system/piraeus-operator/charts/piraeus/templates/validating-webhook-configuration.yaml index 5a6c5c9e6..04d05bfaf 100644 --- a/packages/system/piraeus-operator/charts/piraeus/templates/validating-webhook-configuration.yaml +++ b/packages/system/piraeus-operator/charts/piraeus/templates/validating-webhook-configuration.yaml @@ -9,6 +9,7 @@ apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: {{ include "piraeus-operator.fullname" . }} + namespace: {{ .Release.Namespace }} labels: {{- include "piraeus-operator.labels" . | nindent 4 }} spec: @@ -36,6 +37,7 @@ apiVersion: v1 kind: Secret metadata: name: {{ include "piraeus-operator.certifcateName" . }} + namespace: {{ .Release.Namespace }} labels: {{- include "piraeus-operator.labels" . | nindent 4 }} type: kubernetes.io/tls diff --git a/packages/system/piraeus-operator/charts/piraeus/templates/webhook-service.yaml b/packages/system/piraeus-operator/charts/piraeus/templates/webhook-service.yaml index d360269bc..57f8966c9 100644 --- a/packages/system/piraeus-operator/charts/piraeus/templates/webhook-service.yaml +++ b/packages/system/piraeus-operator/charts/piraeus/templates/webhook-service.yaml @@ -2,6 +2,7 @@ apiVersion: v1 kind: Service metadata: name: {{ include "piraeus-operator.fullname" . }}-webhook-service + namespace: {{ .Release.Namespace }} labels: {{- include "piraeus-operator.labels" . | nindent 4 }} spec: diff --git a/packages/system/piraeus-operator/charts/piraeus/values.yaml b/packages/system/piraeus-operator/charts/piraeus/values.yaml index bcc4dc9af..ff0d92b5a 100644 --- a/packages/system/piraeus-operator/charts/piraeus/values.yaml +++ b/packages/system/piraeus-operator/charts/piraeus/values.yaml @@ -29,6 +29,12 @@ operator: # cpu: 100m # memory: 128Mi + ## Additional volume mounts + extraVolumeMounts: [ ] + # - name: clusterapi-kubeconfig + # mountPath: /etc/operator + # readOnly: true + kubeRbacProxy: enabled: false image: @@ -54,6 +60,16 @@ kubeRbacProxy: # cpu: 100m # memory: 128Mi + ## Additional volume mounts + extraVolumeMounts: [ ] + + +## Additional volumes +extraVolumes: [] +# - name: clusterapi-kubeconfig +# secret: +# secretName: super-secret + webhook: timeoutSeconds: 2 failurePolicy: Fail @@ -111,3 +127,30 @@ imageConfigOverride: [] # tag: my-custom-tag # Results in the image example.com/piraeus/linstor-csi:my-custom-tag being used. # See templates/config.yaml for available components. + +## Extra manifests to deploy. Can be of type dict or list. +## If dict, keys are ignored and only values are used. +## Items contained within extraObjects can be defined as dict or string and are passed through tpl. +extraManifests: null + # - apiVersion: v1 + # kind: ConfigMap + # metadata: + # labels: + # name: piraeus-extra + # data: + # extra-data: "value" + # + # can also be defined as a string, useful for templating field names + # - | + # apiVersion: v1 + # kind: Secret + # type: Opaque + # metadata: + # name: super-secret + # labels: + # {{- range $key, $value := .Values.commonLabels }} + # {{ $key }}: {{ $value }} + # {{- end }} + # data: + # plaintext: Zm9vYmFy +# templated: '{{ print "foobar" | upper | b64enc }}' From e1ed8c771938e31c9130b06931d649ef85761e49 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 02:23:32 +0300 Subject: [PATCH 473/518] fix(linstor): update linstor-csi to v1.13.2 piraeus-operator v2.12.1 is released against LINSTOR CSI v1.13.2, and the release line carries IO QoS limits through the qos.linbit.com/* StorageClass parameters. The operator gives the CSI pods their client certificate as mounted files through LS_USER_CERTIFICATE_FILE and LS_USER_KEY_FILE, instead of inline values in LS_USER_CERTIFICATE and LS_USER_KEY. The linstor-wait-until v0.3.1 in this image is built on a golinstor release that reads only the inline variables, so its init containers present no client certificate, the LINSTOR API closes the TLS connection, and they retry on EOF forever. Move to linstor-wait-until v1.0.1, the version upstream v1.13.2 pins. The Protocol C override for dual attach is dropped. Since v1.12 the driver no longer sets allow-two-primaries itself: it asks LINSTOR to manage dual-primary through make-available, and LINSTOR 1.35 sets protocol C on the connections that need it. Upstream closed the equivalent change as superseded for that reason (piraeusdatastore/linstor-csi#435). The relocate-after-clone patch is carried forward. Volume IDs became a struct for consistency groups; Clone and VolFromSnap handle standalone resources, so relocation keys on the resource name. The enumer output is regenerated rather than hand-patched. Upstream now pins grpc v1.84.0, which builds on Go 1.27 and includes the fix for GO-2026-6443, so the grpc override is no longer needed. The golden-image clone guard in vm-disk was re-checked: parameter parsing for allowRemoteVolumeAccess and layerList, and the default derived from the layer list, are unchanged in this range. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- packages/system/linstor/Makefile | 2 +- .../linstor/images/linstor-csi/Dockerfile | 11 +- .../001-relocate-after-clone-restore.diff | 103 +++++++------- ...2-protocol-c-override-for-dual-attach.diff | 132 ------------------ 4 files changed, 57 insertions(+), 191 deletions(-) delete mode 100644 packages/system/linstor/images/linstor-csi/patches/002-protocol-c-override-for-dual-attach.diff diff --git a/packages/system/linstor/Makefile b/packages/system/linstor/Makefile index 74e3f5aef..5e44ebade 100644 --- a/packages/system/linstor/Makefile +++ b/packages/system/linstor/Makefile @@ -8,7 +8,7 @@ LINSTOR_VERSION ?= 1.33.3 # The golden-image clone guard in packages/apps/vm-disk/templates/dv.yaml # reads allowRemoteVolumeAccess and layerList the way this release does and # has to be re-checked against the new one. -LINSTOR_CSI_VERSION ?= v1.11.2 +LINSTOR_CSI_VERSION ?= v1.13.2 test: helm unittest . diff --git a/packages/system/linstor/images/linstor-csi/Dockerfile b/packages/system/linstor/images/linstor-csi/Dockerfile index 50e755090..3ff1d154d 100644 --- a/packages/system/linstor/images/linstor-csi/Dockerfile +++ b/packages/system/linstor/images/linstor-csi/Dockerfile @@ -1,8 +1,8 @@ FROM --platform=$BUILDPLATFORM golang:1.27@sha256:3680233e3204827fbdc66088528ae6d4b3d034f51d03a99d454f6de034888244 AS builder ARG GOPROXY -ARG VERSION=v1.11.2 -ARG LINSTOR_WAIT_UNTIL_VERSION=v0.3.1 +ARG VERSION=v1.13.2 +ARG LINSTOR_WAIT_UNTIL_VERSION=v1.0.1 ARG TARGETARCH ARG TARGETOS @@ -15,13 +15,6 @@ RUN git apply /patches/*.diff RUN for attempt in 1 2 3; do go mod download && break; [ "$attempt" = 3 ] && exit 1; sleep $((attempt * 10)); done -# Upstream pins grpc v1.81.0, which uses http2.TrailerPrefix; on Go 1.27 -# golang.org/x/net builds http2 on top of net/http and no longer defines it. -# v1.83.2 also carries the fix for GO-2026-6443, a server panic reachable -# through the CSI gRPC endpoint. -RUN go get google.golang.org/grpc@v1.83.2 && \ - go mod tidy - # go build never compiles a _test.go, so nothing else runs the tests of the # packages the patches touch. Each package is tested from its own directory: # that finds the module it belongs to, and an empty list runs nothing. diff --git a/packages/system/linstor/images/linstor-csi/patches/001-relocate-after-clone-restore.diff b/packages/system/linstor/images/linstor-csi/patches/001-relocate-after-clone-restore.diff index 0e2726708..29c209444 100644 --- a/packages/system/linstor/images/linstor-csi/patches/001-relocate-after-clone-restore.diff +++ b/packages/system/linstor/images/linstor-csi/patches/001-relocate-after-clone-restore.diff @@ -1,38 +1,37 @@ diff --git a/pkg/client/linstor.go b/pkg/client/linstor.go -index ac4651d..46d565d 100644 --- a/pkg/client/linstor.go +++ b/pkg/client/linstor.go -@@ -462,6 +462,14 @@ func (s *Linstor) Clone(ctx context.Context, vol, src *volume.Info, params *volu +@@ -498,6 +498,14 @@ return err } + if params.RelocateAfterClone { + logger.Debug("relocate resources to optimal nodes") + -+ if err := s.relocateResources(ctx, vol.ID, rGroup.Name); err != nil { ++ if err := s.relocateResources(ctx, vol.ResourceName, rGroup.Name); err != nil { + logger.WithError(err).Warn("resource relocation failed, volume is still usable") + } + } + logger.Debug("reconcile extra properties") - err = s.client.ResourceDefinitions.Modify(ctx, vol.ID, lapi.GenericPropsModify{OverrideProps: vol.Properties}) -@@ -1297,6 +1305,14 @@ func (s *Linstor) VolFromSnap(ctx context.Context, snap *volume.Snapshot, vol *v + err = s.client.ResourceDefinitions.Modify(ctx, vol.ResourceName, lapi.GenericPropsModify{OverrideProps: vol.Properties}) +@@ -1614,6 +1622,14 @@ return err } + if snapParams != nil && snapParams.RelocateAfterRestore { + logger.Debug("relocate resources to optimal nodes") + -+ if err := s.relocateResources(ctx, vol.ID, rGroup.Name); err != nil { ++ if err := s.relocateResources(ctx, vol.ResourceName, rGroup.Name); err != nil { + logger.WithError(err).Warn("resource relocation failed, volume is still usable") + } + } + logger.Debug("reconcile extra properties") - err = s.client.ResourceDefinitions.Modify(ctx, vol.ID, lapi.GenericPropsModify{OverrideProps: vol.Properties}) -@@ -1487,9 +1503,8 @@ func (s *Linstor) reconcileSnapshotResources(ctx context.Context, snapshot *volu + err = s.client.ResourceDefinitions.Modify(ctx, vol.ResourceName, lapi.GenericPropsModify{OverrideProps: vol.Properties}) +@@ -1818,9 +1834,8 @@ return fmt.Errorf("snapshot '%s' not deployed on any node", snap.Name) } @@ -44,7 +43,7 @@ index ac4651d..46d565d 100644 for _, snapNode := range snap.Nodes { if err := s.NodeAvailable(ctx, snapNode); err != nil { logger.WithField("selected node candidate", snapNode).WithError(err).Debug("node is not available") -@@ -1497,13 +1512,23 @@ func (s *Linstor) reconcileSnapshotResources(ctx context.Context, snapshot *volu +@@ -1828,15 +1843,25 @@ } if slices.Contains(preferredNodes, snapNode) { @@ -58,8 +57,8 @@ index ac4651d..46d565d 100644 } + + available = append(available, snapNode) -+ } -+ + } + + // Pick a random node from preferred candidates, falling back to any available node. + // Randomization distributes restore load across snapshot nodes, preventing all + // clones of the same source from concentrating on a single node. @@ -71,13 +70,20 @@ index ac4651d..46d565d 100644 + var selectedNode string + if len(candidates) > 0 { + selectedNode = candidates[rand.Intn(len(candidates))] - } - ++ } ++ if selectedNode == "" { -@@ -1696,6 +1721,114 @@ func (s *Linstor) reconcileResourcePlacement(ctx context.Context, vol *volume.In - return nil - } - + return fmt.Errorf("no node [%s] available for snapshot '%s'", strings.Join(snap.Nodes, ","), snap.Name) + } +@@ -2030,6 +2055,114 @@ + err = volumeScheduler.Create(ctx, vol.ResourceName, params, topologies) + if err != nil { + return err ++ } ++ ++ return nil ++} ++ +const propertyRelocationTriggered = "Aux/csi-relocation-triggered" + +// relocateResources migrates replicas from their current nodes to the nodes that @@ -181,19 +187,13 @@ index ac4651d..46d565d 100644 + logger.WithError(err).Warn("migrate-disk failed, skipping this pair") + continue + } -+ } -+ -+ return nil -+} -+ - // FindSnapsByID searches the snapshot in the backend - func (s *Linstor) FindSnapsByID(ctx context.Context, id string) ([]*volume.Snapshot, error) { - snapshotId, err := volume.ParseSnapshotId(id) + } + + return nil diff --git a/pkg/volume/parameter.go b/pkg/volume/parameter.go -index 39acd95..54d1dfb 100644 --- a/pkg/volume/parameter.go +++ b/pkg/volume/parameter.go -@@ -50,6 +50,7 @@ const ( +@@ -51,6 +51,7 @@ nfsservicename nfssquash nfsrecoveryvolumesize @@ -201,24 +201,24 @@ index 39acd95..54d1dfb 100644 ) // Parameters configuration for linstor volumes. -@@ -118,6 +119,8 @@ type Parameters struct { - // NfsRecoveryVolumeSize sets the volume size (in bytes) of the recovery volume used by the NFS server. - // Defaults to 300MiB. - NfsRecoveryVolumeBytes int64 +@@ -122,6 +123,8 @@ + // IOQoSLimits holds the IO QoS limits parsed from the qos.linbit.com/* parameters. They are stored on + // the LINSTOR resource definition and passed to the nri-volume-qos plugin via VolumeAttachment metadata. + IOQoSLimits meta.Limits + // RelocateAfterClone triggers asynchronous relocation of replicas to optimal nodes after cloning. + RelocateAfterClone bool } const DefaultDisklessStoragePoolName = "DfltDisklessStorPool" -@@ -140,6 +143,7 @@ func NewParameters(params map[string]string, topologyPrefix string) (Parameters, +@@ -144,6 +147,7 @@ NfsServiceName: "linstor-csi-nfs", NfsSquash: "no_root_squash", NfsRecoveryVolumeBytes: 300 * 1024 * 1024, + RelocateAfterClone: true, } - for k, v := range params { -@@ -287,6 +291,13 @@ func NewParameters(params map[string]string, topologyPrefix string) (Parameters, + limits, err := meta.ParseLimits(params) +@@ -298,6 +302,13 @@ } p.NfsRecoveryVolumeBytes = s.Value() @@ -233,10 +233,9 @@ index 39acd95..54d1dfb 100644 } diff --git a/pkg/volume/paramkey_enumer.go b/pkg/volume/paramkey_enumer.go -index 5474963..2eb8a7c 100644 --- a/pkg/volume/paramkey_enumer.go +++ b/pkg/volume/paramkey_enumer.go -@@ -7,11 +7,11 @@ import ( +@@ -7,11 +7,11 @@ "strings" ) @@ -251,7 +250,7 @@ index 5474963..2eb8a7c 100644 func (i paramKey) String() string { if i < 0 || i >= paramKey(len(_paramKeyIndex)-1) { -@@ -50,9 +50,10 @@ func _paramKeyNoOp() { +@@ -50,9 +50,10 @@ _ = x[nfsservicename-(23)] _ = x[nfssquash-(24)] _ = x[nfsrecoveryvolumesize-(25)] @@ -262,17 +261,24 @@ index 5474963..2eb8a7c 100644 +var _paramKeyValues = []paramKey{allowremotevolumeaccess, autoplace, clientlist, disklessonremaining, disklessstoragepool, donotplacewithregex, encryption, fsopts, layerlist, mountopts, nodelist, placementcount, placementpolicy, replicasondifferent, replicasonsame, sizekib, storagepool, postmountxfsopts, resourcegroup, usepvcname, overprovision, xreplicasondifferent, nfsconfigtemplate, nfsservicename, nfssquash, nfsrecoveryvolumesize, relocateafterclone} var _paramKeyNameToValueMap = map[string]paramKey{ - _paramKeyName[0:23]: allowremotevolumeaccess, -@@ -107,6 +108,8 @@ var _paramKeyNameToValueMap = map[string]paramKey{ + _paramKeyName[0:23]: allowremotevolumeaccess, +@@ -81,6 +82,7 @@ + _paramKeyName[310:324]: nfsservicename, + _paramKeyName[324:333]: nfssquash, + _paramKeyName[333:354]: nfsrecoveryvolumesize, ++ _paramKeyName[354:372]: relocateafterclone, + } + + var _paramKeyLowerNameToValueMap = map[string]paramKey{ +@@ -110,6 +112,7 @@ + _paramKeyLowerName[310:324]: nfsservicename, _paramKeyLowerName[324:333]: nfssquash, - _paramKeyName[333:354]: nfsrecoveryvolumesize, _paramKeyLowerName[333:354]: nfsrecoveryvolumesize, -+ _paramKeyName[354:372]: relocateafterclone, + _paramKeyLowerName[354:372]: relocateafterclone, } var _paramKeyNames = []string{ -@@ -136,6 +139,7 @@ var _paramKeyNames = []string{ +@@ -139,6 +142,7 @@ _paramKeyName[310:324], _paramKeyName[324:333], _paramKeyName[333:354], @@ -281,18 +287,17 @@ index 5474963..2eb8a7c 100644 // paramKeyString retrieves an enum value from the enum constants string name. diff --git a/pkg/volume/snapshot_params.go b/pkg/volume/snapshot_params.go -index d167cb8..50b70fb 100644 --- a/pkg/volume/snapshot_params.go +++ b/pkg/volume/snapshot_params.go -@@ -35,6 +35,7 @@ type SnapshotParameters struct { - LinstorTargetUrl string `json:"linstor-target-url,omitempty"` - LinstorTargetClusterID string `json:"linstor-target-cluster-id,omitempty"` - LinstorTargetStoragePool string `json:"linstor-target-storage-pool,omitempty"` -+ RelocateAfterRestore bool `json:"relocate-after-restore,omitempty"` +@@ -38,6 +38,7 @@ + LinstorTargetUrl string `json:"linstor-target-url,omitempty"` + LinstorTargetClusterID string `json:"linstor-target-cluster-id,omitempty"` + LinstorTargetStoragePool string `json:"linstor-target-storage-pool,omitempty"` ++ RelocateAfterRestore bool `json:"relocate-after-restore,omitempty"` } func NewSnapshotParameters(params, secrets map[string]string) (*SnapshotParameters, error) { -@@ -91,6 +92,13 @@ func NewSnapshotParameters(params, secrets map[string]string) (*SnapshotParamete +@@ -116,6 +117,13 @@ p.LinstorTargetClusterID = v case "/linstor-target-storage-pool": p.LinstorTargetStoragePool = v diff --git a/packages/system/linstor/images/linstor-csi/patches/002-protocol-c-override-for-dual-attach.diff b/packages/system/linstor/images/linstor-csi/patches/002-protocol-c-override-for-dual-attach.diff deleted file mode 100644 index 279c636b1..000000000 --- a/packages/system/linstor/images/linstor-csi/patches/002-protocol-c-override-for-dual-attach.diff +++ /dev/null @@ -1,132 +0,0 @@ -diff --git a/pkg/client/linstor.go b/pkg/client/linstor.go -index ac4651d..9d75c79 100644 ---- a/pkg/client/linstor.go -+++ b/pkg/client/linstor.go -@@ -653,11 +653,35 @@ func (s *Linstor) Attach(ctx context.Context, volId, node string, rwxBlock bool) - } - - if otherResInUse > 0 && rwxBlock { -- rdPropsModify := lapi.GenericPropsModify{OverrideProps: map[string]string{ -+ rdProps := map[string]string{ - linstor.PropertyAllowTwoPrimaries: "yes", -- }} -+ } - -- err = s.client.ResourceDefinitions.Modify(ctx, volId, rdPropsModify) -+ // DRBD requires Protocol C whenever allow-two-primaries is enabled. -+ // If the resource is configured with Protocol A or B (typically -+ // through its resource-group), drbdadm adjust on the satellites -+ // fails with "Protocol C required" (errno 139) and live migration -+ // cannot proceed. Override Protocol to C on the resource-definition -+ // itself so it applies to every connection (including diskless -+ // peers, e.g. a TieBreaker). The marker lets Detach revert exactly -+ // the override we installed without disturbing operator-set props. -+ proto, protoErr := s.getEffectiveDrbdProtocol(ctx, volId) -+ if protoErr != nil { -+ s.log.WithError(protoErr).WithField("volume", volId). -+ Warn("failed to determine effective DRBD protocol; skipping Protocol=C override for dual-attach") -+ } else if proto == "A" || proto == "B" { -+ s.log.WithFields(logrus.Fields{ -+ "volume": volId, -+ "protocol": proto, -+ }).Info("installing Protocol=C override on resource-definition for dual-attach") -+ -+ rdProps[linstor.PropertyDrbdNetProtocol] = "C" -+ rdProps[linstor.PropertyCsiProtocolOverride] = "yes" -+ } -+ -+ err = s.client.ResourceDefinitions.Modify(ctx, volId, lapi.GenericPropsModify{ -+ OverrideProps: rdProps, -+ }) - if err != nil { - return "", err - } -@@ -774,11 +798,26 @@ func (s *Linstor) Detach(ctx context.Context, volId, node string) error { - } - - if resInUse == 1 { -- rdPropsModify := lapi.GenericPropsModify{DeleteProps: []string{ -- linstor.PropertyAllowTwoPrimaries, -- }} -+ deleteProps := []string{linstor.PropertyAllowTwoPrimaries} -+ -+ // Drop the Protocol=C override only if Attach installed it (gated by -+ // our marker), so an operator-set Protocol property on the -+ // resource-definition is preserved. -+ rd, getErr := s.client.ResourceDefinitions.Get(ctx, volId) -+ if getErr != nil { -+ log.WithError(getErr).Warn("failed to fetch resource-definition props; skipping Protocol=C override removal") -+ } else if rd.Props[linstor.PropertyCsiProtocolOverride] != "" { -+ log.Info("removing Protocol=C override from resource-definition") - -- err = s.client.ResourceDefinitions.Modify(ctx, volId, rdPropsModify) -+ deleteProps = append(deleteProps, -+ linstor.PropertyDrbdNetProtocol, -+ linstor.PropertyCsiProtocolOverride, -+ ) -+ } -+ -+ err = s.client.ResourceDefinitions.Modify(ctx, volId, lapi.GenericPropsModify{ -+ DeleteProps: deleteProps, -+ }) - if err != nil { - return nil404(err) - } -@@ -805,6 +844,33 @@ func (s *Linstor) Detach(ctx context.Context, volId, node string) error { - return nil404(s.client.Resources.Delete(ctx, volId, node)) - } - -+// getEffectiveDrbdProtocol returns the DRBD network protocol ("A", "B" or "C") -+// that LINSTOR will hand to drbdadm for the given resource. Resource-definition -+// properties take precedence; otherwise the value is inherited from the -+// resource-group. An empty string is returned when neither level sets the -+// property (LINSTOR's compiled-in default is "C"). -+func (s *Linstor) getEffectiveDrbdProtocol(ctx context.Context, volId string) (string, error) { -+ rd, err := s.client.ResourceDefinitions.Get(ctx, volId) -+ if err != nil { -+ return "", err -+ } -+ -+ if v, ok := rd.Props[linstor.PropertyDrbdNetProtocol]; ok { -+ return v, nil -+ } -+ -+ if rd.ResourceGroupName == "" { -+ return "", nil -+ } -+ -+ rg, err := s.client.ResourceGroups.Get(ctx, rd.ResourceGroupName) -+ if err != nil { -+ return "", err -+ } -+ -+ return rg.Props[linstor.PropertyDrbdNetProtocol], nil -+} -+ - // CapacityBytes returns the amount of free space in the storage pool specified by the params and topology. - func (s *Linstor) CapacityBytes(ctx context.Context, storagePools []string, overProvision *float64, segments map[string]string) (int64, error) { - log := s.log.WithField("storage-pools", storagePools).WithField("segments", segments) -diff --git a/pkg/linstor/const.go b/pkg/linstor/const.go -index 9a5f79c..c8bc9c3 100644 ---- a/pkg/linstor/const.go -+++ b/pkg/linstor/const.go -@@ -44,6 +44,19 @@ const ( - // PropertyAllowTwoPrimaries is DRBD option to allow second primary. Mainly used for live-migration. - PropertyAllowTwoPrimaries = lc.NamespcDrbdNetOptions + "/allow-two-primaries" - -+ // PropertyDrbdNetProtocol is the DRBD network replication protocol option -+ // (A = async, B = semi-sync, C = sync). DRBD requires Protocol C whenever -+ // allow-two-primaries is enabled, otherwise drbdadm adjust fails with -+ // "Protocol C required" (errno 139). -+ PropertyDrbdNetProtocol = lc.NamespcDrbdNetOptions + "/protocol" -+ -+ // PropertyCsiProtocolOverride marks a resource-connection where this driver -+ // has installed a temporary Protocol=C override to make a Protocol-A/B -+ // resource compatible with allow-two-primaries during live migration. The -+ // marker lets us safely remove only the overrides we set, without touching -+ // connection-level overrides installed by the operator. -+ PropertyCsiProtocolOverride = lc.NamespcAuxiliary + "/csi-protocol-override" -+ - // CreatedForTemporaryDisklessAttach marks a resource as temporary, i.e. it should be removed after it is no longer - // needed. - CreatedForTemporaryDisklessAttach = "temporary-diskless-attach" From c1652a730d77bd3217597a152cdcaafcf1e68970 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 02:41:08 +0300 Subject: [PATCH 474/518] fix(linstor): update LINSTOR server to v1.35.3 piraeus-operator v2.12.1 ships LINSTOR 1.35.3 as its controller and satellite version, and the in-repo piraeus-server build has to match it. LINSTOR 1.34 raised the minimum Java to 21, so the build and runtime stages move from bookworm to trixie, and the system Gradle used by debian/rules follows the upstream wrapper to 8.14.2. Six carried patches are part of upstream by v1.35.3 and are dropped; the stale-bitmap adjust retry is rebased onto the new DrbdAdm. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- packages/system/linstor/Makefile | 2 +- .../linstor/images/piraeus-server/Dockerfile | 9 +- .../images/piraeus-server/patches/README.md | 34 +- .../patches/allow-toggle-disk-retry.diff | 261 ------- .../patches/fix-duplicate-tcp-ports.diff | 137 ---- .../patches/fix-luks-header-size.diff | 570 --------------- .../patches/fix-min-io-probe-device-wait.diff | 660 ------------------ .../piraeus-server/patches/native-protoc.diff | 10 +- .../retry-adjust-after-stale-bitmap.diff | 25 +- .../patches/retry-secondary-after-mkfs.diff | 104 --- .../snapshot-restore-drbd-initialized.diff | 67 -- 11 files changed, 33 insertions(+), 1846 deletions(-) delete mode 100644 packages/system/linstor/images/piraeus-server/patches/allow-toggle-disk-retry.diff delete mode 100644 packages/system/linstor/images/piraeus-server/patches/fix-duplicate-tcp-ports.diff delete mode 100644 packages/system/linstor/images/piraeus-server/patches/fix-luks-header-size.diff delete mode 100644 packages/system/linstor/images/piraeus-server/patches/fix-min-io-probe-device-wait.diff delete mode 100644 packages/system/linstor/images/piraeus-server/patches/retry-secondary-after-mkfs.diff delete mode 100644 packages/system/linstor/images/piraeus-server/patches/snapshot-restore-drbd-initialized.diff diff --git a/packages/system/linstor/Makefile b/packages/system/linstor/Makefile index 5e44ebade..e58fc8578 100644 --- a/packages/system/linstor/Makefile +++ b/packages/system/linstor/Makefile @@ -4,7 +4,7 @@ export NAMESPACE=cozy-$(NAME) include ../../../hack/common-envs.mk include ../../../hack/package.mk -LINSTOR_VERSION ?= 1.33.3 +LINSTOR_VERSION ?= 1.35.3 # The golden-image clone guard in packages/apps/vm-disk/templates/dv.yaml # reads allowRemoteVolumeAccess and layerList the way this release does and # has to be re-checked against the new one. diff --git a/packages/system/linstor/images/piraeus-server/Dockerfile b/packages/system/linstor/images/piraeus-server/Dockerfile index 2c8a5500a..631303b72 100644 --- a/packages/system/linstor/images/piraeus-server/Dockerfile +++ b/packages/system/linstor/images/piraeus-server/Dockerfile @@ -1,9 +1,10 @@ -ARG DISTRO=bookworm@sha256:f37a335e82bca302e955fa39f9dfe28f1be618f016f8a2b56318e5a5111afc26 +# LINSTOR 1.35 builds and runs on Java 21, which bookworm does not ship. +ARG DISTRO=trixie@sha256:9c6e381421d0f48d1b8133e573f33c185ee16cbf505eda9c4a8a4791b25ea8a6 ARG LINSTOR_VERSION # ------------------------------------------------------------------------------ # Build linstor-server from source -FROM --platform=$BUILDPLATFORM debian:bookworm@sha256:f37a335e82bca302e955fa39f9dfe28f1be618f016f8a2b56318e5a5111afc26 AS builder +FROM --platform=$BUILDPLATFORM debian:trixie@sha256:9c6e381421d0f48d1b8133e573f33c185ee16cbf505eda9c4a8a4791b25ea8a6 AS builder ARG LINSTOR_VERSION ARG VERSION=${LINSTOR_VERSION} @@ -13,10 +14,10 @@ ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update \ && apt-get -y upgrade \ && apt-get -y install build-essential git default-jdk-headless python3-all debhelper wget unzip && \ - wget https://services.gradle.org/distributions/gradle-8.9-bin.zip -O /tmp/gradle.zip && \ + wget https://services.gradle.org/distributions/gradle-8.14.2-bin.zip -O /tmp/gradle.zip && \ unzip -d /opt /tmp/gradle.zip && \ rm /tmp/gradle.zip && \ - ln -s /opt/gradle-8.9/bin/gradle /usr/local/bin/gradle + ln -s /opt/gradle-8.14.2/bin/gradle /usr/local/bin/gradle RUN git clone https://github.com/LINBIT/linstor-server.git /linstor-server WORKDIR /linstor-server diff --git a/packages/system/linstor/images/piraeus-server/patches/README.md b/packages/system/linstor/images/piraeus-server/patches/README.md index d2ded4285..f0b80dc92 100644 --- a/packages/system/linstor/images/piraeus-server/patches/README.md +++ b/packages/system/linstor/images/piraeus-server/patches/README.md @@ -1,36 +1,22 @@ # LINSTOR Server Patches -Custom patches for piraeus-server (linstor-server) v1.33.3. +Custom patches for piraeus-server (linstor-server) v1.35.3. -- **allow-toggle-disk-retry.diff** — Backport maintainer implementation of toggle-disk retry/abort - - Source PR/comment: [#475](https://github.com/LINBIT/linstor-server/pull/475), [maintainer note](https://github.com/LINBIT/linstor-server/pull/475#issuecomment-3949630419) - - Backported from upstream commit: [`3d97f71c9`](https://github.com/LINBIT/linstor-server/commit/3d97f71c95a493588d3d521c63eac4d846935fb3) -- **fix-duplicate-tcp-ports.diff** — Preserve DRBD TCP ports during toggle-disk and avoid redundant `ensureStackDataExists()` - - Source PR/review: [#476](https://github.com/LINBIT/linstor-server/pull/476), [review suggestion](https://github.com/LINBIT/linstor-server/pull/476#discussion_r3007725079) - - Backported from commits: [`79d6375c5`](https://github.com/kvaps/linstor-server/commit/79d6375c55d6181b35a7b7f0fe8dbdfb86e126cd), [`bcc89902f`](https://github.com/kvaps/linstor-server/commit/bcc89902f4f61ac1589dd07ebb7f5aae1935370d) -- **fix-luks-header-size.diff** — Account for LUKS2 `--offset`, metadata/keyslots sizing, and device `optimal_io_size` - - Source PR/comment: [#472](https://github.com/LINBIT/linstor-server/pull/472), [maintainer note](https://github.com/LINBIT/linstor-server/pull/472#issuecomment-3949687603) - - Backported from commits: [`ccc85fbd2`](https://github.com/LINBIT/linstor-server/commit/ccc85fbd2c65f0b97c52403fa80f1efdb886ec4e), [`71b601554`](https://github.com/LINBIT/linstor-server/commit/71b601554f41bcb50cd5bd06989c5b0d3a814acd) - - Note: upstream commit [`3d0402a0c`](https://github.com/LINBIT/linstor-server/commit/3d0402a0c25f0a4b57b380321f10e89982f26e7a) is already included in `v1.33.1` - **fix-k8s-create-already-exists.diff** — Adopt the existing object when a create in the Kubernetes CRD backend returns 409 AlreadyExists. The HTTP client retries a POST after a 5xx or a dropped connection, so a create the apiserver already persisted comes back as 409; a single-object transaction has no rollback entry, the key is dropped from memory while the CR stays, and every later write of that key fails until the controller restarts. Seen as a PV stuck in `Released` because linstor-csi could not set `BalanceResourcesEnabled` on the resource definition. - Source PR: [#531](https://github.com/LINBIT/linstor-server/pull/531) - Backported from commit: [`8badf8f05`](https://github.com/lexfrei/linstor-server/commit/8badf8f0526ff3b1b391857e4a046ed46b95a79c) - **retry-adjust-after-stale-bitmap.diff** — Retry `drbdadm adjust` after detaching a stale local bitmap state - Source PR: [#491](https://github.com/LINBIT/linstor-server/pull/491) - Backported from commit: [`51ae50a84`](https://github.com/kvaps/linstor-server/commit/51ae50a84dcb98093f543b819652c750a94d96c9) -- **retry-secondary-after-mkfs.diff** — Retry `drbdadm secondary` after mkfs when DRBD reports the device as held open by an external probe (Talos block-controller, udev, multipathd, etc.). Without this retry, a transient `Device is held open by someone` aborts resource initialization, leaves the satellite in an intermediate state, and prevents the controller from receiving the final `UpToDate` event — orphan PVs in `Released` then cannot be cleaned up. - - Related upstream issue: [#268](https://github.com/LINBIT/linstor-server/issues/268) - - Related upstream issue: [drbd #74](https://github.com/LINBIT/drbd/issues/74) (same EBUSY pattern with multipathd) - -- **fix-min-io-probe-device-wait.diff** — Wait for temporary probe devices before reading pool I/O properties; retry failures and fall back to a probe when ZFS volumes have no usable device path. - - Source PR: [#528](https://github.com/LINBIT/linstor-server/pull/528), fixes [#527](https://github.com/LINBIT/linstor-server/issues/527) - - Adapted for v1.33.3; inactive LVM volumes retain the existing no-probe behavior. - + - Rebased onto v1.35.3, where `DrbdAdm` gained the `DFLT_WINDOWS_TIMEOUT_MS` constant; `DrbdadmAdjust` still receives the failed `OutputData` through `ExtCmdFailedException`, so its own retry on "Resize not allowed during resync" keeps working on top of this one. - **native-protoc.diff** — Download the protoc build for the build host's architecture instead of always `linux-x86_64`. The builder stage runs on `$BUILDPLATFORM`, so on a native arm64 builder the x86_64 protoc cannot execute and `./gradlew generateJava` fails with `Syntax error: ")" unexpected`. - Not proposed upstream yet. -- **snapshot-restore-drbd-initialized.diff** — Mark the volume definitions of a restored snapshot as DRBD-initialized, unless the restored resource definition carries `DrbdOptions/ForceInitialSyncPermanent`, which keeps the existing path that recreates metadata, as for a backup. Without the flag the initial UpToDate race is still open on a restored volume, and `Linstor/Drbd/InitialUptoDateOn`, copied from the source volume through the snapshot, names whichever node won it for the source. When linstor-csi autoplaces a replica on that node, the empty new volume wins the race, gets mkfs and goes StandAlone against the restored replica, so a pod attached there sees an empty filesystem. With the flag set, nothing wins the race, and an autoplaced replica resyncs from the restored one like any replica added to an existing volume. - - Backported from upstream commit: [`9d8cdf511`](https://github.com/LINBIT/linstor-server/commit/9d8cdf5118338fc72a34bb99799e452e031ea8f6), first released in v1.34.0; the `CHANGELOG.md` hunk is left out - - Drop when LINSTOR is upgraded to v1.34.0 or later - - On ZFS and thin pools the autoplaced replicas still skip the initial sync and rely on the source volume's day0 UUID being in the restored replica's history, upstream included; if it is not, DRBD reports unrelated data and goes StandAlone rather than syncing - - Fixes [cozystack/cozystack#4862](https://github.com/cozystack/cozystack/issues/4862) +Dropped on the move from v1.33.3, because v1.35.3 already carries them: + +- `allow-toggle-disk-retry.diff` — upstream [`3d97f71c9`](https://github.com/LINBIT/linstor-server/commit/3d97f71c95a493588d3d521c63eac4d846935fb3), v1.34.0 +- `fix-luks-header-size.diff` — upstream [`ccc85fbd2`](https://github.com/LINBIT/linstor-server/commit/ccc85fbd2c65f0b97c52403fa80f1efdb886ec4e) and [`71b601554`](https://github.com/LINBIT/linstor-server/commit/71b601554f41bcb50cd5bd06989c5b0d3a814acd), v1.34.0 +- `snapshot-restore-drbd-initialized.diff` — upstream [`9d8cdf511`](https://github.com/LINBIT/linstor-server/commit/9d8cdf5118338fc72a34bb99799e452e031ea8f6), v1.34.0 +- `fix-duplicate-tcp-ports.diff` — upstream [`521c6d211`](https://github.com/LINBIT/linstor-server/commit/521c6d21121dbbcc26cf6eda723c24a5b34889be) (from [#476](https://github.com/LINBIT/linstor-server/pull/476)), v1.34.0 +- `retry-secondary-after-mkfs.diff` — upstream [`677789c02`](https://github.com/LINBIT/linstor-server/commit/677789c028ac), v1.34.0 +- `fix-min-io-probe-device-wait.diff` — upstream [`6e5557839`](https://github.com/LINBIT/linstor-server/commit/6e5557839c911c7a1db240dcb6fc348383a74266) (from [#528](https://github.com/LINBIT/linstor-server/pull/528)), v1.35.1 diff --git a/packages/system/linstor/images/piraeus-server/patches/allow-toggle-disk-retry.diff b/packages/system/linstor/images/piraeus-server/patches/allow-toggle-disk-retry.diff deleted file mode 100644 index 952ba16c5..000000000 --- a/packages/system/linstor/images/piraeus-server/patches/allow-toggle-disk-retry.diff +++ /dev/null @@ -1,261 +0,0 @@ -diff --git a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java -index d93a18014..a944cb809 100644 ---- a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java -+++ b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java -@@ -111,6 +111,14 @@ import reactor.util.function.Tuple2; - @Singleton - public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionListener - { -+ private enum ToggleDiskAction -+ { -+ NOOP, -+ NORMAL, -+ ABORT, -+ RETRY -+ } -+ - private final AccessContext apiCtx; - private final ScopeRunner scopeRunner; - private final BackgroundRunner backgroundRunner; -@@ -386,69 +394,33 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL - "Toggle Disk on %s/%s %s", nodeNameStr, rscNameStr, removeDisk ? "removing disk" : "adding disk"); - - Resource rsc = ctrlApiDataLoader.loadRsc(nodeName, rscName, true); -+ ResourceDefinition rscDfn = rsc.getResourceDefinition(); - -- if (hasDiskAddRequested(rsc)) -- { -- throw new ApiRcException(ApiCallRcImpl.simpleEntry( -- ApiConsts.FAIL_RSC_BUSY, -- "Addition of disk to resource already requested", -- true -- )); -- } -- if (hasDiskRemoveRequested(rsc)) -- { -- throw new ApiRcException(ApiCallRcImpl.simpleEntry( -- ApiConsts.FAIL_RSC_BUSY, -- "Removal of disk from resource already requested", -- true -- )); -- } -+ ToggleDiskAction action = determineToggleDiskAction(rsc, removeDisk); -+ errorReporter.logDebug("Toggle Disk action: %s", action); - -- if (!removeDisk && !ctrlVlmCrtApiHelper.isDiskless(rsc)) -+ switch (action) - { -- throw new ApiRcException(ApiCallRcImpl.simpleEntry( -- ApiConsts.WARN_RSC_ALREADY_HAS_DISK, -- "Resource already has disk", -- true -- )); -- } -- if (removeDisk && ctrlVlmCrtApiHelper.isDiskless(rsc)) -- { -- throw new ApiRcException(ApiCallRcImpl.simpleEntry( -- ApiConsts.WARN_RSC_ALREADY_DISKLESS, -- "Resource already diskless", -- true -- )); -+ case NOOP: -+ return handleNoopAction(rsc, removeDisk); -+ case RETRY: -+ return handleRetryAction(rsc, removeDisk, toggleIntoTiebreakerRef, context); -+ case ABORT: -+ clearToggleDiskFlags(rsc); -+ errorReporter.logInfo( -+ "Aborting previous toggle disk transition, starting new transition to %s", -+ removeDisk ? "diskless" : "diskful" -+ ); -+ break; -+ case NORMAL: -+ break; -+ default: -+ throw new ImplementationError("Unhandled case: " + action); - } - -- ResourceDefinition rscDfn = rsc.getResourceDefinition(); -- AccessContext peerCtx = peerAccCtx.get(); -- if (removeDisk) -- { -- // Prevent removal of the last disk -- int haveDiskCount = countDisksAndIsOnline(rscDfn); -- if (haveDiskCount <= 1) -- { -- throw new ApiRcException(ApiCallRcImpl.simpleEntry( -- ApiConsts.FAIL_INSUFFICIENT_REPLICA_COUNT, -- "Cannot remove the disk from the only online resource with a disk", -- true -- )); -- } -+ validateToggleDiskPreconditions(rsc, removeDisk); - -- if (!LayerUtils.hasLayer(getLayerData(peerCtx, rsc), DeviceLayerKind.DRBD)) -- { -- throw new ApiRcException(ApiCallRcImpl.simpleEntry( -- ApiConsts.FAIL_INVLD_LAYER_STACK, -- "Toggle disk is only supported in combination with DRBD", -- true -- )); -- } -- } -- else -- { -- ensureAllPeersHavePeerSlotLeft(rscDfn); -- } -+ AccessContext peerCtx = peerAccCtx.get(); - - // Save the requested storage pool in the resource properties. - // This does not cause the storage pool to be used automatically. -@@ -628,10 +600,10 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL - - ctrlTransactionHelper.commit(); - -- String action = removeDisk ? "Removal of disk from" : "Addition of disk to"; -+ String actionStr = removeDisk ? "Removal of disk from" : "Addition of disk to"; - responses.addEntry(ApiCallRcImpl.simpleEntry( - ApiConsts.MODIFIED, -- action + " resource '" + rscDfn.getName().displayValue + "' " + -+ actionStr + " resource '" + rscDfn.getName().displayValue + "' " + - "on node '" + rsc.getNode().getName().displayValue + "' registered" - )); - -@@ -651,6 +623,40 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL - ); - } - -+ private Flux handleNoopAction(Resource rscRef, boolean removeDiskRef) -+ { -+ String state = removeDiskRef ? "diskless" : "diskful"; -+ ApiCallRcImpl responses = new ApiCallRcImpl(); -+ responses.addEntry(ApiCallRcImpl.simpleEntry( -+ ApiConsts.INFO_NOOP, -+ "Resource '" + rscRef.getResourceDefinition().getName().displayValue + "' on node '" + -+ rscRef.getNode().getName().displayValue + "' is already " + state -+ )); -+ return Flux.just(responses); -+ } -+ -+ private Flux handleRetryAction( -+ Resource rscRef, -+ boolean removeDisk, -+ boolean toggleIntoTiebreakerRef, -+ ResponseContext context -+ ) -+ { -+ String direction = removeDisk ? "diskless" : "diskful"; -+ ApiCallRcImpl responses = new ApiCallRcImpl(); -+ NodeName nodeName = rscRef.getNode().getName(); -+ ResourceName rscName = rscRef.getResourceDefinition().getName(); -+ responses.addEntry(ApiCallRcImpl.simpleEntry( -+ ApiConsts.INFO_NOOP, -+ "Retrying toggle disk to " + direction + " for resource '" + rscName.displayValue + -+ "' on node '" + nodeName.displayValue + "'" -+ )); -+ -+ return Flux -+ .just(responses) -+ .concatWith(updateAndAdjustDisk(nodeName, rscName, removeDisk, toggleIntoTiebreakerRef, context)); -+ } -+ - private long getVlmDfnSizePrivileged(VolumeDefinition vlmDfnRef) - { - try -@@ -781,6 +787,96 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL - } - } - -+ private ToggleDiskAction determineToggleDiskAction(Resource rsc, boolean removeDisk) -+ { -+ boolean isDiskless = ctrlVlmCrtApiHelper.isDiskless(rsc); -+ boolean diskAddRequested = hasDiskAddRequested(rsc); -+ boolean diskRemoveRequested = hasDiskRemoveRequested(rsc); -+ -+ ToggleDiskAction action; -+ if (isDiskless) -+ { -+ if (diskAddRequested) -+ { -+ action = removeDisk ? ToggleDiskAction.ABORT : ToggleDiskAction.RETRY; -+ } -+ else if (diskRemoveRequested) -+ { -+ action = removeDisk ? ToggleDiskAction.RETRY : ToggleDiskAction.ABORT; -+ } -+ else -+ { -+ action = removeDisk ? ToggleDiskAction.NOOP : ToggleDiskAction.NORMAL; -+ } -+ } -+ else -+ { -+ if (diskRemoveRequested) -+ { -+ action = removeDisk ? ToggleDiskAction.RETRY : ToggleDiskAction.ABORT; -+ } -+ else -+ { -+ action = removeDisk ? ToggleDiskAction.NORMAL : ToggleDiskAction.NOOP; -+ } -+ } -+ -+ return action; -+ } -+ -+ private void validateToggleDiskPreconditions(Resource rsc, boolean removeDisk) -+ { -+ ResourceDefinition rscDfn = rsc.getResourceDefinition(); -+ if (removeDisk) -+ { -+ validateDiskRemovalAllowed(rsc, rscDfn); -+ } -+ else -+ { -+ ensureAllPeersHavePeerSlotLeft(rscDfn); -+ } -+ } -+ -+ private void clearToggleDiskFlags(Resource rsc) -+ { -+ try -+ { -+ rsc.getStateFlags().disableFlags( -+ apiCtx, -+ Resource.Flags.DISK_ADD_REQUESTED, -+ Resource.Flags.DISK_ADDING, -+ Resource.Flags.DISK_REMOVE_REQUESTED, -+ Resource.Flags.DISK_REMOVING -+ ); -+ } -+ catch (AccessDeniedException | DatabaseException exc) -+ { -+ throw new ImplementationError(exc); -+ } -+ } -+ -+ private void validateDiskRemovalAllowed(Resource rsc, ResourceDefinition rscDfn) -+ { -+ int haveDiskCount = countDisksAndIsOnline(rscDfn); -+ if (haveDiskCount <= 1) -+ { -+ throw new ApiRcException(ApiCallRcImpl.simpleEntry( -+ ApiConsts.FAIL_INSUFFICIENT_REPLICA_COUNT, -+ "Cannot remove the disk from the only online resource with a disk", -+ true -+ )); -+ } -+ -+ if (!LayerUtils.hasLayer(getLayerData(peerAccCtx.get(), rsc), DeviceLayerKind.DRBD)) -+ { -+ throw new ApiRcException(ApiCallRcImpl.simpleEntry( -+ ApiConsts.FAIL_INVLD_LAYER_STACK, -+ "Toggle disk is only supported in combination with DRBD", -+ true -+ )); -+ } -+ } -+ - /** - * Although we need to rebuild the layerData as the layerList might have changed, if we do not - * deactivate (i.e. down) the current resource, we need to make sure that deleting DrbdRscData diff --git a/packages/system/linstor/images/piraeus-server/patches/fix-duplicate-tcp-ports.diff b/packages/system/linstor/images/piraeus-server/patches/fix-duplicate-tcp-ports.diff deleted file mode 100644 index ed106f98c..000000000 --- a/packages/system/linstor/images/piraeus-server/patches/fix-duplicate-tcp-ports.diff +++ /dev/null @@ -1,137 +0,0 @@ -diff --git a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java -index d93a18014..01cfbbacf 100644 ---- a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java -+++ b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlRscToggleDiskApiCallHandler.java -@@ -37,6 +37,7 @@ import com.linbit.linstor.core.objects.StorPool; - import com.linbit.linstor.core.objects.Volume; - import com.linbit.linstor.core.objects.VolumeDefinition; - import com.linbit.linstor.core.objects.utils.MixedStorPoolHelper; -+import com.linbit.linstor.core.types.TcpPortNumber; - import com.linbit.linstor.dbdrivers.DatabaseException; - import com.linbit.linstor.event.EventWaiter; - import com.linbit.linstor.event.ObjectIdentifier; -@@ -85,6 +86,7 @@ import java.util.List; - import java.util.Map; - import java.util.Map.Entry; - import java.util.Set; -+import java.util.TreeSet; - - import org.reactivestreams.Publisher; - import reactor.core.publisher.Flux; -@@ -575,12 +577,13 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL - - /* - * We also have to remove the currently diskless DrbdRscData and free up the node-id as now we must -- * use the shared resource's node-id -+ * use the shared resource's node-id. We still need to preserve TCP ports though. - */ -+ copyDrbdTcpPortsIfExists(rsc, payload); - } - else - { -- copyDrbdNodeIdIfExists(rsc, payload); -+ copyDrbdSettings(rsc, payload); - } - /* - * rebuilds the layerdata in case we just removed it.. -@@ -782,10 +785,20 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL - } - - /** -- * Although we need to rebuild the layerData as the layerList might have changed, if we do not -- * deactivate (i.e. down) the current resource, we need to make sure that deleting DrbdRscData -- * and recreating a new DrbdRscData ends up with the same node-id as before. -+ * Copies DRBD settings (node-id and TCP ports) from the existing DrbdRscData into the payload -+ * before removeLayerData() deletes them. This ensures that recreated DrbdRscData ends up with -+ * the same node-id and TCP ports as before. -+ * -+ * TCP ports must be preserved because if the satellite misses the update (e.g. due to controller -+ * restart or connectivity issues), it will keep the old ports while peers receive the new ones, -+ * causing DRBD connections to fail with StandAlone state. - */ -+ private void copyDrbdSettings(Resource rsc, LayerPayload payload) throws ImplementationError -+ { -+ copyDrbdNodeIdIfExists(rsc, payload); -+ copyDrbdTcpPortsIfExists(rsc, payload); -+ } -+ - private void copyDrbdNodeIdIfExists(Resource rsc, LayerPayload payload) throws ImplementationError - { - Set> drbdRscDataSet = LayerRscUtils.getRscDataByLayer( -@@ -804,6 +817,28 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL - } - } - -+ private void copyDrbdTcpPortsIfExists(Resource rsc, LayerPayload payload) throws ImplementationError -+ { -+ Set> drbdRscDataSet = LayerRscUtils.getRscDataByLayer( -+ getLayerData(apiCtx, rsc), -+ DeviceLayerKind.DRBD -+ ); -+ if (!drbdRscDataSet.isEmpty()) -+ { -+ DrbdRscData drbdRscData = (DrbdRscData) drbdRscDataSet.iterator().next(); -+ Collection tcpPorts = drbdRscData.getTcpPortList(); -+ if (tcpPorts != null && !tcpPorts.isEmpty()) -+ { -+ Set portInts = new TreeSet<>(); -+ for (TcpPortNumber port : tcpPorts) -+ { -+ portInts.add(port.value); -+ } -+ payload.drbdRsc.tcpPorts = portInts; -+ } -+ } -+ } -+ - private List removeLayerData(Resource rscRef) - { - List layerList; -@@ -1058,15 +1093,15 @@ public class CtrlRscToggleDiskApiCallHandler implements CtrlSatelliteConnectionL - /* - * We also have to remove the possible meta-children of previous StorageRscData. - * LayerData will be recreated with ensureStackDataExists. -- * However, we still need to remember our node-id if we had / have DRBD in the list -+ * However, we still need to remember our DRBD settings if we had / have DRBD in the list - */ -- copyDrbdNodeIdIfExists(rsc, payload); -+ copyDrbdSettings(rsc, payload); - layerList = removeLayerData(rsc); - } - else - { - markDiskAdded(rsc); -- ctrlLayerStackHelper.resetStoragePools(rsc); -+ ctrlLayerStackHelper.resetStoragePools(rsc, false); - } - ctrlLayerStackHelper.ensureStackDataExists(rsc, layerList, payload); - -diff --git a/controller/src/main/java/com/linbit/linstor/layer/resource/CtrlRscLayerDataFactory.java b/controller/src/main/java/com/linbit/linstor/layer/resource/CtrlRscLayerDataFactory.java -index 3538b380c..f9733b6f1 100644 ---- a/controller/src/main/java/com/linbit/linstor/layer/resource/CtrlRscLayerDataFactory.java -+++ b/controller/src/main/java/com/linbit/linstor/layer/resource/CtrlRscLayerDataFactory.java -@@ -263,6 +263,11 @@ public class CtrlRscLayerDataFactory - } - - public void resetStoragePools(Resource rscRef) -+ { -+ resetStoragePools(rscRef, true); -+ } -+ -+ public void resetStoragePools(Resource rscRef, boolean callEnsureStackDataExistsRef) - { - try - { -@@ -276,8 +281,10 @@ public class CtrlRscLayerDataFactory - - rscDataToProcess.addAll(rscData.getChildren()); - } -- -- ensureStackDataExists(rscRef, null, new LayerPayload()); -+ if (callEnsureStackDataExistsRef) -+ { -+ ensureStackDataExists(rscRef, null, new LayerPayload()); -+ } - } - catch (AccessDeniedException exc) - { diff --git a/packages/system/linstor/images/piraeus-server/patches/fix-luks-header-size.diff b/packages/system/linstor/images/piraeus-server/patches/fix-luks-header-size.diff deleted file mode 100644 index 8611825bc..000000000 --- a/packages/system/linstor/images/piraeus-server/patches/fix-luks-header-size.diff +++ /dev/null @@ -1,570 +0,0 @@ -diff --git a/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java b/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java ---- a/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java -+++ b/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java -@@ -2081,7 +2081,6 @@ public abstract class AbsStorageProvider< - ) - throws IOException, AccessDeniedException - { -- final StorPoolName storPoolObjName = storPoolObj.getName(); - if (storDevicePath != null) - { - errorReporter.logDebug("updateMinIoSize: Have storDevicePath \"%s\"", storDevicePath); -@@ -2094,35 +2093,18 @@ public abstract class AbsStorageProvider< - "updateMinIoSize: Block device path is \"%s\"", - blockDevicePath.toString() - ); -- final long minIoSize = BlockSizeInfo.getBlockSize(blockDevicePath); -- -- boolean updateValue = true; -- -- final String propKey = StorageConstants.NAMESPACE_INTERNAL + '/' + -- StorageConstants.BLK_DEV_MIN_IO_SIZE; -- final Props storPoolProps = storPoolObj.getProps(storDriverAccCtx); -- final @Nullable String currentPropValue = storPoolProps.getProp(propKey); -- if (currentPropValue != null) -- { -- try -- { -- final long currentPropMinIoSize = Long.parseLong(currentPropValue); -- updateValue = currentPropMinIoSize != minIoSize; -- } -- catch (NumberFormatException ignored) -- { -- } -- } -- -- if (updateValue) -- { -- final String propValue = Long.toString(minIoSize); -- errorReporter.logDebug( -- "Storage pool \"%s\": Set property \"%s\" = \"%s\"", -- storPoolObjName.displayValue, propKey, propValue -- ); -- propsChange.changeStorPoolProp(storPoolObj, propKey, propValue); -- } -+ updatePropIfNeeded( -+ propsChange, -+ storPoolObj, -+ StorageConstants.NAMESPACE_INTERNAL + '/' + StorageConstants.BLK_DEV_MIN_IO_SIZE, -+ Long.toString(BlockSizeInfo.getBlockSize(blockDevicePath)) -+ ); -+ updatePropIfNeeded( -+ propsChange, -+ storPoolObj, -+ StorageConstants.NAMESPACE_INTERNAL + '/' + StorageConstants.BLK_DEV_OPT_IO_SIZE, -+ Long.toString(BlockSizeInfo.getOptimalIoSize(blockDevicePath)) -+ ); - } - else - { -@@ -2130,6 +2112,28 @@ public abstract class AbsStorageProvider< - } - } - -+ private void updatePropIfNeeded( -+ LocalPropsChangePojo propsChangeRef, -+ StorPool storPoolObjRef, -+ String propKeyRef, -+ String propValueRef -+ ) -+ throws AccessDeniedException -+ { -+ final Props storPoolProps = storPoolObjRef.getProps(storDriverAccCtx); -+ final @Nullable String currentPropValue = storPoolProps.getProp(propKeyRef); -+ if (currentPropValue == null || !currentPropValue.equals(propValueRef)) -+ { -+ errorReporter.logDebug( -+ "Storage pool \"%s\": Set property \"%s\" = \"%s\"", -+ storPoolObjRef.getName().displayValue, -+ propKeyRef, -+ propValueRef -+ ); -+ propsChangeRef.changeStorPoolProp(storPoolObjRef, propKeyRef, propValueRef); -+ } -+ } -+ - @SuppressWarnings("unchecked") - @Override - public void updateAllocatedSize(VlmProviderObject vlmDataRef) -diff --git a/satellite/src/main/java/com/linbit/linstor/layer/storage/utils/BlockSizeInfo.java b/satellite/src/main/java/com/linbit/linstor/layer/storage/utils/BlockSizeInfo.java ---- a/satellite/src/main/java/com/linbit/linstor/layer/storage/utils/BlockSizeInfo.java -+++ b/satellite/src/main/java/com/linbit/linstor/layer/storage/utils/BlockSizeInfo.java -@@ -3,8 +3,11 @@ package com.linbit.linstor.layer.storage.utils; - import com.linbit.utils.MathUtils; - import com.linbit.utils.SymbolicLinkResolver; - -+import static com.linbit.linstor.layer.storage.BlockSizeConsts.DFLT_OPT_IO_SIZE; - import static com.linbit.linstor.layer.storage.BlockSizeConsts.DFLT_IO_SIZE; -+import static com.linbit.linstor.layer.storage.BlockSizeConsts.MAX_OPT_IO_SIZE; - import static com.linbit.linstor.layer.storage.BlockSizeConsts.MAX_IO_SIZE; -+import static com.linbit.linstor.layer.storage.BlockSizeConsts.MIN_OPT_IO_SIZE; - import static com.linbit.linstor.layer.storage.BlockSizeConsts.MIN_IO_SIZE; - - import java.io.FileInputStream; -@@ -13,6 +16,8 @@ import java.nio.file.Path; - - public class BlockSizeInfo - { -+ private static final int NUMBER_BUFFER_SIZE = 32; -+ - /** - * Determines the blocksize, aka minimum I/O size, for the specified backing storage path. - * -@@ -51,7 +56,7 @@ public class BlockSizeInfo - final Path infoSourceName = blockDevice.getFileName(); - final Path infoSource = Path.of("/sys/block", infoSourceName.toString(), "queue/physical_block_size"); - -- final byte[] data = new byte[32]; -+ final byte[] data = new byte[NUMBER_BUFFER_SIZE]; - try (final FileInputStream fileIn = new FileInputStream(infoSource.toString())) - { - final int readCount = fileIn.read(data); -@@ -75,4 +80,38 @@ public class BlockSizeInfo - } - return blockSize; - } -+ -+ public static long getOptimalIoSize(final Path storageObj) -+ { -+ long optIoSize = DFLT_OPT_IO_SIZE; -+ try -+ { -+ final Path blockDevice = SymbolicLinkResolver.resolveSymLink(storageObj); -+ final Path infoSourceName = blockDevice.getFileName(); -+ final Path infoSource = Path.of("/sys/block", infoSourceName.toString(), "queue/optimal_io_size"); -+ -+ final byte[] data = new byte[NUMBER_BUFFER_SIZE]; -+ try (final FileInputStream fileIn = new FileInputStream(infoSource.toString())) -+ { -+ final int readCount = fileIn.read(data); -+ if (readCount > 0) -+ { -+ String numberStr = new String(data, 0, readCount); -+ numberStr = numberStr.trim(); -+ try -+ { -+ final long unboundedOptIoSize = Long.parseLong(numberStr); -+ optIoSize = MathUtils.bounds(MIN_OPT_IO_SIZE, unboundedOptIoSize, MAX_OPT_IO_SIZE); -+ } -+ catch (NumberFormatException ignored) -+ { -+ } -+ } -+ } -+ } -+ catch (IOException ignored) -+ { -+ } -+ return optIoSize; -+ } - } -diff --git a/server/src/main/java/com/linbit/SizeConv.java b/server/src/main/java/com/linbit/SizeConv.java ---- a/server/src/main/java/com/linbit/SizeConv.java -+++ b/server/src/main/java/com/linbit/SizeConv.java -@@ -16,6 +16,7 @@ public class SizeConv - public enum SizeUnit - { - UNIT_B, -+ UNIT_SECTORS, - UNIT_KiB, - UNIT_MiB, - UNIT_GiB, -@@ -41,6 +42,9 @@ public class SizeConv - case UNIT_B: - factor = FACTOR_B; - break; -+ case UNIT_SECTORS: -+ factor = FACTOR_SECTORS; -+ break; - case UNIT_KiB: - factor = FACTOR_KiB; - break; -@@ -111,6 +115,9 @@ public class SizeConv - case "b": - unit = SizeUnit.UNIT_B; - break; -+ case "s": -+ unit = SizeUnit.UNIT_SECTORS; -+ break; - case "k": - // fall-through - case "kb": -@@ -217,6 +224,9 @@ public class SizeConv - // Factor 1 - public static final BigInteger FACTOR_B = BigInteger.valueOf(1L); - -+ // Factor 512 -+ public static final BigInteger FACTOR_SECTORS = BigInteger.valueOf(512L); -+ - // Factor 1,024 - // Naming convention exception: SI unit capitalization rules - @SuppressWarnings("checkstyle:constantname") -diff --git a/server/src/main/java/com/linbit/linstor/layer/luks/LuksLayerSizeCalculator.java b/server/src/main/java/com/linbit/linstor/layer/luks/LuksLayerSizeCalculator.java ---- a/server/src/main/java/com/linbit/linstor/layer/luks/LuksLayerSizeCalculator.java -+++ b/server/src/main/java/com/linbit/linstor/layer/luks/LuksLayerSizeCalculator.java -@@ -1,27 +1,62 @@ - package com.linbit.linstor.layer.luks; - -+import com.linbit.SizeConv; -+import com.linbit.SizeConv.SizeUnit; - import com.linbit.exceptions.InvalidSizeException; -+import com.linbit.linstor.PriorityProps; -+import com.linbit.linstor.annotation.Nullable; -+import com.linbit.linstor.api.ApiConsts; -+import com.linbit.linstor.core.objects.AbsResource; -+import com.linbit.linstor.core.objects.AbsVolume; -+import com.linbit.linstor.core.objects.Resource; -+import com.linbit.linstor.core.objects.ResourceDefinition; -+import com.linbit.linstor.core.objects.ResourceGroup; -+import com.linbit.linstor.core.objects.Snapshot; -+import com.linbit.linstor.core.objects.SnapshotVolume; -+import com.linbit.linstor.core.objects.StorPool; -+import com.linbit.linstor.core.objects.Volume; -+import com.linbit.linstor.core.objects.VolumeDefinition; - import com.linbit.linstor.dbdrivers.DatabaseException; - import com.linbit.linstor.layer.AbsLayerSizeCalculator; -+import com.linbit.linstor.netcom.Peer; -+import com.linbit.linstor.propscon.InvalidKeyException; -+import com.linbit.linstor.propscon.ReadOnlyProps; - import com.linbit.linstor.security.AccessDeniedException; -+import com.linbit.linstor.storage.StorageConstants; - import com.linbit.linstor.storage.data.adapter.luks.LuksVlmData; - import com.linbit.linstor.storage.interfaces.categories.resource.VlmProviderObject; - import com.linbit.linstor.storage.kinds.DeviceLayerKind; - import com.linbit.linstor.storage.kinds.ExtTools; - import com.linbit.linstor.storage.kinds.ExtToolsInfo; - import com.linbit.linstor.storage.kinds.ExtToolsInfo.Version; -+import com.linbit.linstor.utils.layer.LayerVlmUtils; -+import com.linbit.utils.ShellUtils; -+import com.linbit.utils.SignedAlign; - - import javax.inject.Inject; - import javax.inject.Singleton; - -+import java.util.Iterator; -+import java.util.LinkedList; -+import java.util.List; -+import java.util.Set; -+import java.util.regex.Matcher; -+import java.util.regex.Pattern; -+ - @Singleton - public class LuksLayerSizeCalculator extends AbsLayerSizeCalculator> - { -+ public static final String LUKS2_OPT_METADATA_SIZE = "--luks2-metadata-size"; -+ public static final String LUKS2_OPT_KEYSLOTS_SIZE = "--luks2-keyslots-size"; -+ public static final String LUKS2_OPT_OFFSET = "--offset"; -+ private static final String LUKS2_OPT_ALIGN_PAYLOAD = "--align-payload"; -+ private static final long DFLT_LUKS2_METADATA_SIZE_IN_BYTES = 16L << 10; -+ private static final long DFLT_ALIGNMENT_1MIB_IN_BYTES = 1L << 20; -+ private static final long LUKS1_HEADER_SIZE_IN_KIB = 2L << 10; -+ private static final long LUKS2_HEADER_SIZE_IN_KIB = 16L << 10; -+ private static final long LUKS2_HEADER_SIZE_IN_BYTES = LUKS2_HEADER_SIZE_IN_KIB << 10; - -- // linstor calculates in KiB -- private static final int MIB = 1024; -- private static final int LUKS1_HEADER_SIZE = 2 * MIB; -- private static final int LUKS2_HEADER_SIZE = 16 * MIB; -+ private static final Pattern PATTERN_SIZE = Pattern.compile("(\\d+)([kKmMgGtT]i?[bB]?|[sS]|)"); - - @Inject - public LuksLayerSizeCalculator(AbsLayerSizeCalculatorInit initRef) -@@ -74,30 +109,266 @@ public class LuksLayerSizeCalculator extends AbsLayerSizeCalculator vlmDataRef) -- throws AccessDeniedException -+ throws AccessDeniedException, InvalidSizeException - { -- ExtToolsInfo cryptSetupInfo = vlmDataRef.getRscLayerObject() -+ @Nullable Peer peer = vlmDataRef.getRscLayerObject() - .getAbsResource() - .getNode() -- .getPeer(sysCtx) -- .getExtToolsManager() -+ .getPeer(sysCtx); -+ if (peer == null) -+ { -+ throw new InvalidSizeException( -+ "Could not calculate size of LUKS volume, since cryptsetup's version could not be determined", -+ null -+ ); -+ } -+ ExtToolsInfo cryptSetupInfo = peer.getExtToolsManager() - .getExtToolInfo(ExtTools.CRYPT_SETUP); -- long luksHeaderSize; -+ -+ final long luksHeaderSize; - if (cryptSetupInfo != null && cryptSetupInfo.isSupported()) - { - if (cryptSetupInfo.hasVersionOrHigher(new Version(2, 1))) - { -- luksHeaderSize = LUKS2_HEADER_SIZE; -+ luksHeaderSize = calcLuks2HeaderSize(vlmDataRef); - } - else - { -- luksHeaderSize = LUKS1_HEADER_SIZE; -+ luksHeaderSize = LUKS1_HEADER_SIZE_IN_KIB; - } - } - else - { -- luksHeaderSize = -1; -+ throw new InvalidSizeException( -+ "Could not calculate size of LUKS volume, since cryptsetup's version could not be determined", -+ null -+ ); - } - return luksHeaderSize; - } -+ -+ private long calcLuks2HeaderSize(VlmProviderObject vlmDataRef) throws AccessDeniedException -+ { -+ PriorityProps prioProps = getPrioProps(vlmDataRef); -+ -+ final @Nullable String userOptProp = prioProps.getProp( -+ ApiConsts.KEY_STOR_DRIVER_LUKS_FORMAT_OPTIONS, -+ ApiConsts.NAMESPC_STORAGE_DRIVER -+ ); -+ List userOptions = userOptProp != null ? -+ ShellUtils.shellSplit(userOptProp) : -+ new LinkedList<>(); -+ -+ final long alignedHeaderSizeInBytes; -+ final long unalignedHeaderSizeInBytes; -+ @Nullable Long cryptsetupOffsetInBytes = getLongOptionValue( -+ userOptions, -+ LUKS2_OPT_OFFSET, -+ SizeUnit.UNIT_SECTORS -+ ); -+ if (cryptsetupOffsetInBytes != null) -+ { -+ alignedHeaderSizeInBytes = cryptsetupOffsetInBytes; -+ } -+ else -+ { -+ @Nullable Long cryptsetupLuks2KeyslotsSize = getLongOptionValue( -+ userOptions, -+ LUKS2_OPT_KEYSLOTS_SIZE -+ ); -+ if (cryptsetupLuks2KeyslotsSize == null) -+ { -+ unalignedHeaderSizeInBytes = LUKS2_HEADER_SIZE_IN_BYTES; -+ } -+ else -+ { -+ @Nullable Long cryptsetupLuks2MetadataSize = getLongOptionValue( -+ userOptions, -+ LUKS2_OPT_METADATA_SIZE -+ ); -+ cryptsetupLuks2MetadataSize = cryptsetupLuks2MetadataSize == null ? -+ DFLT_LUKS2_METADATA_SIZE_IN_BYTES : -+ cryptsetupLuks2MetadataSize; -+ -+ unalignedHeaderSizeInBytes = 2 * cryptsetupLuks2MetadataSize + cryptsetupLuks2KeyslotsSize; -+ } -+ -+ long alignment = getAlignment(vlmDataRef, userOptions); -+ alignedHeaderSizeInBytes = new SignedAlign(alignment).ceiling(unalignedHeaderSizeInBytes); -+ } -+ return SizeConv.convert( -+ alignedHeaderSizeInBytes, -+ SizeUnit.UNIT_B, -+ SizeUnit.UNIT_KiB -+ ); -+ } -+ -+ private long getAlignment(VlmProviderObject vlmDataRef, List userOptions) throws AccessDeniedException -+ { -+ @Nullable Long cryptsetupAlignPayloadInBytes = getLongOptionValue( -+ userOptions, -+ LUKS2_OPT_ALIGN_PAYLOAD, -+ SizeUnit.UNIT_SECTORS -+ ); -+ -+ long alignment = DFLT_ALIGNMENT_1MIB_IN_BYTES; -+ if (cryptsetupAlignPayloadInBytes != null) -+ { -+ alignment = cryptsetupAlignPayloadInBytes; -+ } -+ else -+ { -+ final long maxOptIoSize = getMaxOptIoSize(vlmDataRef); -+ alignment = Math.max(alignment, maxOptIoSize); -+ } -+ return alignment; -+ } -+ -+ private long getMaxOptIoSize(VlmProviderObject vlmDataRef) throws InvalidKeyException, AccessDeniedException -+ { -+ long ret = 0; -+ Set storPoolSet = LayerVlmUtils.getStorPoolSet(vlmDataRef, sysCtx); -+ for (StorPool sp : storPoolSet) -+ { -+ @Nullable String strValue = sp.getProps(sysCtx) -+ .getProp( -+ StorageConstants.BLK_DEV_OPT_IO_SIZE, -+ StorageConstants.NAMESPACE_INTERNAL -+ ); -+ if (strValue != null) -+ { -+ try -+ { -+ long parsed = Long.parseLong(strValue); -+ ret = Math.max(parsed, ret); -+ } -+ catch (NumberFormatException ignored) -+ { -+ errorReporter.logWarning( -+ "LuksHeaderSize: Failed to parse '%s' from prop %s. Defaulting to 0 opt_io_size " + -+ "(no recommendation/hint)", -+ strValue, -+ StorageConstants.NAMESPACE_INTERNAL + "/" + StorageConstants.BLK_DEV_OPT_IO_SIZE -+ ); -+ } -+ } -+ } -+ return ret; -+ } -+ -+ private @Nullable Long getLongOptionValue(List userOptionsRef, String optRef) -+ { -+ return getLongOptionValue(userOptionsRef, optRef, SizeUnit.UNIT_B); -+ } -+ -+ @SuppressWarnings("checkstyle:magicnumber") -+ private @Nullable Long getLongOptionValue(List userOptionsRef, String optRef, SizeUnit dfltSizeUnit) -+ { -+ @Nullable Long ret = null; -+ @Nullable String val = findLastValue(userOptionsRef, optRef); -+ if (val != null && !val.isBlank()) -+ { -+ Matcher matcher = PATTERN_SIZE.matcher(val); -+ if (matcher.matches()) -+ { -+ try -+ { -+ ret = Long.parseLong(matcher.group(1)); -+ String unit = matcher.group(2); -+ SizeUnit sizeUnit; -+ if (unit.isBlank()) -+ { -+ sizeUnit = dfltSizeUnit; -+ } -+ else -+ { -+ boolean forcePowerOfTwo = unit.length() == 1 || unit.length() == 3; -+ sizeUnit = SizeUnit.parse(unit, forcePowerOfTwo); -+ } -+ -+ ret = SizeConv.convert(ret, sizeUnit, SizeUnit.UNIT_B); -+ } -+ catch (NumberFormatException ignored) -+ { -+ errorReporter.logWarning( -+ "LuksHeaderSize: Failed to parse '%s' from option '%s %s'.", -+ matcher.group(1), -+ optRef, -+ val -+ ); -+ } -+ } -+ } -+ return ret; -+ } -+ -+ private @Nullable String findLastValue(List userOptionsRef, String optRef) -+ { -+ Iterator it = userOptionsRef.iterator(); -+ @Nullable String val = null; -+ while (it.hasNext()) -+ { -+ String opt = it.next(); -+ if (opt.equals(optRef)) -+ { -+ if (it.hasNext()) -+ { -+ val = it.next(); -+ } -+ else -+ { -+ val = null; -+ } -+ } -+ else if (opt.startsWith(optRef + "=")) -+ { -+ val = opt.substring(optRef.length() + 1); -+ if (val.isBlank()) -+ { -+ val = null; -+ } -+ } -+ } -+ return val; -+ } -+ -+ private PriorityProps getPrioProps(VlmProviderObject vlmDataRef) throws AccessDeniedException -+ { -+ final AbsVolume vlm = vlmDataRef.getVolume(); -+ final AbsResource rsc = vlm.getAbsResource(); -+ final ResourceDefinition rscDfn = vlm.getResourceDefinition(); -+ final VolumeDefinition vlmDfn = vlm.getVolumeDefinition(); -+ final ResourceGroup rscGrp = rscDfn.getResourceGroup(); -+ -+ final ReadOnlyProps vlmProps; -+ final ReadOnlyProps rscProps; -+ if (vlm instanceof Volume) -+ { -+ vlmProps = ((Volume) vlm).getProps(sysCtx); -+ rscProps = ((Resource) rsc).getProps(sysCtx); -+ } -+ else -+ { -+ vlmProps = ((SnapshotVolume) vlm).getVlmProps(sysCtx); -+ rscProps = ((Snapshot) rsc).getRscProps(sysCtx); -+ } -+ -+ final PriorityProps prioProps = new PriorityProps( -+ vlmProps, -+ rscProps -+ ); -+ for (StorPool storPool : LayerVlmUtils.getStorPoolSet(vlmDataRef, sysCtx)) -+ { -+ prioProps.addProps(storPool.getProps(sysCtx)); -+ } -+ prioProps.addProps( -+ rsc.getNode().getProps(sysCtx), -+ vlmDfn.getProps(sysCtx), -+ rscDfn.getProps(sysCtx), -+ rscGrp.getVolumeGroupProps(sysCtx, vlmDfn.getVolumeNumber()), -+ rscGrp.getProps(sysCtx), -+ stltProps -+ ); -+ return prioProps; -+ } - } -diff --git a/server/src/main/java/com/linbit/linstor/layer/storage/BlockSizeConsts.java b/server/src/main/java/com/linbit/linstor/layer/storage/BlockSizeConsts.java ---- a/server/src/main/java/com/linbit/linstor/layer/storage/BlockSizeConsts.java -+++ b/server/src/main/java/com/linbit/linstor/layer/storage/BlockSizeConsts.java -@@ -13,4 +13,9 @@ public class BlockSizeConsts - - // Default value for the minimum_io_size value of non-storage layers - public static final long DFLT_SPECIAL_IO_SIZE = (1L << 12); -+ -+ // optimal_io_size may be 0 to indicate no recommendation. -+ public static final long MIN_OPT_IO_SIZE = 0; -+ public static final long DFLT_OPT_IO_SIZE = 0; -+ public static final long MAX_OPT_IO_SIZE = Long.MAX_VALUE; - } -diff --git a/server/src/main/java/com/linbit/linstor/storage/StorageConstants.java b/server/src/main/java/com/linbit/linstor/storage/StorageConstants.java ---- a/server/src/main/java/com/linbit/linstor/storage/StorageConstants.java -+++ b/server/src/main/java/com/linbit/linstor/storage/StorageConstants.java -@@ -11,6 +11,7 @@ public class StorageConstants - public static final String NAMESPACE_NVME = ApiConsts.NAMESPC_STORAGE_DRIVER + "/NVME"; - public static final String NAMESPACE_INTERNAL = NAMESPACE_STOR_DRIVER + "/internal/"; - -+ public static final String BLK_DEV_OPT_IO_SIZE = "optIoSize"; - public static final String BLK_DEV_MIN_IO_SIZE = "minIoSize"; - public static final String BLK_DEV_MIN_IO_SIZE_AUTO = "minIoSizeAuto"; - public static final String BLK_DEV_MAX_BIO_SIZE = "maxBioSize"; diff --git a/packages/system/linstor/images/piraeus-server/patches/fix-min-io-probe-device-wait.diff b/packages/system/linstor/images/piraeus-server/patches/fix-min-io-probe-device-wait.diff deleted file mode 100644 index bb6185b88..000000000 --- a/packages/system/linstor/images/piraeus-server/patches/fix-min-io-probe-device-wait.diff +++ /dev/null @@ -1,660 +0,0 @@ -diff --git a/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java b/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java ---- a/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java -+++ b/satellite/src/main/java/com/linbit/linstor/layer/storage/AbsStorageProvider.java -@@ -163,6 +163,8 @@ - } - - private static final long DFLT_WAIT_UNTIL_DEVICE_CREATED_TIMEOUT_IN_MS = 5000; -+ private static final int PROBE_VLM_ATTEMPTS = 3; -+ private static final long PROBE_VLM_RETRY_DELAY_IN_MS = 1000; - protected static final int DFLT_STRIPES = 1; - public static final long SIZE_OF_NOT_FOUND_STOR_POOL = ApiConsts.VAL_STOR_POOL_SPACE_NOT_FOUND; - -@@ -184,6 +186,7 @@ - protected final Set changedStoragePoolStrings = new HashSet<>(); - private final String typeDescr; - private final FileSystemWatch fsWatch; -+ private final Object probeVlmLock = new Object(); - protected final DeviceProviderKind kind; - private final DrbdInvalidateUtils drbdInvalidateUtils; - -@@ -2022,61 +2025,126 @@ - final StorPoolName storPoolObjName = storPoolObj.getName(); - errorReporter.logDebug("ENTER updateMinIoSize method: Storage pool \"%s\"", storPoolObjName.displayValue); - Collection> vlmProviderList = storPoolObj.getVolumes(storDriverAccCtx); -+ boolean haveInfo = false; - if (vlmProviderList != null && vlmProviderList.size() >= 1) - { - errorReporter.logDebug("updateMinIoSize: Have vlmProviderList with %d items", vlmProviderList.size()); - Iterator> vlmProviderIter = vlmProviderList.iterator(); -- boolean haveInfo = false; - while (vlmProviderIter.hasNext() && !haveInfo) - { - VlmProviderObject vlmProvider = vlmProviderIter.next(); - errorReporter.logDebug("updateMinIoSize: Have vlmProvider"); - final String storDevicePath = vlmProvider.getDevicePath(); -+ if (storDevicePath != null) -+ { -+ try -+ { -+ updateMinIoSizeByDevice(storPoolObj, storDevicePath, propsChange); -+ haveInfo = true; -+ } -+ catch (IOException ignored) -+ { -+ } -+ } -+ } -+ } -+ // An inactive shared LVM volume can be owned by another node; do not probe it by creating a volume. -+ if (!haveInfo && this instanceof ProbeVlmStorageProvider && -+ (vlmProviderList == null || vlmProviderList.isEmpty() || -+ kind == DeviceProviderKind.ZFS || kind == DeviceProviderKind.ZFS_THIN)) -+ { -+ updateMinIoSizeByProbeVlm((ProbeVlmStorageProvider) this, storPoolObj, propsChange); -+ } -+ errorReporter.logDebug("EXIT updateMinIoSize method"); -+ } -+ -+ private void updateMinIoSizeByProbeVlm( -+ ProbeVlmStorageProvider storPrv, -+ StorPool storPoolObj, -+ LocalPropsChangePojo propsChange -+ ) -+ throws AccessDeniedException -+ { -+ // Providers use a fixed probe name. Keep creation, probing and cleanup in one critical section. -+ synchronized (probeVlmLock) -+ { -+ boolean haveInfo = false; -+ boolean interrupted = false; -+ String lastFailure = ""; -+ int attempts = 0; -+ while (attempts < PROBE_VLM_ATTEMPTS && !haveInfo && !interrupted) -+ { -+ if (attempts > 0) -+ { -+ try -+ { -+ Thread.sleep(PROBE_VLM_RETRY_DELAY_IN_MS); -+ } -+ catch (InterruptedException exc) -+ { -+ Thread.currentThread().interrupt(); -+ lastFailure = "interrupted before the next probe attempt"; -+ break; -+ } -+ } -+ attempts++; -+ boolean cleaned = false; - try - { -- updateMinIoSizeByDevice(storPoolObj, storDevicePath, propsChange); -- haveInfo = true; -- } -- catch (IOException ignored) -- { -- } -- } -- } -- else -- { -- errorReporter.logDebug("updateMinIoSize: Don't have vlmProviderList, using temporary volumes"); -- if (this instanceof ProbeVlmStorageProvider) -- { -- final ProbeVlmStorageProvider storPrv = (ProbeVlmStorageProvider) this; -- try -- { -- final String storDevicePath = storPrv.createTmpProbeVlm(storPoolObj); -+ @Nullable String storDevicePath = storPrv.createTmpProbeVlm(storPoolObj); -+ if (storDevicePath == null) -+ { -+ lastFailure = "temporary volume has no device path"; -+ } -+ else -+ { -+ waitUntilDeviceCreated(storDevicePath, getWaitTimeoutAfterCreate(storPoolObj)); -+ updateMinIoSizeByDevice(storPoolObj, storDevicePath, propsChange); -+ haveInfo = true; -+ } -+ } -+ catch (StorageException | IOException exc) -+ { -+ lastFailure = exc.getMessage(); -+ interrupted = exc.getCause() instanceof InterruptedException; -+ errorReporter.logDebug("Temporary volume probe attempt %d failed: %s", attempts, lastFailure); -+ } -+ finally -+ { - try - { -- updateMinIoSizeByDevice(storPoolObj, storDevicePath, propsChange); -- } -- catch (IOException ignored) -- { -- } -- } -- catch (StorageException ignored) -- { -- errorReporter.logDebug("updateMinIoSize: Temporary volume creation failed"); -- } -- finally -- { -- try -- { - storPrv.deleteTmpProbeVlm(storPoolObj); -- } -- catch (StorageException ignored) -- { -- errorReporter.logDebug("updateMinIoSize: Temporary volume deletion failed"); -- } -- } -- } -- } -- errorReporter.logDebug("EXIT updateMinIoSize method"); -+ cleaned = true; -+ } -+ catch (StorageException exc) -+ { -+ interrupted |= exc.getCause() instanceof InterruptedException; -+ errorReporter.logWarning( -+ "Failed to delete temporary probe volume for storage pool '%s': %s", -+ storPoolObj.getName().displayValue, -+ exc.getMessage() -+ ); -+ } -+ } -+ if (interrupted) -+ { -+ Thread.currentThread().interrupt(); -+ } -+ if (!cleaned) -+ { -+ break; -+ } -+ } -+ if (!haveInfo) -+ { -+ errorReporter.logWarning( -+ "Could not determine block device information for storage pool '%s' after %d probe attempts: %s", -+ storPoolObj.getName().displayValue, -+ attempts, -+ lastFailure -+ ); -+ } -+ } - } - - private void updateMinIoSizeByDevice( -diff --git a/src/test/java/com/linbit/fsevent/ProbeVolumeDeviceWaitTest.java b/src/test/java/com/linbit/fsevent/ProbeVolumeDeviceWaitTest.java -new file mode 100644 ---- /dev/null -+++ b/src/test/java/com/linbit/fsevent/ProbeVolumeDeviceWaitTest.java -@@ -0,0 +1,467 @@ -+package com.linbit.fsevent; -+ -+import com.linbit.Platform; -+import com.linbit.linstor.core.identifier.StorPoolName; -+import com.linbit.linstor.core.objects.Resource; -+import com.linbit.linstor.core.objects.StorPool; -+import com.linbit.linstor.core.pojos.LocalPropsChangePojo; -+import com.linbit.linstor.layer.storage.AbsStorageProvider.AbsStorageProviderInit; -+import com.linbit.linstor.layer.storage.BlockSizeConsts; -+import com.linbit.linstor.layer.storage.lvm.LvmProvider; -+import com.linbit.linstor.layer.storage.zfs.ZfsThinProvider; -+import com.linbit.linstor.logging.ErrorReporter; -+import com.linbit.linstor.propscon.Props; -+import com.linbit.linstor.storage.StorageConstants; -+import com.linbit.linstor.storage.StorageException; -+import com.linbit.linstor.storage.interfaces.categories.resource.VlmProviderObject; -+ -+import java.io.IOException; -+import java.nio.file.Files; -+import java.nio.file.Path; -+import java.util.Collections; -+import java.util.List; -+import java.util.Map; -+import java.util.concurrent.Callable; -+import java.util.concurrent.CountDownLatch; -+import java.util.concurrent.ExecutorService; -+import java.util.concurrent.Executors; -+import java.util.concurrent.Future; -+import java.util.concurrent.ScheduledExecutorService; -+import java.util.concurrent.ScheduledFuture; -+import java.util.concurrent.TimeUnit; -+import java.util.concurrent.atomic.AtomicInteger; -+ -+import org.junit.After; -+import org.junit.Before; -+import org.junit.Rule; -+import org.junit.Test; -+import org.junit.rules.TemporaryFolder; -+import org.junit.rules.Timeout; -+ -+import static org.junit.Assert.assertEquals; -+import static org.junit.Assert.assertFalse; -+import static org.junit.Assert.assertTrue; -+import static org.junit.Assume.assumeTrue; -+import static org.mockito.Mockito.doReturn; -+import static org.mockito.Mockito.mock; -+import static org.mockito.Mockito.mockingDetails; -+import static org.mockito.Mockito.when; -+ -+public class ProbeVolumeDeviceWaitTest -+{ -+ @Rule -+ public final TemporaryFolder directory = new TemporaryFolder(); -+ -+ @Rule -+ public final Timeout timeout = Timeout.seconds(15); -+ -+ private FileSystemWatch watch; -+ private Path device; -+ private Path link; -+ private StorPool pool; -+ private ProbeProvider provider; -+ private ErrorReporter reporter; -+ private ScheduledExecutorService executor; -+ -+ @Before -+ public void setUp() throws Exception -+ { -+ assumeTrue(Platform.isLinux()); -+ device = directory.newFile("zd-test").toPath(); -+ link = directory.getRoot().toPath().resolve("probe"); -+ executor = Executors.newSingleThreadScheduledExecutor(); -+ reporter = mock(ErrorReporter.class); -+ watch = new FileSystemWatch(reporter); -+ watch.start(); -+ pool = mock(StorPool.class); -+ when(pool.getName()).thenReturn(new StorPoolName("data")); -+ when(pool.getVolumes(null)).thenReturn(Collections.emptyList()); -+ when(pool.getProps(null)).thenReturn(mock(Props.class)); -+ provider = new ProbeProvider(providerInit()); -+ } -+ -+ @After -+ public void tearDown() throws Exception -+ { -+ if (executor != null) -+ { -+ executor.shutdownNow(); -+ assertTrue(executor.awaitTermination(2, TimeUnit.SECONDS)); -+ } -+ if (watch != null) -+ { -+ watch.shutdown(false); -+ watch.awaitShutdown(2000); -+ watch.cancelAllWatchKeys(); -+ } -+ } -+ -+ @Test -+ public void delayedDevicePublishesPropertiesBeforeCleanup() throws Exception -+ { -+ provider.delayMillis = 50; -+ assertPublished(probe()); -+ assertCleaned(1); -+ assertEquals(0, warningCount()); -+ } -+ -+ @Test -+ public void existingProbeDevicePublishesProperties() throws Exception -+ { -+ assertPublished(probe()); -+ assertCleaned(1); -+ } -+ -+ @Test -+ public void missingDeviceTimesOutAndCleansEveryAttempt() throws Exception -+ { -+ provider.delayMillis = -1; -+ provider.timeoutMillis = 20; -+ assertTrue(probe().isEmpty()); -+ assertCleaned(3); -+ assertEquals(1, warningCount()); -+ } -+ -+ @Test -+ public void failedCreationStillCleansEveryAttempt() throws Exception -+ { -+ provider.failCreateTimes = Integer.MAX_VALUE; -+ assertTrue(probe().isEmpty()); -+ assertCleaned(3); -+ assertEquals(1, warningCount()); -+ } -+ -+ @Test -+ public void transientCreationFailureIsRetried() throws Exception -+ { -+ provider.failCreateTimes = 1; -+ provider.delayMillis = 50; -+ assertPublished(probe()); -+ assertCleaned(2); -+ assertEquals(0, warningCount()); -+ } -+ -+ @Test -+ public void nullProbePathDoesNotPublishProperties() throws Exception -+ { -+ provider.nullPath = true; -+ assertTrue(probe().isEmpty()); -+ assertCleaned(3); -+ assertEquals(1, warningCount()); -+ } -+ -+ @Test -+ public void nullFirstPathDoesNotSkipUsableVolume() throws Exception -+ { -+ Files.createSymbolicLink(link, device); -+ doReturn(List.of(volume(null), volume(link))).when(pool).getVolumes(null); -+ assertPublished(probe()); -+ assertEquals(0, provider.createCalls); -+ assertEquals(0, provider.deleteCalls); -+ } -+ -+ @Test -+ public void missingFirstPathDoesNotSkipUsableVolume() throws Exception -+ { -+ Files.createSymbolicLink(link, device); -+ doReturn(List.of(volume(link.resolveSibling("missing")), volume(link))).when(pool).getVolumes(null); -+ assertPublished(probe()); -+ assertEquals(0, provider.createCalls); -+ assertEquals(0, provider.deleteCalls); -+ } -+ -+ @Test -+ public void allNullPathsFallBackToProbe() throws Exception -+ { -+ doReturn(List.of(volume(null))).when(pool).getVolumes(null); -+ provider.delayMillis = 50; -+ assertPublished(probe()); -+ assertCleaned(1); -+ } -+ -+ @Test -+ public void allMissingPathsFallBackToProbe() throws Exception -+ { -+ doReturn(List.of( -+ volume(link.resolveSibling("missing-a")), volume(link.resolveSibling("missing-b")) -+ )).when(pool).getVolumes(null); -+ provider.delayMillis = 50; -+ assertPublished(probe()); -+ assertCleaned(1); -+ } -+ -+ @Test -+ public void cleanupFailureAfterSuccessIsReported() throws Exception -+ { -+ provider.failDelete = true; -+ assertPublished(probe()); -+ assertEquals(1, provider.createCalls); -+ assertEquals(1, provider.deleteCalls); -+ assertEquals(1, warningCount()); -+ } -+ -+ @Test -+ public void cleanupFailurePreventsAnotherCreationAttempt() throws Exception -+ { -+ provider.delayMillis = -1; -+ provider.timeoutMillis = 20; -+ provider.failDelete = true; -+ assertTrue(probe().isEmpty()); -+ assertEquals(1, provider.createCalls); -+ assertEquals(1, provider.deleteCalls); -+ assertEquals(2, warningCount()); -+ } -+ -+ @Test -+ public void interruptedDeviceWaitCleansUpAndPreservesInterrupt() throws Exception -+ { -+ provider.delayMillis = -1; -+ Thread callingThread = Thread.currentThread(); -+ executor.schedule(callingThread::interrupt, 50, TimeUnit.MILLISECONDS); -+ try -+ { -+ assertTrue(probe().isEmpty()); -+ assertTrue(callingThread.isInterrupted()); -+ assertCleaned(1); -+ assertEquals(1, warningCount()); -+ } -+ finally -+ { -+ Thread.interrupted(); -+ } -+ } -+ -+ @Test -+ public void interruptedRetryDelayDoesNotCreateAnotherProbe() throws Exception -+ { -+ provider.failCreateTimes = Integer.MAX_VALUE; -+ Thread callingThread = Thread.currentThread(); -+ executor.schedule(callingThread::interrupt, 100, TimeUnit.MILLISECONDS); -+ try -+ { -+ assertTrue(probe().isEmpty()); -+ assertTrue(callingThread.isInterrupted()); -+ assertCleaned(1); -+ assertEquals(1, warningCount()); -+ } -+ finally -+ { -+ Thread.interrupted(); -+ } -+ } -+ -+ @Test -+ public void interruptedCleanupPreservesInterruptAndStopsRetries() throws Exception -+ { -+ provider.failCreateTimes = Integer.MAX_VALUE; -+ provider.interruptDelete = true; -+ try -+ { -+ assertTrue(probe().isEmpty()); -+ assertTrue(Thread.currentThread().isInterrupted()); -+ assertEquals(1, provider.createCalls); -+ assertEquals(1, provider.deleteCalls); -+ } -+ finally -+ { -+ Thread.interrupted(); -+ } -+ } -+ -+ @Test -+ public void concurrentProbesDoNotShareTheTemporaryVolume() throws Exception -+ { -+ provider.creationDelayMillis = 200; -+ ExecutorService callers = Executors.newFixedThreadPool(2); -+ CountDownLatch ready = new CountDownLatch(2); -+ CountDownLatch start = new CountDownLatch(1); -+ Callable call = () -> -+ { -+ ready.countDown(); -+ start.await(); -+ return probe(); -+ }; -+ try -+ { -+ Future first = callers.submit(call); -+ Future second = callers.submit(call); -+ assertTrue(ready.await(2, TimeUnit.SECONDS)); -+ start.countDown(); -+ assertFalse(first.get(10, TimeUnit.SECONDS).isEmpty()); -+ assertFalse(second.get(10, TimeUnit.SECONDS).isEmpty()); -+ assertEquals(1, provider.peakActiveProbes.get()); -+ assertCleaned(2); -+ } -+ finally -+ { -+ callers.shutdownNow(); -+ assertTrue(callers.awaitTermination(2, TimeUnit.SECONDS)); -+ } -+ } -+ -+ @Test -+ public void inactiveLvmVolumesDoNotCreateProbe() throws Exception -+ { -+ doReturn(List.of(volume(null))).when(pool).getVolumes(null); -+ LvmProvider lvm = new LvmProvider(providerInit()) -+ { -+ @Override -+ public String createTmpProbeVlm(StorPool ignored) -+ { -+ throw new AssertionError("Inactive shared LVM volumes must not trigger a probe volume"); -+ } -+ -+ @Override -+ public void deleteTmpProbeVlm(StorPool ignored) -+ { -+ throw new AssertionError("No probe volume was created"); -+ } -+ }; -+ LocalPropsChangePojo changes = new LocalPropsChangePojo(); -+ lvm.updateMinIoSize(pool, changes); -+ assertTrue(changes.isEmpty()); -+ } -+ -+ private AbsStorageProviderInit providerInit() -+ { -+ return new AbsStorageProviderInit( -+ reporter, null, null, null, null, null, null, null, null, null, watch, null, null, null -+ ); -+ } -+ -+ private LocalPropsChangePojo probe() throws Exception -+ { -+ LocalPropsChangePojo changes = new LocalPropsChangePojo(); -+ provider.updateMinIoSize(pool, changes); -+ return changes; -+ } -+ -+ private void assertPublished(LocalPropsChangePojo changesRef) -+ { -+ String namespace = StorageConstants.NAMESPACE_INTERNAL + '/'; -+ assertEquals(Map.of( -+ namespace + StorageConstants.BLK_DEV_MIN_IO_SIZE, Long.toString(BlockSizeConsts.DFLT_IO_SIZE), -+ namespace + StorageConstants.BLK_DEV_OPT_IO_SIZE, Long.toString(BlockSizeConsts.DFLT_OPT_IO_SIZE) -+ ), changesRef.changedStorPoolProps.get(pool.getName())); -+ } -+ -+ private void assertCleaned(int attemptsRef) -+ { -+ assertEquals(attemptsRef, provider.createCalls); -+ assertEquals(attemptsRef, provider.deleteCalls); -+ assertFalse(Files.exists(link)); -+ } -+ -+ @SuppressWarnings("unchecked") -+ private VlmProviderObject volume(Path pathRef) -+ { -+ VlmProviderObject volume = mock(VlmProviderObject.class); -+ when(volume.getDevicePath()).thenReturn(pathRef == null ? null : pathRef.toString()); -+ return volume; -+ } -+ -+ private long warningCount() -+ { -+ return mockingDetails(reporter).getInvocations().stream() -+ .filter(invocation -> invocation.getMethod().getName().equals("logWarning")) -+ .count(); -+ } -+ -+ private class ProbeProvider extends ZfsThinProvider -+ { -+ private long delayMillis; -+ private long creationDelayMillis; -+ private long timeoutMillis = 4000; -+ private int failCreateTimes; -+ private boolean nullPath; -+ private boolean failDelete; -+ private boolean interruptDelete; -+ private int createCalls; -+ private int deleteCalls; -+ private ScheduledFuture create; -+ private final AtomicInteger activeProbes = new AtomicInteger(); -+ private final AtomicInteger peakActiveProbes = new AtomicInteger(); -+ -+ ProbeProvider(AbsStorageProviderInit initRef) -+ { -+ super(initRef); -+ } -+ -+ @Override -+ protected long getWaitTimeoutAfterCreate(StorPool ignored) -+ { -+ return timeoutMillis; -+ } -+ -+ @Override -+ public String createTmpProbeVlm(StorPool ignored) throws StorageException -+ { -+ peakActiveProbes.accumulateAndGet(activeProbes.incrementAndGet(), Math::max); -+ createCalls++; -+ if (creationDelayMillis > 0) -+ { -+ try -+ { -+ Thread.sleep(creationDelayMillis); -+ } -+ catch (InterruptedException exc) -+ { -+ throw new StorageException("Test probe creation interrupted", exc); -+ } -+ } -+ if (failCreateTimes > 0) -+ { -+ failCreateTimes--; -+ throw new StorageException("Probe creation failed"); -+ } -+ if (nullPath) -+ { -+ return null; -+ } -+ if (delayMillis == 0) -+ { -+ try -+ { -+ Files.createSymbolicLink(link, device); -+ } -+ catch (IOException exc) -+ { -+ throw new StorageException("Test device creation failed", exc); -+ } -+ } -+ else if (delayMillis > 0) -+ { -+ create = executor.schedule(() -> Files.createSymbolicLink(link, device), -+ delayMillis, TimeUnit.MILLISECONDS); -+ } -+ return link.toString(); -+ } -+ -+ @Override -+ public void deleteTmpProbeVlm(StorPool ignored) throws StorageException -+ { -+ deleteCalls++; -+ if (interruptDelete) -+ { -+ throw new StorageException("Probe cleanup interrupted", new InterruptedException()); -+ } -+ if (failDelete) -+ { -+ throw new StorageException("Probe cleanup failed"); -+ } -+ try -+ { -+ if (create != null && !create.cancel(false)) -+ { -+ create.get(); -+ } -+ Files.deleteIfExists(link); -+ activeProbes.decrementAndGet(); -+ } -+ catch (Exception exc) -+ { -+ throw new StorageException("Test device cleanup failed", exc); -+ } -+ } -+ } -+} diff --git a/packages/system/linstor/images/piraeus-server/patches/native-protoc.diff b/packages/system/linstor/images/piraeus-server/patches/native-protoc.diff index f3e601698..fa8596177 100644 --- a/packages/system/linstor/images/piraeus-server/patches/native-protoc.diff +++ b/packages/system/linstor/images/piraeus-server/patches/native-protoc.diff @@ -1,8 +1,8 @@ diff --git a/build.gradle b/build.gradle -index 75b7f64..d2dfec9 100644 +index 9df6c840cdc0..9894a436712c 100644 --- a/build.gradle +++ b/build.gradle -@@ -37,6 +37,9 @@ configurations { +@@ -34,6 +34,9 @@ configurations { } def protobufVersion = '31.1' @@ -12,7 +12,7 @@ index 75b7f64..d2dfec9 100644 def protoc = file("${projectDir}/tools/protoc-" + protobufVersion + '/bin/protoc') def logbackConfig = "logback.xml" def jvmOpts = ["-Xms32M"] -@@ -577,7 +580,7 @@ tasks.register('buildTestCommands', Exec) { +@@ -650,7 +653,7 @@ tasks.register('buildTestCommands', Exec) { tasks.register('unzipProtoc', Copy) { dependsOn(downloadProtoc) @@ -21,7 +21,7 @@ index 75b7f64..d2dfec9 100644 def outputDir = file("tools/protoc-" + protobufVersion) from zipTree(zipFile) -@@ -591,7 +594,7 @@ tasks.register('unzipProtoc', Copy) { +@@ -664,7 +667,7 @@ tasks.register('unzipProtoc', Copy) { tasks.register('downloadProtoc') { outputs.dir "tools" @@ -30,7 +30,7 @@ index 75b7f64..d2dfec9 100644 outputs.file protozip doLast { -@@ -599,7 +602,7 @@ tasks.register('downloadProtoc') { +@@ -672,7 +675,7 @@ tasks.register('downloadProtoc') { mkdir "tools" println "downloading protoc..." new URL('https://github.com/google/protobuf/releases/download/v' diff --git a/packages/system/linstor/images/piraeus-server/patches/retry-adjust-after-stale-bitmap.diff b/packages/system/linstor/images/piraeus-server/patches/retry-adjust-after-stale-bitmap.diff index 009591674..e5e0ec013 100644 --- a/packages/system/linstor/images/piraeus-server/patches/retry-adjust-after-stale-bitmap.diff +++ b/packages/system/linstor/images/piraeus-server/patches/retry-adjust-after-stale-bitmap.diff @@ -1,28 +1,27 @@ diff --git a/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java b/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java -index 5627d1be8..ece191292 100644 +index 0a19f875b702..d44e19b70d88 100644 --- a/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java +++ b/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java -@@ -42,6 +42,9 @@ import java.util.Arrays; - import java.util.List; +@@ -45,6 +45,8 @@ import java.util.List; + import java.util.Set; import java.util.concurrent.ArrayBlockingQueue; import java.util.concurrent.TimeUnit; -+import java.nio.charset.StandardCharsets; +import java.util.regex.Matcher; +import java.util.regex.Pattern; import java.util.stream.Collectors; @Singleton -@@ -56,6 +58,9 @@ public class DrbdAdm - - public static final int WAIT_CONNECT_RES_TIME = 10; - private static final long DOWN_WAIT_TIMEOUT_SEC = 5; +@@ -68,6 +70,9 @@ public class DrbdAdm + private static final long SECONDARY_RETRY_INITIAL_BACKOFF_MS = 250L; + private static final long SECONDARY_RETRY_MAX_BACKOFF_MS = 2_000L; + private static final String DEVICE_HELD_OPEN_ERR_MSG = "Device is held open by someone"; + private static final long FORCE_DETACH_RETRY_WAIT_MS = 250; + private static final String BITMAP_LEAK_ERR_MSG = "already has a bitmap, this should not happen"; + private static final Pattern BITMAP_LEAK_MINOR_PATTERN = Pattern.compile("\\bminor\\s+(\\d+)\\b"); private final ExtCmdFactory extCmdFactory; - private final AccessContext sysCtx; -@@ -131,8 +136,38 @@ public class DrbdAdm + private final StltConfigAccessor stltCfgAccessor; +@@ -143,8 +148,38 @@ public class DrbdAdm // command.add(resName); command.add(drbdRscData.getSuffixedResourceName()); // execute(Arrays.asList("drbdsetup", "show", drbdRscData.getSuffixedResourceName())); @@ -35,7 +34,7 @@ index 5627d1be8..ece191292 100644 + ExtCmd extCmd = extCmdFactory.create(); + if (Platform.isWindows()) + { -+ extCmd.setTimeout(TimeoutType.WAIT, 5 * 60 * 1000); ++ extCmd.setTimeout(TimeoutType.WAIT, DFLT_WINDOWS_TIMEOUT_MS); + } + + OutputData outputData = extCmd.pipeExec(ProcessBuilder.Redirect.from(nullDevice), commandArr); @@ -63,8 +62,8 @@ index 5627d1be8..ece191292 100644 drbdRscData.setAdjustRequired(false); } -@@ -805,6 +840,75 @@ public class DrbdAdm - } +@@ -955,6 +990,75 @@ public class DrbdAdm + return ret; } + static boolean isBitmapLeakOnAttach(OutputData outputData) diff --git a/packages/system/linstor/images/piraeus-server/patches/retry-secondary-after-mkfs.diff b/packages/system/linstor/images/piraeus-server/patches/retry-secondary-after-mkfs.diff deleted file mode 100644 index f213e9037..000000000 --- a/packages/system/linstor/images/piraeus-server/patches/retry-secondary-after-mkfs.diff +++ /dev/null @@ -1,104 +0,0 @@ -diff --git a/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java b/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java -index 71c1df6..f91903b 100644 ---- a/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java -+++ b/satellite/src/main/java/com/linbit/linstor/layer/drbd/utils/DrbdAdm.java -@@ -62,6 +62,10 @@ public class DrbdAdm - private static final long FORCE_DETACH_RETRY_WAIT_MS = 250; - private static final String BITMAP_LEAK_ERR_MSG = "already has a bitmap, this should not happen"; - private static final Pattern BITMAP_LEAK_MINOR_PATTERN = Pattern.compile("\\bminor\\s+(\\d+)\\b"); -+ private static final int SECONDARY_RETRY_MAX_ATTEMPTS = 7; -+ private static final long SECONDARY_RETRY_INITIAL_BACKOFF_MS = 250L; -+ private static final long SECONDARY_RETRY_MAX_BACKOFF_MS = 2000L; -+ private static final String DEVICE_HELD_OPEN_ERR_MSG = "Device is held open by someone"; - - private final ExtCmdFactory extCmdFactory; - private final AccessContext sysCtx; -@@ -945,6 +949,79 @@ public class DrbdAdm - return true; - } - -+ /** -+ * Switches a resource to secondary mode after mkfs, retrying when DRBD reports the device as held -+ * open by another process. On Talos and other systems with active block-device probing -+ * (machined/udev), the kernel uevent for a freshly-promoted device may cause an external probe -+ * to open the device just before LINSTOR demotes it back to Secondary, producing -+ * "Device is held open by someone" (errno -12) and aborting resource initialization. -+ * Probes typically release the fd within milliseconds, so a short retry with exponential backoff -+ * is sufficient to win the race without changing the contract of {@link #secondary}. -+ */ -+ public void secondaryAfterMkfs(DrbdRscData drbdRscData) throws ExtCmdFailedException -+ { -+ String[] commandArr = new String[] { -+ DRBDADM_UTIL, "-vvv", "secondary", drbdRscData.getSuffixedResourceName() -+ }; -+ -+ File nullDevice = new File(Platform.nullDevice()); -+ -+ OutputData lastOutput = null; -+ for (int attempt = 1; attempt <= SECONDARY_RETRY_MAX_ATTEMPTS; attempt++) -+ { -+ try -+ { -+ ExtCmd extCmd = extCmdFactory.create(); -+ if (Platform.isWindows()) -+ { -+ extCmd.setTimeout(TimeoutType.WAIT, 5 * 60 * 1000); -+ } -+ -+ lastOutput = extCmd.pipeExec(ProcessBuilder.Redirect.from(nullDevice), commandArr); -+ if (lastOutput.exitCode == 0) -+ { -+ return; -+ } -+ if (!isDeviceHeldOpenError(lastOutput) || attempt == SECONDARY_RETRY_MAX_ATTEMPTS) -+ { -+ throw new ExtCmdFailedException(commandArr, lastOutput); -+ } -+ } -+ catch (ChildProcessTimeoutException timeoutExc) -+ { -+ throw new ExtCmdFailedException(commandArr, timeoutExc); -+ } -+ catch (IOException ioExc) -+ { -+ throw new ExtCmdFailedException(commandArr, ioExc); -+ } -+ -+ long sleepMs = Math.min( -+ SECONDARY_RETRY_INITIAL_BACKOFF_MS * (1L << (attempt - 1)), -+ SECONDARY_RETRY_MAX_BACKOFF_MS -+ ); -+ try -+ { -+ Thread.sleep(sleepMs); -+ } -+ catch (InterruptedException ie) -+ { -+ Thread.currentThread().interrupt(); -+ throw new ExtCmdFailedException(commandArr, lastOutput); -+ } -+ } -+ } -+ -+ static boolean isDeviceHeldOpenError(OutputData outputData) -+ { -+ if (outputData == null || outputData.stderrData == null) -+ { -+ return false; -+ } -+ String stderr = new String(outputData.stderrData, StandardCharsets.UTF_8); -+ return stderr.contains(DEVICE_HELD_OPEN_ERR_MSG); -+ } -+ - public static class DrbdPrimary implements AutoCloseable - { - private final DrbdAdm drbdAdm; -@@ -963,7 +1040,7 @@ public class DrbdAdm - @Override - public void close() throws ExtCmdFailedException - { -- drbdAdm.secondary(drbdRscData); -+ drbdAdm.secondaryAfterMkfs(drbdRscData); - } - } - } diff --git a/packages/system/linstor/images/piraeus-server/patches/snapshot-restore-drbd-initialized.diff b/packages/system/linstor/images/piraeus-server/patches/snapshot-restore-drbd-initialized.diff deleted file mode 100644 index 7c72aab69..000000000 --- a/packages/system/linstor/images/piraeus-server/patches/snapshot-restore-drbd-initialized.diff +++ /dev/null @@ -1,67 +0,0 @@ -diff --git a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java -index a639c27cc..4dabc1a83 100644 ---- a/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java -+++ b/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlSnapshotRestoreApiCallHandler.java -@@ -434,26 +434,39 @@ public class CtrlSnapshotRestoreApiCallHandler - } - } - -- /* -- * When a snapshot was received as a backup, it is marked to re-create its local metadata. Before the -- * snapshot was sent the new DRBD_INITIALIZED flag and the VlmDfn property InitialUpToDateOn were -+ /** -+ *

Sets the {@value InternalApiConsts#KEY_LINSTOR_DRBD_INITIAL_UPTODATE_ON} property to the nodeName -+ * of the first Snapshot (which has to have a disk) of the given SnapshotDefinition.

-+ * -+ *

If this SnapshotDefinition is a backup, we also unset the {@link VolumeDefinition.Flags#DRBD_INITIALIZED} -+ * flag. Otherwise we ensure the same flag is set.

-+ * -+ *

Backup path: When a snapshot was received as a backup, it is marked to re-create its local metadata. -+ * Before the snapshot was sent the new DRBD_INITIALIZED flag and the VlmDfn property InitialUpToDateOn were - * cleared. If DRBD_INITIALZED would be set, no peer would even read the property (on purpose). - * Without the property, no peer would think of itself as a winner, thus the DRBD resource would never - * become UpToDate. - * To avoid this, this method sets the InitialUpToDateOn property to let the first resource win the -- * race, so it can become UpToDate and therefore also pull the other peers eventually into UpToDate state. -+ * race, so it can become UpToDate and therefore also pull the other peers eventually into UpToDate state.

- * -- * If the given SnapDfn had also non-backup snapshots, those should be able to keep their metadata. -+ *

If the given SnapDfn had also non-backup snapshots, those should be able to keep their metadata. - * In that case we do not have to do anything. If however all snapshots are descendants from backups - * we must pick one and decide the UpToDate race upfront so that the winner can set its local DRBD -- * as UpToDate, which also triggers the controller to set the DRBD_INITIALZED flag eventually. -+ * as UpToDate, which also triggers the controller to set the DRBD_INITIALZED flag eventually.

-+ * -+ *

Non-Backup path: When a regular snapshot (i.e. not from a backup) is restored, we can or even should -+ * assume that the backing snapshot LVs have correct metadata which do not need be be initialized. When -+ * a SnapshotVolumeDefinition is created it unfortunately does not properly store the VolumeDefinition's flags, -+ * therefore the restored VolumeDefinition will also not have the DRBD_INITIALIZED flag set, although it -+ * refers to a device with healthy and UpToDate metadata. To fix this issue we simply set the flag in this case.

- */ - private void checkDrbdInitializedState(ResourceDefinition rscDfnRef) throws AccessDeniedException - { - AccessContext accCtx = peerAccCtx.get(); -- if (DrbdLayerUtils.isForceInitialSyncSet(accCtx, rscDfnRef) && rscDfnRef.getDiskfulCount(accCtx) > 0) -+ if (rscDfnRef.getDiskfulCount(accCtx) > 0) - { - String winnerNodeName = rscDfnRef.getDiskfulResources(accCtx).get(0).getNode().getName().value; -+ boolean uninitializeDrbd = DrbdLayerUtils.isForceInitialSyncSet(accCtx, rscDfnRef); - Iterator vlmDfnIt = rscDfnRef.iterateVolumeDfn(accCtx); - while (vlmDfnIt.hasNext()) - { -@@ -462,7 +475,14 @@ public class CtrlSnapshotRestoreApiCallHandler - { - vlmDfn.getProps(accCtx) - .setProp(InternalApiConsts.KEY_LINSTOR_DRBD_INITIAL_UPTODATE_ON, winnerNodeName); -- vlmDfn.getFlags().disableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED); -+ if (uninitializeDrbd) -+ { -+ vlmDfn.getFlags().disableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED); -+ } -+ else -+ { -+ vlmDfn.getFlags().enableFlags(accCtx, VolumeDefinition.Flags.DRBD_INITIALIZED); -+ } - } - catch (InvalidKeyException | InvalidValueException exc) - { From 1619cb4eb94d2d1f82b0a44e59a73c591766cf5c Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 15:42:32 +0500 Subject: [PATCH 475/518] ci: keep the CI-only Go tools from escalating builds and e2e cmd/backport-batch and cmd/api-gate are built into no image and run nowhere in the e2e sandbox, yet a change to either rebuilt every image and ran the full Chainsaw suite, and widened a package change made in the same pull request to everything. Every other directory under cmd/ is a shipped binary and keeps escalating. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- hack/build-matrix.sh | 8 +++++++- hack/build-matrix_test.bats | 17 +++++++++++++++++ hack/select-e2e.sh | 13 +++++++++++-- hack/select-e2e_test.bats | 17 +++++++++++++++++ 4 files changed, 52 insertions(+), 3 deletions(-) diff --git a/hack/build-matrix.sh b/hack/build-matrix.sh index 95e894a7f..d15267631 100755 --- a/hack/build-matrix.sh +++ b/hack/build-matrix.sh @@ -67,7 +67,13 @@ if [ -z "$CHANGED" ] || [ "$CHANGED" = "FULL" ]; then exit 0 fi -if grep -qE "$full_rebuild_pattern" "$CHANGED"; then +# CI-only Go tools that no image builds. Dropped before the full-rebuild check, +# whose cmd/ would otherwise rebuild everything for an edit no image contains. +# hack/select-e2e.sh skips the same two, and a diff made only of them and docs/ +# never reaches this script's build jobs (the plan job's `code` is false). +tooling_pattern='^cmd/(backport-batch|api-gate)/' + +if grep -vE "$tooling_pattern" "$CHANGED" | grep -qE "$full_rebuild_pattern"; then emit_json "$units" exit 0 fi diff --git a/hack/build-matrix_test.bats b/hack/build-matrix_test.bats index 44a9f2c04..c11f2707f 100644 --- a/hack/build-matrix_test.bats +++ b/hack/build-matrix_test.bats @@ -97,6 +97,23 @@ load test_helper rm -f "$tmp" } +@test "the CI-only tools do not force the full matrix" { + tmp=$(mktemp) + # No image builds cmd/backport-batch or cmd/api-gate, so they select nothing + # and leave the rest of the diff to its own rules. + printf '%s\n' cmd/backport-batch/report_test.go cmd/api-gate/main.go > "$tmp" + out=$(hack/build-matrix.sh "$tmp") + [ "$out" = '[]' ] + printf '%s\n' cmd/backport-batch/main.go packages/apps/mariadb/Makefile > "$tmp" + out=$(hack/build-matrix.sh "$tmp") + [ "$out" = '["packages/apps/mariadb"]' ] + # Every other cmd/ directory is a shipped binary and still rebuilds all. + printf '%s\n' cmd/backport-batch/main.go cmd/cozystack-api/main.go > "$tmp" + out=$(hack/build-matrix.sh "$tmp") + [ "$out" = "$(hack/build-matrix.sh)" ] + rm -f "$tmp" +} + @test "a package with no image target selects nothing" { tmp=$(mktemp) # postgres ships no in-repo image build, so it is not a build unit. diff --git a/hack/select-e2e.sh b/hack/select-e2e.sh index fcb7651e3..59db4fe25 100755 --- a/hack/select-e2e.sh +++ b/hack/select-e2e.sh @@ -8,8 +8,9 @@ # Output: # - empty no E2E impact, decided by one of the rules that select # nothing: step 1 for docs/, dashboards/ and *.md, step 2 for -# a *.disabled Chainsaw suite and for an -# examples/backups// with no suite, and +# a *.disabled Chainsaw suite, for an +# examples/backups// with no suite and for the CI-only +# tools cmd/backport-batch/ and cmd/api-gate/, and # inert_config_pattern for repo meta and agent config # - selected per the PackageSource dependency graph # - full list any path that affects all tests, OR an unrecognised @@ -497,6 +498,14 @@ while IFS= read -r file || [ -n "$file" ]; do echo "select-e2e: '$file' belongs to no single backup walkthrough — escalating to the full suite" >&2 trigger_full=1 continue ;; + cmd/backport-batch/*|cmd/api-gate/*) + # CI-only Go tools. No image builds them and nothing in the e2e sandbox + # runs them, so an edit cannot regress a suite. Matched here because + # full_suite_pattern takes all of cmd/ and is checked before the inert + # list. The plan job's `code` output and e2e-fork.yaml skip e2e for a + # diff made only of these and docs/; this arm keeps them from escalating + # a diff that also touches something else. + continue ;; esac # 3. Full-suite trigger diff --git a/hack/select-e2e_test.bats b/hack/select-e2e_test.bats index 6ce198830..f79d6b05c 100644 --- a/hack/select-e2e_test.bats +++ b/hack/select-e2e_test.bats @@ -202,6 +202,23 @@ assert_full_suite() { rm -rf "$tmp" } +@test "the CI-only tools select nothing and do not escalate the rest" { + tmp=$(mktemp -d) + cp -r packages/core/platform/sources "$tmp/sources" + printf '%s\n' cmd/backport-batch/report_test.go cmd/api-gate/main.go > "$tmp/diff" + output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources") + [ -z "$output" ] + # Beside a package, the package alone decides the selection. + printf '%s\n' cmd/backport-batch/main.go packages/apps/postgres/values.yaml > "$tmp/diff" + output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources") + assert_selection "a CI-only tool must not widen a package selection" "$output" "postgres" + # Any other cmd/ directory is a shipped binary and still escalates. + echo "cmd/cozystack-api/main.go" > "$tmp/diff" + output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources" 2>/dev/null) + assert_full_suite "$output" + rm -rf "$tmp" +} + @test "kubernetes-application maps to the two kubernetes suites" { tmp=$(mktemp -d) cp -r packages/core/platform/sources "$tmp/sources" From 48eae9ad361206355ddb8224470ab5053cedd0e3 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Thu, 12 Nov 2026 15:43:05 +0500 Subject: [PATCH 476/518] ci: run only the backport-batch tests for a backport-batch change A pull request that changed only cmd/backport-batch built every image, installed the platform for e2e and ran the whole unit lane, so a one-line test fix waited hours on suites that cannot exercise it. The backport-batch tests themselves ran on every code pull request, although only a change to the tool, to what it imports or to the pages its tests read can break them. Their inputs are listed beside the tool and checked against its import graph, because a hand-kept list that fell behind would switch the tests off without anyone noticing. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- .github/workflows/e2e-fork.yaml | 21 +++-- .github/workflows/pull-requests.yaml | 51 +++++++++--- Makefile | 5 +- cmd/backport-batch/inputs_test.go | 112 ++++++++++++++++++++++++++ cmd/backport-batch/test-inputs.txt | 26 ++++++ docs/agents/e2e-testing.md | 2 +- docs/agents/overview.md | 2 +- hack/backport-batch-tests-needed.bats | 87 ++++++++++++++++++++ hack/backport-batch-tests-needed.sh | 44 ++++++++++ hack/downstream-trigger-map.bats | 16 +++- hack/gate-concurrency-contract.bats | 40 ++++++++- 11 files changed, 378 insertions(+), 28 deletions(-) create mode 100644 cmd/backport-batch/inputs_test.go create mode 100644 cmd/backport-batch/test-inputs.txt create mode 100644 hack/backport-batch-tests-needed.bats create mode 100755 hack/backport-batch-tests-needed.sh diff --git a/.github/workflows/e2e-fork.yaml b/.github/workflows/e2e-fork.yaml index 65b78230d..03c568395 100644 --- a/.github/workflows/e2e-fork.yaml +++ b/.github/workflows/e2e-fork.yaml @@ -61,7 +61,8 @@ jobs: # base-repo PR file list — NOT from the presence/absence of an artifact the # fork's own build controls. Fail-closed if the fork build failed or the # pr-patch artifact is missing; green-and-stop only for a PR whose file list is - # entirely under docs/ (verified against the base-repo API). + # entirely under docs/ or the CI-only tools cmd/backport-batch/ and + # cmd/api-gate/ (verified against the base-repo API). resolve: name: Resolve fork PR # Only fork PRs: head_repository differs from the base repository. Same-repo @@ -266,15 +267,21 @@ jobs: await decide('failure', 'PR file list hit the 3000-file API cap — cannot verify docs-only'); return; } + // Paths with nothing to build and nothing for e2e to install: docs, + // and the CI-only Go tools no image builds. pull-requests.yaml's + // plan job answers the same question with the same prefixes, and + // hack/gate-concurrency-contract.bats holds the two lists equal. + // // A rename reports the NEW path in `filename` and the old one in // `previous_filename`. Checking only `filename` would read a // packages/… -> docs/… rename as docs-only, when it in fact removes - // a code path. BOTH sides must be under docs/ to count as docs-only. - const isDocs = p => typeof p === 'string' && p.startsWith('docs/'); - const hasNonDocs = files.some(f => - !isDocs(f.filename) || (f.previous_filename && !isDocs(f.previous_filename))); - if (!hasNonDocs) { - await decide('success', 'docs-only PR (verified via PR file list) — e2e not required'); + // a code path. BOTH sides must be under these prefixes to count. + const noRuntimePrefixes = ['docs/', 'cmd/backport-batch/', 'cmd/api-gate/']; + const isNoRuntime = p => typeof p === 'string' && noRuntimePrefixes.some(x => p.startsWith(x)); + const hasRuntime = files.some(f => + !isNoRuntime(f.filename) || (f.previous_filename && !isNoRuntime(f.previous_filename))); + if (!hasRuntime) { + await decide('success', 'docs or CI-only tools (verified via PR file list) — e2e not required'); return; } diff --git a/.github/workflows/pull-requests.yaml b/.github/workflows/pull-requests.yaml index 24ee473ef..62cbbb1ad 100644 --- a/.github/workflows/pull-requests.yaml +++ b/.github/workflows/pull-requests.yaml @@ -75,6 +75,8 @@ jobs: statuses: write outputs: code: ${{ steps.p.outputs.code }} + docs: ${{ steps.p.outputs.docs }} + backport_batch: ${{ steps.p.outputs.backport_batch }} matrix: ${{ steps.p.outputs.matrix }} any: ${{ steps.p.outputs.any }} touched: ${{ steps.p.outputs.touched }} @@ -167,11 +169,27 @@ jobs: # alone here rather than widened by a side effect. git diff --name-only --no-renames "origin/${BASE_REF}...HEAD" > /tmp/changed_norenames.txt echo "Changed files:"; sed 's/^/ /' /tmp/changed_norenames.txt - if grep -qvE '^docs/' /tmp/changed_norenames.txt; then + # `code` is false when every path is under docs/ or in a CI-only Go + # tool that no image builds (cmd/backport-batch, cmd/api-gate): there + # is nothing to build and nothing for e2e to install. e2e-fork.yaml + # answers the same question from the PR file list with the same + # prefixes, and hack/gate-concurrency-contract.bats holds the two lists + # equal. + if grep -qvE '^(docs/|cmd/backport-batch/|cmd/api-gate/)' /tmp/changed_norenames.txt; then echo "code=true" >> "$GITHUB_OUTPUT" else echo "code=false" >> "$GITHUB_OUTPUT" fi + # The Bats contracts read docs, so a docs change runs them even when + # `code` is false; a change confined to the CI-only tools does not. + if grep -qE '^docs/' /tmp/changed_norenames.txt; then + echo "docs=true" >> "$GITHUB_OUTPUT" + else + echo "docs=false" >> "$GITHUB_OUTPUT" + fi + backport_batch="$(hack/backport-batch-tests-needed.sh /tmp/changed_norenames.txt)" + echo "backport-batch tests needed: $backport_batch" + echo "backport_batch=$backport_batch" >> "$GITHUB_OUTPUT" matrix="$(hack/build-matrix.sh /tmp/changed.txt)" echo "Build matrix: $matrix" echo "matrix=$matrix" >> "$GITHUB_OUTPUT" @@ -293,9 +311,11 @@ jobs: contents: read needs: ["plan"] # Unit contracts read both code and documentation. Run this job for every - # non-release PR: code changes take the complete unit path, while docs-only - # changes run the Bats lane that owns those cross-tree contracts. Builds and - # E2E remain gated on `code` below. + # non-release PR: code changes take the complete unit path, docs changes run + # the Bats lane that owns those cross-tree contracts, and the backport-batch + # tests run whenever one of their inputs changed and at no other time. A + # change confined to the CI-only tools runs no other lane. Builds and E2E + # remain gated on `code` below. if: | !contains(github.event.pull_request.labels.*.name, 'release') steps: @@ -306,20 +326,21 @@ jobs: # The module cache is restored from the key build-main.yaml writes on # main, which every PR run can read, so a PR's first run starts warm. The # download fetches whatever the cache lacks, the whole set on a miss. The - # Go steps are for code changes only: a docs-only run is the Bats lane alone. + # Go steps are for code changes and the backport-batch tests only: a + # docs-only run is the Bats lane alone. - name: Locate Go module cache - if: needs.plan.outputs.code == 'true' + if: needs.plan.outputs.code == 'true' || needs.plan.outputs.backport_batch == 'true' id: gomodcache run: echo "dir=$(go env GOMODCACHE)" >> "$GITHUB_OUTPUT" - name: Cache Go modules - if: needs.plan.outputs.code == 'true' + if: needs.plan.outputs.code == 'true' || needs.plan.outputs.backport_batch == 'true' uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ${{ steps.gomodcache.outputs.dir }} key: go-mod-${{ runner.os }}-${{ hashFiles('**/go.sum') }} restore-keys: go-mod-${{ runner.os }}- - name: Download Go modules - if: needs.plan.outputs.code == 'true' + if: needs.plan.outputs.code == 'true' || needs.plan.outputs.backport_batch == 'true' run: | for mod in $(git ls-files go.mod '*/go.mod' | xargs -n1 dirname); do (cd "$mod" && "$GITHUB_WORKSPACE/hack/go-mod-download.sh") @@ -327,8 +348,9 @@ jobs: # Ephemeral runners don't carry the helm-unittest plugin the persistent # self-hosted runner accreted. The plugin embeds its own Helm, so its # version is pinned: a plugin release must not change test results. Bats is - # installed on every run, docs-only ones included. + # installed whenever a Bats lane runs, docs-only ones included. - name: Set up test toolchain + if: needs.plan.outputs.code == 'true' || needs.plan.outputs.docs == 'true' env: CODE_CHANGED: ${{ needs.plan.outputs.code }} HELM_UNITTEST_VERSION: "1.1.2" @@ -374,8 +396,15 @@ jobs: if: needs.plan.outputs.code == 'true' run: make unit-tests test-controllers test-controllers-envtest -j4 -k --output-sync=target - name: Run Bats unit tests for docs-only changes - if: needs.plan.outputs.code == 'false' + if: needs.plan.outputs.code == 'false' && needs.plan.outputs.docs == 'true' run: make bats-unit-tests + # Not part of `make unit-tests`: hack/backport-batch-tests-needed.sh + # decides from cmd/backport-batch/test-inputs.txt, whatever `code` is. + # `!cancelled()` for the reason the unit lane runs with -k: a red step + # above must not hide this suite's own result. + - name: Run backport-batch tests + if: ${{ !cancelled() && needs.plan.outputs.backport_batch == 'true' }} + run: make test-backport-batch - name: Upload unit test report if: always() uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 @@ -1427,7 +1456,7 @@ jobs: description = `release packages candidate did not verify (result: ${CANDIDATE_RESULT})`; } else if (CODE !== 'true') { state = 'success'; - description = 'docs-only PR — e2e not required'; + description = 'docs or CI-only tools — e2e not required'; } else if (IS_RELEASE === 'true' && IS_FULL_E2E !== 'true') { // Release promotion PRs deliberately do not repeat e2e: the rc was // already verified against the immutable candidate whose digests diff --git a/Makefile b/Makefile index 2feea00e9..a32e33af4 100644 --- a/Makefile +++ b/Makefile @@ -120,7 +120,7 @@ test: make -C packages/core/testing apply make -C packages/core/testing e2e -unit-tests: helm-unit-tests bats-unit-tests bats-posix-compat-tests go-unit-tests go-module-tests rd-presets-check test-check-readiness test-backport-batch migrations-target-check +unit-tests: helm-unit-tests bats-unit-tests bats-posix-compat-tests go-unit-tests go-module-tests rd-presets-check test-check-readiness migrations-target-check helm-unit-tests: hack/helm-unit-tests.sh @@ -196,6 +196,9 @@ test-check-readiness: go test ./test/check-readiness/ -count=1 # ./cmd/... is excluded from go-unit-tests, so the batch tool needs its own target. +# It is not a prerequisite of unit-tests: the checks job in pull-requests.yaml +# runs it only when the diff touches a path listed in +# cmd/backport-batch/test-inputs.txt. test-backport-batch: go test ./cmd/backport-batch/... -count=1 diff --git a/cmd/backport-batch/inputs_test.go b/cmd/backport-batch/inputs_test.go new file mode 100644 index 000000000..0632f80ef --- /dev/null +++ b/cmd/backport-batch/inputs_test.go @@ -0,0 +1,112 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "os" + "os/exec" + "strings" + "testing" +) + +const modulePath = "github.com/cozystack/cozystack/" + +// readTestInputs returns the entries of test-inputs.txt, the list CI reads to +// decide whether a pull request runs these tests. +func readTestInputs(t *testing.T) []string { + t.Helper() + data, err := os.ReadFile("test-inputs.txt") + if err != nil { + t.Fatal(err) + } + var entries []string + for _, line := range strings.Split(string(data), "\n") { + line = strings.TrimSpace(line) + if line == "" || strings.HasPrefix(line, "#") { + continue + } + entries = append(entries, line) + } + if len(entries) == 0 { + t.Fatal("test-inputs.txt lists no path") + } + return entries +} + +// covered reports whether a change to path matches an entry the way the plan +// job matches it: a trailing slash is a directory prefix, anything else is +// one file. +func covered(entries []string, path string) bool { + for _, e := range entries { + if strings.HasSuffix(e, "/") && strings.HasPrefix(path, e) || path == e { + return true + } + } + return false +} + +// A pull request that changes an input of these tests has to run them, and CI +// knows the inputs only from test-inputs.txt. An import added here, or added to +// a package this tool imports, lands in a pull request that touches a listed +// path, so it runs this test, which fails until the new package is listed. +func TestInputsListed(t *testing.T) { + entries := readTestInputs(t) + + out, err := exec.Command("go", "list", "-deps", "-test", "-f", "{{.ImportPath}}", ".").Output() + if err != nil { + t.Fatalf("go list: %v", err) + } + var deps int + for _, pkg := range strings.Split(strings.TrimSpace(string(out)), "\n") { + // A test variant prints as "pkg [pkg.test]"; the synthesized main + // package as "pkg.test". Both are this directory. + pkg, _, _ = strings.Cut(pkg, " ") + pkg = strings.TrimSuffix(pkg, ".test") + rel, ok := strings.CutPrefix(pkg, modulePath) + if !ok { + continue + } + deps++ + if !covered(entries, rel+"/x.go") { + t.Errorf("these tests import %s, which test-inputs.txt does not list: add %s/", pkg, rel) + } + } + if deps == 0 { + t.Fatalf("go list named no package of %s, so nothing was checked:\n%s", modulePath, out) + } + + for _, page := range []string{policyPage, "docs/release/backport-tasks/x.md"} { + if !covered(entries, page) { + t.Errorf("the tests read %s, which test-inputs.txt does not cover", page) + } + } + for _, f := range []string{"go.mod", "go.sum"} { + if !covered(entries, f) { + t.Errorf("test-inputs.txt does not list %s", f) + } + } +} + +// A listed path that no longer exists makes the list read as broader than it +// is, and usually means a dependency moved and its new home is missing. +func TestInputsExist(t *testing.T) { + for _, e := range readTestInputs(t) { + if _, err := os.Stat("../../" + e); err != nil { + t.Errorf("test-inputs.txt lists %s: %v", e, err) + } + } +} diff --git a/cmd/backport-batch/test-inputs.txt b/cmd/backport-batch/test-inputs.txt new file mode 100644 index 000000000..4262b0e1f --- /dev/null +++ b/cmd/backport-batch/test-inputs.txt @@ -0,0 +1,26 @@ +# Every path whose change can change the result of +# `go test ./cmd/backport-batch/...`. The plan job in +# .github/workflows/pull-requests.yaml runs those tests when, and only when, the +# pull request touches one of these paths. +# +# One repository-relative path per line. A trailing slash names a directory and +# matches everything below it; anything else matches that file alone. Blank lines +# and lines starting with # are ignored. +# +# TestInputsListed in inputs_test.go keeps this list in step with the code: it +# fails when the tests' in-repo package dependencies (go list -deps -test) or the +# docs pages they read are missing from it. + +cmd/backport-batch/ +go.mod +go.sum + +# In-repo packages the tool and its tests import. +internal/apigate/ +pkg/apis/apps/v1alpha1/ +pkg/apis/apps/validation/ +pkg/config/ + +# Pages the tests read from the tree. +docs/release/backports.md +docs/release/backport-tasks/ diff --git a/docs/agents/e2e-testing.md b/docs/agents/e2e-testing.md index 9ad31d967..2d37ab274 100644 --- a/docs/agents/e2e-testing.md +++ b/docs/agents/e2e-testing.md @@ -121,7 +121,7 @@ Be precise about TIA's scope — it is narrower than "skip E2E for unrelated PRs - **It trims only which suites `chainsaw test` runs, not the expensive stages.** `Prepare environment` and `Install Cozystack` (the full platform install) run regardless of the selection. TIA only decides the `CHAINSAW_SUITES` passed to the final E2E step. A perfect narrow saves the suite tail, not the bulk of wall-time. (Package image builds are scoped separately by `hack/build-matrix.sh`; the Talos leg runs when its direct or shared inputs changed and remains unconditional for forks, while installer/finalize runs for every non-docs PR. That is a distinct mechanism from TIA's suite trimming.) - **`Select E2E tests` runs *after* `Install Cozystack` and has no `always()`.** GitHub implicitly ANDs a step's `if:` with `success()`, so if install fails the selector (and `Run E2E tests`) are skipped entirely. On a red install you will not see TIA narrow anything — that is the ordering, not a selector bug. - **Most PRs legitimately escalate to the full suite.** Platform manifests live under `packages/core/`, Go under `api/`/`internal/`, helpers under `hack/`, plus the `Makefile` — touching any one triggers the full suite. So "all suites selected" is usually correct, not a failure to narrow. -- **Two skip layers, with a gap.** The `plan` job computes a coarse docs-only gate (`code`, true for any change outside `docs/**`) that skips the whole build+E2E pipeline for docs-only PRs, plus a per-package build matrix (`hack/build-matrix.sh`) that scopes which images build. TIA's finer skip (`select-e2e.sh` ignoring `docs/`, `dashboards/`, `*.md` and the rest of `inert_config_pattern`) only trims the suite selection. A PR touching only `dashboards/*.json` therefore still runs install, then skips just the Chainsaw suites — near-zero savings. If you need such PRs to skip the pipeline, widen the `plan` docs-gate, not `select-e2e.sh`. +- **Two skip layers, with a gap.** The `plan` job computes a coarse gate (`code`, false only when every changed path is under `docs/` or one of the CI-only Go tools no image builds, `cmd/backport-batch/` and `cmd/api-gate/`) that skips the whole build+E2E pipeline for such PRs, and `e2e-fork.yaml` skips on the same prefixes (`hack/gate-concurrency-contract.bats` holds the two lists equal), plus a per-package build matrix (`hack/build-matrix.sh`) that scopes which images build. TIA's finer skip (`select-e2e.sh` ignoring `docs/`, `dashboards/`, `*.md` and the rest of `inert_config_pattern`) only trims the suite selection. A PR touching only `dashboards/*.json` therefore still runs install, then skips just the Chainsaw suites — near-zero savings. If you need such PRs to skip the pipeline, widen the `plan` docs-gate, not `select-e2e.sh`. ### 8. Failure must be self-explanatory — diagnostics are first-class diff --git a/docs/agents/overview.md b/docs/agents/overview.md index 9ab67c41d..3c0b50d14 100644 --- a/docs/agents/overview.md +++ b/docs/agents/overview.md @@ -106,7 +106,7 @@ A chart also declares the limits on its own resource name there, with `## @name ## Testing -- **Helm and BATS unit tests:** `make helm-unit-tests` runs `hack/helm-unit-tests.sh` over every package that defines a `test` target. A `test` target is expected to run helm-unittest: the sweep fails a package whose run reports no suite, because exiting 0 having asserted nothing is indistinguishable from passing. A target that drives something else belongs under another name, as `packages/core/testing` already does with its sandbox flow. `make bats-unit-tests` requires bats-core 1.5 or newer; GNU parallel is optional and the target runs serially without it. Run the complete local BATS and POSIX-compatibility pass with `pre-commit run bats-unit-tests --hook-stage manual --all-files`; pull-request CI runs both lanes for code changes and the BATS contracts for docs-only changes. On a host whose `/bin/sh` differs from CI, `make BATS_POSIX_SHELL=dash bats-posix-compat-tests` selects the same interpreter explicitly. `make unit-tests` runs the full unit suite — Helm, BATS, the targeted POSIX-shell compatibility pass, Go, and the preset/readiness checks. It does not include the controller tests under `./internal/...`; run `make test-controllers` as well. CI invokes both targets together. +- **Helm and BATS unit tests:** `make helm-unit-tests` runs `hack/helm-unit-tests.sh` over every package that defines a `test` target. A `test` target is expected to run helm-unittest: the sweep fails a package whose run reports no suite, because exiting 0 having asserted nothing is indistinguishable from passing. A target that drives something else belongs under another name, as `packages/core/testing` already does with its sandbox flow. `make bats-unit-tests` requires bats-core 1.5 or newer; GNU parallel is optional and the target runs serially without it. Run the complete local BATS and POSIX-compatibility pass with `pre-commit run bats-unit-tests --hook-stage manual --all-files`; pull-request CI runs both lanes for code changes and the BATS contracts for docs-only changes. On a host whose `/bin/sh` differs from CI, `make BATS_POSIX_SHELL=dash bats-posix-compat-tests` selects the same interpreter explicitly. `make unit-tests` runs the full unit suite — Helm, BATS, the targeted POSIX-shell compatibility pass, Go, and the preset/readiness checks. It does not include the controller tests under `./internal/...`; run `make test-controllers` as well. CI invokes both targets together. The backport-batch tests are not part of it: run `make test-backport-batch` after changing anything listed in `cmd/backport-batch/test-inputs.txt`, which is also what CI keys them on. - **E2E tests:** Kyverno Chainsaw suites in `hack/e2e-chainsaw/` (one directory per app), run with `chainsaw test`. Cluster bootstrap (`hack/e2e-install-cozystack.bats`) and the OpenAPI checks (`hack/e2e-test-openapi.bats`) remain BATS. Conventions for writing and stabilising them — and the CI that runs them — live in [`e2e-testing.md`](./e2e-testing.md). - **Go tests:** standard `go test`, with Ginkgo/Gomega for controllers. diff --git a/hack/backport-batch-tests-needed.bats b/hack/backport-batch-tests-needed.bats new file mode 100644 index 000000000..9f37e768f --- /dev/null +++ b/hack/backport-batch-tests-needed.bats @@ -0,0 +1,87 @@ +#!/usr/bin/env bats +# Unit tests for hack/backport-batch-tests-needed.sh, which decides whether a +# pull request runs `make test-backport-batch`, and for its wiring into the +# checks job of pull-requests.yaml. +# +# Test-level EXIT traps replace Bats' own handler and hide failing TAP results. +# Cleanup follows aborting assertions; see docs/agents/e2e-testing.md. + +load test_helper + +CLASSIFIER="hack/backport-batch-tests-needed.sh" +WF=".github/workflows/pull-requests.yaml" + +classify() { + list="$1" + shift + printf '%s\n' "$@" > "$list" + "$CLASSIFIER" "$list" +} + +@test "a change to the tool, an imported package, a page its tests read or the module runs the tests" { + tmp=$(mktemp) + [ "$(classify "$tmp" cmd/backport-batch/report_test.go)" = true ] + [ "$(classify "$tmp" internal/apigate/surface.go)" = true ] + [ "$(classify "$tmp" pkg/config/config.go)" = true ] + [ "$(classify "$tmp" docs/release/backports.md)" = true ] + [ "$(classify "$tmp" docs/release/backport-tasks/conflict.md)" = true ] + [ "$(classify "$tmp" go.sum)" = true ] + # One input among unrelated paths is enough. + [ "$(classify "$tmp" packages/apps/mariadb/values.yaml pkg/config/config.go)" = true ] + rm -f "$tmp" +} + +@test "a change that reaches none of its inputs does not run the tests" { + tmp=$(mktemp) + [ "$(classify "$tmp" packages/apps/mariadb/values.yaml)" = false ] + [ "$(classify "$tmp" cmd/api-gate/main.go)" = false ] + [ "$(classify "$tmp" hack/backport-batch-scheduled.sh)" = false ] + [ "$(classify "$tmp" docs/release.md docs/release/other.md)" = false ] + # A file entry is not a prefix: backports.md does not cover backports.md.bak, + # and a directory entry does not cover a sibling sharing its name as prefix. + [ "$(classify "$tmp" docs/release/backports.md.bak)" = false ] + [ "$(classify "$tmp" pkg/configx/x.go)" = false ] + [ "$(classify "$tmp" internal/apigate-old/x.go)" = false ] + rm -f "$tmp" +} + +@test "entries are trimmed as the Go test trims them" { + tmp=$(mktemp -d) + printf '%s\n' cmd/backport-batch/main.go > "$tmp/diff" + printf ' cmd/backport-batch/ \r\n' > "$tmp/inputs" + [ "$("$CLASSIFIER" "$tmp/diff" "$tmp/inputs")" = true ] + rm -rf "$tmp" +} + +@test "an inputs file with no entry is refused rather than read as never" { + tmp=$(mktemp -d) + printf '%s\n' cmd/backport-batch/main.go > "$tmp/diff" + printf '# only a comment\n\n' > "$tmp/inputs" + if "$CLASSIFIER" "$tmp/diff" "$tmp/inputs" >"$tmp/out" 2>"$tmp/err"; then + echo "an empty inputs file must fail, it printed: $(cat "$tmp/out")" >&2 + exit 1 + fi + grep -q 'lists no path' "$tmp/err" + rm -rf "$tmp" +} + +@test "the checks job runs the target on the classifier's answer alone" { + plan_run=$(yq -r '.jobs.plan.steps[] | select(.id == "p") | .run' "$WF") + printf '%s\n' "$plan_run" | grep -qF 'backport_batch="$(hack/backport-batch-tests-needed.sh /tmp/changed_norenames.txt)"' + [ "$(yq -r '.jobs.plan.outputs.backport_batch' "$WF")" = '${{ steps.p.outputs.backport_batch }}' ] + + step_if=$(yq -r '.jobs.checks.steps[] | select(.name == "Run backport-batch tests") | .if' "$WF") + step_run=$(yq -r '.jobs.checks.steps[] | select(.name == "Run backport-batch tests") | .run' "$WF") + # !cancelled(): a failed unit lane above must not skip this suite too. + [ "$step_if" = "\${{ !cancelled() && needs.plan.outputs.backport_batch == 'true' }}" ] + [ "$step_run" = 'make test-backport-batch' ] + + # And nowhere else: the unit lane every code PR runs must not carry it. + prereqs=$(sed -n 's/^unit-tests:[[:space:]]*//p' Makefile) + [ -n "$prereqs" ] + case " $prereqs " in + *" test-backport-batch "*) + echo "make unit-tests still runs test-backport-batch on every code PR: $prereqs" >&2 + exit 1 ;; + esac +} diff --git a/hack/backport-batch-tests-needed.sh b/hack/backport-batch-tests-needed.sh new file mode 100755 index 000000000..71803e730 --- /dev/null +++ b/hack/backport-batch-tests-needed.sh @@ -0,0 +1,44 @@ +#!/bin/sh +# Print whether a changed-path list touches an input of +# `go test ./cmd/backport-batch/...`, as listed in +# cmd/backport-batch/test-inputs.txt. An entry ending in / matches every path +# below it, any other entry matches that file alone. The caller supplies the +# rename-blind list so both sides of a rename are classified. +# +# Usage: hack/backport-batch-tests-needed.sh [] +set -eu + +inputs="${2:-cmd/backport-batch/test-inputs.txt}" +if [ "$#" -lt 1 ] || [ "$#" -gt 2 ] || [ ! -r "$1" ] || [ ! -r "$inputs" ]; then + echo "usage: backport-batch-tests-needed.sh []" >&2 + exit 2 +fi + +# An inputs file with no entry would answer false for every diff and switch the +# tests off without a word; refuse instead. +if ! grep -qvE '^[[:space:]]*(#|$)' "$inputs"; then + echo "backport-batch-tests-needed: $inputs lists no path" >&2 + exit 2 +fi + +# Entries are trimmed the way TestInputsListed trims them, so a trailing space +# or a CR cannot make the Go test accept an entry this never matches. +awk ' + NR == FNR { + e = $0 + sub(/^[[:space:]]+/, "", e) + sub(/[[:space:]]+$/, "", e) + if (e != "" && e !~ /^#/) entries[++n] = e + next + } + { + for (i = 1; i <= n; i++) { + e = entries[i] + if ((substr(e, length(e)) == "/" && index($0, e) == 1) || $0 == e) { + found = 1 + exit + } + } + } + END { print (found ? "true" : "false") } +' "$inputs" "$1" diff --git a/hack/downstream-trigger-map.bats b/hack/downstream-trigger-map.bats index c5ebf6af3..39e1d5433 100644 --- a/hack/downstream-trigger-map.bats +++ b/hack/downstream-trigger-map.bats @@ -35,8 +35,9 @@ # a human's job, and reviewers should not assume CI did it for them. # # A docs-only PR skips builds and E2E, but the `checks` job still runs the Bats -# lane. Otherwise editing this map, or another document audited by a unit -# contract, would exempt the changed input from its own guard. +# lane whenever a docs/ path changed. Otherwise editing this map, or another +# document audited by a unit contract, would exempt the changed input from its +# own guard. load test_helper @@ -117,9 +118,18 @@ map_section() { [ "$job_if" = "!contains(github.event.pull_request.labels.*.name, 'release')" ] [ "$code_if" = "needs.plan.outputs.code == 'true'" ] - [ "$docs_if" = "needs.plan.outputs.code == 'false'" ] + [ "$docs_if" = "needs.plan.outputs.code == 'false' && needs.plan.outputs.docs == 'true'" ] [ "$docs_run" = 'make bats-unit-tests' ] [ "$code_run" = 'make unit-tests test-controllers test-controllers-envtest -j4 -k --output-sync=target' ] + + # `code` is false for the CI-only tools too, so the docs lane is keyed on a + # docs/ path being present, not on `code` alone. Pin that the output means + # exactly that and that Bats gets installed for it. + plan_run=$(yq -r '.jobs.plan.steps[] | select(.id == "p") | .run' "$WORKFLOW") + printf '%s\n' "$plan_run" | grep -qF "if grep -qE '^docs/' /tmp/changed_norenames.txt; then" + [ "$(yq -r '.jobs.plan.outputs.docs' "$WORKFLOW")" = '${{ steps.p.outputs.docs }}' ] + toolchain_if=$(yq -r '.jobs.checks.steps[] | select(.name == "Set up test toolchain") | .if' "$WORKFLOW") + [ "$toolchain_if" = "needs.plan.outputs.code == 'true' || needs.plan.outputs.docs == 'true'" ] } @test "the trigger map and the PR-template checklist list the same repositories" { diff --git a/hack/gate-concurrency-contract.bats b/hack/gate-concurrency-contract.bats index 5133a4cd8..0c4c90f15 100644 --- a/hack/gate-concurrency-contract.bats +++ b/hack/gate-concurrency-contract.bats @@ -266,24 +266,56 @@ resolve_silent_conclusions() { plan_block="$(job_block plan "$PULL_REQUESTS" | code_lines)" [ -n "$plan_block" ] - count="$(printf '%s\n' "$plan_block" | grep -c "grep -qvE '\^docs/' /tmp/changed_norenames.txt" || true)" + count="$(printf '%s\n' "$plan_block" | grep -c "grep -qvE '\^(.*)' /tmp/changed_norenames.txt" || true)" [ "${count:-0}" -eq 1 ] # The rename-detected list may still feed the build matrix, but never this. - count="$(printf '%s\n' "$plan_block" | grep -c "grep -qvE '\^docs/' /tmp/changed.txt" || true)" + count="$(printf '%s\n' "$plan_block" | grep -c "grep -qvE '\^(.*)' /tmp/changed.txt" || true)" [ "${count:-0}" -eq 0 ] # The fork half of the same agreement: both sides of a rename are inspected. - # Pinned on the docs-only expression itself — `previous_filename` also appears + # Pinned on the skip expression itself — `previous_filename` also appears # in the TIA file list a few lines down, so counting the identifier would stay # satisfied by that one with this check deleted. block="$(job_block resolve "$FORK" | code_lines)" [ -n "$block" ] count="$(printf '%s\n' "$block" \ - | grep -c 'f\.previous_filename && !isDocs(f\.previous_filename)' || true)" + | grep -c 'f\.previous_filename && !isNoRuntime(f\.previous_filename)' || true)" [ "${count:-0}" -eq 1 ] } +@test "both lanes skip e2e for the same path prefixes" { + # The rename test above pins HOW each lane reads its list; this one pins + # WHAT it skips. A prefix added to one lane only is a fork PR that one lane + # reads as nothing-to-build and the other as needing the pr-patch artifact, + # which nothing will ever produce. + plan_prefixes="$(job_block plan "$PULL_REQUESTS" | code_lines \ + | sed -nE "s,.*grep -qvE '\^\(([^)]*)\)' /tmp/changed_norenames\.txt.*,\1,p" \ + | tr '|' '\n' | sort)" + fork_prefixes="$(job_block resolve "$FORK" | code_lines \ + | sed -nE "s,.*const noRuntimePrefixes = \[([^]]*)\];.*,\1,p" \ + | tr ',' '\n' | tr -d " '" | sort)" + [ -n "$plan_prefixes" ] + printf '%s\n' "$plan_prefixes" | grep -qx 'docs/' + if [ "$plan_prefixes" != "$fork_prefixes" ]; then + echo "plan skips:" >&2; printf ' %s\n' $plan_prefixes >&2 + echo "fork skips:" >&2; printf ' %s\n' $fork_prefixes >&2 + exit 1 + fi + + # The tools in that list must also be the ones the build matrix and the suite + # selector leave alone in a mixed diff; otherwise a tool skipped on its own + # still rebuilds everything beside a one-line package change. + tools="$(printf '%s\n' "$plan_prefixes" | grep -vx 'docs/')" + [ -n "$tools" ] + matrix_tools="$(sed -nE "s,^tooling_pattern='\^cmd/\(([^)]*)\)/'$,\1,p" "$REPO_ROOT/hack/build-matrix.sh" \ + | tr '|' '\n' | sed 's,^,cmd/,; s,$,/,' | sort)" + select_tools="$(sed -nE 's,^ (cmd/[^)]*)\)$,\1,p' "$REPO_ROOT/hack/select-e2e.sh" \ + | tr '|' '\n' | sed 's,\*$,,' | sort)" + [ "$tools" = "$matrix_tools" ] + [ "$tools" = "$select_tools" ] +} + @test "same-repo gate: every publishing guard names the same label set" { # `plan`, `verify-release-candidate`, `e2e-report` and the concurrency key # each decide from the label name whether this run owns the status. Renaming From 9e861c117840cbc4f3886bc7f53dbdd0fb692427 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Fri, 13 Nov 2026 09:19:25 +0100 Subject: [PATCH 477/518] fix(seaweedfs): stop the master crashing on the crowded volume map SeaweedFS 4.45 updates the crowded-volume map of a volume layout under the read lock, while Assign reads that map under the same read lock to count writable volumes. Under a steady write load the Go runtime aborts the master with "concurrent map read and map write", a fatal error that recover cannot catch, and the S3 gateway answers 500 until a master is elected again. Taking the write lock for the update, as every other mutation of the map already does, removes the race. Upstream fixed it in seaweedfs/seaweedfs#11216, released in 4.46. The patch carries that change and its regression test unchanged, so the build runs the test and the patch stops applying, and is dropped, once the vendored release carries the fix. Signed-off-by: Mattia Eleuteri Assisted-by: LLM --- .../seaweedfs/images/seaweedfs/Dockerfile | 9 ++ .../patches/volume-layout-crowded-lock.diff | 87 +++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 packages/system/seaweedfs/images/seaweedfs/patches/volume-layout-crowded-lock.diff diff --git a/packages/system/seaweedfs/images/seaweedfs/Dockerfile b/packages/system/seaweedfs/images/seaweedfs/Dockerfile index 8f42f3ea2..d52ee7329 100644 --- a/packages/system/seaweedfs/images/seaweedfs/Dockerfile +++ b/packages/system/seaweedfs/images/seaweedfs/Dockerfile @@ -16,6 +16,15 @@ # seaweedfs/seaweedfs#11674 after 4.48; drop the patch on the first vendored # release that carries it. # +# patches/volume-layout-crowded-lock.diff makes SetVolumeCrowded take the write +# lock of the volume layout. Upstream 4.45 mutates the crowded map under the +# read lock, while Assign reads it under the same read lock through +# GetWritableVolumeCount, so under a steady stream of writes the master dies on +# a fatal "concurrent map read and map write", which recover cannot catch, and +# S3 answers 500 until it is back. Backport of seaweedfs/seaweedfs#11216, +# released in 4.46; drop the patch on the first vendored release that carries +# it. +# # `make update` re-pins VERSION from the vendored chart, SOURCE_SHA256 from the # source archive of that tag, UPSTREAM_DIGEST from the upstream image of that # tag and the builder from upstream's docker/Dockerfile.go_build, and checks diff --git a/packages/system/seaweedfs/images/seaweedfs/patches/volume-layout-crowded-lock.diff b/packages/system/seaweedfs/images/seaweedfs/patches/volume-layout-crowded-lock.diff new file mode 100644 index 000000000..12e09ca67 --- /dev/null +++ b/packages/system/seaweedfs/images/seaweedfs/patches/volume-layout-crowded-lock.diff @@ -0,0 +1,87 @@ +diff --git a/weed/topology/crowded_writable_test.go b/weed/topology/crowded_writable_test.go +--- a/weed/topology/crowded_writable_test.go ++++ b/weed/topology/crowded_writable_test.go +@@ -1,6 +1,7 @@ + package topology + + import ( ++ "sync" + "testing" + "time" + +@@ -73,3 +74,52 @@ func TestCrowdedCountStillMatchesWritables(t *testing.T) { + t.Errorf("writable=%d crowded=%d, want 1 and 1", writable, crowded) + } + } ++ ++// SetVolumeCrowded mutates the crowded map while GetWritableVolumeCount reads ++// it on the Assign hot path. Under RLock both could run at once and the Go ++// runtime aborts with "concurrent map read and map write". Run with -race to ++// reproduce the original bug; the write lock makes this pass. ++func TestSetVolumeCrowdedNoRaceWithGetWritableVolumeCount(t *testing.T) { ++ rp, _ := super_block.NewReplicaPlacementFromString("000") ++ vl := NewVolumeLayout(rp, needle.EMPTY_TTL, types.HardDriveType, 10000, false) ++ // GetWritableVolumeCount iterates vl.writables, so give it something to ++ // read while the crowded map is being mutated. ++ vl.writables = []needle.VolumeId{1, 2, 3} ++ ++ var wg sync.WaitGroup ++ stop := make(chan struct{}) ++ for range 8 { ++ wg.Add(1) ++ go func() { ++ defer wg.Done() ++ for { ++ select { ++ case <-stop: ++ return ++ default: ++ vl.SetVolumeCrowded(needle.VolumeId(2)) ++ } ++ } ++ }() ++ } ++ for range 8 { ++ wg.Add(1) ++ go func() { ++ defer wg.Done() ++ for { ++ select { ++ case <-stop: ++ return ++ default: ++ vl.GetWritableVolumeCount() ++ } ++ } ++ }() ++ } ++ ++ // The race detector needs only a brief window to flag a concurrent ++ // map read/write; let the goroutines hammer the lock briefly. ++ time.Sleep(50 * time.Millisecond) ++ close(stop) ++ wg.Wait() ++} +diff --git a/weed/topology/volume_layout.go b/weed/topology/volume_layout.go +--- a/weed/topology/volume_layout.go ++++ b/weed/topology/volume_layout.go +@@ -1019,11 +1019,14 @@ func (vl *VolumeLayout) setVolumeCrowded(vid needle.VolumeId) { + } + + func (vl *VolumeLayout) SetVolumeCrowded(vid needle.VolumeId) { +- // since delete is guarded by accessLock.Lock(), +- // and is always called in sequential order, +- // RLock() should be safe enough +- vl.accessLock.RLock() +- defer vl.accessLock.RUnlock() ++ // setVolumeCrowded mutates the crowded map, and GetWritableVolumeCount ++ // reads it under RLock on the Assign hot path. Two concurrent RLock ++ // holders with one writing the map triggers a fatal ++ // "concurrent map read and map write". Take the write lock: this path ++ // is a low-frequency single consumer driven by the crowded-volume ++ // event loop, and every other mutation of crowded already holds Lock(). ++ vl.accessLock.Lock() ++ defer vl.accessLock.Unlock() + + vl.setVolumeCrowded(vid) + } From 1725226d2332efc2cf0085a77e4f4aa19f01a798 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Fri, 13 Nov 2026 09:30:36 +0100 Subject: [PATCH 478/518] fix(linstor): restore snapshots on a schedulable node A volume restored from a snapshot is pinned to the one node it is restored on. linstor-csi picked that node among every ONLINE snapshot node: its preferred-node filter matched all nodes, so the node the scheduler selected got no priority, and a cordoned node was as good as any other. A pod whose volume landed on a cordoned node stayed Pending until the node was uncordoned, which made Velero data mover backups time out on every storage node cordoned ahead of maintenance, even when schedulable replicas held the same snapshot. Prefer the selected node when it holds the snapshot, and skip cordoned nodes while another node holds it. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../003-snapshot-restore-node-selection.diff | 249 ++++++++++++++++++ 1 file changed, 249 insertions(+) create mode 100644 packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff diff --git a/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff b/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff new file mode 100644 index 000000000..0ad54da77 --- /dev/null +++ b/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff @@ -0,0 +1,249 @@ +diff --git a/cmd/linstor-csi/linstor-csi.go b/cmd/linstor-csi/linstor-csi.go +index bd28e06..5bed73b 100644 +--- a/cmd/linstor-csi/linstor-csi.go ++++ b/cmd/linstor-csi/linstor-csi.go +@@ -135,6 +135,7 @@ func main() { + client.LogOut(logOut), + client.PropertyNamespace(*propNs), + client.LabelBySP(*labelBySP), ++ client.ConfigureKubernetesIfAvailable(), + ) + if err != nil { + log.Fatal(err) +diff --git a/pkg/client/linstor.go b/pkg/client/linstor.go +index 5d71c37..e8e67ec 100644 +--- a/pkg/client/linstor.go ++++ b/pkg/client/linstor.go +@@ -46,6 +46,8 @@ import ( + "github.com/pborman/uuid" + "github.com/sirupsen/logrus" + "golang.org/x/sys/unix" ++ apierrors "k8s.io/apimachinery/pkg/api/errors" ++ metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/mount-utils" + utilexec "k8s.io/utils/exec" + "k8s.io/utils/ptr" +@@ -72,6 +74,9 @@ type Linstor struct { + mounter mount.Interface + resizer *mount.ResizeFs + labelBySP bool ++ // nodeSchedulable reports whether new pods can be scheduled on a node. ++ // It is nil when not running in Kubernetes: every node then counts as schedulable. ++ nodeSchedulable func(ctx context.Context, node string) (bool, error) + } + + // NewLinstor returns a high-level linstor client for CSI applications to interact with +@@ -125,6 +130,34 @@ func APIClient(c *lc.HighLevelClient) func(*Linstor) error { + } + } + ++// ConfigureKubernetesIfAvailable lets the client read Kubernetes Node objects, so that a snapshot is ++// not restored on a cordoned node when another node holds it. Outside of Kubernetes it does nothing. ++func ConfigureKubernetesIfAvailable() func(*Linstor) error { ++ return func(l *Linstor) error { ++ cl, _, err := utils.KubernetesClient() ++ if err != nil { ++ // Not running in kubernetes ++ return nil ++ } ++ ++ l.nodeSchedulable = func(ctx context.Context, node string) (bool, error) { ++ n, err := cl.CoreV1().Nodes().Get(ctx, node, metav1.GetOptions{}) ++ if err != nil { ++ if apierrors.IsNotFound(err) { ++ // A LINSTOR node without a Kubernetes Node: nothing says it is cordoned. ++ return true, nil ++ } ++ ++ return true, err ++ } ++ ++ return !n.Spec.Unschedulable, nil ++ } ++ ++ return nil ++ } ++} ++ + // LogOut sets the Linstor client to write logs to the provided io.Writer + // instead of discarding logs. + func LogOut(out io.Writer) func(*Linstor) error { +@@ -1567,34 +1600,29 @@ func (s *Linstor) reconcileSnapshotResources(ctx context.Context, snapshot *volu + return fmt.Errorf("snapshot '%s' not deployed on any node", snap.Name) + } + +- // Collect available snapshot nodes, preferring those matching the topology hints. +- var preferred, available []string ++ // Collect the snapshot nodes LINSTOR can restore on, and whether Kubernetes can schedule pods there. ++ var available []string ++ schedulable := make(map[string]bool, len(snap.Nodes)) + for _, snapNode := range snap.Nodes { + if err := s.NodeAvailable(ctx, snapNode); err != nil { + logger.WithField("selected node candidate", snapNode).WithError(err).Debug("node is not available") + continue + } + +- if slices.Contains(preferredNodes, snapNode) { +- preferred = append(preferred, snapNode) +- } +- + available = append(available, snapNode) +- } ++ schedulable[snapNode] = true + +- // Pick a random node from preferred candidates, falling back to any available node. +- // Randomization distributes restore load across snapshot nodes, preventing all +- // clones of the same source from concentrating on a single node. +- candidates := preferred +- if len(candidates) == 0 { +- candidates = available +- } ++ if s.nodeSchedulable != nil { ++ ok, err := s.nodeSchedulable(ctx, snapNode) ++ if err != nil { ++ logger.WithField("selected node candidate", snapNode).WithError(err).Warn("could not check if node is cordoned, assuming it is not") ++ } + +- var selectedNode string +- if len(candidates) > 0 { +- selectedNode = candidates[rand.Intn(len(candidates))] ++ schedulable[snapNode] = ok ++ } + } + ++ selectedNode := selectRestoreNode(available, preferredNodes, func(node string) bool { return schedulable[node] }) + if selectedNode == "" { + return fmt.Errorf("no node [%s] available for snapshot '%s'", strings.Join(snap.Nodes, ","), snap.Name) + } +@@ -1893,6 +1921,39 @@ func (s *Linstor) relocateResources(ctx context.Context, volID, rgName string) e + return nil + } + ++// selectRestoreNode picks the node a snapshot is restored on, among the available nodes holding it. ++// ++// The restored volume is pinned to that node, so the consumer pod has to run there: ++// - preferredNodes[0] is the node the scheduler selected for a WaitForFirstConsumer volume. If it holds the ++// snapshot, restoring there lets the pod start without another scheduling round. ++// - Otherwise a random node is picked, which spreads restores of the same source across its snapshot nodes. ++// Cordoned nodes are skipped as long as another node holds the snapshot: a pod can never be scheduled on ++// them, so a volume restored there stays unusable until the node is uncordoned. ++// ++// Only nodes that are also in preferredNodes are considered, unless none is. ++func selectRestoreNode(available, preferredNodes []string, schedulable func(node string) bool) string { ++ candidates := slices.DeleteFunc(slices.Clone(available), func(node string) bool { ++ return !slices.Contains(preferredNodes, node) ++ }) ++ if len(candidates) == 0 { ++ candidates = available ++ } ++ ++ if usable := slices.DeleteFunc(slices.Clone(candidates), func(node string) bool { return !schedulable(node) }); len(usable) > 0 { ++ candidates = usable ++ } ++ ++ if len(candidates) == 0 { ++ return "" ++ } ++ ++ if len(preferredNodes) > 0 && slices.Contains(candidates, preferredNodes[0]) { ++ return preferredNodes[0] ++ } ++ ++ return candidates[rand.Intn(len(candidates))] ++} ++ + // FindSnapsByID searches the snapshot in the backend + func (s *Linstor) FindSnapsByID(ctx context.Context, id string) ([]*volume.Snapshot, error) { + snapshotId, err := volume.ParseSnapshotId(id) +diff --git a/pkg/client/linstor_test.go b/pkg/client/linstor_test.go +index 4f3405b..84cfca2 100644 +--- a/pkg/client/linstor_test.go ++++ b/pkg/client/linstor_test.go +@@ -577,3 +577,87 @@ func TestGetSnapshotRemoteAndReadiness(t *testing.T) { + }) + } + } ++ ++func TestLinstor_reconcileSnapshotResourcesSkipsCordonedNodes(t *testing.T) { ++ // A replicated volume has a snapshot on three nodes, one of them cordoned. The scheduler selected a node ++ // without the snapshot. Restoring on the cordoned node pins the new volume to a node no pod can be scheduled ++ // on, so the restore must always go to one of the other two. ++ const ( ++ sourceRD = "pvc-source" ++ snapName = "snapshot-1" ++ targetRD = "pvc-target" ++ cordoned = "node-3" ++ ) ++ ++ snapNodes := []string{"node-1", "node-2", cordoned} ++ preferred := []string{"node-9", "node-1", "node-2", cordoned} ++ ++ restoredOn := map[string]int{} ++ ++ for range 50 { ++ res := mocks.ResourceProvider{} ++ res.On("GetAll", mock.Anything, targetRD).Return([]lapi.Resource{}, nil) ++ res.On("GetSnapshot", mock.Anything, sourceRD, snapName).Return(lapi.Snapshot{Name: snapName, Nodes: snapNodes}, nil) ++ res.On("RestoreSnapshot", mock.Anything, sourceRD, snapName, mock.Anything).Run(func(args mock.Arguments) { ++ restoredOn[args.Get(3).(lapi.SnapshotRestore).Nodes[0]]++ ++ }).Return(nil) ++ ++ nodes := mocks.NodeProvider{} ++ for _, n := range snapNodes { ++ nodes.On("Get", mock.Anything, n).Return(lapi.Node{Name: n, ConnectionStatus: "ONLINE"}, nil) ++ } ++ ++ cl := Linstor{ ++ client: &lc.HighLevelClient{Client: &lapi.Client{Resources: &res, Nodes: &nodes}}, ++ log: logrus.WithField("test", t.Name()), ++ nodeSchedulable: func(_ context.Context, node string) (bool, error) { ++ return node != cordoned, nil ++ }, ++ } ++ ++ snap := &volume.Snapshot{SnapshotId: volume.SnapshotId{SourceName: sourceRD, SnapshotName: snapName}} ++ assert.NoError(t, cl.reconcileSnapshotResources(context.Background(), snap, targetRD, preferred)) ++ } ++ ++ assert.Zero(t, restoredOn[cordoned], "restored on the cordoned node: %v", restoredOn) ++ assert.Equal(t, 50, restoredOn["node-1"]+restoredOn["node-2"]) ++} ++ ++func TestSelectRestoreNode(t *testing.T) { ++ all := func(string) bool { return true } ++ notNode3 := func(n string) bool { return n != "node-3" } ++ ++ t.Run("selected node holds the snapshot", func(t *testing.T) { ++ for range 20 { ++ assert.Equal(t, "node-2", selectRestoreNode([]string{"node-1", "node-2", "node-3"}, []string{"node-2", "node-1", "node-3"}, all)) ++ } ++ }) ++ ++ t.Run("selected node is cordoned", func(t *testing.T) { ++ for range 20 { ++ assert.NotEqual(t, "node-3", selectRestoreNode([]string{"node-1", "node-3"}, []string{"node-3", "node-1"}, notNode3)) ++ } ++ }) ++ ++ t.Run("only cordoned nodes hold the snapshot", func(t *testing.T) { ++ assert.Equal(t, "node-3", selectRestoreNode([]string{"node-3"}, []string{"node-1", "node-3"}, notNode3)) ++ }) ++ ++ t.Run("snapshot nodes outside the preferred nodes are a fallback", func(t *testing.T) { ++ assert.Equal(t, "node-2", selectRestoreNode([]string{"node-1", "node-2"}, []string{"node-2"}, all)) ++ assert.Equal(t, "node-1", selectRestoreNode([]string{"node-1"}, []string{"node-2"}, all)) ++ }) ++ ++ t.Run("no available node", func(t *testing.T) { ++ assert.Empty(t, selectRestoreNode(nil, []string{"node-1"}, all)) ++ }) ++ ++ t.Run("restores are spread when the selected node lacks the snapshot", func(t *testing.T) { ++ seen := map[string]bool{} ++ for range 200 { ++ seen[selectRestoreNode([]string{"node-1", "node-2"}, []string{"node-9", "node-1", "node-2"}, all)] = true ++ } ++ ++ assert.Equal(t, map[string]bool{"node-1": true, "node-2": true}, seen) ++ }) ++} From 0f16bf9c7336c03f3236ad68528106964d02c2c0 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Fri, 13 Nov 2026 09:30:50 +0100 Subject: [PATCH 479/518] feat(redis): add maxmemory, eviction policy and snapshot toggle A Redis used purely as a cache had no durable way to bound its memory or stop writing RDB snapshots: the chart only disabled snapshots together with the persistent volume, and a CONFIG SET is lost when the pod restarts. Snapshots and replica full syncs then filled the volume and left the instance crashlooping. maxmemory, maxmemory-policy and save are always rendered with an explicit value, because the operator applies customConfig with CONFIG SET on every reconcile and never resets a directive whose line disappears; leaving the defaults implicit would make reverting a setting a no-op until the next pod restart. maxmemory is refused above 90% of the memory request, the threshold above which the operator silently skips the line and leaves Redis unbounded. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- api/apps/v1alpha1/redis/types.go | 11 ++ .../v1alpha1/redis/zz_generated.deepcopy.go | 1 + packages/apps/redis/README.md | 13 +- .../apps/redis/templates/redisfailover.yaml | 25 ++++ .../apps/redis/tests/cache_config_test.yaml | 118 ++++++++++++++++++ .../apps/redis/tests/min_replicas_test.yaml | 6 +- packages/apps/redis/values.schema.json | 33 +++++ packages/apps/redis/values.yaml | 18 +++ packages/system/redis-rd/cozyrds/redis.yaml | 4 +- 9 files changed, 219 insertions(+), 10 deletions(-) create mode 100644 packages/apps/redis/tests/cache_config_test.yaml diff --git a/api/apps/v1alpha1/redis/types.go b/api/apps/v1alpha1/redis/types.go index c2363749c..03ff0175d 100644 --- a/api/apps/v1alpha1/redis/types.go +++ b/api/apps/v1alpha1/redis/types.go @@ -51,6 +51,14 @@ type ConfigSpec struct { // Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it. // +kubebuilder:validation:Minimum=0 MinReplicasMaxLag *int `json:"minReplicasMaxLag,omitempty"` + // Redis `maxmemory`: the dataset size at which Redis applies `maxmemoryPolicy`. Unset by default, which renders `maxmemory 0` (no limit, the Redis default), so a value set by hand with `CONFIG SET` is reset on the next reconcile. Must be at most 90% of the memory request of each Redis pod (from `resources` or `resourcesPreset`): the operator silently drops a larger value, so the chart refuses it. Leave headroom below that for replication and snapshot buffers. + Maxmemory resource.Quantity `json:"maxmemory,omitempty"` + // Redis `maxmemory-policy`: what Redis does once `maxmemory` is reached. `noeviction` refuses writes; `allkeys-lru` evicts the least recently used keys, which suits an instance used purely as a cache. Has no effect while `maxmemory` is unset. + // +kubebuilder:default:="noeviction" + MaxmemoryPolicy MaxmemoryPolicy `json:"maxmemoryPolicy"` + // Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save ""` is rendered and the volume is kept, so the data is lost on restart but replicas still have room for the full-sync RDB they write to disk. Always off when `size` is empty. + // +kubebuilder:default:=true + Snapshots bool `json:"snapshots"` // Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. // +kubebuilder:default:={} Sentinel Sentinel `json:"sentinel,omitempty"` @@ -80,6 +88,9 @@ type TLS struct { // +kubebuilder:validation:Enum="no";"optional";"yes" type AuthClients string +// +kubebuilder:validation:Enum="noeviction";"allkeys-lru";"allkeys-lfu";"allkeys-random";"volatile-lru";"volatile-lfu";"volatile-random";"volatile-ttl" +type MaxmemoryPolicy string + // +kubebuilder:validation:Enum="t1.nano";"t1.micro";"t1.small";"t1.medium";"t1.large";"t1.xlarge";"t1.2xlarge";"t1.4xlarge";"c1.nano";"c1.micro";"c1.small";"c1.medium";"c1.large";"c1.xlarge";"c1.2xlarge";"c1.4xlarge";"s1.nano";"s1.micro";"s1.small";"s1.medium";"s1.large";"s1.xlarge";"s1.2xlarge";"s1.4xlarge";"u1.nano";"u1.micro";"u1.small";"u1.medium";"u1.large";"u1.xlarge";"u1.2xlarge";"u1.4xlarge";"m1.nano";"m1.micro";"m1.small";"m1.medium";"m1.large";"m1.xlarge";"m1.2xlarge";"m1.4xlarge";"nano";"micro";"small";"medium";"large";"xlarge";"2xlarge" type ResourcesPreset string diff --git a/api/apps/v1alpha1/redis/zz_generated.deepcopy.go b/api/apps/v1alpha1/redis/zz_generated.deepcopy.go index 489c28e85..86437ea58 100644 --- a/api/apps/v1alpha1/redis/zz_generated.deepcopy.go +++ b/api/apps/v1alpha1/redis/zz_generated.deepcopy.go @@ -66,6 +66,7 @@ func (in *ConfigSpec) DeepCopyInto(out *ConfigSpec) { *out = new(int) **out = **in } + out.Maxmemory = in.Maxmemory.DeepCopy() in.Sentinel.DeepCopyInto(&out.Sentinel) } diff --git a/packages/apps/redis/README.md b/packages/apps/redis/README.md index 84cb04930..4044ce951 100644 --- a/packages/apps/redis/README.md +++ b/packages/apps/redis/README.md @@ -39,11 +39,14 @@ Service utilizes the freshworks-oss Redis Operator (a maintained fork of the arc ### Application-specific parameters -| Name | Description | Type | Value | -| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ------ | -| `authEnabled` | Enable password generation. | `bool` | `true` | -| `minReplicasToWrite` | Redis `min-replicas-to-write`: the master refuses writes unless at least this many replicas are connected and within `minReplicasMaxLag`. Unset by default, which leaves the Redis default (disabled). With `replicas: 2` and a value of 1 the master rejects writes while its only replica is down, trading availability for a bound on the writes lost if an isolated master is failed over. Replication stays asynchronous, so this bounds the loss, it does not remove it. Must be 0 or more and below `replicas` (a master with `replicas` pods has `replicas - 1` replicas), otherwise every write is refused; the chart refuses such a value. | `*int` | `null` | -| `minReplicasMaxLag` | Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it. | `*int` | `null` | +| Name | Description | Type | Value | +| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------ | +| `authEnabled` | Enable password generation. | `bool` | `true` | +| `minReplicasToWrite` | Redis `min-replicas-to-write`: the master refuses writes unless at least this many replicas are connected and within `minReplicasMaxLag`. Unset by default, which leaves the Redis default (disabled). With `replicas: 2` and a value of 1 the master rejects writes while its only replica is down, trading availability for a bound on the writes lost if an isolated master is failed over. Replication stays asynchronous, so this bounds the loss, it does not remove it. Must be 0 or more and below `replicas` (a master with `replicas` pods has `replicas - 1` replicas), otherwise every write is refused; the chart refuses such a value. | `*int` | `null` | +| `minReplicasMaxLag` | Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it. | `*int` | `null` | +| `maxmemory` | Redis `maxmemory`: the dataset size at which Redis applies `maxmemoryPolicy`. Unset by default, which renders `maxmemory 0` (no limit, the Redis default), so a value set by hand with `CONFIG SET` is reset on the next reconcile. Must be at most 90% of the memory request of each Redis pod (from `resources` or `resourcesPreset`): the operator silently drops a larger value, so the chart refuses it. Leave headroom below that for replication and snapshot buffers. | `quantity` | `""` | +| `maxmemoryPolicy` | Redis `maxmemory-policy`: what Redis does once `maxmemory` is reached. `noeviction` refuses writes; `allkeys-lru` evicts the least recently used keys, which suits an instance used purely as a cache. Has no effect while `maxmemory` is unset. | `string` | `noeviction` | +| `snapshots` | Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save ""` is rendered and the volume is kept, so the data is lost on restart but replicas still have room for the full-sync RDB they write to disk. Always off when `size` is empty. | `bool` | `true` | ### Sentinel parameters diff --git a/packages/apps/redis/templates/redisfailover.yaml b/packages/apps/redis/templates/redisfailover.yaml index 05a96e3cc..ba36e3f6c 100644 --- a/packages/apps/redis/templates/redisfailover.yaml +++ b/packages/apps/redis/templates/redisfailover.yaml @@ -116,8 +116,33 @@ spec: - loglevel notice {{- if not .Values.size }} - appendonly no + {{- end }} + {{- /* The operator applies these lines with CONFIG SET on every reconcile and never + resets a line that disappears, so each one is always rendered with an explicit + value: switching back to the default must reach the running pods too. The + schedule is the Redis 7 and 8 default. */}} + {{- if and .Values.size .Values.snapshots }} + - save 3600 1 300 100 60 10000 + {{- else }} - save "" {{- end }} + {{- $maxmemory := 0 }} + {{- if not (kindIs "invalid" .Values.maxmemory) }} + {{- $maxmemory = include "cozy-lib.resources.toFloat" (.Values.maxmemory | toString) | float64 | int64 }} + {{- $redisResources := include "cozy-lib.resources.defaultingSanitize" (list .Values.resourcesPreset .Values.resources $) | fromYaml }} + {{- $memoryRequest := include "cozy-lib.resources.toFloat" (dig "requests" "memory" "0" $redisResources | toString) | float64 }} + {{- /* The operator compares maxmemory with 90% of the pod's memory request + (reservedPodMemoryPercent) and silently skips the line when it is larger. */}} + {{- $allowed := mulf $memoryRequest 0.9 | int64 }} + {{- if lt $maxmemory 0 }} + {{- fail (printf "maxmemory (%v) must not be negative" .Values.maxmemory) }} + {{- end }} + {{- if gt $maxmemory $allowed }} + {{- fail (printf "maxmemory (%v, %d bytes) must be at most 90%% of the Redis memory request (%d bytes): the operator ignores a larger value" .Values.maxmemory $maxmemory $allowed) }} + {{- end }} + {{- end }} + - maxmemory {{ $maxmemory }} + - maxmemory-policy {{ .Values.maxmemoryPolicy | default "noeviction" }} {{- /* A master with `replicas` pods has replicas-1 replicas, so a requirement of `replicas` or more can never be met and the master would refuse every write. */}} {{- if and (not (kindIs "invalid" .Values.minReplicasToWrite)) (ge (.Values.minReplicasToWrite | int) (.Values.replicas | int)) }} diff --git a/packages/apps/redis/tests/cache_config_test.yaml b/packages/apps/redis/tests/cache_config_test.yaml new file mode 100644 index 000000000..557b5d3d4 --- /dev/null +++ b/packages/apps/redis/tests/cache_config_test.yaml @@ -0,0 +1,118 @@ +suite: Redis cache tuning (maxmemory, eviction, snapshots) + +# The operator applies customConfig with CONFIG SET and never resets a line +# that disappears, so every directive here is rendered with an explicit value +# even at its default: reverting a setting has to reach the running pods. + +release: + name: my-redis + namespace: tenant-test + +set: + _cluster: + cluster-domain: cluster.local + _namespace: + host: example.com + +templates: + - templates/redisfailover.yaml + +tests: + - it: "renders the Redis defaults explicitly" + documentSelector: + path: kind + value: RedisFailover + asserts: + - contains: + path: spec.redis.customConfig + content: save 3600 1 300 100 60 10000 + - contains: + path: spec.redis.customConfig + content: maxmemory 0 + - contains: + path: spec.redis.customConfig + content: maxmemory-policy noeviction + - notContains: + path: spec.redis.customConfig + content: appendonly no + + - it: "disables snapshots while keeping the volume" + set: + snapshots: false + documentSelector: + path: kind + value: RedisFailover + asserts: + - contains: + path: spec.redis.customConfig + content: save "" + - notContains: + path: spec.redis.customConfig + content: save 3600 1 300 100 60 10000 + - equal: + path: spec.redis.storage.persistentVolumeClaim.spec.resources.requests.storage + value: 1Gi + + - it: "keeps snapshots off without a volume even when enabled" + set: + size: null + snapshots: true + documentSelector: + path: kind + value: RedisFailover + asserts: + - contains: + path: spec.redis.customConfig + content: save "" + - contains: + path: spec.redis.customConfig + content: appendonly no + - notExists: + path: spec.redis.storage + + - it: "renders maxmemory in bytes with the chosen policy" + set: + resourcesPreset: t1.large + maxmemory: 1Gi + maxmemoryPolicy: allkeys-lru + documentSelector: + path: kind + value: RedisFailover + asserts: + - contains: + path: spec.redis.customConfig + content: maxmemory 1073741824 + - contains: + path: spec.redis.customConfig + content: maxmemory-policy allkeys-lru + + - it: "sizes the maxmemory ceiling from explicit resources" + set: + resources: + memory: 1Gi + maxmemory: 900Mi + documentSelector: + path: kind + value: RedisFailover + asserts: + - contains: + path: spec.redis.customConfig + content: maxmemory 943718400 + + # The operator compares maxmemory with 90% of the pod's memory request and + # silently skips a larger line, which would leave Redis unbounded. + - it: "refuses maxmemory above 90% of the memory request" + set: + resources: + memory: 1Gi + maxmemory: 1000Mi + asserts: + - failedTemplate: + errorPattern: must be at most 90% of the Redis memory request + + - it: "refuses a negative maxmemory" + set: + maxmemory: -1Mi + asserts: + - failedTemplate: + errorPattern: must not be negative diff --git a/packages/apps/redis/tests/min_replicas_test.yaml b/packages/apps/redis/tests/min_replicas_test.yaml index f0de83e86..13217884d 100644 --- a/packages/apps/redis/tests/min_replicas_test.yaml +++ b/packages/apps/redis/tests/min_replicas_test.yaml @@ -1,7 +1,7 @@ suite: Redis min-replicas write safety -# Both settings are opt-in: unset keeps the Redis defaults, so existing -# instances render the same customConfig as before. +# Both settings are opt-in: unset keeps the Redis defaults, so neither +# directive is rendered. release: name: my-redis @@ -30,7 +30,7 @@ tests: pattern: ^tcp-keepalive 0$ - lengthEqual: path: spec.redis.customConfig - count: 2 + count: 5 - it: "renders min-replicas-to-write when set" set: diff --git a/packages/apps/redis/values.schema.json b/packages/apps/redis/values.schema.json index 1a301615f..b3b143320 100644 --- a/packages/apps/redis/values.schema.json +++ b/packages/apps/redis/values.schema.json @@ -171,6 +171,39 @@ "type": "integer", "minimum": 0 }, + "maxmemory": { + "description": "Redis `maxmemory`: the dataset size at which Redis applies `maxmemoryPolicy`. Unset by default, which renders `maxmemory 0` (no limit, the Redis default), so a value set by hand with `CONFIG SET` is reset on the next reconcile. Must be at most 90% of the memory request of each Redis pod (from `resources` or `resourcesPreset`): the operator silently drops a larger value, so the chart refuses it. Leave headroom below that for replication and snapshot buffers.", + "pattern": "^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$", + "anyOf": [ + { + "type": "integer" + }, + { + "type": "string" + } + ], + "x-kubernetes-int-or-string": true + }, + "maxmemoryPolicy": { + "description": "Redis `maxmemory-policy`: what Redis does once `maxmemory` is reached. `noeviction` refuses writes; `allkeys-lru` evicts the least recently used keys, which suits an instance used purely as a cache. Has no effect while `maxmemory` is unset.", + "type": "string", + "default": "noeviction", + "enum": [ + "noeviction", + "allkeys-lru", + "allkeys-lfu", + "allkeys-random", + "volatile-lru", + "volatile-lfu", + "volatile-random", + "volatile-ttl" + ] + }, + "snapshots": { + "description": "Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save \"\"` is rendered and the volume is kept, so the data is lost on restart but replicas still have room for the full-sync RDB they write to disk. Always off when `size` is empty.", + "type": "boolean", + "default": true + }, "sentinel": { "description": "Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.", "type": "object", diff --git a/packages/apps/redis/values.yaml b/packages/apps/redis/values.yaml index 7917ca34d..0f8380ae7 100644 --- a/packages/apps/redis/values.yaml +++ b/packages/apps/redis/values.yaml @@ -119,6 +119,24 @@ authEnabled: true ## @param {*int} [minReplicasMaxLag] - Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it. ## @minimum 0 +## @param {quantity} [maxmemory] - Redis `maxmemory`: the dataset size at which Redis applies `maxmemoryPolicy`. Unset by default, which renders `maxmemory 0` (no limit, the Redis default), so a value set by hand with `CONFIG SET` is reset on the next reconcile. Must be at most 90% of the memory request of each Redis pod (from `resources` or `resourcesPreset`): the operator silently drops a larger value, so the chart refuses it. Leave headroom below that for replication and snapshot buffers. + +## @enum {string} MaxmemoryPolicy - Values accepted by the Redis `maxmemory-policy` directive. +## @value noeviction +## @value allkeys-lru +## @value allkeys-lfu +## @value allkeys-random +## @value volatile-lru +## @value volatile-lfu +## @value volatile-random +## @value volatile-ttl + +## @param {MaxmemoryPolicy} maxmemoryPolicy="noeviction" - Redis `maxmemory-policy`: what Redis does once `maxmemory` is reached. `noeviction` refuses writes; `allkeys-lru` evicts the least recently used keys, which suits an instance used purely as a cache. Has no effect while `maxmemory` is unset. +maxmemoryPolicy: "noeviction" + +## @param {bool} snapshots - Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save ""` is rendered and the volume is kept, so the data is lost on restart but replicas still have room for the full-sync RDB they write to disk. Always off when `size` is empty. +snapshots: true + ## ## @section Sentinel parameters ## diff --git a/packages/system/redis-rd/cozyrds/redis.yaml b/packages/system/redis-rd/cozyrds/redis.yaml index cfcaeb01d..02046b97e 100644 --- a/packages/system/redis-rd/cozyrds/redis.yaml +++ b/packages/system/redis-rd/cozyrds/redis.yaml @@ -8,7 +8,7 @@ spec: plural: redises singular: redis openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.nano","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster. Can be enabled only when the instance is created, together with `tls.enabled: true`; turning it on for an existing instance, or changing `tls.enabled` while it is on, is refused at admission.","type":"boolean","default":false},"version":{"description":"Redis major version to deploy. The Redis and Sentinel pods run the same image.","type":"string","default":"v8","enum":["v8","v7"]},"tls":{"description":"TLS configuration. TLS is opt-in and is not inferred from `external`.","type":"object","default":{},"properties":{"authClients":{"description":"Maps to the Redis `tls-auth-clients` directive. Defaults to `no` — the server certificate is presented but client certificates are not validated. `yes` requires a client certificate signed by this release's CA, which the platform does not issue. Changing it restarts the Redis and Sentinel pods, because the directive is read from the config file once at startup.","type":"string","enum":["no","optional","yes"]},"enabled":{"description":"Enable TLS for Redis and Sentinel connections. Disabled unless set. Enabling it moves Redis and Sentinel to a TLS-only listener, so existing plaintext clients must be migrated at the same time. Encryption is provided by the redis-operator fork that mounts the certificate Secret into both Redis and Sentinel pods at `/tls`.","type":"boolean"}}},"authEnabled":{"description":"Enable password generation.","type":"boolean","default":true},"minReplicasToWrite":{"description":"Redis `min-replicas-to-write`: the master refuses writes unless at least this many replicas are connected and within `minReplicasMaxLag`. Unset by default, which leaves the Redis default (disabled). With `replicas: 2` and a value of 1 the master rejects writes while its only replica is down, trading availability for a bound on the writes lost if an isolated master is failed over. Replication stays asynchronous, so this bounds the loss, it does not remove it. Must be 0 or more and below `replicas` (a master with `replicas` pods has `replicas - 1` replicas), otherwise every write is refused; the chart refuses such a value.","type":"integer","minimum":0},"minReplicasMaxLag":{"description":"Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it.","type":"integer","minimum":0},"sentinel":{"description":"Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.","type":"object","default":{},"properties":{"resources":{"description":"Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`.","type":"object","properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}} + {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.nano","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster. Can be enabled only when the instance is created, together with `tls.enabled: true`; turning it on for an existing instance, or changing `tls.enabled` while it is on, is refused at admission.","type":"boolean","default":false},"version":{"description":"Redis major version to deploy. The Redis and Sentinel pods run the same image.","type":"string","default":"v8","enum":["v8","v7"]},"tls":{"description":"TLS configuration. TLS is opt-in and is not inferred from `external`.","type":"object","default":{},"properties":{"authClients":{"description":"Maps to the Redis `tls-auth-clients` directive. Defaults to `no` — the server certificate is presented but client certificates are not validated. `yes` requires a client certificate signed by this release's CA, which the platform does not issue. Changing it restarts the Redis and Sentinel pods, because the directive is read from the config file once at startup.","type":"string","enum":["no","optional","yes"]},"enabled":{"description":"Enable TLS for Redis and Sentinel connections. Disabled unless set. Enabling it moves Redis and Sentinel to a TLS-only listener, so existing plaintext clients must be migrated at the same time. Encryption is provided by the redis-operator fork that mounts the certificate Secret into both Redis and Sentinel pods at `/tls`.","type":"boolean"}}},"authEnabled":{"description":"Enable password generation.","type":"boolean","default":true},"minReplicasToWrite":{"description":"Redis `min-replicas-to-write`: the master refuses writes unless at least this many replicas are connected and within `minReplicasMaxLag`. Unset by default, which leaves the Redis default (disabled). With `replicas: 2` and a value of 1 the master rejects writes while its only replica is down, trading availability for a bound on the writes lost if an isolated master is failed over. Replication stays asynchronous, so this bounds the loss, it does not remove it. Must be 0 or more and below `replicas` (a master with `replicas` pods has `replicas - 1` replicas), otherwise every write is refused; the chart refuses such a value.","type":"integer","minimum":0},"minReplicasMaxLag":{"description":"Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it.","type":"integer","minimum":0},"maxmemory":{"description":"Redis `maxmemory`: the dataset size at which Redis applies `maxmemoryPolicy`. Unset by default, which renders `maxmemory 0` (no limit, the Redis default), so a value set by hand with `CONFIG SET` is reset on the next reconcile. Must be at most 90% of the memory request of each Redis pod (from `resources` or `resourcesPreset`): the operator silently drops a larger value, so the chart refuses it. Leave headroom below that for replication and snapshot buffers.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"maxmemoryPolicy":{"description":"Redis `maxmemory-policy`: what Redis does once `maxmemory` is reached. `noeviction` refuses writes; `allkeys-lru` evicts the least recently used keys, which suits an instance used purely as a cache. Has no effect while `maxmemory` is unset.","type":"string","default":"noeviction","enum":["noeviction","allkeys-lru","allkeys-lfu","allkeys-random","volatile-lru","volatile-lfu","volatile-random","volatile-ttl"]},"snapshots":{"description":"Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save \"\"` is rendered and the volume is kept, so the data is lost on restart but replicas still have room for the full-sync RDB they write to disk. Always off when `size` is empty.","type":"boolean","default":true},"sentinel":{"description":"Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.","type":"object","default":{},"properties":{"resources":{"description":"Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`.","type":"object","properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}} release: prefix: redis- labels: @@ -25,7 +25,7 @@ spec: tags: - cache icon: PHN2ZyB3aWR0aD0iMTQ0IiBoZWlnaHQ9IjE0NCIgdmlld0JveD0iMCAwIDE0NCAxNDQiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CjxyZWN0IHdpZHRoPSIxNDQiIGhlaWdodD0iMTQ0IiByeD0iMjQiIGZpbGw9InVybCgjcGFpbnQwX2xpbmVhcl82ODNfMzIxMykiLz4KPHBhdGggZD0iTTEyMC4xNDkgOTUuNTQ5MUMxMTQuNTg2IDk4LjQ0ODUgODUuNzcwOSAxMTAuMjk2IDc5LjYzNjMgMTEzLjQ5NUM3My41MDE2IDExNi42OTMgNzAuMDkzNyAxMTYuNjYyIDY1LjI0NzIgMTE0LjM0NkM2MC40MDEyIDExMi4wMjkgMjkuNzM2NCA5OS42NDIzIDI0LjIxMjUgOTcuMDAxOUMyMS40NTE5IDk1LjY4MjcgMjAgOTQuNTY4NyAyMCA5My41MTY2VjgyLjk4MDlDMjAgODIuOTgwOSA1OS45MjIgNzQuMjkwMSA2Ni4zNjY5IDcxLjk3NzhDNzIuODExNSA2OS42NjU2IDc1LjA0NzYgNjkuNTgyMSA4MC41MzIgNzEuNTkxQzg2LjAxNzMgNzMuNjAwOCAxMTguODEyIDc5LjUxNzYgMTI0LjIzMyA4MS41MDI5TDEyNC4yMyA5MS44ODk2QzEyNC4yMzEgOTIuOTMxMSAxMjIuOTggOTQuMDczNiAxMjAuMTQ5IDk1LjU0OTFaIiBmaWxsPSIjOTEyNjI2Ii8+CjxwYXRoIGQ9Ik0xMjAuMTQ3IDg1LjA3NTJDMTE0LjU4NSA4Ny45NzM0IDg1Ljc3IDk5LjgyMTggNzkuNjM1NCAxMDMuMDJDNzMuNTAxMSAxMDYuMjE5IDcwLjA5MzIgMTA2LjE4NyA2NS4yNDcyIDEwMy44NzFDNjAuNDAwNyAxMDEuNTU1IDI5LjczNzEgODkuMTY2OCAyNC4yMTM2IDg2LjUyOEMxOC42OTAxIDgzLjg4NzYgMTguNTc0NCA4Mi4wNzA0IDI0LjAwMDMgNzkuOTQ1OEMyOS40MjYxIDc3LjgyMDUgNTkuOTIxNSA2NS44NTYxIDY2LjM2NzIgNjMuNTQzOEM3Mi44MTE4IDYxLjIzMjQgNzUuMDQ3NSA2MS4xNDgxIDgwLjUzMTkgNjMuMTU3OEM4Ni4wMTY4IDY1LjE2NjggMTE0LjY2IDc2LjU2NzYgMTIwLjA3OSA3OC41NTI1QzEyNS41MDEgODAuNTM5OSAxMjUuNzA5IDgyLjE3NjMgMTIwLjE0NyA4NS4wNzUyWiIgZmlsbD0iI0M2MzAyQiIvPgo8cGF0aCBkPSJNMTIwLjE0OSA3OC41MDJDMTE0LjU4NiA4MS40MDE4IDg1Ljc3MDkgOTMuMjQ5MyA3OS42MzYzIDk2LjQ0ODhDNzMuNTAxNiA5OS42NDYyIDcwLjA5MzcgOTkuNjE1MiA2NS4yNDcyIDk3LjI5ODVDNjAuNDAwOCA5NC45ODMgMjkuNzM2NCA4Mi41OTUyIDI0LjIxMjUgNzkuOTU0N0MyMS40NTE5IDc4LjYzNTUgMjAgNzcuNTIzMiAyMCA3Ni40NzA3VjY1LjkzMzhDMjAgNjUuOTMzOCA1OS45MjIgNTcuMjQzNCA2Ni4zNjY5IDU0LjkzMTFDNzIuODExNSA1Mi42MTg5IDc1LjA0NzYgNTIuNTM1IDgwLjUzMiA1NC41NDQzQzg2LjAxNzcgNTYuNTUzNiAxMTguODEzIDYyLjQ2OTMgMTI0LjIzMyA2NC40NTVMMTI0LjIzIDc0Ljg0MjhDMTI0LjIzMSA3NS44ODQgMTIyLjk4IDc3LjAyNjQgMTIwLjE0OSA3OC41MDJaIiBmaWxsPSIjOTEyNjI2Ii8+CjxwYXRoIGQ9Ik0xMjAuMTQ3IDY4LjAyODJDMTE0LjU4NSA3MC45MjcxIDg1Ljc3IDgyLjc3NDcgNzkuNjM1NCA4NS45NzM3QzczLjUwMTEgODkuMTcxNiA3MC4wOTMyIDg5LjE0MDIgNjUuMjQ3MiA4Ni44MjM1QzYwLjQwMDcgODQuNTA4NCAyOS43MzcxIDcyLjEyMDEgMjQuMjEzNiA2OS40ODA5QzE4LjY5MDEgNjYuODQxMyAxOC41NzQ0IDY1LjAyMzcgMjQuMDAwMyA2Mi44OTg0QzI5LjQyNjEgNjAuNzc0MiA1OS45MjE5IDQ4LjgwOSA2Ni4zNjcyIDQ2LjQ5NzJDNzIuODExOCA0NC4xODUzIDc1LjA0NzUgNDQuMTAxNCA4MC41MzE5IDQ2LjExMDhDODYuMDE2OCA0OC4xMTk3IDExNC42NiA1OS41MTk3IDEyMC4wNzkgNjEuNTA1NUMxMjUuNTAxIDYzLjQ5MjQgMTI1LjcwOSA2NS4xMjkyIDEyMC4xNDcgNjguMDI4MloiIGZpbGw9IiNDNjMwMkIiLz4KPHBhdGggZD0iTTEyMC4xNDkgNjAuODIyNEMxMTQuNTg2IDYzLjcyMTQgODUuNzcwOSA3NS41Njk4IDc5LjYzNjMgNzguNzY5MkM3My41MDE2IDgxLjk2NzEgNzAuMDkzNyA4MS45MzU3IDY1LjI0NzIgNzkuNjE5QzYwLjQwMDggNzcuMzAzNSAyOS43MzY0IDY0LjkxNTIgMjQuMjEyNSA2Mi4yNzZDMjEuNDUxOSA2MC45NTU2IDIwIDU5Ljg0MjggMjAgNTguNzkxNVY0OC4yNTQyQzIwIDQ4LjI1NDIgNTkuOTIyIDM5LjU2NDIgNjYuMzY2OSAzNy4yNTI0QzcyLjgxMTUgMzQuOTM5NyA3NS4wNDc2IDM0Ljg1NjcgODAuNTMyIDM2Ljg2NTZDODYuMDE3NyAzOC44NzQ5IDExOC44MTMgNDQuNzkwNSAxMjQuMjMzIDQ2Ljc3NjNMMTI0LjIzIDU3LjE2MzdDMTI0LjIzMSA1OC4yMDQgMTIyLjk4IDU5LjM0NjUgMTIwLjE0OSA2MC44MjI0WiIgZmlsbD0iIzkxMjYyNiIvPgo8cGF0aCBkPSJNMTIwLjE0NyA1MC4zNDlDMTE0LjU4NSA1My4yNDc5IDg1Ljc2OTggNjUuMDk2MyA3OS42MzUyIDY4LjI5NDFDNzMuNTAwOSA3MS40OTIgNzAuMDkzIDcxLjQ2MDYgNjUuMjQ2OSA2OS4xNDUxQzYwLjQwMDkgNjYuODI4MyAyOS43MzY5IDU0LjQ0MDkgMjQuMjEzOCA1MS44MDEzQzE4LjY4OTkgNDkuMTYyMSAxOC41NzQ2IDQ3LjM0NDEgMjQgNDUuMjE5MkMyOS40MjU5IDQzLjA5NDYgNTkuOTIxNyAzMS4xMzEgNjYuMzY3IDI4LjgxODRDNzIuODExNiAyNi41MDYxIDc1LjA0NzMgMjYuNDIzIDgwLjUzMTcgMjguNDMyNEM4Ni4wMTY2IDMwLjQ0MTcgMTE0LjY1OSA0MS44NDE4IDEyMC4wNzkgNDMuODI3NUMxMjUuNTAxIDQ1LjgxMjggMTI1LjcwOSA0Ny40NSAxMjAuMTQ3IDUwLjM0OVoiIGZpbGw9IiNDNjMwMkIiLz4KPHBhdGggZD0iTTg0Ljg1NDEgNDAuMDk5NEw3NS44OTI2IDQxLjAyOThMNzMuODg2NSA0NS44NTdMNzAuNjQ2MyA0MC40NzAzTDYwLjI5ODMgMzkuNTQwNEw2OC4wMTk3IDM2Ljc1NThMNjUuNzAzIDMyLjQ4MTRMNzIuOTMyMSAzNS4zMDg4TDc5Ljc0NzEgMzMuMDc3NUw3Ny45MDUyIDM3LjQ5NzJMODQuODU0MSA0MC4wOTk0Wk03My4zNTE1IDYzLjUxODRMNTYuNjI2NiA1Ni41ODE2TDgwLjU5MiA1Mi45MDI5TDczLjM1MTUgNjMuNTE4NFpNNTAuMTYzNyA0Mi43ODI2QzU3LjIzODEgNDIuNzgyNiA2Mi45NzMgNDUuMDA1NyA2Mi45NzMgNDcuNzQ3NUM2Mi45NzMgNTAuNDkwMSA1Ny4yMzgxIDUyLjcxMjggNTAuMTYzNyA1Mi43MTI4QzQzLjA4OTMgNTIuNzEyOCAzNy4zNTQ1IDUwLjQ4OTcgMzcuMzU0NSA0Ny43NDc1QzM3LjM1NDUgNDUuMDA1NyA0My4wODkzIDQyLjc4MjYgNTAuMTYzNyA0Mi43ODI2WiIgZmlsbD0id2hpdGUiLz4KPHBhdGggZD0iTTk1LjQ0MzQgNDEuNDE3NEwxMDkuNjI3IDQ3LjAyMjRMOTUuNDU1NiA1Mi42MjJMOTUuNDQzNCA0MS40MTc0WiIgZmlsbD0iIzYyMUIxQyIvPgo8cGF0aCBkPSJNNzkuNzUyOSA0Ny42MjYxTDk1LjQ0NDkgNDEuNDE4OUw5NS40NTcxIDUyLjYyMzZMOTMuOTE4NCA1My4yMjU0TDc5Ljc1MjkgNDcuNjI2MVoiIGZpbGw9IiM5QTI5MjgiLz4KPGRlZnM+CjxsaW5lYXJHcmFkaWVudCBpZD0icGFpbnQwX2xpbmVhcl82ODNfMzIxMyIgeDE9IjE4OSIgeTE9IjIxMC41IiB4Mj0iMCIgeTI9IjAiIGdyYWRpZW50VW5pdHM9InVzZXJTcGFjZU9uVXNlIj4KPHN0b3Agc3RvcC1jb2xvcj0iI0E4MDAwMCIvPgo8c3RvcCBvZmZzZXQ9IjEiIHN0b3AtY29sb3I9IiNGRkNGQ0YiLz4KPC9saW5lYXJHcmFkaWVudD4KPC9kZWZzPgo8L3N2Zz4K - keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "replicas"], ["spec", "resources"], ["spec", "resourcesPreset"], ["spec", "size"], ["spec", "storageClass"], ["spec", "external"], ["spec", "version"], ["spec", "tls"], ["spec", "authEnabled"], ["spec", "sentinel"]] + keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "replicas"], ["spec", "resources"], ["spec", "resourcesPreset"], ["spec", "size"], ["spec", "storageClass"], ["spec", "external"], ["spec", "version"], ["spec", "tls"], ["spec", "authEnabled"], ["spec", "maxmemoryPolicy"], ["spec", "snapshots"], ["spec", "sentinel"]] secrets: # Backstop for the label-only include below, which matches on the label # alone with no name constraint. These two Secrets are the ones whose From 4c3a4b42ccc70343895237cce4de69fa6b7ea16b Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:27:04 +0300 Subject: [PATCH 480/518] style: clear unused code and style findings reported by staticcheck Remove helpers and constants nothing references, escape format characters in test literals, and take the simplifications staticcheck suggests where they read the same. Error strings no longer start with a capital: they keep their meaning, with the kind or component name moved out of first position. Callers match errors by type, not text; the one test that matches text, for Kafka, keeps the phrase it looks for. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- cmd/backport-batch/injection_test.go | 6 ++--- cmd/backport-batch/onto_test.go | 4 +-- cmd/backport-batch/policy_test.go | 4 +-- cmd/cozypkg/cmd/add.go | 2 +- cmd/cozystack-operator/main.go | 2 +- .../altinitystrategy_controller.go | 2 +- .../etcdstrategy_controller.go | 4 +-- .../kafkastrategy_controller.go | 6 ++--- .../rabbitmqstrategy_controller.go | 2 +- internal/controller/kubeovnplunger/metrics.go | 2 +- internal/controller/workload_controller.go | 5 ---- internal/fluxinstall/install.go | 2 +- internal/lineagecontrollerwebhook/webhook.go | 2 +- internal/migrationcontroller/inventory.go | 4 +-- internal/operator/package_reconciler_test.go | 9 ------- internal/telemetry/collector.go | 4 +-- internal/telemetry/operator_collector.go | 22 +++++++-------- pkg/apiserver/apiserver.go | 6 ++--- pkg/cmd/server/openapi.go | 4 +-- pkg/ovnstatus/ovnstatus.go | 27 ------------------- pkg/ovnstatus/ovnstatus_test.go | 4 +-- pkg/reservation/evaluator.go | 6 ++--- 22 files changed, 43 insertions(+), 86 deletions(-) diff --git a/cmd/backport-batch/injection_test.go b/cmd/backport-batch/injection_test.go index f1d12b004..a1f5abbe6 100644 --- a/cmd/backport-batch/injection_test.go +++ b/cmd/backport-batch/injection_test.go @@ -142,11 +142,11 @@ func TestOptionInjection(t *testing.T) { // no bidi override reorders it and no zero-width character hides in it. func TestDeferredBlockCleansFileNames(t *testing.T) { l, _ := parseLine("release-1.6") - files := []string{"app/‮txt.exe", "app/zero​width.yaml"} + files := []string{"app/\u202etxt.exe", "app/zero\u200bwidth.yaml"} conflict := pick{v: &verdict{o: &original{number: 5}}, files: files, failed: strings.Repeat("a", 40)} waiting := pick{v: &verdict{o: &original{number: 6}}, files: files} block := deferredBlock(l, []pick{conflict}, nil, []pick{waiting}) - if strings.ContainsAny(block, "‮​") { + if strings.ContainsAny(block, "\u202e\u200b") { t.Errorf("the block carries format characters: %q", block) } want := []string{"app/txt.exe", "app/zerowidth.yaml"} @@ -154,7 +154,7 @@ func TestDeferredBlockCleansFileNames(t *testing.T) { if err != nil || !slices.Equal(got[5].Files, want) || !slices.Equal(got[6].Files, want) { t.Errorf("parseDeferred = %+v, %v; want both originals with %q", got, err, want) } - if files[0] != "app/‮txt.exe" { + if files[0] != "app/\u202etxt.exe" { t.Errorf("deferredBlock rewrote the pick's own file names: %q", files) } } diff --git a/cmd/backport-batch/onto_test.go b/cmd/backport-batch/onto_test.go index ee4d52d33..994059235 100644 --- a/cmd/backport-batch/onto_test.go +++ b/cmd/backport-batch/onto_test.go @@ -41,12 +41,12 @@ func TestOntoContinues(t *testing.T) { f.lines() head := pushed(t, f) f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) - code, stdout, stderr := f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--onto", head, "--date", "2026-10-29") + code, _, stderr := f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--onto", head, "--date", "2026-10-29") if code != 2 || !strings.Contains(stderr, "not the head of backport-batch/release-1.1-20261029") { t.Errorf("--onto with another date = %d (%s), want a refusal", code, stderr) } f.now = f.now.AddDate(0, 0, 3) - code, stdout, stderr = f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--onto", head) + code, stdout, stderr := f.exec(&fakeGH{}, "run", "--line", "release-1.1", "--onto", head) if code != 0 { t.Fatalf("run --onto = %d\n%s", code, stderr) } diff --git a/cmd/backport-batch/policy_test.go b/cmd/backport-batch/policy_test.go index 920c18a0b..2cd7d20f3 100644 --- a/cmd/backport-batch/policy_test.go +++ b/cmd/backport-batch/policy_test.go @@ -176,7 +176,7 @@ func TestPolicyPageMatchesCode(t *testing.T) { t.Errorf("the page's block gives %d released lines, want 2 written out", pol.released) } - rule2 := regexp.MustCompile("(?m)^2\\. Its title type is (.*)$").FindStringSubmatch(page) + rule2 := regexp.MustCompile(`(?m)^2\. Its title type is (.*)$`).FindStringSubmatch(page) if rule2 == nil { t.Fatal("the page has no rule 2") } @@ -187,7 +187,7 @@ func TestPolicyPageMatchesCode(t *testing.T) { if !slices.Equal(types, takenTypes) { t.Errorf("rule 2 on the page takes %v, the code %v", types, takenTypes) } - never := regexp.MustCompile("(?m)^### What waits for the next minor\\n\\n(.*)$").FindStringSubmatch(page) + never := regexp.MustCompile(`(?m)^### What waits for the next minor\n\n(.*)$`).FindStringSubmatch(page) if never == nil { t.Fatal("the page has no list of what never goes into a patch release") } diff --git a/cmd/cozypkg/cmd/add.go b/cmd/cozypkg/cmd/add.go index d882ce287..8c5e22026 100644 --- a/cmd/cozypkg/cmd/add.go +++ b/cmd/cozypkg/cmd/add.go @@ -390,7 +390,7 @@ func createPackageFromFile(ctx context.Context, k8sClient client.Client, filePat } } - return fmt.Errorf("Package %s not found in file", packageName) + return fmt.Errorf("no Package %s in file", packageName) } func installPackage(ctx context.Context, k8sClient client.Client, packageSourceName string) error { diff --git a/cmd/cozystack-operator/main.go b/cmd/cozystack-operator/main.go index e11290aed..e1a2aa1de 100644 --- a/cmd/cozystack-operator/main.go +++ b/cmd/cozystack-operator/main.go @@ -663,7 +663,7 @@ func validateGitRef(refMap map[string]string) error { return fmt.Errorf("commit SHA should be at least 7 characters, got: %s", commit) } for _, c := range commit { - if !((c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F')) { + if (c < '0' || c > '9') && (c < 'a' || c > 'f') && (c < 'A' || c > 'F') { return fmt.Errorf("commit SHA should be a hexadecimal string, got: %s", commit) } } diff --git a/internal/backupcontroller/altinitystrategy_controller.go b/internal/backupcontroller/altinitystrategy_controller.go index f0c597583..4314c3dbf 100644 --- a/internal/backupcontroller/altinitystrategy_controller.go +++ b/internal/backupcontroller/altinitystrategy_controller.go @@ -631,7 +631,7 @@ func (r *BackupJobReconciler) createAltinityBackupArtifact( // used name meets the earlier run's Backup here. Only one recording // this run's archive was created by an earlier reconcile of this run. if got := existing.Spec.DriverMetadata[altinityArchiveMetaKey]; got != archive { - return nil, fmt.Errorf("Backup %s/%s already exists and records archive %q, not %q that this run wrote", + return nil, fmt.Errorf("existing Backup %s/%s records archive %q, not %q that this run wrote", backup.Namespace, backup.Name, got, archive) } return existing, nil diff --git a/internal/backupcontroller/etcdstrategy_controller.go b/internal/backupcontroller/etcdstrategy_controller.go index 2f2f2732c..4f530ca2f 100644 --- a/internal/backupcontroller/etcdstrategy_controller.go +++ b/internal/backupcontroller/etcdstrategy_controller.go @@ -134,14 +134,14 @@ var etcdClusterGVR = schema.GroupVersionResource{ // the wrong CRD via the typed client. func validateEtcdApplicationRef(ref corev1.TypedLocalObjectReference) error { if ref.Kind != etcdAppKind { - return fmt.Errorf("Etcd strategy supports applicationRef.kind=%q, got %q", etcdAppKind, ref.Kind) + return fmt.Errorf("the Etcd strategy supports applicationRef.kind=%q, got %q", etcdAppKind, ref.Kind) } apiGroup := "" if ref.APIGroup != nil { apiGroup = *ref.APIGroup } if apiGroup != "" && apiGroup != etcdapp.GroupName { - return fmt.Errorf("Etcd strategy supports applicationRef.apiGroup=%q, got %q", etcdapp.GroupName, apiGroup) + return fmt.Errorf("the Etcd strategy supports applicationRef.apiGroup=%q, got %q", etcdapp.GroupName, apiGroup) } return nil } diff --git a/internal/backupcontroller/kafkastrategy_controller.go b/internal/backupcontroller/kafkastrategy_controller.go index 70f594d65..f7c34ce53 100644 --- a/internal/backupcontroller/kafkastrategy_controller.go +++ b/internal/backupcontroller/kafkastrategy_controller.go @@ -104,10 +104,10 @@ func kafkaClusterName(appName string) string { // apps.cozystack.io/Kafka application. func validateKafkaApplicationRef(ref corev1.TypedLocalObjectReference) error { if ref.Kind != kafkaApplicationKind { - return fmt.Errorf("Kafka strategy only supports applicationRef.kind=%q, got %q", kafkaApplicationKind, ref.Kind) + return fmt.Errorf("the Kafka strategy only supports applicationRef.kind=%q, got %q", kafkaApplicationKind, ref.Kind) } if ref.APIGroup != nil && *ref.APIGroup != "" && *ref.APIGroup != backupsv1alpha1.DefaultApplicationAPIGroup { - return fmt.Errorf("Kafka strategy only supports applicationRef.apiGroup=%q, got %q", backupsv1alpha1.DefaultApplicationAPIGroup, *ref.APIGroup) + return fmt.Errorf("the Kafka strategy only supports applicationRef.apiGroup=%q, got %q", backupsv1alpha1.DefaultApplicationAPIGroup, *ref.APIGroup) } return nil } @@ -528,7 +528,7 @@ func (r *BackupJobReconciler) createKafkaBackupArtifact( // RabbitMQ driver's guard). if existing.Spec.ApplicationRef.Kind != backup.Spec.ApplicationRef.Kind || existing.Spec.ApplicationRef.Name != backup.Spec.ApplicationRef.Name { - return nil, fmt.Errorf("Backup %s/%s already exists for a different application (%s/%s, not %s/%s)", + return nil, fmt.Errorf("existing Backup %s/%s belongs to a different application (%s/%s, not %s/%s)", backup.Namespace, backup.Name, existing.Spec.ApplicationRef.Kind, existing.Spec.ApplicationRef.Name, backup.Spec.ApplicationRef.Kind, backup.Spec.ApplicationRef.Name) diff --git a/internal/backupcontroller/rabbitmqstrategy_controller.go b/internal/backupcontroller/rabbitmqstrategy_controller.go index adfe11d83..4c3d9a55c 100644 --- a/internal/backupcontroller/rabbitmqstrategy_controller.go +++ b/internal/backupcontroller/rabbitmqstrategy_controller.go @@ -513,7 +513,7 @@ func (r *BackupJobReconciler) createRabbitmqBackupArtifact( // this run's success while describing something else. if existing.Spec.ApplicationRef.Kind != backup.Spec.ApplicationRef.Kind || existing.Spec.ApplicationRef.Name != backup.Spec.ApplicationRef.Name { - return nil, fmt.Errorf("Backup %s/%s already exists for a different application (%s/%s, not %s/%s)", + return nil, fmt.Errorf("existing Backup %s/%s belongs to a different application (%s/%s, not %s/%s)", backup.Namespace, backup.Name, existing.Spec.ApplicationRef.Kind, existing.Spec.ApplicationRef.Name, backup.Spec.ApplicationRef.Kind, backup.Spec.ApplicationRef.Name) diff --git a/internal/controller/kubeovnplunger/metrics.go b/internal/controller/kubeovnplunger/metrics.go index 1330db814..523467fcd 100644 --- a/internal/controller/kubeovnplunger/metrics.go +++ b/internal/controller/kubeovnplunger/metrics.go @@ -219,7 +219,7 @@ func (r *KubeOVNPlunger) WriteClusterMetrics(db string, snaps []ovnstatus.Health r.metrics.allAgree.WithLabelValues(db, cid).Set(b2f(ecv.AllAgree)) r.metrics.membersExpected.WithLabelValues(db, cid).Set(float64(expectedReplicas)) r.metrics.membersObserved.WithLabelValues(db, cid).Set(float64(ecv.MembersCount)) - r.metrics.ipsExpected.WithLabelValues(db, cid).Set(float64(len(ecv.ConsensusResult.TruthView.Members))) // optional; or len(hints.ExpectedIPs) + r.metrics.ipsExpected.WithLabelValues(db, cid).Set(float64(len(ecv.TruthView.Members))) // optional; or len(hints.ExpectedIPs) r.metrics.ipsObserved.WithLabelValues(db, cid).Set(float64(ecv.DistinctIPCount)) r.metrics.excessMembers.WithLabelValues(db, cid).Set(float64(ecv.ExpectedExcess)) r.metrics.missingMembers.WithLabelValues(db, cid).Set(float64(ecv.ExpectedShortfall)) diff --git a/internal/controller/workload_controller.go b/internal/controller/workload_controller.go index d0d9e7430..052358a05 100644 --- a/internal/controller/workload_controller.go +++ b/internal/controller/workload_controller.go @@ -3,7 +3,6 @@ package controller import ( "context" "strings" - "time" corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" @@ -15,10 +14,6 @@ import ( cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1" ) -const ( - deletionRequeueDelay = 30 * time.Second -) - // WorkloadMonitorReconciler reconciles a WorkloadMonitor object type WorkloadReconciler struct { client.Client diff --git a/internal/fluxinstall/install.go b/internal/fluxinstall/install.go index d16d7d854..042bdfd5e 100644 --- a/internal/fluxinstall/install.go +++ b/internal/fluxinstall/install.go @@ -171,7 +171,7 @@ func extractNamespace(objects []*unstructured.Unstructured) (string, error) { if obj.GetKind() == "Namespace" { namespace := obj.GetName() if namespace == "" { - return "", fmt.Errorf("Namespace object has no name") + return "", fmt.Errorf("the Namespace object has no name") } return namespace, nil } diff --git a/internal/lineagecontrollerwebhook/webhook.go b/internal/lineagecontrollerwebhook/webhook.go index ea8b0d01b..7b29e1f76 100644 --- a/internal/lineagecontrollerwebhook/webhook.go +++ b/internal/lineagecontrollerwebhook/webhook.go @@ -176,7 +176,7 @@ func (h *LineageControllerWebhook) computeLabels(ctx context.Context, obj *unstr return s } s = s[:63] - for b := s[62]; !((b >= 'a' && b <= 'z') || (b >= 'A' && b <= 'Z') || (b >= '0' && b <= '9')); s = s[:len(s)-1] { + for b := s[62]; (b < 'a' || b > 'z') && (b < 'A' || b > 'Z') && (b < '0' || b > '9'); s = s[:len(s)-1] { b = s[len(s)-1] } return s diff --git a/internal/migrationcontroller/inventory.go b/internal/migrationcontroller/inventory.go index d56bec2da..52a5e4cd8 100644 --- a/internal/migrationcontroller/inventory.go +++ b/internal/migrationcontroller/inventory.go @@ -146,7 +146,7 @@ func (c *inventoryClient) VMs(ctx context.Context, providerUID string) ([]invent defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { - return nil, fmt.Errorf("Forklift inventory returned %s listing VMs", resp.Status) + return nil, fmt.Errorf("listing VMs: the Forklift inventory returned %s", resp.Status) } var vms []inventoryVMRef @@ -195,7 +195,7 @@ func (c *inventoryClient) VM(ctx context.Context, providerUID, vmID string) (*in case http.StatusNotFound: return nil, ¬FoundError{vmID: vmID} default: - return nil, fmt.Errorf("Forklift inventory returned %s for VM %q", resp.Status, vmID) + return nil, fmt.Errorf("VM %q: the Forklift inventory returned %s", vmID, resp.Status) } vm := &inventoryVM{} diff --git a/internal/operator/package_reconciler_test.go b/internal/operator/package_reconciler_test.go index c9923e7dc..fd8a15fe5 100644 --- a/internal/operator/package_reconciler_test.go +++ b/internal/operator/package_reconciler_test.go @@ -790,15 +790,6 @@ func scanRecordingClient(scheme *runtime.Scheme, listed *[]string, writes *int, }).Build() } -func listedKind(listed []string, kind string) bool { - for _, l := range listed { - if strings.Contains(l, kind) { - return true - } - } - return false -} - // The comparison guarding that write has to be semantic rather than structural, or a // LimitRange written in a different but equivalent notation would re-apply on every // reconcile — invisible in a test that only looked at the resulting object, since every one diff --git a/internal/telemetry/collector.go b/internal/telemetry/collector.go index f4bb8dd88..a948c7495 100644 --- a/internal/telemetry/collector.go +++ b/internal/telemetry/collector.go @@ -118,11 +118,11 @@ func (c *Collector) collect(ctx context.Context) { // Write application count metrics for kind, count := range appKindCounts { - metrics.WriteString(fmt.Sprintf( + fmt.Fprintf(&metrics, "cozy_application_count{kind=\"%s\"} %d\n", kind, count, - )) + ) } // Send metrics only if there's something to send diff --git a/internal/telemetry/operator_collector.go b/internal/telemetry/operator_collector.go index 8b4ec3028..2c0e1b9bb 100644 --- a/internal/telemetry/operator_collector.go +++ b/internal/telemetry/operator_collector.go @@ -133,11 +133,11 @@ func (c *OperatorCollector) collect(ctx context.Context) { var metrics strings.Builder // Add cluster info metric - metrics.WriteString(fmt.Sprintf( + fmt.Fprintf(&metrics, "cozy_cluster_info{cozystack_version=\"%s\",kubernetes_version=\"%s\"} 1\n", version.Version, k8sVersion.GitVersion, - )) + ) // Collect node metrics grouped by OS and kernel nodeOSCount := make(map[string]map[string]int) // os -> kernel -> count @@ -153,12 +153,12 @@ func (c *OperatorCollector) collect(ctx context.Context) { for osKey, kernels := range nodeOSCount { for kernel, count := range kernels { - metrics.WriteString(fmt.Sprintf( + fmt.Fprintf(&metrics, "cozy_nodes_count{os=\"%s\",kernel=\"%s\"} %d\n", osKey, kernel, count, - )) + ) } } @@ -174,11 +174,11 @@ func (c *OperatorCollector) collect(ctx context.Context) { } for resourceName, total := range capacityTotals { - metrics.WriteString(fmt.Sprintf( + fmt.Fprintf(&metrics, "cozy_cluster_capacity{resource=\"%s\"} %d\n", resourceName, total, - )) + ) } // Collect LoadBalancer services metrics @@ -192,7 +192,7 @@ func (c *OperatorCollector) collect(ctx context.Context) { lbCount++ } } - metrics.WriteString(fmt.Sprintf("cozy_loadbalancers_count %d\n", lbCount)) + fmt.Fprintf(&metrics, "cozy_loadbalancers_count %d\n", lbCount) } // Collect PV metrics grouped by driver and size @@ -224,12 +224,12 @@ func (c *OperatorCollector) collect(ctx context.Context) { for size, drivers := range pvMetrics { for driver, count := range drivers { - metrics.WriteString(fmt.Sprintf( + fmt.Fprintf(&metrics, "cozy_pvs_count{driver=\"%s\",size=\"%s\"} %d\n", driver, size, count, - )) + ) } } } @@ -244,11 +244,11 @@ func (c *OperatorCollector) collect(ctx context.Context) { if variant == "" { variant = "default" } - metrics.WriteString(fmt.Sprintf( + fmt.Fprintf(&metrics, "cozy_package_info{name=\"%s\",variant=\"%s\"} 1\n", pkg.Name, variant, - )) + ) } } diff --git a/pkg/apiserver/apiserver.go b/pkg/apiserver/apiserver.go index c3555e46c..06f024bc6 100644 --- a/pkg/apiserver/apiserver.go +++ b/pkg/apiserver/apiserver.go @@ -78,7 +78,7 @@ func init() { // Register HelmRelease types. if err := helmv2.AddToScheme(mgrScheme); err != nil { - panic(fmt.Errorf("Failed to add HelmRelease types to scheme: %w", err)) + panic(fmt.Errorf("failed to add HelmRelease types to scheme: %w", err)) } // Register the in-tree CiliumNetworkPolicy mirror backing SecurityGroup. @@ -87,10 +87,10 @@ func init() { } if err := corev1.AddToScheme(mgrScheme); err != nil { - panic(fmt.Errorf("Failed to add core types to scheme: %w", err)) + panic(fmt.Errorf("failed to add core types to scheme: %w", err)) } if err := rbacv1.AddToScheme(mgrScheme); err != nil { - panic(fmt.Errorf("Failed to add RBAC types to scheme: %w", err)) + panic(fmt.Errorf("failed to add RBAC types to scheme: %w", err)) } // Register Cozystack types for WorkloadMonitor queries. diff --git a/pkg/cmd/server/openapi.go b/pkg/cmd/server/openapi.go index 90af89cd0..d0cbe3289 100644 --- a/pkg/cmd/server/openapi.go +++ b/pkg/cmd/server/openapi.go @@ -319,7 +319,7 @@ func BuildPostProcessV3(kindSchemas map[string]string) func(*spec3.OpenAPI) (*sp base, ok1 := doc.Components.Schemas[baseRef] list, ok2 := doc.Components.Schemas[baseListRef] stat, ok3 := doc.Components.Schemas[baseStatusRef] - if !(ok1 && ok2 && ok3) { + if !ok1 || !ok2 || !ok3 { return doc, nil // not the apps GV — nothing to patch } @@ -463,7 +463,7 @@ func BuildPostProcessV2(kindSchemas map[string]string) func(*spec.Swagger) (*spe base, ok1 := defs[baseRef] list, ok2 := defs[baseListRef] stat, ok3 := defs[baseStatusRef] - if !(ok1 && ok2 && ok3) { + if !ok1 || !ok2 || !ok3 { return sw, nil // not the apps GV — nothing to patch } diff --git a/pkg/ovnstatus/ovnstatus.go b/pkg/ovnstatus/ovnstatus.go index e6d682e51..b50c3f7ed 100644 --- a/pkg/ovnstatus/ovnstatus.go +++ b/pkg/ovnstatus/ovnstatus.go @@ -277,33 +277,6 @@ func defaultRunner(ctx context.Context, bin string, args ...string) (string, err return out, nil } -/************** Helpers **************/ - -func parseClusterStatusJSON(out string) (ClusterStatus, bool) { - var cs ClusterStatus - if json.Unmarshal([]byte(out), &cs) == nil && len(cs.Servers) > 0 { - return cs, true - } - var wrap struct { - Data ClusterStatus `json:"data"` - } - if json.Unmarshal([]byte(out), &wrap) == nil && len(wrap.Data.Servers) > 0 { - return wrap.Data, true - } - return ClusterStatus{}, false -} - -func portOf(db DB) string { - switch db { - case DBNorthbound: - return "6641" - case DBSouthbound: - return "6642" - default: - return "0" - } -} - /************** Minimal text fallback for "Servers" **************/ // Accepts variants like: diff --git a/pkg/ovnstatus/ovnstatus_test.go b/pkg/ovnstatus/ovnstatus_test.go index 8dde66b3a..8c44f5413 100644 --- a/pkg/ovnstatus/ovnstatus_test.go +++ b/pkg/ovnstatus/ovnstatus_test.go @@ -28,9 +28,7 @@ var expectedServersBlock = `` + func TestExtractServersBlock(t *testing.T) { if actual := extractServersBlock(testStdout); actual != expectedServersBlock { - fmt.Println([]byte(actual)) - fmt.Println([]byte(expectedServersBlock)) - t.Errorf("error extracting servers block from following string:\n%s\nexpected:\n%s\ngot:\n%s\n", testStdout, expectedServersBlock, actual) + t.Errorf("error extracting servers block from following string:\n%s\nexpected:\n%q\ngot:\n%q\n", testStdout, expectedServersBlock, actual) } } diff --git a/pkg/reservation/evaluator.go b/pkg/reservation/evaluator.go index 6927b0bf2..40d5c2ced 100644 --- a/pkg/reservation/evaluator.go +++ b/pkg/reservation/evaluator.go @@ -284,9 +284,9 @@ func mulQuantity(q, n ref.Val) ref.Val { if !ok { return types.MaybeNoSuchOverloadErr(n) } - out := qq.Quantity.DeepCopy() + out := qq.DeepCopy() if !out.Mul(int64(nn)) { - return types.NewErr("quantity %s * %d overflows", qq.Quantity.String(), int64(nn)) + return types.NewErr("quantity %s * %d overflows", qq.String(), int64(nn)) } return apiservercel.Quantity{Quantity: &out} } @@ -294,7 +294,7 @@ func mulQuantity(q, n ref.Val) ref.Val { func toQuantity(v ref.Val) (resource.Quantity, error) { switch t := v.(type) { case apiservercel.Quantity: - return t.Quantity.DeepCopy(), nil + return t.DeepCopy(), nil case types.Int: return *resource.NewQuantity(int64(t), resource.DecimalSI), nil case types.Uint: From 238e85577e63211c8124603ae87d4cab01931098 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:28:51 +0300 Subject: [PATCH 481/518] refactor(lineage): resolve a HelmRelease's application in a helper The walk used a loop that always ended in its first pass to get "break to the fallback" control flow. A function returning whether it resolved the application says the same without the loop. A test pins that each failure to resolve still falls back to the HelmRelease's own label and that a resolved one ends the walk. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- pkg/lineage/helmrelease_test.go | 70 +++++++++++++++++++++++++++++++++ pkg/lineage/lineage.go | 59 ++++++++++++++------------- 2 files changed, 99 insertions(+), 30 deletions(-) create mode 100644 pkg/lineage/helmrelease_test.go diff --git a/pkg/lineage/helmrelease_test.go b/pkg/lineage/helmrelease_test.go new file mode 100644 index 000000000..c8ef2676e --- /dev/null +++ b/pkg/lineage/helmrelease_test.go @@ -0,0 +1,70 @@ +package lineage + +import ( + "context" + "fmt" + "strings" + "testing" + + helmv2 "github.com/fluxcd/helm-controller/api/v2" + "k8s.io/apimachinery/pkg/api/meta" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + dynamicfake "k8s.io/client-go/dynamic/fake" +) + +// namedAppMapper maps a HelmRelease to the Kubernetes application it names; +// a HelmRelease missing from the map fails to map. +type namedAppMapper map[string]string + +func (m namedAppMapper) Map(hr *helmv2.HelmRelease) (string, string, string, error) { + app, ok := m[hr.Name] + if !ok { + return "", "", "", fmt.Errorf("no application for HelmRelease %s", hr.Name) + } + return "apps.cozystack.io/v1alpha1", "Kubernetes", strings.TrimSuffix(hr.Name, app), nil +} + +// A HelmRelease that cannot be resolved to its application hands the walk to +// the HelmRelease its label names; one that resolves ends the walk there. +func TestWalkHelmReleaseFallsBackToItsLabel(t *testing.T) { + parent := map[string]string{HRLabel: "kubernetes-b"} + unconvertible := capkObject(HRAPIVersion, HRKind, "kubernetes-a", parent, nil) + unconvertible.Object["spec"] = "not an object" + for _, tc := range []struct { + name string + hr *unstructured.Unstructured + mapped namedAppMapper + want []string + }{ + {name: "mapped", hr: capkObject(HRAPIVersion, HRKind, "kubernetes-a", parent, nil), mapped: namedAppMapper{"kubernetes-a": "a", "kubernetes-b": "b"}, want: []string{"a"}}, + {name: "unmappable", hr: capkObject(HRAPIVersion, HRKind, "kubernetes-a", parent, nil), mapped: namedAppMapper{"kubernetes-b": "b"}, want: []string{"b"}}, + {name: "application missing", hr: capkObject(HRAPIVersion, HRKind, "kubernetes-a", parent, nil), mapped: namedAppMapper{"kubernetes-a": "absent", "kubernetes-b": "b"}, want: []string{"b"}}, + {name: "unconvertible", hr: unconvertible, mapped: namedAppMapper{"kubernetes-a": "a", "kubernetes-b": "b"}, want: []string{"b"}}, + {name: "unmappable without a label", hr: capkObject(HRAPIVersion, HRKind, "kubernetes-a", nil, nil), mapped: namedAppMapper{"kubernetes-b": "b"}, want: nil}, + } { + t.Run(tc.name, func(t *testing.T) { + objs := []runtime.Object{ + capkObject("apps.cozystack.io/v1alpha1", "Kubernetes", "a", nil, nil), + capkObject("apps.cozystack.io/v1alpha1", "Kubernetes", "b", nil, nil), + capkObject(HRAPIVersion, HRKind, "kubernetes-b", nil, nil), + } + mapper := meta.NewDefaultRESTMapper(nil) + for _, o := range objs { + mapper.Add(o.GetObjectKind().GroupVersionKind(), meta.RESTScopeNamespace) + } + dyn := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme(), objs...) + var got []string + for _, id := range WalkOwnershipGraph(context.Background(), dyn, mapper, tc.mapped, tc.hr) { + got = append(got, id.Kind+"/"+id.Name) + } + var want []string + for _, name := range tc.want { + want = append(want, "Kubernetes/"+name) + } + if strings.Join(got, ",") != strings.Join(want, ",") { + t.Fatalf("WalkOwnershipGraph(%s) = %v, want %v", tc.name, got, want) + } + }) + } +} diff --git a/pkg/lineage/lineage.go b/pkg/lineage/lineage.go index 1546df647..8d8614b68 100644 --- a/pkg/lineage/lineage.go +++ b/pkg/lineage/lineage.go @@ -113,36 +113,8 @@ func WalkOwnershipGraph( return } - // I want "if err1 != nil go to next block, if err2 != nil, go to next block, etc semantics", - // like an early return from a function, but if all checks succeed, I don't want to do the rest - // of the function, so it's a `for { if err { break } if othererr { break } if allgood { return } - for { - if obj.GetAPIVersion() != HRAPIVersion || obj.GetKind() != HRKind { - break - } - hr := helmReleaseFromUnstructured(obj) - if hr == nil { - break - } - a, k, p, err := appMapper.Map(hr) - if err != nil { - l.Error(err, "failed to map HelmRelease to app") - break - } - ownerObj, err := getUnstructuredObject(ctx, client, mapper, state.cache, a, k, obj.GetNamespace(), strings.TrimPrefix(obj.GetName(), p)) - if err != nil { - l.Error(err, "couldn't get unstructured object", "APIVersion", a, "Kind", k, "Name", strings.TrimPrefix(obj.GetName(), p)) - break - } - // successfully mapped a HelmRelease to a custom app, no need to continue - out = append(out, - ObjectID{ - APIVersion: ownerObj.GetAPIVersion(), - Kind: ownerObj.GetKind(), - Namespace: ownerObj.GetNamespace(), - Name: ownerObj.GetName(), - }, - ) + if app, ok := helmReleaseApp(ctx, client, mapper, appMapper, state.cache, obj); ok { + out = append(out, app) return } @@ -160,6 +132,33 @@ func WalkOwnershipGraph( return } +// helmReleaseApp resolves a HelmRelease to the application it deploys. When +// it cannot, the walk goes on through the HelmRelease's own label instead. +func helmReleaseApp(ctx context.Context, client dynamic.Interface, mapper meta.RESTMapper, appMapper AppMapper, cache *ObjectCache, obj *unstructured.Unstructured) (ObjectID, bool) { + hr := helmReleaseFromUnstructured(obj) + if hr == nil { + return ObjectID{}, false + } + l := log.FromContext(ctx) + a, k, p, err := appMapper.Map(hr) + if err != nil { + l.Error(err, "failed to map HelmRelease to app") + return ObjectID{}, false + } + name := strings.TrimPrefix(obj.GetName(), p) + app, err := getUnstructuredObject(ctx, client, mapper, cache, a, k, obj.GetNamespace(), name) + if err != nil { + l.Error(err, "couldn't get unstructured object", "APIVersion", a, "Kind", k, "Name", name) + return ObjectID{}, false + } + return ObjectID{ + APIVersion: app.GetAPIVersion(), + Kind: app.GetKind(), + Namespace: app.GetNamespace(), + Name: app.GetName(), + }, true +} + // kubevirtMachineName returns the KubevirtMachine a CAPK-created // VirtualMachine belongs to. It only follows a KubevirtMachine in the VM's own // namespace: CAPK can also run against an external infrastructure cluster, From 04fa87ccd1edf6dfec048717a3bd1ee7d3ab1a1d Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:30:02 +0300 Subject: [PATCH 482/518] style(backups): clear staticcheck findings in overlapping controllers Drop a field-manager constant nothing uses in the MariaDB driver, switch on the MongoDB operator's state as a tagged switch, and reword the error strings of the MongoDB driver and the workload monitor so they do not start with a capital. The errors keep their meaning and nothing matches their text. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .../backupcontroller/mariadbstrategy_controller.go | 2 -- .../backupcontroller/psmdbstrategy_controller.go | 14 +++++++------- internal/controller/workloadmonitor_controller.go | 4 ++-- 3 files changed, 9 insertions(+), 11 deletions(-) diff --git a/internal/backupcontroller/mariadbstrategy_controller.go b/internal/backupcontroller/mariadbstrategy_controller.go index 9fd6780db..452fff071 100644 --- a/internal/backupcontroller/mariadbstrategy_controller.go +++ b/internal/backupcontroller/mariadbstrategy_controller.go @@ -30,8 +30,6 @@ import ( // --------------------------------------------------------------------------- const ( - mariadbFieldManager = "cozystack-mariadb-backup-driver" - // mariadb-operator's MariaDB ApplicationDefinition (release.prefix). The // cozystack mariadb-rd renders the HelmRelease with releaseName = // "mariadb-" + appName (see packages/system/mariadb-rd/cozyrds/mariadb.yaml, diff --git a/internal/backupcontroller/psmdbstrategy_controller.go b/internal/backupcontroller/psmdbstrategy_controller.go index 9caa218a7..ccf3e7735 100644 --- a/internal/backupcontroller/psmdbstrategy_controller.go +++ b/internal/backupcontroller/psmdbstrategy_controller.go @@ -528,8 +528,8 @@ func (r *BackupJobReconciler) reconcileMongoDB(ctx context.Context, j *backupsv1 } } - switch state := mdbBackup.Status.State; { - case state == psmdbtypes.StateReady: + switch state := mdbBackup.Status.State; state { + case psmdbtypes.StateReady: // Backup completed successfully. Materialise the Cozystack artifact and // finalise the BackupJob. Idempotent: reuse the existing artifact if a // previous reconcile created it and then raced on the status update. @@ -570,7 +570,7 @@ func (r *BackupJobReconciler) reconcileMongoDB(ctx context.Context, j *backupsv1 } return ctrl.Result{}, nil - case state == psmdbtypes.StateError || state == psmdbtypes.StateRejected: + case psmdbtypes.StateError, psmdbtypes.StateRejected: message := mdbBackup.Status.Error // The residual half of the injection race: on the useSystemBucket flow the // operator can latch state=error because it resolved the injected storage @@ -1740,8 +1740,8 @@ func (r *RestoreJobReconciler) reconcileMongoDBRestore(ctx context.Context, rest return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("failed to ensure psmdb.percona.com/PerconaServerMongoDBRestore: %v", err)) } - switch state := mdbRestore.Status.State; { - case state == psmdbtypes.StateReady: + switch state := mdbRestore.Status.State; state { + case psmdbtypes.StateReady: now := metav1.Now() restoreJob.Status.CompletedAt = &now restoreJob.Status.Phase = backupsv1alpha1.RestoreJobPhaseSucceeded @@ -1756,7 +1756,7 @@ func (r *RestoreJobReconciler) reconcileMongoDBRestore(ctx context.Context, rest } return ctrl.Result{}, nil - case state == psmdbtypes.StateError || state == psmdbtypes.StateRejected: + case psmdbtypes.StateError, psmdbtypes.StateRejected: message := mdbRestore.Status.Error if message == "" { message = fmt.Sprintf("psmdb.percona.com/PerconaServerMongoDBRestore %s/%s reported state=%s", mdbRestore.Namespace, mdbRestore.Name, state) @@ -1855,7 +1855,7 @@ func (r *RestoreJobReconciler) resolveMongoDBBackupSource(ctx context.Context, b } if destination == "" { return nil, fmt.Errorf( - "Backup has no restorable destination (the operator PerconaServerMongoDBBackup was reaped and no snapshot destination is persisted); re-run the BackupJob") + "no restorable destination for the Backup (the operator PerconaServerMongoDBBackup was reaped and no snapshot destination is persisted); re-run the BackupJob") } src := &psmdbtypes.BackupSource{ Type: psmdbtypes.BackupTypeLogical, diff --git a/internal/controller/workloadmonitor_controller.go b/internal/controller/workloadmonitor_controller.go index 984ab7b21..217fd0bbf 100644 --- a/internal/controller/workloadmonitor_controller.go +++ b/internal/controller/workloadmonitor_controller.go @@ -528,7 +528,7 @@ func (r *WorkloadMonitorReconciler) queryAllBucketMetrics(ctx context.Context, p defer resp.Body.Close() if resp.StatusCode != http.StatusOK { - return nil, fmt.Errorf("Prometheus returned status %d", resp.StatusCode) + return nil, fmt.Errorf("unexpected status %d from Prometheus", resp.StatusCode) } body, err := io.ReadAll(io.LimitReader(resp.Body, 4<<20)) @@ -549,7 +549,7 @@ func (r *WorkloadMonitorReconciler) queryAllBucketMetrics(ctx context.Context, p return nil, fmt.Errorf("parsing Prometheus response: %w", err) } if promResp.Status != "success" { - return nil, fmt.Errorf("Prometheus query finished with status %q", promResp.Status) + return nil, fmt.Errorf("the Prometheus query finished with status %q", promResp.Status) } for _, r := range promResp.Data.Result { From 2cda7d692e1bb2343ca6b71fc2a1335eefc2aade Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:30:02 +0300 Subject: [PATCH 483/518] test(backport): drop an unused fixture helper Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- cmd/backport-batch/fixture_test.go | 9 --------- 1 file changed, 9 deletions(-) diff --git a/cmd/backport-batch/fixture_test.go b/cmd/backport-batch/fixture_test.go index ac1307c46..08aa5f2ac 100644 --- a/cmd/backport-batch/fixture_test.go +++ b/cmd/backport-batch/fixture_test.go @@ -522,15 +522,6 @@ func (f *fixture) batchPR(n int, base, branch, body string) pullRequest { return pr } -// botBatchPR is batchPR opened by the bot the scheduled batch runs as. -func (f *fixture) botBatchPR(gh *fakeGH, n int, base, branch, body string) pullRequest { - f.t.Helper() - gh.bot = defaultBatchBot - pr := f.batchPR(n, base, branch, body) - pr.User.Login, pr.User.Type, pr.AuthorAssociation = defaultBatchBot, "Bot", "CONTRIBUTOR" - return pr -} - // mergeBatch merges the open batch into release-1.1 with a merge commit, as // GitHub does, and deletes its branch, as the repository's // delete_branch_on_merge does. The pull request stays, merged, with its body From d308e9e5e59aea5be1a41952fef9f8c3035845c5 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Fri, 13 Nov 2026 09:43:05 +0100 Subject: [PATCH 484/518] fix(redis): refuse sub-byte maxmemory and apply the policy first A quantity such as 512m is 0.512 bytes; the integer cast truncated it, or -512m, to 0, which Redis reads as no limit, so the bound a tenant set could vanish without an error. The operator also sends the directives in list order, and lowering maxmemory below the dataset evicts at once, so the policy has to be set before the bound. Turning snapshots off does not remove an RDB already on the volume, which Redis still loads on restart; the parameter no longer says the data is lost. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- api/apps/v1alpha1/redis/types.go | 2 +- packages/apps/redis/README.md | 2 +- .../apps/redis/templates/redisfailover.yaml | 18 ++++++++--- .../apps/redis/tests/cache_config_test.yaml | 32 +++++++++++++++++++ packages/apps/redis/values.schema.json | 2 +- packages/apps/redis/values.yaml | 2 +- packages/system/redis-rd/cozyrds/redis.yaml | 2 +- 7 files changed, 50 insertions(+), 10 deletions(-) diff --git a/api/apps/v1alpha1/redis/types.go b/api/apps/v1alpha1/redis/types.go index 03ff0175d..41cf03c14 100644 --- a/api/apps/v1alpha1/redis/types.go +++ b/api/apps/v1alpha1/redis/types.go @@ -56,7 +56,7 @@ type ConfigSpec struct { // Redis `maxmemory-policy`: what Redis does once `maxmemory` is reached. `noeviction` refuses writes; `allkeys-lru` evicts the least recently used keys, which suits an instance used purely as a cache. Has no effect while `maxmemory` is unset. // +kubebuilder:default:="noeviction" MaxmemoryPolicy MaxmemoryPolicy `json:"maxmemoryPolicy"` - // Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save ""` is rendered and the volume is kept, so the data is lost on restart but replicas still have room for the full-sync RDB they write to disk. Always off when `size` is empty. + // Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save ""` is rendered and the volume is kept, so no further scheduled snapshot is written while replicas still have room for the full-sync RDB they write to disk. An RDB file already on the volume is not removed, and Redis loads it on restart, so turning snapshots off does not by itself empty the dataset. Always off when `size` is empty. // +kubebuilder:default:=true Snapshots bool `json:"snapshots"` // Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does. diff --git a/packages/apps/redis/README.md b/packages/apps/redis/README.md index 4044ce951..be03bbea3 100644 --- a/packages/apps/redis/README.md +++ b/packages/apps/redis/README.md @@ -46,7 +46,7 @@ Service utilizes the freshworks-oss Redis Operator (a maintained fork of the arc | `minReplicasMaxLag` | Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it. | `*int` | `null` | | `maxmemory` | Redis `maxmemory`: the dataset size at which Redis applies `maxmemoryPolicy`. Unset by default, which renders `maxmemory 0` (no limit, the Redis default), so a value set by hand with `CONFIG SET` is reset on the next reconcile. Must be at most 90% of the memory request of each Redis pod (from `resources` or `resourcesPreset`): the operator silently drops a larger value, so the chart refuses it. Leave headroom below that for replication and snapshot buffers. | `quantity` | `""` | | `maxmemoryPolicy` | Redis `maxmemory-policy`: what Redis does once `maxmemory` is reached. `noeviction` refuses writes; `allkeys-lru` evicts the least recently used keys, which suits an instance used purely as a cache. Has no effect while `maxmemory` is unset. | `string` | `noeviction` | -| `snapshots` | Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save ""` is rendered and the volume is kept, so the data is lost on restart but replicas still have room for the full-sync RDB they write to disk. Always off when `size` is empty. | `bool` | `true` | +| `snapshots` | Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save ""` is rendered and the volume is kept, so no further scheduled snapshot is written while replicas still have room for the full-sync RDB they write to disk. An RDB file already on the volume is not removed, and Redis loads it on restart, so turning snapshots off does not by itself empty the dataset. Always off when `size` is empty. | `bool` | `true` | ### Sentinel parameters diff --git a/packages/apps/redis/templates/redisfailover.yaml b/packages/apps/redis/templates/redisfailover.yaml index ba36e3f6c..330a2d517 100644 --- a/packages/apps/redis/templates/redisfailover.yaml +++ b/packages/apps/redis/templates/redisfailover.yaml @@ -128,21 +128,29 @@ spec: {{- end }} {{- $maxmemory := 0 }} {{- if not (kindIs "invalid" .Values.maxmemory) }} - {{- $maxmemory = include "cozy-lib.resources.toFloat" (.Values.maxmemory | toString) | float64 | int64 }} + {{- $maxmemoryBytes := include "cozy-lib.resources.toFloat" (.Values.maxmemory | toString) | float64 }} + {{- /* Checked before the int64 cast, which would truncate a sub-byte quantity such + as 512m (0.512 bytes) or -512m to 0, and Redis reads 0 as no limit. */}} + {{- if lt $maxmemoryBytes 0.0 }} + {{- fail (printf "maxmemory (%v) must not be negative" .Values.maxmemory) }} + {{- end }} + {{- if ne (floor $maxmemoryBytes) $maxmemoryBytes }} + {{- fail (printf "maxmemory (%v) must be a whole number of bytes, e.g. 512Mi rather than 512m (millibytes)" .Values.maxmemory) }} + {{- end }} + {{- $maxmemory = $maxmemoryBytes | int64 }} {{- $redisResources := include "cozy-lib.resources.defaultingSanitize" (list .Values.resourcesPreset .Values.resources $) | fromYaml }} {{- $memoryRequest := include "cozy-lib.resources.toFloat" (dig "requests" "memory" "0" $redisResources | toString) | float64 }} {{- /* The operator compares maxmemory with 90% of the pod's memory request (reservedPodMemoryPercent) and silently skips the line when it is larger. */}} {{- $allowed := mulf $memoryRequest 0.9 | int64 }} - {{- if lt $maxmemory 0 }} - {{- fail (printf "maxmemory (%v) must not be negative" .Values.maxmemory) }} - {{- end }} {{- if gt $maxmemory $allowed }} {{- fail (printf "maxmemory (%v, %d bytes) must be at most 90%% of the Redis memory request (%d bytes): the operator ignores a larger value" .Values.maxmemory $maxmemory $allowed) }} {{- end }} {{- end }} - - maxmemory {{ $maxmemory }} + {{- /* The operator sends these in list order, and lowering maxmemory below the + dataset evicts at once, so the policy has to be in place first. */}} - maxmemory-policy {{ .Values.maxmemoryPolicy | default "noeviction" }} + - maxmemory {{ $maxmemory }} {{- /* A master with `replicas` pods has replicas-1 replicas, so a requirement of `replicas` or more can never be met and the master would refuse every write. */}} {{- if and (not (kindIs "invalid" .Values.minReplicasToWrite)) (ge (.Values.minReplicasToWrite | int) (.Values.replicas | int)) }} diff --git a/packages/apps/redis/tests/cache_config_test.yaml b/packages/apps/redis/tests/cache_config_test.yaml index 557b5d3d4..1b618bcac 100644 --- a/packages/apps/redis/tests/cache_config_test.yaml +++ b/packages/apps/redis/tests/cache_config_test.yaml @@ -116,3 +116,35 @@ tests: asserts: - failedTemplate: errorPattern: must not be negative + + # A sub-byte quantity would truncate to maxmemory 0, which Redis reads as no + # limit, so the bound the tenant asked for would silently disappear. + - it: "refuses a maxmemory that is not a whole number of bytes" + set: + maxmemory: 512m + asserts: + - failedTemplate: + errorPattern: must be a whole number of bytes + + - it: "refuses a negative sub-byte maxmemory" + set: + maxmemory: -512m + asserts: + - failedTemplate: + errorPattern: must not be negative + + - it: "applies the eviction policy before the memory bound" + set: + resourcesPreset: t1.large + maxmemory: 1Gi + maxmemoryPolicy: allkeys-lru + documentSelector: + path: kind + value: RedisFailover + asserts: + - equal: + path: spec.redis.customConfig[3] + value: maxmemory-policy allkeys-lru + - equal: + path: spec.redis.customConfig[4] + value: maxmemory 1073741824 diff --git a/packages/apps/redis/values.schema.json b/packages/apps/redis/values.schema.json index b3b143320..183fb6a53 100644 --- a/packages/apps/redis/values.schema.json +++ b/packages/apps/redis/values.schema.json @@ -200,7 +200,7 @@ ] }, "snapshots": { - "description": "Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save \"\"` is rendered and the volume is kept, so the data is lost on restart but replicas still have room for the full-sync RDB they write to disk. Always off when `size` is empty.", + "description": "Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save \"\"` is rendered and the volume is kept, so no further scheduled snapshot is written while replicas still have room for the full-sync RDB they write to disk. An RDB file already on the volume is not removed, and Redis loads it on restart, so turning snapshots off does not by itself empty the dataset. Always off when `size` is empty.", "type": "boolean", "default": true }, diff --git a/packages/apps/redis/values.yaml b/packages/apps/redis/values.yaml index 0f8380ae7..ed14c4f46 100644 --- a/packages/apps/redis/values.yaml +++ b/packages/apps/redis/values.yaml @@ -134,7 +134,7 @@ authEnabled: true ## @param {MaxmemoryPolicy} maxmemoryPolicy="noeviction" - Redis `maxmemory-policy`: what Redis does once `maxmemory` is reached. `noeviction` refuses writes; `allkeys-lru` evicts the least recently used keys, which suits an instance used purely as a cache. Has no effect while `maxmemory` is unset. maxmemoryPolicy: "noeviction" -## @param {bool} snapshots - Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save ""` is rendered and the volume is kept, so the data is lost on restart but replicas still have room for the full-sync RDB they write to disk. Always off when `size` is empty. +## @param {bool} snapshots - Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save ""` is rendered and the volume is kept, so no further scheduled snapshot is written while replicas still have room for the full-sync RDB they write to disk. An RDB file already on the volume is not removed, and Redis loads it on restart, so turning snapshots off does not by itself empty the dataset. Always off when `size` is empty. snapshots: true ## diff --git a/packages/system/redis-rd/cozyrds/redis.yaml b/packages/system/redis-rd/cozyrds/redis.yaml index 02046b97e..8c3045b0f 100644 --- a/packages/system/redis-rd/cozyrds/redis.yaml +++ b/packages/system/redis-rd/cozyrds/redis.yaml @@ -8,7 +8,7 @@ spec: plural: redises singular: redis openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.nano","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster. Can be enabled only when the instance is created, together with `tls.enabled: true`; turning it on for an existing instance, or changing `tls.enabled` while it is on, is refused at admission.","type":"boolean","default":false},"version":{"description":"Redis major version to deploy. The Redis and Sentinel pods run the same image.","type":"string","default":"v8","enum":["v8","v7"]},"tls":{"description":"TLS configuration. TLS is opt-in and is not inferred from `external`.","type":"object","default":{},"properties":{"authClients":{"description":"Maps to the Redis `tls-auth-clients` directive. Defaults to `no` — the server certificate is presented but client certificates are not validated. `yes` requires a client certificate signed by this release's CA, which the platform does not issue. Changing it restarts the Redis and Sentinel pods, because the directive is read from the config file once at startup.","type":"string","enum":["no","optional","yes"]},"enabled":{"description":"Enable TLS for Redis and Sentinel connections. Disabled unless set. Enabling it moves Redis and Sentinel to a TLS-only listener, so existing plaintext clients must be migrated at the same time. Encryption is provided by the redis-operator fork that mounts the certificate Secret into both Redis and Sentinel pods at `/tls`.","type":"boolean"}}},"authEnabled":{"description":"Enable password generation.","type":"boolean","default":true},"minReplicasToWrite":{"description":"Redis `min-replicas-to-write`: the master refuses writes unless at least this many replicas are connected and within `minReplicasMaxLag`. Unset by default, which leaves the Redis default (disabled). With `replicas: 2` and a value of 1 the master rejects writes while its only replica is down, trading availability for a bound on the writes lost if an isolated master is failed over. Replication stays asynchronous, so this bounds the loss, it does not remove it. Must be 0 or more and below `replicas` (a master with `replicas` pods has `replicas - 1` replicas), otherwise every write is refused; the chart refuses such a value.","type":"integer","minimum":0},"minReplicasMaxLag":{"description":"Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it.","type":"integer","minimum":0},"maxmemory":{"description":"Redis `maxmemory`: the dataset size at which Redis applies `maxmemoryPolicy`. Unset by default, which renders `maxmemory 0` (no limit, the Redis default), so a value set by hand with `CONFIG SET` is reset on the next reconcile. Must be at most 90% of the memory request of each Redis pod (from `resources` or `resourcesPreset`): the operator silently drops a larger value, so the chart refuses it. Leave headroom below that for replication and snapshot buffers.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"maxmemoryPolicy":{"description":"Redis `maxmemory-policy`: what Redis does once `maxmemory` is reached. `noeviction` refuses writes; `allkeys-lru` evicts the least recently used keys, which suits an instance used purely as a cache. Has no effect while `maxmemory` is unset.","type":"string","default":"noeviction","enum":["noeviction","allkeys-lru","allkeys-lfu","allkeys-random","volatile-lru","volatile-lfu","volatile-random","volatile-ttl"]},"snapshots":{"description":"Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save \"\"` is rendered and the volume is kept, so the data is lost on restart but replicas still have room for the full-sync RDB they write to disk. Always off when `size` is empty.","type":"boolean","default":true},"sentinel":{"description":"Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.","type":"object","default":{},"properties":{"resources":{"description":"Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`.","type":"object","properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}} + {"title":"Chart Values","type":"object","properties":{"replicas":{"description":"Number of Redis replicas. Three Sentinels watch them, or a single one when `replicas` is 1, since a lone pod has no replica to fail over to.","type":"integer","default":2},"resources":{"description":"Explicit CPU and memory configuration for each Redis replica. When omitted, the preset defined in `resourcesPreset` is applied.","type":"object","default":{},"properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Default sizing preset used when `resources` is omitted.","type":"string","default":"t1.nano","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]},"size":{"description":"Persistent Volume Claim size available for application data.","default":"1Gi","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"storageClass":{"description":"StorageClass used to store the data.","type":"string","default":"","x-kubernetes-validations":[{"rule":"self == oldSelf","message":"storageClass is immutable"}],"x-cozystack-options":{"source":"storageclass"}},"external":{"description":"Enable external access from outside the cluster. Can be enabled only when the instance is created, together with `tls.enabled: true`; turning it on for an existing instance, or changing `tls.enabled` while it is on, is refused at admission.","type":"boolean","default":false},"version":{"description":"Redis major version to deploy. The Redis and Sentinel pods run the same image.","type":"string","default":"v8","enum":["v8","v7"]},"tls":{"description":"TLS configuration. TLS is opt-in and is not inferred from `external`.","type":"object","default":{},"properties":{"authClients":{"description":"Maps to the Redis `tls-auth-clients` directive. Defaults to `no` — the server certificate is presented but client certificates are not validated. `yes` requires a client certificate signed by this release's CA, which the platform does not issue. Changing it restarts the Redis and Sentinel pods, because the directive is read from the config file once at startup.","type":"string","enum":["no","optional","yes"]},"enabled":{"description":"Enable TLS for Redis and Sentinel connections. Disabled unless set. Enabling it moves Redis and Sentinel to a TLS-only listener, so existing plaintext clients must be migrated at the same time. Encryption is provided by the redis-operator fork that mounts the certificate Secret into both Redis and Sentinel pods at `/tls`.","type":"boolean"}}},"authEnabled":{"description":"Enable password generation.","type":"boolean","default":true},"minReplicasToWrite":{"description":"Redis `min-replicas-to-write`: the master refuses writes unless at least this many replicas are connected and within `minReplicasMaxLag`. Unset by default, which leaves the Redis default (disabled). With `replicas: 2` and a value of 1 the master rejects writes while its only replica is down, trading availability for a bound on the writes lost if an isolated master is failed over. Replication stays asynchronous, so this bounds the loss, it does not remove it. Must be 0 or more and below `replicas` (a master with `replicas` pods has `replicas - 1` replicas), otherwise every write is refused; the chart refuses such a value.","type":"integer","minimum":0},"minReplicasMaxLag":{"description":"Redis `min-replicas-max-lag`, in seconds: a replica whose last acknowledgement is older than this does not count towards `minReplicasToWrite`. Unset by default, which leaves the Redis default (10). Has no effect unless `minReplicasToWrite` is set. Must be at least 1 when `minReplicasToWrite` is set: Redis treats 0 as disabling the check, so the chart refuses it.","type":"integer","minimum":0},"maxmemory":{"description":"Redis `maxmemory`: the dataset size at which Redis applies `maxmemoryPolicy`. Unset by default, which renders `maxmemory 0` (no limit, the Redis default), so a value set by hand with `CONFIG SET` is reset on the next reconcile. Must be at most 90% of the memory request of each Redis pod (from `resources` or `resourcesPreset`): the operator silently drops a larger value, so the chart refuses it. Leave headroom below that for replication and snapshot buffers.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"maxmemoryPolicy":{"description":"Redis `maxmemory-policy`: what Redis does once `maxmemory` is reached. `noeviction` refuses writes; `allkeys-lru` evicts the least recently used keys, which suits an instance used purely as a cache. Has no effect while `maxmemory` is unset.","type":"string","default":"noeviction","enum":["noeviction","allkeys-lru","allkeys-lfu","allkeys-random","volatile-lru","volatile-lfu","volatile-random","volatile-ttl"]},"snapshots":{"description":"Write RDB snapshots to the persistent volume (the Redis `save` directive, with the Redis default schedule). Set to `false` for an instance used purely as a cache: `save \"\"` is rendered and the volume is kept, so no further scheduled snapshot is written while replicas still have room for the full-sync RDB they write to disk. An RDB file already on the volume is not removed, and Redis loads it on restart, so turning snapshots off does not by itself empty the dataset. Always off when `size` is empty.","type":"boolean","default":true},"sentinel":{"description":"Sentinel configuration. When `sentinel.resources` is empty and `sentinel.resourcesPreset` is unset, the Sentinel pods are sized like the Redis pods (`resources` and `resourcesPreset`). An empty `sentinel.resources` map counts as unset, as an empty top-level `resources` does.","type":"object","default":{},"properties":{"resources":{"description":"Explicit CPU and memory configuration for each Sentinel pod. Keys left out are filled from the preset in `sentinel.resourcesPreset`.","type":"object","properties":{"cpu":{"description":"CPU available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true},"memory":{"description":"Memory (RAM) available to each replica.","pattern":"^(\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\\+|-)?(([0-9]+(\\.[0-9]*)?)|(\\.[0-9]+))))?$","anyOf":[{"type":"integer"},{"type":"string"}],"x-kubernetes-int-or-string":true}}},"resourcesPreset":{"description":"Sizing preset for the Sentinel pods. Defaults to `resourcesPreset`.","type":"string","enum":["t1.nano","t1.micro","t1.small","t1.medium","t1.large","t1.xlarge","t1.2xlarge","t1.4xlarge","c1.nano","c1.micro","c1.small","c1.medium","c1.large","c1.xlarge","c1.2xlarge","c1.4xlarge","s1.nano","s1.micro","s1.small","s1.medium","s1.large","s1.xlarge","s1.2xlarge","s1.4xlarge","u1.nano","u1.micro","u1.small","u1.medium","u1.large","u1.xlarge","u1.2xlarge","u1.4xlarge","m1.nano","m1.micro","m1.small","m1.medium","m1.large","m1.xlarge","m1.2xlarge","m1.4xlarge","nano","micro","small","medium","large","xlarge","2xlarge"]}}}}} release: prefix: redis- labels: From c6711d6fe9fdeb4b83e7812823408b5d7c3075f8 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Fri, 13 Nov 2026 09:43:20 +0100 Subject: [PATCH 485/518] fix(linstor): prefer a schedulable snapshot node over topology hints The preferred nodes come from the requisite topology, which may leave out a snapshot node. Filtering on them first kept a cordoned node when the only schedulable snapshot node was outside that list, so the restored volume could still be unusable. Schedulability now decides before the topology hints, which only break ties. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../003-snapshot-restore-node-selection.diff | 34 ++++++++++++------- 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff b/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff index 0ad54da77..75351fe8e 100644 --- a/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff +++ b/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff @@ -11,7 +11,7 @@ index bd28e06..5bed73b 100644 if err != nil { log.Fatal(err) diff --git a/pkg/client/linstor.go b/pkg/client/linstor.go -index 5d71c37..e8e67ec 100644 +index 5d71c37..43129c9 100644 --- a/pkg/client/linstor.go +++ b/pkg/client/linstor.go @@ -46,6 +46,8 @@ import ( @@ -115,7 +115,7 @@ index 5d71c37..e8e67ec 100644 if selectedNode == "" { return fmt.Errorf("no node [%s] available for snapshot '%s'", strings.Join(snap.Nodes, ","), snap.Name) } -@@ -1893,6 +1921,39 @@ func (s *Linstor) relocateResources(ctx context.Context, volID, rgName string) e +@@ -1893,6 +1921,43 @@ func (s *Linstor) relocateResources(ctx context.Context, volID, rgName string) e return nil } @@ -128,17 +128,21 @@ index 5d71c37..e8e67ec 100644 +// Cordoned nodes are skipped as long as another node holds the snapshot: a pod can never be scheduled on +// them, so a volume restored there stays unusable until the node is uncordoned. +// -+// Only nodes that are also in preferredNodes are considered, unless none is. ++// Nodes in preferredNodes come first, but a schedulable node outside them wins over a cordoned one inside them. +func selectRestoreNode(available, preferredNodes []string, schedulable func(node string) bool) string { -+ candidates := slices.DeleteFunc(slices.Clone(available), func(node string) bool { -+ return !slices.Contains(preferredNodes, node) -+ }) -+ if len(candidates) == 0 { -+ candidates = available ++ isPreferred := func(node string) bool { return slices.Contains(preferredNodes, node) } ++ both := func(node string) bool { return isPreferred(node) && schedulable(node) } ++ ++ var candidates []string ++ for _, keep := range []func(string) bool{both, schedulable, isPreferred} { ++ candidates = slices.DeleteFunc(slices.Clone(available), func(node string) bool { return !keep(node) }) ++ if len(candidates) > 0 { ++ break ++ } + } + -+ if usable := slices.DeleteFunc(slices.Clone(candidates), func(node string) bool { return !schedulable(node) }); len(usable) > 0 { -+ candidates = usable ++ if len(candidates) == 0 { ++ candidates = available + } + + if len(candidates) == 0 { @@ -156,10 +160,10 @@ index 5d71c37..e8e67ec 100644 func (s *Linstor) FindSnapsByID(ctx context.Context, id string) ([]*volume.Snapshot, error) { snapshotId, err := volume.ParseSnapshotId(id) diff --git a/pkg/client/linstor_test.go b/pkg/client/linstor_test.go -index 4f3405b..84cfca2 100644 +index 4f3405b..9dedb3c 100644 --- a/pkg/client/linstor_test.go +++ b/pkg/client/linstor_test.go -@@ -577,3 +577,87 @@ func TestGetSnapshotRemoteAndReadiness(t *testing.T) { +@@ -577,3 +577,93 @@ func TestGetSnapshotRemoteAndReadiness(t *testing.T) { }) } } @@ -225,6 +229,12 @@ index 4f3405b..84cfca2 100644 + } + }) + ++ t.Run("schedulable node outside the preferred nodes wins over a cordoned preferred node", func(t *testing.T) { ++ for range 20 { ++ assert.Equal(t, "node-2", selectRestoreNode([]string{"node-3", "node-2"}, []string{"node-3"}, notNode3)) ++ } ++ }) ++ + t.Run("only cordoned nodes hold the snapshot", func(t *testing.T) { + assert.Equal(t, "node-3", selectRestoreNode([]string{"node-3"}, []string{"node-1", "node-3"}, notNode3)) + }) From 05818aa3b221a670d94c332f461ae3bb046a5de3 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:34:44 +0300 Subject: [PATCH 486/518] fix(cli): fail cozypkg and check-readiness on unwritable output For the commands this change touches (cozypkg list, the validation report of validate, push and tap, tap's summary, and check-readiness), a write error the process observes is a failure: the command exits non-zero and says whether the action it reports on already happened. Progress lines printed before the decisive step stay best-effort. A closed pipe ends the process with SIGPIPE, as for other Unix tools. `cozypkg list` and the validation report dropped write errors, and the list deferred its tabwriter flush, so a closed or full stdout still reported success with nothing printed. The list now writes to the command's writer and returns the flush error, and the report returns its write error. When validation refuses a push or a tap, or fails a validate run, the refusal stays the reported reason and the report's write error is joined to it. A push whose report cannot be written stops before flux runs and says "nothing was pushed"; a tap whose summary cannot be written fails with "tap applied, but writing its summary failed". check-readiness fails --help, --version and its readiness result the same way, in one-shot, --wait and --watch mode, with exit 1 as for a result that is not ready, which its help now lists among the exit statuses along with SIGPIPE for a closed pipe; the watch loop stops at the first frame it cannot write, and a timed-out wait still ends with its verdict. push writes its "Pushing" line before it runs flux and stops with "nothing was pushed" when that line cannot be written, with validation skipped or after the report. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- cmd/check-readiness/main.go | 55 ++++-- cmd/check-readiness/main_test.go | 68 +++++++ cmd/cozypkg/cmd/list.go | 48 +++-- cmd/cozypkg/cmd/output_test.go | 185 ++++++++++++++++++ cmd/cozypkg/cmd/push.go | 12 +- cmd/cozypkg/cmd/tap.go | 34 ++-- cmd/cozypkg/cmd/validate.go | 17 +- test/check-readiness/main_test.go | 79 ++++++++ .../testdata/golden/help-long.stdout | 4 + .../testdata/golden/help-short.stdout | 4 + .../testdata/golden/unknown-arg.stderr | 4 + .../golden/watch-nonnumeric-interval.stderr | 4 + 12 files changed, 458 insertions(+), 56 deletions(-) create mode 100644 cmd/check-readiness/main_test.go create mode 100644 cmd/cozypkg/cmd/output_test.go diff --git a/cmd/check-readiness/main.go b/cmd/check-readiness/main.go index 50fb8aaf8..eb13f62e1 100644 --- a/cmd/check-readiness/main.go +++ b/cmd/check-readiness/main.go @@ -22,6 +22,7 @@ package main import ( "fmt" + "io" "os" "os/exec" "strconv" @@ -58,6 +59,10 @@ Options: --no-color Disable color output (auto-disabled on non-TTY) -h, --help Show this help +Exit status: 0 when everything is ready, 1 when something is not ready or a +write to stdout fails, 2 on invalid arguments. A closed pipe ends the process +with SIGPIPE, like other Unix tools. + ` // entry is a resource catalog item. @@ -181,10 +186,16 @@ func run(args []string) int { cfg.coreOnly = true i++ case "-h", "--help": - fmt.Fprint(os.Stdout, usageText) + if _, err := fmt.Fprint(os.Stdout, usageText); err != nil { + fmt.Fprintf(os.Stderr, "writing usage: %v\n", err) + return 1 + } return 0 case "--version": - fmt.Fprintln(os.Stdout, Version) + if _, err := fmt.Fprintln(os.Stdout, Version); err != nil { + fmt.Fprintf(os.Stderr, "writing version: %v\n", err) + return 1 + } return 0 default: fmt.Fprintf(os.Stderr, "Unknown argument: %s\n", a) @@ -213,14 +224,14 @@ func run(args []string) int { cfg.timeoutS = sec if cfg.waitMode { - return cfg.runWait() + return cfg.runWait(os.Stdout) } if cfg.watch { - cfg.runWatch() - return 0 + return cfg.runWatch(os.Stdout) + } + if writeOutput(os.Stdout, cfg.runOnce()) != nil { + return 1 } - out := cfg.runOnce() - fmt.Print(out) if cfg.foundIssues { return 1 } @@ -281,18 +292,31 @@ func parseTimeout(raw string) (int, bool) { return 0, false } -func (cfg *config) runWait() int { +// writeOutput prints a result. A result its reader never got fails the run, +// whatever the resources' state, since the exit code alone would vouch for it. +func writeOutput(w io.Writer, parts ...string) error { + for _, part := range parts { + if _, err := io.WriteString(w, part); err != nil { + fmt.Fprintf(os.Stderr, "writing output: %v\n", err) + return err + } + } + return nil +} + +func (cfg *config) runWait(w io.Writer) int { start := time.Now() for { output := cfg.runOnce() elapsed := int(time.Since(start).Seconds()) if strings.Contains(output, "All resources are ready.") { - fmt.Print(output) - fmt.Printf("%sReady after %ds.%s\n", cfg.green, elapsed, cfg.reset) + if writeOutput(w, output, fmt.Sprintf("%sReady after %ds.%s\n", cfg.green, elapsed, cfg.reset)) != nil { + return 1 + } return 0 } if elapsed >= cfg.timeoutS { - fmt.Print(output) + _ = writeOutput(w, output) fmt.Fprintf(os.Stderr, "%s%sTimeout after %ds — resources still not ready.%s\n", cfg.red, cfg.bold, cfg.timeoutS, cfg.reset) return 1 } @@ -300,13 +324,14 @@ func (cfg *config) runWait() int { } } -func (cfg *config) runWatch() { +func (cfg *config) runWatch(w io.Writer) int { for { output := cfg.runOnce() // Smooth clear is cosmetic and untested; just reprint. - fmt.Printf("%sLast updated: %s (refreshing every %ds, Ctrl+C to stop)%s\n", cfg.bold, time.Now().Format(time.UnixDate), cfg.interval, cfg.reset) - fmt.Println() - fmt.Print(output) + header := fmt.Sprintf("%sLast updated: %s (refreshing every %ds, Ctrl+C to stop)%s\n\n", cfg.bold, time.Now().Format(time.UnixDate), cfg.interval, cfg.reset) + if writeOutput(w, header, output) != nil { + return 1 + } time.Sleep(time.Duration(cfg.interval) * time.Second) } } diff --git a/cmd/check-readiness/main_test.go b/cmd/check-readiness/main_test.go new file mode 100644 index 000000000..9afddeedf --- /dev/null +++ b/cmd/check-readiness/main_test.go @@ -0,0 +1,68 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "io" + "os" + "strings" + "testing" +) + +func TestHelpAndVersionSucceedOnWritableStdout(t *testing.T) { + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = r.Close() }) + stdout := os.Stdout + os.Stdout = w + t.Cleanup(func() { os.Stdout = stdout }) + read := make(chan string, 1) + go func() { + b, _ := io.ReadAll(r) + read <- string(b) + }() + + for _, arg := range []string{"--help", "--version"} { + if code := run([]string{arg}); code != 0 { + t.Errorf("run(%s) = %d, want 0", arg, code) + } + } + _ = w.Close() + if out := <-read; !strings.Contains(out, "Usage:") || !strings.Contains(out, Version) { + t.Errorf("stdout = %q, want the usage text and the version", out) + } +} + +func TestHelpAndVersionFailOnUnwritableStdout(t *testing.T) { + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + _ = r.Close() + _ = w.Close() + stdout := os.Stdout + os.Stdout = w + t.Cleanup(func() { os.Stdout = stdout }) + + for _, arg := range []string{"--help", "--version"} { + if code := run([]string{arg}); code == 0 { + t.Errorf("run(%s) with an unwritable stdout = 0, want a failure", arg) + } + } +} diff --git a/cmd/cozypkg/cmd/list.go b/cmd/cozypkg/cmd/list.go index 6f7375026..4b08a602b 100644 --- a/cmd/cozypkg/cmd/list.go +++ b/cmd/cozypkg/cmd/list.go @@ -19,7 +19,7 @@ package cmd import ( "context" "fmt" - "os" + "io" "strings" "text/tabwriter" @@ -78,24 +78,25 @@ Use --components flag to show components on separate lines.`, } if listCmdFlags.installed { - return listPackages(ctx, k8sClient, listCmdFlags.components) + return listPackages(ctx, k8sClient, cmd.OutOrStdout(), listCmdFlags.components) } - return listPackageSources(ctx, k8sClient, listCmdFlags.components) + return listPackageSources(ctx, k8sClient, cmd.OutOrStdout(), listCmdFlags.components) }, } -func listPackageSources(ctx context.Context, k8sClient client.Client, showComponents bool) error { +func listPackageSources(ctx context.Context, k8sClient client.Client, out io.Writer, showComponents bool) error { var psList cozyv1alpha1.PackageSourceList if err := k8sClient.List(ctx, &psList); err != nil { return fmt.Errorf("failed to list PackageSources: %w", err) } // Use tabwriter for better column alignment - w := tabwriter.NewWriter(os.Stdout, 0, 0, 3, ' ', 0) - defer w.Flush() + w := tabwriter.NewWriter(out, 0, 0, 3, ' ', 0) // Print header - fmt.Fprintln(w, "NAME\tVARIANTS\tREADY\tSTATUS") + if _, err := fmt.Fprintln(w, "NAME\tVARIANTS\tREADY\tSTATUS"); err != nil { + return err + } // Print rows for _, ps := range psList.Items { @@ -123,24 +124,26 @@ func listPackageSources(ctx context.Context, k8sClient client.Client, showCompon } } - fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", ps.Name, variantsStr, ready, status) + if _, err := fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", ps.Name, variantsStr, ready, status); err != nil { + return err + } // Show components if requested if showComponents { for _, variant := range ps.Spec.Variants { for _, component := range variant.Components { - fmt.Fprintf(w, " %s\t%s\t\t\n", - fmt.Sprintf("%s.%s", ps.Name, component.Name), - variant.Name) + if _, err := fmt.Fprintf(w, " %s.%s\t%s\t\t\n", ps.Name, component.Name, variant.Name); err != nil { + return err + } } } } } - return nil + return w.Flush() } -func listPackages(ctx context.Context, k8sClient client.Client, showComponents bool) error { +func listPackages(ctx context.Context, k8sClient client.Client, out io.Writer, showComponents bool) error { var pkgList cozyv1alpha1.PackageList if err := k8sClient.List(ctx, &pkgList); err != nil { return fmt.Errorf("failed to list Packages: %w", err) @@ -160,11 +163,12 @@ func listPackages(ctx context.Context, k8sClient client.Client, showComponents b } // Use tabwriter for better column alignment - w := tabwriter.NewWriter(os.Stdout, 0, 0, 3, ' ', 0) - defer w.Flush() + w := tabwriter.NewWriter(out, 0, 0, 3, ' ', 0) // Print header - fmt.Fprintln(w, "NAME\tVARIANT\tREADY\tSTATUS") + if _, err := fmt.Fprintln(w, "NAME\tVARIANT\tREADY\tSTATUS"); err != nil { + return err + } // Print rows for _, pkg := range pkgList.Items { @@ -187,7 +191,9 @@ func listPackages(ctx context.Context, k8sClient client.Client, showComponents b } } - fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", pkg.Name, variant, ready, status) + if _, err := fmt.Fprintf(w, "%s\t%s\t%s\t%s\n", pkg.Name, variant, ready, status); err != nil { + return err + } // Show components if requested if showComponents { @@ -197,9 +203,9 @@ func listPackages(ctx context.Context, k8sClient client.Client, showComponents b for _, v := range ps.Spec.Variants { if v.Name == variant { for _, component := range v.Components { - fmt.Fprintf(w, " %s\t%s\t\t\n", - fmt.Sprintf("%s.%s", pkg.Name, component.Name), - variant) + if _, err := fmt.Fprintf(w, " %s.%s\t%s\t\t\n", pkg.Name, component.Name, variant); err != nil { + return err + } } break } @@ -208,7 +214,7 @@ func listPackages(ctx context.Context, k8sClient client.Client, showComponents b } } - return nil + return w.Flush() } func init() { diff --git a/cmd/cozypkg/cmd/output_test.go b/cmd/cozypkg/cmd/output_test.go new file mode 100644 index 000000000..9eed21639 --- /dev/null +++ b/cmd/cozypkg/cmd/output_test.go @@ -0,0 +1,185 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package cmd + +import ( + "context" + "errors" + "fmt" + "io" + "os" + "path/filepath" + "strings" + "testing" + + cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1" + "github.com/spf13/cobra" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + utilruntime "k8s.io/apimachinery/pkg/util/runtime" + "sigs.k8s.io/controller-runtime/pkg/client/fake" +) + +type failingWriter struct{} + +func (failingWriter) Write([]byte) (int, error) { return 0, errors.New("broken pipe") } + +func TestPrintReportReturnsWriteError(t *testing.T) { + r := &Report{Findings: []Finding{{Severity: SeverityError, Code: "X", Message: "m"}}} + if err := printReport(r, failingWriter{}); err == nil { + t.Fatal("printReport to an unwritable writer returned nil") + } +} + +// The table is buffered until the tabwriter flushes, so a failing reader shows +// up only as a Flush error, which list has to return. +func TestListReturnsWriteError(t *testing.T) { + s := runtime.NewScheme() + utilruntime.Must(cozyv1alpha1.AddToScheme(s)) + ps := &cozyv1alpha1.PackageSource{ObjectMeta: metav1.ObjectMeta{Name: "acme.app"}} + pkg := &cozyv1alpha1.Package{ObjectMeta: metav1.ObjectMeta{Name: "acme.app"}} + cl := fake.NewClientBuilder().WithScheme(s).WithObjects(ps, pkg).Build() + + for name, list := range map[string]func(io.Writer) error{ + "sources": func(w io.Writer) error { return listPackageSources(context.Background(), cl, w, true) }, + "packages": func(w io.Writer) error { return listPackages(context.Background(), cl, w, true) }, + } { + t.Run(name, func(t *testing.T) { + if err := list(failingWriter{}); err == nil { + t.Fatal("list to an unwritable writer returned nil") + } + var out strings.Builder + if err := list(&out); err != nil || !strings.Contains(out.String(), "acme.app") { + t.Fatalf("list = %v, output %q; want the row", err, out.String()) + } + }) + } +} + +// A command that cannot print its validation report still fails, and a +// refusal keeps validation as its reason with the write error joined to it. +func TestValidationCommandsReportUnwritableReport(t *testing.T) { + fakeFlux := t.TempDir() + if err := os.WriteFile(filepath.Join(fakeFlux, "flux"), []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", fakeFlux+string(os.PathListSeparator)+os.Getenv("PATH")) + + emptyRepo := func(t *testing.T) string { + root := t.TempDir() + if err := os.Mkdir(filepath.Join(root, "packages"), 0o755); err != nil { + t.Fatal(err) + } + return root + } + run := func(c *cobra.Command, args ...string) error { + c.SetOut(failingWriter{}) + t.Cleanup(func() { c.SetOut(nil) }) + return c.RunE(c, args) + } + push := func(root string) error { + pushCmdFlags.path = root + t.Cleanup(func() { pushCmdFlags.path = "." }) + return run(pushCmd, "oci://registry.example/acme/apps:v1") + } + + for _, tc := range []struct { + name string + run func(t *testing.T) error + refusal string + }{ + {name: "validate, valid", run: func(t *testing.T) error { return run(validateCmd, validRepo(t)) }}, + {name: "validate, invalid", run: func(t *testing.T) error { return run(validateCmd, emptyRepo(t)) }, refusal: "validation failed"}, + {name: "push, valid", run: func(t *testing.T) error { return push(validRepo(t)) }, refusal: "nothing was pushed"}, + {name: "push, invalid", run: func(t *testing.T) error { return push(emptyRepo(t)) }, refusal: "refusing to push"}, + {name: "tap, invalid", run: func(t *testing.T) error { return run(tapCmd, "oci://registry.example/acme/apps:v1") }, refusal: "refusing to tap"}, + } { + t.Run(tc.name, func(t *testing.T) { + err := tc.run(t) + if err == nil || !strings.Contains(err.Error(), "broken pipe") { + t.Fatalf("err = %v, want the write error", err) + } + if tc.refusal != "" && !strings.Contains(err.Error(), tc.refusal) { + t.Errorf("err = %v, want the refusal %q kept", err, tc.refusal) + } + }) + } +} + +// A tap that applied is reported as applied even when its summary is lost, +// and the lost summary still fails the command. +func TestTapSummaryReportsUnwritableOutput(t *testing.T) { + err := writeTapSummary(failingWriter{}, "oci://registry.example/acme/apps:v1", "tap-acme", []string{"acme.app"}) + if err == nil || !strings.Contains(err.Error(), "tap applied") || !strings.Contains(err.Error(), "broken pipe") { + t.Fatalf("err = %v, want the write error with the tap reported as applied", err) + } + var out strings.Builder + if err := writeTapSummary(&out, "oci://registry.example/acme/apps:v1", "tap-acme", []string{"acme.app"}); err != nil || !strings.Contains(out.String(), "cozypkg add acme.app") { + t.Fatalf("writeTapSummary = %v, output %q", err, out.String()) + } +} + +// A clean report has no finding lines, so its summary is its only write and +// alone decides the result. +func TestPrintReportReturnsSummaryWriteError(t *testing.T) { + if err := printReport(&Report{}, failingWriter{}); err == nil { + t.Fatal("printReport of a clean report to an unwritable writer returned nil") + } +} + +// push runs flux only after its status line was written, with validation +// skipped and after a report that was written. +func TestPushStopsBeforeFluxWhenItsStatusIsLost(t *testing.T) { + fakeFlux := t.TempDir() + ran := filepath.Join(t.TempDir(), "flux-ran") + if err := os.WriteFile(filepath.Join(fakeFlux, "flux"), []byte("#!/bin/sh\ntouch "+ran+"\n"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", fakeFlux+string(os.PathListSeparator)+os.Getenv("PATH")) + for _, skip := range []bool{true, false} { + t.Run(fmt.Sprintf("skip-validate=%v", skip), func(t *testing.T) { + pushCmdFlags.path, pushCmdFlags.skipValidate = validRepo(t), skip + pushCmdFlags.source, pushCmdFlags.revision = "https://example.com/acme.git", "v1" + t.Cleanup(func() { + pushCmdFlags.path, pushCmdFlags.skipValidate = ".", false + pushCmdFlags.source, pushCmdFlags.revision = "", "" + }) + pushCmd.SetOut(&failOn{want: "Pushing"}) + t.Cleanup(func() { pushCmd.SetOut(nil) }) + err := pushCmd.RunE(pushCmd, []string{"oci://registry.example/acme/apps:v1"}) + if err == nil || !strings.Contains(err.Error(), "nothing was pushed") { + t.Fatalf("push losing its status = %v, want nothing pushed", err) + } + if _, statErr := os.Stat(ran); statErr == nil { + t.Fatal("flux ran after the status line was lost") + } + }) + } +} + +// failOn fails every write that contains want, and takes every other one. +type failOn struct { + want string + buf strings.Builder +} + +func (w *failOn) Write(p []byte) (int, error) { + if strings.Contains(string(p), w.want) { + return 0, errors.New("broken pipe") + } + return w.buf.Write(p) +} diff --git a/cmd/cozypkg/cmd/push.go b/cmd/cozypkg/cmd/push.go index 3f8e65cac..f60004034 100644 --- a/cmd/cozypkg/cmd/push.go +++ b/cmd/cozypkg/cmd/push.go @@ -17,6 +17,7 @@ limitations under the License. package cmd import ( + "errors" "fmt" "os" "os/exec" @@ -106,10 +107,13 @@ published. Source URL and revision are derived from git when not given.`, if err != nil { return err } - printReport(rep, cmd.OutOrStdout()) + printErr := printReport(rep, cmd.OutOrStdout()) if rep.HasErrors() { errs, _, _ := rep.Counts() - return fmt.Errorf("refusing to push: validation failed with %d error(s) (use --skip-validate to override)", errs) + return errors.Join(fmt.Errorf("refusing to push: validation failed with %d error(s) (use --skip-validate to override)", errs), printErr) + } + if printErr != nil { + return fmt.Errorf("nothing was pushed: writing the validation report failed: %w", printErr) } } @@ -129,7 +133,9 @@ published. Source URL and revision are derived from git when not given.`, } fluxArgs := buildFluxPushArgs(ref, packagesDir, source, revision, pushCmdFlags.reproducible) - fmt.Fprintf(cmd.OutOrStdout(), "Pushing %s (source=%s revision=%s)\n", ref, source, revision) + if _, err := fmt.Fprintf(cmd.OutOrStdout(), "Pushing %s (source=%s revision=%s)\n", ref, source, revision); err != nil { + return fmt.Errorf("nothing was pushed: writing the push status failed: %w", err) + } flux := exec.Command("flux", fluxArgs...) flux.Stdout = cmd.OutOrStdout() flux.Stderr = cmd.ErrOrStderr() diff --git a/cmd/cozypkg/cmd/tap.go b/cmd/cozypkg/cmd/tap.go index e04edebd7..2d5f7bb9b 100644 --- a/cmd/cozypkg/cmd/tap.go +++ b/cmd/cozypkg/cmd/tap.go @@ -18,6 +18,7 @@ package cmd import ( "context" + "errors" "fmt" "io" "regexp" @@ -402,9 +403,8 @@ runs its charts in your management cluster: tap only sources you trust.`, return err } if rep.HasErrors() { - printReport(rep, cmd.OutOrStdout()) errs, _, _ := rep.Counts() - return fmt.Errorf("refusing to tap: artifact failed validation with %d error(s) (use --skip-validate to override)", errs) + return errors.Join(fmt.Errorf("refusing to tap: artifact failed validation with %d error(s) (use --skip-validate to override)", errs), printReport(rep, cmd.OutOrStdout())) } } @@ -487,19 +487,29 @@ runs its charts in your management cluster: tap only sources you trust.`, _, _ = fmt.Fprintf(cmd.ErrOrStderr(), "warning: could not reset materialization state on OCIRepository/%s: %v\n", srcName, err) } - fmt.Fprintf(cmd.OutOrStdout(), "Tapped %s\n OCIRepository/%s\n", fullRef, srcName) - for _, n := range names { - fmt.Fprintf(cmd.OutOrStdout(), " PackageSource/%s\n", n) - } - _, _ = fmt.Fprintf(cmd.OutOrStdout(), "Tapping installs nothing. Install an app with:\n") - for _, n := range names { - fmt.Fprintf(cmd.OutOrStdout(), " cozypkg add %s\n", n) - } - _, _ = fmt.Fprintf(cmd.OutOrStdout(), "A privileged install component is not installed until you confirm it with 'cozypkg add --allow-privileged'.\n") - return nil + return writeTapSummary(cmd.OutOrStdout(), fullRef, srcName, names) }, } +// writeTapSummary reports what a tap created. The tap is applied by then, so a +// summary that cannot be written fails the command while saying so. +func writeTapSummary(w io.Writer, fullRef, srcName string, names []string) error { + var b strings.Builder + fmt.Fprintf(&b, "Tapped %s\n OCIRepository/%s\n", fullRef, srcName) + for _, n := range names { + fmt.Fprintf(&b, " PackageSource/%s\n", n) + } + b.WriteString("Tapping installs nothing. Install an app with:\n") + for _, n := range names { + fmt.Fprintf(&b, " cozypkg add %s\n", n) + } + b.WriteString("A privileged install component is not installed until you confirm it with 'cozypkg add --allow-privileged'.\n") + if _, err := io.WriteString(w, b.String()); err != nil { + return fmt.Errorf("tap applied, but writing its summary failed: %w", err) + } + return nil +} + var untapCmd = &cobra.Command{ Use: "untap ", Short: "Remove a tapped repository", diff --git a/cmd/cozypkg/cmd/validate.go b/cmd/cozypkg/cmd/validate.go index 809f0f246..f781cfd8b 100644 --- a/cmd/cozypkg/cmd/validate.go +++ b/cmd/cozypkg/cmd/validate.go @@ -21,6 +21,7 @@ import ( "bytes" "context" "encoding/json" + "errors" "fmt" "io" "io/fs" @@ -535,16 +536,19 @@ func pullOCIArtifact(ref string) (string, func(), error) { return dir, cleanup, nil } -func printReport(r *Report, w io.Writer) { +func printReport(r *Report, w io.Writer) error { for _, f := range r.Findings { loc := f.Location if loc == "" { loc = "-" } - fmt.Fprintf(w, "[%s] %s (%s): %s\n", f.Severity, loc, f.Code, f.Message) + if _, err := fmt.Fprintf(w, "[%s] %s (%s): %s\n", f.Severity, loc, f.Code, f.Message); err != nil { + return err + } } errs, warns, infos := r.Counts() - fmt.Fprintf(w, "\nSummary: %d error(s), %d warning(s), %d info\n", errs, warns, infos) + _, err := fmt.Fprintf(w, "\nSummary: %d error(s), %d warning(s), %d info\n", errs, warns, infos) + return err } var validateCmdFlags struct { @@ -663,10 +667,13 @@ runs "helm lint" on every component chart.`, if err != nil { return err } - printReport(rep, cmd.OutOrStdout()) + printErr := printReport(rep, cmd.OutOrStdout()) if rep.HasErrors() { errs, _, _ := rep.Counts() - return fmt.Errorf("validation failed: %d error(s)", errs) + return errors.Join(fmt.Errorf("validation failed: %d error(s)", errs), printErr) + } + if printErr != nil { + return fmt.Errorf("writing the validation report failed: %w", printErr) } return nil }, diff --git a/test/check-readiness/main_test.go b/test/check-readiness/main_test.go index 622340c7f..01e6c1505 100644 --- a/test/check-readiness/main_test.go +++ b/test/check-readiness/main_test.go @@ -13,6 +13,7 @@ package checkreadiness_test import ( "bytes" + "context" "errors" "flag" "fmt" @@ -23,6 +24,7 @@ import ( "runtime" "strings" "testing" + "time" ) var update = flag.Bool("update", false, "regenerate golden files") @@ -335,3 +337,80 @@ func repoRoot(t *testing.T) string { } return root } + +// A result that never reaches stdout fails the run, whatever the cluster's +// state, and the watch loop stops instead of refreshing into the void. stdout +// is a read-only descriptor, so every write fails without a SIGPIPE. +func TestUnwritableStdoutFails(t *testing.T) { + root := repoRoot(t) + fakeBin := filepath.Join(root, "test", "check-readiness", "fakekubectl") + base := filepath.Join(root, "test", "check-readiness", "testdata", "fixtures", "_base") + for _, tc := range []struct { + name string + args []string + }{ + {name: "once", args: []string{"--no-color"}}, + {name: "wait", args: []string{"--wait", "--no-color"}}, + {name: "watch", args: []string{"-w", "1", "--no-color"}}, + } { + t.Run(tc.name, func(t *testing.T) { + stdout, err := os.Open(os.DevNull) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = stdout.Close() }) + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + t.Cleanup(cancel) + argv := subjectArgv(tc.args) + cmd := exec.CommandContext(ctx, argv[0], argv[1:]...) + cmd.Env = testEnv(fakeBin, base, base, filepath.Join(t.TempDir(), "kubectl.log")) + var stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = stdout, &stderr + runErr := cmd.Run() + if ctx.Err() != nil { + t.Fatalf("still running after its output failed\nstderr:\n%s", stderr.String()) + } + if code := exitCode(t, runErr); code != 1 { + t.Errorf("exit code = %d, want 1\nstderr:\n%s", code, stderr.String()) + } + if !strings.Contains(stderr.String(), "writing output") { + t.Errorf("stderr does not say the output was lost:\n%s", stderr.String()) + } + }) + } +} + +// A wait that times out ends with its verdict: the table comes first and the +// timeout line last, also when the table could not be written. +func TestWaitTimeoutVerdictComesLast(t *testing.T) { + root := repoRoot(t) + fakeBin := filepath.Join(root, "test", "check-readiness", "fakekubectl") + fixtures := filepath.Join(root, "test", "check-readiness", "testdata", "fixtures") + args := []string{"-w", "1", "--wait", "--timeout", "1s", "--no-color"} + for _, unwritable := range []bool{false, true} { + t.Run(fmt.Sprintf("unwritable=%v", unwritable), func(t *testing.T) { + cmd := scriptCommand(args) + cmd.Env = testEnv(fakeBin, filepath.Join(fixtures, "helmrelease-not-ready"), filepath.Join(fixtures, "_base"), filepath.Join(t.TempDir(), "kubectl.log")) + var combined bytes.Buffer + cmd.Stdout, cmd.Stderr = &combined, &combined + if unwritable { + stdout, err := os.Open(os.DevNull) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = stdout.Close() }) + cmd.Stdout = stdout + } + if code := exitCode(t, cmd.Run()); code != 1 { + t.Errorf("exit code = %d, want 1\n%s", code, combined.String()) + } + lines := strings.Split(strings.TrimRight(combined.String(), "\n"), "\n") + if last := lines[len(lines)-1]; !strings.HasPrefix(last, "Timeout after") { + t.Errorf("last line = %q, want the timeout verdict\n%s", last, combined.String()) + } + if unwritable && !strings.Contains(combined.String(), "writing output") { + t.Errorf("stderr does not say the table was lost:\n%s", combined.String()) + } + }) + } +} diff --git a/test/check-readiness/testdata/golden/help-long.stdout b/test/check-readiness/testdata/golden/help-long.stdout index f1f289e08..428f135aa 100644 --- a/test/check-readiness/testdata/golden/help-long.stdout +++ b/test/check-readiness/testdata/golden/help-long.stdout @@ -21,3 +21,7 @@ Options: --no-color Disable color output (auto-disabled on non-TTY) -h, --help Show this help +Exit status: 0 when everything is ready, 1 when something is not ready or a +write to stdout fails, 2 on invalid arguments. A closed pipe ends the process +with SIGPIPE, like other Unix tools. + diff --git a/test/check-readiness/testdata/golden/help-short.stdout b/test/check-readiness/testdata/golden/help-short.stdout index f1f289e08..428f135aa 100644 --- a/test/check-readiness/testdata/golden/help-short.stdout +++ b/test/check-readiness/testdata/golden/help-short.stdout @@ -21,3 +21,7 @@ Options: --no-color Disable color output (auto-disabled on non-TTY) -h, --help Show this help +Exit status: 0 when everything is ready, 1 when something is not ready or a +write to stdout fails, 2 on invalid arguments. A closed pipe ends the process +with SIGPIPE, like other Unix tools. + diff --git a/test/check-readiness/testdata/golden/unknown-arg.stderr b/test/check-readiness/testdata/golden/unknown-arg.stderr index d6ec7c0db..ca652d9ea 100644 --- a/test/check-readiness/testdata/golden/unknown-arg.stderr +++ b/test/check-readiness/testdata/golden/unknown-arg.stderr @@ -22,3 +22,7 @@ Options: --no-color Disable color output (auto-disabled on non-TTY) -h, --help Show this help +Exit status: 0 when everything is ready, 1 when something is not ready or a +write to stdout fails, 2 on invalid arguments. A closed pipe ends the process +with SIGPIPE, like other Unix tools. + diff --git a/test/check-readiness/testdata/golden/watch-nonnumeric-interval.stderr b/test/check-readiness/testdata/golden/watch-nonnumeric-interval.stderr index 3cb2c43bb..cc23fc457 100644 --- a/test/check-readiness/testdata/golden/watch-nonnumeric-interval.stderr +++ b/test/check-readiness/testdata/golden/watch-nonnumeric-interval.stderr @@ -22,3 +22,7 @@ Options: --no-color Disable color output (auto-disabled on non-TTY) -h, --help Show this help +Exit status: 0 when everything is ready, 1 when something is not ready or a +write to stdout fails, 2 on invalid arguments. A closed pipe ends the process +with SIGPIPE, like other Unix tools. + From f99e511eea92e311eaf68234f1d8e16cdcfb07d4 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:42:56 +0300 Subject: [PATCH 487/518] fix(backport): fail a run whose output could not be written Every report, check and verify result goes to stdout, and every write error there was dropped, so a run whose output never reached its reader still exited 0. The terminal writer now keeps the first stdout error, and the run fails with it unless it already failed. Only stdout is tracked; a write error on stderr is still discarded. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- cmd/backport-batch/README.md | 2 ++ cmd/backport-batch/assemble.go | 24 ++++++------- cmd/backport-batch/check.go | 16 ++++----- cmd/backport-batch/main.go | 36 +++++++++++++++----- cmd/backport-batch/notify.go | 14 ++++---- cmd/backport-batch/output_test.go | 56 +++++++++++++++++++++++++++++++ cmd/backport-batch/report.go | 30 ++++++++--------- cmd/backport-batch/verify.go | 10 +++--- 8 files changed, 133 insertions(+), 55 deletions(-) create mode 100644 cmd/backport-batch/output_test.go diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index 81afd0ef6..4919035c0 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -27,6 +27,8 @@ Requires `git` and an authenticated `gh` on PATH, and a remote called `origin`. Build the binary wherever an exit code matters: `go run` turns any non-zero exit into its own `1`, so under it a check that could not be completed looks the same as one that found something missing. +A command whose output on stdout could not be written exits 2 where it would have exited 0, and says on stderr that the output was lost; anything the run did before that, a push or a published pull request included, has still happened. + ### run Switches the current working tree to `backport-batch/-`, created at `origin/`, and picks onto it one original at a time, in the order they merged into `main`, as one unit each on the branch's first-parent line. A pull request merged into `main` with a merge commit `M` goes on as its own commits, `git rev-list --reverse --topo-order --no-merges M^1..M^2`, each picked with `git cherry-pick -x --signoff`, so each keeps its author, message and sign-offs and gains the `(cherry picked from commit …)` line; the merges that brought `main` into its branch are left out, since `main` already has what they brought. A merge commit then joins the picks to the batch: its first parent is the batch before the original, its second the last pick, and its message is `M`'s with a paragraph of its own, `Backported-from: M`. A pull request squash-merged into `main` goes on as one pick. When any commit of an original conflicts, the pick is aborted, the branch goes back to where it was before that original, and the original is recorded with the commit that failed and its files; the run goes on with the next one. A commit the branch already carries is skipped as empty, and an original none of whose commits changes anything is reported as carried. The tree has to be clean. A local branch of that name is reset to where the run starts; when that drops commits only it has, the run first prints its tip and the `git branch` command that restores them, and so does `--defer`. Signing follows your git configuration; hooks are off for the picks and the merge, so each commit carries its message as git writes it. diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index 3c025517e..a0aab98db 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -153,7 +153,7 @@ func (a *app) run(args []string) int { } // The scheduled job has half an hour for every line; this says where it // goes. - fmt.Fprintf(a.stderr, "prepared %s in %s, %s of it judging partial units and what originals build on\n", l.branch(), time.Since(started).Round(time.Millisecond), b.detecting.Round(time.Millisecond)) + _, _ = fmt.Fprintf(a.stderr, "prepared %s in %s, %s of it judging partial units and what originals build on\n", l.branch(), time.Since(started).Round(time.Millisecond), b.detecting.Round(time.Millisecond)) if drops != nil { return a.redrop(b, drops, reason, *undropArg != "", *push) } @@ -449,20 +449,20 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r } body, left := r.body(template) p.body = body - fmt.Fprint(a.stdout, body) + _, _ = fmt.Fprint(a.stdout, body) skipped := 0 for _, items := range r.skipped { skipped += len(items) } for _, rule := range left { for _, v := range r.skipped[rule] { - fmt.Fprintf(a.stderr, "skipped by %s: %s %s (%s)\n", ruleName(rule), v.o.url(), terminalSafe(v.o.title), terminalSafe(v.reason)) + _, _ = fmt.Fprintf(a.stderr, "skipped by %s: %s %s (%s)\n", ruleName(rule), v.o.url(), terminalSafe(v.o.title), terminalSafe(v.reason)) } } for _, p := range r.waiting { - fmt.Fprintf(a.stderr, "waiting for a maintainer: %s %s changes %s\n", p.v.o.url(), terminalSafe(p.v.o.title), terminalSafe(strings.Join(p.files, ", "))) + _, _ = fmt.Fprintf(a.stderr, "waiting for a maintainer: %s %s changes %s\n", p.v.o.url(), terminalSafe(p.v.o.title), terminalSafe(strings.Join(p.files, ", "))) } - fmt.Fprintf(a.stderr, "%s at %.12s: %d on the branch, %d deferred, %d waiting for a maintainer, %d skipped\ntitle: %s\nreport: %s\n", + _, _ = fmt.Fprintf(a.stderr, "%s at %.12s: %d on the branch, %d deferred, %d waiting for a maintainer, %d skipped\ntitle: %s\nreport: %s\n", branch, head, len(r.onBranch), len(r.deferred)+len(r.held), len(r.waiting), skipped, p.title, doc.ID) if !push { return 0 @@ -471,8 +471,8 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r // Picking nothing is not a failure of the run; check still reports // whatever the line lacks. Merged and closed batches are never // written, so there is nowhere to keep what the run found. - fmt.Fprintf(a.stderr, "%s carries nothing beyond %s; nothing to push\n", branch, b.line.branch()) - fmt.Fprintf(a.stderr, "no batch pull request is open on %s; this run's classification is not retained\n", b.line.branch()) + _, _ = fmt.Fprintf(a.stderr, "%s carries nothing beyond %s; nothing to push\n", branch, b.line.branch()) + _, _ = fmt.Fprintf(a.stderr, "no batch pull request is open on %s; this run's classification is not retained\n", b.line.branch()) return 0 } if err := a.publish(b, p, rebuilt); err != nil { @@ -537,7 +537,7 @@ func (a *app) warnReset(branch, start string) error { if err != nil || kept { return err } - fmt.Fprintf(a.stderr, "resetting the local %s from %s, dropping commits %.12s does not carry; git branch %s-%.12s %s restores them\n", branch, old, start, branch, old, old) + _, _ = fmt.Fprintf(a.stderr, "resetting the local %s from %s, dropping commits %.12s does not carry; git branch %s-%.12s %s restores them\n", branch, old, start, branch, old, old) return nil } @@ -733,7 +733,7 @@ func (a *app) publish(b *batch, p publication, rebuilt bool) error { if err != nil { return err } - fmt.Fprintf(a.stderr, "opened draft %s\n", url) + _, _ = fmt.Fprintf(a.stderr, "opened draft %s\n", url) if err := a.postReport(n, p.report); err != nil { return fmt.Errorf("posting report %.12s on #%d: %w; the body keeps it, and the next run --push posts it", p.report.doc.ID, n, err) } @@ -745,7 +745,7 @@ func (a *app) publish(b *batch, p publication, rebuilt bool) error { if err := a.gh.editPull(b.pr.Number, p.title, name); err != nil { return err } - fmt.Fprintf(a.stderr, "updated %s\n", pullURL(b.pr)) + _, _ = fmt.Fprintf(a.stderr, "updated %s\n", pullURL(b.pr)) return nil } @@ -861,7 +861,7 @@ func (a *app) finishClose(l line, url string, c closing) error { if err := a.deleteBranch(l, c); err != nil { return err } - fmt.Fprintf(a.stderr, "closed %s: without what is deferred it carries nothing beyond %s; the next batch tries them again\n", url, l.branch()) + _, _ = fmt.Fprintf(a.stderr, "closed %s: without what is deferred it carries nothing beyond %s; the next batch tries them again\n", url, l.branch()) return nil } @@ -933,7 +933,7 @@ func (a *app) resumeClose(b *batch, push, changing bool) (bool, int) { if err := a.deleteBranch(b.line, c); err != nil { return false, a.fail(err) } - fmt.Fprintf(a.stderr, "deleted %s, which the close of #%d left\n", c.Branch, c.PR) + _, _ = fmt.Fprintf(a.stderr, "deleted %s, which the close of #%d left\n", c.Branch, c.PR) return false, a.finished(c, changing) } diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index d8acf4e37..2101d251f 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -73,14 +73,14 @@ func (a *app) check(args []string) int { case d.kind == outcomeWarning: a.warn("%s", d.text) case d.kind == outcomeNote: - fmt.Fprintf(a.stderr, "note: %s\n", d.text) + _, _ = fmt.Fprintf(a.stderr, "note: %s\n", d.text) } } if len(missing)+len(failing) == 0 { if *cut { - fmt.Fprintf(a.stdout, "%s: no change the policy takes is missing before the cut\n", l.branch()) + _, _ = fmt.Fprintf(a.stdout, "%s: no change the policy takes is missing before the cut\n", l.branch()) } else { - fmt.Fprintf(a.stdout, "%s: every change the policy takes is on the line or on the batch\n", l.branch()) + _, _ = fmt.Fprintf(a.stdout, "%s: every change the policy takes is on the line or on the batch\n", l.branch()) } return 0 } @@ -89,13 +89,13 @@ func (a *app) check(args []string) int { where = "nor on batch pull request #" + strconv.Itoa(b.pr.Number) } if len(missing) > 0 { - fmt.Fprintf(a.stdout, "%s: %d changes the policy takes are not on the line, %s:\n", l.branch(), len(missing), where) + _, _ = fmt.Fprintf(a.stdout, "%s: %d changes the policy takes are not on the line, %s:\n", l.branch(), len(missing), where) for _, text := range missing { - fmt.Fprintf(a.stdout, " %s\n", text) + _, _ = fmt.Fprintf(a.stdout, " %s\n", text) } } for _, text := range failing { - fmt.Fprintf(a.stdout, "%s: %s\n", l.branch(), text) + _, _ = fmt.Fprintf(a.stdout, "%s: %s\n", l.branch(), text) } return 1 } @@ -326,10 +326,10 @@ func (a *app) fold(args []string) int { return a.fail(err) } if why != "" { - fmt.Fprintf(a.stdout, "refused: %s\n", why) + _, _ = fmt.Fprintf(a.stdout, "refused: %s\n", why) return 1 } - fmt.Fprintln(a.stdout, "ok") + _, _ = fmt.Fprintln(a.stdout, "ok") return 0 } diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index 23b357dff..19102e57f 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -111,6 +111,9 @@ type app struct { policyRef string // tool is the commit backport-batch was built from, for the record. tool string + // unwritten is the first error writing to stdout, which fails the run: + // what it printed never reached its reader. + unwritten error } func main() { @@ -119,10 +122,21 @@ func main() { } func (a *app) main(args []string) int { - a.stdout, a.stderr = textWriter{a.stdout}, textWriter{a.stderr} + a.stdout, a.stderr = textWriter{w: a.stdout, failed: &a.unwritten}, textWriter{w: a.stderr} + code := a.command(args) + if a.unwritten != nil { + _, _ = fmt.Fprintf(a.stderr, "backport-batch: output was lost: %v; anything the run did, such as a push or a publish, still happened\n", a.unwritten) + if code == 0 { + code = 2 + } + } + return code +} + +func (a *app) command(args []string) int { a.gh = &permCache{github: a.gh, perms: map[string]string{}} if len(args) == 0 { - fmt.Fprint(a.stderr, usageText) + _, _ = fmt.Fprint(a.stderr, usageText) return 2 } if bot := a.gh.batchBot(); bot != "" && !botLoginRE.MatchString(bot) { @@ -144,20 +158,26 @@ func (a *app) main(args []string) int { case "lines": return a.lines(args[1:]) case "-h", "--help", "help": - fmt.Fprint(a.stdout, usageText) + _, _ = fmt.Fprint(a.stdout, usageText) return 0 } - fmt.Fprintf(a.stderr, "unknown command %q\n\n%s", args[0], usageText) + _, _ = fmt.Fprintf(a.stderr, "unknown command %q\n\n%s", args[0], usageText) return 2 } // textWriter keeps output fit for a terminal. Titles, file names and the // messages git relays come from other people, and a control sequence among // them would move the cursor or rewrite what was printed before. -type textWriter struct{ w io.Writer } +type textWriter struct { + w io.Writer + failed *error +} func (t textWriter) Write(p []byte) (int, error) { if _, err := io.WriteString(t.w, clean(string(p), true)); err != nil { + if t.failed != nil && *t.failed == nil { + *t.failed = err + } return 0, err } return len(p), nil @@ -166,7 +186,7 @@ func (t textWriter) Write(p []byte) (int, error) { func (a *app) flags(name string) *flag.FlagSet { fs := flag.NewFlagSet(name, flag.ContinueOnError) fs.SetOutput(a.stderr) - fs.Usage = func() { fmt.Fprint(a.stderr, usageText) } + fs.Usage = func() { _, _ = fmt.Fprint(a.stderr, usageText) } return fs } @@ -189,12 +209,12 @@ func parse(fs *flag.FlagSet, args []string) error { } func (a *app) fail(err error) int { - fmt.Fprintf(a.stderr, "backport-batch: %v\n", err) + _, _ = fmt.Fprintf(a.stderr, "backport-batch: %v\n", err) return 2 } func (a *app) warn(format string, args ...any) { - fmt.Fprintf(a.stderr, "warning: "+format+"\n", args...) + _, _ = fmt.Fprintf(a.stderr, "warning: "+format+"\n", args...) } // verdict is what the policy says about one original for the line. diff --git a/cmd/backport-batch/notify.go b/cmd/backport-batch/notify.go index fe301a2ff..3460e287d 100644 --- a/cmd/backport-batch/notify.go +++ b/cmd/backport-batch/notify.go @@ -135,11 +135,11 @@ func (a *app) notify(args []string) int { stalled := days >= stalledAfter var text []string if stalled { - fmt.Fprintf(a.stdout, "%s: batch pull request #%d has been open since %s, %d days\n", l.branch(), pr.Number, opened.Format(time.DateOnly), days) + _, _ = fmt.Fprintf(a.stdout, "%s: batch pull request #%d has been open since %s, %d days\n", l.branch(), pr.Number, opened.Format(time.DateOnly), days) text = append(text, fmt.Sprintf("This batch has been open since %s, %d days, through at least one scheduled run. Merge it, or close it and the next run starts a new one; until then each run adds what the rules take to it.", opened.Format(time.DateOnly), days)) } if len(partial) > 0 { - fmt.Fprintf(a.stdout, "%s: %d units on #%d carry only part of their originals\n", l.branch(), len(partial), pr.Number) + _, _ = fmt.Fprintf(a.stdout, "%s: %d units on #%d carry only part of their originals\n", l.branch(), len(partial), pr.Number) var b strings.Builder b.WriteString("These are on the batch but carry only part of their originals on this line; take each out with `backport-batch run --defer` before the cut, and port what applies as a pull request of its own:\n") for _, n := range partial { @@ -148,7 +148,7 @@ func (a *app) notify(args []string) int { text = append(text, b.String()) } if len(conflicts) > 0 { - fmt.Fprintf(a.stdout, "%s: %d deferred conflicts on #%d wait for a maintainer\n", l.branch(), count(conflicts), pr.Number) + _, _ = fmt.Fprintf(a.stdout, "%s: %d deferred conflicts on #%d wait for a maintainer\n", l.branch(), count(conflicts), pr.Number) var b strings.Builder fmt.Fprintf(&b, "These conflicts wait for a maintainer, as [resolving a conflict](%sresolve-conflict.md) describes; unresolved, each goes to the next batch, and so does what waits on one:\n", tasksURL) for _, n := range conflicts { @@ -158,9 +158,9 @@ func (a *app) notify(args []string) int { } if len(waiting) > 0 { if count(waiting) == len(waiting) { - fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s and wait for a maintainer\n", l.branch(), len(waiting), pr.Number, workflowsDir) + _, _ = fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s and wait for a maintainer\n", l.branch(), len(waiting), pr.Number, workflowsDir) } else { - fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s or wait on one that does\n", l.branch(), len(waiting), pr.Number, workflowsDir) + _, _ = fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s or wait on one that does\n", l.branch(), len(waiting), pr.Number, workflowsDir) } var b strings.Builder fmt.Fprintf(&b, "These change `%s`, which the batch's token may not push, and wait for a maintainer to bring them over, as [changes to workflows](%s#changes-to-workflows) describes, or wait on one that does:\n", workflowsDir, policyURL) @@ -170,7 +170,7 @@ func (a *app) notify(args []string) int { text = append(text, b.String()) } if len(parks) > 0 { - fmt.Fprintf(a.stdout, "%s: %d parked originals on #%d, named once\n", l.branch(), len(parks), pr.Number) + _, _ = fmt.Fprintf(a.stdout, "%s: %d parked originals on #%d, named once\n", l.branch(), len(parks), pr.Number) var b strings.Builder fmt.Fprintf(&b, "These cannot land on this line whole, so the batch parked them; each is named here once. A maintainer ports what applies as a pull request of its own, as [fixes that exist only on a release line](%s#fixes-that-exist-only-on-a-release-line) describes, or drops the original with `backport-batch run --drop`:\n", policyURL) for _, p := range parks { @@ -354,7 +354,7 @@ func (a *app) lines(args []string) int { return a.fail(err) } for _, l := range slices.Backward(lines) { - fmt.Fprintln(a.stdout, l.branch()) + _, _ = fmt.Fprintln(a.stdout, l.branch()) } return 0 } diff --git a/cmd/backport-batch/output_test.go b/cmd/backport-batch/output_test.go new file mode 100644 index 000000000..0362f44b7 --- /dev/null +++ b/cmd/backport-batch/output_test.go @@ -0,0 +1,56 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "errors" + "strings" + "testing" + "time" +) + +type failingWriter struct{} + +func (failingWriter) Write([]byte) (int, error) { return 0, errors.New("broken pipe") } + +// Output that never reached its reader fails the command, and stderr says why. +func TestUnwritableOutputFailsTheCommand(t *testing.T) { + var stderr strings.Builder + a := &app{gh: &fakeGH{}, stdout: failingWriter{}, stderr: &stderr} + if code := a.main([]string{"help"}); code != 2 { + t.Fatalf("help with an unwritable stdout = %d, want 2", code) + } + if !strings.Contains(stderr.String(), "broken pipe") { + t.Errorf("stderr does not name the write failure:\n%s", stderr.String()) + } +} + +// A command that already failed keeps its own exit code when its output is +// lost as well: check still says something is missing. +func TestUnwritableOutputKeepsTheCommandsFailure(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + var stderr strings.Builder + a := &app{gh: &fakeGH{remoteHead: f.remoteHead}, stdout: failingWriter{}, stderr: &stderr, now: func() time.Time { return f.now }, tool: f.tool} + if code := a.main([]string{"check", "--line", "release-1.1"}); code != 1 { + t.Fatalf("check with #6 missing and an unwritable stdout = %d, want 1\n%s", code, stderr.String()) + } + if !strings.Contains(stderr.String(), "broken pipe") { + t.Errorf("stderr does not name the write failure:\n%s", stderr.String()) + } +} diff --git a/cmd/backport-batch/report.go b/cmd/backport-batch/report.go index d1ae4a4b5..ee072e332 100644 --- a/cmd/backport-batch/report.go +++ b/cmd/backport-batch/report.go @@ -708,7 +708,7 @@ func (a *app) report(args []string) int { return a.fail(fmt.Errorf("batch pull request #%d keeps no report %.12s", pr.Number, *id)) } if *asJSON { - fmt.Fprintln(a.stdout, s.raw) + _, _ = fmt.Fprintln(a.stdout, s.raw) return 0 } a.printReport(&pr, s, inBody, inComment) @@ -721,29 +721,29 @@ func (a *app) report(args []string) int { func (a *app) printReport(pr *pullRequest, s storedReport, inBody *storedReport, inComment bool) { d, b := s.doc, s.doc.Built safe := terminalSafe - fmt.Fprintf(a.stdout, "report %s for %s, made %s by a %s run", safe(d.ID), safe(d.Line), safe(b.At), safe(b.Options.Mode)) + _, _ = fmt.Fprintf(a.stdout, "report %s for %s, made %s by a %s run", safe(d.ID), safe(d.Line), safe(b.At), safe(b.Options.Mode)) if b.Options.NoWorkflowFiles { - fmt.Fprint(a.stdout, " with --no-workflow-files") + _, _ = fmt.Fprint(a.stdout, " with --no-workflow-files") } - fmt.Fprintln(a.stdout) - fmt.Fprintf(a.stdout, " main %s, line %s, newer line %s, policy %s\n batch %s, tool %s, run %s\n", safe(b.Main), safe(b.Line), safe(or(b.Newer, "none")), safe(b.Policy), safe(b.Batch), safe(or(b.Tool, "unknown")), safe(or(b.Run, "none"))) + _, _ = fmt.Fprintln(a.stdout) + _, _ = fmt.Fprintf(a.stdout, " main %s, line %s, newer line %s, policy %s\n batch %s, tool %s, run %s\n", safe(b.Main), safe(b.Line), safe(or(b.Newer, "none")), safe(b.Policy), safe(b.Batch), safe(or(b.Tool, "unknown")), safe(or(b.Run, "none"))) current := "none" if inBody != nil { current = safe(inBody.doc.ID) } switch { case inBody != nil && inBody.doc.ID == d.ID: - fmt.Fprintf(a.stdout, " #%d's body carries this report\n", pr.Number) + _, _ = fmt.Fprintf(a.stdout, " #%d's body carries this report\n", pr.Number) case pr.Head.SHA == b.Batch: - fmt.Fprintf(a.stdout, " #%d's head is %.12s, the batch this report describes, but its body carries report %.12s\n", pr.Number, safe(b.Batch), current) + _, _ = fmt.Fprintf(a.stdout, " #%d's head is %.12s, the batch this report describes, but its body carries report %.12s\n", pr.Number, safe(b.Batch), current) default: - fmt.Fprintf(a.stdout, " publication unconfirmed: #%d's head is %.12s, not %.12s, and its body carries report %.12s\n", pr.Number, safe(pr.Head.SHA), safe(b.Batch), current) + _, _ = fmt.Fprintf(a.stdout, " publication unconfirmed: #%d's head is %.12s, not %.12s, and its body carries report %.12s\n", pr.Number, safe(pr.Head.SHA), safe(b.Batch), current) } if !inComment { if prState(pr) == "open" { - fmt.Fprintln(a.stdout, " comment not posted; the next backport-batch run --push posts it") + _, _ = fmt.Fprintln(a.stdout, " comment not posted; the next backport-batch run --push posts it") } else { - fmt.Fprintf(a.stdout, " comment not posted; kept in the body only, since nothing is written to a %s pull request\n", prState(pr)) + _, _ = fmt.Fprintf(a.stdout, " comment not posted; kept in the body only, since nothing is written to a %s pull request\n", prState(pr)) } } type line struct { @@ -802,22 +802,22 @@ func (a *app) printReport(pr *pullRequest, s storedReport, inBody *storedReport, } slices.SortFunc(lines, func(x, y line) int { return cmp.Compare(x.n, y.n) }) for _, l := range lines { - fmt.Fprintf(a.stdout, "#%d %s\n", l.n, terminalSafe(l.text)) + _, _ = fmt.Fprintf(a.stdout, "#%d %s\n", l.n, terminalSafe(l.text)) } overlay := func(name string, list []int) { if len(list) > 0 { - fmt.Fprintf(a.stdout, "%s: %s\n", name, strings.Trim(fmt.Sprint(list), "[]")) + _, _ = fmt.Fprintf(a.stdout, "%s: %s\n", name, strings.Trim(fmt.Sprint(list), "[]")) } } overlay("stale, still on the batch", d.Overlays.Stale) overlay("only part of it applies, still on the batch", d.Overlays.Partial) for _, e := range d.Overlays.Dropped { - fmt.Fprintf(a.stdout, "dropped: #%d, merged as %.12s, on #%d: %s\n", e.Original, terminalSafe(e.Merge), e.On, terminalSafe(e.Reason)) + _, _ = fmt.Fprintf(a.stdout, "dropped: #%d, merged as %.12s, on #%d: %s\n", e.Original, terminalSafe(e.Merge), e.On, terminalSafe(e.Reason)) } overlay("changed in the pick", d.Overlays.Changed) for _, n := range slices.Sorted(maps.Keys(d.Diag)) { for _, ln := range d.Diag[n] { - fmt.Fprintf(a.stdout, "#%s: %s\n", terminalSafe(n), terminalSafe(ln)) + _, _ = fmt.Fprintf(a.stdout, "#%s: %s\n", terminalSafe(n), terminalSafe(ln)) } } if len(d.Other) > 0 { @@ -825,7 +825,7 @@ func (a *app) printReport(pr *pullRequest, s storedReport, inBody *storedReport, for i, sha := range d.Other { other[i] = safe(sha) } - fmt.Fprintf(a.stdout, "commits on main that merged no pull request: %s\n", strings.Join(other, ", ")) + _, _ = fmt.Fprintf(a.stdout, "commits on main that merged no pull request: %s\n", strings.Join(other, ", ")) } } diff --git a/cmd/backport-batch/verify.go b/cmd/backport-batch/verify.go index 0691df641..03b0e8955 100644 --- a/cmd/backport-batch/verify.go +++ b/cmd/backport-batch/verify.go @@ -66,7 +66,7 @@ func (a *app) verify(args []string) int { return a.fail(err) } if pr == nil { - fmt.Fprintf(a.stdout, "%s: no batch pull request is open; nothing to verify\n", l.branch()) + _, _ = fmt.Fprintf(a.stdout, "%s: no batch pull request is open; nothing to verify\n", l.branch()) return 0 } if head, err = batchHead(pr); err != nil { @@ -102,7 +102,7 @@ func (a *app) verify(args []string) int { } if why != "" { refused = append(refused, why) - fmt.Fprintf(a.stdout, "refused %.12s: %s\n", u.commit, why) + _, _ = fmt.Fprintf(a.stdout, "refused %.12s: %s\n", u.commit, why) continue } seen[v] = u.commit @@ -110,13 +110,13 @@ func (a *app) verify(args []string) int { if !u.merge() { n = 1 } - fmt.Fprintf(a.stdout, "ok %.12s #%d, %d %s\n", u.commit, v.o.number, n, plural(n, "commit")) + _, _ = fmt.Fprintf(a.stdout, "ok %.12s #%d, %d %s\n", u.commit, v.o.number, n, plural(n, "commit")) } if len(refused) > 0 { - fmt.Fprintf(a.stdout, "%s: %d of %d units at %.12s are not originals in the batch's shape\n", l.branch(), len(refused), len(units), head) + _, _ = fmt.Fprintf(a.stdout, "%s: %d of %d units at %.12s are not originals in the batch's shape\n", l.branch(), len(refused), len(units), head) return 1 } - fmt.Fprintf(a.stdout, "%s: %d %s at %.12s, each one original in the batch's shape\n", l.branch(), len(units), plural(len(units), "unit"), head) + _, _ = fmt.Fprintf(a.stdout, "%s: %d %s at %.12s, each one original in the batch's shape\n", l.branch(), len(units), plural(len(units), "unit"), head) return 0 } From 20ce6c691e50f03e30b31c012d3ba3dd63c62fa8 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 00:14:38 +0300 Subject: [PATCH 488/518] fix(sse): close a client's channel once when the server shuts down Shutdown closed every open client's channel, and the client's handler closed the same channel again when its request ended, so any stream still open at shutdown panicked with a close of a closed channel. Stopping a client is now idempotent. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- internal/sse/server.go | 11 ++++- internal/sse/server_test.go | 98 +++++++++++++++++++++++++++++++++++++ 2 files changed, 107 insertions(+), 2 deletions(-) create mode 100644 internal/sse/server_test.go diff --git a/internal/sse/server.go b/internal/sse/server.go index 0c55032ec..f65817fb1 100644 --- a/internal/sse/server.go +++ b/internal/sse/server.go @@ -60,6 +60,7 @@ type Server struct { type client struct { ch chan string closeCh chan struct{} + closeOnce sync.Once flusher http.Flusher w http.ResponseWriter req *http.Request @@ -122,7 +123,7 @@ func (s *Server) ListenAndServe() error { func (s *Server) Shutdown(ctx context.Context) error { s.clientsMu.Lock() for c := range s.clients { - close(c.closeCh) + c.stop() } s.clientsMu.Unlock() return s.http.Shutdown(ctx) @@ -206,12 +207,18 @@ func (s *Server) handleStream(w http.ResponseWriter, r *http.Request) { <-r.Context().Done() // Unregister client - close(c.closeCh) + c.stop() s.clientsMu.Lock() delete(s.clients, c) s.clientsMu.Unlock() } +// stop ends the client's pump. Shutdown and the client's own handler both +// stop it, in either order. +func (c *client) stop() { + c.closeOnce.Do(func() { close(c.closeCh) }) +} + func (c *client) pump() { t := time.NewTicker(c.heartbeat) defer t.Stop() diff --git a/internal/sse/server_test.go b/internal/sse/server_test.go new file mode 100644 index 000000000..be7c21e34 --- /dev/null +++ b/internal/sse/server_test.go @@ -0,0 +1,98 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package sse + +import ( + "context" + "errors" + "io" + "log" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" + "time" +) + +// streamWriter accepts the first okWrites writes and fails every one after. +type streamWriter struct { + mu sync.Mutex + header http.Header + okWrites int + body strings.Builder +} + +func (w *streamWriter) Header() http.Header { return w.header } +func (w *streamWriter) WriteHeader(int) {} +func (w *streamWriter) Flush() {} + +func (w *streamWriter) Write(p []byte) (int, error) { + w.mu.Lock() + defer w.mu.Unlock() + if w.okWrites == 0 { + return 0, errors.New("connection reset") + } + w.okWrites-- + return w.body.Write(p) +} + +func (w *streamWriter) String() string { + w.mu.Lock() + defer w.mu.Unlock() + return w.body.String() +} + +func (s *Server) snapshotClients() []*client { + s.clientsMu.RLock() + defer s.clientsMu.RUnlock() + var out []*client + for c := range s.clients { + out = append(out, c) + } + return out +} + +func newTestServer(heartbeat time.Duration) *Server { + return New(Options{Heartbeat: heartbeat, Logger: log.New(io.Discard, "", 0)}) +} + +// A stream still open when the server shuts down ends when its client +// disconnects, without closing the client a second time. +func TestShutdownThenDisconnect(t *testing.T) { + s := newTestServer(time.Hour) + ctx, cancel := context.WithCancel(context.Background()) + r := httptest.NewRequest(http.MethodGet, "/stream", nil).WithContext(ctx) + done := make(chan struct{}) + go func() { + s.handleStream(&streamWriter{header: http.Header{}, okWrites: 100}, r) + close(done) + }() + deadline := time.Now().Add(time.Second) + for len(s.snapshotClients()) == 0 && time.Now().Before(deadline) { + time.Sleep(time.Millisecond) + } + if err := s.Shutdown(context.Background()); err != nil { + t.Fatal(err) + } + cancel() + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("the stream did not end after its client disconnected") + } +} From 41ad82842a395b238d98889f76378b7e611b1fe2 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 00:15:00 +0300 Subject: [PATCH 489/518] fix(sse): drop a client as soon as a write to it fails Write errors on the event stream were ignored, so a client whose connection broke kept its handler, pump and registration until the request context ended, and every event was written into the dead connection meanwhile. A failed write now ends the stream and unregisters the client, and the handler also returns once its pump ends, so Shutdown no longer leaves streams open. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- internal/sse/server.go | 52 +++++++++++++------- internal/sse/server_test.go | 97 ++++++++++++++++++++++++++++++++++--- 2 files changed, 126 insertions(+), 23 deletions(-) diff --git a/internal/sse/server.go b/internal/sse/server.go index f65817fb1..166b01f49 100644 --- a/internal/sse/server.go +++ b/internal/sse/server.go @@ -187,8 +187,16 @@ func (s *Server) handleStream(w http.ResponseWriter, r *http.Request) { s.clients[c] = struct{}{} s.clientsMu.Unlock() + defer func() { + s.clientsMu.Lock() + delete(s.clients, c) + s.clientsMu.Unlock() + }() + // Initial comment to open the stream for some proxies - fmt.Fprintf(w, ": connected %s\n\n", time.Now().Format(time.RFC3339)) + if _, err := fmt.Fprintf(w, ": connected %s\n\n", time.Now().Format(time.RFC3339)); err != nil { + return + } flusher.Flush() // Send latest if any @@ -196,21 +204,24 @@ func (s *Server) handleStream(w http.ResponseWriter, r *http.Request) { latest := s.latest s.latestMu.RUnlock() if latest != "" { - writeSSE(w, latest) + if err := writeSSE(w, latest); err != nil { + return + } flusher.Flush() } - // Start pump - go c.pump() - - // Block until client disconnects - <-r.Context().Done() - - // Unregister client + // A write the client no longer takes ends the pump; the handler then + // gives the client up without waiting for the request to end. + pumped := make(chan struct{}) + go func() { + c.pump() + close(pumped) + }() + select { + case <-r.Context().Done(): + case <-pumped: + } c.stop() - s.clientsMu.Lock() - delete(s.clients, c) - s.clientsMu.Unlock() } // stop ends the client's pump. Shutdown and the client's own handler both @@ -227,23 +238,30 @@ func (c *client) pump() { case <-c.closeCh: return case msg := <-c.ch: - writeSSE(c.w, msg) + if err := writeSSE(c.w, msg); err != nil { + return + } c.flusher.Flush() case <-t.C: // heartbeat comment (keeps connections alive through proxies) - fmt.Fprint(c.w, ": hb\n\n") + if _, err := fmt.Fprint(c.w, ": hb\n\n"); err != nil { + return + } c.flusher.Flush() } } } -func writeSSE(w http.ResponseWriter, msg string) { +func writeSSE(w http.ResponseWriter, msg string) error { // Split on lines; each needs its own "data:" field per the SSE spec lines := strings.SplitSeq(strings.TrimRight(msg, "\n"), "\n") for ln := range lines { - fmt.Fprintf(w, "data: %s\n", ln) + if _, err := fmt.Fprintf(w, "data: %s\n", ln); err != nil { + return err + } } - fmt.Fprint(w, "\n") + _, err := fmt.Fprint(w, "\n") + return err } // Minimal index page with live updates diff --git a/internal/sse/server_test.go b/internal/sse/server_test.go index be7c21e34..5ae5ad966 100644 --- a/internal/sse/server_test.go +++ b/internal/sse/server_test.go @@ -57,6 +57,37 @@ func (w *streamWriter) String() string { return w.body.String() } +// serve runs the stream handler for a client that never disconnects and +// reports whether the handler gave the client up within a second. +func serve(t *testing.T, s *Server, w *streamWriter, publish string) bool { + t.Helper() + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + r := httptest.NewRequest(http.MethodGet, "/stream", nil).WithContext(ctx) + done := make(chan struct{}) + go func() { + s.handleStream(w, r) + close(done) + }() + if publish != "" { + deadline := time.Now().Add(time.Second) + for len(s.snapshotClients()) == 0 && time.Now().Before(deadline) { + time.Sleep(time.Millisecond) + } + // Onto the client's queue, which only the pump reads, so the failing + // write is the pump's and not the handler's write of latest. + for _, c := range s.snapshotClients() { + c.ch <- publish + } + } + select { + case <-done: + return len(s.snapshotClients()) == 0 + case <-time.After(time.Second): + return false + } +} + func (s *Server) snapshotClients() []*client { s.clientsMu.RLock() defer s.clientsMu.RUnlock() @@ -71,15 +102,70 @@ func newTestServer(heartbeat time.Duration) *Server { return New(Options{Heartbeat: heartbeat, Logger: log.New(io.Discard, "", 0)}) } -// A stream still open when the server shuts down ends when its client -// disconnects, without closing the client a second time. -func TestShutdownThenDisconnect(t *testing.T) { +// A client whose connection stops taking writes is dropped at once, at +// whichever write first fails, instead of holding its handler until the +// request context ends. +func TestStreamDropsClientOnWriteFailure(t *testing.T) { + for _, tc := range []struct { + name string + okWrites int + latest string + publish string + heartbeat time.Duration + }{ + {name: "connect comment", okWrites: 0, heartbeat: time.Hour}, + {name: "latest payload", okWrites: 1, latest: "state", heartbeat: time.Hour}, + {name: "published event", okWrites: 1, publish: "event", heartbeat: time.Hour}, + {name: "heartbeat", okWrites: 1, heartbeat: 10 * time.Millisecond}, + } { + t.Run(tc.name, func(t *testing.T) { + s := newTestServer(tc.heartbeat) + if tc.latest != "" { + s.Publish(tc.latest) + } + if !serve(t, s, &streamWriter{header: http.Header{}, okWrites: tc.okWrites}, tc.publish) { + t.Fatal("the handler kept a client whose writes fail") + } + }) + } +} + +// Each line of a payload is its own data field, and the event ends with an +// empty line. +func TestStreamFramesMultilinePayload(t *testing.T) { s := newTestServer(time.Hour) + s.Publish("one\ntwo\n") + w := &streamWriter{header: http.Header{}, okWrites: 100} ctx, cancel := context.WithCancel(context.Background()) r := httptest.NewRequest(http.MethodGet, "/stream", nil).WithContext(ctx) done := make(chan struct{}) go func() { - s.handleStream(&streamWriter{header: http.Header{}, okWrites: 100}, r) + s.handleStream(w, r) + close(done) + }() + deadline := time.Now().Add(time.Second) + for !strings.Contains(w.String(), "data: two\n\n") && time.Now().Before(deadline) { + time.Sleep(time.Millisecond) + } + cancel() + <-done + if got := w.String(); !strings.Contains(got, "data: one\ndata: two\n\n") { + t.Fatalf("stream = %q, want both lines framed as one event", got) + } + if len(s.snapshotClients()) != 0 { + t.Fatal("a disconnected client stayed registered") + } +} + +// Shutdown ends every open stream, and the handler that then returns does +// not close the client a second time. +func TestShutdownEndsOpenStreams(t *testing.T) { + s := newTestServer(time.Hour) + w := &streamWriter{header: http.Header{}, okWrites: 100} + r := httptest.NewRequest(http.MethodGet, "/stream", nil) + done := make(chan struct{}) + go func() { + s.handleStream(w, r) close(done) }() deadline := time.Now().Add(time.Second) @@ -89,10 +175,9 @@ func TestShutdownThenDisconnect(t *testing.T) { if err := s.Shutdown(context.Background()); err != nil { t.Fatal(err) } - cancel() select { case <-done: case <-time.After(time.Second): - t.Fatal("the stream did not end after its client disconnected") + t.Fatal("Shutdown left the stream open") } } From bd998d9653e1f1b4ab7045599044ccaa744314a8 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:46:11 +0300 Subject: [PATCH 490/518] fix(kube-ovn): fail the plunger when its status server cannot start The status stream was started in a bare goroutine whose error was dropped, and the runnable's registration error was discarded too, so a port that could not be bound left the plunger running with no status endpoint and nothing in its logs. The server now runs inside its runnable: a listen error fails the manager, as a metrics server bind failure does, and stopping the manager is not reported as an error. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .../kubeovnplunger/kubeovn_plunger.go | 30 ++++++++--- .../kubeovnplunger/status_server_test.go | 51 +++++++++++++++++++ 2 files changed, 75 insertions(+), 6 deletions(-) create mode 100644 internal/controller/kubeovnplunger/status_server_test.go diff --git a/internal/controller/kubeovnplunger/kubeovn_plunger.go b/internal/controller/kubeovnplunger/kubeovn_plunger.go index a45fba073..30aa42c00 100644 --- a/internal/controller/kubeovnplunger/kubeovn_plunger.go +++ b/internal/controller/kubeovnplunger/kubeovn_plunger.go @@ -3,8 +3,10 @@ package kubeovnplunger import ( "bytes" "context" + "errors" "fmt" "io" + "net/http" "strings" "time" @@ -153,12 +155,9 @@ func (r *KubeOVNPlunger) SetupWithManager(mgr ctrl.Manager, kubeOVNNamespace, ap Complete(r); err != nil { return err } - _ = mgr.Add(manager.RunnableFunc(func(ctx context.Context) error { - go srv.ListenAndServe() - <-ctx.Done() - _ = srv.Shutdown(context.Background()) - return nil - })) + if err := mgr.Add(serveStatus(srv)); err != nil { + return err + } return mgr.Add(manager.RunnableFunc(func(ctx context.Context) error { ticker := time.NewTicker(rescanInterval) defer ticker.Stop() @@ -180,6 +179,25 @@ func (r *KubeOVNPlunger) SetupWithManager(mgr ctrl.Manager, kubeOVNNamespace, ap })) } +// serveStatus runs the status stream for as long as the manager does. A server +// that cannot start fails the manager, as the metrics server does. +func serveStatus(s *sse.Server) manager.RunnableFunc { + return func(ctx context.Context) error { + served := make(chan error, 1) + go func() { served <- s.ListenAndServe() }() + select { + case err := <-served: + return err + case <-ctx.Done(): + } + _ = s.Shutdown(context.Background()) + if err := <-served; !errors.Is(err, http.ErrServerClosed) { + return err + } + return nil + } +} + type ExecOptions struct { Namespace string Pod string diff --git a/internal/controller/kubeovnplunger/status_server_test.go b/internal/controller/kubeovnplunger/status_server_test.go new file mode 100644 index 000000000..c894ecf74 --- /dev/null +++ b/internal/controller/kubeovnplunger/status_server_test.go @@ -0,0 +1,51 @@ +package kubeovnplunger + +import ( + "context" + "io" + "log" + "net" + "testing" + "time" + + "github.com/cozystack/cozystack/internal/sse" +) + +func runStatusServer(ctx context.Context, t *testing.T, addr string) (bool, error) { + t.Helper() + ctx, cancel := context.WithCancel(ctx) + t.Cleanup(cancel) + s := sse.New(sse.Options{Addr: addr, Logger: log.New(io.Discard, "", 0)}) + done := make(chan error, 1) + go func() { done <- serveStatus(s)(ctx) }() + select { + case err := <-done: + return true, err + case <-time.After(2 * time.Second): + return false, nil + } +} + +// A status server that cannot listen fails its runnable, and with it the +// manager, rather than leaving the plunger running without it. +func TestStatusServerReportsListenFailure(t *testing.T) { + taken, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = taken.Close() }) + returned, err := runStatusServer(context.Background(), t, taken.Addr().String()) + if !returned || err == nil { + t.Fatalf("serveStatus on an occupied address = %v (returned %v), want an error", err, returned) + } +} + +// Stopping the manager stops the server, and that is not an error. +func TestStatusServerStopsWithoutError(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + time.AfterFunc(100*time.Millisecond, cancel) + returned, err := runStatusServer(ctx, t, "127.0.0.1:0") + if !returned || err != nil { + t.Fatalf("serveStatus after cancel = %v (returned %v), want nil", err, returned) + } +} From 87b794a9925e2f9c4813f5da6425f8344f408c03 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:53:11 +0300 Subject: [PATCH 491/518] chore: handle errors dropped in cleanup paths and tests The drain tests now assert that RunThenDrain returns the run's nil error, the idle-tail test fails on any error other than the stream ending or being cut, and a test fixture file fails the test when it cannot be closed. Removing a temporary directory and closing a response body that has been read are best-effort: a failure leaves nothing for the caller to act on and does not change any result, so those errors are discarded explicitly. The same holds for the stub servers' response writes, whose failure surfaces on the client side of the test. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .../cnpg_artifact_size_test.go | 4 ++-- .../mariadb_artifact_size_test.go | 2 +- .../fluxplunger/drain_outcome_test.go | 18 ++++++++++++------ internal/crdinstall/install.go | 2 +- internal/fluxinstall/install.go | 2 +- internal/observabilityproxy/server_test.go | 8 +++++++- internal/telemetry/collector.go | 2 +- internal/telemetry/operator_collector.go | 2 +- pkg/cmd/server/watch_initial_events_test.go | 2 +- pkg/reservation/instancetype_test.go | 4 +++- 10 files changed, 30 insertions(+), 16 deletions(-) diff --git a/internal/backupcontroller/cnpg_artifact_size_test.go b/internal/backupcontroller/cnpg_artifact_size_test.go index bdf9c5d7b..4afef7194 100644 --- a/internal/backupcontroller/cnpg_artifact_size_test.go +++ b/internal/backupcontroller/cnpg_artifact_size_test.go @@ -222,13 +222,13 @@ func cnpgListStub(prefix string, seen func(*http.Request)) http.HandlerFunc { return func(w http.ResponseWriter, req *http.Request) { w.Header().Set("Content-Type", "application/xml") if _, ok := req.URL.Query()["location"]; ok { - fmt.Fprint(w, ``) + _, _ = fmt.Fprint(w, ``) return } if seen != nil { seen(req) } - fmt.Fprintf(w, ` + _, _ = fmt.Fprintf(w, ` bkt%[1]s21000false %[1]sbackup.info7"a"2026-10-21T10:01:00.000Z diff --git a/internal/backupcontroller/mariadb_artifact_size_test.go b/internal/backupcontroller/mariadb_artifact_size_test.go index 9d649c15c..ed4a2b7bc 100644 --- a/internal/backupcontroller/mariadb_artifact_size_test.go +++ b/internal/backupcontroller/mariadb_artifact_size_test.go @@ -464,7 +464,7 @@ func TestMariaDBArtifactSizeBytes_MinioClient(t *testing.T) { gotPath, gotPrefix, gotAuth = req.URL.Path, req.URL.Query().Get("prefix"), req.Header.Get("Authorization") gotDelimiter = req.URL.Query().Get("delimiter") w.Header().Set("Content-Type", "application/xml") - fmt.Fprintf(w, ` + _, _ = fmt.Fprintf(w, ` bkttenant/db/11000false %s31337"x"2026-10-21T10:01:00.000Z diff --git a/internal/controller/fluxplunger/drain_outcome_test.go b/internal/controller/fluxplunger/drain_outcome_test.go index cb74e0177..152091dec 100644 --- a/internal/controller/fluxplunger/drain_outcome_test.go +++ b/internal/controller/fluxplunger/drain_outcome_test.go @@ -65,12 +65,14 @@ func (s captureSink) WithName(string) logr.LogSink { return s } func TestRunThenDrain_FencesLeftBehindAreAnErrorNamingThem(t *testing.T) { logger, entries := newCaptureLogger() ctx := log.IntoContext(context.Background(), logger) - RunThenDrain(ctx, + if err := RunThenDrain(ctx, func(context.Context) error { return nil }, func(context.Context) (int, error) { return 1, errors.Join(context.DeadlineExceeded, errors.New("still fenced: tenant-root/vm-disk-0")) }, - time.Minute) + time.Minute); err != nil { + t.Fatalf("RunThenDrain = %v, want the run's nil error", err) + } for _, e := range *entries { if e.err != nil && strings.Contains(e.err.Error(), "tenant-root/vm-disk-0") { return @@ -84,10 +86,12 @@ func TestRunThenDrain_FencesLeftBehindAreAnErrorNamingThem(t *testing.T) { func TestRunThenDrain_AnUnansweredListingIsNotACountOfMinusOne(t *testing.T) { logger, entries := newCaptureLogger() ctx := log.IntoContext(context.Background(), logger) - RunThenDrain(ctx, + if err := RunThenDrain(ctx, func(context.Context) error { return nil }, func(context.Context) (int, error) { return -1, context.DeadlineExceeded }, - time.Minute) + time.Minute); err != nil { + t.Fatalf("RunThenDrain = %v, want the run's nil error", err) + } found := false for _, e := range *entries { for i := 0; i+1 < len(e.kv); i += 2 { @@ -108,10 +112,12 @@ func TestRunThenDrain_AnUnansweredListingIsNotACountOfMinusOne(t *testing.T) { func TestRunThenDrain_ACompleteDrainSaysSo(t *testing.T) { logger, entries := newCaptureLogger() ctx := log.IntoContext(context.Background(), logger) - RunThenDrain(ctx, + if err := RunThenDrain(ctx, func(context.Context) error { return nil }, func(context.Context) (int, error) { return 0, nil }, - time.Minute) + time.Minute); err != nil { + t.Fatalf("RunThenDrain = %v, want the run's nil error", err) + } for _, e := range *entries { if e.err == nil && strings.Contains(e.msg, "Released every uninstall fence") { return diff --git a/internal/crdinstall/install.go b/internal/crdinstall/install.go index 5760beb6b..12706060b 100644 --- a/internal/crdinstall/install.go +++ b/internal/crdinstall/install.go @@ -40,7 +40,7 @@ func Install(ctx context.Context, k8sClient client.Client, writeEmbeddedManifest if err != nil { return fmt.Errorf("failed to create temp directory: %w", err) } - defer os.RemoveAll(tmpDir) + defer func() { _ = os.RemoveAll(tmpDir) }() manifestsDir := filepath.Join(tmpDir, "manifests") if err := os.MkdirAll(manifestsDir, 0755); err != nil { diff --git a/internal/fluxinstall/install.go b/internal/fluxinstall/install.go index 042bdfd5e..63c977803 100644 --- a/internal/fluxinstall/install.go +++ b/internal/fluxinstall/install.go @@ -41,7 +41,7 @@ func Install(ctx context.Context, k8sClient client.Client, writeEmbeddedManifest if err != nil { return fmt.Errorf("failed to create temp directory: %w", err) } - defer os.RemoveAll(tmpDir) + defer func() { _ = os.RemoveAll(tmpDir) }() // Extract embedded manifests (generated by cozypkg) manifestsDir := filepath.Join(tmpDir, "manifests") diff --git a/internal/observabilityproxy/server_test.go b/internal/observabilityproxy/server_test.go index df3e23bc4..2117b1551 100644 --- a/internal/observabilityproxy/server_test.go +++ b/internal/observabilityproxy/server_test.go @@ -18,6 +18,7 @@ package observabilityproxy import ( "bufio" + "errors" "io" "net/http" "net/http/httptest" @@ -413,7 +414,12 @@ func TestServerTailCapsAndIdle(t *testing.T) { closed <- err }() select { - case <-closed: + case err := <-closed: + // The server cuts an idle tail off mid-stream rather than ending the + // response, so the client reads an unexpected EOF; a clean end is fine too. + if err != nil && !errors.Is(err, io.ErrUnexpectedEOF) { + t.Fatalf("idle tail ended with %v, want the stream to end or be cut", err) + } case <-time.After(3 * time.Second): t.Fatal("idle tail not closed") } diff --git a/internal/telemetry/collector.go b/internal/telemetry/collector.go index a948c7495..636bbc85a 100644 --- a/internal/telemetry/collector.go +++ b/internal/telemetry/collector.go @@ -147,7 +147,7 @@ func (c *Collector) sendMetrics(clusterID, metrics string) error { if err != nil { return fmt.Errorf("failed to send request: %w", err) } - defer resp.Body.Close() + defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { return fmt.Errorf("unexpected status code: %d", resp.StatusCode) diff --git a/internal/telemetry/operator_collector.go b/internal/telemetry/operator_collector.go index 2c0e1b9bb..9f2f9fcbe 100644 --- a/internal/telemetry/operator_collector.go +++ b/internal/telemetry/operator_collector.go @@ -272,7 +272,7 @@ func (c *OperatorCollector) sendMetrics(clusterID, metrics string) error { if err != nil { return fmt.Errorf("failed to send request: %w", err) } - defer resp.Body.Close() + defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { return fmt.Errorf("unexpected status code: %d", resp.StatusCode) diff --git a/pkg/cmd/server/watch_initial_events_test.go b/pkg/cmd/server/watch_initial_events_test.go index 331417545..75d43c3e1 100644 --- a/pkg/cmd/server/watch_initial_events_test.go +++ b/pkg/cmd/server/watch_initial_events_test.go @@ -104,7 +104,7 @@ func watchStream(t *testing.T, ts *httptest.Server, cli client.Client, query str if err != nil { t.Fatalf("watch: %v", err) } - defer resp.Body.Close() + defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { t.Fatalf("watch: status %d", resp.StatusCode) } diff --git a/pkg/reservation/instancetype_test.go b/pkg/reservation/instancetype_test.go index 0d072b548..f837af8e3 100644 --- a/pkg/reservation/instancetype_test.go +++ b/pkg/reservation/instancetype_test.go @@ -56,7 +56,9 @@ func shippedInstancetypes(t *testing.T) []map[string]any { out = append(out, obj) } } - fh.Close() + if err := fh.Close(); err != nil { + t.Fatalf("%s: %v", f, err) + } } return out } From 7a481c47656c041ef2369ef1ca798367c0713629 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:53:12 +0300 Subject: [PATCH 492/518] chore(workloadmonitor): discard best-effort close errors explicitly The Prometheus response body is closed after it has been read, and the test stubs' writes fail on the client side of the test, so neither error changes a result. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- internal/controller/workloadmonitor_controller.go | 2 +- internal/controller/workloadmonitor_controller_test.go | 10 +++++----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/internal/controller/workloadmonitor_controller.go b/internal/controller/workloadmonitor_controller.go index 217fd0bbf..b8b0bac13 100644 --- a/internal/controller/workloadmonitor_controller.go +++ b/internal/controller/workloadmonitor_controller.go @@ -525,7 +525,7 @@ func (r *WorkloadMonitorReconciler) queryAllBucketMetrics(ctx context.Context, p if err != nil { return nil, fmt.Errorf("querying Prometheus: %w", err) } - defer resp.Body.Close() + defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { return nil, fmt.Errorf("unexpected status %d from Prometheus", resp.StatusCode) diff --git a/internal/controller/workloadmonitor_controller_test.go b/internal/controller/workloadmonitor_controller_test.go index 353066c4b..87f9dd269 100644 --- a/internal/controller/workloadmonitor_controller_test.go +++ b/internal/controller/workloadmonitor_controller_test.go @@ -870,7 +870,7 @@ func TestReconcileNoBucketClaimSkips(t *testing.T) { func TestQueryAllBucketMetrics(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - fmt.Fprint(w, `{"status":"success","data":{"resultType":"vector","result":[ + _, _ = fmt.Fprint(w, `{"status":"success","data":{"resultType":"vector","result":[ {"metric":{"__name__":"SeaweedFS_s3_bucket_size_bytes","bucket":"bucket-aaa"},"value":[1713000000,"10485864"]}, {"metric":{"__name__":"SeaweedFS_s3_bucket_physical_size_bytes","bucket":"bucket-aaa"},"value":[1713000000,"20971728"]}, {"metric":{"__name__":"SeaweedFS_s3_bucket_size_bytes","bucket":"bucket-bbb"},"value":[1713000000,"0"]} @@ -909,7 +909,7 @@ func TestQueryAllBucketMetrics(t *testing.T) { func TestQueryAllBucketMetricsEmpty(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - fmt.Fprint(w, `{"status":"success","data":{"resultType":"vector","result":[]}}`) + _, _ = fmt.Fprint(w, `{"status":"success","data":{"resultType":"vector","result":[]}}`) })) defer srv.Close() @@ -953,7 +953,7 @@ func TestQueryAllBucketMetricsPathPrefixAndUserinfo(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotPath = r.URL.Path gotUser, gotPass, gotOK = r.BasicAuth() - fmt.Fprint(w, `{"status":"success","data":{"resultType":"vector","result":[]}}`) + _, _ = fmt.Fprint(w, `{"status":"success","data":{"resultType":"vector","result":[]}}`) })) defer srv.Close() @@ -1219,7 +1219,7 @@ func TestReconcileBucketMetricsFromEndpointOverride(t *testing.T) { if r.URL.Path != "/vm/select/0/prometheus/api/v1/query" { t.Errorf("unexpected query path %q", r.URL.Path) } - fmt.Fprint(w, `{"status":"success","data":{"resultType":"vector","result":[ + _, _ = fmt.Fprint(w, `{"status":"success","data":{"resultType":"vector","result":[ {"metric":{"__name__":"SeaweedFS_s3_bucket_size_bytes","bucket":"cosi-abc123"},"value":[1713000000,"4096"]} ]}}`) })) @@ -1331,7 +1331,7 @@ func TestReconcileRetainsLastKnownSizesOnQueryFailure(t *testing.T) { func TestReconcileRetainsLastKnownSizesWhenBucketMissingFromResult(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - fmt.Fprint(w, `{"status":"success","data":{"resultType":"vector","result":[]}}`) + _, _ = fmt.Fprint(w, `{"status":"success","data":{"resultType":"vector","result":[]}}`) })) defer srv.Close() From 50959ccca20fa212c836cc2ccccf7583f1bc279a Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Thu, 12 Nov 2026 23:53:20 +0300 Subject: [PATCH 493/518] chore(backport): discard temp-file cleanup errors explicitly A temporary file or directory that cannot be removed, or closed after its write already failed, leaves nothing to act on: the run's result is the same, and the primary error is the one returned. The test that unsets an environment variable now fails if it cannot. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- cmd/backport-batch/apisurface.go | 2 +- cmd/backport-batch/assemble.go | 4 ++-- cmd/backport-batch/effects.go | 2 +- cmd/backport-batch/notify.go | 6 +++--- cmd/backport-batch/report.go | 8 ++++---- cmd/backport-batch/report_test.go | 4 +++- cmd/backport-batch/shape.go | 4 ++-- 7 files changed, 16 insertions(+), 14 deletions(-) diff --git a/cmd/backport-batch/apisurface.go b/cmd/backport-batch/apisurface.go index 706e4be62..6966bce74 100644 --- a/cmd/backport-batch/apisurface.go +++ b/cmd/backport-batch/apisurface.go @@ -99,7 +99,7 @@ func apiSnapshot(commit string) (apigate.Snapshot, error) { if err != nil { return nil, err } - defer os.RemoveAll(dir) + defer func() { _ = os.RemoveAll(dir) }() for name, blob := range paths { file := filepath.Join(dir, filepath.FromSlash(name)) if err := os.MkdirAll(filepath.Dir(file), 0o700); err != nil { diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index a0aab98db..2ebca3483 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -727,7 +727,7 @@ func (a *app) publish(b *batch, p publication, rebuilt bool) error { if err != nil { return err } - defer os.Remove(name) + defer func() { _ = os.Remove(name) }() if b.pr == nil { n, url, err := a.gh.createPull(b.line.branch(), p.branch, p.title, name) if err != nil { @@ -828,7 +828,7 @@ func (a *app) closeBatch(b *batch, p publication) error { if err != nil { return err } - defer os.Remove(name) + defer func() { _ = os.Remove(name) }() if err := a.stillOpen(c.PR); err != nil { return err } diff --git a/cmd/backport-batch/effects.go b/cmd/backport-batch/effects.go index 350f9a961..6388c42ae 100644 --- a/cmd/backport-batch/effects.go +++ b/cmd/backport-batch/effects.go @@ -367,7 +367,7 @@ func (b *batch) alreadyApplied(pre, cur, post string) (applied, unsupported bool if err != nil { return false, false, err } - defer os.RemoveAll(dir) + defer func() { _ = os.RemoveAll(dir) }() names := map[string]string{} for _, f := range []struct{ name, oid string }{{"cur", cur}, {"pre", pre}, {"post", post}} { names[f.name] = filepath.Join(dir, f.name) diff --git a/cmd/backport-batch/notify.go b/cmd/backport-batch/notify.go index 3460e287d..e9ee23234 100644 --- a/cmd/backport-batch/notify.go +++ b/cmd/backport-batch/notify.go @@ -229,7 +229,7 @@ func (a *app) announced(pr *pullRequest, rec *lineRecord, parks []parkedEntry) e if err != nil { return err } - defer os.Remove(name) + defer func() { _ = os.Remove(name) }() return a.gh.editBody(pr.Number, name) } @@ -309,9 +309,9 @@ func (a *app) comment(pr *pullRequest, mention string, today time.Time, text []s if err != nil { return a.fail(err) } - defer os.Remove(f.Name()) + defer func() { _ = os.Remove(f.Name()) }() if _, err := f.WriteString(b.String()); err != nil { - f.Close() + _ = f.Close() return a.fail(err) } if err := f.Close(); err != nil { diff --git a/cmd/backport-batch/report.go b/cmd/backport-batch/report.go index ee072e332..fbfc4d569 100644 --- a/cmd/backport-batch/report.go +++ b/cmd/backport-batch/report.go @@ -850,7 +850,7 @@ func (a *app) postReport(n int, s storedReport) error { if err != nil { return err } - defer os.Remove(name) + defer func() { _ = os.Remove(name) }() return a.gh.createComment(n, name) } @@ -873,12 +873,12 @@ func writeTemp(pattern, content string) (string, error) { return "", err } if _, err := f.WriteString(content); err != nil { - f.Close() - os.Remove(f.Name()) + _ = f.Close() + _ = os.Remove(f.Name()) return "", err } if err := f.Close(); err != nil { - os.Remove(f.Name()) + _ = os.Remove(f.Name()) return "", err } return f.Name(), nil diff --git a/cmd/backport-batch/report_test.go b/cmd/backport-batch/report_test.go index a496f68e0..754b599bb 100644 --- a/cmd/backport-batch/report_test.go +++ b/cmd/backport-batch/report_test.go @@ -224,7 +224,9 @@ func TestReportPartition(t *testing.T) { } t.Setenv("GIT_COMMITTER_DATE", "2026-01-01T00:00:00Z") f.pr(4, "fix(app): four, before the start", "", map[string]string{"app/four.txt": "4\n"}) - os.Unsetenv("GIT_COMMITTER_DATE") + if err := os.Unsetenv("GIT_COMMITTER_DATE"); err != nil { + t.Fatal(err) + } f.setPolicy("release-1.0: 2026-06-01\n") f.onLine("release-1.0", "fix(app): a line-only eleven", map[string]string{"app/eleven.txt": "line\n"}) pr(5, "fix(app): five", map[string]string{"app/five.txt": "5\n"}, true) diff --git a/cmd/backport-batch/shape.go b/cmd/backport-batch/shape.go index ee0a6940d..31d2da501 100644 --- a/cmd/backport-batch/shape.go +++ b/cmd/backport-batch/shape.go @@ -121,9 +121,9 @@ func commitMerge(base, tip, message string) (string, error) { if err != nil { return "", err } - defer os.Remove(f.Name()) + defer func() { _ = os.Remove(f.Name()) }() if _, err := f.WriteString(message); err != nil { - f.Close() + _ = f.Close() return "", err } if err := f.Close(); err != nil { From 939a1ec44b940566cc1267c5b43ffee4f6d32007 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 00:23:41 +0300 Subject: [PATCH 494/518] refactor: read managed fields through FieldsV1 accessors Direct access to FieldsV1.Raw is deprecated; GetRawBytes returns the same bytes and is nil-safe, and NewFieldsV1 builds the test fixtures. The write-options test still covers a raw-only orphanDependents delete option, now decoded from its wire form, and asserts that the converted options keep it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- internal/controller/fluxplunger/flux_plunger.go | 5 +++-- .../controller/fluxplunger/flux_plunger_test.go | 4 ++-- pkg/registry/write_options_test.go | 16 +++++++++++++--- 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/internal/controller/fluxplunger/flux_plunger.go b/internal/controller/fluxplunger/flux_plunger.go index d53ec3eef..22775837b 100644 --- a/internal/controller/fluxplunger/flux_plunger.go +++ b/internal/controller/fluxplunger/flux_plunger.go @@ -486,11 +486,12 @@ func ownsSuspensionManagedFields(hr *helmv2.HelmRelease) bool { // fieldsManageSuspend reports whether a managed-fields set includes spec.suspend, // encoded by the apiserver as {"f:spec":{"f:suspend":{}}}. func fieldsManageSuspend(fs *metav1.FieldsV1) bool { - if fs == nil || len(fs.Raw) == 0 { + raw := fs.GetRawBytes() + if len(raw) == 0 { return false } var top map[string]json.RawMessage - if err := json.Unmarshal(fs.Raw, &top); err != nil { + if err := json.Unmarshal(raw, &top); err != nil { return false } specRaw, ok := top["f:spec"] diff --git a/internal/controller/fluxplunger/flux_plunger_test.go b/internal/controller/fluxplunger/flux_plunger_test.go index 365250621..fc1f8adad 100644 --- a/internal/controller/fluxplunger/flux_plunger_test.go +++ b/internal/controller/fluxplunger/flux_plunger_test.go @@ -83,7 +83,7 @@ func suspendManagedFields(manager string, op metav1.ManagedFieldsOperationType) Operation: op, APIVersion: helmv2.GroupVersion.String(), FieldsType: "FieldsV1", - FieldsV1: &metav1.FieldsV1{Raw: []byte(`{"f:spec":{"f:suspend":{}}}`)}, + FieldsV1: metav1.NewFieldsV1(`{"f:spec":{"f:suspend":{}}}`), }} } @@ -286,7 +286,7 @@ func TestOwnsSuspensionManagedFields(t *testing.T) { {"no managed fields", nil, false}, {"plunger apply of other fields", []metav1.ManagedFieldsEntry{{ Manager: suspendFieldManager, Operation: metav1.ManagedFieldsOperationApply, - FieldsType: "FieldsV1", FieldsV1: &metav1.FieldsV1{Raw: []byte(`{"f:spec":{"f:chart":{}}}`)}, + FieldsType: "FieldsV1", FieldsV1: metav1.NewFieldsV1(`{"f:spec":{"f:chart":{}}}`), }}, false}, } for _, tc := range cases { diff --git a/pkg/registry/write_options_test.go b/pkg/registry/write_options_test.go index 488d7a51a..8d6b375f9 100644 --- a/pkg/registry/write_options_test.go +++ b/pkg/registry/write_options_test.go @@ -17,7 +17,9 @@ limitations under the License. package registry import ( + "encoding/json" "reflect" + "strings" "testing" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" @@ -26,7 +28,6 @@ import ( func TestClientWriteOptionsPreserveAPISemantics(t *testing.T) { grace := int64(7) - orphan := true propagation := metav1.DeletePropagationForeground uid := types.UID("expected-uid") rv := "42" @@ -74,11 +75,20 @@ func TestClientWriteOptionsPreserveAPISemantics(t *testing.T) { t.Fatalf("conversion mutated API options: create=%#v update=%#v delete=%#v", create, update, deleteOptions) } - orphanDelete := &metav1.DeleteOptions{OrphanDependents: &orphan, DryRun: []string{metav1.DryRunAll}} + // orphanDependents is deprecated in Go but still accepted on the wire, + // which is how a raw-only option reaches the registry. + orphanDelete := &metav1.DeleteOptions{} + if err := json.Unmarshal([]byte(`{"orphanDependents":true,"dryRun":["All"]}`), orphanDelete); err != nil { + t.Fatal(err) + } orphanExpected := orphanDelete.DeepCopy() - if got := ClientDeleteOptions(orphanDelete).AsDeleteOptions(); !reflect.DeepEqual(got, orphanExpected) { + got := ClientDeleteOptions(orphanDelete).AsDeleteOptions() + if !reflect.DeepEqual(got, orphanExpected) { t.Fatalf("raw-only delete option lost: got %#v, want %#v", got, orphanExpected) } + if wire, err := json.Marshal(got); err != nil || !strings.Contains(string(wire), `"orphanDependents":true`) { + t.Fatalf("converted delete options = %s (%v), want orphanDependents kept", wire, err) + } if !reflect.DeepEqual(orphanDelete, orphanExpected) { t.Fatalf("raw-only delete conversion mutated input: got %#v, want %#v", orphanDelete, orphanExpected) } From 648928f27299118af5d2d1ee1b559cce007e3a18 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 00:23:43 +0300 Subject: [PATCH 495/518] refactor(workloadmonitor): use k8s.io/utils/ptr instead of pointer The pointer package is deprecated; its Bool and BoolPtr are aliases of ptr.To, so the values written are unchanged. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- internal/controller/workloadmonitor_controller.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/internal/controller/workloadmonitor_controller.go b/internal/controller/workloadmonitor_controller.go index b8b0bac13..120ba2533 100644 --- a/internal/controller/workloadmonitor_controller.go +++ b/internal/controller/workloadmonitor_controller.go @@ -28,7 +28,7 @@ import ( "k8s.io/client-go/tools/record" "k8s.io/client-go/util/retry" "k8s.io/client-go/util/workqueue" - "k8s.io/utils/pointer" + "k8s.io/utils/ptr" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/cache" "sigs.k8s.io/controller-runtime/pkg/client" @@ -423,8 +423,8 @@ func updateOwnerReferences(obj metav1.Object, monitor client.Object) { Name: monitor.GetName(), UID: monitor.GetUID(), // Set Controller to false to avoid conflict as multiple controllers are not allowed - Controller: pointer.BoolPtr(false), - BlockOwnerDeletion: pointer.BoolPtr(true), + Controller: ptr.To(false), + BlockOwnerDeletion: ptr.To(true), } owners = append(owners, newOwnerRef) } @@ -1174,9 +1174,9 @@ func (r *WorkloadMonitorReconciler) Reconcile(ctx context.Context, req ctrl.Requ // Default to operational = true, but check MinReplicas if set. // Use fresh.Spec to avoid making decisions based on a stale cached copy // when the spec was updated between the initial read and this retry. - fresh.Status.Operational = pointer.Bool(true) + fresh.Status.Operational = ptr.To(true) if fresh.Spec.MinReplicas != nil && availableReplicas < *fresh.Spec.MinReplicas { - fresh.Status.Operational = pointer.Bool(false) + fresh.Status.Operational = ptr.To(false) } // Only the DataVolume check writes Message and Reason, so the stored ones // are the last DataVolume verdict: kept while no reader is installed yet, so a @@ -1189,7 +1189,7 @@ func (r *WorkloadMonitorReconciler) Reconcile(ctx context.Context, req ctrl.Requ } } if fresh.Status.Message != "" { - fresh.Status.Operational = pointer.Bool(false) + fresh.Status.Operational = ptr.To(false) } if equality.Semantic.DeepEqual(stored, &fresh.Status) { return nil From 86da6c7a5bbfd11aa5a0f5c61481ddfb475e6c63 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 00:26:47 +0300 Subject: [PATCH 496/518] style(lineage-webhook): prefix sentinel error names with Err Go names sentinel error variables Err...; the package is internal and the webhook is their only user. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- internal/lineagecontrollerwebhook/webhook.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/internal/lineagecontrollerwebhook/webhook.go b/internal/lineagecontrollerwebhook/webhook.go index 7b29e1f76..c3efd84bd 100644 --- a/internal/lineagecontrollerwebhook/webhook.go +++ b/internal/lineagecontrollerwebhook/webhook.go @@ -30,8 +30,8 @@ import ( ) var ( - NoAncestors = errors.New("no managed apps found in lineage") - AncestryAmbiguous = errors.New("object ancestry is ambiguous") + ErrNoAncestors = errors.New("no managed apps found in lineage") + ErrAncestryAmbiguous = errors.New("object ancestry is ambiguous") ) const ( @@ -115,9 +115,9 @@ func (h *LineageControllerWebhook) Handle(ctx context.Context, req admission.Req owner, err := h.getOwner(ctx, obj) switch { - case err != nil && errors.Is(err, AncestryAmbiguous): + case err != nil && errors.Is(err, ErrAncestryAmbiguous): warn = append(warn, "object ancestry ambiguous, using first ancestor found") - case err != nil && errors.Is(err, NoAncestors): + case err != nil && errors.Is(err, ErrNoAncestors): // not a problem, mark object as unmanaged case err != nil: logger.Error(err, "error computing lineage labels") @@ -147,14 +147,14 @@ func (h *LineageControllerWebhook) Handle(ctx context.Context, req admission.Req func (h *LineageControllerWebhook) getOwner(ctx context.Context, o *unstructured.Unstructured) (*unstructured.Unstructured, error) { owners := lineage.WalkOwnershipGraphWithCache(ctx, h.dynClient, h.mapper, h, h.ownerCache, o) if len(owners) == 0 { - return nil, NoAncestors + return nil, ErrNoAncestors } obj, err := owners[0].GetUnstructuredCached(ctx, h.dynClient, h.mapper, h.ownerCache) if err != nil { return nil, err } if len(owners) > 1 { - err = AncestryAmbiguous + err = ErrAncestryAmbiguous } return obj, err } From 40d340a3fc8147e5d07e6026f75c628db849e91f Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 00:56:58 +0300 Subject: [PATCH 497/518] fix(sse): wait for the pump before returning When a client disconnected, the handler stopped its pump and returned at once, while the pump could still be writing an event or a heartbeat to the response. net/http does not allow a write to a response after ServeHTTP has returned. The handler now waits for the pump to exit; a pump blocked in a write returns once the connection is gone. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- internal/sse/server.go | 1 + internal/sse/server_test.go | 63 +++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) diff --git a/internal/sse/server.go b/internal/sse/server.go index 166b01f49..67e19cbc6 100644 --- a/internal/sse/server.go +++ b/internal/sse/server.go @@ -222,6 +222,7 @@ func (s *Server) handleStream(w http.ResponseWriter, r *http.Request) { case <-pumped: } c.stop() + <-pumped } // stop ends the client's pump. Shutdown and the client's own handler both diff --git a/internal/sse/server_test.go b/internal/sse/server_test.go index 5ae5ad966..ecb0bcd54 100644 --- a/internal/sse/server_test.go +++ b/internal/sse/server_test.go @@ -181,3 +181,66 @@ func TestShutdownEndsOpenStreams(t *testing.T) { t.Fatal("Shutdown left the stream open") } } + +// blockingWriter holds every write after the first until release is closed. +type blockingWriter struct { + streamWriter + writing chan struct{} + release chan struct{} + returned chan struct{} +} + +func (w *blockingWriter) Write(p []byte) (int, error) { + if w.String() == "" { + return w.streamWriter.Write(p) + } + select { + case w.writing <- struct{}{}: + default: + } + <-w.release + return w.streamWriter.Write(p) +} + +// The handler does not return while its pump is still writing: net/http does +// not allow a write to a response after ServeHTTP has returned. +func TestStreamWaitsForPumpBeforeReturning(t *testing.T) { + s := newTestServer(time.Hour) + w := &blockingWriter{ + streamWriter: streamWriter{header: http.Header{}, okWrites: 100}, + writing: make(chan struct{}, 1), + release: make(chan struct{}), + returned: make(chan struct{}), + } + ctx, cancel := context.WithCancel(context.Background()) + r := httptest.NewRequest(http.MethodGet, "/stream", nil).WithContext(ctx) + go func() { + s.handleStream(w, r) + close(w.returned) + }() + deadline := time.Now().Add(time.Second) + for len(s.snapshotClients()) == 0 && time.Now().Before(deadline) { + time.Sleep(time.Millisecond) + } + // Straight onto the client's queue, which only the pump reads: Publish + // would also set latest, which the handler itself may still write. + s.snapshotClients()[0].ch <- "event" + select { + case <-w.writing: + case <-time.After(time.Second): + t.Fatal("the pump never wrote the event") + } + cancel() + select { + case <-w.returned: + close(w.release) + t.Fatal("the handler returned while its pump was writing") + case <-time.After(100 * time.Millisecond): + } + close(w.release) + select { + case <-w.returned: + case <-time.After(time.Second): + t.Fatal("the handler did not return once its pump finished") + } +} From 6f2d6ed17b7a712c91f33c49f771db7a2f352af1 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 01:46:20 +0300 Subject: [PATCH 498/518] fix(sse): refuse streams that open while the server shuts down Shutdown stopped the clients registered at that moment and then shut the HTTP server down. A stream accepted in between registered after the stop loop, so nothing ended it, and http.Server.Shutdown waited for its handler. The kube-ovn plunger calls Shutdown without a deadline, so such a stream delayed stopping until the manager's graceful-shutdown timeout and then failed it. Shutdown now marks the server as closing under the same lock, and a stream that opens after that is answered 503 without being registered. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- internal/sse/server.go | 9 +++++++++ internal/sse/server_test.go | 29 +++++++++++++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/internal/sse/server.go b/internal/sse/server.go index 67e19cbc6..3c121a08a 100644 --- a/internal/sse/server.go +++ b/internal/sse/server.go @@ -51,6 +51,9 @@ type Server struct { clientsMu sync.RWMutex clients map[*client]struct{} + // closing is set by Shutdown under clientsMu, so no stream registers + // after the clients it stops. + closing bool // latest holds the most recent payload (sent to new clients on connect). latestMu sync.RWMutex @@ -122,6 +125,7 @@ func (s *Server) ListenAndServe() error { // Shutdown gracefully stops the server. func (s *Server) Shutdown(ctx context.Context) error { s.clientsMu.Lock() + s.closing = true for c := range s.clients { c.stop() } @@ -184,6 +188,11 @@ func (s *Server) handleStream(w http.ResponseWriter, r *http.Request) { // Register client s.clientsMu.Lock() + if s.closing { + s.clientsMu.Unlock() + http.Error(w, "server is shutting down", http.StatusServiceUnavailable) + return + } s.clients[c] = struct{}{} s.clientsMu.Unlock() diff --git a/internal/sse/server_test.go b/internal/sse/server_test.go index ecb0bcd54..577d19308 100644 --- a/internal/sse/server_test.go +++ b/internal/sse/server_test.go @@ -244,3 +244,32 @@ func TestStreamWaitsForPumpBeforeReturning(t *testing.T) { t.Fatal("the handler did not return once its pump finished") } } + +// A stream that opens once Shutdown has run is refused instead of being +// registered after the clients were stopped, where nothing would end it. +func TestStreamOpenedDuringShutdownIsRefused(t *testing.T) { + s := newTestServer(time.Hour) + if err := s.Shutdown(context.Background()); err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancel(context.Background()) + t.Cleanup(cancel) + rec := httptest.NewRecorder() + r := httptest.NewRequest(http.MethodGet, "/stream", nil).WithContext(ctx) + done := make(chan struct{}) + go func() { + s.handleStream(rec, r) + close(done) + }() + select { + case <-done: + case <-time.After(time.Second): + t.Fatal("a stream opened after Shutdown stayed open") + } + if rec.Code != http.StatusServiceUnavailable { + t.Errorf("status = %d, want %d", rec.Code, http.StatusServiceUnavailable) + } + if len(s.snapshotClients()) != 0 { + t.Error("a stream opened after Shutdown was registered") + } +} From a3305e8dd3cd1401f844bc31eb495057fb886831 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 03:00:27 +0300 Subject: [PATCH 499/518] fix(backport): push and publish nothing once output is lost backport-batch writes its body, summary, findings and warnings before it pushes, opens or edits a pull request, closes a batch or posts a comment. A run that kept going after that output was lost acted on what nobody read, and a closed pipe was worse: with Go's default SIGPIPE handling the process died silently mid-run. Lines that confirm an action after it happened, such as the URL of an opened pull request, go to stderr, so a lost stderr matters as much as a lost stdout. A failed write to either stream is kept, and from then on every push and every GitHub write is refused at the point it would be made: the GitHub client is wrapped so that each write checks first, and each push checks before it runs. Once output is lost, the run changes nothing more: no push, no GitHub write, no local branch move. No new unit starts after output is lost; a unit already in flight finishes its rollback. A reset whose own hint, the only way back to the commits it drops, cannot be written does not run either. The run exits 2 saying the output was lost; a confirmation lost after an action fails the run too. Such a confirmation is lost only when stderr itself fails, so a lost confirmation can only be reported by the exit code. A test holds every method of the GitHub interface to being a known read or a guarded write. The tool catches SIGPIPE with signal.Notify, so a closed pipe is a write error like any other. It does not use signal.Ignore, whose ignored disposition the git and gh children would inherit across exec; a test checks that a child still dies of SIGPIPE. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- cmd/backport-batch/README.md | 2 +- cmd/backport-batch/assemble.go | 17 +- cmd/backport-batch/defer.go | 2 +- cmd/backport-batch/github.go | 48 +++++ cmd/backport-batch/main.go | 33 ++- cmd/backport-batch/output_test.go | 329 ++++++++++++++++++++++++++++++ cmd/backport-batch/shape.go | 17 ++ 7 files changed, 439 insertions(+), 9 deletions(-) diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index 4919035c0..7137bdaf0 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -27,7 +27,7 @@ Requires `git` and an authenticated `gh` on PATH, and a remote called `origin`. Build the binary wherever an exit code matters: `go run` turns any non-zero exit into its own `1`, so under it a check that could not be completed looks the same as one that found something missing. -A command whose output on stdout could not be written exits 2 where it would have exited 0, and says on stderr that the output was lost; anything the run did before that, a push or a published pull request included, has still happened. +Output is written before a push or a publish. Once a write to stdout or stderr fails, a closed pipe included, the run changes nothing more: no push, no GitHub write, no local branch move, and no new pick; a pick already in flight still finishes its rollback. It exits 2 where it would have exited 0, saying on stderr, while it still can, that the output was lost. A confirmation lost after a push or a publish fails the run the same way. ### run diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index 2ebca3483..7f5529bcd 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -273,7 +273,7 @@ func (a *app) run(args []string) int { continue } } - p, err := pickOriginal(v, func(base, head string) ([]string, error) { + p, err := a.pickUnit(v, func(base, head string) ([]string, error) { omitted, diag, err := b.omittedEffects(v, base, head) v.addDiag(diag...) return omitted, err @@ -529,6 +529,9 @@ func cleanTree() error { // the move drops commits of and how to get them back: a clean tree says // nothing about commits only a local branch has. func (a *app) warnReset(branch, start string) error { + if err := a.mayAct(); err != nil { + return err + } old, err := lookup("refs/heads/" + branch) if err != nil || old == "" || old == start { return err @@ -537,7 +540,11 @@ func (a *app) warnReset(branch, start string) error { if err != nil || kept { return err } - _, _ = fmt.Fprintf(a.stderr, "resetting the local %s from %s, dropping commits %.12s does not carry; git branch %s-%.12s %s restores them\n", branch, old, start, branch, old, old) + // The hint is the only way back to the dropped commits, so the reset does + // not run when it could not be written. + if _, err := fmt.Fprintf(a.stderr, "resetting the local %s from %s, dropping commits %.12s does not carry; git branch %s-%.12s %s restores them\n", branch, old, start, branch, old, old); err != nil { + return fmt.Errorf("not resetting the local %s: %w", branch, err) + } return nil } @@ -719,6 +726,9 @@ func (a *app) publish(b *batch, p publication, rebuilt bool) error { } } if pushed != p.head { + if err := a.mayAct(); err != nil { + return err + } if _, err := git(pushArgs(p.branch, p.head, lease)...); err != nil { return err } @@ -878,6 +888,9 @@ func (a *app) deleteBranch(l line, c closing) error { case h != c.Head: return movedBranch(c, h) } + if err := a.mayAct(); err != nil { + return err + } if _, err := git(deleteArgs(c.Branch, c.Head)...); err != nil { return fmt.Errorf("deleting %s, left by the close of #%d: %w; backport-batch run --line %s --push deletes it", c.Branch, c.PR, err, l.branch()) } diff --git a/cmd/backport-batch/defer.go b/cmd/backport-batch/defer.go index de77d5d5c..84e05d5e8 100644 --- a/cmd/backport-batch/defer.go +++ b/cmd/backport-batch/defer.go @@ -102,7 +102,7 @@ func (a *app) reassemble(b *batch, out, hold map[int]string, drops []droppedEntr return a.fail(err) } for _, u := range replay { - failed, files, err := replayUnit(u) + failed, files, err := a.replayNext(u) switch { case err != nil: return a.fail(err) diff --git a/cmd/backport-batch/github.go b/cmd/backport-batch/github.go index 7161ef15b..0e549b1f6 100644 --- a/cmd/backport-batch/github.go +++ b/cmd/backport-batch/github.go @@ -137,6 +137,54 @@ type github interface { batchBot() string } +// guardedGitHub refuses every GitHub write once the run's output was lost. +type guardedGitHub struct { + github + a *app +} + +func (g guardedGitHub) createPull(base, head, title, bodyFile string) (int, string, error) { + if err := g.a.mayAct(); err != nil { + return 0, "", err + } + return g.github.createPull(base, head, title, bodyFile) +} + +func (g guardedGitHub) editPull(number int, title, bodyFile string) error { + if err := g.a.mayAct(); err != nil { + return err + } + return g.github.editPull(number, title, bodyFile) +} + +func (g guardedGitHub) editBody(number int, bodyFile string) error { + if err := g.a.mayAct(); err != nil { + return err + } + return g.github.editBody(number, bodyFile) +} + +func (g guardedGitHub) closePull(number int) error { + if err := g.a.mayAct(); err != nil { + return err + } + return g.github.closePull(number) +} + +func (g guardedGitHub) createComment(number int, bodyFile string) error { + if err := g.a.mayAct(); err != nil { + return err + } + return g.github.createComment(number, bodyFile) +} + +func (g guardedGitHub) editComment(id int64, bodyFile string) error { + if err := g.a.mayAct(); err != nil { + return err + } + return g.github.editComment(id, bodyFile) +} + type ghCLI struct{ bot string } func (g ghCLI) batchBot() string { return g.bot } diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index 19102e57f..c49fa951f 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -24,7 +24,9 @@ import ( "fmt" "io" "os" + "os/signal" "strings" + "syscall" "time" ) @@ -111,21 +113,31 @@ type app struct { policyRef string // tool is the commit backport-batch was built from, for the record. tool string - // unwritten is the first error writing to stdout, which fails the run: - // what it printed never reached its reader. + // unwritten is the first error writing to stdout or stderr. Output goes + // out before a push or a publish, so a run that lost it acts on nothing, + // and one that lost a confirmation afterwards fails. unwritten error } func main() { + catchSIGPIPE() a := &app{gh: ghCLI{bot: batchBotLogin(os.LookupEnv)}, stdout: os.Stdout, stderr: os.Stderr, now: time.Now, tool: toolVersion()} os.Exit(a.main(os.Args[1:])) } +// catchSIGPIPE makes a write to a closed stdout or stderr fail with EPIPE like +// any other write, so the run stops before it pushes or publishes and says +// why, instead of dying silently. signal.Ignore would do the same here but +// would also leave SIGPIPE ignored in the git and gh children, across exec. +func catchSIGPIPE() { + signal.Notify(make(chan os.Signal, 1), syscall.SIGPIPE) +} + func (a *app) main(args []string) int { - a.stdout, a.stderr = textWriter{w: a.stdout, failed: &a.unwritten}, textWriter{w: a.stderr} + a.stdout, a.stderr = textWriter{w: a.stdout, failed: &a.unwritten}, textWriter{w: a.stderr, failed: &a.unwritten} code := a.command(args) if a.unwritten != nil { - _, _ = fmt.Fprintf(a.stderr, "backport-batch: output was lost: %v; anything the run did, such as a push or a publish, still happened\n", a.unwritten) + _, _ = fmt.Fprintf(a.stderr, "backport-batch: output was lost: %v\n", a.unwritten) if code == 0 { code = 2 } @@ -134,7 +146,7 @@ func (a *app) main(args []string) int { } func (a *app) command(args []string) int { - a.gh = &permCache{github: a.gh, perms: map[string]string{}} + a.gh = guardedGitHub{github: &permCache{github: a.gh, perms: map[string]string{}}, a: a} if len(args) == 0 { _, _ = fmt.Fprint(a.stderr, usageText) return 2 @@ -208,6 +220,17 @@ func parse(fs *flag.FlagSet, args []string) error { return nil } +// mayAct refuses any further change once output was lost: the run would act +// on what it printed while nobody read it. Every push and every move of a +// local batch branch go through it, and every GitHub write through +// guardedGitHub. +func (a *app) mayAct() error { + if a.unwritten != nil { + return fmt.Errorf("output was lost: %w; nothing more is pushed or written to GitHub", a.unwritten) + } + return nil +} + func (a *app) fail(err error) int { _, _ = fmt.Fprintf(a.stderr, "backport-batch: %v\n", err) return 2 diff --git a/cmd/backport-batch/output_test.go b/cmd/backport-batch/output_test.go index 0362f44b7..a094d4b75 100644 --- a/cmd/backport-batch/output_test.go +++ b/cmd/backport-batch/output_test.go @@ -18,6 +18,10 @@ package main import ( "errors" + "io" + "os" + "os/exec" + "reflect" "strings" "testing" "time" @@ -54,3 +58,328 @@ func TestUnwritableOutputKeepsTheCommandsFailure(t *testing.T) { t.Errorf("stderr does not name the write failure:\n%s", stderr.String()) } } + +// A closed pipe on stdout does not kill the run before it can say what +// already happened: the write fails like any other, and the run exits 2. +func TestClosedStdoutPipeIsReported(t *testing.T) { + if os.Getenv("BACKPORT_BATCH_CLOSED_STDOUT_HELPER") == "1" { + os.Args = []string{"backport-batch", "help"} + main() + return + } + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + _ = r.Close() + t.Cleanup(func() { _ = w.Close() }) + cmd := exec.Command(os.Args[0], "-test.run=^TestClosedStdoutPipeIsReported$") + cmd.Env = append(os.Environ(), "BACKPORT_BATCH_CLOSED_STDOUT_HELPER=1") + var stderr strings.Builder + cmd.Stdout, cmd.Stderr = w, &stderr + err = cmd.Run() + var exit *exec.ExitError + if !errors.As(err, &exit) || exit.ExitCode() != 2 { + t.Fatalf("help into a closed pipe = %v, want exit 2\nstderr:\n%s", err, stderr.String()) + } + if !strings.Contains(stderr.String(), "output was lost") { + t.Errorf("stderr does not say the output was lost:\n%s", stderr.String()) + } +} + +// failOn fails every write that contains want, and takes every other one. +type failOn struct { + want string + buf strings.Builder +} + +func (w *failOn) Write(p []byte) (int, error) { + if strings.Contains(string(p), w.want) { + return 0, errors.New("broken pipe") + } + return w.buf.Write(p) +} + +func runWith(f *fixture, gh *fakeGH, stdout, stderr io.Writer, args ...string) int { + f.t.Helper() + if gh.remoteHead == nil { + gh.remoteHead = f.remoteHead + } + a := &app{gh: gh, stdout: stdout, stderr: stderr, now: func() time.Time { return f.now }, tool: f.tool} + return a.main(args) +} + +// Output is written before the push; when it is lost, on either stream, +// nothing is pushed and no pull request is opened. +func TestLostOutputPushesNothing(t *testing.T) { + for _, stream := range []string{"stdout", "stderr"} { + t.Run(stream, func(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{} + var stdout, stderr io.Writer = &strings.Builder{}, &strings.Builder{} + if stream == "stdout" { + stdout = failingWriter{} + } else { + stderr = failingWriter{} + } + if code := runWith(f, gh, stdout, stderr, "run", "--line", "release-1.1", "--push"); code != 2 { + t.Fatalf("run --push with a failing %s = %d, want 2", stream, code) + } + if head := f.remoteHead(batchBranch1); head != "" { + t.Errorf("the batch was pushed at %.12s", head) + } + if len(gh.prs) != 0 { + t.Errorf("a pull request was opened: %+v", gh.prs) + } + }) + } +} + +// A confirmation lost after the push fails the run: the pull request was +// opened, and the run does not exit 0 without having said so. +func TestLostConfirmationFailsTheRun(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{} + if code := runWith(f, gh, &strings.Builder{}, &failOn{want: "opened draft"}, "run", "--line", "release-1.1", "--push"); code != 2 { + t.Fatalf("run --push losing its confirmation = %d, want 2", code) + } + if len(gh.prs) != 1 { + t.Errorf("%d pull requests, want the one the run opened", len(gh.prs)) + } + if len(gh.issueComments) != 0 { + t.Errorf("the report was posted after the confirmation was lost: %+v", gh.issueComments) + } +} + +// notify writes what it found before it comments; when that is lost, it +// does not comment. +func TestLostOutputCommentsNothing(t *testing.T) { + gh := &fakeGH{prs: []pullRequest{notifyPR("20261019", "")}} + var stderr strings.Builder + a := &app{gh: gh, stdout: failingWriter{}, stderr: &stderr, now: func() time.Time { return time.Date(2026, 10, 26, 6, 23, 0, 0, time.UTC) }} + if code := a.main([]string{"notify", "--line", "release-1.1", "--push"}); code != 2 { + t.Fatalf("notify --push with a failing stdout = %d, want 2\n%s", code, stderr.String()) + } + if len(gh.issueComments) != 0 { + t.Errorf("notify commented: %+v", gh.issueComments) + } +} + +// writesGitHub records the GitHub writes that reach it. +type writesGitHub struct { + github + writes []string +} + +func (g *writesGitHub) createPull(string, string, string, string) (int, string, error) { + g.writes = append(g.writes, "createPull") + return 1, "", nil +} +func (g *writesGitHub) editPull(int, string, string) error { + g.writes = append(g.writes, "editPull") + return nil +} +func (g *writesGitHub) editBody(int, string) error { + g.writes = append(g.writes, "editBody") + return nil +} +func (g *writesGitHub) closePull(int) error { + g.writes = append(g.writes, "closePull") + return nil +} +func (g *writesGitHub) createComment(int, string) error { + g.writes = append(g.writes, "createComment") + return nil +} +func (g *writesGitHub) editComment(int64, string) error { + g.writes = append(g.writes, "editComment") + return nil +} + +// Every GitHub write is refused once output was lost. Every method of +// github is either a read or in this table, so a write added later without +// a guard fails here. +func TestEveryGitHubWriteIsGuarded(t *testing.T) { + reads := map[string]bool{"openPulls": true, "recentPulls": true, "permission": true, "labelled": true, "pull": true, "comments": true, "batchBot": true} + inner := &writesGitHub{} + g := guardedGitHub{github: inner, a: &app{unwritten: errors.New("broken pipe")}} + writes := map[string]func() error{ + "createPull": func() error { _, _, err := g.createPull("b", "h", "t", "f"); return err }, + "editPull": func() error { return g.editPull(1, "t", "f") }, + "editBody": func() error { return g.editBody(1, "f") }, + "closePull": func() error { return g.closePull(1) }, + "createComment": func() error { return g.createComment(1, "f") }, + "editComment": func() error { return g.editComment(1, "f") }, + } + iface := reflect.TypeFor[github]() + for i := range iface.NumMethod() { + name := iface.Method(i).Name + if _, ok := writes[name]; !ok && !reads[name] { + t.Errorf("github.%s is neither a known read nor a guarded write", name) + } + } + for name, write := range writes { + if err := write(); err == nil || !strings.Contains(err.Error(), "output was lost") { + t.Errorf("%s with output lost = %v, want a refusal", name, err) + } + } + if len(inner.writes) != 0 { + t.Errorf("writes reached GitHub after output was lost: %q", inner.writes) + } +} + +// A warning lost while the open batch is being updated stops the update: +// nothing is pushed, posted or edited. +func TestLostWarningLeavesTheOpenBatchAlone(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{} + mustRun(t, f, gh, "run", "--line", "release-1.1", "--push") + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + bad := issueComment{ID: 8, Body: reportMarker + "names none\n"} + bad.User.Login, bad.User.Type = "maintainer", "User" + gh.issueComments = append(gh.issueComments, bad) + head, comments, body := f.remoteHead(batchBranch1), len(gh.issueComments), gh.find(90).Body + f.now = f.now.AddDate(0, 0, 1) + if code := runWith(f, gh, &strings.Builder{}, &failOn{want: "does not count as a report"}, "run", "--line", "release-1.1", "--continue", "--push"); code != 2 { + t.Fatalf("run --continue --push losing a warning = %d, want 2", code) + } + if got := f.remoteHead(batchBranch1); got != head { + t.Errorf("the batch moved to %.12s", got) + } + if len(gh.issueComments) != comments { + t.Errorf("a comment was posted: %d comments, want %d", len(gh.issueComments), comments) + } + if gh.find(90).Body != body { + t.Error("the batch pull request was edited") + } +} + +// A warning lost while a batch is being closed leaves it open, with no +// closing record half written. +func TestLostWarningLeavesTheBatchUnclosed(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{} + mustRun(t, f, gh, "run", "--line", "release-1.1", "--push") + bad := issueComment{ID: 8, Body: reportMarker + "names none\n"} + bad.User.Login, bad.User.Type = "maintainer", "User" + gh.issueComments = append(gh.issueComments, bad) + body := gh.find(90).Body + if code := runWith(f, gh, &strings.Builder{}, &failOn{want: "does not count as a report"}, "run", "--line", "release-1.1", "--defer", "5", "--reason", "x", "--push"); code != 2 { + t.Fatalf("run --defer losing a warning = %d, want 2", code) + } + if gh.find(90).Body != body { + t.Error("the batch pull request was edited") + } + if !gh.isOpen(90) { + t.Error("the batch pull request was closed") + } +} + +// The git and gh children keep the default SIGPIPE: catching it here must not +// leave it ignored in them, as signal.Ignore would through exec. +func TestChildrenKeepTheDefaultSIGPIPE(t *testing.T) { + catchSIGPIPE() + out, err := exec.Command("sh", "-c", "kill -PIPE $$; echo survived").Output() + if err == nil || strings.Contains(string(out), "survived") { + t.Fatalf("a child survived SIGPIPE: %q, %v", out, err) + } +} + +// The branch a close left is not deleted once output was lost: a deletion +// is a push like any other. +func TestLostOutputDeletesNoBranch(t *testing.T) { + f, gh, h1 := closeWorld(t) + f.git("switch", "-q", "main") + l, err := parseLine("release-1.1") + if err != nil { + t.Fatal(err) + } + a := &app{gh: gh, unwritten: errors.New("broken pipe")} + if err := a.deleteBranch(l, closing{PR: 90, Branch: batchBranch1, Head: h1}); err == nil || !strings.Contains(err.Error(), "output was lost") { + t.Fatalf("deleteBranch with output lost = %v, want a refusal", err) + } + if got := f.remoteHead(batchBranch1); got != h1 { + t.Errorf("the branch is at %q, want it left at %.12s", got, h1) + } +} + +// A park line lost on stdout stops notify before it writes: no comment, and +// the record is not marked as announced. +func TestLostParkLineWritesNothing(t *testing.T) { + park := parkedEntry{Original: 5, Merge: strings.Repeat("c", 40), Kind: parkDependency, Needs: []int{3}} + gh := &fakeGH{prs: []pullRequest{notifyPR("20261026", parkedBody(park))}} + body := gh.prs[0].Body + var stderr strings.Builder + a := &app{gh: gh, stdout: &failOn{want: "parked originals"}, stderr: &stderr, now: func() time.Time { return time.Date(2026, 10, 26, 6, 23, 0, 0, time.UTC) }} + if code := a.main([]string{"notify", "--line", "release-1.1", "--push"}); code != 2 { + t.Fatalf("notify --push losing the park line = %d, want 2\n%s", code, stderr.String()) + } + if len(gh.issueComments) != 0 { + t.Errorf("notify commented: %+v", gh.issueComments) + } + if gh.prs[0].Body != body { + t.Error("notify marked the park as announced") + } +} + +// The hint naming the commits a reset drops is the only way back to them, so +// the local branch is not reset when the hint is lost. +func TestLostResetHintKeepsTheBranch(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.git("switch", "-q", "-C", batchBranch1, "origin/release-1.1") + f.write("app/mine.txt", "unpushed\n") + f.git("add", "-A") + f.git("commit", "-q", "-m", "an unpushed resolution") + tip := f.git("rev-parse", "HEAD") + f.git("switch", "-q", "main") + if code := runWith(f, &fakeGH{}, &strings.Builder{}, &failOn{want: "resetting the local"}, "run", "--line", "release-1.1"); code != 2 { + t.Fatalf("run losing the reset hint = %d, want 2", code) + } + if got := f.git("rev-parse", batchBranch1); got != tip { + t.Errorf("the local %s moved to %.12s, want it kept at %.12s", batchBranch1, got, tip) + } +} + +// Once output is lost the run moves no local branch, even one a reset would +// only fast-forward. +func TestLostOutputMovesNoLocalBranch(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.git("branch", "-f", batchBranch1, "origin/release-1.1^") + before := f.git("rev-parse", batchBranch1) + if code := runWith(f, &fakeGH{}, &strings.Builder{}, &failOn{want: "prepared"}, "run", "--line", "release-1.1"); code != 2 { + t.Fatalf("run losing an early line = %d, want 2", code) + } + if got := f.git("rev-parse", batchBranch1); got != before { + t.Errorf("the local %s moved to %.12s, want it kept at %.12s", batchBranch1, got, before) + } +} + +// No new unit starts once output is lost: neither a pick nor a replay touches +// the branch. +func TestLostOutputStartsNoUnit(t *testing.T) { + f := newFixture(t) + f.lines() + before := f.git("rev-parse", "HEAD") + a := &app{unwritten: errors.New("broken pipe")} + if _, err := a.pickUnit(&verdict{}, nil); err == nil || !strings.Contains(err.Error(), "output was lost") { + t.Errorf("pickUnit with output lost = %v, want a refusal", err) + } + if _, _, err := a.replayNext(unit{}); err == nil || !strings.Contains(err.Error(), "output was lost") { + t.Errorf("replayNext with output lost = %v, want a refusal", err) + } + if got := f.git("rev-parse", "HEAD"); got != before { + t.Errorf("HEAD moved to %.12s", got) + } +} diff --git a/cmd/backport-batch/shape.go b/cmd/backport-batch/shape.go index 31d2da501..b4a1e759c 100644 --- a/cmd/backport-batch/shape.go +++ b/cmd/backport-batch/shape.go @@ -114,6 +114,23 @@ func mergeMessage(message, merge string) string { return strings.TrimRight(message, " \t\r\n") + "\n\n" + batchTrailer + ": " + merge + "\n" } +// pickUnit and replayNext start the next unit only while the run's output is +// intact: once it was lost no new unit starts, and a unit already in flight +// still finishes its rollback. +func (a *app) pickUnit(v *verdict, validate func(base, head string) ([]string, error)) (pick, error) { + if err := a.mayAct(); err != nil { + return pick{}, err + } + return pickOriginal(v, validate) +} + +func (a *app) replayNext(u unit) (string, []string, error) { + if err := a.mayAct(); err != nil { + return "", nil, err + } + return replayUnit(u) +} + // commitMerge makes the merge of tip onto base, whose tree is tip's since tip // descends from base, and moves the current branch from tip to it. func commitMerge(base, tip, message string) (string, error) { From 456ea6b98b7d90e25a120f4b1d4afd24f1993261 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 11:58:30 +0300 Subject: [PATCH 500/518] ci: run the standalone pkg and cmd unit tests in go-unit-tests go-unit-tests ran only pkg/registry, pkg/config and pkg/cmd/server, so the tests in pkg/lineage, pkg/ovnstatus, pkg/reservation, cmd/cozypkg and cmd/check-readiness ran on no CI job. None of them needs a cluster or the network. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- Makefile | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/Makefile b/Makefile index 2feea00e9..adc652dd7 100644 --- a/Makefile +++ b/Makefile @@ -139,13 +139,14 @@ rd-presets-check: migrations-target-check: hack/check-migrations-target.sh -# Scoped go test over the cozystack-api surface that this repo owns. Kept -# narrow intentionally - running `go test ./...` pulls in generated code -# round-trip suites whose behavior depends on tool versions outside this -# repo's control (kubebuilder, openapi-gen, etc.) and is better exercised -# from their generator workflows. +# Go tests of the cozystack-api surface and of the standalone packages and +# commands listed here, whose tests need no cluster. Kept narrow +# intentionally - running `go test ./...` pulls in generated code round-trip +# suites whose behavior depends on tool versions outside this repo's control +# (kubebuilder, openapi-gen, etc.) and is better exercised from their +# generator workflows. go-unit-tests: - go test ./pkg/registry/... ./pkg/config/... ./pkg/cmd/server/... + go test ./pkg/registry/... ./pkg/config/... ./pkg/cmd/server/... ./pkg/lineage/... ./pkg/ovnstatus/... ./pkg/reservation/... ./cmd/cozypkg/... ./cmd/check-readiness/... # The nested Go modules (image sources and the published API types) are # outside ./... of the root module, so no other target reaches their tests. @@ -158,9 +159,7 @@ go-module-tests: done # Go tests for the controllers and supporting packages under ./internal. -# Excludes ./pkg/... and ./cmd/... — those are run separately by -# go-unit-tests above (pkg subset) and skipped (cmd) until their tests -# stabilise. CI schedules this target in the same four-slot make invocation as +# CI schedules this target in the same four-slot make invocation as # unit-tests; locally invoke it directly or chain the two targets. test-controllers: go test ./internal/... -count=1 @@ -195,7 +194,8 @@ test-controllers-envtest: $(SETUP_ENVTEST) test-check-readiness: go test ./test/check-readiness/ -count=1 -# ./cmd/... is excluded from go-unit-tests, so the batch tool needs its own target. +# The batch tool's suite drives real git repositories for minutes, so it is +# kept out of go-unit-tests in a target of its own. test-backport-batch: go test ./cmd/backport-batch/... -count=1 From 6cab97f0107c641638b884278e24eff7e5a54022 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 12:03:19 +0300 Subject: [PATCH 501/518] style: use tagged switches and drop unused test assignments Use tagged switches where staticcheck suggests them and drop two test assignments whose values are never read. Behaviour is unchanged. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- cmd/backport-batch/effects_test.go | 4 ++-- cmd/backport-batch/notify.go | 12 ++++++------ cmd/backport-batch/report.go | 4 ++-- .../backupcontroller/cnpgstrategy_controller_test.go | 2 +- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/cmd/backport-batch/effects_test.go b/cmd/backport-batch/effects_test.go index f96f5a596..9dfe0de95 100644 --- a/cmd/backport-batch/effects_test.go +++ b/cmd/backport-batch/effects_test.go @@ -53,7 +53,7 @@ func emptyBatch(f *fixture, gh *fakeGH, body string) *foldWorld { // and every check of the batch's shape takes the old name as the same file: // verify, a hand fold, and a rebuild by --defer. func TestRenamePickThroughFoldAndVerify(t *testing.T) { - f, mc5 := renameWorld(t) + f, _ := renameWorld(t) f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) gh := &fakeGH{} head := openBatch(t, f, gh) @@ -74,7 +74,7 @@ func TestRenamePickThroughFoldAndVerify(t *testing.T) { t.Errorf("verify after the rebuild = %d\n%s\n%s", code, out, stderr) } - f, mc5 = renameWorld(t) + f, mc5 := renameWorld(t) w := emptyBatch(f, &fakeGH{}, "") pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/old.txt": strings.Replace(appFile, "three", "THREE", 1)}) fold := w.merge(w.head, pick, "fix(app): fix the app file (#5)\n\n"+batchTrailer+": "+mc5) diff --git a/cmd/backport-batch/notify.go b/cmd/backport-batch/notify.go index e9ee23234..b2a8f7c18 100644 --- a/cmd/backport-batch/notify.go +++ b/cmd/backport-batch/notify.go @@ -102,10 +102,10 @@ func (a *app) notify(args []string) int { var conflicts, waiting []int after := map[int][]int{} for n, e := range deferred { - switch place := placement(deferred, n, map[int]bool{}); { - case place == "workflow": + switch placement(deferred, n, map[int]bool{}) { + case "workflow": waiting = append(waiting, n) - case place == "conflict": + case "conflict": conflicts = append(conflicts, n) } if len(e.Needs) > 0 { @@ -259,10 +259,10 @@ func placement(deferred map[int]deferredEntry, n int, seen map[int]bool) string seen[n] = true place := "" for _, f := range e.Needs { - switch p := placement(deferred, f, seen); { - case p == "workflow": + switch p := placement(deferred, f, seen); p { + case "workflow": return p - case p == "conflict": + case "conflict": place = p } } diff --git a/cmd/backport-batch/report.go b/cmd/backport-batch/report.go index fbfc4d569..27fd8860c 100644 --- a/cmd/backport-batch/report.go +++ b/cmd/backport-batch/report.go @@ -780,8 +780,8 @@ func (a *app) printReport(pr *pullRequest, s storedReport, inBody *storedReport, } for _, e := range d.Primary.Parked { text := "parked: " - switch { - case e.Kind == parkPartial: + switch e.Kind { + case parkPartial: text += "only part of it applies, it would leave " + strings.Join(e.Files, ", ") + " without effect" default: var needs []string diff --git a/internal/backupcontroller/cnpgstrategy_controller_test.go b/internal/backupcontroller/cnpgstrategy_controller_test.go index 5cd990f18..d88408100 100644 --- a/internal/backupcontroller/cnpgstrategy_controller_test.go +++ b/internal/backupcontroller/cnpgstrategy_controller_test.go @@ -2830,7 +2830,7 @@ func TestReconcileCNPG_ObjectStoreApplyRecovers(t *testing.T) { } failing = false - j, _ = f.pass() + _, _ = f.pass() f.settleDestination() j, backups := f.pass() if len(backups) != 1 { From e6127f7e423f4839d5d22262f98e2b91fbc8865f Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Fri, 13 Nov 2026 10:42:20 +0100 Subject: [PATCH 502/518] fix(controller, platform): copy the LimitRange only under a compute-limit quota The tenant chart renders tenant-range-limits under any non-empty resourceQuotas, including one that bounds no compute, such as services.loadbalancers alone. Copying it into an unbounded member whose allocated quota bounds neither limits.cpu nor limits.memory imposes a 128Mi default memory limit the quota never asked for: a container that requests more and sets no limit, virt-launcher among them, is refused where it was admitted before. Only a quota on limits.cpu or limits.memory makes admission reject a container that omits a limit, so the copy is now limited to that case, and removed otherwise. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- internal/controller/tenantquota/reconciler.go | 18 ++++++++-- .../controller/tenantquota/reconciler_test.go | 33 +++++++++++++++++++ 2 files changed, 48 insertions(+), 3 deletions(-) diff --git a/internal/controller/tenantquota/reconciler.go b/internal/controller/tenantquota/reconciler.go index 6c77eff6d..ed18499ce 100644 --- a/internal/controller/tenantquota/reconciler.go +++ b/internal/controller/tenantquota/reconciler.go @@ -113,8 +113,8 @@ func (r *Reconciler) Reconcile(ctx context.Context, _ ctrl.Request) (ctrl.Result pools := ComputePools(tenants) desired := map[string]corev1.ResourceList{} - // limitRangeSource maps each unbounded pool member to the pool root whose - // chart LimitRange it must inherit. + // limitRangeSource maps each unbounded pool member whose quota bounds a + // compute limit to the pool root whose chart LimitRange it must inherit. limitRangeSource := map[string]string{} for _, p := range pools { if over := p.Overcommitted(); len(over) > 0 { @@ -134,7 +134,7 @@ func (r *Reconciler) Reconcile(ctx context.Context, _ ctrl.Request) (ctrl.Result for _, ns := range p.Members { if existing[ns] { desired[ns] = buffered.EnforcedHard(ns, usedByNS) - if ns != p.Root { + if ns != p.Root && boundsComputeLimits(desired[ns]) { limitRangeSource[ns] = p.Root } } @@ -352,6 +352,18 @@ func ownedNamespace(hrNamespace, appName string) string { } } +// boundsComputeLimits reports whether a quota bounds limits.cpu or +// limits.memory, the only case in which admission rejects a container that +// omits a limit. The chart renders its LimitRange under any non-empty +// resourceQuotas, so the root may carry one under a quota that bounds no +// compute at all; copying it then would impose default limits the quota +// never asked for. +func boundsComputeLimits(hard corev1.ResourceList) bool { + _, cpu := hard[corev1.ResourceLimitsCPU] + _, memory := hard[corev1.ResourceLimitsMemory] + return cpu || memory +} + // maxResourceList returns the per-resource maximum of a and b. func maxResourceList(a, b corev1.ResourceList) corev1.ResourceList { out := corev1.ResourceList{} diff --git a/internal/controller/tenantquota/reconciler_test.go b/internal/controller/tenantquota/reconciler_test.go index 4624b2e36..5c5778e63 100644 --- a/internal/controller/tenantquota/reconciler_test.go +++ b/internal/controller/tenantquota/reconciler_test.go @@ -363,3 +363,36 @@ func TestReconcile_PoolRootWithoutLimitRange(t *testing.T) { t.Fatalf("no LimitRange can be copied from a root that has none") } } + +// TestReconcile_NoLimitRangeWithoutComputeLimitQuota: the chart renders its +// LimitRange for any non-empty resourceQuotas, compute or not. A member whose +// allocated quota bounds neither limits.cpu nor limits.memory is admitted +// without limits today; copying the root's defaults there would cap or refuse +// pods (one requesting more than the default memory limit, for instance) +// that the quota itself never constrains. +func TestReconcile_NoLimitRangeWithoutComputeLimitQuota(t *testing.T) { + stale := &corev1.LimitRange{ + ObjectMeta: metav1.ObjectMeta{ + Name: allocatedLimitRangeName, + Namespace: "tenant-foo-bar", + Labels: map[string]string{managedByLabel: managedByValue}, + }, + } + r, c := newReconciler(t, + tenantHR("foo", "tenant-root"), + tenantHR("bar", "tenant-foo"), + ns("tenant-foo"), ns("tenant-foo-bar"), + chartQuota("tenant-foo", map[string]string{"services.loadbalancers": "1"}, nil), + chartLimitRange("tenant-foo", "250m"), + stale, + ) + if _, err := r.Reconcile(context.Background(), sweepKey); err != nil { + t.Fatalf("reconcile: %v", err) + } + if _, ok := allocatedHard(t, c, "tenant-foo-bar", "services.loadbalancers"); !ok { + t.Fatalf("tenant-foo-bar must still get its allocated quota") + } + if _, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName); ok { + t.Fatalf("tenant-foo-bar's quota bounds no compute limit; it must carry no LimitRange copy") + } +} From 21fd2b7b6909811045d40aa5c8140b1757fe6672 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 9 Nov 2026 13:57:59 +0100 Subject: [PATCH 503/518] feat(bucket): add publicRead to serve a bucket without credentials A bucket cannot serve its objects without a signature: every BucketAccess issues a key, and the keys grant neither WriteAcp nor PutBucketPolicy, so a tenant cannot open the bucket themselves with put-bucket-acl. Static assets then need a proxy, presigned URLs or an operator running weed shell by hand, which no reconcile tracks and which outlives the bucket. publicRead renders one more BucketAccess on a new -public access class whose accessPolicy is public-read. The COSI driver handles that policy by adding Read: to the gateway's anonymous identity and removing only that action on revoke, so deleting the bucket or turning the flag off closes it again without touching other public buckets. Anonymous callers can read objects but cannot list the bucket or write to it. The access is named -public-read, the name a user called public-read would take, so that combination fails to render instead of fighting over one BucketAccess. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- api/apps/v1alpha1/bucket/types.go | 3 + packages/apps/bucket/README.md | 15 ++-- .../apps/bucket/templates/bucketclaim.yaml | 15 ++++ .../apps/bucket/tests/public_read_test.yaml | 82 +++++++++++++++++++ packages/apps/bucket/values.schema.json | 5 ++ packages/apps/bucket/values.yaml | 3 + .../templates/client/cosi-bucket-class.yaml | 9 ++ .../storage-pool-bucket-classes.yaml | 9 ++ .../tests/public_access_class_test.yaml | 50 +++++++++++ packages/system/bucket-rd/cozyrds/bucket.yaml | 4 +- .../templates/cosi/cosi-bucket-class.yaml | 9 ++ .../cosi-bucket-class-lock-readonly.patch | 13 ++- .../tests/cosi_public_class_test.yaml | 25 ++++++ 13 files changed, 231 insertions(+), 11 deletions(-) create mode 100644 packages/apps/bucket/tests/public_read_test.yaml create mode 100644 packages/extra/seaweedfs/tests/public_access_class_test.yaml create mode 100644 packages/system/seaweedfs/tests/cosi_public_class_test.yaml diff --git a/api/apps/v1alpha1/bucket/types.go b/api/apps/v1alpha1/bucket/types.go index 6632ca530..8a4f331d5 100644 --- a/api/apps/v1alpha1/bucket/types.go +++ b/api/apps/v1alpha1/bucket/types.go @@ -25,6 +25,9 @@ type ConfigSpec struct { // Deploys the web UI (s3manager) with its Service and its Ingress or HTTPRoute at `.`. Set to `false` to run the bucket without them. // +kubebuilder:default:=true Ui bool `json:"ui"` + // Serves the bucket's objects to anyone without credentials at `//`. Anonymous callers can read objects but cannot list the bucket or write to it. + // +kubebuilder:default:=false + PublicRead bool `json:"publicRead"` // Users configuration map. // +kubebuilder:default:={} Users map[string]User `json:"users,omitempty"` diff --git a/packages/apps/bucket/README.md b/packages/apps/bucket/README.md index 13e52d108..4db481810 100644 --- a/packages/apps/bucket/README.md +++ b/packages/apps/bucket/README.md @@ -4,11 +4,12 @@ ### Parameters -| Name | Description | Type | Value | -| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ------- | -| `locking` | Provisions bucket from the `-lock` BucketClass (with object lock enabled). | `bool` | `false` | -| `storagePool` | Selects a specific BucketClass by storage pool name. | `string` | `""` | -| `ui` | Deploys the web UI (s3manager) with its Service and its Ingress or HTTPRoute at `.`. Set to `false` to run the bucket without them. | `bool` | `true` | -| `users` | Users configuration map. | `map[string]object` | `{}` | -| `users[name].readonly` | Whether the user has read-only access. | `bool` | `false` | +| Name | Description | Type | Value | +| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ------- | +| `locking` | Provisions bucket from the `-lock` BucketClass (with object lock enabled). | `bool` | `false` | +| `storagePool` | Selects a specific BucketClass by storage pool name. | `string` | `""` | +| `ui` | Deploys the web UI (s3manager) with its Service and its Ingress or HTTPRoute at `.`. Set to `false` to run the bucket without them. | `bool` | `true` | +| `publicRead` | Serves the bucket's objects to anyone without credentials at `//`. Anonymous callers can read objects but cannot list the bucket or write to it. | `bool` | `false` | +| `users` | Users configuration map. | `map[string]object` | `{}` | +| `users[name].readonly` | Whether the user has read-only access. | `bool` | `false` | diff --git a/packages/apps/bucket/templates/bucketclaim.yaml b/packages/apps/bucket/templates/bucketclaim.yaml index 0639916f0..8ce03d454 100644 --- a/packages/apps/bucket/templates/bucketclaim.yaml +++ b/packages/apps/bucket/templates/bucketclaim.yaml @@ -22,3 +22,18 @@ spec: credentialsSecretName: {{ $.Release.Name }}-{{ $name }} protocol: s3 {{- end }} +{{- if .Values.publicRead }} +{{- if hasKey .Values.users "public-read" }} +{{- fail "publicRead creates the BucketAccess -public-read, which the user \"public-read\" already owns: rename that user" }} +{{- end }} +--- +apiVersion: objectstorage.k8s.io/v1alpha1 +kind: BucketAccess +metadata: + name: {{ .Release.Name }}-public-read +spec: + bucketAccessClassName: {{ $seaweedfs }}{{- if $pool }}-{{ $pool }}{{- end }}-public + bucketClaimName: {{ .Release.Name }} + credentialsSecretName: {{ .Release.Name }}-public-read + protocol: s3 +{{- end }} diff --git a/packages/apps/bucket/tests/public_read_test.yaml b/packages/apps/bucket/tests/public_read_test.yaml new file mode 100644 index 000000000..1bdf5d82b --- /dev/null +++ b/packages/apps/bucket/tests/public_read_test.yaml @@ -0,0 +1,82 @@ +suite: bucket publicRead +templates: + - templates/bucketclaim.yaml + +release: + name: bucket-my-bucket + namespace: tenant-root + +set: + _namespace: + seaweedfs: tenant-root + +tests: + - it: renders no public BucketAccess by default + asserts: + - hasDocuments: + count: 1 + - isKind: + of: BucketClaim + + - it: renders a public-read BucketAccess on the -public class + set: + publicRead: true + documentSelector: + path: metadata.name + value: bucket-my-bucket-public-read + asserts: + - isKind: + of: BucketAccess + - equal: + path: spec.bucketAccessClassName + value: tenant-root-public + - equal: + path: spec.bucketClaimName + value: bucket-my-bucket + - equal: + path: spec.credentialsSecretName + value: bucket-my-bucket-public-read + + - it: uses the storage pool's -public class + set: + publicRead: true + storagePool: fast + documentSelector: + path: metadata.name + value: bucket-my-bucket-public-read + asserts: + - equal: + path: spec.bucketAccessClassName + value: tenant-root-fast-public + + - it: keeps the access class free of -lock on a locked bucket + set: + publicRead: true + locking: true + documentSelector: + path: metadata.name + value: bucket-my-bucket-public-read + asserts: + - equal: + path: spec.bucketAccessClassName + value: tenant-root-public + + - it: keeps the users' BucketAccesses next to the public one + set: + publicRead: true + users: + app: {} + reader: + readonly: true + asserts: + - hasDocuments: + count: 4 + + - it: refuses a user that would own the public BucketAccess name + set: + publicRead: true + users: + public-read: {} + asserts: + - failedTemplate: + errorPattern: user "public-read" already owns diff --git a/packages/apps/bucket/values.schema.json b/packages/apps/bucket/values.schema.json index 249169395..641822c51 100644 --- a/packages/apps/bucket/values.schema.json +++ b/packages/apps/bucket/values.schema.json @@ -20,6 +20,11 @@ "type": "boolean", "default": true }, + "publicRead": { + "description": "Serves the bucket's objects to anyone without credentials at `\u003cendpoint\u003e/\u003cbucket\u003e/\u003ckey\u003e`. Anonymous callers can read objects but cannot list the bucket or write to it.", + "type": "boolean", + "default": false + }, "users": { "description": "Users configuration map.", "type": "object", diff --git a/packages/apps/bucket/values.yaml b/packages/apps/bucket/values.yaml index 3b79909fc..a5d87b9b3 100644 --- a/packages/apps/bucket/values.yaml +++ b/packages/apps/bucket/values.yaml @@ -8,6 +8,9 @@ storagePool: "" ## @param {bool} ui=true - Deploys the web UI (s3manager) with its Service and its Ingress or HTTPRoute at `.`. Set to `false` to run the bucket without them. ui: true +## @param {bool} publicRead=false - Serves the bucket's objects to anyone without credentials at `//`. Anonymous callers can read objects but cannot list the bucket or write to it. +publicRead: false + ## @typedef {struct} User - Bucket user configuration. ## @field {bool} [readonly] - Whether the user has read-only access. diff --git a/packages/extra/seaweedfs/templates/client/cosi-bucket-class.yaml b/packages/extra/seaweedfs/templates/client/cosi-bucket-class.yaml index ce049b316..0b262f47d 100644 --- a/packages/extra/seaweedfs/templates/client/cosi-bucket-class.yaml +++ b/packages/extra/seaweedfs/templates/client/cosi-bucket-class.yaml @@ -35,4 +35,13 @@ driverName: {{ .Release.Namespace }}.seaweedfs.objectstorage.k8s.io authenticationType: KEY parameters: accessPolicy: readonly +--- +kind: BucketAccessClass +apiVersion: objectstorage.k8s.io/v1alpha1 +metadata: + name: {{ .Release.Namespace }}-public +driverName: {{ .Release.Namespace }}.seaweedfs.objectstorage.k8s.io +authenticationType: KEY +parameters: + accessPolicy: public-read {{- end }} diff --git a/packages/extra/seaweedfs/templates/storage-pool-bucket-classes.yaml b/packages/extra/seaweedfs/templates/storage-pool-bucket-classes.yaml index bf58340a2..faa99f137 100644 --- a/packages/extra/seaweedfs/templates/storage-pool-bucket-classes.yaml +++ b/packages/extra/seaweedfs/templates/storage-pool-bucket-classes.yaml @@ -51,5 +51,14 @@ driverName: {{ $.Release.Namespace }}.seaweedfs.objectstorage.k8s.io authenticationType: KEY parameters: accessPolicy: readonly +--- +kind: BucketAccessClass +apiVersion: objectstorage.k8s.io/v1alpha1 +metadata: + name: {{ $.Release.Namespace }}-{{ $poolName }}-public +driverName: {{ $.Release.Namespace }}.seaweedfs.objectstorage.k8s.io +authenticationType: KEY +parameters: + accessPolicy: public-read {{- end }} {{- end }} diff --git a/packages/extra/seaweedfs/tests/public_access_class_test.yaml b/packages/extra/seaweedfs/tests/public_access_class_test.yaml new file mode 100644 index 000000000..67ea3239c --- /dev/null +++ b/packages/extra/seaweedfs/tests/public_access_class_test.yaml @@ -0,0 +1,50 @@ +suite: seaweedfs public-read BucketAccessClass +release: + name: seaweedfs + namespace: tenant-root + +set: + filer: + grpcHost: seaweedfs-filer.tenant-other.svc + _namespace: + host: example.org + ingress: tenant-root + _cluster: + issuer-name: letsencrypt-prod + solver: http01 + +tests: + - it: offers a -public access class to a Client instance + template: templates/client/cosi-bucket-class.yaml + set: + topology: Client + documentSelector: + path: metadata.name + value: tenant-root-public + asserts: + - isKind: + of: BucketAccessClass + - equal: + path: parameters.accessPolicy + value: public-read + - equal: + path: driverName + value: tenant-root.seaweedfs.objectstorage.k8s.io + + - it: offers a -public access class per storage pool + template: templates/storage-pool-bucket-classes.yaml + set: + topology: Simple + volume: + pools: + fast: + diskType: ssd + documentSelector: + path: metadata.name + value: tenant-root-fast-public + asserts: + - isKind: + of: BucketAccessClass + - equal: + path: parameters.accessPolicy + value: public-read diff --git a/packages/system/bucket-rd/cozyrds/bucket.yaml b/packages/system/bucket-rd/cozyrds/bucket.yaml index f75c736bd..0f05f4dfe 100644 --- a/packages/system/bucket-rd/cozyrds/bucket.yaml +++ b/packages/system/bucket-rd/cozyrds/bucket.yaml @@ -8,7 +8,7 @@ spec: plural: buckets singular: bucket openAPISchema: |- - {"title":"Chart Values","type":"object","properties":{"locking":{"description":"Provisions bucket from the `-lock` BucketClass (with object lock enabled).","type":"boolean","default":false},"storagePool":{"description":"Selects a specific BucketClass by storage pool name.","type":"string","default":"","x-cozystack-options":{"source":"storagepool"}},"ui":{"description":"Deploys the web UI (s3manager) with its Service and its Ingress or HTTPRoute at `.`. Set to `false` to run the bucket without them.","type":"boolean","default":true},"users":{"description":"Users configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"readonly":{"description":"Whether the user has read-only access.","type":"boolean"}}}}}} + {"title":"Chart Values","type":"object","properties":{"locking":{"description":"Provisions bucket from the `-lock` BucketClass (with object lock enabled).","type":"boolean","default":false},"storagePool":{"description":"Selects a specific BucketClass by storage pool name.","type":"string","default":"","x-cozystack-options":{"source":"storagepool"}},"ui":{"description":"Deploys the web UI (s3manager) with its Service and its Ingress or HTTPRoute at `.`. Set to `false` to run the bucket without them.","type":"boolean","default":true},"publicRead":{"description":"Serves the bucket's objects to anyone without credentials at `//`. Anonymous callers can read objects but cannot list the bucket or write to it.","type":"boolean","default":false},"users":{"description":"Users configuration map.","type":"object","default":{},"additionalProperties":{"type":"object","properties":{"readonly":{"description":"Whether the user has read-only access.","type":"boolean"}}}}}} release: prefix: bucket- labels: @@ -26,7 +26,7 @@ spec: tags: - storage icon: PHN2ZyB3aWR0aD0iMTQ0IiBoZWlnaHQ9IjE0NCIgdmlld0JveD0iMCAwIDE0NCAxNDQiIGZpbGw9Im5vbmUiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CjxyZWN0IHdpZHRoPSIxNDQiIGhlaWdodD0iMTQ0IiByeD0iMjQiIGZpbGw9InVybCgjcGFpbnQwX2xpbmVhcl82ODNfMzA5MSkiLz4KPHBhdGggZmlsbC1ydWxlPSJldmVub2RkIiBjbGlwLXJ1bGU9ImV2ZW5vZGQiIGQ9Ik03MiAzMC4xNjQxTDExNy45ODMgMzYuNzc4OVY0MC42NzM5QzExNy45ODMgNDYuNDY1MyA5Ny4zODYyIDUxLjEzMzIgNzEuOTgyNyA1MS4xMzMyQzQ2LjU3OTIgNTEuMTMzMiAyNiA0Ni40NjUzIDI2IDQwLjY3MzlWMzYuNDQzMUw3MiAzMC4xNjQxWk03MiA1OC4yNjc4QzkxLjIwODQgNTguMjY3OCAxMDcuNjU4IDU1LjU5ODYgMTE0LjU0NyA1MS44MDQ4TDExNi44MDMgNDguMTExTDExNy43MjMgNDQuNzUzVjQ4LjkxNzFMMTAyLjY3OSAxMTEuMDMzQzEwMi42NzkgMTE0Ljg5NSA4OC45NTMzIDExOCA3Mi4wMTcyIDExOEM1NS4wODEyIDExOCA0MS4zNzQzIDExNC44OTUgNDEuMzc0MyAxMTEuMDMzTDI2LjMzIDQ4LjkxNzFWNDQuODM2OUwyOS44MDA3IDUxLjkzODJDMzYuNzA2NSA1NS42NjUzIDUyLjk5OTcgNTguMjY3OCA3MiA1OC4yNjc4WiIgZmlsbD0iIzhDMzEyMyIvPgo8cGF0aCBmaWxsLXJ1bGU9ImV2ZW5vZGQiIGNsaXAtcnVsZT0iZXZlbm9kZCIgZD0iTTcyLjAwMDMgMjZDOTcuNDAzOCAyNiAxMTggMzAuNjgzOSAxMTggMzYuNDQyQzExOCA0Mi4yIDk3LjM4NjYgNDYuODUwNyA3Mi4wMDAzIDQ2Ljg1MDdDNDYuNjE0MSA0Ni44NTA3IDI2LjAxNzYgNDIuMjM0NSAyNi4wMTc2IDM2LjQ0MkMyNi4wMTc2IDMwLjY0OTQgNDYuNTk2OCAyNiA3Mi4wMDAzIDI2Wk03Mi4wMDAzIDU0LjEwMzdDOTUuNjg1NyA1NC4xMDM3IDExNS4xNzIgNTAuMDU4IDExNy43MDYgNDQuODE5N0wxMDIuNjYyIDEwNi45MzdDMTAyLjY2MiAxMTAuNzk5IDg4LjkzNjQgMTEzLjkwNSA3Mi4wMDAzIDExMy45MDVDNTUuMDY0MyAxMTMuOTA1IDQxLjMzOSAxMTAuODE2IDQxLjMzOSAxMDYuOTU0TDI2LjI5NTkgNDQuODM3QzI4Ljg0NjYgNTAuMDU4IDQ4LjMzMzMgNTQuMTAzNyA3Mi4wMDAzIDU0LjEwMzdaIiBmaWxsPSIjRTA1MjQzIi8+CjxwYXRoIGZpbGwtcnVsZT0iZXZlbm9kZCIgY2xpcC1ydWxlPSJldmVub2RkIiBkPSJNNjEuMTcyNSA2MC4wMjkzSDgxLjA5MjhWNzkuMTY3Nkg2MS4xNzI1VjYwLjAyOTNaTTQ1LjMzMDEgOTUuMzY4OEM0NS4zMzAxIDkwLjE0MiA0OS43MTA0IDg1LjkzNDIgNTUuMTUxMSA4NS45MzQyQzYwLjU5MTcgODUuOTM0MiA2NC45NzIxIDkwLjE0MiA2NC45NzIxIDk1LjM2ODhDNjQuOTcyMSAxMDAuNTk2IDYwLjU5MTcgMTA0LjgwMyA1NS4xNTExIDEwNC44MDNDNDkuNzEwNCAxMDQuODAzIDQ1LjMzMDEgMTAwLjU5NiA0NS4zMzAxIDk1LjM2ODhaTTk2LjQ0ODcgMTA0LjM2OEg3Ni43NzIyTDg2LjYxMDUgODYuNzczN0w5Ni40NDg3IDEwNC4zNjhaIiBmaWxsPSJ3aGl0ZSIvPgo8ZGVmcz4KPGxpbmVhckdyYWRpZW50IGlkPSJwYWludDBfbGluZWFyXzY4M18zMDkxIiB4MT0iMCIgeTE9IjAiIHgyPSIxNTEiIHkyPSIxODAiIGdyYWRpZW50VW5pdHM9InVzZXJTcGFjZU9uVXNlIj4KPHN0b3Agc3RvcC1jb2xvcj0iI0ZGRjBFRSIvPgo8c3RvcCBvZmZzZXQ9IjEiIHN0b3AtY29sb3I9IiNFQzg4N0QiLz4KPC9saW5lYXJHcmFkaWVudD4KPC9kZWZzPgo8L3N2Zz4K - keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "locking"], ["spec", "storagePool"], ["spec", "ui"], ["spec", "users"]] + keysOrder: [["apiVersion"], ["appVersion"], ["kind"], ["metadata"], ["metadata", "name"], ["spec", "locking"], ["spec", "storagePool"], ["spec", "ui"], ["spec", "publicRead"], ["spec", "users"]] secrets: exclude: [] include: diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml index 1ff0e7768..11d7fb779 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml @@ -39,4 +39,13 @@ driverName: {{ .Values.cosi.driverName }} authenticationType: KEY parameters: accessPolicy: readonly +--- +kind: BucketAccessClass +apiVersion: objectstorage.k8s.io/v1alpha1 +metadata: + name: {{ .Values.cosi.bucketClassName }}-public +driverName: {{ .Values.cosi.driverName }} +authenticationType: KEY +parameters: + accessPolicy: public-read {{- end }} diff --git a/packages/system/seaweedfs/patches/cosi-bucket-class-lock-readonly.patch b/packages/system/seaweedfs/patches/cosi-bucket-class-lock-readonly.patch index ddca7873e..84ca4fcf2 100644 --- a/packages/system/seaweedfs/patches/cosi-bucket-class-lock-readonly.patch +++ b/packages/system/seaweedfs/patches/cosi-bucket-class-lock-readonly.patch @@ -1,8 +1,8 @@ diff --git a/packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml b/packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml -index 4d19cc128..1ff0e7768 100644 +index 4d19cc128..11d7fb779 100644 --- a/packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml +++ b/packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml -@@ -11,10 +11,32 @@ parameters: +@@ -11,10 +11,41 @@ {{- toYaml . | nindent 2 }} {{- end }} --- @@ -34,4 +34,13 @@ index 4d19cc128..1ff0e7768 100644 +authenticationType: KEY +parameters: + accessPolicy: readonly ++--- ++kind: BucketAccessClass ++apiVersion: objectstorage.k8s.io/v1alpha1 ++metadata: ++ name: {{ .Values.cosi.bucketClassName }}-public ++driverName: {{ .Values.cosi.driverName }} ++authenticationType: KEY ++parameters: ++ accessPolicy: public-read {{- end }} diff --git a/packages/system/seaweedfs/tests/cosi_public_class_test.yaml b/packages/system/seaweedfs/tests/cosi_public_class_test.yaml new file mode 100644 index 000000000..72c815e0d --- /dev/null +++ b/packages/system/seaweedfs/tests/cosi_public_class_test.yaml @@ -0,0 +1,25 @@ +suite: vendored COSI classes include public-read +templates: + - charts/seaweedfs/templates/cosi/cosi-bucket-class.yaml + +release: + name: seaweedfs-system + namespace: tenant-root + +tests: + - it: renders a -public BucketAccessClass with the public-read policy + set: + seaweedfs: + cosi: + enabled: true + bucketClassName: tenant-root + driverName: tenant-root.seaweedfs.objectstorage.k8s.io + documentSelector: + path: metadata.name + value: tenant-root-public + asserts: + - isKind: + of: BucketAccessClass + - equal: + path: parameters.accessPolicy + value: public-read From c2d370e95c621dab9910658bacb7368ccf62dda0 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 13:02:13 +0300 Subject: [PATCH 504/518] fix(seaweedfs): pin the COSI driver release with public-read access The public BucketAccessClass relies on the public-read accessPolicy, which the driver supports from v0.5.0; v0.4.0 would hand out keys instead of anonymous read access. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag | 2 +- packages/system/seaweedfs/tests/cosi_image_test.yaml | 8 ++++---- packages/system/seaweedfs/values.yaml | 5 +++-- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag b/packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag index bf4084277..b84ea7484 100644 --- a/packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag +++ b/packages/extra/seaweedfs/images/seaweedfs-cosi-driver.tag @@ -1 +1 @@ -ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.4.0 +ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.5.0 diff --git a/packages/system/seaweedfs/tests/cosi_image_test.yaml b/packages/system/seaweedfs/tests/cosi_image_test.yaml index d43aa176e..4057c989e 100644 --- a/packages/system/seaweedfs/tests/cosi_image_test.yaml +++ b/packages/system/seaweedfs/tests/cosi_image_test.yaml @@ -7,8 +7,8 @@ suite: seaweedfs COSI driver image pin # BucketAccess and ignores the accessPolicy parameter. The chart ships a # -readonly BucketAccessClass with parameters.accessPolicy: readonly, so on # v0.1.2 a credential meant to be read-only is silently granted read-write. -# The package values.yaml overrides cosi.image to v0.4.0, which honours -# accessPolicy: readonly. +# The package values.yaml overrides cosi.image to v0.5.0, which honours +# accessPolicy: readonly and public-read. # # This test does not set the image itself; it relies on the package values.yaml # override, so it goes red if that override is removed and the image falls back @@ -25,7 +25,7 @@ release: namespace: cozy-seaweedfs tests: - - it: pins the cosi-driver container to v0.4.0 (read-only access enforcement) + - it: pins the cosi-driver container to v0.5.0 (read-only and public-read access policies) template: charts/seaweedfs/templates/cosi/cosi-deployment.yaml set: seaweedfs.cosi.enabled: true @@ -35,4 +35,4 @@ tests: value: seaweedfs-cosi-driver - equal: path: spec.template.spec.containers[0].image - value: ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.4.0 + value: ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.5.0 diff --git a/packages/system/seaweedfs/values.yaml b/packages/system/seaweedfs/values.yaml index 64e2562ba..24881e6f8 100644 --- a/packages/system/seaweedfs/values.yaml +++ b/packages/system/seaweedfs/values.yaml @@ -313,8 +313,9 @@ seaweedfs: # actually issues read-only (Read,List) credentials instead of read-write. # v0.4.0 also writes each grant to the per-identity IAM file: v0.3.1 # rewrites only the legacy identity.json, which the S3 gateways migrate - # concurrently, so a grant could be lost for good. - image: "ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.4.0" + # concurrently, so a grant could be lost for good. v0.5.0 adds the + # public-read accessPolicy the -public BucketAccessClass relies on. + image: "ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.5.0" podLabels: policy.cozystack.io/allow-to-apiserver: "true" driverName: "seaweedfs.objectstorage.k8s.io" From fa1cf776708758aac18f5daf5a64462a2d478b9f Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Fri, 13 Nov 2026 11:04:57 +0100 Subject: [PATCH 505/518] fix(seaweedfs): reserve the -public suffix in pool names A pool's classes take its name as a prefix, so a pool named -public renders a readwrite BucketAccessClass --public with the same name as pool 's public-read class, and whichever object wins gives one of the two pools the wrong policy. A pool named public takes -public, the instance's own public-read class. The pool name check already reserves -lock and -readonly for the same reason. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../extra/seaweedfs/templates/seaweedfs.yaml | 8 +-- .../tests/public_access_class_test.yaml | 63 +++++++++++++++++++ 2 files changed, 67 insertions(+), 4 deletions(-) diff --git a/packages/extra/seaweedfs/templates/seaweedfs.yaml b/packages/extra/seaweedfs/templates/seaweedfs.yaml index 92b40ef4d..ed24b8fe4 100644 --- a/packages/extra/seaweedfs/templates/seaweedfs.yaml +++ b/packages/extra/seaweedfs/templates/seaweedfs.yaml @@ -32,8 +32,8 @@ {{- if not (regexMatch "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$" $poolName) }} {{- fail (printf "volume.pools key '%s' must be a valid DNS label (lowercase alphanumeric and hyphens, no dots)." $poolName) }} {{- end }} -{{- if or (hasSuffix "-lock" $poolName) (hasSuffix "-readonly" $poolName) }} -{{- fail (printf "volume.pools key '%s' must not end with '-lock' or '-readonly' (reserved suffixes for COSI resources)." $poolName) }} +{{- if or (hasSuffix "-lock" $poolName) (hasSuffix "-readonly" $poolName) (hasSuffix "-public" $poolName) (eq "public" $poolName) }} +{{- fail (printf "volume.pools key '%s' must not be 'public' or end with '-lock', '-readonly' or '-public' (reserved for COSI resources)." $poolName) }} {{- end }} {{- if not $pool.diskType }} {{- fail (printf "volume.pools.%s.diskType is required." $poolName) }} @@ -52,8 +52,8 @@ {{- if not (regexMatch "^[a-z0-9]([a-z0-9-]*[a-z0-9])?$" $poolName) }} {{- fail (printf "volume.zones.%s.pools key '%s' must be a valid DNS label." $zoneName $poolName) }} {{- end }} -{{- if or (hasSuffix "-lock" $poolName) (hasSuffix "-readonly" $poolName) }} -{{- fail (printf "volume.zones.%s.pools key '%s' must not end with '-lock' or '-readonly' (reserved suffixes for COSI resources)." $zoneName $poolName) }} +{{- if or (hasSuffix "-lock" $poolName) (hasSuffix "-readonly" $poolName) (hasSuffix "-public" $poolName) (eq "public" $poolName) }} +{{- fail (printf "volume.zones.%s.pools key '%s' must not be 'public' or end with '-lock', '-readonly' or '-public' (reserved for COSI resources)." $zoneName $poolName) }} {{- end }} {{- if not $pool.diskType }} {{- fail (printf "volume.zones.%s.pools.%s.diskType is required." $zoneName $poolName) }} diff --git a/packages/extra/seaweedfs/tests/public_access_class_test.yaml b/packages/extra/seaweedfs/tests/public_access_class_test.yaml index 67ea3239c..dbbcc84e7 100644 --- a/packages/extra/seaweedfs/tests/public_access_class_test.yaml +++ b/packages/extra/seaweedfs/tests/public_access_class_test.yaml @@ -48,3 +48,66 @@ tests: - equal: path: parameters.accessPolicy value: public-read + + # A pool named -public would render a readwrite BucketAccessClass + # --public, the same name as pool 's public-read class; a pool named + # public would take -public, the instance's own public-read class. + - it: refuses a pool name ending in -public + template: templates/seaweedfs.yaml + set: + volume: + pools: + fast: + diskType: ssd + fast-public: + diskType: nvme + asserts: + - failedTemplate: + errorPattern: volume.pools key 'fast-public' must not be 'public' or end with + + - it: refuses a pool named public + template: templates/seaweedfs.yaml + set: + volume: + pools: + public: + diskType: ssd + asserts: + - failedTemplate: + errorPattern: volume.pools key 'public' must not be 'public' or end with + + - it: refuses a zone pool name ending in -public + template: templates/seaweedfs.yaml + set: + topology: MultiZone + replicationFactor: 1 + volume: + zones: + a: + pools: + fast-public: + diskType: ssd + asserts: + - failedTemplate: + errorPattern: volume.zones.a.pools key 'fast-public' must not be 'public' or end with + + - it: still refuses a pool name ending in -readonly + template: templates/seaweedfs.yaml + set: + volume: + pools: + fast-readonly: + diskType: ssd + asserts: + - failedTemplate: + errorPattern: volume.pools key 'fast-readonly' must not be 'public' or end with + + - it: accepts a pool name that only contains public + template: templates/seaweedfs.yaml + set: + volume: + pools: + publicfast: + diskType: ssd + asserts: + - notFailedTemplate: {} From 0fed7c51d271fa56ae58b02b0e13c5d72a51ac52 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Fri, 13 Nov 2026 15:22:34 +0500 Subject: [PATCH 506/518] fix(api): warn when a Kubernetes spec still carries talos The Kubernetes chart no longer reads talos, but a stored or hand-edited spec that keeps the block is accepted without a word. List it with the other keys the worker pool split made inert, so the API answers with the same warning it gives for nodeGroups. Assisted-by: LLM Signed-off-by: Myasnikov Daniil --- pkg/registry/apps/application/rest.go | 2 +- pkg/registry/apps/application/rest_validation_test.go | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/pkg/registry/apps/application/rest.go b/pkg/registry/apps/application/rest.go index aa0e24a62..6cd544994 100644 --- a/pkg/registry/apps/application/rest.go +++ b/pkg/registry/apps/application/rest.go @@ -2155,7 +2155,7 @@ const kubernetesKind = "Kubernetes" // upgraded cluster keeps them in its values until re-edited, and the adoption // migration deliberately leaves them in place rather than rewriting parent // values mid-upgrade -- but they no longer render anything on the Kubernetes CR. -var removedKubernetesFields = []string{"nodeGroups", "nodeHealthCheck", "maxNodeProvisionTime"} +var removedKubernetesFields = []string{"nodeGroups", "nodeHealthCheck", "maxNodeProvisionTime", "talos"} // warnRemovedKubernetesFields emits a client-facing admission warning for each // Phase 2-removed field still present in a Kubernetes CR's values, so an diff --git a/pkg/registry/apps/application/rest_validation_test.go b/pkg/registry/apps/application/rest_validation_test.go index f48f40d26..51a9f48ce 100644 --- a/pkg/registry/apps/application/rest_validation_test.go +++ b/pkg/registry/apps/application/rest_validation_test.go @@ -466,6 +466,12 @@ func TestWarnRemovedKubernetesFields(t *testing.T) { specJSON: `{"nodeGroups":{},"nodeHealthCheck":{"maxUnhealthy":"50%"},"maxNodeProvisionTime":"10m"}`, wantKeys: []string{"nodeGroups", "nodeHealthCheck", "maxNodeProvisionTime"}, }, + { + name: "kubernetes with a stored talos block warns", + kindName: "Kubernetes", + specJSON: `{"version":"v1.35","talos":{"version":"v1.13.6"}}`, + wantKeys: []string{"talos"}, + }, { name: "kubernetes without removed fields does not warn", kindName: "Kubernetes", From e3e20180752219d23aff6c1ac26b909b9085b965 Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Mon, 14 Sep 2026 10:25:13 +0200 Subject: [PATCH 507/518] fix(seaweedfs, cozy-lib): cap the Go heap so components survive their memory limit Every SeaweedFS component is a Go binary running under a cgroup memory limit, and none of them set GOMEMLIMIT. The collector then works from GOGC alone, a target relative to the live heap with no absolute ceiling, so a burst of allocation grows the heap past the limit and the kernel kills the container. Seen on an S3 gateway holding 6.4 GiB of heap_sys against an 8 GiB limit while only 389 MiB was actually in use: ten kills in seven days, each one an interruption of the object store every tenant writes its backups to. Raising the limit does not help, since nothing bounds the growth; it only moves the ceiling the heap climbs to. Derive the value from the limit already computed for each component, at the same 75% the javaHeap helper uses for the JVM. The remaining quarter covers what GOMEMLIMIT does not account for, goroutine stacks and mmap'd regions among them. Volume pools and zones carry their own value: the subchart merges volume into every entry of volumes, so a pool would otherwise inherit the value sized for the default volume's limit, above its own. The lifecycle admin and worker run the same binary under limits the parent chart does not set, so their value is pinned next to those limits. A limit under 2Mi floors to 0MiB, which Go accepts and then collects almost continuously; the render fails instead. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../extra/seaweedfs/templates/seaweedfs.yaml | 40 ++++- .../seaweedfs/tests/gomemlimit_test.yaml | 162 ++++++++++++++++++ .../library/cozy-lib/templates/_resources.tpl | 20 +++ .../seaweedfs/tests/lifecycle_test.yaml | 30 ++++ packages/system/seaweedfs/values.yaml | 8 + .../templates/tests/resources-gomemlimit.yaml | 8 + .../tests/resources_gomemlimit_test.yaml | 61 +++++++ .../tests/resources_gomemlimit_values.yaml | 16 ++ 8 files changed, 338 insertions(+), 7 deletions(-) create mode 100644 packages/extra/seaweedfs/tests/gomemlimit_test.yaml create mode 100644 packages/tests/cozy-lib-tests/templates/tests/resources-gomemlimit.yaml create mode 100644 packages/tests/cozy-lib-tests/tests/resources_gomemlimit_test.yaml create mode 100644 packages/tests/cozy-lib-tests/tests/resources_gomemlimit_values.yaml diff --git a/packages/extra/seaweedfs/templates/seaweedfs.yaml b/packages/extra/seaweedfs/templates/seaweedfs.yaml index 92b40ef4d..d69a3e493 100644 --- a/packages/extra/seaweedfs/templates/seaweedfs.yaml +++ b/packages/extra/seaweedfs/templates/seaweedfs.yaml @@ -248,13 +248,19 @@ spec: {{- /* An explicit null is dropped before schema validation, so neither bound sees it, and the master exits on the empty flag it renders. */}} volumeSizeLimitMB: {{ required "master.volumeSizeLimitMB must be set" .Values.master.volumeSizeLimitMB }} - resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.master.resourcesPreset .Values.master.resources $) | nindent 10 }} +{{- $masterResources := include "cozy-lib.resources.defaultingSanitize" (list .Values.master.resourcesPreset .Values.master.resources $) | fromYaml }} + resources: {{- toYaml $masterResources | nindent 10 }} + extraEnvironmentVars: + GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $masterResources | quote }} volume: {{ if eq .Values.topology "MultiZone" }} enabled: false {{- end }} replicas: {{ .Values.volume.replicas }} - resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.volume.resourcesPreset .Values.volume.resources $) | nindent 10 }} +{{- $volumeResources := include "cozy-lib.resources.defaultingSanitize" (list .Values.volume.resourcesPreset .Values.volume.resources $) | fromYaml }} + resources: {{- toYaml $volumeResources | nindent 10 }} + extraEnvironmentVars: + GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $volumeResources | quote }} dataDirs: - name: data1 type: "persistentVolumeClaim" @@ -273,7 +279,13 @@ spec: {{- range $poolName, $pool := .Values.volume.pools }} {{ $poolName }}: replicas: {{ ternary $pool.replicas $.Values.volume.replicas (hasKey $pool "replicas") }} - resources: {{- include "cozy-lib.resources.defaultingSanitize" (list ($pool.resourcesPreset | default $.Values.volume.resourcesPreset) (default dict $pool.resources) $) | nindent 12 }} +{{- $poolResources := include "cozy-lib.resources.defaultingSanitize" (list ($pool.resourcesPreset | default $.Values.volume.resourcesPreset) (default dict $pool.resources) $) | fromYaml }} + resources: {{- toYaml $poolResources | nindent 12 }} + {{- /* The subchart merges volume into every entry of volumes, so a + pool that does not set its own value runs on the default + volume's, sized for a different limit. */}} + extraEnvironmentVars: + GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $poolResources | quote }} dataDirs: - name: data1 type: "persistentVolumeClaim" @@ -289,7 +301,9 @@ spec: {{- range $zoneName, $zone := .Values.volume.zones }} {{ $zoneName }}: replicas: {{ ternary $zone.replicas $.Values.volume.replicas (hasKey $zone "replicas") }} - resources: {{- include "cozy-lib.resources.defaultingSanitize" (list $.Values.volume.resourcesPreset $.Values.volume.resources $) | nindent 12 }} + resources: {{- toYaml $volumeResources | nindent 12 }} + extraEnvironmentVars: + GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $volumeResources | quote }} dataDirs: - name: data1 type: "persistentVolumeClaim" @@ -314,7 +328,13 @@ spec: {{- range $poolName, $pool := (dig "pools" dict $zone) }} {{ $zoneName }}-{{ $poolName }}: replicas: {{ ternary $pool.replicas (ternary $zone.replicas $.Values.volume.replicas (hasKey $zone "replicas")) (hasKey $pool "replicas") }} - resources: {{- include "cozy-lib.resources.defaultingSanitize" (list ($pool.resourcesPreset | default $.Values.volume.resourcesPreset) (default dict $pool.resources) $) | nindent 12 }} +{{- $poolResources := include "cozy-lib.resources.defaultingSanitize" (list ($pool.resourcesPreset | default $.Values.volume.resourcesPreset) (default dict $pool.resources) $) | fromYaml }} + resources: {{- toYaml $poolResources | nindent 12 }} + {{- /* The subchart merges volume into every entry of volumes, so a + pool that does not set its own value runs on the default + volume's, sized for a different limit. */}} + extraEnvironmentVars: + GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $poolResources | quote }} dataDirs: - name: data1 type: "persistentVolumeClaim" @@ -345,7 +365,10 @@ spec: replicas: {{ .Values.filer.replicas }} s3: domainName: {{ .Values.host | default (printf "s3.%s" $host) }} - resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.filer.resourcesPreset .Values.filer.resources $) | nindent 10 }} +{{- $filerResources := include "cozy-lib.resources.defaultingSanitize" (list .Values.filer.resourcesPreset .Values.filer.resources $) | fromYaml }} + resources: {{- toYaml $filerResources | nindent 10 }} + extraEnvironmentVars: + GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $filerResources | quote }} s3: replicas: {{ .Values.s3.replicas }} ingress: @@ -381,7 +404,10 @@ spec: {{- if not $wildcardSecret }} secretName: {{ .Release.Name }}-s3-ingress-tls {{- end }} - resources: {{- include "cozy-lib.resources.defaultingSanitize" (list .Values.s3.resourcesPreset .Values.s3.resources $) | nindent 10 }} +{{- $s3Resources := include "cozy-lib.resources.defaultingSanitize" (list .Values.s3.resourcesPreset .Values.s3.resources $) | fromYaml }} + resources: {{- toYaml $s3Resources | nindent 10 }} + extraEnvironmentVars: + GOMEMLIMIT: {{ include "cozy-lib.resources.goMemLimit" $s3Resources | quote }} admin: enabled: {{ .Values.lifecycle.enabled }} secret: diff --git a/packages/extra/seaweedfs/tests/gomemlimit_test.yaml b/packages/extra/seaweedfs/tests/gomemlimit_test.yaml new file mode 100644 index 000000000..2b298b10e --- /dev/null +++ b/packages/extra/seaweedfs/tests/gomemlimit_test.yaml @@ -0,0 +1,162 @@ +suite: seaweedfs components carry a GOMEMLIMIT derived from their memory limit +templates: + - templates/seaweedfs.yaml + +# Every SeaweedFS component is a Go binary. Without GOMEMLIMIT the runtime +# collects on a relative target and keeps no absolute ceiling, so a burst of +# allocation grows the heap past the cgroup limit and the kernel kills the +# container. Each component sizes its own value from its own memory limit, +# so the assertions below track the limit rather than a fixed string. + +tests: + - it: derives GOMEMLIMIT from the default preset on every component + documentSelector: + path: kind + value: HelmRelease + set: + topology: Simple + replicationFactor: 2 + _namespace: + host: example.org + ingress: tenant-root + _cluster: + issuer-name: letsencrypt-prod + solver: http01 + asserts: + - equal: + path: spec.values.seaweedfs.master.resources.limits.memory + value: 512Mi + - equal: + path: spec.values.seaweedfs.master.extraEnvironmentVars.GOMEMLIMIT + value: 384MiB + - equal: + path: spec.values.seaweedfs.volume.extraEnvironmentVars.GOMEMLIMIT + value: 384MiB + - equal: + path: spec.values.seaweedfs.filer.extraEnvironmentVars.GOMEMLIMIT + value: 384MiB + - equal: + path: spec.values.seaweedfs.s3.extraEnvironmentVars.GOMEMLIMIT + value: 384MiB + + - it: follows the memory limit when a component is resized + documentSelector: + path: kind + value: HelmRelease + set: + topology: Simple + replicationFactor: 2 + _namespace: + host: example.org + ingress: tenant-root + _cluster: + issuer-name: letsencrypt-prod + solver: http01 + s3: + resources: + memory: 8Gi + volume: + resources: + memory: 2Gi + asserts: + - equal: + path: spec.values.seaweedfs.s3.resources.limits.memory + value: 8Gi + - equal: + path: spec.values.seaweedfs.s3.extraEnvironmentVars.GOMEMLIMIT + value: 6144MiB + - equal: + path: spec.values.seaweedfs.volume.extraEnvironmentVars.GOMEMLIMIT + value: 1536MiB + - equal: + path: spec.values.seaweedfs.master.extraEnvironmentVars.GOMEMLIMIT + value: 384MiB + + - it: keeps GOMEMLIMIT below the cgroup limit it protects + documentSelector: + path: kind + value: HelmRelease + set: + topology: Simple + replicationFactor: 2 + _namespace: + host: example.org + ingress: tenant-root + _cluster: + issuer-name: letsencrypt-prod + solver: http01 + s3: + resources: + memory: 1Gi + asserts: + - equal: + path: spec.values.seaweedfs.s3.resources.limits.memory + value: 1Gi + - equal: + path: spec.values.seaweedfs.s3.extraEnvironmentVars.GOMEMLIMIT + value: 768MiB + + # The subchart merges volume into every entry of volumes, so a pool that + # did not carry its own value would run on the default volume's, sized for + # a larger limit than its own. + - it: sizes a Simple pool from its own limit, not the default volume's + documentSelector: + path: kind + value: HelmRelease + set: + topology: Simple + replicationFactor: 2 + _namespace: + host: example.org + ingress: tenant-root + _cluster: + issuer-name: letsencrypt-prod + solver: http01 + volume: + pools: + small: + diskType: ssd + resourcesPreset: t1.nano + asserts: + - equal: + path: spec.values.seaweedfs.volumes.small.resources.limits.memory + value: 128Mi + - equal: + path: spec.values.seaweedfs.volumes.small.extraEnvironmentVars.GOMEMLIMIT + value: 96MiB + - equal: + path: spec.values.seaweedfs.volume.extraEnvironmentVars.GOMEMLIMIT + value: 384MiB + + - it: sizes zones and zone pools from their own limits in MultiZone + documentSelector: + path: kind + value: HelmRelease + set: + topology: MultiZone + replicationFactor: 1 + _namespace: + host: example.org + ingress: tenant-root + _cluster: + issuer-name: letsencrypt-prod + solver: http01 + volume: + resources: + memory: 2Gi + zones: + a: + pools: + small: + diskType: ssd + resourcesPreset: t1.nano + asserts: + - equal: + path: spec.values.seaweedfs.volumes.a.extraEnvironmentVars.GOMEMLIMIT + value: 1536MiB + - equal: + path: spec.values.seaweedfs.volumes.a-small.resources.limits.memory + value: 128Mi + - equal: + path: spec.values.seaweedfs.volumes.a-small.extraEnvironmentVars.GOMEMLIMIT + value: 96MiB diff --git a/packages/library/cozy-lib/templates/_resources.tpl b/packages/library/cozy-lib/templates/_resources.tpl index 48b0718fd..9b621fe0d 100644 --- a/packages/library/cozy-lib/templates/_resources.tpl +++ b/packages/library/cozy-lib/templates/_resources.tpl @@ -202,6 +202,26 @@ {{- printf `-Xms%dm -Xmx%dm` $xmsMi $xmxMi }} {{- end }} +{{- /* + goMemLimit takes a sanitized .Values.resources and returns a GOMEMLIMIT + value set to 75% of the memory limit. Without it the Go runtime collects + on a relative target (GOGC) with no absolute ceiling, so a burst of + allocation grows the heap past the cgroup limit and the kernel kills the + container. The 25% headroom covers what GOMEMLIMIT does not account for: + goroutine stacks and any mmap'd region. Accepts only sanitized resource + maps. +*/}} +{{- define "cozy-lib.resources.goMemLimit" }} +{{- $memoryLimitInt := include "cozy-lib.resources.toFloat" .limits.memory | float64 | int64 }} +{{- /* 4194304 is 4Mi */}} +{{- $limitMiB := div (mul $memoryLimitInt 3) 4194304 }} +{{- /* Go accepts 0, and then collects almost without pause. */}} +{{- if lt $limitMiB 1 }} +{{- fail (printf "ERROR: a memory limit of %s leaves no room for a Go heap; GOMEMLIMIT needs at least 2Mi." .limits.memory) }} +{{- end }} +{{- printf `%dMiB` $limitMiB }} +{{- end }} + {{- define "cozy-lib.resources.flatten" -}} {{- $out := dict -}} {{- $res := include "cozy-lib.resources.sanitize" . | fromYaml -}} diff --git a/packages/system/seaweedfs/tests/lifecycle_test.yaml b/packages/system/seaweedfs/tests/lifecycle_test.yaml index 7f3cec8ff..0fd02b8a8 100644 --- a/packages/system/seaweedfs/tests/lifecycle_test.yaml +++ b/packages/system/seaweedfs/tests/lifecycle_test.yaml @@ -57,6 +57,36 @@ tests: path: spec.template.spec.containers[0].command[2] pattern: '-admin=seaweedfs-admin\.tenant-root:23646\.33646 ' + # Both are the weed binary under a fixed memory limit the parent chart does + # not set, so their GOMEMLIMIT is pinned here rather than derived: without + # one the heap has no ceiling and a large lifecycle pass grows it past the + # limit. + - it: caps the admin heap below its memory limit + template: charts/seaweedfs/templates/admin/admin-statefulset.yaml + set: *enabled + asserts: + - equal: + path: spec.template.spec.containers[0].resources.limits.memory + value: 256Mi + - contains: + path: spec.template.spec.containers[0].env + content: + name: GOMEMLIMIT + value: 192MiB + + - it: caps the worker heap below its memory limit + template: charts/seaweedfs/templates/worker/worker-deployment.yaml + set: *enabled + asserts: + - equal: + path: spec.template.spec.containers[0].resources.limits.memory + value: 512Mi + - contains: + path: spec.template.spec.containers[0].env + content: + name: GOMEMLIMIT + value: 384MiB + # The worker executes every job type it registers for. Lifecycle is the only # one this release asks for: vacuum, balance and erasure coding move data and # are not something enabling lifecycle should start doing as a side effect. diff --git a/packages/system/seaweedfs/values.yaml b/packages/system/seaweedfs/values.yaml index 64e2562ba..81b486002 100644 --- a/packages/system/seaweedfs/values.yaml +++ b/packages/system/seaweedfs/values.yaml @@ -349,6 +349,11 @@ seaweedfs: limits: cpu: "500m" memory: "256Mi" + # 75% of the limit above, as cozy-lib.resources.goMemLimit derives it for + # the components whose resources the parent chart sets. Without it the + # heap has no absolute ceiling and grows past the limit. + extraEnvironmentVars: + GOMEMLIMIT: "192MiB" worker: enabled: false # Vacuum, balance and erasure coding move data; enabling lifecycle must @@ -378,6 +383,9 @@ seaweedfs: limits: cpu: "1" memory: "512Mi" + # 75% of the limit above; see admin.extraEnvironmentVars. + extraEnvironmentVars: + GOMEMLIMIT: "384MiB" certificates: commonName: "SeaweedFS CA" ipAddresses: [] diff --git a/packages/tests/cozy-lib-tests/templates/tests/resources-gomemlimit.yaml b/packages/tests/cozy-lib-tests/templates/tests/resources-gomemlimit.yaml new file mode 100644 index 000000000..f9d3739b0 --- /dev/null +++ b/packages/tests/cozy-lib-tests/templates/tests/resources-gomemlimit.yaml @@ -0,0 +1,8 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: {{ .Release.Name }}-gomemlimit +data: +{{- range $name, $resources := .Values.sanitizedResources }} + {{ $name }}: {{ include "cozy-lib.resources.goMemLimit" $resources | quote }} +{{- end }} diff --git a/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_test.yaml b/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_test.yaml new file mode 100644 index 000000000..17a7d0b97 --- /dev/null +++ b/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_test.yaml @@ -0,0 +1,61 @@ +# ./tests/resources_gomemlimit_test.yaml +suite: resources goMemLimit helper + +# The helper turns a memory limit into a GOMEMLIMIT the Go runtime accepts, +# at 75% of the limit. Two properties matter and neither is visible from a +# single value: the result carries the MiB suffix Go parses, and it stays +# strictly under the limit it protects, whatever unit the quantity came in. +# A value that came out at or above the limit would leave the runtime with +# no headroom and the container would still be killed. + +templates: + - templates/tests/resources-gomemlimit.yaml + +tests: + - it: takes 75% of the limit and keeps the unit Go parses + values: + - resources_gomemlimit_values.yaml + + release: + name: myrelease + namespace: default + + asserts: + - equal: + path: data.binaryGi + value: "6144MiB" + + - equal: + path: data.binaryMi + value: "384MiB" + + # A decimal quantity is smaller than its binary lookalike: 1G is + # 1000000000 bytes, so 75% of it lands below 768MiB. + - equal: + path: data.decimalG + value: "715MiB" + + - equal: + path: data.bareBytes + value: "768MiB" + + # The smallest limit that still yields a usable value: 75% of 2Mi + # floors to 1MiB. + - equal: + path: data.twoMiB + value: "1MiB" + + # Integer division floors, so a limit under 2Mi comes out at 0MiB. Go + # accepts that and then collects almost without pause, slowing the + # component instead of protecting it, so the render stops instead. + - it: refuses a limit too small to leave a Go heap + values: + - resources_gomemlimit_values.yaml + set: + sanitizedResources: + belowOneMiB: + limits: + memory: "1Mi" + asserts: + - failedTemplate: + errorPattern: "leaves no room for a Go heap" diff --git a/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_values.yaml b/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_values.yaml new file mode 100644 index 000000000..3e367cc21 --- /dev/null +++ b/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_values.yaml @@ -0,0 +1,16 @@ +sanitizedResources: + binaryGi: + limits: + memory: "8Gi" + binaryMi: + limits: + memory: "512Mi" + decimalG: + limits: + memory: "1G" + bareBytes: + limits: + memory: "1073741824" + twoMiB: + limits: + memory: "2Mi" From 3b893c626af984810b87acae19dd7377f5eb874f Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Fri, 13 Nov 2026 11:43:13 +0100 Subject: [PATCH 508/518] fix(cozy-lib): state the real lower bound in the goMemLimit error The limit that floors to 0MiB is anything under 4/3 Mi, not under 2Mi: 1536Ki still yields 1MiB. The message named the wrong bound and the tests pinned 2Mi and 1Mi, neither of which sits on the edge. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../library/cozy-lib/templates/_resources.tpl | 2 +- .../tests/resources_gomemlimit_test.yaml | 15 ++++++++------- .../tests/resources_gomemlimit_values.yaml | 4 ++-- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/packages/library/cozy-lib/templates/_resources.tpl b/packages/library/cozy-lib/templates/_resources.tpl index 9b621fe0d..5830e1400 100644 --- a/packages/library/cozy-lib/templates/_resources.tpl +++ b/packages/library/cozy-lib/templates/_resources.tpl @@ -217,7 +217,7 @@ {{- $limitMiB := div (mul $memoryLimitInt 3) 4194304 }} {{- /* Go accepts 0, and then collects almost without pause. */}} {{- if lt $limitMiB 1 }} -{{- fail (printf "ERROR: a memory limit of %s leaves no room for a Go heap; GOMEMLIMIT needs at least 2Mi." .limits.memory) }} +{{- fail (printf "ERROR: a memory limit of %s leaves no room for a Go heap; GOMEMLIMIT needs a limit of at least 1398102 bytes (4/3 Mi) to come out at 1MiB." .limits.memory) }} {{- end }} {{- printf `%dMiB` $limitMiB }} {{- end }} diff --git a/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_test.yaml b/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_test.yaml index 17a7d0b97..1753f925f 100644 --- a/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_test.yaml +++ b/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_test.yaml @@ -39,15 +39,16 @@ tests: path: data.bareBytes value: "768MiB" - # The smallest limit that still yields a usable value: 75% of 2Mi - # floors to 1MiB. + # The smallest limit that still yields a usable value: 75% of + # 1398102 bytes (4/3 Mi, rounded up) floors to 1MiB. - equal: - path: data.twoMiB + path: data.smallestUsable value: "1MiB" - # Integer division floors, so a limit under 2Mi comes out at 0MiB. Go - # accepts that and then collects almost without pause, slowing the - # component instead of protecting it, so the render stops instead. + # Integer division floors, so a limit one byte under 4/3 Mi comes out + # at 0MiB. Go accepts that and then collects almost without pause, + # slowing the component instead of protecting it, so the render stops + # instead. - it: refuses a limit too small to leave a Go heap values: - resources_gomemlimit_values.yaml @@ -55,7 +56,7 @@ tests: sanitizedResources: belowOneMiB: limits: - memory: "1Mi" + memory: "1398101" asserts: - failedTemplate: errorPattern: "leaves no room for a Go heap" diff --git a/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_values.yaml b/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_values.yaml index 3e367cc21..f6be917e0 100644 --- a/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_values.yaml +++ b/packages/tests/cozy-lib-tests/tests/resources_gomemlimit_values.yaml @@ -11,6 +11,6 @@ sanitizedResources: bareBytes: limits: memory: "1073741824" - twoMiB: + smallestUsable: limits: - memory: "2Mi" + memory: "1398102" From cc412d44f2f5dbb7f0ed6686feb0e331c6338bcd Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Fri, 13 Nov 2026 13:07:08 +0100 Subject: [PATCH 509/518] test(controller): exercise the missing-root-LimitRange path under a compute-limit quota TestReconcile_PoolRootWithoutLimitRange declared its root quota on cpu, which the compute-limit gate does not count, so the sweep never looked for the root's LimitRange and the test passed whatever that path did. Declaring limits.cpu makes it reach the lookup, and it now fails if a missing root LimitRange produces an empty copy. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- internal/controller/tenantquota/reconciler_test.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/internal/controller/tenantquota/reconciler_test.go b/internal/controller/tenantquota/reconciler_test.go index 5c5778e63..b869825dc 100644 --- a/internal/controller/tenantquota/reconciler_test.go +++ b/internal/controller/tenantquota/reconciler_test.go @@ -351,12 +351,12 @@ func TestReconcile_PoolRootWithoutLimitRange(t *testing.T) { tenantHR("foo", "tenant-root"), tenantHR("bar", "tenant-foo"), ns("tenant-foo"), ns("tenant-foo-bar"), - chartQuota("tenant-foo", map[string]string{"cpu": "10"}, nil), + chartQuota("tenant-foo", map[string]string{"limits.cpu": "10"}, nil), ) if _, err := r.Reconcile(context.Background(), sweepKey); err != nil { t.Fatalf("reconcile: %v", err) } - if _, ok := allocatedHard(t, c, "tenant-foo-bar", "cpu"); !ok { + if _, ok := allocatedHard(t, c, "tenant-foo-bar", "limits.cpu"); !ok { t.Fatalf("tenant-foo-bar must still get its allocated quota") } if _, ok := getLimitRange(c, "tenant-foo-bar", allocatedLimitRangeName); ok { From f3bf92acc1562f0a33374838b6ea49c741cc2cb2 Mon Sep 17 00:00:00 2001 From: "cozystack-ci[bot]" <274107086+cozystack-ci[bot]@users.noreply.github.com> Date: Fri, 13 Nov 2026 12:57:32 +0000 Subject: [PATCH 510/518] docs: add changelog for v1.7.1 Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> --- docs/changelogs/v1.7.1.md | 82 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 docs/changelogs/v1.7.1.md diff --git a/docs/changelogs/v1.7.1.md b/docs/changelogs/v1.7.1.md new file mode 100644 index 000000000..500735d9f --- /dev/null +++ b/docs/changelogs/v1.7.1.md @@ -0,0 +1,82 @@ + + +# Cozystack v1.7.1 + +Cozystack v1.7.1 is a patch release on the v1.7 line. It fixes the COSI driver bug that could leave freshly granted S3 keys rejected for good, makes marketplace taps digest-pinned, makes MariaDB, PostgreSQL and ClickHouse backups report their real outcome, and repairs several tenant-facing issues: tenant cluster metrics, SecurityGroups on Kubernetes workers, Redis with a single replica, the Proxmox cloud-controller-manager and the tcp-balancer. Read the Migration and Upgrades section before upgrading. + +## Fixes + +* **SeaweedFS COSI driver v0.4.0**: bucket users granted together could get S3 keys that never work (`403 InvalidAccessKeyId`) because the v0.3.1 driver rewrote a legacy IAM file the S3 gateways were migrating. The driver is bumped to v0.4.0, which writes each grant to a per-identity IAM file. Keys already lost on running clusters stay broken: remove the user from the Bucket, wait until its BucketAccess is gone, and add it back ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #5284, backport #5290). +* **Marketplace taps are pinned to a digest**: a tapped repository no longer follows a mutable tag, so moving the tag upstream cannot change what runs in the cluster. `tap connect` accepts `oci://...@sha256:...` references and an optional keyless cosign verify policy, anchored to an exact identity and preserved across re-connects; with the verify flags the CLI verifies the signature and pins the verified digest at tap time. Community index entries can record a digest too ([**@IvanHunters**](https://github.com/IvanHunters) in #5082, backport #5246). +* **Tenant cluster metrics reach the tenant's Monitoring**: the vmagent of tenant Kubernetes clusters now writes to the tenant's vminsert and labels series with the owning tenant instead of `cozy-monitoring`; before, no tenant cluster metric reached the tenant's Monitoring ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #5024, backport #5246). +* **SecurityGroups apply to Kubernetes worker nodes**: the virt-launcher pods of Kubernetes worker nodes are now resolved to their Kubernetes application and marked as managed by Cozystack, so a SecurityGroup attached to a Kubernetes cluster applies to its workers, not only to its control plane ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #5124, backport #5246). +* **Membership-only SecurityGroups**: a SecurityGroup with attachments and no rules no longer leaves an invalid CiliumNetworkPolicy behind; its backing policy is accepted by Cilium and still allows or denies nothing ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #5133, backport #5246). +* **Proxmox CCM and CSI images**: the Proxmox cloud-controller-manager runs v0.16.1 and the Proxmox CCM and CSI driver images are pinned by digest. The v0.15.0 tag was re-pushed upstream to a build that panics on start, which left every new Proxmox worker with the uninitialized taint ([**@themoriarti**](https://github.com/themoriarti) in #4917, backport #5246). +* **Autoscaler capacity of KubernetesNodes pools**: on the kubevirt substrate the cluster-autoscaler capacity annotation is planned on the cores the VM actually gets, so a leading-zero `resources.cpu` such as `"010"` (decoded as YAML, eight cores) no longer yields a wrong capacity. No pool rolls ([**@themoriarti**](https://github.com/themoriarti) in #4974, backport #5246). +* **Tenant quota admission counts extended resources**: admission now reads the parent pool's usage of extended resources and hugepages under the `requests.*` key the controller counts, so a child tenant can no longer get a share the parent has already used ([**@lexfrei**](https://github.com/lexfrei) in #5184, backport #5246). +* **Postgres backup jobs fail as soon as the CloudNativePG Backup fails**: a Postgres `BackupJob` now fails with the recorded error instead of after 30 minutes, and reports a `WaitingForCNPGBackup` condition with the Backup's phase while it waits ([**@scooby87**](https://github.com/scooby87) in #5211, backport #5246). +* **MariaDB backups and restores report real failures**: a `BackupJob` or `RestoreJob` whose operator Job failed now ends Failed instead of Succeeded, and a failed backup no longer creates a Ready Backup without an archive ([**@scooby87**](https://github.com/scooby87) in #5207, backport #5246). +* **ClickHouse restores use the right archive**: a `RestoreJob` restores the archive its Backup was taken as, not the newest one. New Backups record the archive name; Backups taken before this release are restored best effort from the newest archive stamped no later than their `takenAt`, with a warning in the restore pod log. A ClickHouse `BackupJob` created under a name whose Backup still exists now fails instead of reporting that Backup as its result ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #5208, backport #5246). +* **Backup S3 endpoint changes are followed**: the backup controller re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new `publishing.host`, backups move to the new endpoint without forcing releases by hand ([**@scooby87**](https://github.com/scooby87) in #5230, backport #5246). +* **Backup clients run on a dedicated image**: the ClickHouse (Altinity) and RabbitMQ backup strategy Pods now run a small `backup-http-client` image instead of the 349 MB `platform-migrations` image, so a platform migration change no longer re-digests the backup clients. Air-gapped mirrors need the new `ghcr.io/cozystack/cozystack/backup-http-client` repository ([**@androndo**](https://github.com/androndo) in #4756, backport #5246). +* **PostgreSQL system databases**: `databases` refuses `postgres`, `template0` and `template1`, and the init job never runs `DROP DATABASE` on them, which used to fail the init job on every upgrade and leave the HelmRelease in `UpgradeFailed` ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #5155, backport #5246). +* **PostgreSQL init script is stable**: the declared databases render in a stable order, so an upgrade that leaves the databases unchanged no longer rewrites the init-script Secret ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #5091, backport #5246). +* **Redis with a single replica**: an instance with `replicas: 1` runs one Sentinel instead of three, without a Sentinel PodDisruptionBudget, so node drains are no longer blocked by it; instances with two replicas or more are unchanged ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #5151, backport #5246). +* **MariaDB operator on a custom cluster domain**: the operator uses the platform's `networking.clusterDomain` instead of a hard-coded `cozy.local`, so replication and User/Database/Grant reconciliation work on clusters with a non-default domain ([**@mattia-eleuteri**](https://github.com/mattia-eleuteri) in #5123, backport #5246). +* **tcp-balancer no longer OOMs**: HAProxy takes its fd and thread caps from the replica's own memory and CPU limits instead of the node, so a `t1.nano` balancer is no longer OOMKilled on nodes with many CPUs ([**@yankawai**](https://github.com/yankawai) in #5005, backport #5246). +* **etcd alert noise**: `etcdHighNumberOfFailedGRPCRequests` counts only the gRPC codes the etcd mixin counts, so the Canceled watch streams of a stopping Kubernetes API server no longer fire it on every member ([**@yankawai**](https://github.com/yankawai) in #4985, backport #5246). +* **Cilium IPAM hook on larger clusters**: the hook passes the CiliumNode list to `jq` on stdin and no longer fails with "Argument list too long" on clusters with more than a few nodes, which blocked the cilium upgrade and every package depending on networking ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #5212, backport #5246). +* **OIDC bootstrap Job with many users**: the Job no longer fails to start for a tenant cluster with about 240 or more `oidc.users`; the bindings now come from a ConfigMap the Job mounts ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #5213, backport #5246). +* **Dashboard tenant list links to the tenant page**: the tenant name in the tenant list opens its detail page, where Delete lives, and top-level tenants get their Edit button back ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #4004, backport #5246). +* **Dead `registries` platform value removed**: the platform chart no longer has the `registries` value or the `patch-containerd` Secret, which nothing read. Mirrors for tenant workers are the per-pool `talos.registryMirrors` of `KubernetesNodes` ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #5190, backport #5246). + +## Security + +* **Marketplace digest pinning and signature verification**: see the marketplace entry under Fixes (#5082). + +## Development, Testing, and CI/CD + +* **Chainsaw asserts survive transient API errors**: an e2e assert keeps polling through an error such as an etcd leader change instead of failing the suite early ([**@lexfrei**](https://github.com/lexfrei) in #4867, backport #5246). +* **Proxmox e2e hygiene**: the Proxmox tenant kubeconfig is kept per run and removed afterwards, the HelmRelease guard reads the `KubernetesNodes` pool release, and the leak allowlist is validated against every suite layout ([**@themoriarti**](https://github.com/themoriarti) in #4971, #4970 and #4972, backports #5246). +* **Frozen-line builds start with `GITHUB_TOKEN`**: the initial packages build dispatched when a release line is frozen now runs ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in #4888, backport #5246). +* **Unit tests after the release image restamp**: two unit tests that read fields the release build rewrites stay green on `release-1.7`; this is a release-branch-only fix ([**@themoriarti**](https://github.com/themoriarti) in #5179). + +## Documentation + +* **[website] Documentation**: the v1.7 docs gain the guides that lived only in the cozystack repository, the pending v1.7 docs PRs, a DRBD note for the replicated storage class, a refreshed Chainsaw e2e guide, restored legacy URLs and a grouping of older doc versions ([**@myasnikovdaniil**](https://github.com/myasnikovdaniil) in cozystack/website#763, cozystack/website#782, cozystack/website#675, cozystack/website#611, cozystack/website#775 and cozystack/website#767). +* **[website] Blog and demo**: a full-stack observability post for v1.7, the Cozystack 1.7 announcement, a three-board Raspberry Pi 5 post, and a collapsible navbar with a closable hint on the live demo page ([**@tym83**](https://github.com/tym83) in cozystack/website#790, cozystack/website#801 and cozystack/website#802; [**@kvaps**](https://github.com/kvaps) in cozystack/website#789). +* **[website] Pages deploy**: the deploy workflow drops the Dart Sass install ([**@kvaps**](https://github.com/kvaps) in cozystack/website#798). +* **[ansible-cozystack] Installer v1.7.0**: tracks the Cozystack installer image at v1.7.0, released in ansible-cozystack v1.7.0 ([**@app/renovate**](https://github.com/apps/renovate) in cozystack/ansible-cozystack#89). + +## Migration and Upgrades + +Platform migrations do not change in this release: `migrations.targetVersion` stays at 61. + +* **Remove system databases from PostgreSQL `databases` before upgrading (#5155).** A release that declares `postgres`, `template0` or `template1` under `databases` is refused at render; remove the entry first. A system database a previous version took over is handed back to the superuser on the next upgrade: the event trigger is removed, ownership and the `public` schema are restored, and its `_admin` and `_readonly` roles are dropped. +* **Tenant quota updates can be rejected after upgrading (#5184).** Admission now counts the parent's usage of extended resources and hugepages, so an update to an existing tenant can be rejected when its quota for these resources exceeds the parent's remaining budget, including an update that leaves the quota unchanged. +* **Air-gapped mirrors need `backup-http-client` (#4756).** Mirror `ghcr.io/cozystack/cozystack/backup-http-client` before upgrading, or ClickHouse and RabbitMQ backup Pods cannot pull their image. +* **tcp-balancer pods roll once (#5005).** The new HAProxy `global` section changes the rendered config; `maxconn` follows the memory limit at about 16 connections per MiB (2035 on `t1.nano`). +* **Redis scaled to one replica loses its Sentinel PDB (#5151).** A post-upgrade hook deletes the Sentinel PodDisruptionBudget the operator created earlier for an instance with `replicas: 1`; scaling an instance between 1 and 2 replicas scales its Sentinel Deployment between 1 and 3. +* **Backups created by failed MariaDB runs are not cleaned up (#5207).** Backups made by failed runs before this fix are not removed; restoring from one can fail or select another dump from the shared prefix. +* **Keys lost to the COSI grant bug stay lost (#5284).** Re-add the affected user to the Bucket as described in the Fixes entry. +* **`registries` in the platform Package is ignored (#5190).** A Package that still sets it keeps installing; the `patch-containerd` Secret in `cozy-system` is deleted by the release. +* **SecurityGroups now reach Kubernetes workers (#5124).** A SecurityGroup attached to a `Kubernetes` application starts applying to its worker nodes after the upgrade, so review attached groups before upgrading. + +## Contributors + +We'd like to thank all contributors who made this release possible: + +* [**@androndo**](https://github.com/androndo) +* [**@IvanHunters**](https://github.com/IvanHunters) +* [**@kvaps**](https://github.com/kvaps) +* [**@lexfrei**](https://github.com/lexfrei) +* [**@mattia-eleuteri**](https://github.com/mattia-eleuteri) +* [**@myasnikovdaniil**](https://github.com/myasnikovdaniil) +* [**@scooby87**](https://github.com/scooby87) +* [**@themoriarti**](https://github.com/themoriarti) +* [**@tym83**](https://github.com/tym83) +* [**@yankawai**](https://github.com/yankawai) + +**Full Changelog**: https://github.com/cozystack/cozystack/compare/v1.7.0...v1.7.1 From 5d453c4d238463244e30d328ddbe862daadbb1c3 Mon Sep 17 00:00:00 2001 From: Aleksei Sviridkin Date: Fri, 13 Nov 2026 16:33:57 +0300 Subject: [PATCH 511/518] test(backups): give the fake S3 bucket a StatObject main added StatObject to the object lister interface, so the fake bucket the MariaDB cleanup tests use has to implement it to compile. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin --- .../backupcontroller/mariadbstrategy_cleanup_test.go | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/internal/backupcontroller/mariadbstrategy_cleanup_test.go b/internal/backupcontroller/mariadbstrategy_cleanup_test.go index 1d2f10cbe..7efb3046f 100644 --- a/internal/backupcontroller/mariadbstrategy_cleanup_test.go +++ b/internal/backupcontroller/mariadbstrategy_cleanup_test.go @@ -1368,3 +1368,12 @@ func TestMinioObjectRemover_ReturnsTheServersError(t *testing.T) { t.Fatal("RemoveObject = nil, want the AccessDenied error") } } + +func (f *fakeS3Bucket) StatObject(_ context.Context, _, key string) (int64, error) { + for _, o := range f.objects { + if o.Key == key { + return o.Size, nil + } + } + return 0, errors.New("not found") +} From 700600ee3c2773f0a96baddef4411600263e6d85 Mon Sep 17 00:00:00 2001 From: Nimesh Kulkarni <167794819+GitNimay@users.noreply.github.com> Date: Mon, 19 Oct 2026 13:31:51 +0530 Subject: [PATCH 512/518] chore(cilium): drop externalIPs.enabled and nodePort.enabled The vendored Cilium chart reads neither key: there is no externalIPs value upstream, and from the nodePort block the templates read only range, addresses, bindProtection, autoProtectPortRange, directRoutingDevice, enableHealthCheck, enableHealthCheckLoadBalancerIP and enableDynamicSourceLookup. Both features follow kubeProxyReplacement, which stays true, so the keys only misled readers into thinking they toggle anything. Fixes #4816 Assisted-by: LLM Signed-off-by: Nimesh Kulkarni <167794819+GitNimay@users.noreply.github.com> --- packages/system/cilium/values.yaml | 4 ---- 1 file changed, 4 deletions(-) diff --git a/packages/system/cilium/values.yaml b/packages/system/cilium/values.yaml index 732fd4151..b8bc0b9ee 100644 --- a/packages/system/cilium/values.yaml +++ b/packages/system/cilium/values.yaml @@ -30,10 +30,6 @@ cilium: enabled: true hubble: enabled: false - externalIPs: - enabled: true - nodePort: - enabled: true loadBalancer: algorithm: maglev serviceTopology: true From f7ee616f6cdcd9cf9828026b84a8df83cf7a83fd Mon Sep 17 00:00:00 2001 From: Mattia Eleuteri Date: Fri, 13 Nov 2026 17:27:48 +0100 Subject: [PATCH 513/518] fix(linstor): rebase the snapshot restore patch on linstor-csi v1.13.2 The import block of pkg/client/linstor.go changed in v1.13.2, so the patch no longer applied and the linstor-csi image failed to build. Assisted-by: LLM Signed-off-by: Mattia Eleuteri --- .../003-snapshot-restore-node-selection.diff | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff b/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff index 75351fe8e..7f96ed5f9 100644 --- a/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff +++ b/packages/system/linstor/images/linstor-csi/patches/003-snapshot-restore-node-selection.diff @@ -1,8 +1,8 @@ diff --git a/cmd/linstor-csi/linstor-csi.go b/cmd/linstor-csi/linstor-csi.go -index bd28e06..5bed73b 100644 +index e260d7b..2062787 100644 --- a/cmd/linstor-csi/linstor-csi.go +++ b/cmd/linstor-csi/linstor-csi.go -@@ -135,6 +135,7 @@ func main() { +@@ -148,6 +148,7 @@ func main() { client.LogOut(logOut), client.PropertyNamespace(*propNs), client.LabelBySP(*labelBySP), @@ -11,7 +11,7 @@ index bd28e06..5bed73b 100644 if err != nil { log.Fatal(err) diff --git a/pkg/client/linstor.go b/pkg/client/linstor.go -index 5d71c37..43129c9 100644 +index 5b52dd4..82f10d0 100644 --- a/pkg/client/linstor.go +++ b/pkg/client/linstor.go @@ -46,6 +46,8 @@ import ( @@ -20,10 +20,10 @@ index 5d71c37..43129c9 100644 "golang.org/x/sys/unix" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/util/sets" "k8s.io/mount-utils" utilexec "k8s.io/utils/exec" - "k8s.io/utils/ptr" -@@ -72,6 +74,9 @@ type Linstor struct { +@@ -73,6 +75,9 @@ type Linstor struct { mounter mount.Interface resizer *mount.ResizeFs labelBySP bool @@ -33,7 +33,7 @@ index 5d71c37..43129c9 100644 } // NewLinstor returns a high-level linstor client for CSI applications to interact with -@@ -125,6 +130,34 @@ func APIClient(c *lc.HighLevelClient) func(*Linstor) error { +@@ -126,6 +131,34 @@ func APIClient(c *lc.HighLevelClient) func(*Linstor) error { } } @@ -68,7 +68,7 @@ index 5d71c37..43129c9 100644 // LogOut sets the Linstor client to write logs to the provided io.Writer // instead of discarding logs. func LogOut(out io.Writer) func(*Linstor) error { -@@ -1567,34 +1600,29 @@ func (s *Linstor) reconcileSnapshotResources(ctx context.Context, snapshot *volu +@@ -1834,34 +1867,29 @@ func (s *Linstor) reconcileSnapshotResources(ctx context.Context, snapshot *volu return fmt.Errorf("snapshot '%s' not deployed on any node", snap.Name) } @@ -115,7 +115,7 @@ index 5d71c37..43129c9 100644 if selectedNode == "" { return fmt.Errorf("no node [%s] available for snapshot '%s'", strings.Join(snap.Nodes, ","), snap.Name) } -@@ -1893,6 +1921,43 @@ func (s *Linstor) relocateResources(ctx context.Context, volID, rgName string) e +@@ -2168,6 +2196,43 @@ func (s *Linstor) relocateResources(ctx context.Context, volID, rgName string) e return nil } @@ -160,10 +160,10 @@ index 5d71c37..43129c9 100644 func (s *Linstor) FindSnapsByID(ctx context.Context, id string) ([]*volume.Snapshot, error) { snapshotId, err := volume.ParseSnapshotId(id) diff --git a/pkg/client/linstor_test.go b/pkg/client/linstor_test.go -index 4f3405b..9dedb3c 100644 +index 54c5242..8da6cad 100644 --- a/pkg/client/linstor_test.go +++ b/pkg/client/linstor_test.go -@@ -577,3 +577,93 @@ func TestGetSnapshotRemoteAndReadiness(t *testing.T) { +@@ -994,3 +994,93 @@ func TestReconcileResourceDefinitionRGConflict(t *testing.T) { }) } } From fd15ad282eb72d8ad6187ff92eb131c99b018fbd Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Sat, 14 Nov 2026 19:51:20 +0700 Subject: [PATCH 514/518] fix(ci): take the main ref for a bare image ref in a drifted file A file whose configuration drifted from the cozystack-packages:main artifact keeps its committed refs. For a digest-pinned ref that runs an older image; for the bare : a new image is first committed with, it names a tag that was never pushed, so the component cannot pull. The merge commit CI tests is fixed when the run starts, so a PR whose merge commit predates a config change to observability-proxy's values.yaml installed observability-proxy:v0.0.0 and failed E2E. In a drifted values file, a bare ref under an image key now takes the artifact line with the same key, the same image name and a digest, when exactly one exists. The rest of the file stays as the PR has it. Assisted-by: LLM Signed-off-by: Maxim Belyy --- hack/overlay-main-images.sh | 89 ++++++++++++++++++++++++++++-- hack/overlay-main-images_test.bats | 85 ++++++++++++++++++++++++++++ 2 files changed, 170 insertions(+), 4 deletions(-) diff --git a/hack/overlay-main-images.sh b/hack/overlay-main-images.sh index 2c13e34e1..b35be70ba 100755 --- a/hack/overlay-main-images.sh +++ b/hack/overlay-main-images.sh @@ -42,9 +42,13 @@ # a change: `yq -i`, which every `make image` uses to stamp the ref, drops one. # If any OTHER line differs — the PR branched from a # main whose config for that file differs from the artifact's base — the file is -# left on its committed ref (safe degradation, logged, never fatal). CI checks -# out the pull_request merge commit (current main + PR), so for an unbuilt file -# the config already matches current main and only ref lines differ. +# left on its committed ref (safe degradation, logged, never fatal), except +# that a bare digest-less ref in it, which no registry serves, still takes the +# artifact's ref (see replace_bare_refs). CI checks out the pull_request merge +# commit (main + PR), so for an unbuilt file the config matches main as of +# that merge commit. It is fixed when the run starts, so a rerun, or a PR not +# pushed to since, can be older than the artifact and drift on files it never +# touched. # # Usage: hack/overlay-main-images.sh [touched-pkg-dirs] # = extracted cozystack-packages:main tree; its root holds @@ -123,8 +127,76 @@ img_line='(@sha256:|^[[:space:]]*(- )?(image|repository|registry|tag|digest):|^[ # match branch for this same shape and for this same reason. bare_ref_line='^[^[:space:]#][^[:space:]]*/[^[:space:]]+$' +# A drifted file keeps its committed refs. For a ref pinned by digest that is a +# pullable older image, which is the safe answer. A bare `:` with no +# digest is different: it is the first commit of a new image +# (docs/agents/image-refs.md), its tag was never pushed, and keeping it makes the +# component fail to pull instead of run an older image. The merge commit CI +# checks out is fixed when the run starts, so a PR whose merge commit predates a +# config change to such a file drifts on it, and every new image is exposed +# until a release stamps its ref (#5328). +# +# replace_bare_refs rewrites, in only, each line whose value is +# such a bare ref under an image key, taking the line from that has the +# same key and indentation, the same image name and an @sha256: digest. Exactly +# one such line must exist; two are ambiguous and none means build-main did not +# stamp it, so there is nothing more pullable to take. Every other line of +# stays as the PR has it. The current image may expect configuration the PR's +# older chart lacks, but the committed ref cannot run at all. Prints the number +# of lines replaced. +replace_bare_refs() { + tmp="$1.overlay-bare.$$" + n=$(awk -v newf="$2" -v out="$tmp" ' + function key_prefix(s) { + if (match(s, /^[[:space:]]*(- )?([A-Za-z]+[Ii]mage|image):[[:space:]]*/)) return substr(s, 1, RLENGTH) + return "" + } + function ref_value(s, p, v) { + v = substr(s, length(p) + 1) + sub(/[[:space:]]+$/, "", v) + gsub(/"/, "", v) + return v + } + function image_name(v) { + sub(/@.*/, "", v) + sub(/.*\//, "", v) + sub(/:.*/, "", v) + return v + } + BEGIN { + while ((getline l < newf) > 0) { + p = key_prefix(l) + if (p == "" || index(l, "@sha256:") == 0) continue + k = p SUBSEP image_name(ref_value(l, p)) + cnt[k]++ + line[k] = l + } + close(newf) + } + { + p = key_prefix($0) + if (p != "") { + v = ref_value($0, p) + if (v ~ /^[^@[:space:]]+\/[^@[:space:]\/]+:[^@[:space:]\/]+$/) { + k = p SUBSEP image_name(v) + if (cnt[k] == 1) { print line[k] > out; r++; next } + } + } + print > out + } + END { close(out); print r + 0 } + ' "$1") + if [ "$n" -gt 0 ]; then + mv "$tmp" "$1" + else + rm -f "$tmp" + fi + echo "$n" +} + overlaid=0 same=0 +bare=0 skipped=0 drift=0 drift_files="" @@ -164,6 +236,15 @@ for new in $(find "$MAINPKGS" -type d -name charts -prune -o \ | grep -vE '^[[:space:]]*$' | grep -qvE "$line_pat"; then echo "drift (non-ref change) in $cur -> keeping committed ref" drift=$((drift + 1)) + case "$cur" in + *.yaml) + r=$(replace_bare_refs "$cur" "$new") || r=0 + if [ "$r" -gt 0 ]; then + echo "bare ref in $cur -> current-main ($r line(s))" + bare=$((bare + r)) + fi + ;; + esac # Name the PACKAGE, not the directory the file happens to sit in: an # images/*.tag drift would otherwise read as "/images", which is not # something anyone can go and look at. @@ -181,7 +262,7 @@ for new in $(find "$MAINPKGS" -type d -name charts -prune -o \ fi done -echo "Overlay current-main images: overlaid=$overlaid same=$same skipped(rebuilt/owned/edited)=$skipped drift=$drift failed=$failed" +echo "Overlay current-main images: overlaid=$overlaid same=$same skipped(rebuilt/owned/edited)=$skipped drift=$drift bare-refs=$bare failed=$failed" # Keeping the committed ref is the safe answer to a config that does not match # the artifact, but it is not a neutral one: that component then runs its last diff --git a/hack/overlay-main-images_test.bats b/hack/overlay-main-images_test.bats index 7cff2d842..7b22b00ff 100644 --- a/hack/overlay-main-images_test.bats +++ b/hack/overlay-main-images_test.bats @@ -502,3 +502,88 @@ load test_helper cd "$root" rm -rf "$w" } + +# A drifted file keeps its committed refs, and for a digest-pinned ref that is +# a pullable older image. A bare : is the first commit of a new image +# (docs/agents/image-refs.md) and names a tag no registry serves, so keeping it +# turns drift into ImagePullBackOff. That hit every fork PR whose merge commit +# predated a config change to observability-proxy's values.yaml (#5328). Only +# such a bare line takes the artifact's ref; the rest of the file stays as the +# PR has it. +@test "a drifted file still takes the artifact ref for a bare (digest-less) ref line" { + root=$(pwd) + w=$(mktemp -d) + mkdir -p "$w/packages/system/proxy" "$w/main/system/proxy" + printf 'proxy:\n image: "ghcr.io/cozystack/cozystack/proxy:v0.0.0"\n lookback: 1h\n' \ + > "$w/packages/system/proxy/values.yaml" + printf 'proxy:\n image: "iad.ocir.io/x/cozystack/proxy:main@sha256:bbbb"\n lookback: 1h\n prefix: cozy-\n' \ + > "$w/main/system/proxy/values.yaml" + ( cd "$w" && "$root/hack/overlay-main-images.sh" main '[]' ) > "$w/out.txt" + grep -q 'drift (non-ref change) in packages/system/proxy/values.yaml' "$w/out.txt" + grep -q ' image: "iad.ocir.io/x/cozystack/proxy:main@sha256:bbbb"' "$w/packages/system/proxy/values.yaml" + # The config stays the PR's: the new key is not introduced. + if grep -q 'prefix:' "$w/packages/system/proxy/values.yaml"; then + echo "FAIL: a config line was taken from the artifact" >&2; exit 1 + fi + grep -q 'bare ref in packages/system/proxy/values.yaml -> current-main' "$w/out.txt" + grep -q 'bare-refs=1' "$w/out.txt" + rm -rf "$w" +} + +@test "a drifted file replaces only its bare ref, not a digest-pinned sibling" { + root=$(pwd) + w=$(mktemp -d) + mkdir -p "$w/packages/system/pair" "$w/main/system/pair" + printf 'tuning: old\nimage: ghcr.io/cozystack/cozystack/pair:v1.5.0@sha256:aaaa\nclientImage: ghcr.io/cozystack/cozystack/pair-client:latest\n' \ + > "$w/packages/system/pair/values.yaml" + printf 'tuning: new\nimage: iad.ocir.io/x/cozystack/pair:main@sha256:bbbb\nclientImage: iad.ocir.io/x/cozystack/pair-client:main@sha256:cccc\n' \ + > "$w/main/system/pair/values.yaml" + ( cd "$w" && "$root/hack/overlay-main-images.sh" main '[]' ) > "$w/out.txt" + grep -q '^image: ghcr.io/cozystack/cozystack/pair:v1.5.0@sha256:aaaa$' "$w/packages/system/pair/values.yaml" + grep -q '^clientImage: iad.ocir.io/x/cozystack/pair-client:main@sha256:cccc$' "$w/packages/system/pair/values.yaml" + grep -q '^tuning: old$' "$w/packages/system/pair/values.yaml" + rm -rf "$w" +} + +@test "a bare ref is not replaced by an artifact line for a different image" { + # The same key can hold another image in the artifact, e.g. after a rename. + # Matching on the key alone would swap in an image the chart never asked for. + root=$(pwd) + w=$(mktemp -d) + mkdir -p "$w/packages/system/ren" "$w/main/system/ren" + printf 'tuning: old\nimage: ghcr.io/cozystack/cozystack/old-name:v0.0.0\n' > "$w/packages/system/ren/values.yaml" + printf 'tuning: new\nimage: iad.ocir.io/x/cozystack/new-name:main@sha256:bbbb\n' > "$w/main/system/ren/values.yaml" + ( cd "$w" && "$root/hack/overlay-main-images.sh" main '[]' ) > "$w/out.txt" + grep -q '^image: ghcr.io/cozystack/cozystack/old-name:v0.0.0$' "$w/packages/system/ren/values.yaml" + grep -q 'bare-refs=0' "$w/out.txt" + rm -rf "$w" +} + +@test "a bare ref is kept when the artifact carries no digest for it either" { + # A ref build-main does not stamp (third-party, or not built yet) has no + # digest in the artifact, so there is nothing more pullable to take. + root=$(pwd) + w=$(mktemp -d) + mkdir -p "$w/packages/system/tp" "$w/main/system/tp" + printf 'tuning: old\nimage: docker.io/library/busybox:1.37.0\n' > "$w/packages/system/tp/values.yaml" + printf 'tuning: new\nimage: docker.io/library/busybox:1.38.0\n' > "$w/main/system/tp/values.yaml" + ( cd "$w" && "$root/hack/overlay-main-images.sh" main '[]' ) > "$w/out.txt" + grep -q '^image: docker.io/library/busybox:1.37.0$' "$w/packages/system/tp/values.yaml" + grep -q 'bare-refs=0' "$w/out.txt" + rm -rf "$w" +} + +@test "a bare ref is kept when the artifact has two candidate lines for it" { + # Two lines with the same key and image name (e.g. one per component in a + # list) cannot be told apart, so neither is taken. + root=$(pwd) + w=$(mktemp -d) + mkdir -p "$w/packages/system/dup" "$w/main/system/dup" + printf 'tuning: old\na:\n image: ghcr.io/cozystack/cozystack/dup:v0.0.0\n' > "$w/packages/system/dup/values.yaml" + printf 'tuning: new\na:\n image: iad.ocir.io/x/cozystack/dup:main@sha256:bbbb\nb:\n image: iad.ocir.io/x/cozystack/dup:main@sha256:cccc\n' \ + > "$w/main/system/dup/values.yaml" + ( cd "$w" && "$root/hack/overlay-main-images.sh" main '[]' ) > "$w/out.txt" + grep -q '^ image: ghcr.io/cozystack/cozystack/dup:v0.0.0$' "$w/packages/system/dup/values.yaml" + grep -q 'bare-refs=0' "$w/out.txt" + rm -rf "$w" +} From 54b76fcdf1ff61402a695eb790381dbf1428e46f Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Sat, 14 Nov 2026 19:51:20 +0700 Subject: [PATCH 515/518] docs(agents): say how PR CI stamps a bare ref in a drifted file Assisted-by: LLM Signed-off-by: Maxim Belyy --- docs/agents/image-refs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agents/image-refs.md b/docs/agents/image-refs.md index 9c9bb3ca6..764b560c2 100644 --- a/docs/agents/image-refs.md +++ b/docs/agents/image-refs.md @@ -42,7 +42,7 @@ If a package stamps a reference somewhere the two globs do not cover, add that f ## Invariants -- **Every first-party reference is tag+digest.** A bare tag with no digest is not reproducible and must not be committed. The one exception is the first commit of a new image: it has no digest until it is built, so it lands as a bare `:` that the producing `make image` replaces. PR CI and release preparation both stamp it before anything pulls it, but a tree applied without a build pulls a tag that does not exist. +- **Every first-party reference is tag+digest.** A bare tag with no digest is not reproducible and must not be committed. The one exception is the first commit of a new image: it has no digest until it is built, so it lands as a bare `:` that the producing `make image` replaces. PR CI and release preparation both stamp it before anything pulls it, but a tree applied without a build pulls a tag that does not exist. In PR CI the stamp comes from `hack/overlay-main-images.sh` when the PR does not rebuild the package, and it takes a bare ref even from a file whose configuration drifted from the artifact, because keeping a tag that was never pushed cannot install. - **Exactly one producer stamps each reference.** A reference no `Makefile` writes will never be refreshed by anything, including release preparation. - **A package that stamps a reference is in the root `Makefile` `build:` list.** `build-matrix.sh`, the main build and release preparation all iterate that one list, so a package outside it is never rebuilt by any path and its pin freezes permanently. `hack/build-matrix_test.bats` computes the set of packages whose Makefile runs `docker buildx build` and fails on any the list misses. - **Every consumer of references reads all three storage shapes**, via `hack/lib/image-refs.sh`. From 854a3f40e40dbcd8b2bdb8c0c8beec6e590b96b5 Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Sat, 14 Nov 2026 19:54:31 +0700 Subject: [PATCH 516/518] fix(ci): limit the drifted-file bare-ref overlay to first-party images Main can bump a third-party image together with its config, so a PR behind it must keep the committed pair rather than take main's image alone. Only ghcr.io/cozystack/cozystack refs, matched against an artifact line for the same first-party image, are replaced now. State the trade-off for first-party refs that do exist, and name the packages that took main's images in their own annotation. Assisted-by: LLM Signed-off-by: Maxim Belyy --- docs/agents/image-refs.md | 2 +- hack/overlay-main-images.sh | 63 +++++++++++++++++++++--------- hack/overlay-main-images_test.bats | 45 +++++++++++++++++---- 3 files changed, 83 insertions(+), 27 deletions(-) diff --git a/docs/agents/image-refs.md b/docs/agents/image-refs.md index 764b560c2..191bc62c2 100644 --- a/docs/agents/image-refs.md +++ b/docs/agents/image-refs.md @@ -42,7 +42,7 @@ If a package stamps a reference somewhere the two globs do not cover, add that f ## Invariants -- **Every first-party reference is tag+digest.** A bare tag with no digest is not reproducible and must not be committed. The one exception is the first commit of a new image: it has no digest until it is built, so it lands as a bare `:` that the producing `make image` replaces. PR CI and release preparation both stamp it before anything pulls it, but a tree applied without a build pulls a tag that does not exist. In PR CI the stamp comes from `hack/overlay-main-images.sh` when the PR does not rebuild the package, and it takes a bare ref even from a file whose configuration drifted from the artifact, because keeping a tag that was never pushed cannot install. +- **Every first-party reference is tag+digest.** A bare tag with no digest is not reproducible and must not be committed. The one exception is the first commit of a new image: it has no digest until it is built, so it lands as a bare `:` that the producing `make image` replaces. PR CI and release preparation both stamp it before anything pulls it, but a tree applied without a build pulls a tag that does not exist. In PR CI the stamp comes from `hack/overlay-main-images.sh` when the PR does not rebuild the package, and it takes a bare first-party ref even from a file whose configuration drifted from the artifact, because a tag that was never pushed cannot install. The cost is that any other digest-less first-party ref in such a file also runs main's image with the PR's older config; third-party refs in it keep their committed pair. - **Exactly one producer stamps each reference.** A reference no `Makefile` writes will never be refreshed by anything, including release preparation. - **A package that stamps a reference is in the root `Makefile` `build:` list.** `build-matrix.sh`, the main build and release preparation all iterate that one list, so a package outside it is never rebuilt by any path and its pin freezes permanently. `hack/build-matrix_test.bats` computes the set of packages whose Makefile runs `docker buildx build` and fails on any the list misses. - **Every consumer of references reads all three storage shapes**, via `hack/lib/image-refs.sh`. diff --git a/hack/overlay-main-images.sh b/hack/overlay-main-images.sh index b35be70ba..adb6b8638 100755 --- a/hack/overlay-main-images.sh +++ b/hack/overlay-main-images.sh @@ -43,7 +43,7 @@ # If any OTHER line differs — the PR branched from a # main whose config for that file differs from the artifact's base — the file is # left on its committed ref (safe degradation, logged, never fatal), except -# that a bare digest-less ref in it, which no registry serves, still takes the +# that a bare first-party ref in it, one with no digest, still takes the # artifact's ref (see replace_bare_refs). CI checks out the pull_request merge # commit (main + PR), so for an unbuilt file the config matches main as of # that merge commit. It is fixed when the run starts, so a rerun, or a PR not @@ -127,23 +127,33 @@ img_line='(@sha256:|^[[:space:]]*(- )?(image|repository|registry|tag|digest):|^[ # match branch for this same shape and for this same reason. bare_ref_line='^[^[:space:]#][^[:space:]]*/[^[:space:]]+$' -# A drifted file keeps its committed refs. For a ref pinned by digest that is a -# pullable older image, which is the safe answer. A bare `:` with no -# digest is different: it is the first commit of a new image -# (docs/agents/image-refs.md), its tag was never pushed, and keeping it makes the -# component fail to pull instead of run an older image. The merge commit CI -# checks out is fixed when the run starts, so a PR whose merge commit predates a -# config change to such a file drifts on it, and every new image is exposed -# until a release stamps its ref (#5328). +# A drifted file keeps its committed refs. For a ref pinned by digest that is an +# older image which pulls and matches the PR's config, so it is the safe answer. +# A first-party ref with no digest is not: docs/agents/image-refs.md forbids it +# except as the first commit of a new image, whose tag was never pushed, and +# keeping that makes the component fail to pull. The merge commit CI checks out +# is fixed when the run starts, so a PR whose merge commit predates a config +# change to such a file drifts on it, and a new image fails every such PR until +# a release stamps its ref (#5328). # # replace_bare_refs rewrites, in only, each line whose value is -# such a bare ref under an image key, taking the line from that has the -# same key and indentation, the same image name and an @sha256: digest. Exactly -# one such line must exist; two are ambiguous and none means build-main did not -# stamp it, so there is nothing more pullable to take. Every other line of -# stays as the PR has it. The current image may expect configuration the PR's -# older chart lacks, but the committed ref cannot run at all. Prints the number -# of lines replaced. +# a bare first-party ref (ghcr.io/cozystack/cozystack/:, no digest) +# under an image key, taking the line from that has the same key and +# indentation, an @sha256: digest and the same first-party image, i.e. a path +# ending in /cozystack/. Exactly one such line must exist; two are +# ambiguous and none means build-main did not stamp it. Every other line of +# stays as the PR has it. +# +# The trade-off is deliberate. A bare first-party ref that does exist (an older +# release tag or a moving `latest`, #4264) also takes main's image, so it runs +# with the PR's older config rather than as committed. It is the same component +# built from the tree, and a digest-less ref is not reproducible anyway; the +# alternative is a ref that may not pull at all. Third-party refs are never +# touched: main can bump one alongside its config, and the committed pair is +# the one known to work together. The split shape (`repository:` + `tag:` +# without `digest:`) is not handled; no package uses it for a bare ref today. +# +# Prints the number of lines replaced. replace_bare_refs() { tmp="$1.overlay-bare.$$" n=$(awk -v newf="$2" -v out="$tmp" ' @@ -167,7 +177,10 @@ replace_bare_refs() { while ((getline l < newf) > 0) { p = key_prefix(l) if (p == "" || index(l, "@sha256:") == 0) continue - k = p SUBSEP image_name(ref_value(l, p)) + v = ref_value(l, p) + name = image_name(v) + if (index(v, "/cozystack/" name ":") == 0) continue + k = p SUBSEP name cnt[k]++ line[k] = l } @@ -177,7 +190,7 @@ replace_bare_refs() { p = key_prefix($0) if (p != "") { v = ref_value($0, p) - if (v ~ /^[^@[:space:]]+\/[^@[:space:]\/]+:[^@[:space:]\/]+$/) { + if (v ~ /^ghcr\.io\/cozystack\/cozystack\/[^@[:space:]\/]+:[^@[:space:]\/]+$/) { k = p SUBSEP image_name(v) if (cnt[k] == 1) { print line[k] > out; r++; next } } @@ -197,6 +210,7 @@ replace_bare_refs() { overlaid=0 same=0 bare=0 +bare_files="" skipped=0 drift=0 drift_files="" @@ -240,8 +254,10 @@ for new in $(find "$MAINPKGS" -type d -name charts -prune -o \ *.yaml) r=$(replace_bare_refs "$cur" "$new") || r=0 if [ "$r" -gt 0 ]; then - echo "bare ref in $cur -> current-main ($r line(s))" + echo "bare first-party ref in $cur -> current-main ($r line(s)); its other refs stay committed" bare=$((bare + r)) + d=$(dirname "${cur#packages/}") + bare_files="$bare_files $d" fi ;; esac @@ -277,3 +293,12 @@ if [ "$drift" -gt 0 ]; then "$(printf '%s\n' "$drift_pkgs" | wc -l | tr -d ' ')" \ "$(printf '%s' "$drift_pkgs" | tr '\n' ' ' | sed 's/ $//')" fi +# Among those, a package whose bare first-party ref took main's image runs that +# image with the PR's older config, not its committed ref; name it separately +# so the warning above does not read as the whole story for it. +if [ "$bare" -gt 0 ]; then + bare_pkgs=$(echo "$bare_files" | tr ' ' '\n' | grep -v '^$' | sort -u) + printf '::warning title=Image overlay bare refs::%s drifted package(s) took current-main images for bare first-party refs, with the config committed in this PR: %s\n' \ + "$(printf '%s\n' "$bare_pkgs" | wc -l | tr -d ' ')" \ + "$(printf '%s' "$bare_pkgs" | tr '\n' ' ' | sed 's/ $//')" +fi diff --git a/hack/overlay-main-images_test.bats b/hack/overlay-main-images_test.bats index 7b22b00ff..28f04526f 100644 --- a/hack/overlay-main-images_test.bats +++ b/hack/overlay-main-images_test.bats @@ -525,7 +525,8 @@ load test_helper if grep -q 'prefix:' "$w/packages/system/proxy/values.yaml"; then echo "FAIL: a config line was taken from the artifact" >&2; exit 1 fi - grep -q 'bare ref in packages/system/proxy/values.yaml -> current-main' "$w/out.txt" + grep -q 'bare first-party ref in packages/system/proxy/values.yaml -> current-main' "$w/out.txt" + grep -q '::warning title=Image overlay bare refs::1 drifted package(s) .*: system/proxy$' "$w/out.txt" grep -q 'bare-refs=1' "$w/out.txt" rm -rf "$w" } @@ -560,15 +561,45 @@ load test_helper } @test "a bare ref is kept when the artifact carries no digest for it either" { - # A ref build-main does not stamp (third-party, or not built yet) has no - # digest in the artifact, so there is nothing more pullable to take. + # A ref build-main does not stamp has no digest in the artifact, so there is + # nothing more pullable to take. root=$(pwd) w=$(mktemp -d) - mkdir -p "$w/packages/system/tp" "$w/main/system/tp" - printf 'tuning: old\nimage: docker.io/library/busybox:1.37.0\n' > "$w/packages/system/tp/values.yaml" - printf 'tuning: new\nimage: docker.io/library/busybox:1.38.0\n' > "$w/main/system/tp/values.yaml" + mkdir -p "$w/packages/system/nd" "$w/main/system/nd" + printf 'tuning: old\nimage: ghcr.io/cozystack/cozystack/nd:v0.0.0\n' > "$w/packages/system/nd/values.yaml" + printf 'tuning: new\nimage: iad.ocir.io/x/cozystack/nd:main\n' > "$w/main/system/nd/values.yaml" ( cd "$w" && "$root/hack/overlay-main-images.sh" main '[]' ) > "$w/out.txt" - grep -q '^image: docker.io/library/busybox:1.37.0$' "$w/packages/system/tp/values.yaml" + grep -q '^image: ghcr.io/cozystack/cozystack/nd:v0.0.0$' "$w/packages/system/nd/values.yaml" + grep -q 'bare-refs=0' "$w/out.txt" + rm -rf "$w" +} + +@test "a bare third-party ref in a drifted file is kept even when main pins one" { + # Main can bump a third-party image together with its config in one file. A + # PR behind it has the older pair, which is the one known to work together; + # taking main's image alone would run it with the config it replaced. + root=$(pwd) + w=$(mktemp -d) + mkdir -p "$w/packages/system/kc" "$w/main/system/kc" + printf 'features: old\nimage: quay.io/keycloak/keycloak:26.6.3\n' > "$w/packages/system/kc/values.yaml" + printf 'features: new\nimage: quay.io/keycloak/keycloak:27.0.0@sha256:abcd\n' > "$w/main/system/kc/values.yaml" + ( cd "$w" && "$root/hack/overlay-main-images.sh" main '[]' ) > "$w/out.txt" + grep -q '^image: quay.io/keycloak/keycloak:26.6.3$' "$w/packages/system/kc/values.yaml" + grep -q 'bare-refs=0' "$w/out.txt" + if grep -q 'Image overlay bare refs' "$w/out.txt"; then + echo "FAIL: a bare-refs annotation with nothing replaced" >&2; exit 1 + fi + rm -rf "$w" +} + +@test "a bare first-party ref is not replaced by a third-party image of the same name" { + root=$(pwd) + w=$(mktemp -d) + mkdir -p "$w/packages/system/same" "$w/main/system/same" + printf 'tuning: old\nimage: ghcr.io/cozystack/cozystack/redis:v0.0.0\n' > "$w/packages/system/same/values.yaml" + printf 'tuning: new\nimage: docker.io/library/redis:8.0@sha256:abcd\n' > "$w/main/system/same/values.yaml" + ( cd "$w" && "$root/hack/overlay-main-images.sh" main '[]' ) > "$w/out.txt" + grep -q '^image: ghcr.io/cozystack/cozystack/redis:v0.0.0$' "$w/packages/system/same/values.yaml" grep -q 'bare-refs=0' "$w/out.txt" rm -rf "$w" } From 2468b309eea6e6b8e6ec3bbd2ea89b3285fb9b8c Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Sat, 14 Nov 2026 19:58:33 +0700 Subject: [PATCH 517/518] fix(ci): count a failed bare-ref rewrite and word the drift log by result A rewrite that cannot replace the file now counts as failed and leaves no temporary file behind, instead of reporting lines as replaced. A drifted file whose bare refs took main's images no longer logs that it kept its committed ref. Assisted-by: LLM Signed-off-by: Maxim Belyy --- hack/overlay-main-images.sh | 36 +++++++++++++++++++----------- hack/overlay-main-images_test.bats | 20 +++++++++++++++++ 2 files changed, 43 insertions(+), 13 deletions(-) diff --git a/hack/overlay-main-images.sh b/hack/overlay-main-images.sh index adb6b8638..2e6c4a1c4 100755 --- a/hack/overlay-main-images.sh +++ b/hack/overlay-main-images.sh @@ -151,7 +151,10 @@ bare_ref_line='^[^[:space:]#][^[:space:]]*/[^[:space:]]+$' # alternative is a ref that may not pull at all. Third-party refs are never # touched: main can bump one alongside its config, and the committed pair is # the one known to work together. The split shape (`repository:` + `tag:` -# without `digest:`) is not handled; no package uses it for a bare ref today. +# without `digest:`) is not handled, nor is a single-quoted value or one +# followed by a `# comment`; such a line stays as committed, and no package +# writes a bare first-party ref that way today. Prints nothing and fails when +# the file could not be rewritten, so the caller counts it as failed. # # Prints the number of lines replaced. replace_bare_refs() { @@ -199,11 +202,11 @@ replace_bare_refs() { } END { close(out); print r + 0 } ' "$1") - if [ "$n" -gt 0 ]; then - mv "$tmp" "$1" - else - rm -f "$tmp" - fi + case "$n" in + ''|*[!0-9]*) rm -f "$tmp"; return 1 ;; + 0) rm -f "$tmp" ;; + *) mv "$tmp" "$1" || { rm -f "$tmp"; return 1; } ;; + esac echo "$n" } @@ -248,19 +251,26 @@ for new in $(find "$MAINPKGS" -type d -name charts -prune -o \ if diff "$cur" "$new" | sed -n 's/^[<>] //p' \ | grep -vE '^[[:space:]]*$' | grep -qvE "$line_pat"; then - echo "drift (non-ref change) in $cur -> keeping committed ref" drift=$((drift + 1)) + r=0 case "$cur" in *.yaml) - r=$(replace_bare_refs "$cur" "$new") || r=0 - if [ "$r" -gt 0 ]; then - echo "bare first-party ref in $cur -> current-main ($r line(s)); its other refs stay committed" - bare=$((bare + r)) - d=$(dirname "${cur#packages/}") - bare_files="$bare_files $d" + if ! r=$(replace_bare_refs "$cur" "$new"); then + echo "WARN: rewriting bare refs failed for $cur (keeping committed refs)" + failed=$((failed + 1)) + r=0 fi ;; esac + if [ "$r" -eq 0 ]; then + echo "drift (non-ref change) in $cur -> keeping committed ref" + else + echo "drift (non-ref change) in $cur -> keeping committed refs except bare first-party ones" + echo "bare first-party ref in $cur -> current-main ($r line(s)); its other refs stay committed" + bare=$((bare + r)) + d=$(dirname "${cur#packages/}") + bare_files="$bare_files $d" + fi # Name the PACKAGE, not the directory the file happens to sit in: an # images/*.tag drift would otherwise read as "/images", which is not # something anyone can go and look at. diff --git a/hack/overlay-main-images_test.bats b/hack/overlay-main-images_test.bats index 28f04526f..4059f9ea1 100644 --- a/hack/overlay-main-images_test.bats +++ b/hack/overlay-main-images_test.bats @@ -618,3 +618,23 @@ load test_helper grep -q 'bare-refs=0' "$w/out.txt" rm -rf "$w" } + +@test "a bare-ref rewrite that cannot write the file is counted as failed, not replaced" { + # Mode bits do not stop root from writing, and the POSIX runner has no + # `skip`, so there is nothing to observe as root. + if [ "$(id -u)" -eq 0 ]; then return 0; fi + root=$(pwd) + w=$(mktemp -d) + mkdir -p "$w/packages/system/ro" "$w/main/system/ro" + printf 'tuning: old\nimage: ghcr.io/cozystack/cozystack/ro:v0.0.0\n' > "$w/packages/system/ro/values.yaml" + printf 'tuning: new\nimage: iad.ocir.io/x/cozystack/ro:main@sha256:bbbb\n' > "$w/main/system/ro/values.yaml" + chmod a-w "$w/packages/system/ro" + ( cd "$w" && "$root/hack/overlay-main-images.sh" main '[]' ) > "$w/out.txt" 2>&1 + chmod u+w "$w/packages/system/ro" + grep -q '^image: ghcr.io/cozystack/cozystack/ro:v0.0.0$' "$w/packages/system/ro/values.yaml" + grep -q 'bare-refs=0 failed=1' "$w/out.txt" + if ls "$w/packages/system/ro" | grep -q 'overlay-bare'; then + echo "FAIL: temp file left behind" >&2; exit 1 + fi + rm -rf "$w" +} From 1597bb609842d961d7d01cf3803dd0ae96f0cb9f Mon Sep 17 00:00:00 2001 From: Maxim Belyy Date: Sat, 14 Nov 2026 20:02:29 +0700 Subject: [PATCH 518/518] fix(ci): keep the drift annotation true when bare refs took main's images Assisted-by: LLM Signed-off-by: Maxim Belyy --- hack/overlay-main-images.sh | 10 +++++++--- hack/overlay-main-images_test.bats | 1 + 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/hack/overlay-main-images.sh b/hack/overlay-main-images.sh index 2e6c4a1c4..6e8feafd2 100755 --- a/hack/overlay-main-images.sh +++ b/hack/overlay-main-images.sh @@ -141,7 +141,9 @@ bare_ref_line='^[^[:space:]#][^[:space:]]*/[^[:space:]]+$' # under an image key, taking the line from that has the same key and # indentation, an @sha256: digest and the same first-party image, i.e. a path # ending in /cozystack/. Exactly one such line must exist; two are -# ambiguous and none means build-main did not stamp it. Every other line of +# ambiguous, and none means build-main did not stamp it, or the key moved +# (another indentation or parent) or the image was renamed on main since the +# PR's merge commit. Every other line of # stays as the PR has it. # # The trade-off is deliberate. A bare first-party ref that does exist (an older @@ -299,8 +301,10 @@ echo "Overlay current-main images: overlaid=$overlaid same=$same skipped(rebuilt # cost three unrelated PRs, one round each. if [ "$drift" -gt 0 ]; then drift_pkgs=$(echo "$drift_files" | tr ' ' '\n' | grep -v '^$' | sort -u) - printf '::warning title=Image overlay drift::%s package(s) kept committed release refs, so they run release images in this lane rather than current main: %s\n' \ - "$(printf '%s\n' "$drift_pkgs" | wc -l | tr -d ' ')" \ + except="" + [ "$bare" -eq 0 ] || except=" (bare first-party refs excepted, see Image overlay bare refs)" + printf '::warning title=Image overlay drift::%s package(s) kept committed release refs%s, so they run release images in this lane rather than current main: %s\n' \ + "$(printf '%s\n' "$drift_pkgs" | wc -l | tr -d ' ')" "$except" \ "$(printf '%s' "$drift_pkgs" | tr '\n' ' ' | sed 's/ $//')" fi # Among those, a package whose bare first-party ref took main's image runs that diff --git a/hack/overlay-main-images_test.bats b/hack/overlay-main-images_test.bats index 4059f9ea1..68438ffa5 100644 --- a/hack/overlay-main-images_test.bats +++ b/hack/overlay-main-images_test.bats @@ -527,6 +527,7 @@ load test_helper fi grep -q 'bare first-party ref in packages/system/proxy/values.yaml -> current-main' "$w/out.txt" grep -q '::warning title=Image overlay bare refs::1 drifted package(s) .*: system/proxy$' "$w/out.txt" + grep -q '::warning title=Image overlay drift::1 package(s) kept committed release refs (bare first-party refs excepted, see Image overlay bare refs), .*: system/proxy$' "$w/out.txt" grep -q 'bare-refs=1' "$w/out.txt" rm -rf "$w" }