mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
ea4968cf4613f696cd955bb1f66a7cfd877529ce
8232
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ea4968cf46 |
fix(apps): keep Job and CronJob names within the Kubernetes limits (#4945)
## What this PR does The apps API admits release names of up to 53 characters, the release prefix included (less for Kafka and Kubernetes, which cap their own names). Kubernetes copies a Job name into the `job-name` label of its pods, and a label value takes at most 63 characters, so the apiserver refuses a Job with a longer name. A CronJob name is refused over 52 characters, because the controller names each Job it starts `<cronjob>-<timestamp>`. I rendered every Job and CronJob in `packages/apps` and `packages/extra` with the longest release name each application admits, and eight of them go over. | Application | Object | Name | Longest on main | After | | --- | --- | --- | --- | --- | | bucket | post-delete Job | `<release>-cleanup-acme` | 66 | 63 | | qdrant | post-delete Job | `<release>-qdrant-cleanup` | 68 | 63 | | kafka | pre-install and pre-upgrade Job | `<release>-kafka-migration` | 66 (release capped at 50) | 63 | | vm-disk | resize Job, rendered when the disk grows | `<release>-volume-resize-hook` | 72 | 63 | | vm-instance | update Job and the `app` label of its pod | `<release>-update-hook` | 65 | 63 | | mariadb | backup CronJob | `<release>-backup` | 60 | 52 | | clickhouse | legacy backup CronJob | `<release>-backup` | 60 | 52 | | tenant | pre-delete Job in `cozy-system` | `<tenant namespace>-cleanup` | 71 (nested namespace of 63) | 63 | Counted in characters of the application name, a Kafka of 42 to 44 cannot be installed today, a VMDisk of 37 or more cannot grow, and a VMInstance of 40 or 41 cannot take an instance type, profile or external change. Deleting a Bucket of 44 or more or a Qdrant of 42 or more fails in the post-delete hook. With the backup CronJob enabled, a MariaDB of 38 or more and a ClickHouse of 35 or more fail the whole release. A nested tenant whose namespace is longer than 55 characters cannot be deleted. The Jobs of bucket, qdrant, kafka, vm-disk and vm-instance are now named `printf "%s-<suffix>" .Release.Name | trunc 63 | trimSuffix "-"`, and vm-instance uses the same value for the pod label. A release name is never longer than 53, so only the fixed suffix is cut, and a brute-force check over every pair of release name lengths found no two release names that render the same Job name. A plain cut to 52 would reach into the release name of a CronJob, and two 53-character names that differ in the last character would share one. So a CronJob name that does not fit becomes the release name cut to 38 characters, the first six hex characters of its sha256 and `-backup`, the shape #4133 gave the ClickHouse Keeper name. The tenant cleanup Job takes the same form with the namespace cut to 48, because it runs in `cozy-system` next to the cleanup Job of every other tenant. There a plain cut gives a parent and a child named `clea` the same name, and the child's `before-hook-creation` would delete the parent's Job while it runs. Names that fit render exactly as before. The hook documents rendered from main and from this branch with the short release names the tests use are identical in all eight charts. A name that did not fit never produced an object, so nothing is renamed on upgrade. The ServiceAccounts, Roles and RoleBindings of these hooks keep their names, since they have no such limit, so the ones a pre-install hook left behind are replaced rather than orphaned. Helm runs the delete hooks stored with a release, so an existing Bucket, Qdrant or tenant picks up the fix with its next upgrade. Every other Job and CronJob fits at the longest name. The cleanup Jobs of clickhouse, harbor, mariadb and opensearch and the Kafka pre-delete Job reach 61, the Postgres init Job 62, and the Kubernetes and KubernetesNodes Jobs either stay under 63 or already cut their names. The etcd and monitoring modules require their chart name as the release name, and the gateway and seaweedfs cleanup Jobs reach 61. Each of the eight charts gets a new helm-unittest suite. It pins a short name, a name that lands exactly on the limit and the longest admitted name, a cut that would end in a dash wherever one can happen, and for the hooks the unchanged ServiceAccount and RBAC names. The CronJob suites also pin two 53-character names that differ in the last character, and the tenant suite pins the parent and child pair. Against the templates on main, 18 of the 41 new cases fail, every one with a name over the limit. With the hash forms replaced by a plain cut, the tenant parent and child both render `tenant-abcdefghijklmnopqrstuvwxyz-0123456789abcdefghijklmn-clea` and the two MariaDB names both render `mariadb-abcdefghijklmnopqrstuvwxyz0123456789abcdefgh`. `hack/helm-unit-tests.sh` passes over all 106 packages with tests. The Bats files that exercise these hooks and charts (`app-cleanup-hooks_test`, `kafka-migration-hook`, `kafka-pre-delete-hook`, `tenant-pre-delete-hook`, `check-hook-request-timeout`, `pod-label-census_test`, `mariadb-first-boot-waits`, `check-render-matrix`) give 96 ok on main and on this branch. `make generate` with cozyvalues-gen v1.7.0 in the eight packages changes nothing. A local kube-apiserver v1.35.0 refuses the longest name each chart renders on main, with `must be no more than 63 bytes` on the job-name label or `must be no more than 52 characters` on the CronJob, and creates the name from this branch. For vm-disk and vm-instance, whose hooks render only after a live lookup, that check used a Job with the same name and labels. This is related to #4875 but does not depend on it. #4875 adds a cozy-lib cleanup Job and cuts its name to 63 the same way, while this PR covers the Jobs already on main. Neither needs the other, and the two merge cleanly. ### Screenshots Not a UI change. ### Downstream repositories Templates and tests only. No values, schema, README, defaults, `ApplicationDefinition`, Secret or Service name changes, and none of the changed Job or CronJob names appears in the website, the Terraform provider, the Ansible collection or the other repositories on the list. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(apps): Job and CronJob names now fit the Kubernetes limits for the longest application names. A Kafka with a name of 42 to 44 characters can now be installed, long VMDisk and VMInstance names no longer fail a resize or an instance type change, long Bucket and Qdrant names and long nested tenant namespaces no longer fail deletion, and long MariaDB and ClickHouse names no longer fail the release with the backup CronJob enabled. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Cleanup, migration, resize, and update Jobs now use names that fit Kubernetes naming limits, including for long release or tenant names. * ClickHouse and MariaDB backup CronJobs now use names within Kubernetes’ length limit. Longer names are shortened with a hash suffix to help distinguish them. * Short names and names at the supported limit retain their existing naming format. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
868b97dbe6 |
fix(kubevirt): let tenant admins upload a VMDisk image with virtctl (#5114)
## What this PR does A VMDisk created with `source: upload` waits in `UploadReady` for an image, but no tenant role can fill it. `virtctl image-upload` reads the disk PVC and CDI's PVC Prime and requests an upload token, and the tenant roles grant neither `get` on persistentvolumeclaims nor `create` on `upload.cdi.kubevirt.io/uploadtokenrequests`, so the command stops on a forbidden PVC read. `cozy:tenant:admin:base` now grants both. Tenant admins and super-admins get them in their own namespaces through the existing RoleBindings, the use and view tiers do not. Reading the DataVolume is already granted for the disk's own name by the vm-disk chart's dashboard role, so no DataVolume rule is added. The vm-disk notes print the command with `--no-create`, because the chart has already created the DataVolume, and the security self-assessment and threat model list the new grants. This is the RBAC part of #3847, split out so tenants can upload while the console upload panel and the upload proxy certificate stay in review there. The rules, their tests, the notes change and the docs lines are from @myasnikovdaniil, who is co-author on the commit. Checked live on a cluster with KubeVirt v1.8.4 and CDI v1.66.1, with virtctl v1.8.4 impersonating a user in the `<tenant>-admin` group. - With the stock roles `virtctl image-upload dv vm-disk-<name> --no-create` fails with `persistentvolumeclaims "vm-disk-<name>" is forbidden`. - With exactly these two rules added as a temporary Role, the same command reads `prime-<uid>`, uploads a 16 MiB image and the DataVolume reaches `Succeeded`. `get` on the DataVolume came from the vm-disk dashboard role. - The upload went through a port-forward to `cdi-uploadproxy` with `--insecure`. A trusted certificate on the published proxy is the other half of #3847 and is not changed here. Unit tests with `helm unittest` 1.1.2, the version CI pins. `packages/system/cozystack-basics` passes 72 of 72, against 66 on main, and `packages/apps/vm-disk` 117 of 117, against 116. Removing the upload token rule fails one case, and dropping `--no-create` from the notes fails one case. Fixes #3759 ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. Two tenant RBAC rules, their tests, the vm-disk notes and two docs files. No values key, schema, default, README or package change, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kubevirt): tenant admins can upload an image into a VMDisk created with source upload using virtctl image-upload. The tenant admin and super-admin roles get get on persistentvolumeclaims and create on upload.cdi.kubevirt.io/uploadtokenrequests, and the vm-disk notes print the command with --no-create. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Tenant administrators can now upload images into existing VM disks. * **Improvements** * The displayed upload command now explicitly targets the release namespace and avoids creating a new disk. * **Documentation** * Security guidance now clarifies the access needed for uploads and notes that persistent-volume permissions remain limited by namespace scope. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6f8171db89 |
fix(workloadmonitor): report a bucket's workloads not ready until its claim is provisioned (#5096)
## What this PR does A Bucket app reports WorkloadsReady=True while its BucketClaim is never provisioned. On a v1.6.5 cluster a tenant got a BucketClaim for bucket class `tenant-root`, the root tenant there runs no SeaweedFS, and for three days the claim had no status while the objectstorage controller kept emitting `FailedCreateBucket` with `bucketclasses.objectstorage.k8s.io "tenant-root" not found`. The Bucket app showed Ready and WorkloadsReady the whole time. The WorkloadMonitor controller already writes a Workload per BucketClaim with `Operational` set from `status.bucketReady`, but the monitor itself counts only pods. The bucket chart renders its monitor with `replicas: 0` and `minReplicas: 0`, so the monitor stays operational whatever happens to the claim. The monitor now names every selected BucketClaim that is not provisioned in `status.message`, sets `status.reason` to the new `BucketClaimNotReady` and turns `Operational` false, the same way it already reports a DataVolume that is not ready. A DataVolume verdict keeps the fields when both apply. BucketClaims are listed on every pass, so a stored BucketClaim verdict is recomputed rather than kept like an unread DataVolume verdict, and it clears once the claim is provisioned. Any monitor whose selector picks a BucketClaim follows its readiness, which is what the Workload objects already say. The Application then shows WorkloadsReady=False with that message and reason. Its Ready condition keeps following the HelmRelease, as `pkg/registry/apps/application/rest.go` intends. Three new tests cover a provisioned claim, a claim not provisioned with and without a class, a claim outside the selector, the verdict clearing after provisioning with and without a DataVolume reader, and the DataVolume verdict winning over the BucketClaim one. Against the controller on main the four not-provisioned and clearing cases fail and the rest pass. `go test ./internal/controller/...` and `go vet` pass. `make generate` does not run on my machine, gengo fails to parse the go1.27.1 standard library, so I ran the controller-gen `object` and `crd` steps of `hack/update-codegen.sh` for `./api/v1alpha1/...` directly. The deepcopy file is unchanged and the generated WorkloadMonitor CRD matches `packages/system/cozystack-controller/definitions/cozystack.io_workloadmonitors.yaml`. I have not run the change on a live cluster. #5083 changes the Workload names of the same BucketClaims and does not touch the monitor status, so the two do not overlap. ### Screenshots Not a UI change. ### Downstream repositories The controller, one exported constant and its tests. No CRD schema, values key, default, telemetry metric or package change, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(workloadmonitor): a Bucket app reports WorkloadsReady=False with reason BucketClaimNotReady while its BucketClaim is not provisioned, instead of WorkloadsReady=True. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Workload monitors now report when a selected bucket claim is not provisioned and mark the monitor as non-operational. Status messages include the bucket class when available and list multiple unprovisioned claims in sorted order. * **Bug Fixes** * The bucket-claim status clears when the claim is provisioned or no longer selected. When DataVolume information is available, its status takes precedence over the bucket-claim status. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
df42104b13 |
fix(seaweedfs, cozy-lib): cap the Go heap so components survive their memory limit (#4234)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Every SeaweedFS component is a Go binary running under a cgroup memory limit, and none of them set `GOMEMLIMIT`. The collector then works from `GOGC` alone, which is a target relative to the live heap with no absolute ceiling, so a burst of allocation grows the heap past the limit and the kernel kills the container. This is not a leak. On the cluster where it showed up, the S3 gateway held 6.4 GiB of `heap_sys` against an 8 GiB limit while `heap_inuse` was 389 MiB: the container was killed over memory the runtime was holding and not using. Ten kills in seven days, each one an interruption of the object store every tenant writes its backups to. Raising the limit does not help, because nothing bounds the growth; it only moves the ceiling the heap climbs to. This PR adds a `cozy-lib.resources.goMemLimit` helper next to the existing `javaHeap` one and wires it into the master, volume, filer and s3 components and into every volume pool and zone, each sized from the memory limit already computed for it. Pools need their own value because the subchart merges `volume` into every entry of `volumes`: without one, a `t1.nano` pool would inherit the value sized for the default volume's 512Mi limit, above its own 128Mi. The value is 75% of the limit, the same fraction `javaHeap` uses for the JVM; the remaining quarter covers what `GOMEMLIMIT` does not account for, goroutine stacks and mmap'd regions among them. The lifecycle admin and worker run the same binary under limits the parent chart does not set (256Mi and 512Mi in `packages/system/seaweedfs/values.yaml`), so their value is pinned next to those limits at the same 75%: 192MiB and 384MiB. A limit under 4/3 Mi (1398102 bytes) floors to `0MiB`, which Go accepts and then collects almost continuously, so the helper fails the render instead. The upstream chart already supports `extraEnvironmentVars` on every one of these components, so nothing had to change there. No values key is added, so this needs no configuration and changes no public schema. Tested with `helm unittest`: a suite on the seaweedfs package asserts the value tracks the memory limit rather than a fixed string, for the default volume, Simple pools, MultiZone zones and zone pools; the system seaweedfs suite asserts the admin and worker env; and a helper suite in `cozy-lib-tests` pins the unit grammar and the rejection of a zero value. Reverting each fix makes its suite fail. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(seaweedfs): set GOMEMLIMIT on master, volume (and each pool and zone), filer, s3 and the lifecycle admin and worker so the Go runtime collects before the container hits its memory limit instead of being OOM-killed ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * SeaweedFS master, volume, pool, zonal volume, filer, and S3 workloads now receive a Go memory limit set to 75% of their configured memory limit. * Pool workloads use their own resource settings when provided, falling back to the default volume preset when needed. * SeaweedFS admin and worker components now have Go memory limits aligned with their configured memory limits. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
4fa6bfe30b |
feat(monitoring): alert when pods of one workload share a node (#4993)
## What this PR does On three 3-node clusters running v1.6.4 and v1.6.5 we found a tenant etcd with two of its three members on one node, a tenant Redis with two of its three Sentinels on the node that also ran the master, and several VictoriaMetrics and VictoriaLogs pairs (vmstorage, vmselect, vmalertmanager, vlstorage) sharing a node. Nothing alerted. Losing one of those nodes would have cost the etcd or Sentinel quorum, or both copies of a storage tier. Spread constraints keep new pods apart once they are in place (#4986 for etcd, #4989 for monitoring, #4990 for Redis), but a placement that already exists, or that comes from a soft rule or a drain, stays as it is, because the scheduler does not move running pods. This adds `PodReplicasColocated`, a warning in a new `workload-placement` group under `packages/system/monitoring-agents/alerts/`. It fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG `Cluster` or `StrimziPodSet` have been on one node for an hour while the cluster has at least as many schedulable nodes as the workload has running pods, so five replicas on three nodes stay quiet. The alert carries `cluster`, `namespace`, `owner_kind`, `owner_name` and `node`, plus the `severity`, `exported_instance` and `service` labels the other platform alerts set, and its value is the number of pods on that node. DaemonSets, Jobs and every other owner kind are not in the list, so a DaemonSet surge pod next to its predecessor does not count. etcd-operator gives every member pod its own `EtcdMember` as controller owner ([etcdmember_controller.go#L546](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L546)), so grouping by owner would see one pod per group. Members get their names from `GenerateName: <cluster>-` ([etcdcluster_controller.go#L434](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L434), [#L839](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L839)), and members adopted from the old StatefulSet keep names like `etcd-0` ([adopt.go#L103](https://github.com/cozystack/etcd-operator/blob/v0.5.6/internal/migrate/adopt.go#L103)). The rule drops the last dash-separated part of the member name and reports the group as `EtcdCluster <name>`. With the etcd package that is one group per tenant namespace, and two etcd clusters in one namespace would still be told apart. CNPG instance pods carry a controller reference to their `Cluster` ([cluster_create.go#L1580](https://github.com/cloudnative-pg/cloudnative-pg/blob/v1.30.0/internal/controller/cluster_create.go#L1580)), and Strimzi pods one to their `StrimziPodSet` ([StrimziPodSetController.java#L369](https://github.com/strimzi/strimzi-kafka-operator/blob/0.45.1-rc1/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/StrimziPodSetController.java#L369)). Owner and node both come from `kube_pod_info`. kube-state-metrics fills `created_by_kind` and `created_by_name` from the controller owner reference and `node` from `spec.nodeName` ([pod.go#L638-L653](https://github.com/kubernetes/kube-state-metrics/blob/v2.15.0/internal/store/pod.go#L638-L653)). `kube_pod_owner` and `kube_pod_status_phase` have no node label of their own, and the kube-state-metrics scrape adds `node` from `__meta_kubernetes_pod_node_name` with `honorLabels: true` (`templates/kube-state-metrics-scrape.yaml`), so on those series `node` is the node of the kube-state-metrics pod. The test data keeps that label on the phase series. A pod counts while `kube_pod_status_phase{phase="Running"}` is 1, so a Failed or Pending pod on the same node does not. `kube_pod_labels` does carry every pod label here (`--metric-labels-allowlist=pods=[*]`), but the rule does not depend on it. A node counts as schedulable unless it has a `NoSchedule` or `NoExecute` taint. A cordon shows up as the `node.kubernetes.io/unschedulable:NoSchedule` taint ([node_lifecycle_controller.go#L549-L555](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L549-L555)), and not ready and unreachable nodes get theirs from their conditions ([#L87-L99](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L87-L99)), so one clause covers all three, and a tainted control plane does not count either. Pods that a drain had to put together on the remaining nodes therefore stay quiet while the node is out, and the hour starts once it is back. The alert waits an hour. A replacement pod runs next to its terminating predecessor only for the predecessor's grace period, and node maintenance takes the drained node out of the count, while a placement that has held for an hour does not fix itself. The alert is about a risk rather than an outage, so the wait costs nothing. The VMAlert sets both `remoteWrite` and `remoteRead`, so the pending state survives a vmalert restart ([vmalert docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerts-state-on-restarts)). Not covered are workloads that run one pod per owner, such as ClickHouse with one StatefulSet per replica ([patterns.go#L45-L53](https://github.com/Altinity/clickhouse-operator/blob/release-0.25.2/pkg/model/chi/namer/patterns.go#L45-L53)) and the KubeVirt VMs behind a tenant Kubernetes node group. They would need grouping by pod labels, which this rule avoids. The node count is cluster-wide, so a workload that a node selector or affinity restricts to fewer nodes than it has pods fires too. `hack/workload-placement-alerts.bats` takes `.spec` of the VMRule and replays it through `vmalert-tool` v1.136.0 pinned by digest, the release vm-operator v0.68.4 deploys by default ([config.go#L38](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38)) and the same image #4963 and #4985 use. `hack/testdata/workload-placement-alerts/unittest.yaml` has 13 cases with 24 checks. Two of three StatefulSet pods on one node fire at 61 minutes and not at 59, with labels and annotations pinned. Spread pods, five ReplicaSet pods on three nodes, a DaemonSet surge pod, two pods of one Job, a Failed pod next to a running one, and one pod each in two clusters that share workload and node names stay quiet. etcd members fire as one `EtcdCluster` per namespace, for generated and adopted member names. A replacement pod next to its terminating predecessor for twenty minutes stays quiet, and the same overlap held past the hour fires and resolves when it ends. A cordoned node keeps the alert quiet until it is uncordoned, and a node with unreachable taints keeps it quiet throughout. ReplicaSet, CNPG `Cluster` and `StrimziPodSet` pods fire with their owner kind. ```console $ bats hack/workload-placement-alerts.bats 1..1 ok 1 workload placement alerts pass their vmalert-tool cases ``` With the expression replaced by one that never fires, the suite prints `not ok 1` and vmalert-tool reports 5 of the 24 checks, which are all the checks that expect an alert. On main, where the rule file does not exist, the suite fails as well. Each of these changes to the rule fails at least one check, with the number of failing checks in brackets. `for: 15m` (5), no phase filter (1), no node count (6), no `cluster` label (6), DaemonSet and Job in the kind list (3), no taint clause (4), `<` instead of `<=` (5), etcd members grouped per member (1). With Bats 1.14.0 at the commit CI pins, the suite passes together with `alert-severity-contract`, `bats-strict-setup`, `bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`, `image-pin-consistency` and `check-commit-trailers`, 109 of 109, and the severity contract lists the new alert as `warning`. helm-unittest 1.1.2, the version CI pins, passes the monitoring-agents chart with 19 of 19 tests in 6 suites, and `helm template` with Helm 3.17.3 renders the VMRule the same as the file. Like the migration suites the new suite needs docker, and only the image pull touches the network. I have not run the rule in vmalert on a live cluster. On release-1.6 kube-state-metrics 2.15.0, its scrape config, the alerts template, the VMAlert and vm-operator v0.68.4 are the same, and etcd-operator v0.5.5 and CNPG 1.28.2 set the same owners and member names. The new files apply there without changes. That backport is a separate PR with one more commit that drops the `load test_helper` line, because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`. ### Screenshots Not a UI change. ### Downstream repositories A new alert file under `packages/system/monitoring-agents/alerts/`, a Bats suite under `hack/` and its test data. No values, schema, README, image or `ApplicationDefinition` change, and the website does not list platform alerts by name, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(monitoring): new warning alert PodReplicasColocated fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG Cluster, StrimziPodSet or etcd cluster have shared a node for an hour while the cluster has a schedulable node for each of them. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added monitoring for workloads whose running pod replicas are colocated on the same node. * Alerts include workload, node, cluster, and severity details, along with quorum-loss guidance. * Coverage includes StatefulSets, ReplicaSets, Clusters, StrimziPodSets, and EtcdMembers. * **Tests** * Added validation for workload-placement alerts across scheduling, availability, cluster-isolation, and pod-replacement scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
5fe356f57e |
fix(postgres): accept a standard five-field backup schedule (#5119)
## What this PR does The postgres app hands `backup.schedule` to a CloudNativePG `ScheduledBackup`, and the field asks for the six-field CloudNativePG cron with leading seconds. The mongodb app takes a standard five-field cron in the same field, and so do backup Plans and the Plan form in the console, so tenants copy a value from one app into the other and support has to explain the difference. CloudNativePG does not reject a five-field value. CloudNativePG 1.30.0, which main pins, parses the schedule with `cron.Parse` from robfig/cron v1.2.0 in both the ScheduledBackup webhook and the controller. That parser takes five or six fields, reads the first one as seconds and makes day-of-week optional, and the webhook only returns a warning when the count is not six. CloudNativePG's own webhook test "warn the user if the schedule has a wrong number of arguments" covers this with `1 2 3 4 5`. So `0 2 * * *`, meant as every day at 02:00, fires every hour at minute 2, and `5 * * * *` fires every minute. The chart now counts the fields. A five-field value gets a leading `0` seconds field, so `0 2 * * *` renders `0 0 2 * * *`. A six-field value and a descriptor such as `@daily` or `@every 6h` render exactly as before. Any other field count fails the render with a message that shows both forms, where before it reached the webhook and was rejected there. An empty schedule still renders no ScheduledBackup. The field description that the dashboard form shows and the README now name both forms. One thing changes on upgrade. A release that already sets a five-field value gets the standard meaning of it, so `0 2 * * *` moves from hourly to daily. Before this change such a value was outside the documented format and drew the webhook warning. `packages/system/keycloak` renders its own CloudNativePG ScheduledBackup from its `backup.schedule`, with a six-field default and a values comment that says so. It is a platform component with its own template, so it is not changed here. The other apps with a backup schedule (mongodb, mariadb, clickhouse) and backup Plans already take five fields. - Ran robfig/cron v1.2.0 `Parse` locally from 10:00:00. `5 * * * *` fires at 10:00:05, 10:01:05 and 10:02:05, and `0 2 * * *` at 10:02:00 and 11:02:00. With the leading `0` they fire at 10:05:00 and 11:05:00, and daily at 02:00. - Rendered the whole chart with helm 3.17.3 and the `configured` render fixture. The output matches main for the defaults, for `0 2 * * * *` and for `@daily`, and for `0 2 * * *` only the `schedule` line differs. Unit tests with `helm unittest` 1.1.2, the version CI pins. `packages/apps/postgres` passes 216 of 216 in 27 suites, against 209 in 26 on main. The new suite has seven cases. With the old template five of them fail. The six-field and descriptor cases pass on both, and they fail when the template prepends to six fields too or drops the descriptor check. ### Screenshots Not a UI change. The dashboard form picks up the new field description from the schema. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. A chart template, a new test suite, one field description and the README, schema, Go types and cozyrd regenerated from it. No field is added, removed or renamed and no default or enum changes. The terraform provider does not model `backup.schedule`. The website reference page for postgres is generated from this README on a stable tag, and the Go types example there uses a six-field schedule, which keeps working. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(postgres): backup.schedule accepts a standard five-field cron such as 0 2 * * *, and the chart prepends a 0 seconds field before handing it to CloudNativePG, which reads the first field as seconds. Six-field values and descriptors render as before, and any other field count fails the render. A release that already sets five fields gets the standard meaning on upgrade, so 0 2 * * * moves from hourly to daily. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * PostgreSQL backup schedules support standard five-field cron expressions as well as six-field expressions with a leading seconds field. Five-field schedules are converted automatically; six-field schedules and cron descriptors are used as provided. * **Schedule Changes** * After upgrading, `0 2 * * *` runs daily at 02:00 instead of hourly at minute 2. To keep the previous hourly schedule, use `0 2 * * * *`. * Invalid schedule formats produce a rendering error. Empty schedules continue to disable chart-managed backups. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
f12300943d |
chore: clear golangci-lint findings and handle the errors they flagged (#5306)
## What this PR does
This clears the golangci-lint findings on main and fixes the errors
several of them pointed at. With the default linter set (errcheck,
govet, ineffassign, staticcheck, unused), main had 197 findings when
this branch started. Rebased on main at
|
||
|
|
a4537127e7 |
fix(vertical-pod-autoscaler): run two admission controllers spread over nodes (#5098)
## What this PR does The VPA admission controller runs as one replica, and its webhook is registered with `failurePolicy: Ignore`. While that one pod is being rescheduled, every pod created in the cluster is admitted without its VPA recommendation, and with `updateMode: Initial` it keeps those resources until it is recreated. On a v1.6.5 cluster `vmstorage-shortterm-0` was created during a node drain while the admission controller moved to another node. It has no `vpaUpdates` annotation and runs with the vm-operator defaults, requests 250m/500Mi and limits 1/1500Mi, while its VPA recommends 920m and 3.42Gi and the other replica got 5.13Gi through VPA. It was OOMKilled 18 days later. This PR sets `replicaCount: 2`, enables the chart's PodDisruptionBudget with `minAvailable: 1` and spreads the two replicas over nodes with a `ScheduleAnyway` topology spread, so single-node clusters still schedule both. A drain then leaves one controller serving. The controller still deletes and recreates its MutatingWebhookConfiguration on start (`selfRegistration` in `pkg/admission-controller/config.go`, VPA 1.5.1), so a restart leaves only that short gap instead of the whole reschedule. `failurePolicy` stays `Ignore`, a VPA outage should not block pod creation. Two new cases in `tests/vpa_test.yaml` check the replica count, the spread selector against the pod labels and the PDB. `helm unittest .` in `packages/system/vertical-pod-autoscaler` passes 18 of 18 with helm-unittest 1.1.2, and against the values on main the two new cases fail and the other 16 pass. `helm template` renders the Deployment with 2 replicas and the PDB with the admission controller selector. I have not run the change on a live cluster. #4540 is the same trap from the other side, a component that keeps a bad size because Initial never revisits it. ### Screenshots Not a UI change. ### Downstream repositories Values of `packages/system/vertical-pod-autoscaler` and its tests only. No platform values, app schema, default exposed to tenants or package change, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(vertical-pod-autoscaler): the admission controller runs two replicas spread over nodes with a PodDisruptionBudget, so pods created during a drain still get their VPA recommendation. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Improvements** * The VPA admission controller now runs with two replicas, increasing availability. * Its pods are spread across hostnames when scheduling permits, helping reduce the impact of node-level disruptions. * A disruption budget requires at least one admission-controller pod to remain available during voluntary disruptions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
24f2b8009f |
feat(redis): add maxmemory, eviction policy and snapshot toggle (#5310)
## What this PR does Lets a tenant run Redis as a pure cache without giving up the persistent volume. Until now the chart only turned snapshots off together with the volume (when `size` is empty), and offered no way to bound memory: a `CONFIG SET maxmemory` is lost when the pod restarts. On a cache-only instance, RDB snapshots and replica full syncs then filled the volume, and each SIGKILL during a save left an orphaned `temp-*.rdb` behind, so the instance crashlooped. New values: - `maxmemory` (quantity, unset by default): rendered in bytes. Unset renders `maxmemory 0`, the Redis default (no limit). - `maxmemoryPolicy` (enum, default `noeviction`): rendered as `maxmemory-policy`. - `snapshots` (bool, default `true`): `false` renders `save ""` while keeping the volume. Snapshots stay off when `size` is empty, as before. Every directive is rendered with an explicit value, defaults included. The operator applies `customConfig` with `CONFIG SET` on every reconcile and never resets a directive whose line disappears (`service/redis/client.go` in freshworks-oss/redis-operator v3.3.5), so leaving the defaults implicit would turn reverting a setting into a no-op until the next pod restart. As a consequence, existing instances gain `save 3600 1 300 100 60 10000`, `maxmemory 0` and `maxmemory-policy noeviction` on upgrade. These equal the Redis 7 and 8 defaults, but they overwrite any value an operator set by hand with `CONFIG SET`. `maxmemory` is refused at render time above 90% of the Redis memory request: above that threshold (`reservedPodMemoryPercent`, default 10) the operator logs an error and skips the line, which would leave Redis unbounded with nothing visible to the tenant. Not changed here: a replica still writes a temporary RDB to disk during a full sync (`repl-diskless-load` defaults to `disabled`), even with `snapshots: false`. `stop-writes-on-bgsave-error` keeps its default, since no background save runs once snapshots are off. Tests: `tests/cache_config_test.yaml` covers the rendered defaults, the snapshot toggle with and without a volume, byte conversion from presets and explicit resources, and both refusals. `helm unittest packages/apps/redis` passes (93 tests). ### Screenshots Not a UI change. ### Downstream repositories The new fields in `values.schema.json` reach `cozystack/terraform-provider-cozystack`, whose redis schema and expand/flatten pair are hand-written; no follow-up has been opened there yet, so the box is left unticked. The website reference page is regenerated from `README.md` by the release bot. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(redis): add `maxmemory`, `maxmemoryPolicy` and `snapshots` values so Redis can run as a pure cache (bounded memory, eviction, no RDB snapshots) while keeping its persistent volume. Existing instances now render the Redis defaults for these directives explicitly, which resets any value set by hand with `CONFIG SET`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added configurable Redis memory limits, with a default of unlimited, and eviction policies, defaulting to `noeviction`. Configured limits must be whole-byte quantities and cannot exceed 90% of the pod’s memory request. * Added controls for scheduled Redis snapshots. Snapshots are enabled by default when storage is configured; disabling them does not remove existing snapshot files, and snapshots are off when storage size is empty. * **Documentation** * Documented the new Redis memory and snapshot settings, including defaults, constraints, and effects. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
4e6ce4655a |
feat(dashboard): add logs and metrics views to the console (#5218)
## What this PR does Adds logs and resource-consumption views to the console, backed by the observability proxy from the design proposals [Tenant application logs](https://github.com/cozystack/community/pull/95) and [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94). This PR is useful only together with #5214, which adds the console nginx route to the proxy. The application tabs need #5219 and the per-application roots from its follow-up PR. Until those are installed, each view shows a quiet "not available on this cluster" state instead of an error, so the order of merging does not break the console. What is added: - A "Logs" page at the namespace level, under a new "Observability" sidebar group, reading the namespace logs root. It lives at `/console/-/observability/logs`: every one- or two-segment console path is an application list or detail page, and `-` cannot start an application plural or name, so the page never shadows an application's own pages. - "Logs" and "Metrics" tabs on application pages (every application kind except VMDisk, which runs no containers), reading the per-application roots. - Logs view: - time range, a LogsQL filter, and pod and container pickers that turn into a stream filter; - "Follow" opens a live tail. The status shows "Live" as soon as the stream is open, even before the first line; on a dropped stream or a load refusal (`429`) it reconnects from the newest line seen, with backoff; it stops on `400`/`401`/`403`/`404`; - the reader can scroll up while lines stream; the newest 1000 lines are kept; - while a new range or filter loads, the previous lines stay on screen with a note saying so, or the view says it is loading; it never reports "no lines" for a query it has not read yet. - Metrics view: per-pod CPU and memory charts and application-wide network charts, refreshed every minute, with ranges from 1 hour to 7 days. The `rate` window is at least 5 minutes and four steps, so queries routed to the long-term store (coarser deduplication) still return data. No query names a namespace or kind; the proxy adds the scope. - Error handling by proxy status: scope-wide answers (`401`/`403`/`404`) replace the whole view with one notice; limit answers keep the proxy's own advice. A Kubernetes `Status` body on a proxy path means the request reached kube-apiserver instead of the proxy and is shown as "not available". - Shared UI pieces: an accessible `Select` and a text `Input` matching the console's controls, and an `EmptyState`. - `K8sClient.raw()` for untyped requests over the same credential path as `/apis`; the client now keeps caller headers given in any form. - Charts use uPlot, loaded in a separate lazy chunk, so the main bundle does not grow. #### How it was tested - `pnpm typecheck`, `pnpm test`, `pnpm lint` and `pnpm build` pass. - Screenshots of the views were taken against a mock API and on a test cluster. - Verified on a test cluster through the full browser chain (OIDC login, gatekeeper, gateway, console nginx, proxy), as a user with the `view` role in one tenant: the Logs page works; Follow keeps a single tail open for more than two minutes without reconnecting; Follow on a quiet filter shows "Live"; the Metrics tab of a Redis instance draws its charts; requests from the same session to another tenant's roots answer `403`. - Not verified: a screen reader (roles and names are asserted in tests only) and viewports narrower than 1440px. #### Known limitation The first lines a newly started pod writes can be missing from an already open tail, although a history query shows them. This comes from the log store's live-tail window relative to log collection delay, not from the console. ### Screenshots <img width="1440" height="900" alt="cluster-follow-live" src="https://github.com/user-attachments/assets/6efd9bfb-f05b-4498-bd20-0af4e5cda01f" /> <img width="1440" height="900" alt="cluster-logs-empty" src="https://github.com/user-attachments/assets/673d8295-cd3d-409d-b871-290705d0a6a2" /> <img width="1440" height="900" alt="cluster-logs" src="https://github.com/user-attachments/assets/621c2312-8427-4353-b199-44bdd12b78f7" /> <img width="1440" height="900" alt="cluster-pod-select" src="https://github.com/user-attachments/assets/92ea9c72-4a3c-4562-9c3a-2170730ab085" /> <img width="1400" height="1000" alt="cluster-redis-metrics" src="https://github.com/user-attachments/assets/3533d069-01b9-41de-bba9-70b940d05c6e" /> <img width="1440" height="900" alt="follow-waiting" src="https://github.com/user-attachments/assets/eb318d13-5188-4632-842a-43c0bbbcff5a" /> <img width="1440" height="900" alt="logs-empty" src="https://github.com/user-attachments/assets/c3d89e81-7249-444d-a3fa-9ab44c1ca86e" /> <img width="1440" height="900" alt="logs" src="https://github.com/user-attachments/assets/ec6caa9c-5eda-4361-99ea-359e5d002f04" /> <img width="1440" height="900" alt="metrics-empty" src="https://github.com/user-attachments/assets/db16da15-a618-4dbe-a531-593bc9b916dc" /> ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: no trigger matched. Candidate: none affected, because the whole diff is inside the vendored console (`packages/system/dashboard/images/console`), which per the trigger map belongs to this repository and has no downstream copy. A user guide page for the new console views on the website would be useful but is not required by the trigger map. ### Release note ```release-note feat(dashboard): the console gains a namespace Logs page and Logs and Metrics tabs on application pages, with live log following, per-pod CPU and memory charts, and application-wide network charts. The views need the observability proxy; without it they show "not available on this cluster". ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added Metrics and Logs tabs to supported application details, plus a tenant-wide Logs page in the sidebar. * View CPU, memory, and network charts over selectable time ranges, with optional sample tables and automatic refresh. * Search logs using pod and container filters, choose a history range, and follow live updates with connection status. * Added distinct loading, empty, and error messages for observability views, including guidance for unavailable services and query limits. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
17bbca2088 |
feat(observability-proxy): serve tenant namespace metrics (#5219)
## What this PR does Depends on #5214. This PR is stacked on that branch; the description covers only what it adds on top. Backward compatibility: this adds a new package and a new HelmRelease; nothing existing is renamed, removed or changed, so no migration is needed. Adds the namespace metrics root to the observability proxy, as described in the design proposal [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94). A tenant user can query the metrics of their own namespace from VictoriaMetrics with their Kubernetes credentials, without seeing other tenants' series. What is added: - Serves `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/{query,query_range,series,labels,label/<name>/values}`, GET and POST. Anything else under the root answers `404`. - Authorization by one `SubjectAccessReview` (`list` on `cozystack.io/metrics` in the namespace); the aggregated tenant `view` role gains that rule. - The upstream request is rebuilt from an allowlist (`query`, `start`, `end`, `step`, `time`, `match[]`, `timeout`). The scope is a single forced `extra_filters[]={namespace="<ns>"}`; client-supplied `extra_filters`, `extra_filters[]` and `extra_label` are dropped, because vmselect would OR them with the forced filter. The query itself is not rewritten. - Label endpoints always carry a `match[]` (the namespace scope when the client sent none), so they stay scoped even if vmselect's `-search.ignoreExtraFiltersAtLabelsAPI` is ever enabled. - Cost limits: maximum range (30 days), maximum points (11000), and a clamped `timeout` (30 seconds); query-shape limits answer `422`, the concurrency limit answers `429`. Times are accepted as unix seconds or RFC3339; durations are rounded up to whole milliseconds before limits are checked. - Store selection between the short-term and long-term VictoriaMetrics stores, configured by new chart values (`observabilityProxy.metrics`). #### Store selection: a known limitation and an open question The proxy has to send each query to the short-term store (3 days of retention, finer deduplication) or the long-term store. The choice depends on how far back the query reads, which is not only the request window but also the windows, subquery steps and offsets inside the expression. The current implementation estimates that lookback with a conservative scan of the expression text, not with a parser. It sums every `[...]` window and subquery step, every positive `offset`, and allowances for implicit subqueries and vmselect's default step and latency offset. Anything it cannot read with certainty (`@`, `WITH` templates, step multiples such as `5i`, template variables, unusual duration syntax) goes to the long-term store. The query is forwarded unchanged, so the estimate affects only the store choice, never the scope. This is a heuristic. It is designed to err only towards the long-term store, which holds the same recent data at coarser deduplication, so a wrong guess returns coarser data rather than missing data in most cases. A very short `rate` window sent to the long-term store can still come back empty. Two review rounds each found new expressions the scan underestimated, which were then fixed; there may be more. Open question for reviewers: should the scan be replaced by parsing the expression with `github.com/VictoriaMetrics/metricsql` (the library vmselect itself uses), only for the store choice? That gives an exact lookback at the cost of a new Go dependency. The namespace scope would still be enforced by vmselect through the single `extra_filters[]`, so the isolation design does not change either way. A user-visible consequence of counting the lookback: a range whose start sits right at the short-term retention edge (for example "last 3 days") now goes to the long-term store. #### How it was tested - `go test -race` and `go vet` for the proxy packages; table tests for the lookback estimate, including mutation checks of the `@` and unparseable-window cases. - helm-unittest for `observability-proxy` (new args and the tenant role rule). - Verified on a test cluster: `401` and `403` where expected; results contain only the caller's namespace; a query naming another tenant's namespace and a client-supplied `extra_filters[]` return nothing extra; `query_range`, `label/<name>/values` and `series` are scoped; an admin endpoint answers `404`. - Not verified: the limits under load, and the store choice on windows longer than 3 days (the test cluster had less than an hour of data). #### Known limitations - Relative times (`start=-1h`, `end=now`) are refused with `400`, while the logs root accepts durations. The console sends unix seconds, so it is not affected. - The proxy bounds the evaluation window, the step, the number of points of `query_range` and the timeout, not how far an expression reads: windows and offsets inside the expression (`up[10y]`, `x[30d:1ms]`, `offset 20d`) are bounded only by the store's retention and vmselect's own limits. - Times and durations are sent in a form vmselect parses exactly (RFC3339 with milliseconds for times, durations that truncate to the checked milliseconds); times outside 1970..2262 are refused with `400`. A test repeats vmselect's parsing of the exact strings the proxy sends. - `--metrics-max-range` defaults to 30 days while the longterm store keeps 14, so the older part of a long range comes back empty rather than refused. The two should be aligned or documented. - The store estimate errs towards longterm: deeply nested expressions with a large step can go to longterm although shortterm holds the data, and longterm deduplicates at 5 minutes, so short windows such as `rate(x[1m])` can come back sparse there. - The `namespace` label is only as trustworthy as the writer. Any pod in a child tenant may write to the vminsert of every ancestor tenant (`packages/apps/tenant/templates/networkpolicy.yaml`), and vminsert does not authenticate writers, so a child tenant can write series labelled with another tenant's namespace. Through this proxy the owner of that namespace would see those series as its own, and a flood of them could push its queries over the series limit. Reading another tenant's data is not possible this way. The write path predates this PR and already affects the root tenant's dashboards and alerts; closing it needs the write side to mark untrusted series, for example a vmauth in front of vminsert that forces an `extra_label` the proxy then excludes. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: no trigger matched. Candidate: none affected, because the diff on top of #5214 is confined to the proxy's Go code and its own chart under `packages/system/observability-proxy` (no platform values, no package under `packages/apps/`, no CRD, no `hack/` change). A user-facing page for the metrics API on the website would be useful but is not required by the trigger map. ### Release note ```release-note feat(observability-proxy): let tenant users query their namespace metrics from VictoriaMetrics with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/api/v1/`. Access requires the tenant `view` role or higher. Ranges are limited to 30 days and 11000 points. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * The observability proxy now supports tenant-scoped metrics queries alongside logs. * Metrics requests are routed to short- or long-term stores based on their time range, with configurable retention and query limits. * Metrics access is subject to tenant authorization, request validation, and concurrency limits. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
8c9f1ef4f9 |
docs(release): add changelog for v1.7.1 (#5318)
This PR adds the changelog for release `v1.7.1`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.7.1.md`.
|
||
|
|
f3bf92acc1 |
docs: add changelog for v1.7.1
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
|
185c5eca72 |
fix(monitoring): route InfoInhibitor to the blackhole receiver (#4948)
## What this PR does `InfoInhibitor` in `packages/system/monitoring-agents/alerts/general.rules.yaml` is the kube-prometheus rule that exists only to feed an Alertmanager inhibit rule, and its own description says it should go to a null receiver. The `update` target that first imported the rules rewrote its `severity: none` to `major`, and the platform Alertmanager config has no inhibit rule, so the alert goes to Alerta one level below critical. Alerta's Telegram and Slack plugins forward it with the default `disabledSeverity: []`, and once email delivery is configured, a `notify_email="true"` label inherited from the info alert gets it emailed as well. It fires whenever an alert with `severity="info"` is active and no warning or critical alert fires in the same namespace and cluster. #3799 moved the last first-party rule off `info`, but rules the platform does not write still use it. CDI v1.66.1 generates `CDIStorageProfilesIncomplete`, KubeVirt v1.9.0 generates `KubeVirtVMGuestMemoryAvailableLow` and `KubeVirtDeprecatedAPIRequested`, all at `info`, and operators add their own. We hit it on a v1.6.4 cluster with one custom `severity: info` rule that is always firing, and Alerta showed a permanent MAJOR `InfoInhibitor`. The info alert itself cannot show up there, because Alerta's default alarm model has no `info`. This puts the label back to `none`, the value kube-prometheus ships. The platform Alertmanager config already routes `severity="none"` to the `blackhole` receiver ahead of every route that delivers, and the Monitoring parameters already document that such alerts never reach email. `hack/alert-severity-contract.bats` refuses `none` on purpose, because an alert carrying it is never delivered, so InfoInhibitor is listed as the one alert for which that is the point. The exemption covers `none` and that name only. The contract now also checks that InfoInhibitor carries `none`, that `none` still fails on any other alert, and that the exemption does not let InfoInhibitor through with a severity Alerta refuses. A new helm-unittest suite in `packages/system/monitoring` pins the route order, so that only the heartbeat route can come ahead of the blackhole, with and without the email routes. There is no inhibit rule here, and the expression still matches `info` only. Wiring the inhibition up would hide every informational alert from Alerta unless a warning fires in the same namespace, which changes behaviour on every cluster and is a separate decision. Dropping InfoInhibitor altogether would be equivalent for delivery, and I can rework the PR that way if you prefer it. v1.6.5 ships the same rule and the same routes, so a backport to `release-1.6` would be appreciated. The commit cherry-picks onto it without conflicts. ### Validation - `bats hack/alert-severity-contract.bats` passes 12 of 12, and so does the same file under `hack/cozytest.sh`. Without the label change the new InfoInhibitor test fails on `[ "$delivered" -eq 0 ]`. With the exemption removed the tree-wide contract fails on `packages/system/monitoring-agents/alerts/general.rules.yaml: InfoInhibitor -> 'none'`. With the exemption widened to any alert, or to any severity, the matching fixture test fails. - `helm unittest` in `packages/system/monitoring` passes 25 suites and 306 tests. Deleting the `severity="none"` route fails both new cases, and an email route placed ahead of it fails the email case. - `helm unittest` in `packages/system/monitoring-agents` passes 6 suites and 19 tests, and the rendered `alerts-general.rules` VMRule carries `severity: none` for InfoInhibitor. - Rendered the Alertmanager config and ran `amtool config routes test` from Alertmanager v0.31.0, the default of VM operator v0.68.4, with InfoInhibitor's labels. `severity=major` routes to `alerta`, and to `alerta,email` once email is configured and `notify_email=true` is inherited. `severity=none` routes to `blackhole` in every case. `amtool check-config` reports 0 inhibit rules. - On `release-1.6` with this commit cherry-picked, the contract and both helm-unittest suites pass. - Not exercised on a live cluster. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Walked the trigger map against the diff. It changes one alert label under `packages/system/monitoring-agents/alerts/`, extends the test in `hack/alert-severity-contract.bats` and adds a test suite under `packages/system/monitoring/tests/`. No package is added or renamed, and no values, schema, CRD, platform values, variant, make target, tool the satellites call, node prerequisite or metric changes. The website already says that `severity="none"` alerts are blackholed and never reach email, and no page mentions InfoInhibitor, so nothing there goes stale. ### Release note ```release-note fix(monitoring): InfoInhibitor no longer reaches Alerta as a major alert. It carries severity none again, which the platform Alertmanager routes to the blackhole, as upstream kube-prometheus intends. Before, any active alert with severity info, from a custom rule or from one CDI or KubeVirt generates, raised a major InfoInhibitor in Alerta and in its Telegram and Slack notifications. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * `InfoInhibitor` alerts are now treated as non-notifying, preventing them from generating alert notifications. Other alerts retain their existing severity handling, and heartbeat alerts continue to follow their configured route. * The non-notifying route takes precedence over email delivery rules that include `InfoInhibitor`. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
dd6ff661bd |
fix(monitoring-agents): make a silent Watchdog an alarm in Alerta (#5102)
## What this PR does
The Watchdog is meant to prove the alerting pipeline is alive, and
Alerta turns it into a heartbeat because it carries `event: Heartbeat`.
When the heartbeat goes stale, nothing alarms.
- The Alerta 9.1.0 heartbeat plugin (`alerta/plugins/heartbeat.py`)
stores the alert severity in the heartbeat attributes, and the Watchdog
has `severity: ok`.
- The `alerta/alerta-web:9.1.0` image sets `HEARTBEAT_SEVERITY=major`,
and supervisord runs `alerta heartbeats --alert --severity major` about
every minute. The client it ships, 8.5.3, takes the severity of a stale
heartbeat from the heartbeat attributes ahead of that flag
(`want_severity = b.attributes.pop('severity', severity)` in
`alertaclient/commands/cmd_heartbeats.py`). The HeartbeatFail it raises
for the Watchdog is therefore severity `ok`, a normal alert.
- The Watchdog has no `timeout` label, so the Alerta webhook leaves the
timeout unset and the heartbeat only goes stale after a day (timeout
86400 on three v1.6.5 clusters), while the `event="Heartbeat"` route
resends it every minute. `/api/management/healthcheck` compares against
four timeouts and never reports it.
The Watchdog now has `severity: major`, matching the image's own
`HEARTBEAT_SEVERITY`, and `timeout: "300"`, five missed resends. The
`event="Heartbeat"` route still sends it only to `alerta_no_resolve` and
stops, so the severity reaches no email or other receiver, and the
template has no inhibit rules it could trigger. In the Alertmanager and
vmalert lists the Watchdog now shows as major instead of ok.
A new suite `tests/watchdog_heartbeat_test.yaml` checks the severity,
the timeout and the event label. `helm unittest .` in
`packages/system/monitoring-agents` passes 21 of 21 with helm-unittest
1.1.2, and against the rules on main the two new cases fail and the
other 19 pass. `hack/alert-severity-contract.bats` passes 9 of 9. I have
not run the change on a live cluster, and I have not read a heartbeat
record through the Alerta API, the chain above is from the sources of
the versions the chart pins.
The rule file is the same on release-1.6, and that backport is a
separate PR.
### Screenshots
Not a UI change.
### Downstream repositories
One rule in
`packages/system/monitoring-agents/alerts/general.rules.yaml` and a
test. No values key, schema, default or package change, so no trigger
map entry applies.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(monitoring-agents): a Watchdog that stops reaching Alerta now raises a major HeartbeatFail about six minutes later. It used to raise an ok alert after a day.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Monitoring**
* The Watchdog alert now reports a major severity and includes a
300-second timeout.
* The alert identifies heartbeat events.
* **Tests**
* Added checks to verify the Watchdog alert’s severity, timeout, and
event.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
a5c58e5e98 |
fix(vm-disk): refuse a disk smaller than the golden image it clones (#4960)
## What this PR does
A VMDisk created from `source.image` with a `storage` smaller than its
golden image never comes up. CDI v1.66.1 checks a clone against its
source in `validateClone`
(`pkg/controller/datavolume/pvc-clone-controller.go`) before it creates
the target claim. When the source PVC is in `Block` mode and requests
more than the DataVolume, CDI records `CloneValidationFailed` ("target
resources requests storage size is smaller than the source"), creates no
claim and leaves the DataVolume `Pending` for good. The webhook admits
such a DataVolume, so nothing fails at install. Raising `storage`
afterwards does not help either, because the chart renders an existing
DataVolume's spec as it is (CDI does not let it change) and the resize
hook only patches a claim that exists.
This is the default path. Every image in `vm-default-images` is 20Gi on
`replicated`, the StorageProfile CDI builds for the LINSTOR driver puts
`Block` first, and the chart defaults `storage` to 5Gi. Seen on a v1.6.4
cluster with CDI v1.66.1. A 10Gi disk from a 20Gi golden image sat
`Pending` with `CloneValidationFailed` while its release kept timing
out, and raising the disk to 25Gi left the DataVolume at 10Gi. The only
way out is to delete the disk and create it again at 20Gi or more.
The chart now refuses such a disk at render time, before any DataVolume
exists, with a message that names both sizes and the minimum to set, for
example `storage 5Gi is smaller than golden image
"vm-default-images-ubuntu" (20Gi), so CDI refuses to clone it and the
disk would stay pending forever. Set storage to 20Gi or more.` It
compares what CDI compares, the golden PVC's request against the disk's
request in bytes, and first raises the disk's request to the target
StorageProfile's `cdi.kubevirt.io/minimumSupportedPvcSize` the way
`GetEffectiveStorageResources` does. For an empty `storageClass` the
target is the class the check from #4796 already resolves, the default
virt class or else the Kubernetes default.
Nothing is compared, and the disk renders as before, when the golden PVC
cannot be read (under `helm template`, or before CDI created it), when
the golden PVC is in `Filesystem` mode, and for any source other than
`image`. For a `Filesystem` source CDI sizes a `Block` target from the
virtual size of the image inside it and otherwise compares requests
inflated by a per-class filesystem overhead, neither of which the chart
reproduces. An existing disk is exempt, as for the other golden checks,
since its DataVolume spec cannot change anyway.
Not covered here. A `source.disk` clone into a smaller disk goes through
the same CDI check and still renders. The chart default of 5Gi stays
below every default image, so a disk created from one at the default
size is now refused with the message above instead of hanging. Whether
the default should follow the images is a separate decision.
A new helm-unittest suite, `tests/dv_golden_size_test.yaml`, has 15
cases. It pins the refusal at the chart default against a 20Gi `Block`
golden with the whole message, the equal and the larger size, the
comparison in bytes (`21G` is refused against `20Gi`, `20480Mi`
renders), a StorageProfile minimum that covers the golden and one that
stays below it, the minimum read from the default class for an empty
`storageClass`, and the cases that render as before, a `Filesystem`
golden, no cluster, a golden PVC that does not exist yet, a disk without
a size, an existing disk, an `http` and an `upload` source. Against the
template on main the 3 refusal cases fail and the other 12 pass. Six
mutations of the new block each turn at least one case red (any volume
mode compared, the minimum dropped, the minimum read from the requested
class only, no check that `storage` is set, an equal size refused, sizes
compared as strings). `helm unittest` passes 129 of 129 for vm-disk,
against 114 on main, with helm-unittest 1.1.2 as CI pins it. Merged with
the current heads of #4802 or #4955, which touch the same template,
vm-disk passes 137 and 131 cases. `helm template` without a cluster
renders an image disk at 5Gi as before. I have not run it on a live
cluster.
The commit does not cherry-pick onto release-1.6 or release-1.7. Neither
carries the check from #4796 it builds on, and release-1.6 has none of
the golden checks, so a backport would be an adapted change.
### Screenshots
Not a UI change.
### Downstream repositories
Template and tests only. No values, schema, README, defaults or
`ApplicationDefinition` change. The golden images guide on the website
already creates its example disk at the image's 20Gi.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(vm-disk): a new disk from a golden image is now refused at render time when its storage is smaller than the image, with a message that names both sizes. Before, CDI refused the clone with CloneValidationFailed, the disk stayed pending forever, and raising its storage afterwards did not help. The default images are 20Gi and the default storage is 5Gi, so a disk from one of them needs storage set to 20Gi or more.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Disk sizes are now checked against the size of Block-mode golden
images when cloning. If the requested size is too small, it is increased
to the target storage profile’s minimum supported size when possible;
otherwise, rendering fails with an error that states the required size.
* Size comparisons account for actual byte capacity, so equivalent size
units are handled correctly. Filesystem-mode sources and other disk
sources are unaffected.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
e5ab1d6a7e |
fix(kubernetes): detach volumes when the infra PVC is missing (#5249)
## What this PR does routes an infra PVC `NotFound` in `ControllerUnpublishVolume` through the existing upstream hotplug detach path. a quota-rejected PVC can still have a persistent entry in the VM spec, so returning success before detach leaves a stale volume after the DataVolume is deleted. adds regression coverage for a pending hotplug volume, an already detached volume, and a VMI that outlives its VM. existing NFS unpublish tests still pass. no dependency update is needed. Fixes https://github.com/cozystack/cozystack/issues/5248 validation in the kubevirt-csi-driver module: `go test -count=1 ./...` and `go vet ./...`. the new regression fails without the production change. root `make generate` and `make manifests` also pass without generated-file changes. ### Screenshots no UI changes. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kubernetes): remove stale hotplug volumes when the infra PVC is missing, preventing deleted volumes from blocking subsequent attachments ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Volume unpublish requests now proceed through standard cleanup when the associated PVC is missing, rather than returning success immediately. This allows remaining teardown steps to run for detached volumes and VM-level resources. Other PVC lookup errors continue to be returned directly, without entering the missing-PVC cleanup path. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
e4d059cedd |
fix(etcd): require an even spread of members over nodes by default (#4986)
## What this PR does The tenant etcd chart spreads members with `maxSkew: 1` on `kubernetes.io/hostname` and `whenUnsatisfiable: ScheduleAnyway`, the default since #119. `ScheduleAnyway` only adds to the score of a node and other scores can outweigh it, so two members of one etcd can still start on the same node, where a node-local volume then keeps them. This PR makes the default `DoNotSchedule` with `nodeTaintsPolicy: Ignore`. The branch for `_cluster.scheduling.globalAppTopologySpreadConstraints` is untouched, so a platform that sets its own policy renders exactly as before, and the soft pod anti-affinity from #3962 stays. Seen on a three-node cluster running v1.6.4, whose `etcd-cluster.yaml` and etcd-operator v0.5.5 are the same as in v1.6.5. A new tenant etcd got the Pods of its first two members on one node, created 26 seconds apart, and the third on another node 18 seconds later. The Pods carry the `ScheduleAnyway` constraint and were scheduled by `linstor-scheduler`, whose profile adds the LINSTOR extender's prioritize score to the same sum with weight 5. The data volumes are on a node-local StorageClass with `WaitForFirstConsumer` and `allowRemoteVolumeAccess: "false"`, so the two members cannot leave that node, and losing it takes the quorum. A second tenant etcd created there the next day came up the same way. On a test cluster four tenant etcds created within one week each had two members on one node, and we moved one member of each by hand. `DoNotSchedule` does not stop an etcd with more members than nodes from starting when every node can run its Pods, unlike a required anti-affinity, because the skew is counted between the nodes that exist. kube-scheduler rejects a node only when the matching Pods on it plus the new one exceed the node with the fewest by more than `maxSkew` ([filtering.go](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/scheduler/framework/plugins/podtopologyspread/filtering.go#L334-L352)). One node takes all three members, two nodes take two and one, and three or more nodes take one each. A member whose only allowed node cannot take it, for example because the storage pool there is full, now waits Pending instead of joining another member. `nodeTaintsPolicy: Ignore` is the Kubernetes default and is written out on purpose. A cordoned node carries `node.kubernetes.io/unschedulable`, and `Honor` would take it out of the count ([common.go](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/scheduler/framework/plugins/podtopologyspread/common.go#L50-L54)), so a member created during a drain would start next to another member and stay there on a node-local volume. With `Ignore` it waits Pending until the node is back, while the other two keep quorum. The cost is that a node no member can run on, such as a tainted control plane, counts as an empty node too, so on a cluster with such a node each schedulable node takes at most one member and the members beyond that do not start. #4947 makes the same choice for the SeaweedFS masters and checked both policies in kwok. A platform that wants the old behaviour can set `scheduling.globalAppTopologySpreadConstraints` to `ScheduleAnyway`, which the other charts that read it pick up as well, and a new test case pins that the chart renders such a policy as written. On upgrade nothing is recreated. The etcd-operator adopts a changed `spec.topologySpreadConstraints` into `status.observed` once the cluster has reached its current target, or its progress deadline has passed ([etcdcluster_controller.go#L203-L214](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L203-L214)). It copies the value from there into the spec of an `EtcdMember` only when it creates one, at bootstrap ([#L446](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L446)) and on scale-up ([#L851](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L851)), and nothing in the operator writes it into an existing `EtcdMember`. The member controller creates a Pod only when it is missing and builds it from that member's own copy ([etcdmember_controller.go#L499-L512](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L499-L512), [#L545-L549](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L545-L549), [#L860](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L860)). So running members keep `ScheduleAnyway`, and so does a member whose Pod is evicted, deleted or replaced after a terminal phase ([#L164-L186](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L164-L186)), since the new Pod is built from the same copy. Only members created after the upgrade get `DoNotSchedule`, which means a scale-up, a crash-loop replacement or a member whose `EtcdMember` was deleted. No member is rolled, neither one at a time nor all at once. This matches the field docs ([etcdcluster_types.go#L542-L549](https://github.com/cozystack/etcd-operator/blob/v0.5.6/api/v1alpha2/etcdcluster_types.go#L542-L549), [etcdmember_types.go#L112-L116](https://github.com/cozystack/etcd-operator/blob/v0.5.6/api/v1alpha2/etcdmember_types.go#L112-L116)). The code paths are the same in v0.5.8, and in v0.5.5, which v1.6.4 and v1.6.5 ship, except that v0.5.5 does not yet replace a Pod left in a terminal phase. Members that already share a node therefore stay there. One of them moves only when a new `EtcdMember` takes its place, and the constraint then sends the replacement to the node without a member. That is how we moved the members on the test cluster, with `DoNotSchedule` set on the `EtcdCluster` by hand. The operator's docs say a scheduling change can be applied by deleting one Pod at a time ([concepts.md#L272](https://github.com/cozystack/etcd-operator/blob/v0.5.6/docs/concepts.md#L272)), but the recreated Pod gets the member's old copy, so that does not move it. The release note now declares two cases from review. In the first, an etcd created while the cluster had fewer nodes than members has two or three members on one node, and once a node is added a recreated Pod of one of them stays Pending, since `maxSkew` rejects its own node and its node-local volume rules out the others. Deleting that member's `EtcdMember` while the other members keep quorum replaces it on a node without a member. In v0.5.5 the member finalizer removes the member from etcd through the remaining members before the object goes ([`handleDeletion`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L195-L231), [`removeMemberFromEtcd`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L243-L310)), its Pod and its `data-<member>` PVC are owned by it and deleted with it ([#L403](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L403), [#L490](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L490)), and the cluster controller waits for the deletion, then adds a member under a new name with the constraints from `status.observed` ([etcdcluster_controller.go#L254-L312](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L254-L312), [`scaleUp`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L820-L851)). Its new PVC binds where its Pod lands, and under `DoNotSchedule` only a node without a member passes. The operator documents the same steps for a broken member, along with the limit that `MemberRemove` cannot run without quorum ([operations.md#L208-L231](https://github.com/cozystack/etcd-operator/blob/v0.5.5/docs/operations.md#L208-L231)), and v0.5.6, which main and `release-1.6` ship now, has the same path. The upstream chart of the operator installs a policy that denies `EtcdMember` deletion without the `etcd-operator.cozystack.io/allow-deletion=true` annotation ([member-deletion-policy.yaml](https://github.com/cozystack/etcd-operator/blob/v0.5.5/charts/etcd-operator/templates/member-deletion-policy.yaml)), and the `etcd-operator` package here does not install it. On a three-node cluster running v0.5.5 a tenant etcd had two members on one node with `DoNotSchedule` in effect. We deleted the `EtcdMember` of one of the two while the other members kept quorum. The replacement's Pod and its new node-local PVC were scheduled on the node without a member, the member joined, and the `EtcdCluster` stayed `Available` with quorum the whole time, polled every 5 seconds. In the second case `nodeTaintsPolicy: Ignore` counts a node the Pods cannot tolerate as one with zero members, so on a cluster with such a node and fewer schedulable nodes than members the members beyond the schedulable nodes do not start. The way out is the soft global policy in the platform `Package`, set before the upgrade. ```yaml apiVersion: cozystack.io/v1alpha1 kind: Package metadata: name: cozystack.cozystack-platform spec: components: platform: values: scheduling: globalAppTopologySpreadConstraints: | topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway ``` The platform chart writes `scheduling` into `_cluster` of the `cozystack-values` Secret ([apps.yaml#L292-L295](https://github.com/cozystack/cozystack/blob/55aad6f91456cb545601a83aaecf39118e943b04/packages/core/platform/templates/apps.yaml#L292-L295)), the tenant chart copies `_cluster` into the Secret of each tenant namespace ([namespace.yaml#L109-L120](https://github.com/cozystack/cozystack/blob/55aad6f91456cb545601a83aaecf39118e943b04/packages/apps/tenant/templates/namespace.yaml#L109-L120)), and this chart renders the value as written, which the new test case pins. Rendering the platform and tenant charts with the value above, on main and on `release-1.6`, gives the same string in both Secrets. The operator adopts a changed spec into `status.observed` only once all members are Ready or the progress deadline has passed ([etcdcluster_controller.go#L196-L213](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L196-L213)), and it clears the deadline when the cluster converges ([#L1448-L1456](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L1448-L1456)). A new etcd stuck short of its members adopts the policy once its first deadline passes, and deleting the `EtcdMember` of the Pending member then brings a replacement that starts. An etcd that had converged before one of its replacements went Pending adopts the policy only after the deadline is forced as the operator docs describe ([operations.md#L197-L206](https://github.com/cozystack/etcd-operator/blob/v0.5.5/docs/operations.md#L197-L206)), which is why the release note says to set it before the upgrade. #4817 moves this block into `_etcd.tpl` in cozy-lib without changing it, so whichever of the two lands second has to carry the two lines over. `tests/topology-spread_test.yaml` now pins the whole default constraint, and a new case pins that a `ScheduleAnyway` policy from `globalAppTopologySpreadConstraints` renders as written, without `nodeTaintsPolicy`. The existing zone override case is unchanged. Against the template on main the suite fails only the default case, 1 failed and 25 passed. Switching the default to `Honor`, to `ScheduleAnyway` or to `maxSkew: 2` fails the default case, and adding `nodeTaintsPolicy` to the override branch fails the new case. ```console $ docker run --rm -v "$PWD":/apps -w /apps/packages/extra/etcd helmunittest/helm-unittest:4.2.3-1.1.2 . Charts: 1 passed, 1 total Test Suites: 9 passed, 9 total Tests: 26 passed, 26 total ``` On main the same command gives 25 tests in 9 suites, with helm-unittest 1.1.2 as CI pins it. With default values `helm template` differs from main in the two lines and in the comment above the affinity block, and with a global policy set only in that comment. The docs commit changes the `affinity` description, from which `make generate` rewrites the README, `values.schema.json` and the `ApplicationDefinition` with nothing but that text, and the suite stays at 26 tests. I have not rolled this chart change out on a cluster. The spread block and the test file are the same on release-1.6 and release-1.7. On release-1.6 the template has no affinity block yet, so the fix commit cherry-picks without conflicts and the docs commit has nothing to change there. That backport is a separate PR. ### Screenshots Not a UI change. ### Downstream repositories One changed and one added line in `packages/extra/etcd/templates/etcd-cluster.yaml` and its test suite, and new text for the `affinity` description and the comment above the affinity block. The description reaches `values.schema.json`, the README and the `ApplicationDefinition` as text only, with no key, type or default change, and the website rebuilds the etcd reference page from the README on release. The website mentions `globalAppTopologySpreadConstraints` only as a platform setting, which stays as it is, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(etcd): tenant etcd members now spread over nodes with whenUnsatisfiable DoNotSchedule by default, so two members of one etcd no longer start on the same node while another node has none. Members that exist are not moved, only members created after the upgrade get the new constraint. An etcd created while the cluster had fewer nodes than members puts two or three members on one node. Once a node is added, such a member whose Pod is recreated (drain, eviction, graceful node shutdown) stays Pending, because maxSkew rejects its own node and its node-local volume rules out the others. With two nodes grown to three the etcd runs two of three members and its PodDisruptionBudget blocks draining the other nodes. With one node grown to three a reboot of that node brings back one member of three and quorum is lost. A stranded member is replaced by deleting its EtcdMember while the other members keep quorum. Its Pod and PVC go with it, and the replacement gets a new Pod and PVC on the node without a member. Right after adding nodes, doing this for one co-located member at a time while all members are Ready avoids the case. With nodeTaintsPolicy Ignore a node the etcd Pods cannot tolerate, such as a tainted control plane, counts as a node with zero members, so no node can take a second member. On a cluster with such a node and fewer schedulable nodes than members, a new etcd starts only as many members as there are schedulable nodes, one of three on a single schedulable node. An existing etcd there keeps its members, but a member it replaces stays Pending the same way. Either case is avoided by setting scheduling.globalAppTopologySpreadConstraints in spec.components.platform.values of the cozystack.cozystack-platform Package to a topologySpreadConstraints entry with maxSkew 1, topologyKey kubernetes.io/hostname and whenUnsatisfiable ScheduleAnyway before the upgrade. Every app chart that reads this value applies it, two members of a new etcd can share a node again, and the operator gives it only to members created after it adopts the changed EtcdCluster spec into status.observed. Once status.observed.topologySpreadConstraints of an EtcdCluster shows ScheduleAnyway, a member of it that stays Pending is replaced by deleting its EtcdMember. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **Bug Fixes** - Default etcd placement now requires even spreading across hosts and ignores node taints when evaluating placement. - Configurable cluster-wide spreading continues to use a flexible scheduling policy when selected. - **Documentation** - Clarified how affinity and topology spreading work together, when required anti-affinity or node-local storage can leave members pending, and that affinity changes take effect as members are replaced. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
76204b10aa |
fix(ingress-nginx): raise client keepalive_time above the 1h default (#5250)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, security (a fix that comes through a security advisory), docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Fixes merged to `main` reach the release lines the backport batch runs for automatically, so do not open backport PRs by hand: https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md --> ## What this PR does ingress-nginx exposes no ConfigMap key for the client-side `keepalive_time`, so nginx keeps its built-in 1h default: every client connection older than an hour is closed, and on HTTP/2 that is a `GOAWAY` (`ErrCode=NO_ERROR`). A client that multiplexes a long operation over one connection loses the request in flight. We hit it with a kopia repository maintenance run by Velero against a SeaweedFS S3 endpoint published through this controller: the run opened its connection at 18:57:05, finished the heavy part, and failed at 19:57:38 while listing blobs with `http2: server sent GOAWAY and closed the connection`, because kopia does not retry that listing. A shorter maintenance on the previous day succeeded on the same repository. This sets `keepalive_time 12h` through `http-snippet` in the controller ConfigMap. Idle connections are still closed after `keep-alive` (75s), so only connections that are actually in use live longer. No cozystack template sets `http-snippet` today; the tenant ingress HelmRelease only sets other `controller.config` keys, which Helm merges with this default, and a test covers that merge. Applied by hand on a production cluster before this PR: the controllers reloaded cleanly and S3 traffic was unaffected. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(ingress-nginx): raise the client keepalive_time from the nginx 1h default to 12h, so long-lived HTTP/2 connections (for example kopia maintenance against S3) are no longer cut by a GOAWAY after one hour ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * The ingress controller now configures NGINX to allow HTTP connections to remain open for up to 12 hours, supporting long-running operations such as S3 transfers. * **Tests** * Added checks confirming the setting remains in place when PROXY protocol and whitelist configuration are enabled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
7ec746a360 |
fix(monitoring): select the vmalert Service the operator creates (#5094)
## What this PR does
`templates/vm/vmalert-scrape.yaml` selects `app.kubernetes.io/instance:
vmalert`. The chart names the VMAlert `vmalert-<first metricsStorages
name>` (`templates/vm/vmalert.yaml` renders only the first storage), and
the operator labels its Service with that name, so with the default
storages the Service `vmalert-vmalert-shortterm` carries
`app.kubernetes.io/instance=vmalert-shortterm`. The VMServiceScrape
matches no Service and the metrics of vmalert itself never reach the
storage.
Seen on three v1.6.5 clusters. The `vmalert` VMServiceScrape in
`tenant-root` selects no Service, `count({__name__=~"vmalert_.*"})` in
vmselect-shortterm returns nothing while
`count(up{job="vmalertmanager-alertmanager"} == 1)` returns 3, and the
`/metrics` endpoint of vmalert does serve those series. No other scrape
object covers it, and no rule under `packages/` uses `vmalert_*`
metrics, so rule evaluation and notifier errors of vmalert are not
visible anywhere in the storage.
The scrape dates from v0.0.1, when the VMAlert was called `vmalert`.
|
||
|
|
900baf2215 |
fix(linstor): update piraeus-operator to v2.12.1 (#5166)
## What this PR does Updates the LINSTOR stack to the current stable piraeus-operator release: operator v2.12.1, LINSTOR server 1.35.3 and LINSTOR CSI v1.13.2, the last two built here with our patches. This started as a fix for the NFS server image hanging up the host console on bare-metal storage nodes (Ubuntu 24.04, Cozystack 1.6.5 and 1.7.0). That needed nfs-server v1.10.6 or later. Operators also asked for per-volume IO limits, which need linstor-csi v1.11.3 or later and piraeus-operator v2.10.8 or later, so the update goes to the current release rather than the minimum. The first commit fixes a problem the update exposed. The operator runs the LINSTOR database migration in a `run-migration` init container, and this package pinned only the main controller container to our build. With the operator update, that init container ran the operator's default LINSTOR 1.35.3 and migrated the database before the controller image changed. The 1.33.3 controller then crash-looped until the linstor release caught up. Both init containers that run the LINSTOR server image (`run-migration` and the satellite's `setup-lvm-configuration`) now use the platform image, with a test pinning it. This protects later upgrades where the operator default is newer than our build. It does not help this upgrade: the pin ships in the linstor release, and Flux reconciles piraeus-operator first. Patches: - linstor-csi: the Protocol C override for dual attach is dropped. Since v1.12 the driver asks LINSTOR to manage dual-primary, and LINSTOR 1.35 sets protocol C on the connections that need it. Upstream closed the same change as superseded (piraeusdatastore/linstor-csi#435). The relocate-after-clone patch is carried forward. - LINSTOR server: six of the nine patches are in upstream by 1.35.3 and are removed. `fix-k8s-create-already-exists`, `retry-adjust-after-stale-bitmap` and `native-protoc` are carried forward. LINSTOR 1.34 requires Java 21, so the image moves from bookworm to trixie. Behaviour that changes for users: - No downgrade. LINSTOR 1.35 migrates the database and drops the `SEC_*` tables, so a cluster that has run this version cannot go back to 1.33. - The LINSTOR controller is down for a short time during this upgrade. piraeus-operator is reconciled before the linstor release, so the database is migrated to 1.35 while the old controller still runs. That controller crash-loops until the linstor release rolls out, and if the rollout fails there is no way back. - A volume on a Protocol A or B resource group that was attached to two nodes while the CSI driver was updated (a live migration in progress) keeps the Protocol C override the old driver set on its resource definition. The new driver does not remove it. Data is not affected, the resource just stays on protocol C. Such resource definitions have the `Aux/csi-protocol-override` property. To remove the override, run `linstor resource-definition set-property <rd> DrbdOptions/Net/protocol` and `linstor resource-definition set-property --aux <rd> csi-protocol-override`. - The CSIDriver moves to `fsGroupPolicy: File`, so `fsGroup` is also applied to ReadWriteMany filesystem volumes. - The operator manages Secrets only in its own namespace, and the CSI controller loses cluster-wide read access to Secrets. Nothing in this repository references snapshotter Secrets. - The csi-health-monitor sidecar is removed. - LINSTOR's default evict timeout goes from 1h to 12h; this package does not set it. - Volume group snapshots turn on only when `groupsnapshot.storage.k8s.io/v1` is served. The snapshot controller here still serves v1alpha1, so they stay off; #5239 changes that. - IO limits can be set through `qos.linbit.com/{rbps,wbps,riops,wiops}` StorageClass parameters. Enforcing them needs NRI enabled on the nodes and the nri-volume-qos plugin, neither of which this PR adds. Tested on a three-node Talos cluster, upgrading from the current main packages to this branch: - LINSTOR database migrated to 1.35.3, no faulty resources, no error reports. - Data written before the upgrade is intact, checked by checksum. - A snapshot taken before the upgrade restores, and all three replicas come up `UpToDate`; cloning and new snapshots work. - KubeVirt live migration works for a ReplicatedBlock volume, including one on a Protocol A resource group: LINSTOR switches the connection to protocol C for the migration and restores A afterwards. - A StorageClass with QoS parameters provisions, and the limits reach the resource definition and the VolumeAttachment metadata. - ZFS userspace in the satellite is now 2.4.4, matching the kernel module. Not tested: an arm64 build. ### Screenshots No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(linstor): update piraeus-operator to v2.12.1, LINSTOR to 1.35.3 and LINSTOR CSI to v1.13.2. Fixes the NFS server breaking console login on storage nodes and adds IO QoS StorageClass parameters. LINSTOR 1.35 migrates its database, and a downgrade to an earlier version is not possible after the upgrade. ``` |
||
|
|
12393ec3d3 |
fix(etcd): size the defrag job for the client it runs (#5100)
## What this PR does The hourly `etcd-defrag` CronJob runs the `ghcr.io/ahrtr/etcd-defrag` client, which only sends defragment calls to the members, but it requested 200m CPU and 256Mi from the tenant quota and limited itself at 500m and 512Mi. On a v1.6.5 sandbox tenant created through the panel, its own etcd and SeaweedFS took 1525m of the 1600m CPU quota, every hourly job failed with `exceeded quota` before a pod was created, and the defrag never ran once. Over a day on another v1.6.5 cluster the client peaked at 5.1 MiB working set, and its runs are too short to show up in a CPU rate. The container now requests 10m and 32Mi and is limited at 100m and 64Mi, so it fits a tight tenant quota and still leaves the client room above what it was seen to use. The tenant LimitRange only sets defaults, so explicit small values are not rejected. A new case in `tests/defrag_test.yaml` pins the resources. `helm unittest .` in `packages/extra/etcd` passes 26 of 26 with helm-unittest 1.1.2, and against the template on main the new case fails and the other 25 pass. I have not run the change on a live cluster. The template is the same on release-1.6, and that backport is a separate PR. ### Screenshots Not a UI change. ### Downstream repositories One template of `packages/extra/etcd` and its test. No values key, schema, default or package change, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(etcd): the tenant etcd defrag job requests 10m and 32Mi instead of 200m and 256Mi, so it runs in tenants whose quota is nearly used up by their own services. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Updates** * The etcd defragmentation job now has lower CPU and memory requests and limits: requests are 10m CPU and 32Mi memory, and limits are 100m CPU and 64Mi memory. This reduces the resources configured for the job. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
7c46e1e384 |
fix(backups): give the barman-cloud sidecar an explicit CPU limit (#5280)
## What this PR does The barman-cloud sidecar of Postgres backups is deliberately left without a CPU limit, so that a throttled sidecar slows WAL archiving down instead of failing it. In a tenant with `resourceQuotas` set that intent does not hold: `packages/apps/tenant/templates/quota.yaml` creates the `tenant-range-limits` LimitRange, whose default CPU limit is `250m`, and the sidecar receives it. Full backups and WAL segments are then gzip-compressed on a quarter of a core. On a production tenant the first full backup of a 1.6 GB database took 30 minutes, uploading at about 0.5 MB/s with the sidecar pinned at its 250m limit, while a small database on the same destination finished in 35 seconds. This sets an explicit `limits.cpu: "1"` on the sidecar, in the two places that build its configuration: `barmanSidecarConfiguration` in the backup driver (platform-managed ObjectStores) and `cozy-lib.barman.sidecarConfiguration` (ObjectStores rendered by the postgres and keycloak charts). This mirrors how the memory request and limit there already override the LimitRange's 128Mi default. One core removes the 250m ceiling without letting the sidecar starve the instance. Inside a tenant it counts against the `limits.cpu` quota, one core per instance pod; the comments now say so. The sidecar is injected when the instance pod is created, so existing instances pick the new limit up at their next restart. Tests: the Go test that asserted the absence of a CPU limit now asserts one core, and so does `packages/apps/postgres/tests/barman_resources_test.yaml`. Both fail with the implementation reverted and pass with it. `go test ./internal/backupcontroller/`, `helm unittest packages/apps/postgres` (220) and `helm unittest packages/system/keycloak` (121) pass. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff (the backup driver, `cozy-lib`'s barman template, two tests): no app, value, schema, default, CRD, annotation or tooling change, so none of the listed repositories restates anything touched here. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the barman-cloud sidecar of Postgres backups now has an explicit one-core CPU limit instead of inheriting the tenant LimitRange's 250m default, which made full backups and WAL archiving several times slower. Each instance pod of a tenant with a CPU quota now counts 750m more against `limits.cpu`; check the headroom before instance pods are recreated. Existing ObjectStores pick the limit up at their next backup (platform-managed) or HelmRelease upgrade (chart-rendered), and running pods at their next restart. ``` |
||
|
|
30c4232be1 |
fix(seaweedfs): stop the master crashing on the crowded volume map (#5308)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Fixes merged to `main` reach the release lines the backport batch runs for automatically, so do not open backport PRs by hand: https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md --> ## What this PR does Backports seaweedfs/seaweedfs#11216 ("topology: fix fatal concurrent map read/write on VolumeLayout.crowded", released in 4.46) as a patch on the in-tree 4.45 image, next to `reader-cache-prefetch-window.diff`. In 4.45, `(*VolumeLayout).SetVolumeCrowded` mutates the `crowded` map under `accessLock.RLock()`, while `(*VolumeLayout).GetWritableVolumeCount` reads the same map under `RLock()` on the `Assign` path. Two read-lock holders, one writing and one reading the map, make the Go runtime abort the master with `fatal error: concurrent map read and map write`. This cannot be recovered, so the process exits and S3 requests that need an assign fail until a master is back. The fix takes the write lock in `SetVolumeCrowded`; every other mutation of `crowded` already holds it. `patches/volume-layout-crowded-lock.diff` is the upstream squash commit 5d5ea18287cfbf9e274f4b78be40a0ae220cb55d (files under `weed/topology/`), regenerated against the 4.45 source so it applies without offset. Its content is unchanged, including the regression test `TestSetVolumeCrowdedNoRaceWithGetWritableVolumeCount`, which the Dockerfile's existing `go test` step runs on every build. #### Seen in production On a production platform running Cozystack v1.7.0 (in-tree SeaweedFS 4.45), with a dedicated S3 cluster used as the Velero backup target: - Since the upgrade to 4.45 on 2026-11-05, all 3 masters crash every night with exit code 2 under the backup load, between 01:00 and 03:00 UTC: 2 crashes on 2026-11-06, about 23 on 2026-11-13, 33 to 43 restarts per master over 8 days. - The stack is always the same: ``` fatal error: concurrent map read and map write ... github.com/seaweedfs/seaweedfs/weed/topology.(*VolumeLayout).GetWritableVolumeCount(...) weed/topology/volume_layout.go:834 github.com/seaweedfs/seaweedfs/weed/server.(*MasterServer).Assign(...) weed/server/master_grpc_server_assign.go:97 ``` - Clients see HTTP 500 `InternalError` from the S3 gateway, and the nightly backups end `Failed` or `PartiallyFailed` (4 `Failed` and 3 `PartiallyFailed` on the night of 2026-11-12 to 13). Upstream issue: seaweedfs/seaweedfs#11211. #### Testing - `git apply --check` of both patches against the 4.45 source archive (`SOURCE_SHA256` in the Dockerfile matches). - Regression test on 4.45 with only the test applied, `CGO_ENABLED=0` as in the image build: `fatal error: concurrent map writes`, FAIL. With the fix: `go test ./weed/topology/` passes, and the new test passes under `-race` too. - `docker buildx build images/seaweedfs --platform linux/amd64` builds, running the patched packages' tests. - `make test` in `packages/system/seaweedfs`: 107 helm unit tests, the naming-guard check and the image/chart version check pass. #### Lifecycle The patch is dropped once the vendored chart reaches 4.46 or later, for example with #4879 (4.48 bump). From that release on it no longer applies, and `make update-image` fails until it is removed, as the Dockerfile header says. This PR is kept small so that it does not wait for #4879. #### Backport This is a `fix` with no CRD, migration or API change, and `release-1.7` carries the same `images/seaweedfs/` as `main`, so the backport batch should take it onto `release-1.7` with a clean cherry-pick for the next v1.7.x. Clusters on v1.7 are the ones that crash today. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff touches only `packages/system/seaweedfs/images/seaweedfs/` (a Go patch and a Dockerfile comment): no package added or renamed, no values, schema, CRD, `hack/` or installer change, so no entry of the trigger map applies. ### Release note ```release-note fix(seaweedfs): stop SeaweedFS masters crashing with "concurrent map read and map write" under write load, which made S3 answer 500 (backport of seaweedfs/seaweedfs#11216 onto 4.45). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability when updating crowded-volume state concurrently with writable-volume queries. * **Documentation** * Added notes about the upstream fix and when the local patch can be removed. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3fce040268 |
Merge branch 'main' into fix/etcd-strict-member-spread
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
054852cfce |
Merge branch 'main' into feat/pod-replicas-colocated-alert
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0bfd98fb86 |
Merge branch 'main' into fix/vm-disk-size-below-image
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
b5543d8260 |
Merge branch 'main' into fix/app-job-names-63
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
dd8649f9cc |
Merge branch 'main' into fix/kubevirt-csi-missing-pvc-unpublish
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
95364b8ba0 |
Merge branch 'main' into fix/tenant-cdi-upload-rbac
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
5543cfc37c |
Merge branch 'main' into fix/watchdog-heartbeat-alarm
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
d1f95e3212 |
Merge branch 'main' into fix/etcd-defrag-small-requests
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
78d79b22f6 |
Merge branch 'main' into fix/vpa-admission-two-replicas
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e8c8fa3ec7 |
Merge branch 'main' into fix/workloadmonitor-bucketclaim-ready
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
4f19e038c9 |
Merge branch 'main' into fix/monitoring-vmalert-scrape
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e8475963ee |
Merge branch 'main' into fix/monitoring-infoinhibitor
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
3b893c626a |
fix(cozy-lib): state the real lower bound in the goMemLimit error
The limit that floors to 0MiB is anything under 4/3 Mi, not under 2Mi: 1536Ki still yields 1MiB. The message named the wrong bound and the tests pinned 2Mi and 1Mi, neither of which sits on the edge. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
97e0618258 |
Merge branch 'main' into fix/postgres-backup-schedule-five-fields
Regenerate the postgres ApplicationDefinition schema on top of main. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e3e2018075 |
fix(seaweedfs, cozy-lib): cap the Go heap so components survive their memory limit
Every SeaweedFS component is a Go binary running under a cgroup memory limit, and none of them set GOMEMLIMIT. The collector then works from GOGC alone, a target relative to the live heap with no absolute ceiling, so a burst of allocation grows the heap past the limit and the kernel kills the container. Seen on an S3 gateway holding 6.4 GiB of heap_sys against an 8 GiB limit while only 389 MiB was actually in use: ten kills in seven days, each one an interruption of the object store every tenant writes its backups to. Raising the limit does not help, since nothing bounds the growth; it only moves the ceiling the heap climbs to. Derive the value from the limit already computed for each component, at the same 75% the javaHeap helper uses for the JVM. The remaining quarter covers what GOMEMLIMIT does not account for, goroutine stacks and mmap'd regions among them. Volume pools and zones carry their own value: the subchart merges volume into every entry of volumes, so a pool would otherwise inherit the value sized for the default volume's limit, above its own. The lifecycle admin and worker run the same binary under limits the parent chart does not set, so their value is pinned next to those limits. A limit under 2Mi floors to 0MiB, which Go accepts and then collects almost continuously; the render fails instead. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
5e07258acb |
Merge branch 'main' into feat/ingress-nginx-keepalive-time
Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
56224a156c |
feat(virtual-machine): let each disk set its own boot order (#5266)
## What this PR does
A `vm-instance` boots from its disks in the order they appear in
`disks`: the template hardcodes `bootOrder: {{ add $i 1 }}`. Booting
from installer or rescue media, such as an ISO attached as a CD-ROM next
to the system disk, therefore means moving that media to the front of
the list, and anything that treats `disks[0]` as the system disk then
finds the ISO there instead.
This adds an optional `bootOrder` to each entry of `disks`:
- When set, it is the disk's boot priority (1 is tried first),
independent of its position in the list.
- When unset, the disk keeps its positional order (`index + 1`), so
existing VMs render exactly as before.
- The values schema requires `bootOrder >= 1`.
- Duplicate effective boot orders are refused at render time with the
names of both disks, rather than surfacing as a KubeVirt admission error
on the VirtualMachine. This includes an explicit value colliding with
another disk's positional one, so booting from a CD-ROM in second
position means setting both disks:
```yaml
disks:
- name: system
bootOrder: 2
- name: install-iso
bus: sata
bootOrder: 1
```
`make generate` was run in the package, so the README,
`values.schema.json`, the Go type in `api/apps/v1alpha1/vminstance`
(with its regenerated deepcopy, since the field is a pointer) and the
`vm-instance-rd` OpenAPI schema are all updated.
A new suite, `tests/disk_boot_order_test.yaml`, mocks the DataVolumes
the template looks up and covers: positional order when no disk sets
`bootOrder`; a CD-ROM in second position booting first while the list
order and volumes stay unchanged; an explicit value colliding with a
positional one; two equal explicit values; and a value below 1 rejected
by the schema. Run against the previous template, the CD-ROM and
collision cases fail. `helm unittest packages/apps/vm-instance`: 6
suites, 77 tests passing.
### Screenshots
No UI change.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
This adds a field to `vm-instance`'s `values.schema.json`, which the
trigger map routes to `cozystack/terraform-provider-cozystack`: its
`vminstance_schema.go` and `vminstance_model.go` model `disks` by hand,
so `bootOrder` needs adding there. The follow-up has not been opened yet
and the box is left blank until it is linked. The website reference page
is regenerated from the README by the release bot; nothing else in the
map is touched.
### Release note
```release-note
feat(virtual-machine): disks in a vm-instance accept an optional `bootOrder`, so a VM can boot from an attached ISO or other rescue media without reordering its disks. Unset keeps the previous positional boot order.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Configure a boot priority for each VM disk. Lower values are tried
first; disks without an explicit priority use their position in the disk
list.
* Disk priorities must be unique, and values below 1 are rejected.
Priority changes do not reorder disks or their associated volumes.
* **Documentation**
* Added guidance and an example for configuring disk boot priorities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
6cab97f010 |
style: use tagged switches and drop unused test assignments
Use tagged switches where staticcheck suggests them and drop two test assignments whose values are never read. Behaviour is unchanged. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
456ea6b98b |
ci: run the standalone pkg and cmd unit tests in go-unit-tests
go-unit-tests ran only pkg/registry, pkg/config and pkg/cmd/server, so the tests in pkg/lineage, pkg/ovnstatus, pkg/reservation, cmd/cozypkg and cmd/check-readiness ran on no CI job. None of them needs a cluster or the network. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
a3305e8dd3 |
fix(backport): push and publish nothing once output is lost
backport-batch writes its body, summary, findings and warnings before it pushes, opens or edits a pull request, closes a batch or posts a comment. A run that kept going after that output was lost acted on what nobody read, and a closed pipe was worse: with Go's default SIGPIPE handling the process died silently mid-run. Lines that confirm an action after it happened, such as the URL of an opened pull request, go to stderr, so a lost stderr matters as much as a lost stdout. A failed write to either stream is kept, and from then on every push and every GitHub write is refused at the point it would be made: the GitHub client is wrapped so that each write checks first, and each push checks before it runs. Once output is lost, the run changes nothing more: no push, no GitHub write, no local branch move. No new unit starts after output is lost; a unit already in flight finishes its rollback. A reset whose own hint, the only way back to the commits it drops, cannot be written does not run either. The run exits 2 saying the output was lost; a confirmation lost after an action fails the run too. Such a confirmation is lost only when stderr itself fails, so a lost confirmation can only be reported by the exit code. A test holds every method of the GitHub interface to being a known read or a guarded write. The tool catches SIGPIPE with signal.Notify, so a closed pipe is a write error like any other. It does not use signal.Ignore, whose ignored disposition the git and gh children would inherit across exec; a test checks that a child still dies of SIGPIPE. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6f2d6ed17b |
fix(sse): refuse streams that open while the server shuts down
Shutdown stopped the clients registered at that moment and then shut the HTTP server down. A stream accepted in between registered after the stop loop, so nothing ended it, and http.Server.Shutdown waited for its handler. The kube-ovn plunger calls Shutdown without a deadline, so such a stream delayed stopping until the manager's graceful-shutdown timeout and then failed it. Shutdown now marks the server as closing under the same lock, and a stream that opens after that is answered 503 without being registered. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
40d340a3fc |
fix(sse): wait for the pump before returning
When a client disconnected, the handler stopped its pump and returned at once, while the pump could still be writing an event or a heartbeat to the response. net/http does not allow a write to a response after ServeHTTP has returned. The handler now waits for the pump to exit; a pump blocked in a write returns once the connection is gone. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
86da6c7a5b |
style(lineage-webhook): prefix sentinel error names with Err
Go names sentinel error variables Err...; the package is internal and the webhook is their only user. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
648928f272 |
refactor(workloadmonitor): use k8s.io/utils/ptr instead of pointer
The pointer package is deprecated; its Bool and BoolPtr are aliases of ptr.To, so the values written are unchanged. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
939a1ec44b |
refactor: read managed fields through FieldsV1 accessors
Direct access to FieldsV1.Raw is deprecated; GetRawBytes returns the same bytes and is nil-safe, and NewFieldsV1 builds the test fixtures. The write-options test still covers a raw-only orphanDependents delete option, now decoded from its wire form, and asserts that the converted options keep it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
50959ccca2 |
chore(backport): discard temp-file cleanup errors explicitly
A temporary file or directory that cannot be removed, or closed after its write already failed, leaves nothing to act on: the run's result is the same, and the primary error is the one returned. The test that unsets an environment variable now fails if it cannot. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |