cozystack

0 Fork 0

mirror of https://github.com/cozystack/cozystack.git synced 2026-11-09 05:25:14 +09:00

Author	SHA1	Message	Date
Yan Bondarenko	cee9065589	fix(linstor): wait for temporary probe devices Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 17:30:00 +04:00
Aleksei Sviridkin	88a7ea0e3c	fix(kubeovn): retry VM migration setup after scheduling (#4253 ) ## What this PR does Kube-OVN can drop VM migration setup when a migration enters `Scheduling` before the target Pod is visible or the source or target node is known. The phase may remain unchanged afterward, leaving the target Pod waiting for network setup. Backport the final fix from kubeovn/kube-ovn#6834 to the pinned v1.15.10 image: return an error for the transient missing-node case so the existing rate-limited queue retries it. Keep the same-node behavior unchanged. That backport can be removed when updating to v1.15.14 or later. Add a separate retry guard for the target-Pod creation race. It returns an error when the informer cache has no target Pod yet, instead of forgetting the migration event. This guard is not part of #6834 and must remain until an equivalent fix is verified in the selected upstream version. Also include the existing kubeovn image target in the root build list. Without it, a kubeovn patch produces an empty PR build matrix and the image is omitted from the regular build and release paths. Validation: - All six regression cases pass on Linux/amd64 using the real controller workqueue. The new missing-target-Pod case fails without its guard and passes after the Pod appears in the informer cache, without changing the VMIM phase. The existing deferred-node, already-scheduled, and same-node cases continue to pass. - All 19 build-matrix tests pass; the full matrix includes kubeovn exactly once. - Both lint steps, pinned golangci-lint v2.10.1 and upstream `modernize -test -skipgenerated ./...`, pass for the Linux/amd64 target using Go 1.26.4. On macOS, modernize is built for the host and run with the Linux target environment, since `go tool` otherwise tries to execute a cross-compiled Linux binary. - All three packaged patches apply cleanly to pinned v1.15.10 with `git apply --check --whitespace=error`; the resulting source matches the tested tree. Formatting and source/build whitespace checks pass. The regression test covers controller queue behavior; the standard container E2E environment does not cover DRBD-backed live migration. The regular image build applies the backport and stamps the resulting digest before packaging. ### Screenshots Not applicable; no UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The change adds an existing image target to the build list; package layout, make target interfaces, chart values, APIs, and published image-reference formats stay the same. CCP discovers package image targets dynamically and does not carry the root build list. The website has no documented workaround for kubeovn/kube-ovn#6823 or #6834 to remove. ### Release note ```release-note fix(kubeovn): retry VM migration network setup when the target Pod or source/target scheduling information is not yet available, and include the Kube-OVN image in the regular build matrix. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes - Improved VM migration handling when source or destination node scheduling information is temporarily unavailable. - Migration events are now retried after scheduling completes instead of being dropped. - Preserved existing behavior when migration source and destination nodes are identical. - Tests - Added coverage for migration scheduling and retry scenarios, including partially assigned and same-node cases. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 17:29:45 +04:00
Aleksei Sviridkin	08a2e76707	fix(opensearch): remove the data PVCs when the application is deleted (#4136 ) ## What this PR does Fixes #4086. The OpenSearch operator provisions its data volumes through a StatefulSet, so the PVCs carry no ownerReference, and the `opensearch` app chart has no cleanup hook. Removing the application therefore leaves every data volume `Bound`, counting against the tenant's storage quota with nothing left in the namespace that could reclaim it. `clickhouse`, `kafka` and `harbor` remove their operator-owned PVCs in a post-delete hook. This adds a post-delete Job to `opensearch` that requests the deletion of the claims labelled `opster.io/opensearch-cluster=<release>` with `--ignore-not-found --wait=false` and exits non-zero when the request fails, with its own ServiceAccount, a Role limited to `persistentvolumeclaims` (`get`, `list`, `delete`), the `clastix/kubectl` image pinned by digest through `cozy-lib.image`, a non-root restricted securityContext, `activeDeadlineSeconds: 120` so a kubectl that never returns cannot hold the uninstall, and `before-hook-creation` alone on the Job so a failed run and its log survive; the ServiceAccount, Role and RoleBinding keep `hook-succeeded`. The delete does not wait for the claims to disappear, on purpose: OpenSearch data pods stop slowly, and a claim they still hold would keep a waiting hook running into helm-controller's 300s uninstall budget. The claims go once their pods have stopped, which is what the log says. This is the shape the monitoring, etcd and seaweedfs hooks use. Retention is not offered as a knob, consistent with the other charts; an opt-out can be added if the maintainers prefer one. `helm unittest`: 64/64 in the chart, including `tests/cleanup_pvc_test.yaml` (hook annotations and deadline, selector and exit path, digest pin and registry prefix, securityContext, Role rules, RoleBinding). `helm template` renders the Job, ServiceAccount, Role and RoleBinding next to the existing objects. ### Downstream repositories A new hook template and its tests only: no values, schema, README or defaults change. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch): reclaim the data PVCs of a removed application with a post-delete hook ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * New Features * Automatically cleans up orphaned OpenSearch persistent volume claims after a release is deleted. * Cleanup waits for associated pods to stop and uses restricted permissions and hardened security settings. * Supports optional container registries with pinned cleanup images. * Tests * Added coverage for cleanup behavior, hook configuration, security settings, registry overrides, and required access permissions. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 17:29:38 +04:00
Timofei Larkin	8f19910988	chore(platform)!: remove the HTTPCache application (#4200 ) ## What this PR does Removes the HTTPCache application. Telemetry says it is the least installed app we ship: 3 instances in June, July and August 2026, 4 so far in September, out of 1162-1418 clusters reporting per month. That is not zero, so I am not going to claim nobody uses it. The maintenance side is what does not hold up. `make update` in this package is broken. All eight sed lines in the `update` target patch `images/nginx/Dockerfile`, but the image lives in `images/nginx-cache/Dockerfile`, so the target fails on the first line. It is the only package under `packages/apps/` pointing at a Dockerfile that does not exist. The pins show what that cost us: ``` nginx 1.25.3 -> 1.31.5 ngx_cache_purge 2.5.3 -> 3.0.2 nginx-module-vts 0.2.2 -> 0.2.7 ip2proxy-c 4.1.2 -> 4.3.0 IP2Location-C 8.6.1 -> 8.7.0 ip2location-nginx 8.6.0 -> 8.7.0 ip2proxy-nginx 8.1.1 -> 8.2.0 51Degrees 3.2.21.1 -> 3.2.21.1 (frozen upstream) ``` So we ship an old nginx with third party C modules compiled into it, and nothing tests it: no chainsaw suite, no `helm unittest` target. I don't think we should keep carrying that for 4 instances. Removed: the chart in `packages/apps/http-cache`, its ResourceDefinition in `packages/system/http-cache-rd`, the values type in `api/apps/v1alpha1/httpcache`, the `cozystack.http-cache-application` PackageSource, its line in the `naas` bundle, the `httpcaches` entry in `cozy:tenant:admin:base`, the dashboard icon, and the entry in the root `Makefile` build list. `cozy:tenant:admin:base` enumerates the `apps.cozystack.io` resources a tenant owner may write, so `httpcaches` comes out of it with the rest of the application. A cluster that freezes gets the grant back from the migration instead, which keeps the chart's list tracking the catalog and leaves no entry behind for a later sweep to read as dead config. ### Existing installations A cluster that still runs an HTTPCache keeps it, served from a frozen source. A cluster that never created one keeps nothing. Without a migration an upgrade would break both. `templates/sources.yaml` is a bare glob with no keep annotation, so Helm prunes the http-cache PackageSource; the ArtifactGenerator is owned by it and goes too, and both ExternalArtifacts go with the generator. Those are the `chartRef` targets of `hr/http-cache-rd` and of every tenant `http-cache-<name>` release, so all of them would sit `Ready=False` for good and `HelmReleaseNotReady` would fire on every cluster with the naas bundle. Migration 58 runs as a pre-upgrade hook, before that prune, and branches on whether any instance exists. Its header comment carries the mechanism in full; this is the summary. If one does, it copies the packages `OCIRepository` and pins the copy to the manifest digest that object resolves at hook time, which is the release the instances are rendering from right then. `spec.ref.digest` is used when the installer pinned one; a tag or semver ref goes through `status.artifact.revision`, whose digest half is the manifest digest. `status.artifact.digest` is the checksum of the downloaded tarball and cannot stand in, and a source with neither aborts the migration rather than publishing a source that points nowhere. Reading the digest live rather than baking one into the script makes the freeze a no-op by construction and asks no question about which release was the last to ship the chart. The copy takes the url, the pull secret, the interval and any verification block from the live object, so an air-gapped or mirrored registry keeps serving exactly what it already served. It is written with no owner references and no Helm metadata, and with the `platform.cozystack.io/no-delete` label the platform chart puts on the original, because it becomes the only chart source those instances have. The PackageSource is then repointed at the copy, annotated `helm.sh/resource-policy: keep` and stripped of its release metadata, so neither Helm nor a later release adopts or prunes it. Helm re-reads that annotation off the live object when it prunes, which is what makes disowning from inside a pre-upgrade hook work. Names stay as they are, deliberately. The ApplicationDefinition inside that artifact hardcodes its `chartRef`, so a renamed PackageSource would leave every release pointing at an artifact nobody produces. Because the names hold, `hr/http-cache-rd` and every tenant release keep the `chartRef` they already have and nothing goes not ready. What changes is where the chart comes from: a pinned digest that no later release moves. The migration also applies `cozy:tenant:admin:http-cache-frozen`, a ClusterRole labelled `rbac.cozystack.io/aggregate-to-tenant-admin`, granting the verbs `cozy:tenant:admin:base` used to carry on `httpcaches`. The hook applies it before the upgrade drops the rule from the chart, so an owner's access never lapses and nothing changes for them. Without it a frozen instance would be visible and undeletable by the person who owns it: the wildcard over `apps.cozystack.io` lives in `cozy:tenant:base`, which aggregates into `cozy:tenant` and not into `cozy:tenant:admin`, `cozy:tenant:view:base` is read only, and no role in the tenant chain carries a verb on `helm.toolkit.fluxcd.io`. If no instance exists, the migration removes the platform side instead: the Package first and waited on, because the operator reconciles off an informer cache and re-creates the release it owns while that Package is still there, then the release, its ApplicationDefinition and its Helm storage. Nothing is pinned or granted on a cluster that never used the application. Every path is fail-closed. A failed fleet scan, a failed read of the source OCIRepository, a source with no digest to freeze on, a failed patch or a failed delete stops the migration before it stamps the version. The migration is numbered 58, because main already carries a 57 from the SeaweedFS volume-size-limit change and its `targetVersion` is already 58. #4287 wants 58 as well, so whichever of the two lands second renumbers and bumps `targetVersion` with it. main is merged in rather than rebased, because the branch carries commits that are not mine to rewrite; the one conflict was `migrations.targetVersion`, resolved as 59, one past this branch's migration number. ### The procedure, written down `docs/deprecating-applications.md` is new, and is the point of doing this one carefully. It covers how to choose between migrating instances onto a successor and freezing them, the chain that makes a bare chart deletion strand every surviving instance, how the freeze reads its digest off the cluster's own source, what the migration owes a cluster on each branch, and where the tenant RBAC grant goes. `AGENTS.md` points at it so the next retirement starts there rather than rediscovering it. ### If you run HTTPCache Nothing changes on upgrade. Your instance keeps running and stays manageable through the Cozystack API and the dashboard, and `kubectl get httpcaches` keeps working. What changes is that it is frozen. The chart now comes from the packages artifact your cluster was already running, pinned by digest, and gets no further updates from us, including security updates to the nginx build. There is no replacement in the catalog. Delete the HTTPCache object when you no longer need it, the way you always did; the migration keeps the tenant permission that lets you. ### Verification `helm template` on `packages/core/platform` before and after, for `isp-full`, `isp-full-generic` and `isp-hosted`. In all three the differences are the `http-cache` Package and PackageSource going away and `migrations.targetVersion` moving from 57 to 58, which on a cluster below 58 also renders the migration hook Job and its RBAC. The migration suite `hack/migration-58-http-cache-freeze.bats` covers both branches, the order of every step and each fail-closed path. Freezing on the tarball checksum instead of the manifest digest, dropping the `no-delete` label, dropping the `managed-by` null, dropping the tenant grant, returning from the Package delete without waiting, freezing unconditionally and granting on a cluster with no instance each turn the matching test red. `packages/system/cozystack-basics/tests/clusterroles-tenant-admin-aggregation_test.yaml` pins the aggregation label the migration's ClusterRole reaches tenant admins through, so renaming it fails in the chart rather than silently on the clusters that need the grant. Green: `helm unittest` for `packages/core/platform` and `packages/system/cozystack-basics`, `make migrations-target-check`, `hack/cozystack-version-stamp.bats`, `go test ./internal/controller/cacert/...`, `go build ./...` in the `api/apps/v1alpha1` module, the console `ServicesTab` and `sidebar-icons` suites with `tsc --noEmit`, `make rd-presets-check`, and the bats suites that enumerate packages and image refs. ### Screenshots The catalog entry and its nginx sidebar icon are gone. Nothing new to show. ### Downstream repositories Two repos are affected. No follow-up is open yet, so no box is ticked. `cozystack/website` carries `content/en/docs//networking/http-cache.md` in eight doc versions and lists `http-cache` in the hardcoded `NETWORKING` list in its Makefile. The `next` page and the Makefile entry are what need to go, released versions stay as history. `cozystack/terraform-provider-cozystack` has a `cozystack_httpcache` resource and data source. The resource keeps working after this: it is hand written against its own terraform-plugin-framework model and imports nothing from here. What breaks, once that repo bumps its `api/apps/v1alpha1` pin past this release, is `internal/provider/httpcache_model_test.go`, the per kind schema guard, which is the only file over there importing this module. The resource and the guard should go together. The other repos have no reference to the package. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note chore(http-cache)!: remove the HTTPCache application from the catalog. New instances can no longer be created from the catalog and the `HTTPCache` values type is gone from the `api/apps/v1alpha1` Go module. A platform migration keeps existing instances working: on a cluster that runs one, the application is frozen on the packages artifact that cluster was already rendering from, pinned by digest, and stays manageable through the Cozystack API and the dashboard, but receives no further updates, including security updates to the nginx build. On a cluster with no instance the application is removed outright. There is no replacement in the catalog; delete the HTTPCache object when you no longer need it. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit Removed * Removed the HTTP Cache application and its deployment, configuration, dashboard metadata, and packaging from the platform. * The HTTP Cache application is no longer included in the NaaS bundle. * HTTP Cache resources and related API definitions are no longer available. * Migration * Existing HTTP Cache installations are frozen during migration; environments without instances are cleaned up. * Platform migration target advanced to version 58. * Documentation * Updated storage immutability documentation to reflect Valkey instead of HTTP Cache. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 17:03:21 +04:00
IvanHunters	4844e1d99c	fix(opensearch-operator): stop excluding a node whose drain cannot finish (#4323 ) ## What this PR does The previous patch let a permanently yellow cluster past the restart health gate. With three or more nodes `packages/apps/opensearch` renders `drainDataNodes: true`, and the restart then reaches `PreparePodForDelete`, which excludes the candidate node from allocation before it can know whether the drain will finish. On the cluster that made the gate wait it never finishes: a shard on that node has nowhere else to go, which is the same condition holding the cluster yellow. The exclusion is only withdrawn after a pod is really deleted (`rollingRestart.go`, `upgrade.go`), so a data node stays out of allocation for as long as the restart is stuck, shedding its shards onto the rest of the cluster and buying nothing. Before the yellow gate this path was unreachable with draining on, so nothing was written at all. `patches/skip-unusable-drain-exclusion.diff` asks the allocator before excluding anything, and only while nothing is moving, so a drain making progress costs nothing extra. A started shard answering `can_move_to_other_node: no` means the drain cannot finish: no exclusion is written, and one left by an earlier pass is withdrawn. A drain that can still progress behaves exactly as before, and the two-data-node path is untouched because it never waits for a full drain. Checking after the exclusion instead would flap: the next reconcile ten seconds later would apply it again. The v2.8.0 base has no allocation-explain call of its own, so the patch adds the client method and a response type carrying only the field the decision reads. The second commit makes the shipped tests real. The image build ran `go build` alone, so a test file that stopped compiling, or stopped asserting, still produced an image. `go test` over the two package trees the patches touch now runs in the builder stage; both run without a cluster. Upstream as opensearch-project/opensearch-k8s-operator#1585, fixed there by opensearch-project/opensearch-k8s-operator#1586. Both patches go away with the first operator release carrying that change. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff: two files under `packages/system/opensearch-operator/images/opensearch-operator/`. No package README or values schema, no API or CRD, no node prerequisites in `hack/`, no vendored chart, no make target. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch-operator): a rolling restart or upgrade that cannot drain its next node no longer leaves that node excluded from shard allocation for as long as the restart is stuck ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * Bug Fixes * Improved node draining during OpenSearch maintenance and restarts. * The operator now checks whether shards can move before excluding a node from allocation. * If a shard cannot relocate, the existing exclusion is withdrawn and draining pauses, helping prevent nodes from remaining unnecessarily unavailable. * Existing behavior is preserved for two-node clusters and while shard relocation is already active. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 15:17:18 +03:00
Yan Bondarenko	54ac8ed135	fix(opensearch): keep the PVC cleanup hook from holding the uninstall kubectl delete waits for the claims to be gone unless told otherwise, and OpenSearch data pods stop slowly, so the post-delete hook could outlive helm-controller's 300s uninstall budget on a claim that would not release and leave the HelmRelease stuck uninstalling. The delete now runs with --wait=false, the shape the monitoring, etcd and seaweedfs hooks use: the deletion is requested and the claims go once their pods have stopped, which the log now says instead of "complete". The Job carries activeDeadlineSeconds 120 as the backstop for a kubectl that never returns and keeps only before-hook-creation, so a failed run and its log survive. The Role drops watch, which nothing needs once the wait is gone. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Assisted-by: LLM	2026-09-18 16:09:19 +04:00
Yan Bondarenko	13be303fbb	fix(opensearch): remove the data PVCs when the application is deleted The OpenSearch operator creates its data volumes through a StatefulSet, so the PVCs carry no ownerReference, and the chart had no cleanup hook. Removing the application left every data volume Bound and counting against the tenant quota, unlike clickhouse, kafka and harbor, whose post-delete hooks reclaim the operator-owned storage. A post-delete hook in the same shape as the clickhouse one now deletes the PVCs labelled opster.io/opensearch-cluster=<release>, with a Role limited to persistentvolumeclaims and the kubectl image pinned by digest. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 16:09:19 +04:00
Aleksei Sviridkin	bf26ce9744	fix(apps): let the cleanup hooks watch the objects they delete (#4135 ) ## What this PR does Fixes #4085. `kubectl delete` waits for the object to disappear unless `--wait=false` is given, and that wait is a watch on the object. The post-delete hook Roles of `harbor`, `clickhouse`, `mariadb` and `qdrant` grant `get`, `list` and `delete` only, so every uninstall logs ``` Failed to watch unstructured.Unstructured: persistentvolumeclaims "..." is forbidden: User "system:serviceaccount:<ns>:<release>-cleanup" cannot watch resource "persistentvolumeclaims" ``` the wait is skipped, and the hook prints `PVC cleanup complete` while the claims are still terminating. Every collection-scoped rule behind a delete in these four hooks now also carries `watch`: `persistentvolumeclaims` in all four; `certificates`, `challenges` and the solver `ingresses` in harbor. The four named Secrets the mariadb hook removes keep `get` and `delete` only: they carry no finalizer, so the delete never reaches the watch, and a `resourceNames` rule does not authorize a collection watch anyway. A real wait needs a bound. kubectl's default is a week, and a claim held by a pod that will not stop, or a stuck detach, would hold the hook until helm-controller's 300s uninstall budget ran out and leave the HelmRelease stuck uninstalling. The PVC deletes now run with `--wait=true --timeout=120s`, so the expiry is reported by the script with the claims still terminating named, and each Job carries `activeDeadlineSeconds: 270` with `terminationGracePeriodSeconds: 10`, the numbers `packages/apps/tenant/templates/cleanup-job.yaml` documents against that budget, as the backstop for a kubectl that never returns. `--wait=false` was not an option: it is exactly the report-complete-while-terminating behaviour this PR removes. The chart unit tests are updated and pass: harbor 21/21, clickhouse 24/24, mariadb 111/111, qdrant 104/104. The behavioural suite `hack/app-cleanup-hooks_test.bats` passes 11/11 against the changed scripts. The second observation in #4085, the harbor hook pod occasionally starting after its ServiceAccount is gone, is a hook-weight question and is not touched here. ### Downstream repositories Hook templates and their tests only: no values, schema, README or defaults change. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(apps): let the harbor, clickhouse, mariadb and qdrant cleanup hooks wait for the deleted objects, with the wait bounded so a stuck claim cannot hang the uninstall ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Enhancements* - Cleanup operations for ClickHouse, Harbor, MariaDB, and Qdrant now wait for persistent volume claim deletion to complete, with a 120-second timeout. - Cleanup jobs are limited to 270 seconds and allow 10 seconds for graceful termination. - Cleanup permissions now support monitoring relevant resources, including persistent volume claims and, where applicable, certificates, challenges, and ingresses. - Named Secret access is limited to retrieval and deletion. - Tests - Updated chart validation to verify cleanup timing, deletion behavior, permissions, and restrictions on unrelated resources. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 16:00:45 +04:00
IvanHunters	000e2e3121	build(opensearch-operator): run the patched tree's tests in the image build The patches under images/opensearch-operator/patches carry their own tests and nothing runs them. The build compiles main.go, and go-unit-tests in the root Makefile covers this repository's own Go tree, so a test file that had stopped compiling still produced an image: printf 'func Broken() { this is not go }' >> restart_gate_test.go go build -a -o manager main.go # exit 0 Run go test over the two package trees the patches touch before building the binary. Both run without a cluster, and the stage takes about eight minutes on an arm64 builder, most of it the module download the build needs anyway. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-09-18 14:21:14 +03:00
IvanHunters	28195d3ef8	fix(opensearch-operator): stop excluding a node whose drain cannot finish The yellow restart gate carried in the previous patch lets a permanently yellow cluster through, and with three or more nodes packages/apps/opensearch turns drainDataNodes on. The restart then reaches PreparePodForDelete, which excludes the candidate from allocation before it can know whether the drain will finish. On the cluster that made the gate wait it never finishes: a shard on that node has nowhere else to go, which is the same condition holding the cluster yellow. The exclusion is withdrawn only after a pod is really deleted, so a data node stays out of allocation for as long as the restart is stuck, shedding its shards onto the rest of the cluster and buying nothing. Ask the allocator before excluding anything, and only while nothing is moving, so a drain that is making progress costs nothing. A started shard answering can_move_to_other_node: no means the drain cannot finish, and then no exclusion is written and one left by an earlier pass is withdrawn. The two-data-node path keeps its behaviour, because it never waits for a full drain. Checking after the fact instead would flap: the next reconcile, ten seconds later, would exclude the node again. Upstream as opensearch-project/opensearch-k8s-operator#1585, fixed there by #1586; the patch goes away with the first release carrying it. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-09-18 14:04:59 +03:00
Timofei Larkin	c398e0e116	Merge branch 'main' into chore/drop-http-cache main took migration 57 and moved migrations.targetVersion to 58 while this branch was open, so that line is the one conflict: 57 at the merge base, 58 on main, 59 here. Resolved as 59, which is what this branch's migration 58 requires, since run-migrations.sh loops up to targetVersion - 1. Merged rather than rebased because the branch carries commits that are not mine to rewrite. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>	2026-09-18 13:20:03 +03:00
Timofei Larkin	2cbc02bd10	fix(platform): freeze HTTPCache on the artifact the cluster runs Pinning the frozen source to a digest written into the migration picks one release for every cluster, and v1.6.0 is not even the last one that ships http-cache: v1.6.1 through v1.6.3 and v1.7.0-alpha.1 all do, and 1.6 patch releases may keep doing so. A cluster on any of those would be rolled back to an older chart by the upgrade that was meant to leave its instances alone, and an air-gapped mirror resolves the constant only if someone mirrored that exact manifest. The hook runs before the upgrade re-points cozystack-packages, so the live object still names the release the instances render from, in whatever registry the operator serves it from and with whatever pull secret and verification they configured. Reading the digest off it makes the freeze a no-op by construction and needs no knowledge of which tag was last. spec.ref.digest is used when the installer pinned one; a tag or semver ref goes through status.artifact.revision, whose digest half is the manifest digest. status.artifact.digest is the checksum of the downloaded tarball and cannot stand in. A source with neither has nothing to freeze on and the migration stops there. The copy also carries the no-delete label the platform chart puts on the original, because it is the only chart source those instances have left. The tenant grant travels with the freeze as well. cozy:tenant:admin:base enumerates the apps.cozystack.io resources a tenant owner may write, and an application that has left the catalog has no business in that list: the next sweep over it takes the entry as dead config, and the failure is silent and remote, a Forbidden for the one person who still needs to remove what they have. The chart drops httpcaches, and a cluster that freezes gets a ClusterRole under the tenant admin aggregation label instead, applied before the upgrade drops the rule so the access never lapses and carrying the verb set the chart had so nothing changes for the owner. A cluster with no instance is granted nothing. The Package delete on the no-instance path has to block. The operator reconciles off an informer cache, so a reconcile in flight against the pre-delete entry re-creates the release it owns, and with it the ApplicationDefinition and the Helm storage the following lines remove. Returning immediately made the ordering a property of the command log rather than of the cluster. The preset check's SIGPIPE note named http-cache-rd as its example of a large enum, and that chart goes away here; kubernetes-rd carries the same enums. Numbered 58 rather than 57: main already carries a migration 57, the SeaweedFS volume-size-limit one, and its targetVersion is already 58. The whole shape of this retirement was worked out here rather than anywhere reusable, so docs/deprecating-applications.md writes it down as a procedure: how to pick between migrating onto a successor and freezing, what the prune takes with it, and what the migration owes a cluster on each branch. The next application to go should not have to rediscover it. Assisted-by: LLM Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>	2026-09-18 13:12:56 +03:00
Aleksei Sviridkin	301a7fed27	fix(nats): merge config.merge.accounts with the generated accounts map (#4134 ) ## What this PR does Fixes #4084. With `users` set, `templates/nats.yaml` rendered `config.merge.accounts.A.users` from the generated passwords and then appended `.Values.config.merge` verbatim. A tenant `config.merge` that also carried `accounts`, which is the only place to set `accounts.A.jetstream` for the generated account, produced the key twice in the HelmRelease values, and the install failed at post-render with `mapping key "accounts" already defined`. The template now builds a single `merge` map: a deep copy of `.Values.config.merge` with the generated users merged in under `accounts.A.users` via `mergeOverwrite`. The generated user list wins for account `A`; other keys under `accounts.A`, other accounts and top-level keys such as `max_payload` are kept; a `config.merge` without `users` passes through unchanged; and no `merge` block is rendered when neither is set. The values contract does not change. `helm unittest`: 67/67 in the chart. The new `tests/config_merge_test.yaml` covers the six shapes above; against the previous template the same suite fails on the three cases that combine `users` with `config.merge.accounts`. ### Downstream repositories Template and tests only: no values, schema, README or defaults change. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(nats): merge config.merge.accounts with the generated accounts instead of rendering the key twice ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes - Improved NATS configuration merging so generated user credentials and custom account settings are combined consistently. - Preserved tenant-listed users, additional accounts, and top-level configuration during merges. - Prevented empty merge sections from being rendered when no users or custom merge settings are configured. - Tests - Added coverage for user generation, account merging, configuration preservation, and empty-configuration scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 13:44:15 +04:00
Aleksei Sviridkin	a04887db3f	fix(linstor): add opt-in graceful satellite shutdown on Talos (#4292 ) ## What this PR does Cozystack's Talos configuration removes the systemd DRBD shutdown guard without a replacement. Add `talos.gracefulShutdown.enabled`, disabled by default, to release unused Secondary DRBD resources before the satellite stops so their backing devices, including ZFS zvols, can be released during normal shutdown. The hook skips Primary resources and open devices, rechecks each candidate, and aborts further actions on ambiguous status, command failure or timeout. It uses no force, demotion, Kubernetes API calls or additional permissions. The hook is available only when both Talos and DRBD are enabled. Keep the value in `spec.components.linstor.values.talos.gracefulShutdown.enabled` on the `cozystack.linstor` package. The chart embeds the script in its existing `cozystack-talos` satellite configuration, which Piraeus merges into the generated DaemonSets. No custom satellite image is required. Default renders remain unchanged for Talos, generic Linux and DRBD-less Talos. This is opt-in because preStop also runs on ordinary satellite restarts. Unused Secondary replicas disconnect until their replacement satellite reattaches them, and Piraeus has one DaemonSet per node without a cluster-wide rollout barrier. Simultaneous restarts can therefore pause I/O. Enable this only with sequential satellite/node maintenance and replication checks between nodes. The README covers the 60-second hook budget, pod versus node shutdown grace periods, rollback behavior and the unsupported forced-reboot/broken-kubelet paths. Related upstream discussion: https://github.com/piraeusdatastore/piraeus-operator/issues/860. The reported outage from unconditional forced demotion in https://github.com/piraeusdatastore/piraeus-operator/issues/860#issuecomment-5635881307 is why this hook never demotes a Primary, even when its devices appear closed. Validation: 22 Helm tests and 20 behavioral Python tests pass. The same 20 tests pass inside the existing Cozystack satellite image. Two of them pin the rendered command form: `python3 -c <script> --execute` reaches `main()` with `["--execute"]` because CPython sets `sys.argv[0]` to `-c`, and a script-name placeholder before `--execute` is rejected. Failure-path tests cover nonzero status/down exits, bounded stderr, partial timeout diagnostics, and stopping before the next resource. Full default renders match the base commit for all three supported substrate variants. The rendered script survives strategic merging against Piraeus 2.10.2 and 2.11.0 satellite manifests; the base chart fails the new feature's regression tests. Runtime validation on Talos 1.13.6 used the script from commit `b8c86f2e70d029166939ff29eab8a45ac191e34b` through per-node satellite configurations: a normal node reboot with drain completed without force or an external reset, and the node returned with its ZFS pool online and DRBD replicas UpToDate. That version is installed on all three nodes of the tested cluster after sequential satellite rollouts and replication checks. The later diagnostic-only update in `527570d8079cba7c91b3c345050cab1b73a290a8` is covered by the local and exact-image tests above but has not been rolled out to that cluster. This does not validate forced reboot, a failed kubelet, or simultaneous satellite restarts. GitHub Actions for this fork require maintainer approval; local checks do not replace that CI. ### Screenshots Not applicable. ### Downstream repositories The diff adds an opt-in system-package value and its implementation/tests. It does not change app schemas, platform variants, node prerequisites, shared tooling contracts or an existing documented workaround. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(linstor): add an opt-in Talos satellite preStop hook that releases unused Secondary DRBD resources without force. Enable it with talos.gracefulShutdown.enabled for sequential node maintenance; existing installations keep their current behavior by default. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * New Features * Added optional graceful shutdown support for LINSTOR satellites running on Talos. * When enabled, unused Secondary DRBD resources are released before termination, with safeguards for active resources and bounded shutdown timing. * Added a 90-second termination grace period and configuration validation for the new setting. * Added documentation covering configuration, operational requirements, and shutdown behavior. * Tests * Added Helm and Python coverage for shutdown behavior, validation, timeouts, and failure scenarios. * The test target now runs both Helm and Python unit tests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 13:43:57 +04:00
Aleksei Sviridkin	a4bdddf030	feat(kubevirt): expose migration configuration through platform values (#4254 ) ## What this PR does Cluster-wide migration settings could only be applied by patching the KubeVirt CR: the kubevirt chart did not render `migrations`, and the platform bundle did not forward it. Add the optional `kubevirt.migrations` platform value and pass it through the generated Package to `spec.configuration.migrations`. For example, set the following under `spec.components.platform.values` on the `cozystack.cozystack-platform` Package: ```yaml kubevirt: migrations: bandwidthPerMigration: 625M parallelMigrationsPerCluster: 2 parallelOutboundMigrationsPerNode: 1 ``` `bandwidthPerMigration` is a quantity in bytes per second. Empty or null settings leave the field unset; existing KubeVirt defaults are unchanged. The map is forwarded intact, including explicit `false` and `0` values. `make update` restores the new parameterization independently after a hand-merge. Validation: - All 207 Helm tests in `packages/system/kubevirt` and `packages/core/platform` passed (42 suites). - Before the template changes, the new limit and false/zero cases failed at both forwarding stages (four expected regression failures). - The GNU sed / make update-idempotency suite passed, including restoration of a missing migrations block, repeated runs, and rejection of a missing insertion anchor. - Standalone KubeVirt rendering and `helm lint` passed. Neither modified chart defines a `make generate` target. Full platform rendering requires its live source repository lookup; the platform template behavior was checked through its existing Helm unit-test fixtures. This is separate from #3989, which exposes `vmStateStorageClass` in the same chart and bundle files. ### Screenshots Not applicable; no UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: [#697](https://github.com/cozystack/website/pull/697) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The website follow-up [#697](https://github.com/cozystack/website/pull/697) documents this new platform value in the unreleased `next` reference and should be merged after this implementation. The merged website PR [#678](https://github.com/cozystack/website/pull/678) documents `kubevirt.disabledFeatureGates` only. The other listed repositories do not need a coordinated change: this adds an optional platform value without changing app schemas, required installer values, CRDs, package layout, source kinds, shared tooling contracts, or node prerequisites. ### Release note ```release-note feat(kubevirt): Configure cluster-wide KubeVirt migration settings through the platform Package value kubevirt.migrations. Empty settings preserve existing defaults. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * New Features * Added optional KubeVirt migration configuration. * Supports migration bandwidth, concurrency limits, auto-convergence, and completion timeout settings. * Migration bandwidth can be specified in bytes per second. * Empty or unset migration settings preserve KubeVirt defaults. * Bug Fixes * Migration values now preserve valid `false` and `0` settings when provided. * Tests * Added coverage for default, empty, null, configured, and repeated-update scenarios. * Added validation to prevent incomplete or duplicated migration configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 13:43:52 +04:00
Aleksei Sviridkin	325a4e3cee	fix(opensearch-operator): let a permanently yellow cluster restart (#4317 ) ## What this PR does A single-replica OpenSearch release stops rolling once any index on it asks for a replica. The copy has nowhere to go on one data node, so the cluster stays yellow for good. The operator's restart gate at v2.8.0 opens on green, or on yellow only when `.opensearch-observability` is the yellow index, and at OpenSearch 2.11.1 that index is `auto_expand_replicas: 0-2`, green on one node. So the gate never opens. Every release gets such an index by itself. The operator sets `plugins.security.audit.type: internal_opensearch`, and the first audited event creates a daily `security-auditlog-` index with one replica. After that nothing that needs a restart is rolled out, a version upgrade or the move of the HTTP listener onto the cert-manager chain included. This is what timed out the `opensearch` e2e suite on some runs. Both failed runs I read show the same thing in the OpenSearch log: the bootstrap node leaves, `Connection reset` on the transport layer, and `[security-auditlog-<date>] creating index ... shards [1]/[1]` within 40 ms. The security plugin audits that exception (`SecuritySSLNettyTransport` hands it to `AuditLogSslExceptionHandler`), so the reset is what creates the index. Then the operator logs `waiting for health to be green` every ten seconds until the assert gives up. Not sure why the green runs had no audit index before the roll, a green run keeps no snapshot. The fix carries opensearch-project/opensearch-k8s-operator#1569 back to v2.8.0 as a second patch on the in-tree operator image, with the upstream tests. The new gate lets a yellow cluster through once allocation is enabled, every data node has joined and no shard is moving, and `PreparePodForDelete` refuses to restart a node that holds the only active copy of a shard meant to have others. A lone data node is exempt. I left out the third commit of that PR (tiered data roles in `HasDataRole`), the chart renders only the `data` role. A bump would not help: the only operator build after v2.8.0 is `3.0.0-alpha` from January, and its gate holds this case the same way. The patch goes away on the first release that includes #1569. The suite now creates an index with an unassignable replica before it moves the listener, so the roll always runs against a yellow cluster. The catch block also reads cluster health and the shard table from the node, because the operator logs that health held a restart back and never what the health was. The image digest in `values.yaml` stays as it is. Closes #4259 ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff: an e2e suite, the operator image's Dockerfile header and patch directory, one comment in the operator package's `values.yaml`. No values key, schema, make target or documented workaround changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch-operator): a single-replica OpenSearch release no longer gets stuck ahead of every rolling restart once an index with a replica, such as the security audit log or any index created with default settings, leaves the cluster permanently yellow ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes* - Improved rolling restart handling for clusters that remain yellow, allowing restarts to proceed safely once shard activity has settled. - Prevented restart operations from becoming blocked by temporary shard-allocation settings. - Added safeguards to ensure data is not disrupted while replicas are relocating or initializing. - Diagnostics - Added automatic cluster health and shard diagnostics when restart-related test operations time out or encounter issues. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 13:43:47 +04:00
Aleksei Sviridkin	74c87247ed	fix(linstor): reconnect only the stalled peer in the satellite plunger (#4184 ) ## What `drbd_stall_act` in `packages/system/linstor/hack/plunger/plunger-satellite.sh` disconnects a single peer with `drbdadm disconnect res:peer` and then calls `drbdadm connect res`, which walks every connection of the resource. Each peer that is still connected makes `drbdsetup connect` fail with `Device has a net-config (use disconnect first)` (exit code 10), the helper only logs `WARN: action failed for res:peer`, and the peer that was just disconnected is never reconnected. ## Observed cozystack 1.4.6 sandbox, LINSTOR 1.33.2 / DRBD 9.3.2 on Talos 1.13.6, resync of a 20 GiB volume while the Primary keeps writing: ``` STALL detected: res=pvc-9d99d1b9-1efe-4340-bbb3-13421c6bc7f2 sync_peer=sb-cs0203 percent_in_sync=90.4 -> disconnect/connect pvc-9d99d1b9-1efe-4340-bbb3-13421c6bc7f2: Failure: (125) Device has a net-config (use disconnect first) Command 'drbdsetup connect pvc-9d99d1b9-1efe-4340-bbb3-13421c6bc7f2 5' terminated with exit code 10 WARN: action failed for pvc-9d99d1b9-1efe-4340-bbb3-13421c6bc7f2:sb-cs0203 ``` ## Fix Reconnect the same `res:peer` pair that was disconnected. Verified on the sandbox by patching the live `linstor-plunger` ConfigMap: the peer reconnects and the resync completes. ## Note Not part of this change: the stall detector compares `percent-in-sync`, which `drbdsetup status --json` reports with one decimal, so a slow resync (for example one throttled down to `c-min-rate` by application I/O) can look stalled for `STALL_ITERS` iterations and trigger a reconnect every couple of minutes. Happy to follow up on that separately if you think it is worth it. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes - Improved reconnection behavior by targeting the explicitly selected resource-peer endpoint instead of reconnecting only at the resource level. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 13:38:19 +04:00
Yan Bondarenko	6e90447091	fix(kubeovn): retry migration setup until target pod exists Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 13:24:49 +04:00
Yan Bondarenko	55a2918fcb	fix(kubeovn): retry VM migration setup after scheduling Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 13:24:49 +04:00
Yan Bondarenko	dcad74d8c0	test(linstor): pin the chart command form of the shutdown hook Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 13:24:40 +04:00
Yan Bondarenko	6be4acad28	fix(linstor): retain shutdown command failure diagnostics Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 13:24:40 +04:00
Yan Bondarenko	ebacd2ea72	fix(linstor): add opt-in graceful satellite shutdown on Talos Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 13:24:40 +04:00
Yan Bondarenko	f7f4eb6656	fix(kubevirt): reject incomplete migrations template blocks Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 13:24:36 +04:00
Yan Bondarenko	5ed076cff8	feat(kubevirt): expose migration configuration through platform values Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 13:24:36 +04:00
Yan Bondarenko	334f6adac6	fix(nats): merge config.merge.accounts with the generated accounts map With users set the chart emitted accounts.A.users from the generated passwords and then appended .Values.config.merge verbatim. A tenant that also put an accounts block into config.merge, which is the only place to enable JetStream for the generated account, got the key twice in the rendered values and the release failed at post-render. The generated users are now merged into config.merge before it is rendered: the generated list wins for account A, everything else the tenant put under config.merge is kept, and a config.merge without users passes through unchanged. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 13:24:31 +04:00
Aleksei Sviridkin	d9fbeab33c	fix(opensearch-operator): let a permanently yellow cluster restart The operator restarts a node only when its health gate opens, and the gate wanted green, or yellow caused by the observability index alone. A cluster whose replicas can never be assigned reaches neither: on a single data node any index with a replica is yellow for good, and the operator itself turns on the audit log that creates one on the first audited event. From then on nothing that needs a restart was ever rolled out, a version upgrade and the move of the HTTP listener onto the cert-manager chain included. Upstream rewrote the gate to let a yellow cluster through once allocation is enabled, every data node has joined and no shard is moving, and to refuse instead to restart a node that holds the only active copy of a shard meant to have others. No released operator carries that change, so it is carried here as a patch against the pinned version, together with the upstream tests for it. Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-by: LLM	2026-09-18 12:01:56 +03:00
Aleksei Sviridkin	d3e228fed8	test(opensearch): roll a single-node cluster held yellow by a replica The roll onto the new mount plan timed out on some runs and passed on others with identical content. What differed was cluster health at the moment the listener moved: on one data node an index that asks for a replica stays yellow for good, and the operator restarts a pod only once its health gate opens. The security audit log creates such an index on its first audited event, and whether that had happened yet was down to timing. Create the index in the suite so the roll always has to get through a yellow cluster, and read the health back so the step cannot keep passing while it proves nothing. The operator logs that health held a restart back, never what the health was, and cluster health is not a Kubernetes object. The catch block now reads it from the node, together with the shard table and the allocation explanation. Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-by: LLM	2026-09-18 12:01:55 +03:00
Aleksei Sviridkin	5713b9774e	fix(foundationdb): let tenant roles read the connection ConfigMap (#4148 ) A tenant cannot connect to its own FoundationDB: the fdb-kubernetes-operator writes the cluster file into the `<release>-config` ConfigMap, but tenant subjects have no RBAC to read it and the dashboard resource map does not expose it, so there is no way to obtain the connection string without cluster-admin access. This PR: - adds `<release>-config` to the dashboard resource map; - adds a Role/RoleBinding granting get/list/watch on exactly that ConfigMap to the tenant `use` access level (via the existing cozy-lib helper); - adds helm unittest coverage (`make test`, 6 tests). Out of scope: FoundationDB currently exposes no Service, so clients still need in-cluster reachability to the coordinator pod IPs from the cluster file — that is a separate topic. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * New Features * Added the FoundationDB configuration resource to the dashboard resource map. * Added read-only access for authorized tenant users and service accounts to view the configuration resource. * Bug Fixes * Ensured configuration access and resource visibility remain available when monitoring is disabled. * Tests * Added coverage to verify resource mapping, access permissions, and rendered chart output. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 11:57:37 +04:00
Yan Bondarenko	9ee2a2375c	fix(foundationdb): let tenant roles read the connection ConfigMap The operator stores the cluster file in the <release>-config ConfigMap, but the dashboard resource map did not list it and no RBAC granted tenant subjects access to it, so a tenant could not retrieve the connection string for its own FoundationDB at all. Add the ConfigMap to the dashboard resource map and create a Role/RoleBinding scoped to exactly that ConfigMap (get/list/watch) for the tenant use-level subjects. Add helm unittest coverage. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 05:54:46 +04:00
Yan Bondarenko	8c649df7de	fix(linstor): reconnect only the stalled peer in the satellite plunger drbd_stall_act disconnects one peer with "drbdadm disconnect res:peer" but then runs "drbdadm connect res", which walks every connection of the resource. The peers that are still connected make drbdsetup fail with "Device has a net-config (use disconnect first)" (exit code 10), the helper only logs "WARN: action failed" and the stalled peer stays disconnected. Reconnect the same res:peer pair that was just disconnected. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>	2026-09-18 05:54:40 +04:00
Aleksei Sviridkin	bd3892253b	fix(clickhouse): keep the Keeper name inside the StatefulSet volume-name limit (#4133 ) ## What this PR does Fixes #4083. clickhouse-operator names the Keeper config volume `chk-<chk>-deploy-confd-<cluster>-<shard>-<replica>`. With the CHK called `<release>-keeper` and `Release.Name` already prefixed with `clickhouse-`, an application name longer than 15 characters pushes that volume name past 63 characters. The StatefulSet is rejected on every reconcile, no Keeper pod appears, and the CHK still reports `Completed`; the only visible signal is the Keeper `WorkloadMonitor` staying at `operational: false`. `templates/_keeper.tpl` now computes the CHK name: `<release>-keeper` as before when it fits, otherwise the release name is truncated and a six-character hash of the release name is appended so two long names cannot collide. The budget accounts for the width of the replica index. `chkeeper.yaml` and the zookeeper `nodes` list in `clickhouse.yaml` both use the helper, so the hosts ClickHouse connects to always match the CHK the operator creates. The pod label, the `VMPodScrape` selector and the `WorkloadMonitor` stay on `<release>-keeper`. Names up to 15 characters render byte-for-byte as before. A release with a longer name never had a Keeper pod; on upgrade its pod-less CHK is replaced by the shortened one and the Keeper comes up. `helm unittest`: 30/30 in the chart, including the new `tests/keeper_name_test.yaml` (short name unchanged, the exact 63-character boundary, one character past it, a very long name, a two-digit replica index, no trailing dash after truncation). ### Downstream repositories Templates and tests only: no values, schema, README, defaults, `ApplicationDefinition` or tenant-facing Secret/Service names change. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(clickhouse): shorten the Keeper name for application names longer than 15 characters so the Keeper StatefulSet can be created ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes - Fixed ClickHouse Keeper resource naming for long release names so generated names remain within Kubernetes’ 63-character limit. - Updated ClickHouse Keeper hostnames to consistently match the generated resource name, including when names are shortened. - Preserved identifying labels and monitoring selectors while applying the corrected Keeper naming behavior. - Tests - Added coverage for standard, boundary-length, and truncated Keeper names, including replica-specific naming scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-18 03:48:56 +04:00
Aleksei Sviridkin	355630a5e7	fix(kubernetes): pin KubeVirt CSI sidecars to SIG Storage releases (#3934 ) ## What this PR does Replaces five mutable OpenShift `latest` image references (four unique sidecar images) in the KubeVirt CSI controller and tenant-node DaemonSet with versioned Kubernetes SIG Storage releases. This addresses the affected references from #3578 without claiming the other references in that issue. The selected controller versions match the sidecars already used by Cozystack's LINSTOR CSI deployment. Render tests pin every changed container name and image so a mutable tag cannot return silently. Validation: - `helm unittest packages/apps/kubernetes`: 247/247 passed. - `helm unittest packages/system/kubevirt-csi-node`: 6/6 passed. - Live MSK-prod guest-cluster canaries on 2026-08-20 and 2026-08-21 reproduced the failure with the stock chart: the `latest` sidecars exited on the old CPU with the x86-64-v3 glibc error and the controller stayed in CrashLoopBackOff. A disposable guest cluster with the pinned images reached a 7/7 Ready controller on 2026-08-19 and was removed after the check. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the four changed template/test files. No package, schema, API, default, CRD, tooling contract, or downstream hardcoded value changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): pin KubeVirt CSI sidecars to versioned SIG Storage images that run on pre-x86-64-v3 CPUs ```	2026-09-18 03:48:47 +04:00
Yan Bondarenko	f886f26ad4	fix(apps): bound the wait the cleanup hooks now perform With watch granted, kubectl delete waits for the claims to be gone, and its default for that wait is a week. A claim held by a pod that will not stop, or a stuck detach, would hold the post-delete hook until helm-controller's 300s uninstall budget ran out and leave the HelmRelease stuck uninstalling. The PVC deletes in the clickhouse, harbor, mariadb and qdrant hooks now wait 120s at most and report the claims still terminating by name, and each Job carries activeDeadlineSeconds 270 with a 10s termination grace period, the numbers packages/apps/tenant/templates/cleanup-job.yaml documents against that budget, so a kubectl that never returns cannot hold the hook either. The mariadb Role no longer grants watch on the named Secrets: they carry no finalizer, so the delete never reaches the watch, and a resourceNames rule does not authorize a collection watch anyway. Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Assisted-by: LLM	2026-09-17 21:49:51 +03:00
Andrey Kolkov	c9aa64cbda	fix(dashboard): let dashboard users read backup CRD schemas (#4236 ) ## What this PR does The dashboard's backup build/edit forms (`BackupPlanCreatePage`, `BackupJobCreatePage`, `BackupRestoreJobCreatePage`, `BackupCreatePage`, `BackupClassCreatePage`, `BackupClassEditPage`, `BackupResourceEditPage`) build their form from the CRD's OpenAPI schema, which `useCRDSchema` fetches directly from `apiextensions.k8s.io` with the logged-in user's token — a cluster-scoped `get` on the CRD by name. No role a tenant user carries granted `customresourcedefinitions`, so every tenant user — tenant super-admin included — hit a 403 on the schema lookup and the form never rendered, even though they already hold write access to the backup CRs through `cozy:backups:admin`. `kubectl` kept working because that only touches the CRs, not their CRDs. The tenant personas (`cozy:tenant:{view,use,admin,super-admin}`) are bound only by RoleBindings, which authorize namespaced requests but never a cluster-scoped one, so the grant belongs on `cozystack-dashboard-readonly` in `packages/system/dashboard/templates/rbac.yaml`: it is bound to `system:authenticated` by a ClusterRoleBinding and already carries the dashboard's other cluster-scoped reads (`storageclasses`, `applicationdefinitions`, `backupclasses`). This adds a `get` on the five `backups.cozystack.io` CRDs, scoped by `resourceNames` to those schemas only — exactly the by-name `get` `useCRDSchema` performs, no wildcard and no collection listing. Verification: `helm template` renders `cozystack-dashboard-readonly` with the new rule cleanly; the real contract is `kubectl auth can-i get customresourcedefinitions.apiextensions.k8s.io/plans.backups.cozystack.io --as-group=<tenant>-super-admin`, which belongs in e2e. Fixes: #4053 ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(dashboard): grant dashboard users read access to the backup CRD schemas so the backup create/edit forms stop returning 403 ```	2026-09-17 18:29:07 +04:00
Andrei Kvapil	b0f50000c7	feat(dashboard): move Repositories from the Marketplace to Admin (#4286 ) ## What this PR does Connecting an External-Apps repository creates a cluster-scoped Tap and runs its charts in the management cluster. That is an operator action, so the Repositories view moves from the Marketplace sidebar to Admin, under Administration. The page is at `/admin/taps` now. `/marketplace/taps` redirects there, otherwise the `:appName` catch-all would render an order page for an app called "taps". Who can open it doesn't change. The entry was ungated in the Marketplace and stays ungated in Administration, Tap RBAC decides the rest. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/700 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: `content/en/docs/next/marketplace/connecting.md` on the website told readers to open the Repositories view "from the sidebar", inside the Marketplace docs. The follow-up points them to Admin and waits for this PR. ### Release note ```release-note feat(dashboard): the Repositories view moved from the Marketplace sidebar to Admin, under Administration. The old /marketplace/taps URL redirects to /admin/taps. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - New Features - Added Repositories to the Administration area at `/admin/taps`. - Added Repositories to the Administration sidebar with a dedicated navigation entry. - Repositories remain accessible without requiring access to the operator area. - Bug Fixes - Legacy Marketplace Repositories links now redirect to `/admin/taps`. - Removed Repositories from the Marketplace sidebar to reflect its new location. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-17 16:11:08 +02:00
Timur Tukaev	1825a2d4b9	docs(gtr): finalize editorial markers and pin submission version (#4221 ) ## What this PR does Finalizes the editorial layer of the General Technical Review ahead of filing the point-in-time snapshot for Incubation Due Diligence (cncf/toc#1916, tracked in cozystack/community#75). Resolves 13 `[maintainers: …]` markers whose answers are settled by repository or documentation evidence, corrects two inaccurate statements, pins the submission version, and assigns the previously-vacant Security Champion role. The DRAFT banner is rewritten so the remaining markers are grouped by why they stay open (measurement / maintainer confirmation / governance / organization action) instead of being mixed with editorial to-dos. Resolved with verified content: end-user case studies (Protofire, Cozyhr, CNCF announcement); compliance = no formal attestation (per self-assessment); API versioning = no stability policy yet, group is `v1alpha1`, tracked in #1251; deprecation communication; rollback = no runbook published; FinOps = not implemented, roadmap; known-failure-modes troubleshooting links; SBOM off by default (#2937); component retirement = GOVERNANCE.md sourcing policy, no formal retirement workflow; SLOs = adopter-defined with exposed SLIs enumerated. Corrections: - Loosening tenant isolation — removed the "shared/flat networking" claim; inter-tenant isolation is enforced by Cilium egress by design. Documented the opt-in policy labels that actually exist (per the tenants guide). - Alpha/beta flagging — removed "marked as such"; there is no per-application maturity field in `ApplicationDefinition` today. Project Version pinned to v1.6.3 (latest stable, 2026-09-04); Date Updated 2026-09-11. Also assigns the previously-vacant Security Champion role to @tym83 (weekly triage pass), which the GTR flagged as the genuine unmet target in the SCA section; surfacing the rotation cadence into `SECURITY.md` is the remaining governance step (CODEOWNER review). What remains open is genuine DD work, unchanged otherwise: measurements (load/scale, baseline overhead, upgrade→downgrade CI matrix), maintainer confirmations (Talos reboot, control-plane restart, CRD cleanup, circuit breaker), the `SECURITY.md` rotation/diversity surfacing, and organization actions (publish the CVE-pipeline policy docs; close the SCA write-back loop). ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note docs(gtr): finalize editorial GTR answers, correct tenant-isolation and alpha/beta wording, pin submission version to v1.6.3, assign Security Champion ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * Documentation * Updated the technical review for Cozystack 1.6.3 and its revision date. * Expanded deployment guidance for Talos, generic Linux/k3s, bring-your-own Kubernetes, and provider-specific installations. * Clarified API versioning, security, rollback, deprecation, scalability, cost management, troubleshooting, compliance, and maturity information. * Updated governance roles and voting information, including removal of the Directors role. * Documented security reporting responsibilities, scanner integration status, component updates, and recent nightly end-to-end test results. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-17 16:10:46 +03:00
Timur Tukaev	eca7401226	docs(gtr): add the General Technical Review for CNCF Incubation Adds GENERAL_TECHNICAL_REVIEW.md answering the CNCF GTR template across Day 0/1/2: deployment model (OS-agnostic, not Talos-only; recommends >=3 servers), architecture, testing and CI health, governance and the contributor ladder, security response and the Security Champion rotation, and compliance mapping pages (guidance for adopters, explicitly not attestations). Repository links target main; doc links are pinned to the versioned /docs/v1.6/ tree. Signed-off-by: Timur Tukaev <6355522@gmail.com>	2026-09-17 17:21:05 +05:00
Andrey Kolkov	3a8a06b44f	fix(dashboard): let dashboard users read backup CRD schemas The dashboard backup build/edit forms build each form from the CRD's OpenAPI schema, which useCRDSchema fetches directly from apiextensions.k8s.io with the logged-in user's token — a cluster-scoped get on the CRD by name. No role a tenant user carries granted customresourcedefinitions, so every tenant user hit a 403 on the schema lookup and the form never rendered, while kubectl kept working through the existing backup CR grants. The tenant personas are bound only by RoleBindings, which cannot authorize a cluster-scoped request, so the grant belongs on cozystack-dashboard-readonly: it is bound to system:authenticated by a ClusterRoleBinding and already carries the dashboard's other cluster-scoped reads. Add a get on the five backups.cozystack.io CRDs, scoped by resourceNames to those schemas only. Fixes: cozystack/cozystack#4053 Assisted-By: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com>	2026-09-17 15:48:14 +04:00
Andrei Kvapil	5692297f94	feat(dashboard): move Repositories from the Marketplace to Admin Connecting an External-Apps repository creates a cluster-scoped Tap and runs its charts in the management cluster. That is an operator action, not something a tenant does while browsing the catalog, so the entry belongs next to the other cluster-wide areas in the Admin portal rather than in the Marketplace sidebar. The page moves to /admin/taps under Administration. /marketplace/taps redirects there, since the :appName catch-all would otherwise render an order page for an application called "taps". Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com>	2026-09-17 14:24:04 +03:00
IvanHunters	8a6fe02ee4	fix(bucket): stop the ACME cleanup hook from talking to localhost, and gate the class (#4289 ) ## What this PR does Fixes #4288, and closes the class behind it and #4276. `packages/apps/bucket/templates/hooks/cleanup-acme.yaml` passed `--request-timeout=30s` on all four of its kubectl calls. A non-zero value there makes kubectl skip its in-cluster credentials: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so every call went to kubectl's built-in default cluster and was refused. This package failed more quietly than kafka. Each call is guarded by `\|\| echo "WARNING: ..."` and the script sets no `-e`, so the hook exited 0 and printed `ACME cleanup complete.`, and `hook-succeeded` then deleted the Job holding those warnings. The ACME solver objects it exists to reap were left behind, with the only record removed moments after it was written. Its own suite matched the rendered command including the flag, so the broken form was pinned in place. The gate is the point of the change rather than a bonus. Two packages carried this, neither was visible in a rendered manifest, and both passed their unit suites. Nothing stops the next hook from reaching for the same flag, because it reads as ordinary defensive practice. `hack/check-hook-request-timeout.bats` fails any non-vendored template whose kubectl call carries a non-zero `--request-timeout`. The coarse part is stated in the file rather than left to be discovered: a template that names `KUBECONFIG` is skipped, because an explicit kubeconfig is not served by the in-cluster branch and the flag is harmless there. That is a per-file judgement, not per-command. It is what lets `apps/kubernetes` keep the flag on two calls that pass their own kubeconfig and stay green, and it is the first thing to tighten if a file ever mixes both styles. Checked in both directions rather than only on the clean tree: the gate passes as committed, fails if the flag is put back on any bucket call, fails if it is put back on the kafka call, and leaves the two `apps/kubernetes` files alone. It also carries a vacuity guard, because a collapsed glob would otherwise report success for having examined nothing. Bounding the work is still available where it belongs: `--timeout` for a watch, `activeDeadlineSeconds` for the Job, which this hook already had at 120s. Depends on #4280 and is branched off it, so the gate is green here. #4280 removes the same flag from the kafka hook; merged before it, this check would be red on `main`. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff. A hook's kubectl flags and a new `hack/.bats` check; no values key, schema, API, default or naming contract moves, and nothing under `hack/` is moved or renamed, nor does any make target change what it does. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(bucket): the ACME cleanup hook now reaches the apiserver instead of reporting success while doing nothing ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit Bug Fixes * Improved ACME cleanup reliability by preventing cleanup commands from targeting an unavailable local Kubernetes endpoint. * Improved Kafka uninstall cleanup reliability by ensuring the pre-delete hook retains access to required credentials throughout execution. * Preserved existing execution time limits and warning-based failure handling for cleanup operations. * Tests * Added coverage to prevent regressions in Kubernetes cleanup command configuration and hook resource lifecycle behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-17 14:21:15 +03:00
IvanHunters	b323d5d88c	fix(kafka): keep the pre-delete hook's credentials as long as its Job (#4280 ) ## What this PR does Fixes #4276. Since #3938 landed, no Kafka release finishes deleting: the pre-delete hook cannot reach the apiserver, exits non-zero, and a non-zero pre-delete hook returns from `Uninstall.Run` before a single resource is removed. It is the inverse of what the hook was written for. #3793 was one release that would not delete; this made every release refuse to. The cause is `--request-timeout=20s` on the delete call. Any non-zero value for that flag makes kubectl skip its in-cluster credentials altogether: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so the branch that would load the ServiceAccount token never runs. kubectl talks to its built-in default cluster, `localhost:8080`, is refused, and the script reads the refusal as a failed delete. Measured on the pinned image rather than argued from documentation. With the token and `ca.crt` mounted and `KUBERNETES_SERVICE_HOST` set, the same delete call reaches the cluster with no flag, goes to `localhost:8080` with `--request-timeout=20s`, goes to `localhost:8080` with `1m`, and reaches the cluster again with `0`. The tree agrees. None of the three hooks that work pass it: `apps/kubernetes` names it in a comment only, `apps/harbor` and `apps/mariadb` never mention it, and `apps/tenant/templates/cleanup-job.yaml` records that its DELETE calls "carry kubectl's default `--request-timeout` of 0". The watch is still bounded by `--timeout`, and the Job by `activeDeadlineSeconds`, which is that same shape. The second commit was a separate change found on the way, and review turned it around. It matched the credentials to the Job on `before-hook-creation` alone, and a ServiceAccount, Role or RoleBinding has nothing like `ttlSecondsAfterFinished` to fall back on: helm removes a hook resource only when a policy says so, and `before-hook-creation` fires on a next activation that after the last uninstall never comes. All three would have sat in the tenant namespace for the life of the cluster, still granting delete over every KafkaTopic in it. They now carry `before-hook-creation,hook-succeeded`, the shape `apps/clickhouse` and `apps/kubernetes-nodes` already use, while the Job keeps `before-hook-creation` alone because the script exits 0 on the degraded path too and the Job's log is the only record that a finalizer outlived the wait. Reaping the credentials cannot strand a running pod: helm applies `hook-succeeded` only once the hook is terminal, and `activeDeadlineSeconds: 270` ends the Job inside the 5m uninstall hook timeout. Neither of these caused #4276, and a cluster run confirmed as much. Both are pinned. Restoring the flag turns the suite red, with `0` as well as with `20s`; the assertion matches `--request-timeout=` rather than the bare name, because the script's own comment names the flag. Dropping the reaping policy from the ServiceAccount alone, or from the Role alone, turns it red too, so the three are three assertions rather than one. Worth stating plainly, because it is the reason this reached `main`: none of this is visible in a rendered manifest. The asymmetry and the flag both rendered cleanly, passed 37 helm-unittest cases and 5 bats cases through two rounds of review and per-branch mutation testing, and only a live uninstall showed either. The e2e lane never ran on #3938 while it was open, because its workflows sat at `action_required` on all five heads a fork PR had. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff. This changes hook annotations and test assertions inside `packages/apps/kafka`; no values key, schema, API, default or naming contract moves, and the app itself is neither added nor renamed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kafka): deleting a Kafka release no longer wedges on a pre-delete hook that cannot reach the apiserver ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - Bug Fixes - Improved Kafka app uninstall cleanup by removing an unnecessary request-timeout constraint while retaining existing overall timeout safeguards. - Updated cleanup resource policies to remove failed-hook retention and preserve resources through successful hook completion, supporting reliable repeated uninstall attempts. - Tests - Added coverage for cleanup command timeout behavior and resource deletion policies. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-17 14:20:49 +03:00
IvanHunters	3a43bcf391	docs(kafka): say why the hook's Job and its credentials are reaped differently The policies are right and nothing here changes them. What is missing is the reason, and this file has now been changed in both directions inside a week by people reasoning from the annotations alone, so the reason is the part worth keeping. The Job carries no hook-succeeded because the script exits 0 on the degraded path as well as the clean one, and its log is the only record that a finalizer outlived the wait; ttlSecondsAfterFinished reaps it on a timescale a human can act within. The ServiceAccount, Role and RoleBinding have no such clock, and before-hook-creation fires on a next activation that after the last uninstall never comes, so without a terminal policy they sit in the tenant namespace for the life of the cluster still granting delete over every KafkaTopic in it. Reaping them cannot strand a running pod, which is the objection the symmetrical shape was written against: helm applies hook-succeeded only once the hook is terminal, and activeDeadlineSeconds ends the Job inside the uninstall hook timeout. hook-failed would decide nothing, because a failing hook already reaps the earlier successful ones under hook-succeeded. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-09-17 13:21:31 +03:00
Timur Tukaev	94885c2e93	docs(governance): remove the unused Directors role (#4296 ) ## What this PR does Removes the Directors role from `GOVERNANCE.md`. The role (non-technical leaders, elected by the maintainers, one vote each in decisions) was defined but never used — no Directors were identified or elected. It was intended for non-technical leadership, but those people ended up maintaining parts of the project directly, so the role stayed empty. Removing it so the documented decision-making body matches actual leadership (maintainers, by consensus with a simple-majority fallback). It can be reintroduced if a real need arises. Addresses cozystack/community#79. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note docs(governance): remove the unused Directors role from GOVERNANCE.md ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * Documentation * Updated governance guidance to remove the Directors role and related leadership descriptions. * Clarified that voting is conducted by maintainers. * Updated consensus language to reflect the revised decision-making process. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-17 13:07:45 +03:00
coderabbitai[bot]	a295e29b8a	fix(kafka): clean up pre-delete hook ServiceAccount and RBAC on success	2026-09-17 09:58:12 +00:00
IvanHunters	3615958240	fix(bucket): stop the ACME cleanup hook from talking to localhost, and gate the class The hook's four kubectl calls carried --request-timeout=30s. A non-zero value there makes kubectl skip its in-cluster credentials: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so every call went to kubectl's built-in default cluster and was refused. This package failed more quietly than kafka did. Each call is guarded by `\|\| echo WARNING` and the script sets no -e, so the hook exited 0 and printed "ACME cleanup complete."; hook-succeeded then deleted the Job that held those warnings. The ACME solver objects it exists to reap were left behind, and the only record went with the Job. Its own suite matched the rendered command including the flag, so the broken form was pinned. The gate is the point of this change. Two packages had it, neither was visible in a rendered manifest, and nothing stops the next hook from reaching for the same flag, since it reads as ordinary defensive practice. hack/check-hook-request-timeout.bats fails any non-vendored template whose kubectl call carries a non-zero --request-timeout, skipping files that name KUBECONFIG because an explicit kubeconfig is not served by the in-cluster branch. That per-file judgement is the coarse part and is written down in the file rather than left to be inferred: apps/kubernetes keeps the flag on two calls that pass their own kubeconfig, and stays green. Bounding the work is still available where it belongs: --timeout for a watch, activeDeadlineSeconds for the Job, which is what this hook already had at 120s. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-09-17 09:32:07 +03:00
IvanHunters	e63b8595d3	fix(kafka): drop the request timeout that hid the apiserver from the hook A non-zero --request-timeout makes kubectl skip its in-cluster credentials altogether. client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so the branch that would load the ServiceAccount token never runs. kubectl falls back to its built-in default cluster, localhost:8080, is refused, and the script reads the refusal as a failed delete and exits 1. A non-zero pre-delete hook returns from Uninstall.Run before any resource is removed, so no Kafka release could finish deleting and helm-controller retried forever. Measured on the pinned image rather than argued from the docs: with the token and ca.crt mounted and KUBERNETES_SERVICE_HOST set, the same delete reaches the cluster without the flag, goes to localhost:8080 with --request-timeout=20s, goes to localhost:8080 with 1m, and reaches the cluster again with 0. The three in-tree hooks that work do not pass it. apps/kubernetes names it in a comment only, apps/harbor and apps/mariadb never mention it, and apps/tenant records that its DELETE calls "carry kubectl's default --request-timeout of 0". The watch stays bounded by --timeout, and the Job by activeDeadlineSeconds, which is that same shape. The suite now rejects the flag. It needs the "=" because the explanation in the script names the flag too, and matching the bare name would fail on the comment rather than on a use. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-09-17 09:31:49 +03:00
IvanHunters	64d5689434	fix(kafka): keep the pre-delete hook's credentials as long as its Job The Job carries no hook-succeeded, so it outlives the hook run and ttlSecondsAfterFinished reclaims it, which is what leaves a readable record when the finalizer outlives the wait. Its ServiceAccount, Role and RoleBinding still carried hook-succeeded and hook-failed, so Helm reaped them at the end of the run either way and left the Job behind them. A Job whose ServiceAccount is gone still starts a pod. kubelet mounts the projected volume without a token, client-go finds no in-cluster config, kubectl falls back to localhost:8080, and the script takes its catch-all branch and exits 1. A non-zero pre-delete hook returns from Uninstall.Run before any resource goes, so the release never finishes deleting and helm-controller retries it: the e2e run on the merge commit shows six such pods in five minutes, in two namespaces, every one of them dead the same way. The hook meant to prevent one stuck Kafka release was instead stopping all of them. Give all four objects the same before-hook-creation policy, so the credentials live exactly as long as the consumer and the next run replaces them. Both other in-tree hooks that reach the apiserver, apps/kubernetes and apps/harbor, are symmetric this way already. The suite now pins the policy on each of the three, because nothing compared them: the asymmetry rendered cleanly, passed 37 helm-unittest cases and 5 bats cases, and only a cluster showed it. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>	2026-09-17 09:31:49 +03:00
IvanHunters	6818795588	fix(seaweedfs): raise the bucket ceiling by sizing volumes smaller (#4251 ) ## What this PR does Raises how many buckets a SeaweedFS instance can serve, by lowering the master's `volumeSizeLimitMB` from 1000 to 400 and its two-copy growth count from 6 to 3. The size limit is exposed on the app chart, where an operator can reach it. Existing instances stay at 1000 unless they already name a limit, so for them the upgrade changes only the growth count. A platform migration does that where migrations run; on the `default` variant it is a manual step before upgrading. A unit test pins each constant separately, and an e2e case creates several buckets and requires each to take a write. Late in a full e2e run every S3 `PUT` starts timing out and a backup suite fails its access preflight. It reads like a credentials or COSI problem and is not one: the instance has no volume slots left, so the master cannot grow a volume for a newly created bucket. The disk is still almost entirely free, which is what makes it look like a flake. The COSI objects say nothing either way, the claim is ready and the access is granted. ### The ceiling A volume server started with `-max 0` takes a slot count derived from disk rather than a fixed one, replication `001` spends a slot on each of the two servers per logical volume, and a collection claims `COPY_2` volumes each time it grows. So the ceiling is set by the product of the size limit and `COPY_2`, and moving either number alone moves it. The mechanism is upstream in `MaybeAdjustVolumeMax` (`weed/storage/store.go`) and `findVolumeCount` (`weed/topology/volume_growth.go`); what matters for choosing the numbers is only that upshot. Do not divide the slot figures out of the nominal PVC size, because what the server counts is the space the filesystem makes available and not the volume's advertised size. At the upstream 1000 and 6 a 10Gi volume PVC gives nine slots per server and two collections, which the platform's own `cozy-backups` bucket and a single tenant bucket already use. That same PVC was seen to give nineteen slots at 500, which puts 400 at twenty-three to twenty-four by the same division, or eight collections. The span is the observation's own: the free space behind nineteen at 500 fits more than one filesystem geometry, and those divide differently at 400. Eight either way, because a collection that cannot get a whole batch of `COPY_2` keeps the volumes it did get and writes into them, so the count rounds up rather than down. Every figure here is for the shipped `replicationFactor: 2`. The master picks which of its four growth counts to apply from the replication the filer asks for, and the filer takes that from `replicationFactor`, so at 1 or 3 copies it reads `COPY_1` or `COPY_3` and the `COPY_2` this PR sets does not apply. Lowering the size limit still raises the ceiling there, since it is the divisor whatever the replication is. I measured this instead of only deriving it, on the real `weed` 4.31 binary: one master, two volume servers with `-max 0`, each on its own 1 GB filesystem image, replication `001`, then one `assign` per fresh collection until the master refused. Scaled 10:1 against the 10Gi PVC, so the limits scale with it. \| `volumeSizeLimitMB` \| `COPY_2` \| slots reported \| collections served \| \| --- \| --- \| --- \| --- \| \| 100 \| 6 \| 10 \| 2 \| \| 100 \| 3 \| 10 \| 4 \| \| 50 \| 3 \| 20 \| 7 \| \| 40 \| 3 \| 25 \| 9 \| \| 25 \| 6 \| 40 \| 7 \| The fourth row is the pair this PR ships. The refusal arrives as `No writable volumes and no free volumes left`, with `volumes == max` on both servers and the disk nearly empty. The reported `max` matched the space the filesystem made available over the limit in every run, which is the rule the PVC figures above are divided with. Those runs sit on a 1 GB image, so their counts are the rule's, not the PVC's: a 10Gi volume has its own filesystem geometry and divides to its own numbers. ### Why both constants move The last two rows are the argument: lowering the size limit and lowering the growth count reach the same ceiling. They do not cost the same. Open descriptors scale with stored bytes rather than with the PVC, so at 400 a 10 TiB server keeps roughly 52k of them against 21k upstream, where taking the whole reduction out of the size limit would have put it at 105k, well past a 65536 `nofile` limit where one is set. Two stay open per volume, a `.dat` and an `.idx`; the `.vif` beside them is written and closed. That cost lands hardest on large instances, which never had a ceiling problem in the first place: at any of these limits a multi-TiB PVC has slots to spare. Lowering the growth count costs no descriptors at all. It costs write spread, since a bucket then spreads its writes over three volumes rather than six, and that cost is flat, it does not grow with the instance. So the reduction is split across both. Neither number is a new bet: 3 is what upstream already uses for three-copy replication. One price is shared by both routes. A collection absorbs `volumeSizeLimitMB × COPY_2` before it needs another grow, so growth requests arrive five times as often as they did at 6000 MB. A collection also starts with half as many writable volumes at `replicationFactor: 2`, and a volume server takes the writes into each volume one at a time, so a heavily parallel writer spreads over fewer volumes until the collection grows again. That holds for instances kept at 1000 too. ### On upgrade On a variant that runs platform migrations, the upgrade does not change the limit on an existing instance unless its operator already set one. Platform migration 57 writes `master.volumeSizeLimitMB: 1000` into every existing SeaweedFS instance that has not chosen a limit, except one whose `master` is not an object (below), so 400 applies only to instances created after this release. The `default` variant runs no platform migrations. A cluster there that installed SeaweedFS by hand has to apply the same patch before upgrading, and the operations guide has the command. Both guide steps end with a `complete` line. `jq` prints it only after reading the whole list, and the pinning loop stops at the first failed patch before it, so a run without that line is incomplete, whatever else it printed. A chosen limit is a whole number from 1 to 30000 and is kept. Anything else counts as not chosen and becomes 1000: the previous chart ignored the key and ran 1000 in its place, and the new schema would refuse it. A kept value other than 1000 takes effect with the upgrade itself. A lower one can strand buckets and a higher one takes the room new buckets grow into, so the guide has a query that lists those instances before upgrading. An instance kept at 1000 still gains: at `replicationFactor: 2` a grow now claims three volumes instead of six, so it has room for about twice as many new buckets. Three shapes get their own line in the Job log, with what they hold. A non-object `spec.values` is pinned too. helm-controller v1.5.0 ignores values it cannot read as a map and renders the chart defaults, so that instance runs at 1000 today and would drop to 400. The merge patch replaces the unreadable value, and nothing that applies is lost. A non-object `master` is left alone. Every v1 schema already refuses it, so that release does not render today, and the log says to set the limit in the same edit that repairs it. A `valuesFrom` with entries besides the `cozystack-values` Secret is named and pinned anyway. helm-controller merges `spec.values` over those entries, so a limit set there does not apply, unless it is set through `targetPath`, which can still win depending on the entries before it. The guide steps print the same three. The value goes into the application's HelmRelease, whose `spec.values` is the application spec, so it covers an instance created through the API and one the tenant module renders alike, the platform's own in `tenant-root` included. The tenant module renders that release without `spec.values`, which a new tenant chart test pins, and a Helm upgrade leaves alone a key that neither the previous nor the new manifest carries: I measured that on a kind cluster with the real HelmRelease CRD and Helm 4.3.0, on client-side, server-side and forced server-side apply, and read `createPatch` at the v4.1.1 that helm-controller v1.5.0 runs. Where migrations run, the order comes from the platform release itself. The migration is its pre-upgrade hook, and the `cozystack-packages` source every application chart is built from is a manifest of that same release, applied only after the hooks finish, so no instance meets the new chart unstamped and a failed migration stops the platform upgrade before that source moves. That holds while the platform source is pinned by digest, as it is out of the box; a source that follows a tag, a semver range or a branch lets source-controller move `cozystack-packages` on its own schedule, and an instance there can run at 400 until the migration has run. Until `cozystack-packages` moves, each patched release still runs its old chart, which ignores the key: no v1 release reads it, and none closes the `master` object in its values schema. The migration's test runs the script under the migrations image's shell against a fake `kubectl` that refuses any selector but the one cozystack-api lists SeaweedFS with, because an empty fleet is a legitimate answer and a mistyped selector would otherwise pass while pinning nothing. The same test runs the two guide steps, found by a marker comment above each, against the same fake and checks that they select the same releases as the migration. Lowering an existing instance is left to its operator, because it is not free. Its volumes between the new limit and the old go oversized on the next heartbeat and leave the master's writable set, and each affected collection needs one successful grow before its next write lands. Nothing is lost and the volumes stay readable. Those grows compete for one pool, and the budget is per collection rather than per server. An oversized volume is charged no headroom while one below the limit is charged all of its unfilled room, so the slots open to a stranded collection are free space less that room, divided by the limit; a grow then claims a whole `COPY_2` batch and takes as much of it as is there, and the volumes it creates are empty and charge their headroom straight back. Measured on the rig: two oversized collections with two free slots between them, and the first to write took both, leaving the second refused afterwards with the disk still reporting a quarter of itself free. Collections that keep writing are not free either: they were grown six at a time, and each of their volumes below 400 MB can hold back a whole 400 MB. Measured at a scaled pair, a server with free space for one batch per collection refused the only grow it needed, because a second, nearly empty collection held the room, and the same free space without that collection let the grow through. The master also grows, on its own schedule and with no write, any collection with no writable volume at or below 90 percent of the limit, which at 400 is 360 MB; measured at the same scaled pair, a collection with every volume in that band got two new volumes per server within one pass while a roomy one was left alone. So the precondition on every volume server is a batch for each collection with no volume at or below that line plus 400 MB for each volume still below the limit, and no volume size exempts an instance from it, since a larger volume carries more buckets and the budget scales with them. The `minFreeSpacePercent` reserve only comes into it on a server whose writable volumes have all passed the limit, since nothing there is charged headroom. It is five percent of what the filesystem reports rather than of the PVC's nominal size, so `All >= 8000 MiB` is where it reaches one 400 MB limit; a 10Gi PVC measures 9979 MiB and clears that, and such a server then refuses a grow only once its disk is down to the reserve. Everywhere else the headroom charge decides, and a refusal can come with the disk far from full whatever the reserve. Deleting a collection gives back a slot for each of its volumes, however little it stored, since the headroom charged to them leaves with them; only a server already at its reserve, where every volume is read-only and charged nothing, gets back no more than the bytes. Deleted objects return room only through a vacuum. The room left inside those volumes for further writes is given up and does not return as capacity, since such a volume is charged no headroom while its own bytes still come out of the budget. Lowering the growth count takes nothing away by contrast: no path removes a volume from the writable set for being surplus to the batch size, so an existing collection keeps what it has and converges as those volumes fill. [`docs/operations/seaweedfs-lowering-the-volume-size-limit.md`](https://github.com/cozystack/cozystack/blob/main/docs/operations/seaweedfs-lowering-the-volume-size-limit.md) has the check for it, and the way back. The ceiling cannot come out lower at any supported `replicationFactor`, which is the part worth stating plainly. The divisor halves, and the space the slot count is derived from grows at the same time, because a volume past the new limit stops being charged headroom against it. `COPY_1` and `COPY_3` are untouched, so at 1 and 3 copies the limit alone still moves the ceiling upward. ### Reaching the value Neither constant was reachable by an operator. The data-plane chart is pulled by `ExternalArtifact`, and the component publishing it declares no `install` section, so the reconciler drops a `Package`-level values override for it before any merge happens. Editing the chart was the only route, on a value whose right setting depends on the PVC size the operator chose. `master.volumeSizeLimitMB` is on the app chart now, with a schema entry. Worth being precise about which file the two numbers end up living in, because exposing one of them moves it. The app chart's HelmRelease sends the size limit on every release, so `packages/extra/seaweedfs/values.yaml` holds the 400 installs run with, and `packages/system/seaweedfs/values.yaml` keeps upstream's 1000 as the fallback a render without it gets. That fallback is deliberate: a platform upgrade can re-render the data-plane release from its artifact before the parent has sent anything new, and a lower value there would then reach the master first and drop the limit ahead of an operator's own setting. Nothing sends the growth count, so the data-plane chart's value is the live one for that half, and it is the part that does land in that window, where a grow costs 3000 MB against the 6000 MB it cost before. The budget the pair produces is written out next to the growth count, with a pointer from the size limit to it, so the two do not drift apart. The field is bounded at both ends, because either end takes an instance down. Zero is not a smaller limit but an off switch: the volume server leaves its slot calculation untouched at zero, keeps reporting no capacity, and nothing is ever grown. Past 30000 the master does not start at all, exiting on `volumeSizeLimitMB should be smaller than 30000`. The bound lives in the values schema, through the generator's `@minimum` and `@maximum` annotations, so Helm refuses the value before any template runs. The apps API reads that schema only for defaults, so it accepts a value out of range, and the release then fails to render. A case pins each edge along with the value just outside it. An explicit null gets past both bounds, because Helm drops a null key before it validates, and would reach the master as an empty `-volumeSizeLimitMB=` that the shipped image exits on at start, so the template requires the value and a case pins that as well. There is no second bound against the disk, and not for want of trying: a limit at or above the volume's own size does not break an instance. `MaybeAdjustVolumeMax` floors the slot count at one, and a 255 MiB disk against a 1000 MB limit reported `Max 1`, took an assign for a fresh collection and served back what was written to it. Such a server is capped at a single slot rather than dead, so a render refusal would stop a working release on every reconcile. The field description says instead that a value close to or above `volume.size` leaves each volume server a single slot. The growth count is deliberately not exposed beside the limit. A field wired to `COPY_2` would do nothing at 1 or 3 copies, for the reason above, while still looking like a setting, and a knob that is silently inert for two of the three supported values is worse than no knob. Making it track the replication factor is a feature with its own matrix, and it would rest on a path where the master's own default replication is separately broken. The size limit has no such dependency, so it is the one that ships. ### Picking a value 400 is sized for the shipped `volume.size: 10Gi`, which is where the slot and collection figures above come from. Going up, it holds until the two descriptors per volume meet a 65536 `nofile` limit, roughly 12.5 TiB per volume server. Going down, an instance needs 1600 MB of filesystem per volume server to serve a second collection at all, since the first claims the opening three slots. ### What the tests cover The existing bucket case creates one bucket, which sits inside the ceiling at any sizing, so it could not tell a working ceiling from an exhausted one. The new e2e case creates three beside the platform's own backup bucket and requires each to accept an object, which is the only signal the exhaustion produces. Each `mc` call there carries its own timeout, since a hung PUT is exactly that signal and the step timeout alone would not end it. It was observed red at the old sizing in [run 34948279363](https://github.com/cozystack/cozystack/actions/runs/34948279363), which ran the case against the unchanged constants: the pre-existing single-bucket case passed in 18s beside it, and the new one failed on the write to the second bucket after the first succeeded. Three rather than two so the count clears the ceiling without depending on how many slots the unnamed default collection holds, which the suite cannot observe. The unit tests pin each constant on its own, because losing one is a silent capacity regression rather than something a render would catch. The app-chart test asserts an overridden size limit as well as the default, since asserting the default alone passes just as well against a template that ignores the value and writes the number as a literal. It also pins both ends of the accepted range and the first value outside each, the `-max 0` zero on all four render paths it is written to, and the 10Gi volume default the slot figures above are measured for, which nothing else guarded. The data-plane suite pins the five percent free-space reserve as well, since the 8000 MiB threshold above is computed from it and nothing else reads it. The capacity case also reads the master's slot accounting in its `catch`. A grow failure in the master log does not say whether the shipped limit is too coarse for the PVC or an earlier collection in the same run is still holding the budget, and the per-server `Max` against `Volumes` does. Closes #4239, closes #4245, closes #4248 ### Screenshots No UI change. ### Downstream repositories Walked against the diff. The new field lands on `packages/extra/seaweedfs`, and the provider models bucket, generic, qdrant, redis and tenant, with no SeaweedFS resource among them, so there is no schema or expand/flatten pair over there for it to drift from. No package is added, renamed or removed, so the website app lists are untouched. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(seaweedfs): raise how many buckets a new SeaweedFS instance can serve on the default 10Gi volume PVC from two to eight at `replicationFactor: 2`, by defaulting the master volume size limit to 400 MB instead of 1000 and lowering the two-copy growth count from 6 to 3, and expose the limit as `master.volumeSizeLimitMB` on the app chart. Action required on the `default` variant only: it runs no platform migrations, so before upgrading, set `master.volumeSizeLimitMB: 1000` on every SeaweedFS instance that has no valid value and whose `master` is an object, with the step in [`docs/operations/seaweedfs-lowering-the-volume-size-limit.md`](https://github.com/cozystack/cozystack/blob/main/docs/operations/seaweedfs-lowering-the-volume-size-limit.md). If this is skipped, every such instance drops to 400 on the upgrade, and a bucket whose volumes have all reached 400 MB takes no writes until it can grow. On the variants that run platform migrations, migration 57 does this: it sets `master.volumeSizeLimitMB: 1000` on every instance that has not chosen a value, except one whose `master` is not an object, which no v1 chart renders and which it names in its Job's log instead, so nothing changes for an instance unless it already names a different value, which the previous chart accepted and ignored and this one applies; the same guide lists such instances before the upgrade, and the migration names them in its Job's log. An existing value that is not a whole number from 1 to 30000 is replaced with 1000, which is what the previous chart applied in its place. Instances at 1000 still get roughly twice the room for new buckets from the lower growth count. The upgrade rolls every SeaweedFS master StatefulSet once, since the growth count is part of the master's environment, and an instance that names a value other than 1000 rolls a second time when that value arrives. The pin lives in the instance's spec, so a tool that replaces the whole resource from a manifest without `master.volumeSizeLimitMB` removes it and the instance takes 400; add the field to such manifests before upgrading. Lowering the limit on an instance that already holds data takes every volume at or above the new value out of the master's writable set, and raising it takes the room new buckets grow into, so the same guide describes the free-space check to run first and how to set it back. ```	2026-09-17 09:29:05 +03:00
IvanHunters	c983390b39	ci(e2e): surface overlay drift and remove both causes of it (#4261 ) ## What this PR does Six commits against one script: one reports a silent degradation, three remove causes of it, two repair the tests and the reasons recorded beside them. Why it matters. A file whose config differs from the current-main artifact beyond its image lines keeps its committed ref. That is the right safe answer, and it is not a neutral one: the component then runs its last RELEASE image in a lane whose job is to exercise the tree under review, and the only trace is one line among hundreds. Seven packages took that path on the run behind #4257. One of them, `backupstrategy-controller`, served an image from v1.6.0 (22 July) against a tree carrying four backup strategy drivers added after it, so five of that run's seven chainsaw failures came from there, and three unrelated pull requests each spent a review round establishing that the red lane was not theirs. After this PR that list is empty. Against the public artifact at its own revision, with the tree checked out at that same revision: ```text before drift=5 overlaid=29 core/platform, cozystack-api, dashboard, kubeovn-webhook, linstor-gui after drift=0 overlaid=34 ``` Naming the drifted packages. One GitHub annotation after the existing tally. Which files get overlaid does not change, and a run with no drift emits nothing. Cause one: a suffixed image key. The classifier accepted a key only when it was exactly `image`, `repository`, `registry`, `tag` or `digest`. A chart rendering several images names the extras after what they are for, so `backupstrategy-controller` carries `chBackupClientImage`, `rabbitmqBackupClientImage` and `redisBackupClientImage`. The first two passed only through the `@sha256:` branch; the third is pinned by tag alone, so it read as a config change and cost the whole file its overlay. The fix accepts a prefix before `image`, and two things keep it narrow. A letter is required before `image`, which a key beginning with it has nothing to supply, so `imagePullPolicy` and `imagePullSecrets` stay config; the trailing colon is not what does that, and they match neither with it nor without. The value must also hold a repository path, which keeps out operator-settable knobs named after an image: `vddkImage: ""` in `core/platform` and `migration-controller` is configuration the build never stamps, and absorbing it would be the opposite of what this script is for. Cause two: a blank line the build removes itself. `yq -i` re-emits the document when it stamps a digest, and a blank line separating two top-level keys does not come back. So a file the PR never touched differs from the artifact on a line carrying no configuration at all, and loses its overlay for it: ```text $ diff repo/system/cozystack-api/values.yaml artifact/system/cozystack-api/values.yaml 4d3 < 6c5 < image: ...cozystack-api:v1.6.0@sha256:c4d374a9... --- > image: ...cozystack-api:main@sha256:2fb87cf8... ``` Running the stamping assignment over the committed file reproduces that diff exactly, which is the second signal behind the diagnosis rather than the first one twice. Blank lines are now discounted before the test; a line with content still counts, so a genuine config difference keeps the committed ref as before. This accounted for all five remaining packages, and it retires the explanation offered in #4265, which ruled `yq` reformatting out. That was wrong: the round-trip check behind it did not cover the blank line. `cozystack-api` and `dashboard` are not peripheral here. An e2e result about either was a result about the last release rather than about `main`. Repairing the tests. `hack/bats-no-exit-trap.bats` audits every `.bats` file under `hack` against the debt its own header declares, and this file was installing three traps beyond it. An EXIT trap inside an `@test` body replaces the one the bats binary installs, so a failing test prints no TAP line at all and the suite reads as green. The answer is to stop installing the handler rather than to raise the number, so those tests clean up at the end of the body instead. ### Tests Eight cases in `hack/overlay-main-images_test.bats`, each held by a mutation of the line it describes: - drift is summarised with the package names and the count; suppressing the annotation or dropping the names reddens the suite - a clean run emits no annotation - a suffixed image key pinned by tag alone overlays instead of drifting; reverting the classifier reddens it - a key merely STARTING with `image` is still config - an `Image` key whose value is operator configuration rather than a path still drifts, so `vddkImage` is not absorbed - two drifted files in one package count as one package, and the name carries no `/images` suffix - a blank line lost to `yq` is not a config change; removing the filter reddens it - a real config line alongside a blank line still counts as drift; widening the filter to drop every line reddens this one and two others The suite's plan and its `ok` count agree at 21, which is the reconciliation the trap audit exists to protect. Green under the `bats` binary and under `hack/cozytest.sh`, which is what `make bats-unit-tests` runs and the only one of the two whose result decides the gate. ### What this does not close `redisBackupClientImage` is still pinned by tag with no digest, and the tag it names does not exist: ```text $ curl .../cozystack/redis-backup/tags/list {"tags":["nightly","0.0.0-nightly.20260911...","0.0.0-nightly.20260912...", ...]} ``` Only nightly tags are published, so `redis-backup:v1.6.0` resolves to nothing. Pinning it needs a published release tag first, which is #4264 rather than this PR. Making the overlay line-wise, so image lines transfer even when other lines genuinely differ, would go further still. It trades away the guarantee the script is built on, that a file comes wholly from one source, and that guarantee is deliberate and commented: a maintainer decision rather than a quiet change from a reviewer. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit New Features * GitHub Actions now warns when image overlay processing detects configuration drift. * Warnings list affected package names in sorted, de-duplicated form. * Image references with alphabetic prefixes are recognized during overlay processing. * Blank-line-only differences no longer trigger drift warnings. * Tests * Added coverage for image references, similarly named configuration keys, blank-line differences, and multi-package drift reporting. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-17 09:28:03 +03:00
Timur Tukaev	69c4fadf8a	docs(security): document Security Champion rotation and response team (#4223 ) ## What this PR does Surfaces the Security Champion rotation policy and the security response team's composition and diversity into the public `SECURITY.md`. Until now these were defined only in the CVE-scanning pipeline's private governance document, which the General Technical Review flagged as a gap (the policy existed but was unpublished, and the rotating role was unassigned). Specifically: - Adds a "Security response team and rotation" subsection: names the response-team maintainers, notes the team spans more than one organization (Ænix, HIDORA) and more than one country, and documents the rotating Security Champion role — its responsibilities, a 6-month rotation among maintainers, and a one-week handoff overlap. - Records @tym83 as the current Security Champion, filling the role this file previously described as unfilled. - Updates the Trivy paragraph accordingly (the triage-clock owner is no longer "currently unfilled"). This is a policy change to a `CODEOWNERS`-protected file and is intended for maintainer (CODEOWNER) review rather than a wording check. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note docs(security): document the Security Champion rotation policy and security response team in SECURITY.md ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * Documentation * Added information about the security response team and its rotating Security Champion role. * Clarified ownership of security triage timing and response coordination. <!-- end of auto-generated comment: release notes by coderabbit.ai -->	2026-09-17 06:07:59 +03:00

1 2 3 4 5 ...