mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 12:43:32 +09:00
1fb4bcf703b0a9613dfedeecf7ae421f4fa03579
1771
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
1fb4bcf703 |
feat(system): add talos-log-collector package
Add a node-local Vector DaemonSet that receives Talos machine and kernel logs (pushed via machine.logging over the node loopback) and forwards them to the tenant VictoriaLogs. Talos keeps its system and kernel logs inside the OS and does not expose them as host files, so the tail-based Fluent Bit in monitoring-agents never collects them. The receiver stays on the pod network so cluster DNS resolves the vlinsert service, and accepts Talos logs via a hostPort bound to hostIP 127.0.0.1, avoiding the host-network DNS breakage a hostNetwork receiver would hit. Shipped as an optional system package. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
b99df11b36 |
fix(kubernetes): bump tenant worker Talos to v1.13.6 (#3269)
## What this PR does Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24. That kernel predates the fixes for two related KVM x86 shadow-paging use-after-free bugs in `arch/x86/kvm/mmu/mmu.c`: CVE-2026-46113, fixed upstream in `0cb2af2ea66ad8ff195c156ea690f11216285bdf` and released in 6.18.30, and CVE-2026-53359, fixed in `81ccda30b4e83d8f5cc4fd50503c44e3a33abfeb` and released in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables nested virtualization inside the VM, and either bug lets a nested guest escape into the worker. This bumps the worker pin to Talos v1.13.6, which ships Linux 6.18.38. Stable kernel branches are cumulative, so v1.13.6 is the first v1.13 patch release carrying both fixes. The pinned v1.13.0 ships 6.18.24 and carries neither; v1.13.5 ships 6.18.36 and carries only CVE-2026-46113. Nested virtualization stays available to tenants; the kernel bump is the fix, not a capability removal. The image-factory schematic is unchanged and resolves for the new version: both the `openstack-amd64.raw.xz` artifact and the `installer/<schematicID>:v1.13.6` manifest are served by the public factory. Talos v1.13 supports Kubernetes 1.31 through 1.36, so the chart's Talos-to-Kubernetes support window is unaffected and no tenant `spec.version` needs to move. Applying this rolls each worker pool once. The new pin changes the rendered worker spec, which rotates the content hash in the `KubevirtMachineTemplate` and `talos-reconcile` Job names, and CAPI replaces the worker VMs to deliver the new kernel. The pinned hashes in the chart tests are recomputed to match. A second commit fixes the support-matrix lookup itself. The matrix dict was keyed on the full patch version, so every Talos patch bump dropped out of the dict and silently disabled the render-time guard — exactly the mismatch the guard exists to catch. The lookup now derives the minor from `talos.version`. The supported-Kubernetes window is unchanged; unknown minors are still skipped, leaving compatibility to whoever overrode the pin. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note fix(kubernetes): bump tenant worker Talos to v1.13.6, picking up the Linux 6.18.38 kernel fixes for CVE-2026-46113 and CVE-2026-53359 (KVM shadow-paging use-after-free, nested-guest escape into the worker VM). Nested virtualization remains available. Applying this upgrade rolls each tenant worker pool once, as the worker template is renamed to carry the new image. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Updated the default Talos worker release to v1.13.6 for Kubernetes cluster provisioning. - Improved compatibility checks to apply consistently across Talos patch releases within the same minor version. - **Bug Fixes** - Corrected generated resource naming expectations when storage settings or GPU configuration change. - **Tests** - Added coverage confirming Kubernetes clusters render successfully with Talos v1.13.6. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
5b9b420ffe | chore(deps): update container image & action digests | ||
|
|
681f4a8313 |
build(flux-shard-operator): read the shared registry build cache
flux-shard-operator was the last image built with the pre-#2937 cache pattern: --cache-from a :latest tag that PR builds no longer publish and --cache-to type=inline, which never caches multistage builder layers. Use the shared cache-args macro so it reads (and, on main, writes) the same mode=max :buildcache ref in CACHE_REGISTRY as every other image. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
b73860d4c9 |
fix(kubernetes): bump tenant worker Talos to v1.13.6
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24. That kernel predates both fixes for a pair of related KVM x86 shadow-paging use-after-free bugs: CVE-2026-46113, fixed in 6.18.30, and CVE-2026-53359, fixed in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables nested virtualization inside the VM, and either bug lets a nested guest escape into the worker. Talos v1.13.6 ships Linux 6.18.38. Stable kernel branches are cumulative, so it is the first v1.13 patch release carrying both fixes: the pinned v1.13.0 ships 6.18.24 and carries neither, while v1.13.5 ships 6.18.36 and carries only CVE-2026-46113. Nested virtualization stays available to tenants — the kernel bump is the fix. The new pin changes the rendered worker spec, so the content hash in the KubevirtMachineTemplate and talos-reconcile Job names rotates and each worker pool rolls once on upgrade. Pinned hashes in the chart tests are recomputed accordingly. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
f91841fce6 |
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3215)
## What this PR does Addresses #3181, where platform log ingestion could fail **silently**: fluent-bit shipped to `vlinsert-generic` while the monitoring `HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName` in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary layers make the failure loud instead of silent: 1. **Readiness gate** — `packages/extra/monitoring/templates/helmrelease.yaml` now sets a `poller` `waitStrategy` and a `healthCheckExprs` entry on the `VLCluster` `status.updateStatus`. A missing/unhealthy log store makes the `monitoring-system` release **not-ready** instead of flipping `Ready` the moment the CR is applied. Mirrors the existing seaweedfs CNPG pattern. 2. **No empty-log-store hole** — `packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render on an empty `logsStorages`. With no entry no `VLCluster` exists, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is now a loud misconfiguration. 3. **No silent fallback** — the `cozy-monitoring → tenant-root` `vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are created only when the root tenant hosts monitoring. The template reads the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth, so it cannot drift from the tenant's real state) and falls back to a new platform value `monitoring.rootEnabled` (via `_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first render). The default preserves the historical always-create behaviour. Covered by helm-unittest across all four affected charts (141 tests): readiness-gate fields, empty-vs-populated `logsStorages` contract, `ExternalName` present/absent fallback contract, and the platform wiring into the `cozystack-values` Secret. Also adds the missing `test:` target to `packages/extra/monitoring/Makefile` so its suite runs in CI. **Known limitation / follow-up:** the legacy platform monitoring release installed directly into `cozy-monitoring` via the `cozystack.monitoring` PackageSource cannot carry the readiness gate, because the engine's `ComponentInstall`/HelmRelease builder and the vendored `helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`. That source is not referenced by any active bundle; full coverage there needs a helm-controller bump plus engine support. ### Release note ```release-note fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a platform setting (`monitoring.rootEnabled`) to control whether root-level monitoring is enabled. * Monitoring redirect ExternalName services are now rendered only when root monitoring is enabled (defaulting to enabled). * **Bug Fixes** * Ensures an explicitly set `monitoring.rootEnabled=false` is preserved when generating platform configuration. * Prevents incomplete monitoring deployments by failing rendering when `logsStorages` is empty. * Improves VictoriaMetrics readiness by delaying “ready” until the operator reports an operational status. * **Tests** * Added/expanded Helm-unittest suites covering monitoring wiring, redirects, readiness gating, and `logsStorages` requirements. * Added a `make test` target to run chart unittests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
f60fe2270b |
Merge remote-tracking branch 'origin/main' into feat/kubernetes-talos-image-factory-override
Signed-off-by: Aleksei Sviridkin <f@lex.la> # Conflicts: # packages/system/kubernetes-rd/cozyrds/kubernetes.yaml |
||
|
|
6c69c0055e |
fix(monitoring)!: clean up orphaned storage PVCs and TLS secrets on tenant module delete (#3091) (#3094)
## What this PR does
When the `monitoring` tenant module is disabled, the
VictoriaMetrics/VictoriaLogs storage PVCs (created by vm-operator) and
the `alerta-tls` / `grafana-ingress-tls` cert-manager secrets were left
orphaned in the tenant namespace.
This adds a `post-delete` cleanup hook
(`packages/extra/monitoring/templates/hooks/cleanup.yaml`) that:
- deletes the VM/VL storage PVCs scoped to this release via `-l
apps.cozystack.io/application.name={{ .Release.Name }}-system`. That
label is stamped on the `storage.volumeClaimTemplate.metadata.labels` of
the VMCluster/VLCluster CRs (`packages/system/monitoring`); vm-operator
copies claim-template labels onto the PVCs (`IntoSTSVolume`), so the
selector matches and stays scoped to this monitoring release.
`managed-by=vm-operator` is intentionally **not** used — vm-operator
does not reliably stamp it onto the storage PVCs;
- deletes the `alerta-tls` and `grafana-ingress-tls` secrets by name
(cert-manager TLS secrets carry only the cluster-wide
`controller.cert-manager.io/fao=true` label, so a label sweep would be
unsafe — they are removed by name).
CNPG PostgreSQL PVCs (`alerta-db-*`, `grafana-db-*`) are intentionally
**not** targeted — CNPG cleans those up itself, and they do not carry
`apps.cozystack.io/application.name`, so the selector leaves them
untouched.
**Upgrade path:** the claim-template label only lands on PVCs created
after the change, so **migration 50**
(`packages/core/platform/images/migrations/migrations/50`) backfills
`apps.cozystack.io/application.name` onto the pre-existing VM/VL storage
PVCs of every cozystack-managed monitoring cluster, so the cleanup
reclaims storage on already-deployed clusters too.
A `test` Makefile target and helm-unittest suites are added for both the
cleanup hook and the claim-template label. The cleanup Job is hardened:
non-root, read-only root fs, dropped capabilities, `seccompProfile:
RuntimeDefault`, resource requests/limits, `activeDeadlineSeconds: 120`,
`backoffLimit: 0`, `--wait=false` on both deletes; RBAC scoped to PVC
get/list/delete and the two named secrets. Failures are logged but never
block teardown.
> [!WARNING]
> **Disabling the monitoring module now permanently deletes its
VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they
hold.** This is intentional and fixes the storage leak. Back up before
disabling if the data matters.
Fixes #3091.
### Release note
```release-note
fix(monitoring): garbage-collect the orphaned VictoriaMetrics/VictoriaLogs storage PVCs and the alerta/grafana TLS secrets left behind when a tenant's monitoring module is disabled; a migration backfills the release label onto pre-existing storage PVCs so already-deployed clusters are covered too. WARNING: disabling monitoring now permanently removes its storage PVCs and the data they hold.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Helm **post-delete** monitoring cleanup job to best-effort
remove managed PVCs and two specific TLS secrets.
* Included dedicated cleanup RBAC and hardened pod security settings for
the cleanup job.
* **Tests**
* Added Helm chart tests validating hook weight/policies, job settings,
exact deletion commands, and negative cases.
* Added a `make test` target to run Helm unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
88c0cc36ad |
feat(keycloak): add Vault Kubernetes and AppRole auth for the encryption proxy (#3239)
## What this PR does Makes the optional `keycloak-kms-proxy` PII-encryption feature production-usable. **Vault auth** — new `encryption.kms.vault.auth` switch: - `kubernetes` — **recommended**. The proxy logs in with its own dedicated ServiceAccount (`keycloak-kms-proxy`; bind the Vault role to `system:serviceaccount:<ns>:keycloak-kms-proxy`), so no credential is stored in a Secret. - `approle` — `KKP_VAULT_ROLE_ID` / `KKP_VAULT_SECRET_ID` from `encryption.kms.vault.appRole.secretName`. - `token` (default, unchanged) — Vault token from `tokenSecretName`. The proxy pod runs under a dedicated ServiceAccount (never the namespace default), with its token mounted only for Kubernetes auth. **Shared DEK set** — `encryption.deksetSecretName` → `KKP_DEKSET_FILE` (mounted read-only), and `encryption.replicas` is now configurable. Without a dekset the proxy mints an ephemeral DEK (only safe on a fresh, empty database, single replica). A shared wrapped DEK set — minted by the proxy's backfill tool — is required to migrate an existing populated Keycloak and to run HA; a guard rejects `replicas > 1` without one. Render-time validation requires each mode's settings and rejects unknown values. helm-unittest covers all auth modes, the ServiceAccount/token toggle, and the dekset/replicas paths. Uses `keycloak-kms-proxy` v0.2.0. ### Release note ```release-note feat(keycloak): Vault Kubernetes/AppRole auth, a dedicated ServiceAccount, and a shared DEK set (migration + HA) for the optional PII-encryption proxy ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Expanded Keycloak KMS proxy Vault authentication to support **token**, **AppRole**, and **Kubernetes** modes. * Added **multi-replica/HA** support via configurable DEK-set reuse, including selecting a custom DEK-set key. * Introduced a dedicated proxy ServiceAccount with **conditional token automount** for Kubernetes auth. * **Bug Fixes** * Added stricter validation for missing/unsupported Vault auth settings. * Prevented Vault credential environment variables from leaking across auth modes. * Updated rendering behavior and test expectations for proxy resources (including KEK/ServiceAccount documents). * **Chores** * Updated the Keycloak KMS proxy container image version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
f700c9043c |
fix(monitoring): backfill release label onto pre-existing VM/VL storage PVCs
Re-review on #3094 (IvanHunters, MAJOR): the volumeClaimTemplate label lands on PVCs only at creation — the StatefulSet controller never re-labels existing PVCs and vm-operator's PVC reconcile is resize-only. So on an already-deployed cluster (exactly the #3091 population) the pre-upgrade VM/VL storage PVCs stay unlabeled and the cleanup hook's selector misses them. Add migration 49 (mirrors the ClickHouse-keeper backfill in migration 48 for the sibling #3072). It relabels the pre-existing vmstorage/vmselect/vlstorage PVCs of every cozystack-managed VMCluster/VLCluster with apps.cozystack.io/application.name read from the CR's own spec.managedMetadata, so only monitoring's clusters are touched (a user's own VMCluster carries no such label) and the value always matches what the cleanup hook selects. Best-effort like #48: a skipped relabel is no worse than the pre-existing leak, so transient apiserver errors never abort the platform upgrade. Bump platform targetVersion 49 -> 50. Also correct the misleading comments (IvanHunters MINOR): spec.managedMetadata is never applied to PVCs by vm-operator at any current version, so the claim-template label is the only mechanism — do not imply a v0.71.0 crossover. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
|
e01b725a48 |
fix(monitoring): stamp release-scoping label on VM/VL storage PVCs so cleanup matches
Re-review on #3094 (IvanHunters, lexfrei): the release-scoped selector added in ba5797f4 was a silent no-op on the shipped vm-operator v0.68.4. The apps.cozystack.io/application.name label lived only under the VMCluster/VLCluster spec.managedMetadata.labels, and v0.68.4 does not propagate managedMetadata onto PVCs (that lands only in v0.71.0). So the delete matched nothing and the #3091 storage leak stayed open. Stamp the label where it actually reaches the PVCs: on spec.{vmselect,vmstorage,vlstorage}.storage.volumeClaimTemplate.metadata.labels of the VMCluster/VLCluster CRs (packages/system/monitoring). vm-operator copies claim-template labels onto the PVCs (IntoSTSVolume) even on v0.68.4, so the existing cleanup selector (managed-by=vm-operator,apps.cozystack.io/application.name= <release>-system) now matches and stays scoped to this release. Verified by rendering the CRs (label resolves to monitoring-system, matching the hook). Also add the explicit subjects[].namespace on the cleanup RoleBinding (lexfrei non-blocking) and a helm-unittest suite asserting the label lands on all three storage claim templates. Corrected the misleading test comment about how the label reaches the PVCs. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
|
f9afc73096 |
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3181)
Platform log ingestion could fail silently: fluent-bit shipped to vlinsert-generic while the monitoring HelmRelease reported Ready, and a vlinsert-generic ExternalName in cozy-monitoring dead-ended at a tenant-root VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary changes: - Readiness (do not stay silently Ready): the monitoring-system HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on its VLCluster (status.updateStatus), so a missing/unhealthy log store makes the release not-ready instead of flipping Ready as soon as the CR is applied. Mirrors the seaweedfs CNPG pattern. - No empty-log-store hole: system/monitoring rejects an empty logsStorages at render time (helm fail). With no entry no VLCluster is created, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is made a loud misconfiguration rather than a documented footgun. - No silent fallback: the cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName redirects in cozystack-basics are now created only when the root tenant hosts monitoring, gated on _cluster.monitoring-enabled — fed from a new platform value monitoring.rootEnabled (default true, preserving the historical always-create behaviour). monitoring.rootEnabled is the single render-time source of truth: changing it re-renders cozystack-basics (helm-controller watches the cozystack-values Secret it flows into). A live `lookup` of Tenant/root spec.monitoring is deliberately NOT used: cozystack-basics is an operator-generated HelmRelease with drift detection off, so it re-renders only on install/upgrade/values-change — never on a reconcile tick or when a lookup result changes — and a lookup would go stale in both directions. Operators must keep monitoring.rootEnabled in sync with Tenant/root spec.monitoring. Covered by helm-unittest across all four charts: readiness gate fields, empty-vs-populated logsStorages contract, ExternalName present/absent and null-guard contract, and the platform wiring into the cozystack-values Secret. The CEL health predicates are exercised live, not in helm-unittest (which cannot evaluate CEL). Adds the missing `test:` target to packages/extra/monitoring/Makefile so its suite runs in CI. Known limitation (follow-up): the legacy cozystack.monitoring PackageSource that installs system/monitoring directly into cozy-monitoring cannot carry the readiness gate, because the engine's ComponentInstall/HelmRelease builder and the vendored helm-controller API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not referenced by any active bundle; full coverage needs a helm-controller bump plus engine support. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
158e85af85 |
chore(keycloak): bump encryption proxy image to 0.2.1
Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
0c7b598304 |
fix(apps/kubernetes): honour nodeGroup minReplicas in MachineDeployment.spec.replicas (#3233)
Refs #3231 (parts 1 + 2 addressed here; part 3 — intra-file `cozy_wait_linstor_pool_free` gate + orphan `default-chk-clickhouse-test-keeper-cluster1-0-2-0` PVC cleanup — deferred to a separate follow-up PR against `hack/e2e-apps/run-kubernetes.sh`). **Do not auto-close the issue on merge** — the hardening work is a distinct concern and belongs on its own tracker entry. ## Root cause chain (recap from the issue) `packages/apps/kubernetes/templates/cluster.yaml` hardcoded `MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's `minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}` bats set `minReplicas: 0` and delete their `Kubernetes` CR without waiting for readiness — but the chart still spun up 2 × 20 GiB `replicated` (DRBD) worker `DataVolume`s each. Their create/delete churn overlapped `kubernetes-previous`, which then saw slow LINSTOR provisioning (my cozyreport from run 28843794670 shows a ~6-minute gap between the `Kubernetes` CR being created and the CDI `ExternalProvisioning` event actually firing for the worker root disk); the tenant node registered late; Cilium's CNI init missed the 12-minute node-join deadline; and the test failed with `cni plugin not initialized`. ## Changes Two edits, applied together — the chart edit is a no-op without the test edit, and vice-versa (dropping `nodeGroups` falls back to the chart helper's default `md0`, which itself has `minReplicas: 0`, so both pieces are needed for zero workers to actually render). ### 1. `packages/apps/kubernetes/templates/cluster.yaml` ```diff spec: clusterName: {{ $.Release.Name }} - replicas: 2 ``` The field is **omitted entirely** so the cluster-autoscaler owns it alone. CAPI's MachineDeployment defaulting webhook seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create (documented at https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling — *"if it's a new MachineDeployment or MachineSet, use min size"*), and Server-Side-Apply field-manager ownership then keeps chart upgrades from overwriting whatever value the autoscaler patched in the meantime. Emitting the field from the chart at all — even with the semantically-correct `{{ $group.minReplicas }}` — would keep the helm-controller ↔ cluster-autoscaler field-ownership fight alive: helm-controller applies with SSA + force-conflicts (https://fluxcd.io/flux/components/helm/helmreleases/), so every routine `helm upgrade` re-asserts the chart's chosen value and drains whatever the autoscaler had patched in the meantime. ### 2. `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats` Drop the explicit `nodeGroups:` block and the paired `storageClass:` from the render-side CR bodies. The tests only assert on the KamajiControlPlane and the cozy-realm Keycloak objects — worker plumbing was never part of their coverage. After the chart fix, the chart helper's default `md0` (with `minReplicas: 0`) renders a MachineDeployment with `spec.replicas: 0` — the resource is still emitted for API-surface completeness but never creates a KubevirtMachine, never provisions a DataVolume, and never touches DRBD. ## Verification ### helm-unittest ``` Charts: 1 passed, 1 total Test Suites: 16 passed, 16 total Tests: 171 passed, 171 total ``` No suite asserted on `MachineDeployment.spec.replicas` directly, so nothing needed updating. ### helm template — verify the omit With `tests/values/common.yaml` (which has `minReplicas: 0`), the rendered MD carries the `cluster-api-autoscaler-node-group-min-size: "0"` annotation and **no `spec.replicas` field**. CAPI's defaulting webhook will fill in `spec.replicas: 0` on Create — no worker DataVolumes provisioned until scale-out. With a user CR that sets `minReplicas: 2` (the shape the `kubernetes-latest` / `kubernetes-previous` bats CRs use), the rendered MD carries the `cluster-api-autoscaler-node-group-min-size: "2"` annotation and again **no `spec.replicas` field**. CAPI's defaulting webhook will fill in `spec.replicas: 2` on Create — byte-identical topology to the pre-PR chart's fresh-install state (2 workers), but without the recurring drain-on-upgrade cycle. ### Render-side render walk `helm template` of the OIDC bats CR bodies (after this PR): 1 MachineDeployment with **no chart-managed `spec.replicas`** and `min-size: "0"` on the autoscaler annotation. CAPI seeds `replicas: 0` on Create, 0 KubevirtMachines emitted, 0 DataVolume provisioning triggered. Test assertions on KamajiControlPlane / KeycloakClient / KeycloakClientScope / AuthenticationConfiguration Secret are unchanged. ## Upgrade impact Existing customers stop losing workers on `helm upgrade`. Prior to this PR the chart re-asserted `MachineDeployment.spec.replicas: 2` on every Helm action, so a customer running with the autoscaler at, say, `replicas: 8` would see 6 workers drained on every routine Cozystack platform bump and then wait for the autoscaler to scale back up in response to unschedulable Pods. With `spec.replicas` no longer emitted by the chart, SSA field-manager ownership preserves the autoscaler's value across upgrades unconditionally. Fresh installs with `nodeGroups: {}` (the schema default) transition from "2 workers by default" to "0 workers until scale-out". The chart helper's default md0 carries `minReplicas: 0`, which is what CAPI's defaulting webhook seeds `spec.replicas` from. The `values.yaml` / README / helper docstring / regenerated Go type doc + openAPISchema call this out explicitly; the release-note repeats it below. See finding 5 in the paired self-review for the specific `addons.ingressNginx.enabled: true` interaction operators need to be aware of on fresh installs. ## Test plan - [x] `helm unittest packages/apps/kubernetes/` — 171/171 passing. - [x] `helm template` with `tests/values/common.yaml` (minReplicas: 0) — MD.replicas: 0. - [x] `helm template` with a `minReplicas: 2` override — MD.replicas: 2 (byte-identical to pre-PR). - [ ] E2E: `kubernetes-latest` unchanged (spawns its two workers as before), `kubernetes-oidc-{system,customconfig}` render zero workers, `kubernetes-previous` no longer flakes on LINSTOR pool pressure originating from the earlier OIDC suite. ## Notes - Part 3 of #3231 (intra-file `cozy_wait_linstor_pool_free` gate between kube sub-tests and the orphan clickhouse keeper PVC cleanup) is **not** in this PR — worth a separate targeted change to `hack/e2e-apps/run-kubernetes.sh`, and it's a hardening-only improvement rather than a root-cause fix. Happy to send it as a follow-up if wanted. ## Release note ```release-note fix(apps/kubernetes): stop the chart from managing `MachineDeployment.spec.replicas`; the cluster-autoscaler owns the field alone. CAPI's defaulting webhook now seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create, and SSA field-manager ownership keeps subsequent `helm upgrade` actions from overwriting whatever value the autoscaler patched. Two behavioural changes worth calling out: - Existing customers: `helm upgrade` no longer drains workers on every platform bump — the autoscaler-held `spec.replicas` is preserved unconditionally. Prior to this change the chart hardcoded `replicas: 2`, which drained back to 2 on every reconcile action. - Fresh installs with `nodeGroups: {}` (the schema default) no longer come up with two workers by default. The chart helper's default md0 carries `minReplicas: 0`, and CAPI's defaulting webhook seeds `spec.replicas` from that. Operators enabling `addons.ingressNginx.enabled: true` on such a CR must either supply an explicit nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or accept that the cluster-autoscaler will bring the default md0 up in response to the ingress-nginx controller Pods becoming Pending on install. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Rendered `MachineDeployment` manifests no longer set `spec.replicas`, avoiding chart upgrades overwriting autoscaler-managed sizing. * **Documentation** * Clarified `nodeGroups: {}` behavior: a default `md0` is emitted with `minReplicas: 0`/`ingress-nginx` role, and workers are provisioned only after an autoscaler-triggering unschedulable Pod or manual scaling; updated related storage and ingress guidance. * **Tests** * Strengthened default node group rendering assertions and adjusted OIDC e2e manifests to reduce CI provisioning churn. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3eb182ddf1 |
feat(apps/kubernetes): make Talos image factory URL and installer repo overridable
Tenant worker VMs boot from a Talos raw disk image that CDI streams over HTTP from the hardcoded public Talos Image Factory, and the in-guest talos-reconcile Job pulls its installer image from the same hardcoded host. Neither was overridable, so air-gapped / rate-limited / flaky-egress environments could not point workers at a mirror (the README flagged this as a Phase-2 gap), and CI had no way to avoid bulk-pulling the OS image from the public internet per worker. Add two chart values, both defaulting to the public factory so rendered output is byte-identical and behaviour is unchanged: - talos.imageFactoryURL (default https://factory.talos.dev) - base URL for the worker OS disk image streamed in by CDI. - talos.installerRepository (default factory.talos.dev/installer) - OCI repository prefix for the in-guest installer image. Regenerated values.schema.json, the aggregated-API Go types, the kubernetes-rd cozyrds schema and the README via `make generate`; added unittests for the override and trailing-slash trimming. Refs #3231 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
2805180dad |
fix(keycloak): correct the encryption proxy image tag (0.2.0, no v)
The release workflow publishes the OCI tag as 0.2.0 (docker/metadata-action strips the leading v from the git tag), so the chart default of v0.2.0 does not exist and pulls fail. Pin the real tag. Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
32986d0dc9 |
test(keycloak): cover the deksetSecretKey override
Assert KKP_DEKSET_FILE honors a non-default deksetSecretKey, closing the last uncovered branch of the dekset wiring. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
0b4b981fa2 |
feat(keycloak): support a shared wrapped DEK set for the encryption proxy
Wire encryption.deksetSecretName -> KKP_DEKSET_FILE (mounted read-only) and make encryption.replicas configurable. Without a dekset the proxy mints an ephemeral DEK, which only works on a fresh database and a single replica. A shared wrapped DEK set is required to migrate an existing populated Keycloak (the proxy must reuse the DEK the backfill wrapped rows with) and to run more than one replica; a guard rejects replicas>1 without one. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
95cdbf216d |
chore(keycloak): pin the encryption proxy to keycloak-kms-proxy v0.2.0
v0.2.0 is the first release carrying the Kubernetes/AppRole auth modes this chart wires, so encryption.kms.vault.auth is now usable with the default image. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
56dacf50c9 |
test(keycloak): assert automountServiceAccountToken on the ServiceAccount
Lock in the SA-object token toggle (off by default, on only for kubernetes auth), complementing the pod-level assertions. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
731020215a |
fix(keycloak): give the encryption proxy a dedicated ServiceAccount
Address review: the proxy pod ran under the namespace default ServiceAccount, which the keycloak StatefulSet also uses. Binding a Vault Kubernetes-auth role to default would hand the keycloak pod itself Transit access, defeating the isolation the proxy exists to provide. Add a dedicated keycloak-kms-proxy ServiceAccount, run the proxy pod under it, and mount its token only in the kubernetes auth mode (off otherwise, to minimise the credential surface). Docs name the SA to bind the Vault role to. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
5744516010 |
feat(keycloak): add Vault Kubernetes auth for the encryption proxy
Extend encryption.kms.vault.auth with "kubernetes" (recommended): the proxy logs in with its own ServiceAccount token, so no credential is stored in a Secret. "approle" and "token" remain. Render-time validation requires kubernetes.role and rejects unknown methods; helm-unittest covers the new path and the mutual absence of the other modes' env vars. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
2b9fedc2f3 |
test(keycloak): make the AppRole/token env negative assertions bite
The notContains checks matched whole array elements, so a name-only content never equaled the valueFrom entry and the assertion could never fail. Add any:true so "AppRole stores no static token" and, symmetrically, "token mode renders no AppRole vars" are actually verified (confirmed by mutation-testing the template). Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
31762644c9 |
feat(keycloak): add Vault AppRole auth for the encryption proxy
The keycloak-kms-proxy Vault Transit backend only supported a static token. Add an encryption.kms.vault.auth switch — "token" (default, unchanged) or "approle" — wiring KKP_VAULT_ROLE_ID / KKP_VAULT_SECRET_ID / KKP_VAULT_APPROLE_MOUNT from a Secret. With AppRole the proxy logs in and re-authenticates on demand, so no long-lived token is stored. Requires a keycloak-kms-proxy image with AppRole support (> v0.1.0). Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
|
8ae9009270 |
chore(linstor): bump linstor-server to 1.33.3 and linstor-csi to v1.11.2 (#2987)
## What this PR does Bumps the LINSTOR storage stack to current upstream, picking up the published security fixes the issue enumerates: - `linstor-server` (piraeus-server) `1.33.2 → 1.33.3` (latest 1.33 patch; v1.34.0 is rc-only). Pulls in the 1.33.3 DRBD fixes (clone-status check before connection states arrive; the secondary-after-mkfs path is now guarded upstream too) plus refreshed Debian packages. Note: 1.33.3 also disables DRBD volume shrink upstream (it never worked correctly) — no effect on cozystack, which does not expose volume shrink. - `linstor-csi` `v1.10.6 → v1.11.2`. Its go.mod requires Go ≥ 1.26, so the CSI builder moves `golang:1.25 → golang:1.26` — the right lever for the Go stdlib / `moby/spdystream` CVEs. Both cozystack images are rebuilt multi-arch (linux/amd64 + linux/arm64) and pinned by digest in `values.yaml`. All five piraeus-server patches re-apply cleanly against v1.33.3 in the Dockerfile glob order, and both linstor-csi patches re-apply against v1.11.2. Adds helm-unittest coverage (the package shipped none) asserting the controller / CSI / satellite images stay digest-pinned (`<image>:<tag>@sha256:<64-hex>`), version-agnostic so it survives the release-prepare re-tag. **Verified on a dev cluster:** a rolling restart of all satellites, controller, and CSI to 1.33.3 / v1.11.2 kept every node Online with zero faulty resources; a replicated PVC provisioned, mkfs'd, and reached UpToDate across 3 nodes with no DRBD "device held open" abort (the cozystack secondary-after-mkfs patch handled the path). The csi v1.11.2 "no RWX without DRBD layer" change is a no-op for cozystack, which always uses the DRBD layer. Closes #2881 ### Release note ```release-note chore(linstor): bump linstor-server to 1.33.3 and linstor-csi to v1.11.2 Upgrade impact: an empty tenant Kubernetes worker `nodeGroups[*].storageClass` now defaults to the application-level (DRBD `replicated`) storageClass instead of the management-cluster default, because linstor-csi v1.11.2 rejects ReadWriteMany volumes on a non-DRBD storage class. This changes the rendered worker template, so existing tenant clusters that relied on the empty-storageClass default roll their worker MachineDeployments once on the first reconcile after upgrade — workers are reprovisioned on the application storageClass and container images are re-pulled. The per-nodeGroup override and the both-empty cluster-default escape hatch are preserved. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated LINSTOR Server to **v1.33.3** and LINSTOR CSI to **v1.11.2** (image tags/digest pins refreshed) * Updated the LINSTOR CSI build toolchain to **Go 1.26** * Added a **test** make target to run Helm unit tests * **Documentation** * Updated the LINSTOR Server patches README to reference **v1.33.3** * **Tests** * Added an image-pinning validation suite to confirm controller and satellite images use **sha256** digests <!-- end of auto-generated comment: release notes by coderabbit.ai --> ## Worker-disk storageClass default (companion to the bump) linstor-csi v1.11.2 rejects ReadWriteMany volumes that are not on a DRBD-layer storage class. Worker VMs live-migrate, so their disks need RWX — the access mode comes from the chosen StorageClass's CDI StorageProfile (a DRBD class yields RWX), it is not set on the worker DataVolume by the chart. With the bump, a worker nodeGroup left on a non-DRBD cluster-default StorageClass would fail provisioning, so an empty per-nodeGroup storageClass now defaults to the application storageClass (replicated/DRBD) instead of the cluster default. The per-nodeGroup override and the both-empty cluster-default escape-hatch are preserved. Upgrade impact: the empty-storageClass default ships on `md0`, and the KubevirtMachineTemplate name is content-addressed over the rendered worker body, so this flip renames the template and triggers a one-time CAPI rolling replacement of worker pools on the first reconcile after upgrade. Two hash-pin tests guard the rename so it stays an intentional, reviewed change. linstor-csi v1.11.2 also defaults DRBD two-primaries to kubevirt-only — a no-op for cozystack, where kubevirt is the target. |
||
|
|
d30a2cfffb |
docs+test(apps/kubernetes): align nodeGroups default docs with actual zero-worker topology + regression-guard the replicas fix
Follow-ups from the self-review of #3233: - `values.yaml` / helper docstring / regenerated README + schema + cozyrds openAPISchema + Go type doc: the previous "one ingress-nginx worker" contract was already misleading (helper default carried `minReplicas: 0` but the chart hardcoded `spec.replicas: 2`), and the replicas fix makes the docstring strictly false — an empty-nodeGroups install now provisions zero workers until either the cluster-autoscaler responds to an unschedulable Pod or the operator explicitly scales the group. Update the docstrings to describe the actual behaviour and to spell out the ingress-nginx-addon implication so operators are not surprised at addon-install time. - `tests/nodegroups_default_test.yaml`: two regression cases pinning `MachineDeployment.spec.replicas == $group.minReplicas` for `minReplicas: 0` (the render-side OIDC bats path — zero worker DataVolumes provisioned) and for `minReplicas: 2` (the kubernetes-latest / kubernetes-previous bats path — byte- identical to pre-PR behaviour). Without these a future refactor can silently re-hardcode `replicas: 2` and turn no tests red. helm-unittest suite grows from 171 to 173, both new cases pass. Ref: cozystack/cozystack#3231. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
cece5c23b5 |
feat(release): immutable tags & rc→stable promotion (#2677) (#3017)
## What this PR does Implements the immutable-tag + rc-promotion flow from #2677. A **stable release becomes a renamed release-candidate**: the bytes shipped as `vX.Y.Z` are bit-for-bit the bytes built and e2e-tested as `vX.Y.Z-rc.N`. No tag is ever force-moved, and stable is never rebuilt — it is *promoted* by retagging the rc's existing images. Layered onto the build matrix from #2937/#2983 (this PR is **stacked on `refactor/build-matrix-2937`** and must merge after it). ## The five force-retag sites, removed | Site | Before | After | |---|---|---| | `tags.yaml` api/apps/v1alpha1 tag | `git tag -f` / `push -f` | write-once (create-if-absent, fail if it would move) | | `tags.yaml` release-X.Y.Z branch | `git branch -f` / `push -f` | compare-before-force (no-op if unchanged; staging branch only) | | `pull-requests-release.yaml` stable tag | `git tag -f` / `push -f` | write-once at the PR merge commit (force impossible by construction) | | `pull-requests-release.yaml` maintenance branch | `updateRef force:true` | fast-forward-only | | `auto-release.yaml` patch tags | delete-recreate (cron) | **workflow deleted** — stable only via explicit promote | ## Version decoupling (the enabler) The operator baked its version into the image at build time, so an rc image self-reported the rc string — blocking retag-promotion. It now reads `COZYSTACK_VERSION` from the environment (threaded via `cozystackOperator.platformVersion` → Deployment env, stamped by `make manifests`), falling back to the build-time value when unset. The same image bits can report any release name. The only runtime reader is the telemetry metric `cozy_cluster_info{cozystack_version=...}`. ## Promotion flow `promote-rc.yaml` (`workflow_dispatch`, `rc_tag=vX.Y.Z-rc.N`): 1. Validate the rc release exists and the stable tag does not. 2. `hack/promote-retag.sh` reads the rc's digest-pinned image refs from `packages/*/*/values.yaml` and `skopeo copy`s each — **by digest** — to `:vX.Y.Z` and `:latest`, verifying with `skopeo inspect`. 3. Rewrite the cosmetic `-rc.N` substring in vendored tags to the stable version (the `@sha256` wins regardless), restamp the version-stamped assets (operator manifests, cozypkg, openapi; the heavy Talos assets are copied verbatim from the rc draft), open a `release-X.Y.Z` PR. 4. Merging the PR reuses the existing release-PR e2e and `pull-requests-release.yaml` finalize to cut the **write-once** stable tag at the merge commit and publish the release. Squash is disallowed (the tag needs a real merge commit). `nightly.yaml` cuts write-once `*-nightly.<date>` tags (gated by `NIGHTLY_ENABLED`); `retention.yaml` keeps the newest 14 per line (dry-run by default). ## Validation **Local end-to-end — the core claim is proven, not asserted.** The whole design rests on "copy-by-digest to a new tag preserves the digest, so stable == the e2e-tested rc, bit-for-bit." This was exercised against a real registry (`ttl.sh`) with real `skopeo`: - Pushed a multi-arch image under a `:v1.4.0-rc.2` tag, built a temp tree exercising all three `values.yaml` digest shapes (single `image:` string, split `repository`/`tag`/`digest` map, and the `platformSourceRef` OCI artifact), then ran the actual `hack/promote-retag.sh v1.4.0`. - **Result: `:v1.4.0-rc.2`, `:v1.4.0`, and `:latest` all resolve to the identical digest** (`sha256:fd8d9aa6…`), across all 17 platform manifests. The script's own `skopeo inspect` post-check passed, and an independent re-inspection confirmed it. Local validation caught (and this PR fixes) two bugs that only surface against a real registry — never in static lint: 1. `skopeo copy --multi-arch all --all` — mutually exclusive flags, fatal error; every retag would have failed on CI. Now `--multi-arch all`. 2. The same digest was retagged twice when an image appeared in two value shapes; the ref set is now deduped on the canonical `repo@digest`. **Other local checks:** `go test ./pkg/version` + `go build`/`vet`; `helm template` renders `COZYSTACK_VERSION` in all 3 operator variants (omitted when unset); `make manifests` stamps the version into the install assets; `shellcheck` clean on `promote-retag.sh`; `actionlint` clean and `act -l` resolves the job graph on all workflows; the rc-tag parser and nightly version-math unit-tested (accept/reject + minor-vs-patch bumps). **Still CI-only (cannot be exercised offline):** the full workflow runtime — github-script API calls, OCIR auth, the rc-release / staging-branch preconditions, the nightly `base_ref` push mechanics, and self-hosted runners. The *logic* inside the steps is unit-tested; the *orchestration* is not. Treat it as unproven until the workflows run. ## Out of scope / required follow-up (needs repo admin — not doable from a PR) - Tag-protection rules: `v*` and `api/apps/v1alpha1/*` create-only by the CI app, no delete/update; "limit branches/tags updated in a single push" = 1. - Enable `NIGHTLY_ENABLED` (and `RETENTION_APPLY` when ready) repo variables. - Registry-side pruning of `*-nightly.*` image tags (no OCIR delete parity yet — tracked TODO in `retention.yaml`). ## Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added nightly publishing (mirror-by-digest, disk build, e2e validation) and nightly retention pruning. * Added RC-to-stable promotion via digest retagging (no rebuild). * Operator/installer and console now expose version/platform metadata, including runtime override via `COZYSTACK_VERSION`. * **Bug Fixes** * Enforced write-once tag behavior and fast-forward-only maintenance updates across release workflows. * Made nightly mirroring/selection and retention pruning more selective and safer. * **Tests** * Added Bats coverage for nightly mirroring and RC retagging, plus new Helm/unit checks. * **Documentation** * Updated the release model to center RC promotion and tag immutability. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
7b65cf2afd |
feat(platform): migrate to etcd-operator v1alpha2 (#2859)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for the newer etcd snapshot workflow (BackupClass/BackupJob/RestoreJob) with in-place restore. * Updated etcd chart/operator packaging and migration flow for the v1alpha2 operator release. * **Bug Fixes** * Switched readiness/condition handling to **Available**. * Improved waits to cover all etcd member pods during restore and scenario steps. * **Documentation** * Refreshed etcd backup/restore docs and example scenarios to match the v1alpha2 APIs and parameters. * **Tests** * Updated e2e coverage to validate v1alpha2 operator runtime contracts and replaced legacy backup-block tests with a snapshot round-trip test. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ff7c8fa1ad |
fix(dashboard): guard empty COZYSTACK_VERSION in the console APP_VERSION build-arg
Address non-blocking review feedback from lexfrei on packages/system/dashboard/Makefile:34. When COZYSTACK_VERSION is empty (a local build with no matching git tag) the build-arg baked a bare `APP_VERSION=v` into the console bundle. Match the guard the root Makefile already uses for platformVersion so an empty version yields an empty APP_VERSION instead of "v". Dev-only cosmetics — the config.json path overrides the displayed version at deploy time — but it removes the mismatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
70adc01e79 |
ci(release): nits — skopeo login via stdin; tighten dashboard version regex
- nightly.yaml: pass the registry tokens to `skopeo login` over stdin (--password-stdin) instead of `-p` on argv, matching promote-rc/finalize and keeping the token out of the process list. - dashboard configmap: require digits-then-dot (^v?[0-9]+\.) when deriving the displayed platform version from the console image tag. The old ^v?[0-9] let a tagless, port-carrying ref (ghcr.io:443/.../console, whose last ":"-segment is "443/.../console") pass as a version. Verified: v1.5.0 / 1.5.0 still surface; the port ref and :latest no longer do. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
727f4e745e |
fix(dashboard): keep the v prefix on the reported platform version
The console config ConfigMap stripped the leading v (trimPrefix "v") while the operator's COZYSTACK_VERSION and the vX.Y.Z release tags keep it, so the dashboard displayed 1.4.0 where the operator reported v1.4.0. Keep the tag verbatim, and build the baked VITE_APP_VERSION fallback as v$(COZYSTACK_ VERSION) too so the deploy-time and build-time version strings stay consistent with each other and with the operator. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
011d6d1639 |
fix FUs
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
ee12295d30 |
fix(etcd-operator): set seccompProfile RuntimeDefault on controller pod
Address review feedback from gemini-code-assist on packages/system/etcd-operator/templates/deployment.yaml: add an explicit seccompProfile (RuntimeDefault) to the controller-manager pod securityContext. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
087a8c4be9 |
chore(etcd-operator): bump to v0.5.2
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
cc5d688e15 |
feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
7b8a330ad4 |
docs(apps/kubernetes): update users[].email docstring — email claim, not preferred_username
Gemini flagged that the `values.yaml` @field docstring on `oidc.users[].email` still described the claim mapping as `preferred_username`, misleading operators after the claim was switched to `email` for real (see the earlier fix and its end-to-end verification on dev3). Rewrite the description to match the actual mechanic: `email` claim, requested via `--oidc-extra-scope=email` in the chart-generated kubeconfig, emitted by every conformant OIDC provider. `make generate` re-propagates the update through README.md, types.go and the cozyrds embedded schema. Note on the other finding in the same review — Gemini also flagged `audiences` as being incorrectly nested under `issuer` in the AuthenticationConfiguration Secret. That is a false positive: per the kube-apiserver v1beta1 spec (apiserver.config.k8s.io/v1beta1, `JWTAuthenticator.Issuer`), `audiences` IS a child of `issuer`, not a sibling of `jwt[i]`. The current shape is correct and verified end-to-end on dev3 (alice/bob tokens carry `aud: [tenant-test-kubernetes-test3, kubernetes]` and apiserver validates them successfully with CRB-driven RBAC). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
9c73b6ef10 |
fix(apps/kubernetes): render-fail when spec.oidc.mode collides with legacy --oidc-* extraArgs
Reviewer flagged a real customer-upgrade regression: the previous values.yaml docstring on `controlPlane.apiServer.extraArgs` documented `--oidc-issuer-url`/`--oidc-client-id` as a valid usage. An existing customer who wired OIDC by hand through that passthrough (following the doc) would see their tenant kube-apiserver BOOT-FAIL the moment they toggle `spec.oidc.mode` to `System` or `CustomConfig`: kube-apiserver refuses to boot with both `--authentication-config` (injected by the new spec.oidc) and any legacy `--oidc-*` flag — `--authentication-config cannot be used with any of --oidc-*`. Add a render-time guard: when `spec.oidc.mode != None`, scan the user-supplied `controlPlane.apiServer.extraArgs` for any flag with prefix `--oidc-` and fail with a clear message pointing at the `spec.oidc.mode: CustomConfig` migration path (they can carry the same issuer/audience inside `customConfig.config`). When `spec.oidc.mode: None` the passthrough still works — customers who manage OIDC entirely by hand keep working until they opt into the new selector explicitly. Also update the values.yaml docstring on `extraArgs` to steer OIDC users to `spec.oidc.mode` and warn against the collision. Two new helm-unittest cases: the collision fails, and mode=None with legacy flags renders normally (backward compat). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
0d70d5ff3e |
feat(system/kubernetes-rd): expose OIDC kubeconfig Secret to the dashboard
Add `kubernetes-<name>-oidc-kubeconfig` to the Kubernetes ApplicationDefinition's `spec.secrets.include[].resourceNames` so the dashboard surfaces the OIDC kubeconfig to tenant viewers alongside the existing admin kubeconfig. The Secret is produced by the post-install,post-upgrade bootstrap Job in `apps/kubernetes` when `spec.oidc.mode: System` is set. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
5950450758 |
feat(apps/kubernetes): add Kubernetes.spec.oidc selector API
Introduce a flat selector — `mode: System | CustomConfig | None` — plus a `users[]` list and a `customConfig` block for tenant-supplied AuthenticationConfiguration. Default `None` preserves today's behavior. The `System` mode trusts the platform `cozy` realm via a per-cluster public client; `CustomConfig` accepts an inline `AuthenticationConfiguration` or a Secret reference (mutually exclusive). The `users[]` list drives per-user ClusterRoleBindings inside the tenant cluster, with two roles: `admin` and `view`. Regenerates deepcopy, JSON schema, README, and the kubernetes-rd cozyrds embedded schema. Implements Phase 1 API of the per-tenant OIDC design proposal (cozystack/community#24). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
|
17002d0d78 |
feat(dashboard): report platform version at deploy time
The console bakes VITE_APP_VERSION into its bundle at build time, so a promote-by-retag image (which is retagged, never rebuilt) would keep reporting the rc version in the UI after promotion to stable. Inject the version at deploy time instead: the chart derives it from the console image tag (build-stamped vX.Y.Z, rewritten rc->stable on promotion by promote-rc.yaml) and writes it into the existing console config.json ConfigMap, which the SPA already fetches. App.tsx now prefers config.version and falls back to the baked VITE_APP_VERSION when absent (dev / non-version tags like :latest). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
59f035ff53 |
fix(kubevirt-instancetypes): restore persistent EFI/TPM state
Reverts #3006. Dropping persistent EFI/TPM was motivated by a live-migration concern that rested on an outdated assumption — that an RWO Filesystem backend-storage PVC pins the VM to its node. KubeVirt has migrated RWO-Filesystem backend storage since v1.4 (kubevirt/kubevirt#12629): on migration it creates a fresh target state PVC and copies the small state blob, so persistent-EFI/TPM VMs on the default replicated storage live-migrate fine (verified: the VM reports LiveMigratable=True and the copy-on-target PVC is created on migration). Restore upstream's persistent firmware state for the windows.* preferences. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
a44d11d2b6 |
fix(cozystack-basics): validate legacy Ingress hostnames against the tenant apex (#3200)
## What this PR does The Gateway (`cozystack-gateway-hostname-policy`) and Route (`cozystack-route-hostname-policy`) admission policies constrain tenant hostnames to the tenant apex on the Gateway API path. Gateway API is opt-in and off by default, so in the default configuration tenant applications publish through a legacy `networking.k8s.io/Ingress` on the shared ingress-nginx, which had no equivalent policy. This adds `cozystack-ingress-hostname-policy`, a fail-closed `ValidatingAdmissionPolicy` + binding for `Ingress` in `tenant-*` namespaces — the legacy-Ingress counterpart of the existing Gateway/Route hostname policies, and a new Layer 8 in the gateway security model. A hostname on `spec.rules[].host` or `spec.tls[].hosts[]` is allowed when it is within the namespace's own apex (`namespace.cozystack.io/host`), or when it is a concrete (non-wildcard) name entirely outside the platform root apex (`_cluster.root-host`). The second case keeps the `kubernetes` app's Proxied `exposeMethod` working, since it routes a user-supplied external domain to a nested cluster. A hostname under the platform root apex but outside the namespace's own apex, a wildcard taken through the outside-root path (which could otherwise match the platform apex itself), a rule with no host (absent or empty — an unbounded catch-all), and `spec.defaultBackend` are rejected. The policy is fail-closed: a `tenant-*` namespace missing its own apex label is denied, and the policy renders only when `_cluster.root-host` is set, so it is never shipped in a state that would admit every hostname. `cozy-*` namespaces are cluster-admin-managed and not gated. This is defense-in-depth in the same family as the Gateway/Route policies — by design tenants hold only read access to `Ingress`. The change ships helm-unittest coverage (render, plus the root-host, empty-host, and wildcard guards) and behavioral admission tests against the live apiserver in `hack/e2e-apps/gateway.bats` (in-apex allow, external-domain allow, under-root-outside-apex deny, wildcard-matching-apex deny, hostless deny, empty-string-host deny, defaultBackend deny), and documents the new layer in `packages/extra/gateway/README.md`. ### Screenshots N/A — no UI change. ### Release note ```release-note fix(cozystack-basics): constrain legacy Ingress hostnames in tenant namespaces to the tenant apex ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added **fail-closed** defense-in-depth Layer 8 (`cozystack-ingress-hostname-policy`) to enforce allowed legacy Ingress rule and TLS hostnames for tenant namespaces, while blocking unsafe catch-all configurations (including missing/empty hosts and `spec.defaultBackend`) when the platform root host is configured. * **Tests** * Added end-to-end and render/deny validation for permitted vs denied hostname patterns, wildcard behavior, required namespace host labeling, and correct deny/fail-closed rendering. * **Documentation** * Updated the gateway security model to document Layer 8 behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
a1c779854b |
fix(vertical-pod-autoscaler): migrate admission-controller TLS to cert-manager to stop reconcile-driven ReplicaSet churn (#3178)
## Summary
The VPA admission-controller webhook TLS Secret is currently rendered by
an in-chart `genCA` + `genSignedCert` template. Those functions are not
idempotent across `helm template` invocations — every Flux HelmRelease
reconcile produces a fresh CA + cert. A `checksum/tls-secret`
podAnnotation on the admission-controller Deployment then flips its
sha, materialising a new ReplicaSet on every reconcile.
Under load — the E2E install phase where cert-manager / CDI / LINSTOR
cascade races repeatedly refire the VPA HelmRelease — this produces
10+ admission-controller ReplicaSets in minutes and blocks the whole
cozystack-platform HR: the admission-controller pod livelocks in
startup (waits on its own election lease which its previous
ReplicaSet is still holding, hits livenessProbe timeout,
CrashLoopBackOffs), VPA never becomes Ready, PackageSources stay
InProgress, and cozystack-platform aborts at its 15-minute install
timeout.
Fix: replace the source of the TLS Secret with a namespace-scoped
cert-manager `Issuer` + `Certificate`. cert-manager only rewrites the
Secret when the certificate is close to expiry (`renewBefore`), so
reconcile-driven cert churn stops. Pod rotation is delegated to a
runtime-lookup-based `checksum/tls-cert` podAnnotation that only
changes when the Secret's `tls.crt` content actually changes — not on
every render, not on every reconcile.
## What
-
`charts/vertical-pod-autoscaler/templates/admission-controller/certificate.yaml`
(new): namespace-scoped selfSigned Issuer + Certificate. The
Certificate mirrors the legacy `genSignedCert` shape (single DNS SAN
`vpa-webhook.<ns>.svc`, 365d validity, matching Secret name), so
cert-manager on the first upgrade adopts the same Secret name the
legacy path wrote — no rename cascade in the Deployment volume mount.
Rendered only when `admissionController.tls.certManagerEnabled=true`
AND no `existingSecret` is supplied.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/tls-secret.yaml`
(modified): wrapped in an extra `certManagerEnabled=false` guard so
the legacy genCA path remains available as an explicit opt-out.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/deployment.yaml`
(modified): the legacy `checksum/tls-secret` annotation is gated
behind `certManagerEnabled=false`. Under the cert-manager path the
Deployment carries a `checksum/tls-cert` annotation computed from
`lookup`-ing the runtime Secret's `tls.crt` — see the template
comment for the full timing analysis and the day-350 renewal
trade-off it documents.
- `charts/vertical-pod-autoscaler/values.yaml` (modified): new
`admissionController.tls.certManagerEnabled` value defaulting to
`true`. Values-block header documents the switch as effectively
one-way (cert-manager writes `type: kubernetes.io/tls`;
`Secret.type` is immutable, so falling back requires `kubectl
delete` first).
- `patches/cert-manager-admission-tls.diff` (new) +
`Makefile update` extended: the four subchart edits above live as a
single reviewable diff that Makefile re-applies over the freshly
vendored chart on every `make update`, matching the fluxcd-operator
package convention. Prevents a bump from silently reverting the
fix.
- `packages/core/platform/sources/vertical-pod-autoscaler.yaml`
(modified): VPA PackageSource now depends on `cozystack.cert-manager`.
Non-circular: cert-manager depends on `cozystack.networking` +
`cozystack.gateway-api-crds`, neither reaches VPA.
- `tests/vpa_test.yaml` (modified): +11 helm-unittest cases (15 total,
4 pre-existing) covering `certManagerEnabled={true,false}`,
`existingSecret` bypass, disabled admission-controller, and
the offline-render behaviour of the `checksum/tls-cert` lookup.
## Why (evidence)
Snapshot from CI run 28587656430 (this branch, pre-workaround-review):
```
admission-controller ReplicaSets after first-install churn window:
vertical-pod-autoscaler-admission-controller-55d888d6d4
vertical-pod-autoscaler-admission-controller-56f8d4c747
vertical-pod-autoscaler-admission-controller-5775dbc9b7
vertical-pod-autoscaler-admission-controller-585b7c56c9
vertical-pod-autoscaler-admission-controller-6fc44944fb
vertical-pod-autoscaler-admission-controller-74bccf4849
vertical-pod-autoscaler-admission-controller-84df98f98b
vertical-pod-autoscaler-admission-controller-8666dffb85
vertical-pod-autoscaler-admission-controller-86b469457c
vertical-pod-autoscaler-admission-controller-9b979f9d6
→ 10 ReplicaSets, one per Flux reconcile, each with a different
genCA-rooted TLS Secret sha in checksum/tls-secret.
```
## Migration for existing clusters
On upgrade the legacy Opaque Secret disappears from the release
manifest, Helm garbage-collects it, and cert-manager immediately
writes a fresh `kubernetes.io/tls` Secret of the same name via the
new Certificate. `Secret.type` is immutable, so in-place adoption is
impossible — the delete/create pass is the correct path, and the
seconds-long window it produces is safe: the `vpa-webhook-config`
MutatingWebhookConfiguration ships with `failurePolicy: Ignore`
(verified on cozy-dev3), so Pod admissions in that window skip VPA
mutation but never get rejected — no cluster-wide disruption.
The Deployment rolls in the same upgrade because the new
`checksum/tls-cert` annotation differs from anything the pod had
before, so the admission-controller re-registers `vpa-webhook-config`
with the fresh caBundle on the same pass.
## Test plan
- [x] `helm unittest packages/system/vertical-pod-autoscaler` — 15/15
pass.
- [x] `helm template` diff between main and this branch is scoped
exactly to the cert-manager path (new Certificate + Issuer,
new `checksum/tls-cert` annotation, guarded legacy Secret, new
values key, patch file + Makefile step, PackageSource
dependsOn). No unrelated lines change.
- [x] `make update`-style simulation (rm -rf charts && helm pull &&
patch) produces a byte-identical subchart to what git tracks —
the patch stays clean across future upstream bumps until it
genuinely conflicts.
- [ ] Live migration on cozy-dev3: `helm upgrade` VPA release from
pre-fix version → observe cert-manager write the new tls Secret
→ verify `checksum/tls-cert` on the new pod's podTemplate →
verify MutatingWebhookConfiguration `vpa-webhook-config`
caBundle updated in place → confirm no ReplicaSet churn on
subsequent Flux reconciles. Blocking the draft's removal.
- [ ] Full E2E on this branch — verify only 1 admission-controller
ReplicaSet after install, VPA HR reaches Ready inside
`cozystack-platform`'s 15m install timeout, and
`kubernetes-latest.bats` / `kubernetes-previous.bats` no longer
time out on VPA-attributable resource churn.
## Review pass — kvaps' CHANGES_REQUESTED addressed
- ✅ Vendored chart mods now carried as `patches/*.diff` re-applied by
`Makefile update` (matches fluxcd-operator convention).
- ✅ Retired the tls-migrator hook — kvaps correctly showed it was
a no-op: Helm's delete decision is driven by the stored release
manifest not by live labels (`ResourcePolicyAnno` is the only opt-
out), and `Secret.type` is immutable so cert-manager could not have
adopted the legacy Opaque Secret in place anyway. ~200 lines
removed.
- ✅ PR description rewritten to match the actual diff (no more
reference to the removed `tls-rollout.yaml` post-upgrade hook, real
test count, dev3 verification honestly listed as pending).
- ✅ `checksum/tls-cert` timing comment corrected: `lookup` runs on
Helm actions only, day-350 renewal trade-off spelled out.
- ✅ `certManagerEnabled=false` documented as effectively one-way
(Secret.type immutable).
- ✅ Dropped the `secretTemplate` labels — `app.kubernetes.io/managed-
by: Helm` on a cert-manager-owned Secret was misleading.
- ✅ Banner comment indentation restored in `vpa_test.yaml`.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* VPA admission-controller webhook TLS now uses cert-manager by default,
with options to disable cert-manager or supply an existing TLS secret.
* Webhook pod restarts are now tied to cert-manager Secret contents via
a content-based checksum.
* **Bug Fixes**
* Improved webhook TLS provisioning reliability when TLS readiness is
delayed.
* Corrected conditional rendering/rollout behavior to avoid mismatched
TLS ownership.
* **Tests**
* Added rendering-invariant tests covering cert-manager, legacy TLS,
existing-secret bypass, and admission-controller disabled behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
d89ad70698 |
fix(cozystack-basics): validate legacy Ingress hostnames against the tenant apex
The Gateway (cozystack-gateway-hostname-policy) and Route (cozystack-route-hostname-policy) admission policies constrain tenant hostnames on the Gateway API path. Gateway API is opt-in and off by default, so in the default configuration tenant applications publish through a legacy networking.k8s.io/Ingress on the shared ingress-nginx, which had no equivalent policy. Add a fail-closed ValidatingAdmissionPolicy + Binding for Ingress in tenant-* namespaces. A hostname on spec.rules[].host or spec.tls[].hosts[] is allowed when it is within the namespace's own apex (namespace.cozystack.io/host), or when it is a concrete (non-wildcard) name entirely outside the platform root apex (_cluster.root-host); a hostname under the platform root apex but outside the namespace's own apex, and a wildcard taken through the outside-root path (which could match the platform apex itself), are rejected. The outside-root branch keeps the kubernetes app's Proxied exposeMethod working, since it routes a user-supplied external domain to a nested cluster. A rule with no host (absent or empty, an unbounded catch-all) and spec.defaultBackend are also rejected. The policy templates the platform root apex into its CEL and renders only when _cluster.root-host is set, so it is never shipped in a state that would admit every hostname. A tenant-* namespace missing its own apex label is denied. Cover the policy with helm-unittest (render, including the root-host, empty-host, and wildcard guards) and behavioral admission tests against the live apiserver (in-apex allow, external-domain allow, own-apex-wildcard allow, external-tls allow, under-root-outside-apex deny, prefix-adjacent-under-root deny, wildcard-matching-apex deny, hostless deny, empty-string-host deny, defaultBackend deny), and document it as an additional layer in the gateway security model. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
223c0ef4fb |
fix(cert-manager): raise cainjector memory limit to unblock caBundle injection
cainjector loads all ValidatingWebhookConfigurations, APIServices and CRDs into its informer caches at startup. On a full cozystack install that working set exceeds the 128Mi limit, so the single leader-elected cainjector Pod is OOMKilled during cache population and CrashLoops before it injects any caBundle. Webhook ValidatingWebhookConfigurations then keep an empty caBundle and every admission call fails with "x509: certificate signed by unknown authority" (failurePolicy: Fail), intermittently blocking Ingress creation and stalling tenant installs. Raise the cainjector memory limit to 512Mi (matching the controller) and the request to 128Mi, and add a helm-unittest that pins the sizing so a blanket resource change or a `make update` cannot silently lower it again. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
|
10f8e2de1f |
feat(kamaji): default to 2 replicas, drop telemetry webhook (#2671)
## What this PR does
`kamaji-controller-manager` runs **9 sub-controllers per
TenantControlPlane** (bootstraptoken, coredns, konnectivity, kubeadm*,
kubelet, kubeproxy, …) inside a single Go process, **and** serves the
admission webhooks for `tenantcontrolplanes` + `datastores` from the
same pod via `kamaji-webhook-service`. With the upstream default
`replicaCount=1`, every webhook round-trip from the apiserver shares CPU
with the reconciler goroutines. On a multi-tenant cluster this pushes
the apiserver-observed p99 of `mtenantcontrolplane.kb.io` and
`vtenantcontrolplane.kb.io` from the kamaji-side `~10ms` (per
`controller_runtime_webhook_latency_seconds_count/sum`) up to `85–95ms`
— purely due to CPU contention.
Three safe defaults are changed:
- **`kamaji.replicaCount: 2`** — HA-safe baseline. `--leader-elect` is
already enabled upstream, so only one replica reconciles tenants; the
follower's CPU is fully available for webhook handlers. On a single-node
cluster both pods still schedule (just colocated).
- **`kamaji.telemetry.disabled: true`** — drops the
`--disable-telemetry` handler **and** removes the
`telemetry.kamaji.clastix.io` entry from the
`ValidatingWebhookConfiguration`. The latter requires `patches/1.diff`
to wrap the telemetry block in `{{- if not .Values.telemetry.disabled }}
… {{- end }}` inside the vendored
`controller-gen/validating-webhook.yaml`. Without the webhook entry, the
apiserver no longer round-trips to a no-op handler on every
TenantControlPlane CREATE/UPDATE/DELETE.
- **`kamaji.affinity` (soft `podAntiAffinity`)** — spread the two
replicas across nodes when possible, while keeping single-node clusters
working (`preferredDuringScheduling`, not required).
The `patches/1.diff` approach (re-applied by `make update`) mirrors the
pattern already used in `packages/system/fluxcd-operator/Makefile` and
`packages/system/ingress-nginx/Makefile`, and survives the next
`clastix/kamaji` chart bump.
## Measured impact
Hikube prod, 22 TenantControlPlanes, before/after a manual rollout of
the same configuration:
| webhook | p99 before | p99 after | Δ |
| --- | --- | --- | --- |
| `mtenantcontrolplane.kb.io` | 85 ms | 25 ms | **-71%** |
| `vtenantcontrolplane.kb.io` | 92 ms | 25 ms | **-73%** |
| `telemetry.kamaji.clastix.io` | 94 ms | _webhook removed_ | **-100%**
|
Cumulative Kamaji admission **impact** (`p99 × calls/s`) dropped from
**830 → 140 (-83%)**. The Kamaji webhooks moved from the top three of
the `apiserver_admission_webhook_admission_duration_seconds_*` impact
ranking down to positions #3-#4.
## Release note
```release-note
feat(kamaji): default to 2 replicas with soft pod anti-affinity and disable telemetry (handler + webhook entry). Reduces apiserver admission p99 for tenantcontrolplane mutating/validating webhooks by ~70% on multi-tenant clusters.
```
## Test plan
- [x] `helm template packages/system/kamaji` renders `replicas: 2`,
`--disable-telemetry`, the soft anti-affinity, and **no**
`telemetry.kamaji.clastix.io` entry in the
`ValidatingWebhookConfiguration`.
- [x] `patches/1.diff` applies cleanly against the freshly extracted
upstream chart (`patch --dry-run -p1 < patches/1.diff`).
- [x] `make update` re-applies the patch automatically.
- [x] Live deploy on a 22-TCP cluster: 3 pods Running (when overriden to
`replicaCount: 3` via cozystack-values), zero pod restarts, leader
election visible, `kamaji-webhook-service` Endpoints serving N IPs.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Telemetry can be disabled via configuration.
* Controller configured for high availability with 2 replicas.
* Pod anti-affinity added to spread replicas across nodes for better
resilience.
* **Bug Fixes**
* Validating webhook is now omitted when telemetry is disabled to avoid
unintended behavior.
* **Chores**
* Update process improved to re-apply chart patches after extraction for
reproducible updates.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2671?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
809a04a220 |
feat(kubevirt-cdi): make cloneStrategyOverride configurable (#2926)
## What this PR does - Exposes the CDI `cloneStrategyOverride` as a Helm value in `packages/system/kubevirt-cdi` instead of hardcoding it in the CR template. - Default stays `csi-clone`, so **behavior is unchanged** for everyone. - It can now be overridden via a cozystack `Package` (e.g. `cloneStrategyOverride: copy`), the same way `uploadProxyURL` already is — no fork needed. ### Why The value was hardcoded, so the only way to change the clone strategy was to fork the package. On some storage backends `csi-clone` is problematic: with LINSTOR/DRBD a CSI clone is created co-located with the source golden image and then relocated to optimal nodes asynchronously; that async relocation (`migrate-disk`) can race the post-clone resize CDI performs, occasionally leaving a volume whose DRBD device is smaller than its PVC capacity. Being able to select `snapshot` or `copy` per cluster, without maintaining a fork, is a useful escape hatch. This only adds a knob; it does not change the default or recommend a strategy. ## Release note ```release-note feat(kubevirt-cdi): make CDI cloneStrategyOverride configurable via values (default csi-clone) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * The KubeVirt CDI clone strategy is now configurable via Helm chart values, allowing selection of `csi-clone`, `snapshot`, or `copy` for DataVolume cloning. * The default remains `csi-clone` to preserve existing behavior and ensure backward compatibility. * **Documentation** * Added the `cloneStrategyOverride` Helm value description and allowed options to the chart’s configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
901bb981ae |
fix(seaweedfs): configure postgres2 connection pool for the filer (#2906)
## What this PR does - Adds `WEED_POSTGRES2_CONNECTION_MAX_IDLE=20`, `WEED_POSTGRES2_CONNECTION_MAX_OPEN=40` and `WEED_POSTGRES2_CONNECTION_MAX_LIFETIME_SECONDS=600` to the filer `extraEnvironmentVars` in `packages/system/seaweedfs/values.yaml`. The vendored seaweedfs chart only ships connection pool settings for the **mysql** store (`WEED_MYSQL_CONNECTION_MAX_*`), which cozystack does not use. The **postgres2** store, which cozystack enables, has no pool settings at all. As a result the Go `database/sql` pool runs with `SetMaxIdleConns(0)`: every filer metadata lookup opens a fresh PostgreSQL connection (TCP + TLS + SCRAM, ~300 ms each). With ~5 path-component lookups per S3 operation, this adds roughly **2 seconds of latency to every S3 request**, even on an idle cluster. ### Measurements (production cluster, 12 nodes, 2 filers, CNPG backend) Before / after applying these three variables, same test (sequential ops through the S3 gateway): | Operation | Before | After | |---|---|---| | PUT 4 KiB (p50) | 10.0 s | 0.28 s | | GET 4 KiB (p50) | 3.0 s | 0.14 s | | HEAD (p50) | 2.6 s | 0.16 s | | Bounded LIST, 100 keys (p50) | 3.8 s | 0.37 s | `pg_stat_activity` confirmed the root cause: before the fix every backend was younger than ~200 ms (permanent connection churn); after the fix connections are reused for minutes. Real-traffic p95 of `SeaweedFS_filerStore_request_seconds` dropped from 1.3–6.5 s to ~0.18 s. `max_open` is capped at 40 so that the default two filer replicas stay below the CNPG default `max_connections` of 100. ## Release note ```release-note fix(seaweedfs): configure postgres2 connection pool for the filer, removing ~2s of per-request S3 latency caused by PostgreSQL connection churn ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Optimized PostgreSQL connection pool configuration in SeaweedFS to reduce latency and improve performance during metadata operations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ec9c29948b |
[reloader] Set resources requests/limits on reloader container (#2631)
## Summary
The upstream Stakater reloader chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `reloader`
Deployment.
Override `packages/system/reloader/values.yaml` at
`reloader.reloader.deployment.resources` with the exact values from the
chart's own commented example:
- `limits` : `cpu: 100m`, `memory: 512Mi`
- `requests` : `cpu: 10m`, `memory: 128Mi`
Reloader is a light controller that watches ConfigMap / Secret changes
and triggers workload restarts — small CPU envelope with enough memory
to track many objects in clusters with lots of namespaces.
## Test plan
- [ ] `helm template . --namespace cozy-reloader` from
`packages/system/reloader/` renders a `resources` block on the
`reloader-reloader` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `reloader-reloader has no
CPU limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the reloader Deployment to clear
the "no CPU limit" conformance warning. Values follow the upstream
chart's commented example (limits 100m/512Mi, requests 10m/128Mi).
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated resource allocation configuration to define CPU and memory
parameters for system component deployments, enhancing infrastructure
stability and resource optimization.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2631)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
f2d66579f1 |
[metrics-server] Set resources requests/limits on metrics-server container (#2630)
## Summary The upstream metrics-server chart ships with `limits` commented out (only `requests cpu 100m / memory 200Mi` are set), which triggers a `no CPU limit` conformance warning on the `metrics-server` Deployment. Override `packages/system/metrics-server/values.yaml` to keep the upstream-recommended baseline `requests` and add `limits`: - `limits` : `cpu: 500m`, `memory: 500Mi` - `requests` : `cpu: 100m`, `memory: 200Mi` Comfortable headroom on top of the [official sizing formula](https://github.com/kubernetes-sigs/metrics-server#configuration) (`100m + 1m/node`, `200Mi + 2Mi/node`), covering clusters up to ~150 nodes without needing to revisit. Larger clusters can override these values further. ## Test plan - [ ] `helm template . --namespace cozy-monitoring` from `packages/system/metrics-server/` renders a `resources` block on the `metrics-server` container of the Deployment (verified locally) - [ ] Re-run the conformance/lint tool — the `metrics-server has no CPU limit` warning should be gone - [ ] CI passes ### Release note \`\`\`release-note Set CPU/memory requests and limits on the metrics-server Deployment to clear the "no CPU limit" conformance warning. Values follow the upstream sizing recommendation (requests 100m/200Mi) with a comfortable limits cap (500m/500Mi). \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Configured explicit resource requests and limits for the metrics-server service (CPU: 100m-500m, Memory: 200Mi-500Mi) to ensure consistent resource allocation. [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2630) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |