mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
8f4c19daaeb00a2aeccf761cd96879867a4d86ce
5128
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 scooby87
|
6c69c0055e |
fix(monitoring)!: clean up orphaned storage PVCs and TLS secrets on tenant module delete (#3091) (#3094)
## What this PR does
When the `monitoring` tenant module is disabled, the
VictoriaMetrics/VictoriaLogs storage PVCs (created by vm-operator) and
the `alerta-tls` / `grafana-ingress-tls` cert-manager secrets were left
orphaned in the tenant namespace.
This adds a `post-delete` cleanup hook
(`packages/extra/monitoring/templates/hooks/cleanup.yaml`) that:
- deletes the VM/VL storage PVCs scoped to this release via `-l
apps.cozystack.io/application.name={{ .Release.Name }}-system`. That
label is stamped on the `storage.volumeClaimTemplate.metadata.labels` of
the VMCluster/VLCluster CRs (`packages/system/monitoring`); vm-operator
copies claim-template labels onto the PVCs (`IntoSTSVolume`), so the
selector matches and stays scoped to this monitoring release.
`managed-by=vm-operator` is intentionally **not** used — vm-operator
does not reliably stamp it onto the storage PVCs;
- deletes the `alerta-tls` and `grafana-ingress-tls` secrets by name
(cert-manager TLS secrets carry only the cluster-wide
`controller.cert-manager.io/fao=true` label, so a label sweep would be
unsafe — they are removed by name).
CNPG PostgreSQL PVCs (`alerta-db-*`, `grafana-db-*`) are intentionally
**not** targeted — CNPG cleans those up itself, and they do not carry
`apps.cozystack.io/application.name`, so the selector leaves them
untouched.
**Upgrade path:** the claim-template label only lands on PVCs created
after the change, so **migration 50**
(`packages/core/platform/images/migrations/migrations/50`) backfills
`apps.cozystack.io/application.name` onto the pre-existing VM/VL storage
PVCs of every cozystack-managed monitoring cluster, so the cleanup
reclaims storage on already-deployed clusters too.
A `test` Makefile target and helm-unittest suites are added for both the
cleanup hook and the claim-template label. The cleanup Job is hardened:
non-root, read-only root fs, dropped capabilities, `seccompProfile:
RuntimeDefault`, resource requests/limits, `activeDeadlineSeconds: 120`,
`backoffLimit: 0`, `--wait=false` on both deletes; RBAC scoped to PVC
get/list/delete and the two named secrets. Failures are logged but never
block teardown.
> [!WARNING]
> **Disabling the monitoring module now permanently deletes its
VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they
hold.** This is intentional and fixes the storage leak. Back up before
disabling if the data matters.
Fixes #3091.
### Release note
```release-note
fix(monitoring): garbage-collect the orphaned VictoriaMetrics/VictoriaLogs storage PVCs and the alerta/grafana TLS secrets left behind when a tenant's monitoring module is disabled; a migration backfills the release label onto pre-existing storage PVCs so already-deployed clusters are covered too. WARNING: disabling monitoring now permanently removes its storage PVCs and the data they hold.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Helm **post-delete** monitoring cleanup job to best-effort
remove managed PVCs and two specific TLS secrets.
* Included dedicated cleanup RBAC and hardened pod security settings for
the cleanup job.
* **Tests**
* Added Helm chart tests validating hook weight/policies, job settings,
exact deletion commands, and negative cases.
* Added a `make test` target to run Helm unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
a28ef9ba13 |
test(e2e): make the hairpin DNS check tolerate a tenant node recycle (#3241)
## What this PR does Makes the ouroboros hairpin DNS assertion in the `kubernetes-latest` e2e deterministic against a tenant worker-node recycle. This is a widespread pre-existing flake: it fails PRs that touch nothing virt-related and passes on re-run. ### Root cause (mechanism, from a failed run's own event stream) The assertion resolves `hairpin-*.example.invalid` from inside the tenant cluster using a single `--restart=Never` netshoot Pod, then waits for that Pod to reach `Succeeded`. The tenant crust-gather snapshot of a failed run shows the pod never ran the DNS loop at all: - `09:00:02` scheduler: `Successfully assigned default/dnscheck to ...-md0-vqcs7-hhfxx` - `09:00:06` kubelet on `hhfxx`: `Pulling image "nicolaka/netshoot:v0.13"` (no subsequent `Pulled`/`Started`) - node `hhfxx` then disappears from the tenant node list; on the management cluster its Machine is gone and a replacement Machine is created at `09:02:01` (`Provisioning`), while the other worker (created `08:43:39`) survives - `09:02:47` `taint-eviction-controller`: `Cancelling deletion of Pod default/dnscheck` The tenant's CAPI `MachineHealthCheck` remediates a node whose `Ready` goes `Unknown` for just 30s by deleting its Machine, KubeVirt VM and Node and provisioning a replacement. On the memory-constrained single-node CI sandbox a worker VM losing its kubelet heartbeat briefly is routine. A bare Pod bound to the deleted node is removed by the node controller and, with `restartPolicy: Never`, is never recreated — so the probe silently vanishes while its (large) image is still pulling and it has never left `Pending`. The single-shot wait then times out and reports `dnscheck pod did not reach Succeeded phase (last seen: <empty>)`, which is indistinguishable from a real DNS-resolution failure. ### The fix Wrap the probe create-and-wait in a bounded outer deadline (7m) that recreates the Pod when it is confirmed `NotFound` — the node-recycle signature — so the check is retried on a surviving node instead of failing the whole test. An empty phase is treated as ambiguous (a transient API blip is not misread as a lost Pod); only a definitive `NotFound` triggers a recreate. A Pod that actually runs to a non-`Succeeded` terminal state, or the deadline elapsing with no successful run, still fails the test. The in-Pod DNS resolution loop that the assertion exists to exercise is unchanged, so the reconciliation coverage is preserved. This fixes the mechanism at the right layer: the check tolerates a transient node recycle of a throwaway diagnostic pod rather than reporting it as a DNS regression. It is deliberately not a timeout bump — the pod was actively deleted, not merely slow, so more time alone would not help. ### Out of scope The `kubernetes-previous` failure seen in the same run is a different mechanism: both tenant worker VMs stayed `Provisioning` for 12m (DataVolume/CDI storage-import stall on the management cluster), so no tenant node ever joined and the run fails at the existing node-join diagnostic block, well before this assertion. Both smell like the same underlying single-node-sandbox resource/storage pressure, but that is an infrastructure-capacity concern, not a harness race this change can fix. Left for a separate follow-up. ### Screenshots N/A — CI/test-harness change, no UI impact. ### Release note ```release-note test(e2e): recreate the tenant hairpin DNS probe pod on a bounded deadline so a worker-node recycle mid-check is retried instead of reported as a spurious DNS failure ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability of the Kubernetes end-to-end DNS check. * The DNS validation now retries more safely during infrastructure-related pod restarts or node churn. * Failures now surface faster with clearer diagnostic output when the check does not pass. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Kirill Ilin
|
88c0cc36ad |
feat(keycloak): add Vault Kubernetes and AppRole auth for the encryption proxy (#3239)
## What this PR does Makes the optional `keycloak-kms-proxy` PII-encryption feature production-usable. **Vault auth** — new `encryption.kms.vault.auth` switch: - `kubernetes` — **recommended**. The proxy logs in with its own dedicated ServiceAccount (`keycloak-kms-proxy`; bind the Vault role to `system:serviceaccount:<ns>:keycloak-kms-proxy`), so no credential is stored in a Secret. - `approle` — `KKP_VAULT_ROLE_ID` / `KKP_VAULT_SECRET_ID` from `encryption.kms.vault.appRole.secretName`. - `token` (default, unchanged) — Vault token from `tokenSecretName`. The proxy pod runs under a dedicated ServiceAccount (never the namespace default), with its token mounted only for Kubernetes auth. **Shared DEK set** — `encryption.deksetSecretName` → `KKP_DEKSET_FILE` (mounted read-only), and `encryption.replicas` is now configurable. Without a dekset the proxy mints an ephemeral DEK (only safe on a fresh, empty database, single replica). A shared wrapped DEK set — minted by the proxy's backfill tool — is required to migrate an existing populated Keycloak and to run HA; a guard rejects `replicas > 1` without one. Render-time validation requires each mode's settings and rejects unknown values. helm-unittest covers all auth modes, the ServiceAccount/token toggle, and the dekset/replicas paths. Uses `keycloak-kms-proxy` v0.2.0. ### Release note ```release-note feat(keycloak): Vault Kubernetes/AppRole auth, a dedicated ServiceAccount, and a shared DEK set (migration + HA) for the optional PII-encryption proxy ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Expanded Keycloak KMS proxy Vault authentication to support **token**, **AppRole**, and **Kubernetes** modes. * Added **multi-replica/HA** support via configurable DEK-set reuse, including selecting a custom DEK-set key. * Introduced a dedicated proxy ServiceAccount with **conditional token automount** for Kubernetes auth. * **Bug Fixes** * Added stricter validation for missing/unsupported Vault auth settings. * Prevented Vault credential environment variables from leaking across auth modes. * Updated rendering behavior and test expectations for proxy resources (including KEK/ServiceAccount documents). * **Chores** * Updated the Keycloak KMS proxy container image version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Алексей Артамонов
|
ff037c04d3 |
test(monitoring): reproduce the real vmselect cache PVC name in the backfill test
/branch-review flagged that the migration comment and its bats test used the simplified vmselect cache PVC name cache-vmselect-<cr>-<ordinal>, while the real name is vmselect-cachedir-vmselect-<cr>-<ordinal>. The anchored regex matches both (it suffix-matches <component>-<cr>-<ordinal>), so behaviour was correct, but the test did not exercise the production name. Feed the realistic name in the fake kubectl so the regex is validated against what vm-operator actually provisions, and reword the migration comment to describe the suffix match generically instead of asserting a specific claim-template prefix. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Алексей Артамонов
|
09c422562a |
test(monitoring): unit-test the VM/VL PVC backfill migration behaviour
helm-unittest only inspects rendered manifests, not the operator's runtime PVC labels, so it cannot prove migration 50 actually relabels the right PVCs — the exact no-op/mis-scope class of bug this PR iterated on. Add a bats test (fake kubectl on PATH, mirroring hack/kubernetes-md0-migration.bats) that pins the runtime behaviour: it relabels the monitoring cluster's vmstorage/vmselect/ vlstorage PVCs with application.name=monitoring-system, never touches the CNPG db PVCs or a user's own VMCluster, and stays a safe version-stamping no-op when the VM/VL CRDs are absent. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
f700c9043c |
fix(monitoring): backfill release label onto pre-existing VM/VL storage PVCs
Re-review on #3094 (IvanHunters, MAJOR): the volumeClaimTemplate label lands on PVCs only at creation — the StatefulSet controller never re-labels existing PVCs and vm-operator's PVC reconcile is resize-only. So on an already-deployed cluster (exactly the #3091 population) the pre-upgrade VM/VL storage PVCs stay unlabeled and the cleanup hook's selector misses them. Add migration 49 (mirrors the ClickHouse-keeper backfill in migration 48 for the sibling #3072). It relabels the pre-existing vmstorage/vmselect/vlstorage PVCs of every cozystack-managed VMCluster/VLCluster with apps.cozystack.io/application.name read from the CR's own spec.managedMetadata, so only monitoring's clusters are touched (a user's own VMCluster carries no such label) and the value always matches what the cleanup hook selects. Best-effort like #48: a skipped relabel is no worse than the pre-existing leak, so transient apiserver errors never abort the platform upgrade. Bump platform targetVersion 49 -> 50. Also correct the misleading comments (IvanHunters MINOR): spec.managedMetadata is never applied to PVCs by vm-operator at any current version, so the claim-template label is the only mechanism — do not imply a v0.71.0 crossover. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
2a78e7b247 |
fix(monitoring): match cleanup PVCs by release label alone, drop managed-by
Re-review on #3094 (IvanHunters): the AND selector kept managed-by=vm-operator, which on the shipped vm-operator v0.68.4 is not reliably present on the storage PVCs (it lives on the StatefulSet selector/pod labels, not the claim template), making the delete fragile. Match by apps.cozystack.io/application.name=<release>-system alone: that label is stamped on the VM/VL claim templates (67b087ab) and copied onto the PVCs by IntoSTSVolume even on v0.68.4, is already release-scoped, and is absent from the CNPG db PVCs (grafana-db/alerta-db carry only policy.cozystack.io + cnpg.io/*), so those stay untouched. Regression guard now forbids re-adding managed-by to the selector. Also apply the review's discipline items: the cleanup Job keeps only before-hook-creation (a Failed run survives for post-mortem; hook-succeeded stays on SA/Role/RoleBinding, cf. #3072) and sets backoffLimit: 0 (best-effort, no retry). Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
e01b725a48 |
fix(monitoring): stamp release-scoping label on VM/VL storage PVCs so cleanup matches
Re-review on #3094 (IvanHunters, lexfrei): the release-scoped selector added in ba5797f4 was a silent no-op on the shipped vm-operator v0.68.4. The apps.cozystack.io/application.name label lived only under the VMCluster/VLCluster spec.managedMetadata.labels, and v0.68.4 does not propagate managedMetadata onto PVCs (that lands only in v0.71.0). So the delete matched nothing and the #3091 storage leak stayed open. Stamp the label where it actually reaches the PVCs: on spec.{vmselect,vmstorage,vlstorage}.storage.volumeClaimTemplate.metadata.labels of the VMCluster/VLCluster CRs (packages/system/monitoring). vm-operator copies claim-template labels onto the PVCs (IntoSTSVolume) even on v0.68.4, so the existing cleanup selector (managed-by=vm-operator,apps.cozystack.io/application.name= <release>-system) now matches and stays scoped to this release. Verified by rendering the CRs (label resolves to monitoring-system, matching the hook). Also add the explicit subjects[].namespace on the cleanup RoleBinding (lexfrei non-blocking) and a helm-unittest suite asserting the label lands on all three storage claim templates. Corrected the misleading test comment about how the label reaches the PVCs. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
69c9e69ff4 |
fix(monitoring): scope VM PVC cleanup selector to this release
Address IvanHunters review on #3094: the PVC delete selector was release-agnostic (-l managed-by=vm-operator), matching the VM storage PVCs of every VMCluster/VLCluster in the namespace. If a namespace ever hosted more than one monitoring release, tearing one down would delete the other's VictoriaMetrics/VictoriaLogs storage. Scope the selector to this release via apps.cozystack.io/application.name= <release>-system, the ownership label the monitoring chart stamps on the VM CRs' managedMetadata and vm-operator propagates onto the PVCs — mirroring the merged seaweedfs hook's release scoping. deletecollection is still not needed: kubectl delete -l lists then deletes individually. Tests: assert the release-scoped selector and add a regression guard that the delete never falls back to the bare managed-by=vm-operator selector. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
b5d1279ccd |
fix(monitoring): make cleanup deletes non-blocking and guard delete scope
Address review on #3094: - Add --wait=false to both kubectl delete calls so PVC finalizers (pvc-protection on pods still terminating) can never block the post-delete hook and stall the HelmRelease teardown. - Add activeDeadlineSeconds: 120 on the cleanup Job as a backstop. - Tests: assert the non-blocking flag, plus negative assertions that the PVC delete stays label-scoped (no --all) and the secret delete stays name-scoped (no -l/--all). deletecollection RBAC verb intentionally not added: kubectl delete -l lists then deletes individually, so the existing delete verb suffices. BREAKING CHANGE: disabling monitoring now permanently removes its VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they hold. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
73937ddff9 |
fix(monitoring): clean up orphaned storage PVCs and TLS secrets on tenant module delete (#3091)
When the monitoring tenant module is disabled, the vm-operator storage PVCs (VictoriaMetrics/VictoriaLogs) and the alerta/grafana TLS secrets were left orphaned. Add a post-delete cleanup hook that deletes the PVCs by the managed-by=vm-operator label and the two TLS secrets by name, plus a test target and helm-unittest suite (the package had neither). CNPG PVCs are not targeted (CNPG cleans them). The cleanup Job is hardened (non-root, read-only root fs, dropped caps, resource limits); failures are logged, never blocking teardown. NOTE: disabling monitoring now permanently removes its storage PVCs and the data they hold. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
f9afc73096 |
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3181)
Platform log ingestion could fail silently: fluent-bit shipped to vlinsert-generic while the monitoring HelmRelease reported Ready, and a vlinsert-generic ExternalName in cozy-monitoring dead-ended at a tenant-root VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary changes: - Readiness (do not stay silently Ready): the monitoring-system HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on its VLCluster (status.updateStatus), so a missing/unhealthy log store makes the release not-ready instead of flipping Ready as soon as the CR is applied. Mirrors the seaweedfs CNPG pattern. - No empty-log-store hole: system/monitoring rejects an empty logsStorages at render time (helm fail). With no entry no VLCluster is created, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is made a loud misconfiguration rather than a documented footgun. - No silent fallback: the cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName redirects in cozystack-basics are now created only when the root tenant hosts monitoring, gated on _cluster.monitoring-enabled — fed from a new platform value monitoring.rootEnabled (default true, preserving the historical always-create behaviour). monitoring.rootEnabled is the single render-time source of truth: changing it re-renders cozystack-basics (helm-controller watches the cozystack-values Secret it flows into). A live `lookup` of Tenant/root spec.monitoring is deliberately NOT used: cozystack-basics is an operator-generated HelmRelease with drift detection off, so it re-renders only on install/upgrade/values-change — never on a reconcile tick or when a lookup result changes — and a lookup would go stale in both directions. Operators must keep monitoring.rootEnabled in sync with Tenant/root spec.monitoring. Covered by helm-unittest across all four charts: readiness gate fields, empty-vs-populated logsStorages contract, ExternalName present/absent and null-guard contract, and the platform wiring into the cozystack-values Secret. The CEL health predicates are exercised live, not in helm-unittest (which cannot evaluate CEL). Adds the missing `test:` target to packages/extra/monitoring/Makefile so its suite runs in CI. Known limitation (follow-up): the legacy cozystack.monitoring PackageSource that installs system/monitoring directly into cozy-monitoring cannot carry the readiness gate, because the engine's ComponentInstall/HelmRelease builder and the vendored helm-controller API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not referenced by any active bundle; full coverage needs a helm-controller bump plus engine support. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Kirill Ilin
|
158e85af85 |
chore(keycloak): bump encryption proxy image to 0.2.1
Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Myasnikov DaniilandClaude Fable 5
|
dea35f7432 |
Merge origin/main into feat/e2e-chainsaw-example (post-#3233)
Main's #3233 landed mid-reconciliation and touched the two OIDC bats again: nodeGroups (and storageClass) dropped from the render-side test CRs — the schema default {} now renders one zero-replica MachineDeployment via CAPI defaulting, so the tests provision zero worker DataVolumes (#3231). Resolve the modify/delete as deleted and mirror the fixture change (and its rationale) into the ported kubernetes-oidc-system / kubernetes-oidc-customconfig Chainsaw suites. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
IvanHunters
|
0c7b598304 |
fix(apps/kubernetes): honour nodeGroup minReplicas in MachineDeployment.spec.replicas (#3233)
Refs #3231 (parts 1 + 2 addressed here; part 3 — intra-file `cozy_wait_linstor_pool_free` gate + orphan `default-chk-clickhouse-test-keeper-cluster1-0-2-0` PVC cleanup — deferred to a separate follow-up PR against `hack/e2e-apps/run-kubernetes.sh`). **Do not auto-close the issue on merge** — the hardening work is a distinct concern and belongs on its own tracker entry. ## Root cause chain (recap from the issue) `packages/apps/kubernetes/templates/cluster.yaml` hardcoded `MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's `minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}` bats set `minReplicas: 0` and delete their `Kubernetes` CR without waiting for readiness — but the chart still spun up 2 × 20 GiB `replicated` (DRBD) worker `DataVolume`s each. Their create/delete churn overlapped `kubernetes-previous`, which then saw slow LINSTOR provisioning (my cozyreport from run 28843794670 shows a ~6-minute gap between the `Kubernetes` CR being created and the CDI `ExternalProvisioning` event actually firing for the worker root disk); the tenant node registered late; Cilium's CNI init missed the 12-minute node-join deadline; and the test failed with `cni plugin not initialized`. ## Changes Two edits, applied together — the chart edit is a no-op without the test edit, and vice-versa (dropping `nodeGroups` falls back to the chart helper's default `md0`, which itself has `minReplicas: 0`, so both pieces are needed for zero workers to actually render). ### 1. `packages/apps/kubernetes/templates/cluster.yaml` ```diff spec: clusterName: {{ $.Release.Name }} - replicas: 2 ``` The field is **omitted entirely** so the cluster-autoscaler owns it alone. CAPI's MachineDeployment defaulting webhook seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create (documented at https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling — *"if it's a new MachineDeployment or MachineSet, use min size"*), and Server-Side-Apply field-manager ownership then keeps chart upgrades from overwriting whatever value the autoscaler patched in the meantime. Emitting the field from the chart at all — even with the semantically-correct `{{ $group.minReplicas }}` — would keep the helm-controller ↔ cluster-autoscaler field-ownership fight alive: helm-controller applies with SSA + force-conflicts (https://fluxcd.io/flux/components/helm/helmreleases/), so every routine `helm upgrade` re-asserts the chart's chosen value and drains whatever the autoscaler had patched in the meantime. ### 2. `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats` Drop the explicit `nodeGroups:` block and the paired `storageClass:` from the render-side CR bodies. The tests only assert on the KamajiControlPlane and the cozy-realm Keycloak objects — worker plumbing was never part of their coverage. After the chart fix, the chart helper's default `md0` (with `minReplicas: 0`) renders a MachineDeployment with `spec.replicas: 0` — the resource is still emitted for API-surface completeness but never creates a KubevirtMachine, never provisions a DataVolume, and never touches DRBD. ## Verification ### helm-unittest ``` Charts: 1 passed, 1 total Test Suites: 16 passed, 16 total Tests: 171 passed, 171 total ``` No suite asserted on `MachineDeployment.spec.replicas` directly, so nothing needed updating. ### helm template — verify the omit With `tests/values/common.yaml` (which has `minReplicas: 0`), the rendered MD carries the `cluster-api-autoscaler-node-group-min-size: "0"` annotation and **no `spec.replicas` field**. CAPI's defaulting webhook will fill in `spec.replicas: 0` on Create — no worker DataVolumes provisioned until scale-out. With a user CR that sets `minReplicas: 2` (the shape the `kubernetes-latest` / `kubernetes-previous` bats CRs use), the rendered MD carries the `cluster-api-autoscaler-node-group-min-size: "2"` annotation and again **no `spec.replicas` field**. CAPI's defaulting webhook will fill in `spec.replicas: 2` on Create — byte-identical topology to the pre-PR chart's fresh-install state (2 workers), but without the recurring drain-on-upgrade cycle. ### Render-side render walk `helm template` of the OIDC bats CR bodies (after this PR): 1 MachineDeployment with **no chart-managed `spec.replicas`** and `min-size: "0"` on the autoscaler annotation. CAPI seeds `replicas: 0` on Create, 0 KubevirtMachines emitted, 0 DataVolume provisioning triggered. Test assertions on KamajiControlPlane / KeycloakClient / KeycloakClientScope / AuthenticationConfiguration Secret are unchanged. ## Upgrade impact Existing customers stop losing workers on `helm upgrade`. Prior to this PR the chart re-asserted `MachineDeployment.spec.replicas: 2` on every Helm action, so a customer running with the autoscaler at, say, `replicas: 8` would see 6 workers drained on every routine Cozystack platform bump and then wait for the autoscaler to scale back up in response to unschedulable Pods. With `spec.replicas` no longer emitted by the chart, SSA field-manager ownership preserves the autoscaler's value across upgrades unconditionally. Fresh installs with `nodeGroups: {}` (the schema default) transition from "2 workers by default" to "0 workers until scale-out". The chart helper's default md0 carries `minReplicas: 0`, which is what CAPI's defaulting webhook seeds `spec.replicas` from. The `values.yaml` / README / helper docstring / regenerated Go type doc + openAPISchema call this out explicitly; the release-note repeats it below. See finding 5 in the paired self-review for the specific `addons.ingressNginx.enabled: true` interaction operators need to be aware of on fresh installs. ## Test plan - [x] `helm unittest packages/apps/kubernetes/` — 171/171 passing. - [x] `helm template` with `tests/values/common.yaml` (minReplicas: 0) — MD.replicas: 0. - [x] `helm template` with a `minReplicas: 2` override — MD.replicas: 2 (byte-identical to pre-PR). - [ ] E2E: `kubernetes-latest` unchanged (spawns its two workers as before), `kubernetes-oidc-{system,customconfig}` render zero workers, `kubernetes-previous` no longer flakes on LINSTOR pool pressure originating from the earlier OIDC suite. ## Notes - Part 3 of #3231 (intra-file `cozy_wait_linstor_pool_free` gate between kube sub-tests and the orphan clickhouse keeper PVC cleanup) is **not** in this PR — worth a separate targeted change to `hack/e2e-apps/run-kubernetes.sh`, and it's a hardening-only improvement rather than a root-cause fix. Happy to send it as a follow-up if wanted. ## Release note ```release-note fix(apps/kubernetes): stop the chart from managing `MachineDeployment.spec.replicas`; the cluster-autoscaler owns the field alone. CAPI's defaulting webhook now seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create, and SSA field-manager ownership keeps subsequent `helm upgrade` actions from overwriting whatever value the autoscaler patched. Two behavioural changes worth calling out: - Existing customers: `helm upgrade` no longer drains workers on every platform bump — the autoscaler-held `spec.replicas` is preserved unconditionally. Prior to this change the chart hardcoded `replicas: 2`, which drained back to 2 on every reconcile action. - Fresh installs with `nodeGroups: {}` (the schema default) no longer come up with two workers by default. The chart helper's default md0 carries `minReplicas: 0`, and CAPI's defaulting webhook seeds `spec.replicas` from that. Operators enabling `addons.ingressNginx.enabled: true` on such a CR must either supply an explicit nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or accept that the cluster-autoscaler will bring the default md0 up in response to the ingress-nginx controller Pods becoming Pending on install. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Rendered `MachineDeployment` manifests no longer set `spec.replicas`, avoiding chart upgrades overwriting autoscaler-managed sizing. * **Documentation** * Clarified `nodeGroups: {}` behavior: a default `md0` is emitted with `minReplicas: 0`/`ingress-nginx` role, and workers are provisioned only after an autoscaler-triggering unschedulable Pod or manual scaling; updated related storage and ingress guidance. * **Tests** * Strengthened default node group rendering assertions and adjusted OIDC e2e manifests to reduce CI provisioning churn. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov DaniilandClaude Fable 5
|
1f9555a3c6 |
test(e2e): resolve outstanding bot review findings
- redis: assert both StatefulSet-templated PVCs Bound (replicas: 2 provisions two, only ordinal 0 was checked) - gateway: || true each delete inside the three cleanup() traps — they run under set -eu, so one transient delete failure skipped the remaining cleanup commands - bucket, backup(.disabled): guard the trap's kill against an unset pf_pid (script can exit before the port-forward starts) Not addressed on purpose: the sort(ports[*].port) suggestions on clickhouse/kafka — sort() is among the JMESPath forms chainsaw v0.2.15 (still the latest release) mis-evaluates, which is why the suites use the ordered (ports[*].port) projection; services keep spec order. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Fable 5
|
dd2f89d208 |
test(sdn): port the securitygroup and serviceexposure suites to Chainsaw
Main added securitygroup.bats (#2922 membership model) and serviceexposure.bats (#3081 ExposureClass/ServiceExposure) after the Chainsaw migration branch was cut. - securitygroup: two Tests — the aggregated-API projection round-trip (backing CNP marker label, empty-value membership endpointSelector, lineage fromApp + membership fromSG peers, 1:1 rule translation, the view hiding storage internals, unmarked-CNP invisibility via an error op, delete propagation) and the controller membership-label stamp/clear lifecycle. Only the attachments annotation stays a jq script (it is a JSON string). Chainsaw auto-cleanup covers every fixture even on mid-test failure — the bats had no teardown hook and leaked its fixtures. - serviceexposure: the externalIPs backend resolve/Ready/assignedIPs path with the no-MetalLB-pool negative as an error op (ordered after resolvedBackend so it cannot false-pass), and the finalizer-removal test collapsed onto the delete op itself, which waits for full disappearance. - selection: kubernetes-application now also maps to the two OIDC render-side suites (they exercise the same app chart — a gap main's bats selection had too); the new dirs enter all_apps discovery automatically. Selector unit tests extended and green (13/13 under cozytest). - README: suites list updated. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Fable 5
|
cac07db8fc |
test(kubernetes): port the Phase-1 OIDC render-side suites to Chainsaw
Main added kubernetes-oidc-system.bats and kubernetes-oidc-customconfig.bats after the Chainsaw migration branch was cut; without a port the 260 lines of coverage would vanish with the hack/e2e-apps/ deletion. Both are light cluster e2e (apply CR -> assert rendered objects; no Kamaji bringup, minReplicas: 0), exactly Chainsaw's declarative shape: - System mode: KamajiControlPlane carries --authentication-config + the chart-owned volume/mount; the AuthenticationConfiguration Secret carries the cozy-realm issuer (regex, script step) + per-cluster audience; per-cluster KeycloakClient/KeycloakClientScope land in the cozy realm - CustomConfig: the flag is still wired; the Secret carries the tenant-supplied issuer and NOT the cozy realm (fail-loud negation); no Keycloak objects and no pre-rendered OIDC kubeconfig Secret, expressed as error ops ordered after the positive asserts so an unreconciled HR cannot false-pass them The port also sheds two latent bats bugs: cozytest defines no skip() (the Keycloak guard would have died with exit 127 had it ever fired) and the api-resources group probe exits 0 even when the group is absent — the chart capability-gates Keycloak rendering and CI always ships the operator, so the asserts are unconditional here. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Fable 5
|
6c76a462a0 |
test(e2e): fold remaining post-branch bats drift into the Chainsaw suites
- qdrant: add the post-delete PVC-reclaim guard (#3059) — delete the CR and poll the ordinal-anchored data PVC to absence via the error op (retries transient API errors internally), with a PVC/jobs/pods dump on failure; Chainsaw auto-cleanup alone would never notice a leaked StatefulSet-templated PVC - kuberture: make the split-horizon negative log assertions fail loud — under set -e a pipeline beginning with ! suppresses errexit, so both leak checks silently passed (the same vacuous-negation bug main fixed in |
||
|
Myasnikov DaniilandClaude Fable 5
|
d6ac62a6d9 |
test(bucket,harbor): fail fast on frozen BucketClaims; enforce readonly denial
Fold main's post-branch bucket/harbor bats drift into the Chainsaw suites: - BucketClaim bucketReady budgets tightened to 2m (bucket had the 5m default, harbor a stale 10m with the reversed 'can lag on a loaded runner' rationale): the objectstorage-controller requeues the claim until the backend Bucket is ready ( |
||
|
Myasnikov DaniilandClaude Fable 5
|
21e21a9234 |
test(kafka): raise fixture presets off the 128Mi OOM threshold
Main raised kafka/zookeeper resourcesPreset nano -> c1.small
(
|
||
|
Myasnikov DaniilandClaude Fable 5
|
7b557db59f |
test(mariadb): port the single-replica webhook-regression guard to Chainsaw
Main's mariadb.bats gained "Create single-replica MariaDB" after the
Chainsaw port was cut: replication is guarded on replicas>1
(
|
||
|
Myasnikov DaniilandClaude Fable 5
|
c8e775d2d0 |
test(gateway): port the 12 ingress-hostname-policy cases to Chainsaw
Main added cozystack-ingress-hostname-policy (
|
||
|
Myasnikov DaniilandClaude Opus 4.8
|
47e7d224d8 |
test(e2e): serve Talos worker OS image from an in-sandbox cache
The kubernetes-* e2e tests fail intermittently and run 18-22 min each because every tenant worker VM imports its Talos OS disk directly from the public factory.talos.dev: that endpoint has no byte-range support (forcing CDI into a full copy-to-scratch) and its egress stream-resets / stalls mid-transfer from the CI runner, so a worker DataVolume import hangs past the chart's 12-minute node-join deadline and the test fails with zero tenant nodes joined (observed on run 28924455578, and on the unrelated branch in run 28929095889 - i.e. not specific to any one PR). Deploy a best-effort in-sandbox mirror (hack/e2e-talos-image-cache.yaml) during install: an initContainer fetches the image once with an aggressive curl retry loop, then busybox httpd serves it locally with range support. Tenant Kubernetes CRs point spec.talos.imageFactoryURL at its Service via a shared helper (hack/e2e-apps/talos-image-cache.sh) that gates on the mirror being Available and falls back to the public factory otherwise, so the mirror can only help, never make CI worse. Deployed early so the seed overlaps the Cozystack install; readiness is resolved once and cached across bats files. Uses the overridable chart value added in the previous commit. Refs #3231 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
3eb182ddf1 |
feat(apps/kubernetes): make Talos image factory URL and installer repo overridable
Tenant worker VMs boot from a Talos raw disk image that CDI streams over HTTP from the hardcoded public Talos Image Factory, and the in-guest talos-reconcile Job pulls its installer image from the same hardcoded host. Neither was overridable, so air-gapped / rate-limited / flaky-egress environments could not point workers at a mirror (the README flagged this as a Phase-2 gap), and CI had no way to avoid bulk-pulling the OS image from the public internet per worker. Add two chart values, both defaulting to the public factory so rendered output is byte-identical and behaviour is unchanged: - talos.imageFactoryURL (default https://factory.talos.dev) - base URL for the worker OS disk image streamed in by CDI. - talos.installerRepository (default factory.talos.dev/installer) - OCI repository prefix for the in-guest installer image. Regenerated values.schema.json, the aggregated-API Go types, the kubernetes-rd cozyrds schema and the README via `make generate`; added unittests for the override and trailing-slash trimming. Refs #3231 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Fable 5
|
ece64ee0a9 |
test(etcd): re-port the Chainsaw suite to the v1alpha2 operator contracts
The suite was ported from the pre-#2859 etcd.bats and asserted etcd.aenix.io/v1alpha1 objects that no longer exist on main — every Test hard-failed at the first assert on an unknown GVK. - drop the etcd-empty-backup and etcd-backup-schedule scenarios with their fixtures: the chart no longer renders EtcdBackupSchedule or etcd-s3-creds (backups moved to the generic backups API), and legacy-values tolerance is pinned by helm-unittest, not e2e - rewrite the remaining scenario into etcd-1-contracts, mirroring main's contracts test: gate on status.readyMembers=3 (Available=True flips at seed-member discovery and races replica-count contracts), assert the operator-owned member Pod shape (strict selector, container name, named metrics port), WorkloadMonitor operational/availableReplicas, and script the three contracts that need imperative probes: the /scale subresource, the plaintext-HTTP metrics probe against the Pod IP, and the on-demand defrag Job - add etcd-2-backup-roundtrip driving examples/backups/etcd/run-all.sh, gated on ETCD_E2E_S3_ROUNDTRIP=1 exactly like main (kind CI has no publicly-trusted S3 endpoint the snapshot agent could verify) - upgrade catch blocks to the v1alpha2 GVKs plus EtcdMember/ EtcdSnapshot/BackupJob describes and the snapshot-Job Pod logs - strengthen _lib/etcd-cleanup.sh with main's etcd_drain deltas: probe the Etcd CR itself (it can linger Terminating and block the next singleton apply) and use the err-sentinel pattern so a transient API error is never misread as "deleted"; add etcd_backup_cleanup wrapping examples/backups/etcd/cleanup.sh for the round-trip's teardown Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Fable 5
|
efbf372d25 |
test(e2e): fix post-merge seams the conflict markers missed
- nightly.yaml still looped over hack/e2e-apps/*.bats with the removed test-apps-% target; switch it to the single chainsaw invocation (empty CHAINSAW_SUITES = full suite) and upload the JUnit report - pin the chainsaw-report upload-artifact step to the same SHA the rest of the workflows use (scorecard/zizmor gate) - repoint hack/run-kubernetes-drain_test.bats at the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh - replicate cozytest.sh's NotReady-pod data-plane capture (hack/e2e-capture-dataplane.sh) in the Chainsaw global error.catch so chainsaw-suite failures keep diagnostic parity with the BATS lanes - refresh stale bats path comments Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Fable 5
|
9297e51170 |
Merge origin/main into feat/e2e-chainsaw-example
Reconcile the Chainsaw e2e migration with ~500 commits of main drift: - keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's _lib/ rename with the Chainsaw finally/snapshot deltas re-applied - resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted; their post-branch assertion drift is re-ported into the Chainsaw suites in follow-up commits - drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup, serviceexposure) from _lib/ relocation; ported as Chainsaw suites in follow-up commits - docs/agents/e2e-testing.md: merge main's plan-job TIA wording with the Chainsaw suite phrasing Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Kirill Ilin
|
2805180dad |
fix(keycloak): correct the encryption proxy image tag (0.2.0, no v)
The release workflow publishes the OCI tag as 0.2.0 (docker/metadata-action strips the leading v from the git tag), so the chart default of v0.2.0 does not exist and pulls fail. Pin the real tag. Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Aleksei Sviridkin
|
699e429187 |
test(e2e): make the tenant hairpin DNS check tolerate a node recycle
The ouroboros hairpin assertion resolves a name from inside the tenant cluster via a single `--restart=Never` netshoot Pod and waits for it to reach Succeeded. On the memory-constrained single-node sandbox a tenant worker node is routinely recycled mid-test: its kubelet heartbeat lapses, the tenant's CAPI MachineHealthCheck marks Ready=Unknown after 30s and deletes the Machine, its KubeVirt VM and the Node, then provisions a replacement. A bare Pod bound to the deleted node is removed by the node controller and, having restartPolicy Never, is never recreated -- so the probe silently vanishes, typically while its large image is still pulling and it has never left Pending. The single-shot wait then times out and reports "dnscheck pod did not reach Succeeded phase (last seen: <empty>)", which is indistinguishable from a real DNS failure and flakes PRs that do not touch virt at all. Keep the DNS resolution asserted exactly once and fail-fast: a probe that runs to phase Failed is the reconciliation regression this test exists to catch and fails the test immediately. Recreate only the probe Pod, and only for the pure-infrastructure node recycle -- when the Pod vanishes and the node it ran on is confirmed gone or no longer Ready, retry on a surviving node within a bounded deadline. A probe that disappears while its node is still Ready is an unexpected deletion and fails loud rather than being retried, so no pod-churn regression is masked, and a transient API error while polling is not mistaken for a deleted Pod. The in-Pod dig loop that the assertion exercises is unchanged. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Kirill Ilin
|
32986d0dc9 |
test(keycloak): cover the deksetSecretKey override
Assert KKP_DEKSET_FILE honors a non-default deksetSecretKey, closing the last uncovered branch of the dekset wiring. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Kirill Ilin
|
0b4b981fa2 |
feat(keycloak): support a shared wrapped DEK set for the encryption proxy
Wire encryption.deksetSecretName -> KKP_DEKSET_FILE (mounted read-only) and make encryption.replicas configurable. Without a dekset the proxy mints an ephemeral DEK, which only works on a fresh database and a single replica. A shared wrapped DEK set is required to migrate an existing populated Keycloak (the proxy must reuse the DEK the backfill wrapped rows with) and to run more than one replica; a guard rejects replicas>1 without one. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Kirill Ilin
|
95cdbf216d |
chore(keycloak): pin the encryption proxy to keycloak-kms-proxy v0.2.0
v0.2.0 is the first release carrying the Kubernetes/AppRole auth modes this chart wires, so encryption.kms.vault.auth is now usable with the default image. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Kirill Ilin
|
56dacf50c9 |
test(keycloak): assert automountServiceAccountToken on the ServiceAccount
Lock in the SA-object token toggle (off by default, on only for kubernetes auth), complementing the pod-level assertions. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Kirill Ilin
|
731020215a |
fix(keycloak): give the encryption proxy a dedicated ServiceAccount
Address review: the proxy pod ran under the namespace default ServiceAccount, which the keycloak StatefulSet also uses. Binding a Vault Kubernetes-auth role to default would hand the keycloak pod itself Transit access, defeating the isolation the proxy exists to provide. Add a dedicated keycloak-kms-proxy ServiceAccount, run the proxy pod under it, and mount its token only in the kubernetes auth mode (off otherwise, to minimise the credential surface). Docs name the SA to bind the Vault role to. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Kirill Ilin
|
5744516010 |
feat(keycloak): add Vault Kubernetes auth for the encryption proxy
Extend encryption.kms.vault.auth with "kubernetes" (recommended): the proxy logs in with its own ServiceAccount token, so no credential is stored in a Secret. "approle" and "token" remain. Render-time validation requires kubernetes.role and rejects unknown methods; helm-unittest covers the new path and the mutual absence of the other modes' env vars. Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Aleksei Sviridkin
|
12c3cd03f8 |
docs(vm-instance): document attaching a VM to an external VLAN (#3196)
## What this PR does Adds `docs/vm-external-vlan.md`, a guide for attaching a `vm-instance` VM directly to an external, physically-routed VLAN via a Linux bridge and a `bridge`-type NetworkAttachmentDefinition. The documented VM networking story is overlay-only (pod network + KubeOVN VPC subnets). Attaching a VM to a real VLAN — so it shares a broadcast domain with external hardware and takes an address from that VLAN's subnet — is undocumented, and the obvious `macvlan` approach silently fails with KubeVirt's default bridge binding: macvlan demultiplexes ingress by the child interface's MAC while bridge binding puts the guest's own MAC on the wire, so gateway replies reach the parent interface and are dropped before the guest. The working recipe is a Linux bridge + `bridge` CNI NAD. The guide covers why bridge and not macvlan, the host bridge, the NAD (one per tenant namespace), assigning a static guest address through cloud-init (the chart has no `networkData`), the always-present dual-homed pod NIC, and the post-recreate ARP staleness from KubeVirt regenerating the guest MAC. Docs-only; no chart or manifest changes. The recipe depends on the `bridge` CNI plugin being present in `/opt/cni/bin`; #3195 makes Cozystack's multus package stage the reference plugins there. ### Screenshots N/A — no UI changes. ### Release note ```release-note docs(vm-instance): add a guide for attaching a VM to an external VLAN via a Linux bridge and a bridge-type NetworkAttachmentDefinition ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added a new guide for connecting a `vm-instance` directly to an externally routed VLAN. * Includes setup steps for creating a bridged VLAN interface, defining a network attachment, and configuring the VM with a static IP. * Notes important networking caveats such as dual-homing behavior, MAC changes after recreation, and host-to-VM connectivity requirements. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
6f5d826777 |
docs(apps/kubernetes): correct upgrade-safety mechanism from SSA to CAPI update-time defaulting
sircthulhu's post-approval note on PR#3233: both the cluster.yaml
comment and the nodegroups_default_test.yaml comment attribute the
upgrade-safety guarantee to Server-Side-Apply field-manager
ownership, but this chart ships as a HelmRelease through Flux
helm-controller, which applies via the Helm SDK three-way merge —
not SSA. The stated conclusion is correct but for the wrong reason
and would send a future maintainer chasing a non-existent field-
manager fight when debugging drift.
Rewrite both comments around the two independent CAPI mechanisms
that actually carry the guarantee:
1. Create — the MachineDeployment defaulting webhook seeds
spec.replicas from the min-size autoscaler annotation.
2. Update — calculateMachineDeploymentReplicas preserves the
existing live value whenever the incoming spec has
replicas: nil (clamping into [min, max] on drift).
The chart guarantee reduces to "leave the field out of the rendered
manifest and let CAPI's update-time defaulting keep the live value
stable" — no field-manager ownership involved.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
Kirill Ilin
|
2b9fedc2f3 |
test(keycloak): make the AppRole/token env negative assertions bite
The notContains checks matched whole array elements, so a name-only content never equaled the valueFrom entry and the assertion could never fail. Add any:true so "AppRole stores no static token" and, symmetrically, "token mode renders no AppRole vars" are actually verified (confirmed by mutation-testing the template). Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
IvanHunters
|
e23d89da03 |
test(apps/kubernetes): assert autoscaler max-size annotation on nonzero-minReplicas case
Symmetry with the zero-minReplicas neighbour, which already covers both min-size and max-size. CAPI's calculateMachineDeploymentReplicas defaults to 1 (not to minSize) if either annotation is missing on a new MachineDeployment, so both keys are load-bearing on the Create-path guarantee and worth asserting in both regression cases. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
6ac52a108a |
docs(e2e): correct OIDC bats comments to match post-fix render behaviour
Both nodeGroups-omitted comments overstated the outcome as "zero
MachineDeployments render", but the chart helper (kubernetes.nodeGroups)
emits a default md0 group whenever the caller's nodeGroups map is
empty, so exactly one MachineDeployment renders. What is actually
zero is the number of KubevirtMachines / worker DataVolumes, because
the chart no longer manages spec.replicas and CAPI's defaulting
webhook seeds it to 0 from the autoscaler min-size annotation.
kubernetes-oidc-system.bats additionally described the bug in the
present tense ("chart hardcodes spec.replicas: 2 ... see #3231"),
which contradicts the sibling customconfig comment and would read
as an outstanding defect post-merge — rewrite it around the post-fix
rationale to match customconfig's shape.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
Kirill Ilin
|
31762644c9 |
feat(keycloak): add Vault AppRole auth for the encryption proxy
The keycloak-kms-proxy Vault Transit backend only supported a static token. Add an encryption.kms.vault.auth switch — "token" (default, unchanged) or "approle" — wiring KKP_VAULT_ROLE_ID / KKP_VAULT_SECRET_ID / KKP_VAULT_APPROLE_MOUNT from a Secret. With AppRole the proxy logs in and re-authenticates on demand, so no long-lived token is stored. Requires a keycloak-kms-proxy image with AppRole support (> v0.1.0). Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Timur Tukaev
|
83cc800f8a |
docs(readme): add OpenSSF Scorecard badge (#3237)
## What this PR does Adds the OpenSSF Scorecard badge to the README, next to the existing OpenSSF Best Practices badge. The `scorecard.yml` workflow already publishes results to the public securityscorecards.dev dataset, so the badge surfaces the project's live supply-chain security score directly in the README. ### Release note ```release-note docs(readme): add OpenSSF Scorecard badge ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Added an OpenSSF Scorecard badge to the README, linking to the repository’s security scorecard page. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
f0e0cec64c |
ci: harden GitHub Actions workflows for OpenSSF Scorecard (pin, permissions, zizmor gate) (#3223)
## What this PR does Restores the two OpenSSF Scorecard checks that dropped the aggregate score below our target, and adds a durable gate so they cannot silently regress again. **Root cause.** 15 of 17 workflows already declare a read-only top-level token and pin their actions by SHA. Two did not — `build-main.yaml` and `ui-test.yaml` (the latter added by the in-tree console move) — and that was enough to zero out two checks: - **Token-Permissions** is scored on the weakest workflow, so a single workflow with an undeclared (default-write) top-level token drops the whole check to `0`. - **Pinned-Dependencies** counted the six tag-pinned action refs as unpinned. **Fixes** - `build-main.yaml`, `ui-test.yaml`: add a top-level read-only `permissions` block (job-level `packages: write` kept where the build cache needs it) and pin every action to the SHA already used elsewhere in the tree. - `.github/zizmor.yml` + `.github/workflows/zizmor.yml` + `.pre-commit-config.yaml`: add [zizmor](https://docs.zizmor.sh/) as a gate. It runs on the PR diff (pre-commit) and tree-wide on every workflow-touching PR, failing on any reintroduced unpinned action or missing/overly-broad permissions block. Scope is intentionally limited to the two Scorecard-relevant audits (`unpinned-uses`, `excessive-permissions`), which are clean tree-wide as of this PR; the remaining audits carry pre-existing debt and are disabled with a documented TODO to burn them down and re-enable one at a time. - `scorecard.yml`: wire an optional `SCORECARD_TOKEN` secret. The default `GITHUB_TOKEN` cannot read branch-protection settings, so the Branch-Protection check is currently scored on partial data. The token is a no-op until the secret is added by a maintainer. - `internal/fluxshardoperator/config_fuzz_test.go`: native Go fuzz targets for the shard config parsers (round-trip and range invariants, not just no-panic), giving the Scorecard Fuzzing check a real signal on a hot-path parser. **Verification** - `zizmor --offline .github/workflows/` → *No findings to report* (gate green). - `go test ./internal/fluxshardoperator/ -run '^Fuzz'` passes; a 15s live fuzz of `FuzzParsePinnedTenants` ran 10.5k execs with no crashers. Follow-ups (tracked separately, not in this PR): merge Renovate's digest-pinning PRs to lift Pinned-Dependencies further, tighten branch protection on `main` and release branches, upgrade the OpenSSF Best Practices badge, and sign releases. ### Release note ```release-note ci: pin GitHub Actions by SHA and scope workflow token permissions to restore the OpenSSF Scorecard Token-Permissions and Pinned-Dependencies checks, and add a zizmor gate (pre-commit + PR workflow) that blocks future regressions ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added automated security scanning for workflow changes via a dedicated CI workflow, along with a matching pre-commit hook. * **Bug Fixes** * Hardened CI workflows with workflow-level read-only token permissions and pinned GitHub Action versions for more reliable execution. * Improved OpenSSF Scorecard configuration to better handle branch-protection checks when a token secret is available. * **Tests** * Added Go fuzz tests to ensure shard parsing/formatting remains non-panicking and preserves round-trip/contract behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Timur Tukaev
|
6a58962eab |
docs(readme): add OpenSSF Scorecard badge
Adds the OpenSSF Scorecard badge next to the existing OpenSSF Best Practices badge. The scorecard.yml workflow already publishes results to the public securityscorecards.dev dataset, so the badge reflects the project's live supply-chain security score. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timur Tukaev <timur.tukaev@aenix.io> |
||
|
Andrei Kvapil
|
8ae9009270 |
chore(linstor): bump linstor-server to 1.33.3 and linstor-csi to v1.11.2 (#2987)
## What this PR does Bumps the LINSTOR storage stack to current upstream, picking up the published security fixes the issue enumerates: - `linstor-server` (piraeus-server) `1.33.2 → 1.33.3` (latest 1.33 patch; v1.34.0 is rc-only). Pulls in the 1.33.3 DRBD fixes (clone-status check before connection states arrive; the secondary-after-mkfs path is now guarded upstream too) plus refreshed Debian packages. Note: 1.33.3 also disables DRBD volume shrink upstream (it never worked correctly) — no effect on cozystack, which does not expose volume shrink. - `linstor-csi` `v1.10.6 → v1.11.2`. Its go.mod requires Go ≥ 1.26, so the CSI builder moves `golang:1.25 → golang:1.26` — the right lever for the Go stdlib / `moby/spdystream` CVEs. Both cozystack images are rebuilt multi-arch (linux/amd64 + linux/arm64) and pinned by digest in `values.yaml`. All five piraeus-server patches re-apply cleanly against v1.33.3 in the Dockerfile glob order, and both linstor-csi patches re-apply against v1.11.2. Adds helm-unittest coverage (the package shipped none) asserting the controller / CSI / satellite images stay digest-pinned (`<image>:<tag>@sha256:<64-hex>`), version-agnostic so it survives the release-prepare re-tag. **Verified on a dev cluster:** a rolling restart of all satellites, controller, and CSI to 1.33.3 / v1.11.2 kept every node Online with zero faulty resources; a replicated PVC provisioned, mkfs'd, and reached UpToDate across 3 nodes with no DRBD "device held open" abort (the cozystack secondary-after-mkfs patch handled the path). The csi v1.11.2 "no RWX without DRBD layer" change is a no-op for cozystack, which always uses the DRBD layer. Closes #2881 ### Release note ```release-note chore(linstor): bump linstor-server to 1.33.3 and linstor-csi to v1.11.2 Upgrade impact: an empty tenant Kubernetes worker `nodeGroups[*].storageClass` now defaults to the application-level (DRBD `replicated`) storageClass instead of the management-cluster default, because linstor-csi v1.11.2 rejects ReadWriteMany volumes on a non-DRBD storage class. This changes the rendered worker template, so existing tenant clusters that relied on the empty-storageClass default roll their worker MachineDeployments once on the first reconcile after upgrade — workers are reprovisioned on the application storageClass and container images are re-pulled. The per-nodeGroup override and the both-empty cluster-default escape hatch are preserved. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated LINSTOR Server to **v1.33.3** and LINSTOR CSI to **v1.11.2** (image tags/digest pins refreshed) * Updated the LINSTOR CSI build toolchain to **Go 1.26** * Added a **test** make target to run Helm unit tests * **Documentation** * Updated the LINSTOR Server patches README to reference **v1.33.3** * **Tests** * Added an image-pinning validation suite to confirm controller and satellite images use **sha256** digests <!-- end of auto-generated comment: release notes by coderabbit.ai --> ## Worker-disk storageClass default (companion to the bump) linstor-csi v1.11.2 rejects ReadWriteMany volumes that are not on a DRBD-layer storage class. Worker VMs live-migrate, so their disks need RWX — the access mode comes from the chosen StorageClass's CDI StorageProfile (a DRBD class yields RWX), it is not set on the worker DataVolume by the chart. With the bump, a worker nodeGroup left on a non-DRBD cluster-default StorageClass would fail provisioning, so an empty per-nodeGroup storageClass now defaults to the application storageClass (replicated/DRBD) instead of the cluster default. The per-nodeGroup override and the both-empty cluster-default escape-hatch are preserved. Upgrade impact: the empty-storageClass default ships on `md0`, and the KubevirtMachineTemplate name is content-addressed over the rendered worker body, so this flip renames the template and triggers a one-time CAPI rolling replacement of worker pools on the first reconcile after upgrade. Two hash-pin tests guard the rename so it stays an intentional, reviewed change. linstor-csi v1.11.2 also defaults DRBD two-primaries to kubevirt-only — a no-op for cozystack, where kubevirt is the target. |
||
|
IvanHunters
|
b3aac80830 |
fix(apps/kubernetes): omit chart-managed spec.replicas so cluster-autoscaler owns the field alone
Structural follow-up to the previous commit's `replicas: {{ $group.minReplicas }}`.
Emitting the field from the chart at all — even with the semantically-
correct value — keeps the helm-controller ↔ cluster-autoscaler field-
ownership fight alive: helm-controller applies with SSA + force-conflicts
(https://fluxcd.io/flux/components/helm/helmreleases/), so every routine
`helm upgrade` action re-asserts `replicas: {{ $group.minReplicas }}` and
drains whatever the autoscaler had patched in the meantime, with a real
disruption window on running clusters.
Drop the field from the manifest entirely. CAPI's MachineDeployment
defaulting webhook seeds `spec.replicas` from the
`cluster-api-autoscaler-node-group-min-size` annotation on Create (see
https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling),
so fresh installs still start with the intended minimum. From that
point on the autoscaler owns the field via SSA field-manager tracking
and chart apply never touches it again.
Behavioural summary:
* Fresh install with `minReplicas: 0` → CAPI defaults `replicas: 0`
on Create → no worker DataVolumes provisioned until scale-out.
* Fresh install with `minReplicas: 2` → CAPI defaults `replicas: 2`
on Create → byte-identical to pre-#3231 behaviour.
* Existing customer (any minReplicas) → next chart upgrade does not
touch spec.replicas — the autoscaler-held value is preserved
unconditionally.
Regression tests updated to assert:
* `notExists: spec.replicas` on the emitted MachineDeployment (the
exact contract this commit establishes).
* Autoscaler `min-size` / `max-size` annotations continue to carry
the nodeGroup's declared bounds (so a future refactor cannot
accidentally drop the wiring the defaulting webhook depends on).
helm-unittest suite stays at 173/173.
Ref: cozystack/cozystack#3231.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
|
||
|
IvanHunters
|
d30a2cfffb |
docs+test(apps/kubernetes): align nodeGroups default docs with actual zero-worker topology + regression-guard the replicas fix
Follow-ups from the self-review of #3233: - `values.yaml` / helper docstring / regenerated README + schema + cozyrds openAPISchema + Go type doc: the previous "one ingress-nginx worker" contract was already misleading (helper default carried `minReplicas: 0` but the chart hardcoded `spec.replicas: 2`), and the replicas fix makes the docstring strictly false — an empty-nodeGroups install now provisions zero workers until either the cluster-autoscaler responds to an unschedulable Pod or the operator explicitly scales the group. Update the docstrings to describe the actual behaviour and to spell out the ingress-nginx-addon implication so operators are not surprised at addon-install time. - `tests/nodegroups_default_test.yaml`: two regression cases pinning `MachineDeployment.spec.replicas == $group.minReplicas` for `minReplicas: 0` (the render-side OIDC bats path — zero worker DataVolumes provisioned) and for `minReplicas: 2` (the kubernetes-latest / kubernetes-previous bats path — byte- identical to pre-PR behaviour). Without these a future refactor can silently re-hardcode `replicas: 2` and turn no tests red. helm-unittest suite grows from 171 to 173, both new cases pass. Ref: cozystack/cozystack#3231. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
myasnikovdaniil
|
cece5c23b5 |
feat(release): immutable tags & rc→stable promotion (#2677) (#3017)
## What this PR does Implements the immutable-tag + rc-promotion flow from #2677. A **stable release becomes a renamed release-candidate**: the bytes shipped as `vX.Y.Z` are bit-for-bit the bytes built and e2e-tested as `vX.Y.Z-rc.N`. No tag is ever force-moved, and stable is never rebuilt — it is *promoted* by retagging the rc's existing images. Layered onto the build matrix from #2937/#2983 (this PR is **stacked on `refactor/build-matrix-2937`** and must merge after it). ## The five force-retag sites, removed | Site | Before | After | |---|---|---| | `tags.yaml` api/apps/v1alpha1 tag | `git tag -f` / `push -f` | write-once (create-if-absent, fail if it would move) | | `tags.yaml` release-X.Y.Z branch | `git branch -f` / `push -f` | compare-before-force (no-op if unchanged; staging branch only) | | `pull-requests-release.yaml` stable tag | `git tag -f` / `push -f` | write-once at the PR merge commit (force impossible by construction) | | `pull-requests-release.yaml` maintenance branch | `updateRef force:true` | fast-forward-only | | `auto-release.yaml` patch tags | delete-recreate (cron) | **workflow deleted** — stable only via explicit promote | ## Version decoupling (the enabler) The operator baked its version into the image at build time, so an rc image self-reported the rc string — blocking retag-promotion. It now reads `COZYSTACK_VERSION` from the environment (threaded via `cozystackOperator.platformVersion` → Deployment env, stamped by `make manifests`), falling back to the build-time value when unset. The same image bits can report any release name. The only runtime reader is the telemetry metric `cozy_cluster_info{cozystack_version=...}`. ## Promotion flow `promote-rc.yaml` (`workflow_dispatch`, `rc_tag=vX.Y.Z-rc.N`): 1. Validate the rc release exists and the stable tag does not. 2. `hack/promote-retag.sh` reads the rc's digest-pinned image refs from `packages/*/*/values.yaml` and `skopeo copy`s each — **by digest** — to `:vX.Y.Z` and `:latest`, verifying with `skopeo inspect`. 3. Rewrite the cosmetic `-rc.N` substring in vendored tags to the stable version (the `@sha256` wins regardless), restamp the version-stamped assets (operator manifests, cozypkg, openapi; the heavy Talos assets are copied verbatim from the rc draft), open a `release-X.Y.Z` PR. 4. Merging the PR reuses the existing release-PR e2e and `pull-requests-release.yaml` finalize to cut the **write-once** stable tag at the merge commit and publish the release. Squash is disallowed (the tag needs a real merge commit). `nightly.yaml` cuts write-once `*-nightly.<date>` tags (gated by `NIGHTLY_ENABLED`); `retention.yaml` keeps the newest 14 per line (dry-run by default). ## Validation **Local end-to-end — the core claim is proven, not asserted.** The whole design rests on "copy-by-digest to a new tag preserves the digest, so stable == the e2e-tested rc, bit-for-bit." This was exercised against a real registry (`ttl.sh`) with real `skopeo`: - Pushed a multi-arch image under a `:v1.4.0-rc.2` tag, built a temp tree exercising all three `values.yaml` digest shapes (single `image:` string, split `repository`/`tag`/`digest` map, and the `platformSourceRef` OCI artifact), then ran the actual `hack/promote-retag.sh v1.4.0`. - **Result: `:v1.4.0-rc.2`, `:v1.4.0`, and `:latest` all resolve to the identical digest** (`sha256:fd8d9aa6…`), across all 17 platform manifests. The script's own `skopeo inspect` post-check passed, and an independent re-inspection confirmed it. Local validation caught (and this PR fixes) two bugs that only surface against a real registry — never in static lint: 1. `skopeo copy --multi-arch all --all` — mutually exclusive flags, fatal error; every retag would have failed on CI. Now `--multi-arch all`. 2. The same digest was retagged twice when an image appeared in two value shapes; the ref set is now deduped on the canonical `repo@digest`. **Other local checks:** `go test ./pkg/version` + `go build`/`vet`; `helm template` renders `COZYSTACK_VERSION` in all 3 operator variants (omitted when unset); `make manifests` stamps the version into the install assets; `shellcheck` clean on `promote-retag.sh`; `actionlint` clean and `act -l` resolves the job graph on all workflows; the rc-tag parser and nightly version-math unit-tested (accept/reject + minor-vs-patch bumps). **Still CI-only (cannot be exercised offline):** the full workflow runtime — github-script API calls, OCIR auth, the rc-release / staging-branch preconditions, the nightly `base_ref` push mechanics, and self-hosted runners. The *logic* inside the steps is unit-tested; the *orchestration* is not. Treat it as unproven until the workflows run. ## Out of scope / required follow-up (needs repo admin — not doable from a PR) - Tag-protection rules: `v*` and `api/apps/v1alpha1/*` create-only by the CI app, no delete/update; "limit branches/tags updated in a single push" = 1. - Enable `NIGHTLY_ENABLED` (and `RETENTION_APPLY` when ready) repo variables. - Registry-side pruning of `*-nightly.*` image tags (no OCIR delete parity yet — tracked TODO in `retention.yaml`). ## Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added nightly publishing (mirror-by-digest, disk build, e2e validation) and nightly retention pruning. * Added RC-to-stable promotion via digest retagging (no rebuild). * Operator/installer and console now expose version/platform metadata, including runtime override via `COZYSTACK_VERSION`. * **Bug Fixes** * Enforced write-once tag behavior and fast-forward-only maintenance updates across release workflows. * Made nightly mirroring/selection and retention pruning more selective and safer. * **Tests** * Added Bats coverage for nightly mirroring and RC retagging, plus new Helm/unit checks. * **Documentation** * Updated the release model to center RC promotion and tag immutability. <!-- end of auto-generated comment: release notes by coderabbit.ai -->v1.6.0-rc.1 |
||
|
IvanHunters
|
6fd98dc510 |
fix(apps/kubernetes): honour nodeGroup minReplicas + drop nodeGroups from render-side OIDC tests
Closes #3231. `packages/apps/kubernetes/templates/cluster.yaml` hardcoded `MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's `minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}` bats — which set `minReplicas: 0` and delete their `Kubernetes` CR without waiting for readiness — still provisioned two 20 GiB `replicated` (DRBD) worker `DataVolume`s each. Their create/delete churn overlapped the very next test, `kubernetes-previous`, and put transient pressure on a LINSTOR satellite's ZFS pool: worker root disks landed late, tenant nodes registered late, Cilium's CNI init missed the 12-minute node-join deadline, and the test failed with `cni plugin not initialized`. Two changes, applied together — the chart fix is a no-op without the test fix, and the test fix is a no-op without the chart fix, because dropping `nodeGroups` falls back to the chart helper's default `md0` (with `minReplicas: 0`). **Chart**: `spec.replicas: {{ $group.minReplicas }}` so the field matches the autoscaler annotations already emitted just above. `minReplicas: 0` groups now genuinely scale from zero and do not silently provision workers; groups with `minReplicas >= 2` (the kubernetes-latest / kubernetes-previous bats CRs, the tests/values/ common.yaml default, the values.yaml default that gets templated into the helper's `md0`) render byte-identical to before. **Render-side OIDC tests**: drop the explicit `nodeGroups:` block and `storageClass:` — falling back to the chart helper's default `md0` (now with an effective `replicas: 0` after the chart change) means the MachineDeployment renders but never creates a KubevirtMachine and no DataVolume is provisioned. The tests only assert on the KamajiControlPlane and the cozy-realm Keycloak objects, so removing worker plumbing does not narrow their coverage. Upgrade impact: any existing customer CR that already carries `nodeGroups.md0.minReplicas: 0` (rare — the default in tests/values and the chart helper both use it, but real user CRs almost always have `minReplicas >= 1`) will see the MD's `spec.replicas` reconcile from `2` down to `0` on the next chart apply. If the cluster- autoscaler is enabled it will scale back up as soon as an unschedulable pod appears; if it is not, the group intentionally becomes empty (which is what `minReplicas: 0` requests literally). This matches the semantics the autoscaler annotations were already advertising — the hardcoded `2` was contradicting them. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |