mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-10 06:24:09 +09:00
ffeb8e5e4f43d287c6539cc1cea26e055ccad973
3077
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ivan Okhotnikov
|
ffeb8e5e4f |
fix(monitoring): close self-review blockers for the OIDC PR
Address the two blocking-class findings from the cozy-review pass on
this PR:
(1) Missing PackageSource dependency on cozystack.keycloak-operator
(MAJOR cold-install race).
Without this edge, the monitoring chart can reconcile before the
v1.edp.epam.com CRDs are registered on a fresh cluster — the
`.Capabilities.APIVersions.Has` guard in
templates/grafana/oidc-keycloak.yaml then silently drops all
Keycloak-side objects for mode: System, and Grafana's
auth.generic_oauth block points at a client Keycloak never
provisions. Add the ordering edge, plus a
sources_monitoring_application_dependson_test.yaml regression
guard so a future edit cannot drop it silently.
(2) allow_assign_grafana_admin driven by `.Release.Name ==
monitoring-system` (MINOR fragile coupling).
Replace the release-name sniff with an explicit
`oidc.grafanaAdmin: false` values field, and flip it on for the
platform Grafana release via
Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin
in packages/core/platform/templates/bundles/system.yaml —
mirroring the pattern monitoring-agents already uses for
per-component overrides. Tenant Grafana releases inherit the
chart default false and stay at org-level Admin.
Two new helm-unittest cases in packages/system/monitoring/tests/
lock the values-driven behaviour on both sides
(grafanaAdmin=true → Admin uplift, default → tenant stays at org
Admin), plus bundles_monitoring_grafana_admin_test.yaml in
packages/core/platform/tests/ regression-guards the Package's
component override so the platform release keeps its uplift on
future edits.
Also document the mode-toggle group-membership loss in
docs/oidc-grafana.md — mode: System → CustomConfig/None deletes the
three chart-owned KeycloakRealmGroups; membership is not restored on
toggle-back. Not a bug, but the operator UX cost of the toggle
deserves a bullet in Failure modes.
Regenerated artifacts (cozyvalues-gen + hack/update-crd.sh):
packages/extra/monitoring/values.schema.json
packages/extra/monitoring/README.md
packages/system/monitoring-rd/cozyrds/monitoring.yaml
Test suites still green: packages/system/monitoring 23/23,
packages/core/platform 81/81.
Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
|
||
|
Ivan Okhotnikov
|
4fe85d039c |
chore(monitoring): move OIDC schema source to packages/extra + regenerate
The Monitoring public API surface (openAPISchema in monitoring-rd
cozyrds) is generated from packages/extra/monitoring/values.yaml by
cozyvalues-gen + hack/update-crd.sh — the foundation commit
(186239044) instead wrote it by hand in
packages/system/monitoring-rd/cozyrds/monitoring.yaml, which the
pre-commit hook then reverts against every push.
Move the oidc annotations into packages/extra/monitoring/values.yaml
(matching the pattern used by the rest of the top-level fields) and
run `make generate` in that directory. That regenerates:
- packages/extra/monitoring/values.schema.json
- packages/extra/monitoring/README.md
- packages/system/monitoring-rd/cozyrds/monitoring.yaml
(openAPISchema + keysOrder, both including the new oidc keys)
No behaviour change — the values.yaml under packages/system/monitoring
still carries the same oidc defaults so `helm template` works
stand-alone, and helm-unittest keeps passing (22/22).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
Ivan Okhotnikov
|
1f44bc06cf |
test(monitoring): helm-unittest coverage for the OIDC selector
15 cases across the three modes plus the two fail-close guards: - mode: None emits nothing OIDC-related — no cozy-realm objects, no client-secret Secret, no auth.generic_oauth key on the Grafana CR, no GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET / GF_PATHS_CUSTOM_INI env. - mode: System renders KeycloakClient + KeycloakClientScope + three KeycloakRealmGroups + persistent client-secret Secret, and pins the Grafana spec.config to the cozy realm with a role_attribute_path that maps the -admin/-editor/-viewer groups to Grafana roles. - mode: System fails the render when the platform-level OIDC feature is off, via the assertSystemEnabled helper. - mode: CustomConfig inline writes the operator-supplied map verbatim into spec.config.auth.generic_oauth without touching cozy. - mode: CustomConfig secretRef mounts the operator Secret under /etc/grafana/oidc + wires GF_PATHS_CUSTOM_INI, and does NOT emit auth.generic_oauth under spec.config (the ini fragment is authoritative). - CustomConfig branch fails-close on both/neither of the two payloads via the assertCustomConfigXor helper. - Platform release (Release.Name == monitoring-system) upgrades the Admin role to server-level GrafanaAdmin through the allow_assign_grafana_admin flag; tenant Grafana stays at org-level Admin. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
8eff03b323 |
feat(monitoring): wire auth.generic_oauth into the Grafana CR
Patch the Grafana CR so that the oidc.mode selector drives the actual Grafana runtime config: - mode: System — emit a full auth.generic_oauth section under spec.config with client_id / client_secret / auth_url / token_url / api_url / scopes / role_attribute_path / allow_assign_grafana_admin built from the helpers, and inject GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET as an env from the persistent Secret introduced in the previous commit. role_attribute_path maps membership in the per-instance -admin / -editor / -viewer groups to Grafana's Admin / Editor / Viewer roles, defaulting to Viewer on authenticated identities with none of them. allow_assign_grafana_admin is set to true only for the platform release (Release.Name == 'monitoring-system'); tenant Grafana stays at org-level Admin. - mode: CustomConfig (inline) — pass the operator-supplied config map verbatim to spec.config.auth.generic_oauth. `cozy` is not in the path. - mode: CustomConfig (secretRef) — mount the operator-managed Secret under /etc/grafana/oidc and load it via GF_PATHS_CUSTOM_INI. The ini fragment is authoritative; the chart does not emit auth.generic_oauth under spec.config in this branch. - mode: None (default) — no change to the rendered manifests. CustomConfig branch fails the render at the XOR helper if both or neither of `config` / `secretRef.name` are set. `disable_login_form` stays false so the admin_user/admin_password Secret remains a documented break-glass path in every mode. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
473b112f66 |
feat(monitoring): persist confidential OIDC client secret for System mode
New oidc-client-secret.yaml renders a Kubernetes Secret with the confidential client_secret used by both the EDP-Keycloak operator (via \$<secret>:client-secret from oidc-keycloak.yaml) and the Grafana pod (via GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET, wired by the next commit). Value is generated at first install (randAlphaNum 32) and preserved across upgrades through the same \`lookup\` pattern used by packages/system/dashboard. secret-hash annotation surfaces the value change to whichever controller is watching, ensuring KeycloakClient + Grafana Deployment re-reconcile if it ever rotates. Rendered only for mode: System — CustomConfig operators bring their own credentials (inline plaintext or a Secret they manage out-of-band). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
9dd9f1d4dd |
feat(monitoring): render Keycloak client + scope + groups for System mode
New oidc-keycloak.yaml template mirrors the tenant kube-apiserver Phase-1 shape (cozystack/cozystack#3044) with three Grafana-specific adjustments: - Client is confidential (public: false, directAccess: false); Grafana is server-side so a client_secret is required. The value is wired via the EDP-Keycloak `$<secret>:<key>` reference; the backing Secret is created by the next commit. - redirectUri is https://grafana.<host>/login/generic_oauth — the Grafana generic_oauth callback path. - Three KeycloakRealmGroups (`<clientId>-{admin,editor,viewer}`) are chart-owned. Membership is set by an operator through the Keycloak UI or a KeycloakRealmUser CR; grafana.yaml will map group membership to Admin / Editor / Viewer via role_attribute_path. Guarded by mode == System AND _cluster.oidc-enabled == true (via the assertSystemEnabled helper). Renders nothing for None or CustomConfig. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
e12bb1a3ba |
feat(monitoring): scaffold OIDC selector on the Grafana instance
Add the plumbing needed for a Phase-1 OIDC integration on Grafana, matching the shape of cozystack/cozystack#3044 for the tenant kube-apiserver: - values.yaml grows a top-level oidc block with a three-value mode enum (None/System/CustomConfig) and a customConfig payload with the XOR-inline/secretRef surface. - monitoring-rd advertises the new selector via its openAPISchema and keysOrder so operators see it on the Monitoring CR. - templates/_helpers.tpl introduces the naming helpers reused by every downstream OIDC template: clientId, audienceScopeName, clientSecretName, grafanaHost, redirectUri, systemIssuerURL, roleAttributePath, allowAssignGrafanaAdmin (platform-only), assertSystemEnabled, assertCustomConfigXor. No behaviour change on default values — mode: None short-circuits and no OIDC objects are rendered. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
ce4b263ca6 |
fix(kubernetes): re-pin deleted centos:stream9 digest for kubevirt-csi-driver
quay.io/centos/centos:stream9 is a rolling tag; upstream deleted the pinned digest (sha256:0c9db821...) when it published a new stream9 build, so every build of image-kubevirt-csi-driver fails with "not found" — on open PRs and on main. Re-pin to the current live stream9 digest (sha256:a93e3316...). This is an emergency CI unblock, not a durable fix: stream9 is garbage-collected, so upstream will delete this digest too on its next rebuild and the same "not found" breakage recurs until the pin is bumped again — the same operation Renovate's weekly, non-automerged digest PR performs. A durable fix (mirror the base image into cozystack's own registry, or move to an immutable-digest base such as UBI9) is out of scope for this hotfix and left as follow-up. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
myasnikovdaniil
|
77389d6e99 |
chore(deps): update github.com/kubevirt/csi-driver digest to 27b52aa (#3066)
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/kubevirt/csi-driver](https://redirect.github.com/kubevirt/csi-driver) | replace | digest | `51c6bca` → `27b52aa` | --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/cozystack/cozystack). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI1OS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0ZWQiLCJkby1ub3QtbWVyZ2UvaG9sZCJdfQ==--> |
||
|
myasnikovdaniil
|
1d1c7c2c38 |
chore(deps): update altinity/clickhouse-backup docker tag to v2.7.4 (#3226)
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [altinity/clickhouse-backup](https://redirect.github.com/Altinity/clickhouse-backup) | final | patch | `2.7.2` → `2.7.4` | --- ### Release Notes <details> <summary>Altinity/clickhouse-backup (altinity/clickhouse-backup)</summary> ### [`v2.7.4`](https://redirect.github.com/Altinity/clickhouse-backup/blob/HEAD/ChangeLog.md#v274) [Compare Source](https://redirect.github.com/Altinity/clickhouse-backup/compare/v2.7.3...v2.7.4) NEW FEATURES - add `clickhouse.parts_columns_max_bytes_before_external_group_by` (env `CLICKHOUSE_PARTS_COLUMNS_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY`, default `100000000`) and `clickhouse.parts_columns_max_memory_usage` (env `CLICKHOUSE_PARTS_COLUMNS_MAX_MEMORY_USAGE`, default `200000000`) — make the memory caps injected into the `system.parts_columns` check query configurable instead of hardcoded, `0` means don't inject the corresponding setting (pre-2.7.2 behavior), fix [#​1420](https://redirect.github.com/Altinity/clickhouse-backup/issues/1420) BUG FIXES - fix `--restore-database-mapping`/`--restore-table-mapping` losing unrelated comma-separated `--tables` pattern items, or applying the mapping to the wrong item when the same source database/table name was repeated across items — each comma-separated pattern item is now matched and substituted independently, fix [#​1421](https://redirect.github.com/Altinity/clickhouse-backup/issues/1421) ### [`v2.7.3`](https://redirect.github.com/Altinity/clickhouse-backup/blob/HEAD/ChangeLog.md#v273) [Compare Source](https://redirect.github.com/Altinity/clickhouse-backup/compare/v2.7.2...v2.7.3) NEW FEATURES - add `general.max_broken_part_ratio` (env `MAX_BROKEN_PART_RATIO`, default `0`) — when >0, `create`/`create_remote` skips broken data parts instead of aborting the whole backup as long as the broken/total part ratio stays at or below the configured fraction; skipped parts are logged as warnings and the backup is marked successful but partial; `0` preserves the legacy any-broken-part-fails behavior, fix [#​1418](https://redirect.github.com/Altinity/clickhouse-backup/issues/1418) - add `clickhouse.rebind_replica_path_if_exists` (env `CLICKHOUSE_REBIND_REPLICA_PATH_IF_EXISTS`, CLI `--rebind-replica-path-if-exists`, also accepted by the REST API `POST /backup/restore` endpoint) — opt-in rebind of the replicated table ZK path to `default_replica_path` during `restore` when the resolved path already has leftover children but no local table uses it (async-stale state after DROP); MUST stay `false` during a concurrent multi-replica restore to avoid split-brain, fix [#​849](https://redirect.github.com/Altinity/clickhouse-backup/issues/849) - add `gcs.disable_http2` (env `GCS_DISABLE_HTTP2`) — force HTTP/1.1 for the GCS client to avoid single-TCP-connection HTTP/2 multiplexing becoming the bottleneck for high-throughput parallel downloads, fix [#​1437](https://redirect.github.com/Altinity/clickhouse-backup/pull/1437) IMPROVEMENTS - switch the S3 client retry mode from `aws.RetryModeStandard` to `aws.RetryModeAdaptive`, so upload/download retries back off with a client-side rate limiter under throttling (`SlowDown`/503) instead of a fixed token bucket - support tables with `ENGINE=Alias` during `restore` — strip the explicit column list (Alias inherits columns from the target table) and add `allow_experimental_alias_table_engine=1` for ClickHouse 25.11+, fix [#​1426](https://redirect.github.com/Altinity/clickhouse-backup/issues/1426) - skip tables from databases with engine `DataLakeCatalog` (read-only external catalogs) during backup, in addition to `MySQL`/`PostgreSQL`/`MaterializedPostgreSQL`, fix [#​1417](https://redirect.github.com/Altinity/clickhouse-backup/issues/1417) - speed up `create` on servers with many tables: batch the in-progress `system.mutations` lookup instead of issuing one query per table (O(N^2) -> O(N)), fix [#​1431](https://redirect.github.com/Altinity/clickhouse-backup/pull/1431) - speed up `restore`/`download` table list resolution: eliminate the O(N^2) linear duplicate scan in `getTableListByPatternLocal`/`getTableListByPatternRemote` via a shared O(1) dedup keyed on `metadata.TableTitle`, fix [#​1430](https://redirect.github.com/Altinity/clickhouse-backup/pull/1430) - extend TestFlows FIPS 140-3 test coverage: dedicated test steps, stronger `--fips-info`/version-output and TLS-handshake assertions, updated requirements documentation, fix [#​1443](https://redirect.github.com/Altinity/clickhouse-backup/pull/1443) BUG FIXES - restore the default `general.compression_use_multi_thread` to `true`: [#​1378](https://redirect.github.com/Altinity/clickhouse-backup/issues/1378) defaulted it to `false`, which silently dropped gzip/zstd compression from multi-threaded (pre-1378 gzip always used `pgzip`) to single-threaded and caused \~30% slower upload throughput for backups dominated by one large table (where `upload_concurrency` provides no per-stream parallelism); the option is now silently ignored instead of failing config validation for formats other than gzip/zstd - fix v2.7.1 regression where `restore --schema` on a second replica rebound the ZK replica path even while the first replica was active and healthy — a foreign/renamed local table occupying the resolved ZK path is still detected automatically via `system.replicas` and rebound to `default_replica_path`, but a non-empty ZK path without a local table now defaults to the HA-safe join and requires the new opt-in `rebind_replica_path_if_exists`, fix [#​1428](https://redirect.github.com/Altinity/clickhouse-backup/issues/1428) - fix inconsistent `--rbac` behavior: `restore --rbac` no longer returns an error when the backup contains no RBAC `access` directory, matching the `download --rbac` behavior, fix [#​1432](https://redirect.github.com/Altinity/clickhouse-backup/issues/1432) - inject `ON CLUSTER` for `LIVE VIEW` and `WINDOW VIEW` restored with explicit column lists when `restore_schema_on_cluster` is set — `enrichQueryWithOnCluster` missed these statements and the views were created only on the local replica, fix [#​1433](https://redirect.github.com/Altinity/clickhouse-backup/pull/1433) - mask sensitive values (passwords, storage credentials, encryption keys) when logging `--env` config overrides, and in the API server action logs and `/backup/status` output, fix [#​1441](https://redirect.github.com/Altinity/clickhouse-backup/pull/1441) </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/cozystack/cozystack). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI0Mi4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJhdXRvbWF0ZWQiLCJkby1ub3QtbWVyZ2UvaG9sZCJdfQ==--> |
||
|
Aleksei Sviridkin
|
249f7b5a5e |
feat(platform, ingress): propagate operator wildcard certificate to per-tenant termination points (#2990)
## What this PR does The operator-provided wildcard certificate (`publishing.certificates.wildcardSecretName`) currently serves the root tenant only: everything is same-namespace, so the root ingress controller and Gateway read the operator's TLS Secret directly. Child tenants run their own ingress controller / Gateway in their own namespace under a default-deny policy and cannot read that Secret across namespaces, so they fall back to per-host ACME. This propagates the operator wildcard to per-tenant termination points by replicating the Secret into each tenant namespace that owns one, keeping the root-MVP invariant intact: the wildcard is deliberately distributed to tenant namespaces, but no tenant RBAC is widened — every consumer reads only its own-namespace copy. A new controller in `cozystack-controller` does the replication with no extra operator input. It reads the same platform values channel the consumers read (`cozy-system/cozystack-values`): the wildcard Secret name from `_cluster.wildcard-secret-name` and the publishing namespace from `_cluster.expose-ingress`. It mirrors that Secret into every tenant namespace whose `namespace.cozystack.io/ingress` or `namespace.cozystack.io/gateway` label equals its own name — i.e. the namespace runs its own controller / Gateway. Because the source is derived from the same value that makes the consumers reference it, the replica is created whenever the consumers expect it — no manual labelling and no upgrade hazard where a child controller references a Secret that nothing creates. Replicas are refreshed on rotation and pruned when a namespace stops terminating TLS. Teardown of every replica happens on exactly one trigger: clearing `wildcardSecretName` (explicit disable). A source that is merely absent or mistyped, or a transiently-missing values channel, leaves existing replicas in place, so a brief gap, a delete+recreate rotation, or a misconfigured publishing namespace never drops tenant TLS. The platform controller's own ServiceAccount gains Secret writes; tenant RBAC is untouched. The controller does not cache every Secret in the cluster: the manager's Secret informer is scoped to managed replicas and the values channel only, and the operator source / foreign collisions are read through the uncached APIReader. The source is not watched (its name is dynamic), so an in-place rotation is picked up by a bounded periodic resync while the publishing tenant serves the source with no lag. On the consumer side, each tenant ingress controller now sets `--default-ssl-certificate` to the wildcard Secret in its own namespace (`.Release.Namespace/<name>`), so a child tenant serves the replica instead of minting per-host ACME. The Gateway path already renders `certMode: existingSecret` with a same-namespace `wildcardSecretRef` for every tenant, so it consumes the replica with no chart change. ### Why not clustersecret-operator The issue proposed the shipped `clustersecret-operator`. It materializes inline template data into selected namespaces; it cannot mirror an existing source Secret, and it is not installed in any default bundle. Driving it would mean embedding the certificate and private key as literal data in a cluster-scoped object rendered from a Helm `lookup` — that broadcasts key material the design keeps off any shared channel, and a `lookup` is invisible to the Flux helm-controller digest so it would not re-render on rotation. Its namespace selector also cannot express "the namespace whose owner label equals its own name", so it would over-replicate the key into every tenant namespace. A small dedicated reconciler targets exactly the termination namespaces and keeps the key off the values channel. ### Coverage caveat Replication delivers the certificate bytes, not SAN coverage: a single `*.<root-host>` does not match `*.<child-apex>`. Operators serving child-tenant subdomains supply a certificate whose SANs cover those apexes; the docs spell this out. The replica carries the wildcard private key into each terminating tenant namespace — the same exposure as the per-host ACME Secret it replaces, now shared across tenants. ### Tests - Go fake-client tests for the reconciler: termination-only replication (ingress-owning and gateway-owning namespaces only, never inheriting ones, never over the source), copy markers, rotation, prune on namespace demotion, the non-destructive source-deletion contract, replica self-heal after out-of-band deletion, the publishing-namespace self-replication guard, disable-via-config teardown, source rename, foreign-Secret collision safety (no overwrite, no requeue, does not block siblings), transient-error requeue, non-TLS source ignored, absent-channel keeps replicas, publishing-namespace mismatch keeps and requeues, the cache-scope pin, the enqueue-wiring map, an `ownsTerminationPoint` table, and the active-only requeue contract. - helm-unittest for the ingress `default-ssl-certificate` wiring (including a child-tenant controller pointing at its own-namespace replica) and for the cozystack-controller ClusterRole's new Secret-write rule. ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(platform): the operator-provided wildcard TLS certificate (`publishing.certificates.wildcardSecretName`) is now served by per-tenant ingress controllers and Gateways automatically — the platform controller replicates it into each tenant namespace that terminates TLS, with no extra operator input and no cross-namespace Secret read ``` Closes #2820 Part of #2811 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **New Features** * Wildcard TLS Secret now replicates from the platform to tenant namespaces that terminate TLS via their own ingress controller or Gateway, including child-tenant scenarios. * **Documentation** * Clarified configuration, timing, and SAN matching requirements for wildcard replication. * Updated gateway and ingress-nginx default certificate documentation for replica mirroring behavior. * **Bug Fixes** * Improved handling for missing/invalid source secrets (no unnecessary pruning) and safe collision behavior. * **Tests** * Added comprehensive reconciliation/replication test coverage. * Added Helm unittest target for the controller chart. * **Chores** * Updated controller RBAC to support wildcard TLS Secret replication. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
3763c528da |
fix(seaweedfs): soften master anti-affinity for small clusters (#3197)
## What this PR does Overrides the seaweedfs master anti-affinity in the package values from a hard `requiredDuringSchedulingIgnoredDuringExecution` hostname term to a strong preference (`preferredDuringSchedulingIgnoredDuringExecution`, weight 100). The selector and topologyKey are unchanged. The vendored subchart pins the master to a hard hostname anti-affinity — its own comment notes that emptying it is what allows single-node deployment. With the master StatefulSet at three replicas, the surplus master Pods stay Pending forever on clusters with fewer than three nodes, so `helm install --wait` never returns and the S3 stack never comes up. Softening to a strong preference keeps the masters spread across distinct nodes whenever the cluster has them (the scheduler only co-locates when there is no alternative — exactly the small-cluster case), while letting single- and dual-node clusters bring the master up. This mirrors the posture already applied to the s3 component in the same values file. A helm-unittest pins the softened term (asserts the hard term is gone and the preferred term carries the right weight, topologyKey, and component selector) so a future subchart bump cannot silently re-harden it. ### Screenshots N/A — no UI changes. ### Release note ```release-note fix(seaweedfs): soften the master anti-affinity to a preference so the S3 stack installs on clusters with fewer than three nodes ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * SeaweedFS master pods now use a softer scheduling preference for hostname separation, improving deployment success on smaller clusters. * **Tests** * Added a Helm chart validation to confirm the anti-affinity behavior is now expressed as a weighted preference (not a hard requirement), including the expected selector and topology key. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
6c9edf5038 |
fix(platform): route OIDC backend calls through the in-cluster Keycloak by default (#3289)
## What this PR does Defaults `authentication.oidc.keycloakInternalUrl` to `http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy` instead of an empty string. With the empty default, oauth2-proxy performed OIDC discovery against the external Keycloak hostname. That request resolves through public DNS and lands on the root ingress controller, which does not serve yet on a fresh install: the root tenant's IngressClass only exists once `tenant-root` is ready, and `tenant-root` is still reconciling while the dashboard starts. Discovery has nothing to talk to, the gatekeeper fails, and the dashboard never comes up — with no obvious diagnosis for a first-time installer. The bundled Keycloak is the only supported provider (the external hostname is templated as `keycloak.<host>` and the realm is fixed), so backend-to-backend calls have no reason to leave the cluster. The new default is the same Service address `keycloak-configure` already uses for its own backend calls. The dashboard and linstor-gui gatekeepers then run with `--skip-oidc-discovery` and send token, jwks, userinfo and logout requests straight to the Service, while browser-facing redirects stay on the external host. Both gatekeeper templates already branch on the value being non-empty, so neither needed a change. The value stays overridable for a Keycloak that does not live at the standard in-cluster address, and setting it back to `""` restores the previous external-discovery behaviour. Token validation is unaffected: Keycloak pins `KC_HOSTNAME` to the external URL, so tokens redeemed over the internal Service still carry the external `iss` and match the unchanged `--oidc-issuer-url`. ### Upgrade behaviour Existing OIDC clusters that never set `keycloakInternalUrl` pick up the new default on upgrade: their gatekeepers roll once and move to the internal path. Clusters that set the value explicitly — including to `""` — keep exactly what they configured. ### Security posture The backend leg (token redemption, which carries the gatekeeper's Keycloak client secret, plus jwks/userinfo/logout) now travels as plaintext HTTP between pods rather than HTTPS to the ingress. This is stated in the `values.yaml` comment rather than left implicit. The `keycloak-http` Service has no TLS listener, so in-cluster HTTPS is not an available alternative, and `keycloak-configure` already sends Keycloak *admin* credentials over that identical address — anyone positioned to read this traffic can already reach a strictly more powerful secret. The externally reachable front door (browser to oauth2-proxy) is unchanged. ### Testing helm-unittest coverage across all three affected charts: the default landing on the `_cluster` channel, both gatekeepers rendering the in-cluster backend URLs while keeping browser URLs external, an explicit override winning, the empty-string escape hatch still rendering the external-discovery shape, and OIDC-disabled clusters (now the common case of "OIDC off, internal URL set") ignoring the value entirely. The E2E OIDC case previously applied this exact URL as a manual workaround. It now sets only `oidc.enabled` and asserts the consequence: it waits for the `auth-proxy` container to appear in the gatekeeper Deployment, then gates on `kubectl rollout status`. Readiness alone would prove nothing there — the HelmRelease is still `Ready` from the token-proxy install, and with `replicas: 1` and `maxUnavailable: 25%` (which rounds down to 0) the Deployment keeps the old ReplicaSet `Available` while a new pod crashloops. Only the rollout catches a regression back to external discovery. Follow-up: the OIDC documentation in the website repo still frames `keycloakInternalUrl` as an optional tweak "for self-signed certificates or restricted external access". It is now the default path and needs a separate docs PR. Closes #3288 ### Screenshots No UI changes. ### Release note ```release-note fix(platform): `authentication.oidc.keycloakInternalUrl` now defaults to the in-cluster Keycloak Service (`http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy`). OIDC backend calls from the dashboard and linstor-gui gatekeepers no longer leave the cluster, so enabling OIDC on a fresh install no longer leaves the dashboard down while the root ingress is not yet serving. Existing clusters that never set the value pick up the new default and roll their gatekeepers once. Set the value explicitly to point at a non-standard Keycloak placement, or to an empty string to restore external OIDC discovery. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - OIDC-enabled applications now use the internal Keycloak service for backend authentication by default, improving reliability and avoiding unnecessary external discovery. - Browser redirects and login flows continue using the external Keycloak address. - Operators can provide a custom internal Keycloak URL or set it to an empty value to restore external OIDC discovery. - **Bug Fixes** - Dashboard and Linstor GUI authentication now correctly apply internal Keycloak settings only when OIDC is enabled. - Dashboard upgrades now wait for authentication components to become ready before completing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
f24b56327b |
fix(platform): route OIDC backend calls through the in-cluster Keycloak by default
authentication.oidc.keycloakInternalUrl defaulted to an empty string, which left oauth2-proxy performing OIDC discovery against the external Keycloak hostname. That request resolves through public DNS and lands on the root ingress controller, which does not serve yet on a fresh install: the root tenant's IngressClass only exists once tenant-root is ready, and tenant-root is still reconciling while the dashboard starts. Discovery has nothing to talk to, the gatekeeper fails, and the dashboard never comes up. The bundled Keycloak is the only supported provider — the external hostname is templated as keycloak.<host> and the realm is fixed — so backend-to-backend calls have no reason to leave the cluster. Default the value to http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy, the same Service address keycloak-configure already uses. The dashboard and linstor-gui gatekeepers then run with --skip-oidc-discovery and send token, jwks, userinfo and logout requests to the Service, while browser redirects stay on the external host. Those calls travel as plaintext HTTP between pods, as keycloak-configure's admin calls to the same address already do. The value remains overridable for a Keycloak that does not live at the standard in-cluster address, and setting it back to an empty string restores external discovery. Existing OIDC clusters that never set it get a gatekeeper rollout onto the internal path on upgrade. Enabling OIDC swaps the dashboard's token-proxy container for oauth2-proxy, so the E2E install now drives that swap to completion: it waits for the auth-proxy container to appear, then gates on the gatekeeper rollout. Readiness alone proves nothing there — the HelmRelease is still Ready from the token-proxy install, and the Deployment keeps the old ReplicaSet Available while a new pod crashloops, so only the rollout catches a regression back to external discovery. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
caf4645cf2 |
docs(harbor): fix SeaweedFS discovery key described as annotation
The prerequisites section described the resolved SeaweedFS class as being surfaced on the tenant namespace as the `namespace.cozystack.io/seaweedfs` annotation. The tenant chart renders that key under `metadata.labels`, alongside the sibling `/etcd`, `/ingress`, `/gateway`, `/monitoring` and `/host` keys; no annotation of that name exists. The wrong noun sends an operator to the wrong field. When a Harbor release stays unreconciled waiting on `BucketInfo`, the documented check is to confirm the tenant chain resolved a SeaweedFS class. Looking for it in `.metadata.annotations` returns nothing even on a correctly configured tenant, which invites the wrong conclusion that the tenant chain is broken and sends the investigation away from the actual cause. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
5d881f111a |
fix(fluxcd): omit empty distribution.artifact in FluxInstance for guest clusters (#3284)
## What this PR does
Guest Kubernetes clusters (`apps/kubernetes`) with the `fluxcd` addon
enabled never became
`Ready`. The `flux-instance` template renders
`spec.distribution.artifact` unconditionally,
unlike the sibling optional fields (`artifactPullSecret`,
`imagePullSecret`, `variant`, which
are all `{{- if }}`-guarded). The umbrella default is empty (`artifact:
""`) so the operator
uses its embedded manifests, which is the intended air-gapped default
(#964, and the upstream
recommendation). But an empty value renders as YAML `null`, and the
flux-operator v0.50.0 CRD
types `distribution.artifact` as a non-nullable `string` (pattern
`^oci://.*$`), so the API
rejects the `FluxInstance`.
Result: the addon's HelmRelease looped install/uninstall
(`remediation.retries: -1`), the
parent `kubernetes` release stayed `InProgress`, and the cluster CR was
stuck `Ready=Unknown`.
This guards the field so an empty value is omitted entirely. Behaviour:
- empty (default) -> field omitted -> operator falls back to embedded
manifests: **air-gapped
default preserved**, and the CRD accepts it (artifact is not in
`distribution.required`);
- non-empty -> renders as before; `valuesOverride` still applies.
The template is a vendored subchart (`make update` runs `helm pull`), so
the change is carried
as `patches/guard-distribution-artifact.diff` and applied by the
`update` target, matching the
convention of the sibling `fluxcd-operator` package (`patch -p1 <
patches/*.diff`).
Verified with `helm template packages/system/fluxcd`:
- default values -> no `artifact:` line (before this PR: `artifact:
null`);
- `--set
flux-instance.instance.distribution.artifact=oci://example.com/x:1` ->
renders.
The affected path is off by default (`addons.fluxcd.enabled: false`) and
not exercised by the
in-repo CI values, which is why default e2e stayed green.
Closes #3283
### Release note
```release-note
fix(fluxcd): guest Kubernetes clusters with the fluxcd addon enabled no longer fail to install; an empty distribution artifact is now omitted so the operator uses its embedded manifests (air-gapped default preserved) instead of rendering an invalid null
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Prevented Flux instance configuration from including an empty
distribution artifact value, which could cause validation failures.
* Improved the generated FluxInstance manifest so the artifact field is
omitted entirely when no distribution artifact is provided.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Ivan Okhotnikov
|
da47b70e2c |
fix(fluxcd): omit empty distribution.artifact in FluxInstance
The flux-instance template rendered spec.distribution.artifact unconditionally, unlike the sibling optional fields (artifactPullSecret, imagePullSecret, variant). The umbrella default is empty to use the operator's embedded manifests (the air-gapped default from #964), but an empty value renders as YAML null. In the flux-operator v0.50.0 CRD the field is optional yet typed 'string' with pattern ^oci://.*$ and is not nullable, so the rendered null (like an empty string) fails schema validation and the API rejects the FluxInstance. Guest Kubernetes clusters with the fluxcd addon enabled could therefore never install it: the sub-release looped install/uninstall, the parent kubernetes release stayed InProgress, and the cluster never reached Ready. Guard the field so an empty value is omitted (the operator falls back to its embedded manifests, preserving the air-gapped default) while a non-empty artifact still renders. The template lives in the vendored subchart, so the change is carried as a patch applied by 'make update'. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
f478915935 |
fix(dashboard): unbreak CORS on expired session for k8s API calls (#2788)
## What this PR does Fix two related issues that cause the dashboard SPA to break with CORS errors once the `kc-access` cookie expires. 1. `KeycloakClient` for dashboard had no `webOrigins` set, so the operator did not touch the field in Keycloak after initial creation. After any `rootUrl` change the existing `webOrigins` value got stuck at an old hostname and blocked CORS from the SPA. Setting it to `+` makes Keycloak derive the allowed origins from `redirectUris`. 2. `oauth2-proxy` answered `302` (redirect to Keycloak `/auth`) for unauthenticated XHR/fetch on `/api` and `/apis`. The login page does not emit `Access-Control-Allow-Origin`, so the browser blocked the fetch with a CORS error and the SPA had no way to recover. Adding `--api-route` makes `oauth2-proxy` answer `401` without a redirect for these paths, so the SPA can handle re-auth itself via a top-level navigation. ### Release note ```release-note fix(dashboard): dashboard SPA no longer fails with CORS errors after the kc-access cookie expires; oauth2-proxy returns 401 (not 302) for /api and /apis, and the KeycloakClient declares webOrigins: ["+"]. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Chores * Updated API routing configuration to properly handle requests to API endpoints * Enhanced web origins configuration for improved authentication system compatibility <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
0044221d31 |
fix(apps/vpn): remove invalid foo field from urls Secret (#3281)
## What this PR does Removes a leftover debug field `foo` from the `<release>-urls` Secret template in the VPN app chart. The `foo` field is not part of the `core/v1` Secret schema (which only allows `data`, `stringData`, `type`, `immutable`, and `metadata`). It went unnoticed under client-side apply, but under server-side apply the stricter schema validation rejects it with: ```text .foo: field not declared in schema ``` As a result, any VPN application upgrade fails and the release is left in `UpgradeFailed` state. ### Release note ```release-note fix(apps/vpn): remove invalid `foo` field from the urls Secret that broke Helm upgrades under server-side apply ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added per-user Shadowsocks connection links to the VPN secret. * Links include the configured encryption method, host, port, and connection name for easier client setup. * **Bug Fixes** * Removed obsolete serialized password data from the VPN secret. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
bf69eff7a5 |
feat(keycloak): add S3 backup for the CNPG database (#3174)
## What this PR does Adds optional scheduled S3 backups for the `keycloak-db` CloudNativePG cluster. When `backup.enabled=true` with a `backup.destinationPath`, the Cluster gets `spec.backup.barmanObjectStore` (gzip-compressed data + WAL streams, a configurable `retentionPolicy`) and a `ScheduledBackup` drives recurring base backups. S3 credentials are either inlined into a rendered `keycloak-db-backup-s3-creds` Secret, or referenced via `backup.existingSecretName` (a Secret carrying `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`). Disabled by default — no change to existing deployments; the object store coordinates and credentials are supplied per-environment by the operator. This is the Keycloak/Postgres half of backing up a Cozystack portal control plane. The portal-side etcd backups (system Talos etcd + api-server etcd) live in the companion downstream PR aenix-org/cozyportal#1008. ### Release note ```release-note feat(keycloak): add optional S3 backups (CNPG Barman) for the Keycloak database, configurable via `backup.*` values and disabled by default. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable scheduled backups for the Keycloak CNPG database to an S3-compatible object store, including destination prefix, optional endpoint URL, retention policy, and cron schedule. * Backups and scheduled backup resources are only created when all required backup settings are provided, avoiding incomplete configuration. * Supports backup credentials via inline values (creates the needed Secret automatically) or via an existing Secret reference. * **Documentation** * Added guidance on the cron schedule format required for the scheduler. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Ivan Okhotnikov
|
70a2284551 |
fix(apps/vpn): remove invalid foo field from urls Secret
A leftover debug field "foo" in the <release>-urls Secret template violates the core/v1 Secret schema. It went unnoticed under client-side apply but breaks Helm upgrades under server-side apply with ".foo: field not declared in schema", leaving the release in UpgradeFailed state. Drop the debug block. Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com> |
||
|
Myasnikov Daniilandmattia-eleuteri
|
db4db4a6e8 |
feat(kubernetes): per-node-group nodeHealthCheck overrides
Add optional nodeGroups[name].maxUnhealthy and nodeGroups[name].nodeStartupTimeout
fields that override the cluster-wide nodeHealthCheck defaults per worker node
group. When unset, a group inherits nodeHealthCheck.{maxUnhealthy,nodeStartupTimeout},
so existing values are unaffected. The MachineHealthCheck render resolves the
per-group value first, reuses the existing int-or-string validation, and names
the offending group in the fail() message when an override is invalid.
Delivers the per-group granularity of the standalone maxUnhealthy proposal on
top of the cluster-wide nodeHealthCheck from the Talos worker bootstrap PR.
Co-authored-by: mattia-eleuteri <mattia@hidora.io>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
renovate[bot]
|
9dc63bd412 | chore(deps): update github.com/kubevirt/csi-driver digest to 27b52aa | ||
|
myasnikovdaniil
|
99a28d82c2 |
test(e2e): migrate app suite from BATS to Chainsaw and wire into CI (#2826)
## What this PR does Migrates the E2E **app** test suite from BATS to [Kyverno Chainsaw](https://github.com/kyverno/chainsaw) — a declarative, Kubernetes-native E2E framework (CNCF, part of the Kyverno project; successor to KUTTL) — and wires it into CI as the replacement for the per-app BATS loop. This grows the original pilot (postgres + bucket) into the full suite. All 22 app suites are ported under `hack/e2e-chainsaw/` and the `hack/e2e-apps/*.bats` files are removed: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`, `clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`, `kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`. **Approach** - **Declarative** suites assert on `status.conditions` and concrete fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait` pair that appeared ~100 times collapses into a single `assert` that polls existence and state together, with a structured diff on failure and automatic `events`/`describe`/`podLogs` capture via `catch` (previously only `harbor.bats` did this, by hand). - **Imperative** suites keep their logic in `script` steps: `openbao` init/unseal, `kuberture` external-dns split-horizon probes, `vminstance`, the `gateway` admission/impersonation cases, and `kubernetes-latest`/`previous`, which wrap the relocated `hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up, LB/NFS/ouroboros checks). - `gateway` tests derive the tenant apex from the namespace `namespace.cozystack.io/host` label at runtime, so they are host-independent. **CI** - The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new `test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`. - The `chainsaw` binary is added to the e2e-sandbox image. - `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap + OpenAPI checks). - The per-app 3-retry loop is dropped — assertion polling replaces the fixed-timeout flakiness it papered over. **Validation** Ran against a development cluster: the DB/app, storage, and VM suites pass; the three suites that depend on platform features not present on that cluster (`gateway`, `kuberture`, `external-dns`) and the two heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a freshly installed platform. Note for reviewers: service-port asserts use the `(ports[*].port)` projection form rather than a number-literal filter (`` ports[?port == `N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal comparisons. **2026-07-09: reconciled with main (~520 commits of drift)** - etcd suite re-ported to the v1alpha2 operator contracts (#2859): `readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip driving `examples/backups/etcd` - 12 new `ingress-hostname-policy` gateway cases ported (apex derived at runtime from the tenant-root namespace label) - post-branch bats drift folded in: `mariadb-single` webhook guard, kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant PVC-reclaim guard (#3059), kuberture fail-loud negations, SC-fallback-default test (#2872 B1) - 4 suites that only exist on main since the branch was cut are ported: `kubernetes-oidc-system`, `kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure` - CI seams: `nightly.yaml` converted to the chainsaw invocation, `run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh` replicated in the Chainsaw global catch ### Release note ```release-note fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * End-to-end testing is now driven by a unified Chainsaw suite (platform apps, Kubernetes, networking, security groups, service exposure). * Test Impact Analysis selects affected suites, with an option to run the full suite. * CI produces a JUnit report artifact and collects suite-scoped diagnostics on failure. * **Bug Fixes** * CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and sandbox bucket state diagnostics. * Improved teardown reliability with bounded wait + finalizer handling to avoid hangs. * **Documentation** * Updated E2E testing docs and local guidance to reflect the new Chainsaw-based workflow and suite paths. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
b99df11b36 |
fix(kubernetes): bump tenant worker Talos to v1.13.6 (#3269)
## What this PR does Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24. That kernel predates the fixes for two related KVM x86 shadow-paging use-after-free bugs in `arch/x86/kvm/mmu/mmu.c`: CVE-2026-46113, fixed upstream in `0cb2af2ea66ad8ff195c156ea690f11216285bdf` and released in 6.18.30, and CVE-2026-53359, fixed in `81ccda30b4e83d8f5cc4fd50503c44e3a33abfeb` and released in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables nested virtualization inside the VM, and either bug lets a nested guest escape into the worker. This bumps the worker pin to Talos v1.13.6, which ships Linux 6.18.38. Stable kernel branches are cumulative, so v1.13.6 is the first v1.13 patch release carrying both fixes. The pinned v1.13.0 ships 6.18.24 and carries neither; v1.13.5 ships 6.18.36 and carries only CVE-2026-46113. Nested virtualization stays available to tenants; the kernel bump is the fix, not a capability removal. The image-factory schematic is unchanged and resolves for the new version: both the `openstack-amd64.raw.xz` artifact and the `installer/<schematicID>:v1.13.6` manifest are served by the public factory. Talos v1.13 supports Kubernetes 1.31 through 1.36, so the chart's Talos-to-Kubernetes support window is unaffected and no tenant `spec.version` needs to move. Applying this rolls each worker pool once. The new pin changes the rendered worker spec, which rotates the content hash in the `KubevirtMachineTemplate` and `talos-reconcile` Job names, and CAPI replaces the worker VMs to deliver the new kernel. The pinned hashes in the chart tests are recomputed to match. A second commit fixes the support-matrix lookup itself. The matrix dict was keyed on the full patch version, so every Talos patch bump dropped out of the dict and silently disabled the render-time guard — exactly the mismatch the guard exists to catch. The lookup now derives the minor from `talos.version`. The supported-Kubernetes window is unchanged; unknown minors are still skipped, leaving compatibility to whoever overrode the pin. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note fix(kubernetes): bump tenant worker Talos to v1.13.6, picking up the Linux 6.18.38 kernel fixes for CVE-2026-46113 and CVE-2026-53359 (KVM shadow-paging use-after-free, nested-guest escape into the worker VM). Nested virtualization remains available. Applying this upgrade rolls each tenant worker pool once, as the worker template is renamed to carry the new image. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Updated the default Talos worker release to v1.13.6 for Kubernetes cluster provisioning. - Improved compatibility checks to apply consistently across Talos patch releases within the same minor version. - **Bug Fixes** - Corrected generated resource naming expectations when storage settings or GPU configuration change. - **Tests** - Added coverage confirming Kubernetes clusters render successfully with Talos v1.13.6. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
renovate[bot]
|
ebecbd71c9 | chore(deps): update altinity/clickhouse-backup docker tag to v2.7.4 | ||
|
Myasnikov DaniilandClaude Opus 4.8
|
302a72f14a |
Merge origin/main into feat/e2e-chainsaw-example
Reconciles the talos-image-cache conflicts created when #3254 merged to main: keep the Chainsaw _lib/ port (hack/e2e-chainsaw/_lib/talos-image-cache.sh, a superset of #3254's logic + the node-join diagnose helper), drop the BATS-era hack/e2e-apps/talos-image-cache.sh, and keep the _lib/-path versions of the mirror manifest, install bats, and unit test. Also picks up main's zizmor action-pinning (#3223, unblocks pre-commit/zizmor) and Talos v1.13.6 (#3240). Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
02b87b63f2 |
fix(apps/kubernetes): bound the pre-delete hook's tenant HelmRelease wait (#3271)
The pre-delete hook (delete.yaml) deleted the tenant's addon HelmReleases with
an unbounded `kubectl delete … --wait=true`. When the tenant has no working
nodes (LINSTOR/ZFS exhaustion, worker VMs that never joined, or a nodeGroup at
minReplicas: 0), those uninstalls cannot converge — in-tenant cilium stays
InstallFailed because cilium-operator has nothing to schedule on — so the hook
hung indefinitely, the parent Helm uninstall failed on the pre-cleanup Job, and
the Kubernetes CR was stuck deleting.
Bound the wait with --timeout=120s; on expiry, force-clear the Flux finalizer
from any child HelmRelease still present so it can be garbage-collected — safe
because the whole tenant (control plane torn down in the steps below) is being
destroyed, so there is nothing left to orphan. A healthy tenant's children
uninstall well within the timeout and never reach the force path.
Also unblocks the render-side kubernetes-oidc-{system,customconfig} Chainsaw
e2e suites, which deterministically hit this teardown hang under the Chainsaw
cleanup budget.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
renovate[bot]
|
5b9b420ffe | chore(deps): update container image & action digests | ||
|
Aleksei Sviridkin
|
a3829ab66c |
fix(talos): bump Talos to v1.13.6 (CVE-2026-53359) (#3240)
## What this PR does Bumps the platform Talos profiles from v1.13.5 to v1.13.6. CVE-2026-53359 and CVE-2026-46113 are related KVM x86 shadow-paging use-after-free bugs that let a guest escape to the host, and the fix is complete only once both upstream patches (`81ccda30b4e8`, `0cb2af2ea66a`) are applied. On the 6.18 stable line CVE-2026-46113 landed in 6.18.30 and CVE-2026-53359 in 6.18.38. Talos v1.13.5 runs Linux 6.18.36, so it carries only the first of the two. Talos v1.13.6 runs Linux 6.18.38 and carries both. Nested virtualization stays enabled. The bugs are fixed in the kernel, so there is no reason to take the feature away from workloads that rely on it — bumping the kernel is the complete fix. Profiles are regenerated with `hack/gen-profiles.sh v1.13.6`. Alongside the Talos version and the installer reference, this refreshes the pinned system-extension digests: the firmware extensions move from the 20260519 to the 20260622 build, and `intel-ucode` keeps its `20260512` tag but resolves to a new digest because upstream re-pushed that tag as an OCI image index (identical config and layer blobs underneath). References: [Talos v1.13.6 release notes](https://github.com/siderolabs/talos/releases/tag/v1.13.6) (`Linux: 6.18.38`) ### Screenshots Not applicable, no UI changes. ### Release note ```release-note fix(talos): bump Talos Linux to v1.13.6, which ships Linux 6.18.38 with the kernel fixes for the CVE-2026-53359 and CVE-2026-46113 KVM guest-to-host escapes ``` |
||
|
Andrei Kvapil
|
681f4a8313 |
build(flux-shard-operator): read the shared registry build cache
flux-shard-operator was the last image built with the pre-#2937 cache pattern: --cache-from a :latest tag that PR builds no longer publish and --cache-to type=inline, which never caches multistage builder layers. Use the shared cache-args macro so it reads (and, on main, writes) the same mode=max :buildcache ref in CACHE_REGISTRY as every other image. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Aleksei Sviridkin
|
3e9b5600e3 |
fix(talos): bump Talos to v1.13.6 (CVE-2026-53359)
Talos v1.13.5 ships Linux 6.18.36, which carries the fix for CVE-2026-46113 but not the one for CVE-2026-53359. The two are related KVM x86 shadow-paging bugs allowing a guest to escape to the host, and both patches are needed for the mitigation to hold. Talos v1.13.6 ships Linux 6.18.38, the first release on the 6.18 stable line that has both. Profiles regenerated with hack/gen-profiles.sh v1.13.6. Alongside the Talos version and the installer reference, this refreshes the pinned system-extension digests: the firmware extensions move from the 20260519 to the 20260622 build, and intel-ucode stays on its 20260512 tag but resolves to a new digest because upstream re-pushed that tag. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
b73860d4c9 |
fix(kubernetes): bump tenant worker Talos to v1.13.6
Tenant worker VMs were pinned to Talos v1.13.0, which ships Linux 6.18.24. That kernel predates both fixes for a pair of related KVM x86 shadow-paging use-after-free bugs: CVE-2026-46113, fixed in 6.18.30, and CVE-2026-53359, fixed in 6.18.38. A worker's own kernel runs KVM whenever a tenant enables nested virtualization inside the VM, and either bug lets a nested guest escape into the worker. Talos v1.13.6 ships Linux 6.18.38. Stable kernel branches are cumulative, so it is the first v1.13 patch release carrying both fixes: the pinned v1.13.0 ships 6.18.24 and carries neither, while v1.13.5 ships 6.18.36 and carries only CVE-2026-46113. Nested virtualization stays available to tenants — the kernel bump is the fix. The new pin changes the rendered worker spec, so the content hash in the KubevirtMachineTemplate and talos-reconcile Job names rotates and each worker pool rolls once on upgrade. Pinned hashes in the chart tests are recomputed accordingly. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
b420d3de02 |
fix(kubernetes): key Talos support matrix on minor version
The Talos<->Kubernetes support window is declared per Talos minor release, and patch releases never move it. Keying the chart's matrix dict on the full patch version meant every Talos patch bump fell out of the dict and silently disabled the render-time guard — precisely the mismatch the guard exists to catch. Derive the minor from talos.version before the lookup, so the guard keeps firing across patch releases of a known minor. Unknown minors are still skipped, leaving compatibility to the operator who overrode the pin. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Myasnikov DaniilandClaude Fable 5
|
1f84d2d49e |
Merge origin/main into feat/e2e-chainsaw-example (post-#3244)
Fold the in-sandbox Talos image cache work into the Chainsaw layout: the new shared helper relocates to hack/e2e-chainsaw/_lib/ talos-image-cache.sh and run-kubernetes.sh sources it from there (the ouroboros DNS-check no-retry rework merges clean). The OIDC bats are resolved as deleted again; their Chainsaw fixtures deliberately skip the talos.imageFactoryURL injection — the helper only emits it when the cache is up, and the zero-worker render-side tests instantiate no DataVolume imports for it to redirect. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
scooby87
|
f91841fce6 |
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3215)
## What this PR does Addresses #3181, where platform log ingestion could fail **silently**: fluent-bit shipped to `vlinsert-generic` while the monitoring `HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName` in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary layers make the failure loud instead of silent: 1. **Readiness gate** — `packages/extra/monitoring/templates/helmrelease.yaml` now sets a `poller` `waitStrategy` and a `healthCheckExprs` entry on the `VLCluster` `status.updateStatus`. A missing/unhealthy log store makes the `monitoring-system` release **not-ready** instead of flipping `Ready` the moment the CR is applied. Mirrors the existing seaweedfs CNPG pattern. 2. **No empty-log-store hole** — `packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render on an empty `logsStorages`. With no entry no `VLCluster` exists, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is now a loud misconfiguration. 3. **No silent fallback** — the `cozy-monitoring → tenant-root` `vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are created only when the root tenant hosts monitoring. The template reads the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth, so it cannot drift from the tenant's real state) and falls back to a new platform value `monitoring.rootEnabled` (via `_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first render). The default preserves the historical always-create behaviour. Covered by helm-unittest across all four affected charts (141 tests): readiness-gate fields, empty-vs-populated `logsStorages` contract, `ExternalName` present/absent fallback contract, and the platform wiring into the `cozystack-values` Secret. Also adds the missing `test:` target to `packages/extra/monitoring/Makefile` so its suite runs in CI. **Known limitation / follow-up:** the legacy platform monitoring release installed directly into `cozy-monitoring` via the `cozystack.monitoring` PackageSource cannot carry the readiness gate, because the engine's `ComponentInstall`/HelmRelease builder and the vendored `helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`. That source is not referenced by any active bundle; full coverage there needs a helm-controller bump plus engine support. ### Release note ```release-note fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a platform setting (`monitoring.rootEnabled`) to control whether root-level monitoring is enabled. * Monitoring redirect ExternalName services are now rendered only when root monitoring is enabled (defaulting to enabled). * **Bug Fixes** * Ensures an explicitly set `monitoring.rootEnabled=false` is preserved when generating platform configuration. * Prevents incomplete monitoring deployments by failing rendering when `logsStorages` is empty. * Improves VictoriaMetrics readiness by delaying “ready” until the operator reports an operational status. * **Tests** * Added/expanded Helm-unittest suites covering monitoring wiring, redirects, readiness gating, and `logsStorages` requirements. * Added a `make test` target to run chart unittests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
c32ac68ab8 |
test(apps/kubernetes): anchor the trailing-slash imageFactoryURL assertion
Match the full rendered worker-disk URL shape (version + artifact + end anchor) like the sibling override assertion, so the trailing-slash normalization case cannot pass on a partial match. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
f60fe2270b |
Merge remote-tracking branch 'origin/main' into feat/kubernetes-talos-image-factory-override
Signed-off-by: Aleksei Sviridkin <f@lex.la> # Conflicts: # packages/system/kubernetes-rd/cozyrds/kubernetes.yaml |
||
|
scooby87
|
6c69c0055e |
fix(monitoring)!: clean up orphaned storage PVCs and TLS secrets on tenant module delete (#3091) (#3094)
## What this PR does
When the `monitoring` tenant module is disabled, the
VictoriaMetrics/VictoriaLogs storage PVCs (created by vm-operator) and
the `alerta-tls` / `grafana-ingress-tls` cert-manager secrets were left
orphaned in the tenant namespace.
This adds a `post-delete` cleanup hook
(`packages/extra/monitoring/templates/hooks/cleanup.yaml`) that:
- deletes the VM/VL storage PVCs scoped to this release via `-l
apps.cozystack.io/application.name={{ .Release.Name }}-system`. That
label is stamped on the `storage.volumeClaimTemplate.metadata.labels` of
the VMCluster/VLCluster CRs (`packages/system/monitoring`); vm-operator
copies claim-template labels onto the PVCs (`IntoSTSVolume`), so the
selector matches and stays scoped to this monitoring release.
`managed-by=vm-operator` is intentionally **not** used — vm-operator
does not reliably stamp it onto the storage PVCs;
- deletes the `alerta-tls` and `grafana-ingress-tls` secrets by name
(cert-manager TLS secrets carry only the cluster-wide
`controller.cert-manager.io/fao=true` label, so a label sweep would be
unsafe — they are removed by name).
CNPG PostgreSQL PVCs (`alerta-db-*`, `grafana-db-*`) are intentionally
**not** targeted — CNPG cleans those up itself, and they do not carry
`apps.cozystack.io/application.name`, so the selector leaves them
untouched.
**Upgrade path:** the claim-template label only lands on PVCs created
after the change, so **migration 50**
(`packages/core/platform/images/migrations/migrations/50`) backfills
`apps.cozystack.io/application.name` onto the pre-existing VM/VL storage
PVCs of every cozystack-managed monitoring cluster, so the cleanup
reclaims storage on already-deployed clusters too.
A `test` Makefile target and helm-unittest suites are added for both the
cleanup hook and the claim-template label. The cleanup Job is hardened:
non-root, read-only root fs, dropped capabilities, `seccompProfile:
RuntimeDefault`, resource requests/limits, `activeDeadlineSeconds: 120`,
`backoffLimit: 0`, `--wait=false` on both deletes; RBAC scoped to PVC
get/list/delete and the two named secrets. Failures are logged but never
block teardown.
> [!WARNING]
> **Disabling the monitoring module now permanently deletes its
VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they
hold.** This is intentional and fixes the storage leak. Back up before
disabling if the data matters.
Fixes #3091.
### Release note
```release-note
fix(monitoring): garbage-collect the orphaned VictoriaMetrics/VictoriaLogs storage PVCs and the alerta/grafana TLS secrets left behind when a tenant's monitoring module is disabled; a migration backfills the release label onto pre-existing storage PVCs so already-deployed clusters are covered too. WARNING: disabling monitoring now permanently removes its storage PVCs and the data they hold.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a Helm **post-delete** monitoring cleanup job to best-effort
remove managed PVCs and two specific TLS secrets.
* Included dedicated cleanup RBAC and hardened pod security settings for
the cleanup job.
* **Tests**
* Added Helm chart tests validating hook weight/policies, job settings,
exact deletion commands, and negative cases.
* Added a `make test` target to run Helm unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Kirill Ilin
|
88c0cc36ad |
feat(keycloak): add Vault Kubernetes and AppRole auth for the encryption proxy (#3239)
## What this PR does Makes the optional `keycloak-kms-proxy` PII-encryption feature production-usable. **Vault auth** — new `encryption.kms.vault.auth` switch: - `kubernetes` — **recommended**. The proxy logs in with its own dedicated ServiceAccount (`keycloak-kms-proxy`; bind the Vault role to `system:serviceaccount:<ns>:keycloak-kms-proxy`), so no credential is stored in a Secret. - `approle` — `KKP_VAULT_ROLE_ID` / `KKP_VAULT_SECRET_ID` from `encryption.kms.vault.appRole.secretName`. - `token` (default, unchanged) — Vault token from `tokenSecretName`. The proxy pod runs under a dedicated ServiceAccount (never the namespace default), with its token mounted only for Kubernetes auth. **Shared DEK set** — `encryption.deksetSecretName` → `KKP_DEKSET_FILE` (mounted read-only), and `encryption.replicas` is now configurable. Without a dekset the proxy mints an ephemeral DEK (only safe on a fresh, empty database, single replica). A shared wrapped DEK set — minted by the proxy's backfill tool — is required to migrate an existing populated Keycloak and to run HA; a guard rejects `replicas > 1` without one. Render-time validation requires each mode's settings and rejects unknown values. helm-unittest covers all auth modes, the ServiceAccount/token toggle, and the dekset/replicas paths. Uses `keycloak-kms-proxy` v0.2.0. ### Release note ```release-note feat(keycloak): Vault Kubernetes/AppRole auth, a dedicated ServiceAccount, and a shared DEK set (migration + HA) for the optional PII-encryption proxy ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Expanded Keycloak KMS proxy Vault authentication to support **token**, **AppRole**, and **Kubernetes** modes. * Added **multi-replica/HA** support via configurable DEK-set reuse, including selecting a custom DEK-set key. * Introduced a dedicated proxy ServiceAccount with **conditional token automount** for Kubernetes auth. * **Bug Fixes** * Added stricter validation for missing/unsupported Vault auth settings. * Prevented Vault credential environment variables from leaking across auth modes. * Updated rendering behavior and test expectations for proxy resources (including KEK/ServiceAccount documents). * **Chores** * Updated the Keycloak KMS proxy container image version. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Алексей Артамонов
|
f700c9043c |
fix(monitoring): backfill release label onto pre-existing VM/VL storage PVCs
Re-review on #3094 (IvanHunters, MAJOR): the volumeClaimTemplate label lands on PVCs only at creation — the StatefulSet controller never re-labels existing PVCs and vm-operator's PVC reconcile is resize-only. So on an already-deployed cluster (exactly the #3091 population) the pre-upgrade VM/VL storage PVCs stay unlabeled and the cleanup hook's selector misses them. Add migration 49 (mirrors the ClickHouse-keeper backfill in migration 48 for the sibling #3072). It relabels the pre-existing vmstorage/vmselect/vlstorage PVCs of every cozystack-managed VMCluster/VLCluster with apps.cozystack.io/application.name read from the CR's own spec.managedMetadata, so only monitoring's clusters are touched (a user's own VMCluster carries no such label) and the value always matches what the cleanup hook selects. Best-effort like #48: a skipped relabel is no worse than the pre-existing leak, so transient apiserver errors never abort the platform upgrade. Bump platform targetVersion 49 -> 50. Also correct the misleading comments (IvanHunters MINOR): spec.managedMetadata is never applied to PVCs by vm-operator at any current version, so the claim-template label is the only mechanism — do not imply a v0.71.0 crossover. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
2a78e7b247 |
fix(monitoring): match cleanup PVCs by release label alone, drop managed-by
Re-review on #3094 (IvanHunters): the AND selector kept managed-by=vm-operator, which on the shipped vm-operator v0.68.4 is not reliably present on the storage PVCs (it lives on the StatefulSet selector/pod labels, not the claim template), making the delete fragile. Match by apps.cozystack.io/application.name=<release>-system alone: that label is stamped on the VM/VL claim templates (67b087ab) and copied onto the PVCs by IntoSTSVolume even on v0.68.4, is already release-scoped, and is absent from the CNPG db PVCs (grafana-db/alerta-db carry only policy.cozystack.io + cnpg.io/*), so those stay untouched. Regression guard now forbids re-adding managed-by to the selector. Also apply the review's discipline items: the cleanup Job keeps only before-hook-creation (a Failed run survives for post-mortem; hook-succeeded stays on SA/Role/RoleBinding, cf. #3072) and sets backoffLimit: 0 (best-effort, no retry). Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
e01b725a48 |
fix(monitoring): stamp release-scoping label on VM/VL storage PVCs so cleanup matches
Re-review on #3094 (IvanHunters, lexfrei): the release-scoped selector added in ba5797f4 was a silent no-op on the shipped vm-operator v0.68.4. The apps.cozystack.io/application.name label lived only under the VMCluster/VLCluster spec.managedMetadata.labels, and v0.68.4 does not propagate managedMetadata onto PVCs (that lands only in v0.71.0). So the delete matched nothing and the #3091 storage leak stayed open. Stamp the label where it actually reaches the PVCs: on spec.{vmselect,vmstorage,vlstorage}.storage.volumeClaimTemplate.metadata.labels of the VMCluster/VLCluster CRs (packages/system/monitoring). vm-operator copies claim-template labels onto the PVCs (IntoSTSVolume) even on v0.68.4, so the existing cleanup selector (managed-by=vm-operator,apps.cozystack.io/application.name= <release>-system) now matches and stays scoped to this release. Verified by rendering the CRs (label resolves to monitoring-system, matching the hook). Also add the explicit subjects[].namespace on the cleanup RoleBinding (lexfrei non-blocking) and a helm-unittest suite asserting the label lands on all three storage claim templates. Corrected the misleading test comment about how the label reaches the PVCs. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
69c9e69ff4 |
fix(monitoring): scope VM PVC cleanup selector to this release
Address IvanHunters review on #3094: the PVC delete selector was release-agnostic (-l managed-by=vm-operator), matching the VM storage PVCs of every VMCluster/VLCluster in the namespace. If a namespace ever hosted more than one monitoring release, tearing one down would delete the other's VictoriaMetrics/VictoriaLogs storage. Scope the selector to this release via apps.cozystack.io/application.name= <release>-system, the ownership label the monitoring chart stamps on the VM CRs' managedMetadata and vm-operator propagates onto the PVCs — mirroring the merged seaweedfs hook's release scoping. deletecollection is still not needed: kubectl delete -l lists then deletes individually. Tests: assert the release-scoped selector and add a regression guard that the delete never falls back to the bare managed-by=vm-operator selector. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
b5d1279ccd |
fix(monitoring): make cleanup deletes non-blocking and guard delete scope
Address review on #3094: - Add --wait=false to both kubectl delete calls so PVC finalizers (pvc-protection on pods still terminating) can never block the post-delete hook and stall the HelmRelease teardown. - Add activeDeadlineSeconds: 120 on the cleanup Job as a backstop. - Tests: assert the non-blocking flag, plus negative assertions that the PVC delete stays label-scoped (no --all) and the secret delete stays name-scoped (no -l/--all). deletecollection RBAC verb intentionally not added: kubectl delete -l lists then deletes individually, so the existing delete verb suffices. BREAKING CHANGE: disabling monitoring now permanently removes its VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they hold. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Алексей Артамонов
|
73937ddff9 |
fix(monitoring): clean up orphaned storage PVCs and TLS secrets on tenant module delete (#3091)
When the monitoring tenant module is disabled, the vm-operator storage PVCs (VictoriaMetrics/VictoriaLogs) and the alerta/grafana TLS secrets were left orphaned. Add a post-delete cleanup hook that deletes the PVCs by the managed-by=vm-operator label and the two TLS secrets by name, plus a test target and helm-unittest suite (the package had neither). CNPG PVCs are not targeted (CNPG cleans them). The cleanup Job is hardened (non-root, read-only root fs, dropped caps, resource limits); failures are logged, never blocking teardown. NOTE: disabling monitoring now permanently removes its storage PVCs and the data they hold. Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
f9afc73096 |
fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3181)
Platform log ingestion could fail silently: fluent-bit shipped to vlinsert-generic while the monitoring HelmRelease reported Ready, and a vlinsert-generic ExternalName in cozy-monitoring dead-ended at a tenant-root VictoriaLogs that did not exist when the root tenant had monitoring disabled. Logs were dropped with no failing signal. Three complementary changes: - Readiness (do not stay silently Ready): the monitoring-system HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on its VLCluster (status.updateStatus), so a missing/unhealthy log store makes the release not-ready instead of flipping Ready as soon as the CR is applied. Mirrors the seaweedfs CNPG pattern. - No empty-log-store hole: system/monitoring rejects an empty logsStorages at render time (helm fail). With no entry no VLCluster is created, the readiness gate has nothing to match, and fluent-bit would silently drop logs — so this is made a loud misconfiguration rather than a documented footgun. - No silent fallback: the cozy-monitoring -> tenant-root vlinsert/vminsert ExternalName redirects in cozystack-basics are now created only when the root tenant hosts monitoring, gated on _cluster.monitoring-enabled — fed from a new platform value monitoring.rootEnabled (default true, preserving the historical always-create behaviour). monitoring.rootEnabled is the single render-time source of truth: changing it re-renders cozystack-basics (helm-controller watches the cozystack-values Secret it flows into). A live `lookup` of Tenant/root spec.monitoring is deliberately NOT used: cozystack-basics is an operator-generated HelmRelease with drift detection off, so it re-renders only on install/upgrade/values-change — never on a reconcile tick or when a lookup result changes — and a lookup would go stale in both directions. Operators must keep monitoring.rootEnabled in sync with Tenant/root spec.monitoring. Covered by helm-unittest across all four charts: readiness gate fields, empty-vs-populated logsStorages contract, ExternalName present/absent and null-guard contract, and the platform wiring into the cozystack-values Secret. The CEL health predicates are exercised live, not in helm-unittest (which cannot evaluate CEL). Adds the missing `test:` target to packages/extra/monitoring/Makefile so its suite runs in CI. Known limitation (follow-up): the legacy cozystack.monitoring PackageSource that installs system/monitoring directly into cozy-monitoring cannot carry the readiness gate, because the engine's ComponentInstall/HelmRelease builder and the vendored helm-controller API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not referenced by any active bundle; full coverage needs a helm-controller bump plus engine support. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Kirill Ilin
|
158e85af85 |
chore(keycloak): bump encryption proxy image to 0.2.1
Assisted-By: Claude AI Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Myasnikov DaniilandClaude Fable 5
|
dea35f7432 |
Merge origin/main into feat/e2e-chainsaw-example (post-#3233)
Main's #3233 landed mid-reconciliation and touched the two OIDC bats again: nodeGroups (and storageClass) dropped from the render-side test CRs — the schema default {} now renders one zero-replica MachineDeployment via CAPI defaulting, so the tests provision zero worker DataVolumes (#3231). Resolve the modify/delete as deleted and mirror the fixture change (and its rationale) into the ported kubernetes-oidc-system / kubernetes-oidc-customconfig Chainsaw suites. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
IvanHunters
|
0c7b598304 |
fix(apps/kubernetes): honour nodeGroup minReplicas in MachineDeployment.spec.replicas (#3233)
Refs #3231 (parts 1 + 2 addressed here; part 3 — intra-file `cozy_wait_linstor_pool_free` gate + orphan `default-chk-clickhouse-test-keeper-cluster1-0-2-0` PVC cleanup — deferred to a separate follow-up PR against `hack/e2e-apps/run-kubernetes.sh`). **Do not auto-close the issue on merge** — the hardening work is a distinct concern and belongs on its own tracker entry. ## Root cause chain (recap from the issue) `packages/apps/kubernetes/templates/cluster.yaml` hardcoded `MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's `minReplicas`. The two render-side `kubernetes-oidc-{system,customconfig}` bats set `minReplicas: 0` and delete their `Kubernetes` CR without waiting for readiness — but the chart still spun up 2 × 20 GiB `replicated` (DRBD) worker `DataVolume`s each. Their create/delete churn overlapped `kubernetes-previous`, which then saw slow LINSTOR provisioning (my cozyreport from run 28843794670 shows a ~6-minute gap between the `Kubernetes` CR being created and the CDI `ExternalProvisioning` event actually firing for the worker root disk); the tenant node registered late; Cilium's CNI init missed the 12-minute node-join deadline; and the test failed with `cni plugin not initialized`. ## Changes Two edits, applied together — the chart edit is a no-op without the test edit, and vice-versa (dropping `nodeGroups` falls back to the chart helper's default `md0`, which itself has `minReplicas: 0`, so both pieces are needed for zero workers to actually render). ### 1. `packages/apps/kubernetes/templates/cluster.yaml` ```diff spec: clusterName: {{ $.Release.Name }} - replicas: 2 ``` The field is **omitted entirely** so the cluster-autoscaler owns it alone. CAPI's MachineDeployment defaulting webhook seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create (documented at https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling — *"if it's a new MachineDeployment or MachineSet, use min size"*), and Server-Side-Apply field-manager ownership then keeps chart upgrades from overwriting whatever value the autoscaler patched in the meantime. Emitting the field from the chart at all — even with the semantically-correct `{{ $group.minReplicas }}` — would keep the helm-controller ↔ cluster-autoscaler field-ownership fight alive: helm-controller applies with SSA + force-conflicts (https://fluxcd.io/flux/components/helm/helmreleases/), so every routine `helm upgrade` re-asserts the chart's chosen value and drains whatever the autoscaler had patched in the meantime. ### 2. `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats` Drop the explicit `nodeGroups:` block and the paired `storageClass:` from the render-side CR bodies. The tests only assert on the KamajiControlPlane and the cozy-realm Keycloak objects — worker plumbing was never part of their coverage. After the chart fix, the chart helper's default `md0` (with `minReplicas: 0`) renders a MachineDeployment with `spec.replicas: 0` — the resource is still emitted for API-surface completeness but never creates a KubevirtMachine, never provisions a DataVolume, and never touches DRBD. ## Verification ### helm-unittest ``` Charts: 1 passed, 1 total Test Suites: 16 passed, 16 total Tests: 171 passed, 171 total ``` No suite asserted on `MachineDeployment.spec.replicas` directly, so nothing needed updating. ### helm template — verify the omit With `tests/values/common.yaml` (which has `minReplicas: 0`), the rendered MD carries the `cluster-api-autoscaler-node-group-min-size: "0"` annotation and **no `spec.replicas` field**. CAPI's defaulting webhook will fill in `spec.replicas: 0` on Create — no worker DataVolumes provisioned until scale-out. With a user CR that sets `minReplicas: 2` (the shape the `kubernetes-latest` / `kubernetes-previous` bats CRs use), the rendered MD carries the `cluster-api-autoscaler-node-group-min-size: "2"` annotation and again **no `spec.replicas` field**. CAPI's defaulting webhook will fill in `spec.replicas: 2` on Create — byte-identical topology to the pre-PR chart's fresh-install state (2 workers), but without the recurring drain-on-upgrade cycle. ### Render-side render walk `helm template` of the OIDC bats CR bodies (after this PR): 1 MachineDeployment with **no chart-managed `spec.replicas`** and `min-size: "0"` on the autoscaler annotation. CAPI seeds `replicas: 0` on Create, 0 KubevirtMachines emitted, 0 DataVolume provisioning triggered. Test assertions on KamajiControlPlane / KeycloakClient / KeycloakClientScope / AuthenticationConfiguration Secret are unchanged. ## Upgrade impact Existing customers stop losing workers on `helm upgrade`. Prior to this PR the chart re-asserted `MachineDeployment.spec.replicas: 2` on every Helm action, so a customer running with the autoscaler at, say, `replicas: 8` would see 6 workers drained on every routine Cozystack platform bump and then wait for the autoscaler to scale back up in response to unschedulable Pods. With `spec.replicas` no longer emitted by the chart, SSA field-manager ownership preserves the autoscaler's value across upgrades unconditionally. Fresh installs with `nodeGroups: {}` (the schema default) transition from "2 workers by default" to "0 workers until scale-out". The chart helper's default md0 carries `minReplicas: 0`, which is what CAPI's defaulting webhook seeds `spec.replicas` from. The `values.yaml` / README / helper docstring / regenerated Go type doc + openAPISchema call this out explicitly; the release-note repeats it below. See finding 5 in the paired self-review for the specific `addons.ingressNginx.enabled: true` interaction operators need to be aware of on fresh installs. ## Test plan - [x] `helm unittest packages/apps/kubernetes/` — 171/171 passing. - [x] `helm template` with `tests/values/common.yaml` (minReplicas: 0) — MD.replicas: 0. - [x] `helm template` with a `minReplicas: 2` override — MD.replicas: 2 (byte-identical to pre-PR). - [ ] E2E: `kubernetes-latest` unchanged (spawns its two workers as before), `kubernetes-oidc-{system,customconfig}` render zero workers, `kubernetes-previous` no longer flakes on LINSTOR pool pressure originating from the earlier OIDC suite. ## Notes - Part 3 of #3231 (intra-file `cozy_wait_linstor_pool_free` gate between kube sub-tests and the orphan clickhouse keeper PVC cleanup) is **not** in this PR — worth a separate targeted change to `hack/e2e-apps/run-kubernetes.sh`, and it's a hardening-only improvement rather than a root-cause fix. Happy to send it as a follow-up if wanted. ## Release note ```release-note fix(apps/kubernetes): stop the chart from managing `MachineDeployment.spec.replicas`; the cluster-autoscaler owns the field alone. CAPI's defaulting webhook now seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create, and SSA field-manager ownership keeps subsequent `helm upgrade` actions from overwriting whatever value the autoscaler patched. Two behavioural changes worth calling out: - Existing customers: `helm upgrade` no longer drains workers on every platform bump — the autoscaler-held `spec.replicas` is preserved unconditionally. Prior to this change the chart hardcoded `replicas: 2`, which drained back to 2 on every reconcile action. - Fresh installs with `nodeGroups: {}` (the schema default) no longer come up with two workers by default. The chart helper's default md0 carries `minReplicas: 0`, and CAPI's defaulting webhook seeds `spec.replicas` from that. Operators enabling `addons.ingressNginx.enabled: true` on such a CR must either supply an explicit nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or accept that the cluster-autoscaler will bring the default md0 up in response to the ingress-nginx controller Pods becoming Pending on install. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Rendered `MachineDeployment` manifests no longer set `spec.replicas`, avoiding chart upgrades overwriting autoscaler-managed sizing. * **Documentation** * Clarified `nodeGroups: {}` behavior: a default `md0` is emitted with `minReplicas: 0`/`ingress-nginx` role, and workers are provisioned only after an autoscaler-triggering unschedulable Pod or manual scaling; updated related storage and ingress guidance. * **Tests** * Strengthened default node group rendering assertions and adjusted OIDC e2e manifests to reduce CI provisioning churn. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |