admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-09 05:25:14 +09:00
Code Issues Packages Projects Releases Wiki Activity
4,932 Commits 569 Branches 294 Tags
3fb8ec5b0ea4d505cfdfd72f0917cb6bb138ee65
Commit Graph
704 Commits
Author SHA1 Message Date
IvanHunters d4f59af624 chore(monitoring): sync PackageSource comment with removed Keycloak groups 
Update the oidc-variant comment on
`packages/core/platform/sources/monitoring-application.yaml` so it
no longer claims `spec.oidc.mode: System` renders three
KeycloakRealmGroups. The dependsOn edge and the fail-close guard
in `templates/grafana/oidc-keycloak.yaml` stay unchanged: the
KeycloakClient + KeycloakClientScope path still needs the
`v1.edp.epam.com` CRDs from `cozystack.keycloak-operator` to be
registered before the monitoring chart reconciles.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-15 15:31:45 +03:00
Ivan Okhotnikov b46543e5f2 fix(monitoring): defer platform GrafanaAdmin lever to a future phase 
The MAJOR finding from the cozy-review agent: the platform-side
`oidc.grafanaAdmin: true` documented in the PR body and in
docs/oidc-grafana.md never actually reached the platform Grafana's
HelmRelease. `Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin`
was set in packages/core/platform/templates/bundles/system.yaml, but
`monitoring-system` (and `monitoring`) have no `install:` block in
packages/core/platform/sources/monitoring-application.yaml — only
`monitoring-rd` does. PackageReconciler skips components without an
`install:` block before ever reaching the assignment that copies
`pkg.Spec.Components[].Values` onto `hr.Spec.Values`
(internal/operator/package_reconciler.go around lines 197-201 and
292-293). The values-override dictionary was silently dropped, no
HR carried `oidc.grafanaAdmin: true`, and the platform Grafana
rendered with the chart default (`false`). The
`bundles_monitoring_grafana_admin_test.yaml` unit test only asserted
against the emitted Package CR spec — exactly the layer where the
value still existed but was about to be discarded — so the broken
propagation slipped past.

Trim the broken promise from Phase 1. The chart-side helper
(`monitoring.oidc.allowAssignGrafanaAdmin`) stays as-is so a future
phase that routes the override through the tenant-root chart's
monitoring HR values (or adds an `install:` block on the
`monitoring-system` component) lands on a green chart-side contract.
The helm-unittest that verifies chart-side emission stays too, but
with a docstring calling out that end-to-end wiring is deferred.
Everything else that promised the lever works — bundle line, docs,
platform bundle test — is either removed or rewritten to match
reality.

Also picks up two MINOR findings from the same review:

* Drop the misleading `secret-hash` annotation on the OIDC
  client-secret Secret. The comment claimed it "rotates the
  KeycloakClient CR and therefore the Grafana Deployment env when
  the client-secret changes", but nothing downstream watches an
  annotation on the Secret itself — no rotation is triggered. The
  `lookup`-based fallback already keeps the value stable across
  reconciles, so no rotation is even needed. Removing the dead
  annotation avoids planting a broken assumption for future readers.

* Add `release.cozystack.io/helm-install-timeout: "15m"` on the
  monitoring cozyrds ApplicationDefinition. Under `mode: System` the
  chart now emits, on top of the existing Grafana + Alerta +
  VMCluster + CNPG stack, four additional Keycloak CRs whose
  reconciliation is asynchronous against an operator in
  `cozy-keycloak` (not the tenant namespace). Flux's default 5m
  install window was already tight for the pre-OIDC stack; 15m
  matches the precedent set by tenant-rd/kubernetes-rd cozyrds.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:45 +03:00
Ivan Okhotnikov be52e80289 fix(monitoring): split monitoring-application PackageSource into default/oidc variants 
Follow-up to the previous self-review-fix commit: the unconditional
`- cozystack.keycloak-operator` in dependsOn was itself a
CRITICAL cold-install regression on any cluster where
`authentication.oidc.enabled: false` (the default). The
keycloak-operator Package is only emitted by
`packages/core/platform/templates/bundles/system.yaml:112` when
that flag is on — a hard dep against it from the default variant
would leave `cozystack.monitoring-application` stuck on
DependenciesNotReady forever, cascading a
`cozystack-platform` HR timeout that hits every OIDC-off customer.

Fix by mirroring the cozystack-engine variant split:

- `default` variant keeps the baseline reconciliation edges
  (no keycloak-operator dep).
- `oidc` variant additionally waits for
  `cozystack.keycloak-operator` so the v1.edp.epam.com CRDs are
  registered before the monitoring chart reconciles and the
  `.Capabilities.APIVersions.Has` guard in
  templates/grafana/oidc-keycloak.yaml can safely emit
  KeycloakClient/Scope/Groups.

The bundle template swings between variants on
`.Values.authentication.oidc.enabled`, mirroring the
cozystack-engine invocation two lines above. The
`monitoring-system` component override
(oidc.grafanaAdmin: true) is preserved across both branches — the
values field is a no-op when mode: None (no auth.generic_oauth
block is emitted at all), so keeping it identical avoids two
divergent code paths.

Also widened the platform helm-unittest coverage to lock:

- default variant does NOT depend on keycloak-operator (permanent
  DependenciesNotReady guard).
- oidc variant DOES.
- Bundle emits the right variant based on
  authentication.oidc.enabled AND keeps the GrafanaAdmin uplift
  on the monitoring-system component in either branch.

83 platform tests + 23 monitoring tests still green.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:44 +03:00
Ivan Okhotnikov ffeb8e5e4f fix(monitoring): close self-review blockers for the OIDC PR 
Address the two blocking-class findings from the cozy-review pass on
this PR:

(1) Missing PackageSource dependency on cozystack.keycloak-operator
    (MAJOR cold-install race).

    Without this edge, the monitoring chart can reconcile before the
    v1.edp.epam.com CRDs are registered on a fresh cluster — the
    `.Capabilities.APIVersions.Has` guard in
    templates/grafana/oidc-keycloak.yaml then silently drops all
    Keycloak-side objects for mode: System, and Grafana's
    auth.generic_oauth block points at a client Keycloak never
    provisions. Add the ordering edge, plus a
    sources_monitoring_application_dependson_test.yaml regression
    guard so a future edit cannot drop it silently.

(2) allow_assign_grafana_admin driven by `.Release.Name ==
    monitoring-system` (MINOR fragile coupling).

    Replace the release-name sniff with an explicit
    `oidc.grafanaAdmin: false` values field, and flip it on for the
    platform Grafana release via
    Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin
    in packages/core/platform/templates/bundles/system.yaml —
    mirroring the pattern monitoring-agents already uses for
    per-component overrides. Tenant Grafana releases inherit the
    chart default false and stay at org-level Admin.

    Two new helm-unittest cases in packages/system/monitoring/tests/
    lock the values-driven behaviour on both sides
    (grafanaAdmin=true → Admin uplift, default → tenant stays at org
    Admin), plus bundles_monitoring_grafana_admin_test.yaml in
    packages/core/platform/tests/ regression-guards the Package's
    component override so the platform release keeps its uplift on
    future edits.

Also document the mode-toggle group-membership loss in
docs/oidc-grafana.md — mode: System → CustomConfig/None deletes the
three chart-owned KeycloakRealmGroups; membership is not restored on
toggle-back. Not a bug, but the operator UX cost of the toggle
deserves a bullet in Failure modes.

Regenerated artifacts (cozyvalues-gen + hack/update-crd.sh):
  packages/extra/monitoring/values.schema.json
  packages/extra/monitoring/README.md
  packages/system/monitoring-rd/cozyrds/monitoring.yaml

Test suites still green: packages/system/monitoring 23/23,
packages/core/platform 81/81.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:44 +03:00
Aleksei Sviridkin 249f7b5a5e feat(platform, ingress): propagate operator wildcard certificate to per-tenant termination points (#2990) 
## What this PR does

The operator-provided wildcard certificate
(`publishing.certificates.wildcardSecretName`) currently serves the root
tenant only: everything is same-namespace, so the root ingress
controller and Gateway read the operator's TLS Secret directly. Child
tenants run their own ingress controller / Gateway in their own
namespace under a default-deny policy and cannot read that Secret across
namespaces, so they fall back to per-host ACME.

This propagates the operator wildcard to per-tenant termination points
by replicating the Secret into each tenant namespace that owns one,
keeping the root-MVP invariant intact: the wildcard is deliberately
distributed to tenant namespaces, but no tenant RBAC is widened — every
consumer reads only its own-namespace copy.

A new controller in `cozystack-controller` does the replication with no
extra operator input. It reads the same platform values channel the
consumers read (`cozy-system/cozystack-values`): the wildcard Secret
name from `_cluster.wildcard-secret-name` and the publishing namespace
from `_cluster.expose-ingress`. It mirrors that Secret into every tenant
namespace whose `namespace.cozystack.io/ingress` or
`namespace.cozystack.io/gateway` label equals its own name — i.e. the
namespace runs its own controller / Gateway. Because the source is
derived from the same value that makes the consumers reference it, the
replica is created whenever the consumers expect it — no manual
labelling and no upgrade hazard where a child controller references a
Secret that nothing creates.

Replicas are refreshed on rotation and pruned when a namespace stops
terminating TLS. Teardown of every replica happens on exactly one
trigger: clearing `wildcardSecretName` (explicit disable). A source that
is merely absent or mistyped, or a transiently-missing values channel,
leaves existing replicas in place, so a brief gap, a delete+recreate
rotation, or a misconfigured publishing namespace never drops tenant
TLS. The platform controller's own ServiceAccount gains Secret writes;
tenant RBAC is untouched.

The controller does not cache every Secret in the cluster: the manager's
Secret informer is scoped to managed replicas and the values channel
only, and the operator source / foreign collisions are read through the
uncached APIReader. The source is not watched (its name is dynamic), so
an in-place rotation is picked up by a bounded periodic resync while the
publishing tenant serves the source with no lag.

On the consumer side, each tenant ingress controller now sets
`--default-ssl-certificate` to the wildcard Secret in its own namespace
(`.Release.Namespace/<name>`), so a child tenant serves the replica
instead of minting per-host ACME. The Gateway path already renders
`certMode: existingSecret` with a same-namespace `wildcardSecretRef` for
every tenant, so it consumes the replica with no chart change.

### Why not clustersecret-operator

The issue proposed the shipped `clustersecret-operator`. It materializes
inline template data into selected namespaces; it cannot mirror an
existing source Secret, and it is not installed in any default bundle.
Driving it would mean embedding the certificate and private key as
literal data in a cluster-scoped object rendered from a Helm `lookup` —
that broadcasts key material the design keeps off any shared channel,
and a `lookup` is invisible to the Flux helm-controller digest so it
would not re-render on rotation. Its namespace selector also cannot
express "the namespace whose owner label equals its own name", so it
would over-replicate the key into every tenant namespace. A small
dedicated reconciler targets exactly the termination namespaces and
keeps the key off the values channel.

### Coverage caveat

Replication delivers the certificate bytes, not SAN coverage: a single
`*.<root-host>` does not match `*.<child-apex>`. Operators serving
child-tenant subdomains supply a certificate whose SANs cover those
apexes; the docs spell this out. The replica carries the wildcard
private key into each terminating tenant namespace — the same exposure
as the per-host ACME Secret it replaces, now shared across tenants.

### Tests

- Go fake-client tests for the reconciler: termination-only replication
(ingress-owning and gateway-owning namespaces only, never inheriting
ones, never over the source), copy markers, rotation, prune on namespace
demotion, the non-destructive source-deletion contract, replica
self-heal after out-of-band deletion, the publishing-namespace
self-replication guard, disable-via-config teardown, source rename,
foreign-Secret collision safety (no overwrite, no requeue, does not
block siblings), transient-error requeue, non-TLS source ignored,
absent-channel keeps replicas, publishing-namespace mismatch keeps and
requeues, the cache-scope pin, the enqueue-wiring map, an
`ownsTerminationPoint` table, and the active-only requeue contract.
- helm-unittest for the ingress `default-ssl-certificate` wiring
(including a child-tenant controller pointing at its own-namespace
replica) and for the cozystack-controller ClusterRole's new Secret-write
rule.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(platform): the operator-provided wildcard TLS certificate (`publishing.certificates.wildcardSecretName`) is now served by per-tenant ingress controllers and Gateways automatically — the platform controller replicates it into each tenant namespace that terminates TLS, with no extra operator input and no cross-namespace Secret read
```

Closes #2820

Part of #2811


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Release Notes

* **New Features**
* Wildcard TLS Secret now replicates from the platform to tenant
namespaces that terminate TLS via their own ingress controller or
Gateway, including child-tenant scenarios.

* **Documentation**
* Clarified configuration, timing, and SAN matching requirements for
wildcard replication.
* Updated gateway and ingress-nginx default certificate documentation
for replica mirroring behavior.

* **Bug Fixes**
* Improved handling for missing/invalid source secrets (no unnecessary
pruning) and safe collision behavior.

* **Tests**
  * Added comprehensive reconciliation/replication test coverage.
  * Added Helm unittest target for the controller chart.

* **Chores**
  * Updated controller RBAC to support wildcard TLS Secret replication.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-15 11:32:08 +03:00
Aleksei Sviridkin f24b56327b fix(platform): route OIDC backend calls through the in-cluster Keycloak by default 
authentication.oidc.keycloakInternalUrl defaulted to an empty string, which
left oauth2-proxy performing OIDC discovery against the external Keycloak
hostname. That request resolves through public DNS and lands on the root
ingress controller, which does not serve yet on a fresh install: the root
tenant's IngressClass only exists once tenant-root is ready, and tenant-root
is still reconciling while the dashboard starts. Discovery has nothing to talk
to, the gatekeeper fails, and the dashboard never comes up.

The bundled Keycloak is the only supported provider — the external hostname is
templated as keycloak.<host> and the realm is fixed — so backend-to-backend
calls have no reason to leave the cluster. Default the value to
http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy, the same Service
address keycloak-configure already uses. The dashboard and linstor-gui
gatekeepers then run with --skip-oidc-discovery and send token, jwks, userinfo
and logout requests to the Service, while browser redirects stay on the
external host. Those calls travel as plaintext HTTP between pods, as
keycloak-configure's admin calls to the same address already do.

The value remains overridable for a Keycloak that does not live at the standard
in-cluster address, and setting it back to an empty string restores external
discovery. Existing OIDC clusters that never set it get a gatekeeper rollout
onto the internal path on upgrade.

Enabling OIDC swaps the dashboard's token-proxy container for oauth2-proxy, so
the E2E install now drives that swap to completion: it waits for the auth-proxy
container to appear, then gates on the gatekeeper rollout. Readiness alone
proves nothing there — the HelmRelease is still Ready from the token-proxy
install, and the Deployment keeps the old ReplicaSet Available while a new pod
crashloops, so only the rollout catches a regression back to external discovery.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-14 14:22:46 +03:00
myasnikovdaniil 99a28d82c2 test(e2e): migrate app suite from BATS to Chainsaw and wire into CI (#2826) 
## What this PR does

Migrates the E2E **app** test suite from BATS to [Kyverno
Chainsaw](https://github.com/kyverno/chainsaw) — a declarative,
Kubernetes-native E2E framework (CNCF, part of the Kyverno project;
successor to KUTTL) — and wires it into CI as the replacement for the
per-app BATS loop. This grows the original pilot (postgres + bucket)
into the full suite.

All 22 app suites are ported under `hack/e2e-chainsaw/` and the
`hack/e2e-apps/*.bats` files are removed:

`postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`,
`clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`,
`external-dns`, `kuberture`, `vminstance`, `gateway`,
`kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`,
`kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`.

**Approach**

- **Declarative** suites assert on `status.conditions` and concrete
fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait`
pair that appeared ~100 times collapses into a single `assert` that
polls existence and state together, with a structured diff on failure
and automatic `events`/`describe`/`podLogs` capture via `catch`
(previously only `harbor.bats` did this, by hand).
- **Imperative** suites keep their logic in `script` steps: `openbao`
init/unseal, `kuberture` external-dns split-horizon probes,
`vminstance`, the `gateway` admission/impersonation cases, and
`kubernetes-latest`/`previous`, which wrap the relocated
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up,
LB/NFS/ouroboros checks).
- `gateway` tests derive the tenant apex from the namespace
`namespace.cozystack.io/host` label at runtime, so they are
host-independent.

**CI**

- The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new
`test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`.
- The `chainsaw` binary is added to the e2e-sandbox image.
- `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap +
OpenAPI checks).
- The per-app 3-retry loop is dropped — assertion polling replaces the
fixed-timeout flakiness it papered over.

**Validation**

Ran against a development cluster: the DB/app, storage, and VM suites
pass; the three suites that depend on platform features not present on
that cluster (`gateway`, `kuberture`, `external-dns`) and the two
heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a
freshly installed platform.

Note for reviewers: service-port asserts use the `(ports[*].port)`
projection form rather than a number-literal filter (`` ports[?port ==
`N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal
comparisons.

**2026-07-09: reconciled with main (~520 commits of drift)**

- etcd suite re-ported to the v1alpha2 operator contracts (#2859):
`readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag
contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip
driving `examples/backups/etcd`
- 12 new `ingress-hostname-policy` gateway cases ported (apex derived at
runtime from the tenant-root namespace label)
- post-branch bats drift folded in: `mariadb-single` webhook guard,
kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket
readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant
PVC-reclaim guard (#3059), kuberture fail-loud negations,
SC-fallback-default test (#2872 B1)
- 4 suites that only exist on main since the branch was cut are ported:
`kubernetes-oidc-system`, `kubernetes-oidc-customconfig`,
`securitygroup`, `serviceexposure`
- CI seams: `nightly.yaml` converted to the chainsaw invocation,
`run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR
pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh`
replicated in the Chainsaw global catch

### Release note

```release-note
fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* End-to-end testing is now driven by a unified Chainsaw suite (platform
apps, Kubernetes, networking, security groups, service exposure).
* Test Impact Analysis selects affected suites, with an option to run
the full suite.
* CI produces a JUnit report artifact and collects suite-scoped
diagnostics on failure.
* **Bug Fixes**
* CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and
sandbox bucket state diagnostics.
* Improved teardown reliability with bounded wait + finalizer handling
to avoid hangs.
* **Documentation**
* Updated E2E testing docs and local guidance to reflect the new
Chainsaw-based workflow and suite paths.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-12 11:58:08 +05:00
Myasnikov DaniilandClaude Opus 4.8 302a72f14a Merge origin/main into feat/e2e-chainsaw-example 
Reconciles the talos-image-cache conflicts created when #3254 merged to
main: keep the Chainsaw _lib/ port (hack/e2e-chainsaw/_lib/talos-image-cache.sh,
a superset of #3254's logic + the node-join diagnose helper), drop the
BATS-era hack/e2e-apps/talos-image-cache.sh, and keep the _lib/-path
versions of the mirror manifest, install bats, and unit test. Also picks
up main's zizmor action-pinning (#3223, unblocks pre-commit/zizmor) and
Talos v1.13.6 (#3240).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-10 12:00:01 +05:00
renovate[bot] 5b9b420ffe chore(deps): update container image & action digests 2026-07-10 05:29:23 +00:00
Aleksei Sviridkin 3e9b5600e3 fix(talos): bump Talos to v1.13.6 (CVE-2026-53359) 
Talos v1.13.5 ships Linux 6.18.36, which carries the fix for
CVE-2026-46113 but not the one for CVE-2026-53359. The two are related
KVM x86 shadow-paging bugs allowing a guest to escape to the host, and
both patches are needed for the mitigation to hold. Talos v1.13.6 ships
Linux 6.18.38, the first release on the 6.18 stable line that has both.

Profiles regenerated with hack/gen-profiles.sh v1.13.6. Alongside the
Talos version and the installer reference, this refreshes the pinned
system-extension digests: the firmware extensions move from the 20260519
to the 20260622 build, and intel-ucode stays on its 20260512 tag but
resolves to a new digest because upstream re-pushed that tag.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-09 20:29:40 +03:00
Myasnikov DaniilandClaude Fable 5 1f84d2d49e Merge origin/main into feat/e2e-chainsaw-example (post-#3244) 
Fold the in-sandbox Talos image cache work into the Chainsaw layout:
the new shared helper relocates to hack/e2e-chainsaw/_lib/
talos-image-cache.sh and run-kubernetes.sh sources it from there (the
ouroboros DNS-check no-retry rework merges clean). The OIDC bats are
resolved as deleted again; their Chainsaw fixtures deliberately skip
the talos.imageFactoryURL injection — the helper only emits it when
the cache is up, and the zero-worker render-side tests instantiate no
DataVolume imports for it to redirect.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-09 10:38:35 +05:00
scooby87 f91841fce6 fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3215) 
## What this PR does

Addresses #3181, where platform log ingestion could fail **silently**:
fluent-bit shipped to `vlinsert-generic` while the monitoring
`HelmRelease` reported `Ready`, and a `vlinsert-generic` `ExternalName`
in `cozy-monitoring` dead-ended at a `tenant-root` VictoriaLogs that did
not exist when the root tenant had monitoring disabled. Logs were
dropped with no failing signal.

Three complementary layers make the failure loud instead of silent:

1. **Readiness gate** —
`packages/extra/monitoring/templates/helmrelease.yaml` now sets a
`poller` `waitStrategy` and a `healthCheckExprs` entry on the
`VLCluster` `status.updateStatus`. A missing/unhealthy log store makes
the `monitoring-system` release **not-ready** instead of flipping
`Ready` the moment the CR is applied. Mirrors the existing seaweedfs
CNPG pattern.
2. **No empty-log-store hole** —
`packages/system/monitoring/templates/vlogs/vlogs.yaml` fails the render
on an empty `logsStorages`. With no entry no `VLCluster` exists, the
readiness gate has nothing to match, and fluent-bit would silently drop
logs — so this is now a loud misconfiguration.
3. **No silent fallback** — the `cozy-monitoring → tenant-root`
`vlinsert`/`vminsert` `ExternalName` redirects in `cozystack-basics` are
created only when the root tenant hosts monitoring. The template reads
the live `Tenant/root` `spec.monitoring` via `lookup` (source of truth,
so it cannot drift from the tenant's real state) and falls back to a new
platform value `monitoring.rootEnabled` (via
`_cluster.monitoring-enabled`) when `lookup` is empty (dry-run / first
render). The default preserves the historical always-create behaviour.

Covered by helm-unittest across all four affected charts (141 tests):
readiness-gate fields, empty-vs-populated `logsStorages` contract,
`ExternalName` present/absent fallback contract, and the platform wiring
into the `cozystack-values` Secret. Also adds the missing `test:` target
to `packages/extra/monitoring/Makefile` so its suite runs in CI.

**Known limitation / follow-up:** the legacy platform monitoring release
installed directly into `cozy-monitoring` via the `cozystack.monitoring`
PackageSource cannot carry the readiness gate, because the engine's
`ComponentInstall`/HelmRelease builder and the vendored
`helm-controller` API v1.4.3 lack `waitStrategy`/`healthCheckExprs`.
That source is not referenced by any active bundle; full coverage there
needs a helm-controller bump plus engine support.

### Release note

```release-note
fix(monitoring): surface missing platform VictoriaLogs as not-ready and stop the dangling vlinsert-generic ExternalName from silently dropping logs (#3181)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a platform setting (`monitoring.rootEnabled`) to control whether
root-level monitoring is enabled.
* Monitoring redirect ExternalName services are now rendered only when
root monitoring is enabled (defaulting to enabled).
* **Bug Fixes**
* Ensures an explicitly set `monitoring.rootEnabled=false` is preserved
when generating platform configuration.
* Prevents incomplete monitoring deployments by failing rendering when
`logsStorages` is empty.
* Improves VictoriaMetrics readiness by delaying “ready” until the
operator reports an operational status.
* **Tests**
* Added/expanded Helm-unittest suites covering monitoring wiring,
redirects, readiness gating, and `logsStorages` requirements.
  * Added a `make test` target to run chart unittests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-09 01:55:01 +03:00
Алексей Артамонов f700c9043c fix(monitoring): backfill release label onto pre-existing VM/VL storage PVCs 
Re-review on #3094 (IvanHunters, MAJOR): the volumeClaimTemplate label lands on
PVCs only at creation — the StatefulSet controller never re-labels existing PVCs
and vm-operator's PVC reconcile is resize-only. So on an already-deployed cluster
(exactly the #3091 population) the pre-upgrade VM/VL storage PVCs stay unlabeled
and the cleanup hook's selector misses them.

Add migration 49 (mirrors the ClickHouse-keeper backfill in migration 48 for the
sibling #3072). It relabels the pre-existing vmstorage/vmselect/vlstorage PVCs of
every cozystack-managed VMCluster/VLCluster with apps.cozystack.io/application.name
read from the CR's own spec.managedMetadata, so only monitoring's clusters are
touched (a user's own VMCluster carries no such label) and the value always
matches what the cleanup hook selects. Best-effort like #48: a skipped relabel is
no worse than the pre-existing leak, so transient apiserver errors never abort the
platform upgrade. Bump platform targetVersion 49 -> 50.

Also correct the misleading comments (IvanHunters MINOR): spec.managedMetadata is
never applied to PVCs by vm-operator at any current version, so the claim-template
label is the only mechanism — do not imply a v0.71.0 crossover.

Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
 2026-07-08 22:27:42 +03:00
Alexey Artamonov f9afc73096 fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3181) 
Platform log ingestion could fail silently: fluent-bit shipped to
vlinsert-generic while the monitoring HelmRelease reported Ready, and a
vlinsert-generic ExternalName in cozy-monitoring dead-ended at a
tenant-root VictoriaLogs that did not exist when the root tenant had
monitoring disabled. Logs were dropped with no failing signal.

Three complementary changes:

- Readiness (do not stay silently Ready): the monitoring-system
  HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on
  its VLCluster (status.updateStatus), so a missing/unhealthy log store
  makes the release not-ready instead of flipping Ready as soon as the
  CR is applied. Mirrors the seaweedfs CNPG pattern.

- No empty-log-store hole: system/monitoring rejects an empty
  logsStorages at render time (helm fail). With no entry no VLCluster is
  created, the readiness gate has nothing to match, and fluent-bit would
  silently drop logs — so this is made a loud misconfiguration rather
  than a documented footgun.

- No silent fallback: the cozy-monitoring -> tenant-root
  vlinsert/vminsert ExternalName redirects in cozystack-basics are now
  created only when the root tenant hosts monitoring, gated on
  _cluster.monitoring-enabled — fed from a new platform value
  monitoring.rootEnabled (default true, preserving the historical
  always-create behaviour). monitoring.rootEnabled is the single
  render-time source of truth: changing it re-renders cozystack-basics
  (helm-controller watches the cozystack-values Secret it flows into). A
  live `lookup` of Tenant/root spec.monitoring is deliberately NOT used:
  cozystack-basics is an operator-generated HelmRelease with drift
  detection off, so it re-renders only on install/upgrade/values-change
  — never on a reconcile tick or when a lookup result changes — and a
  lookup would go stale in both directions. Operators must keep
  monitoring.rootEnabled in sync with Tenant/root spec.monitoring.

Covered by helm-unittest across all four charts: readiness gate fields,
empty-vs-populated logsStorages contract, ExternalName present/absent
and null-guard contract, and the platform wiring into the
cozystack-values Secret. The CEL health predicates are exercised live,
not in helm-unittest (which cannot evaluate CEL). Adds the missing
`test:` target to packages/extra/monitoring/Makefile so its suite runs
in CI.

Known limitation (follow-up): the legacy cozystack.monitoring
PackageSource that installs system/monitoring directly into
cozy-monitoring cannot carry the readiness gate, because the engine's
ComponentInstall/HelmRelease builder and the vendored helm-controller
API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not
referenced by any active bundle; full coverage needs a helm-controller
bump plus engine support.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-07-08 22:27:22 +03:00
Myasnikov DaniilandClaude Fable 5 9297e51170 Merge origin/main into feat/e2e-chainsaw-example 
Reconcile the Chainsaw e2e migration with ~500 commits of main drift:
- keep main's run-kubernetes.sh helpers (kubectl_wait_retry, tenant
  drain, LINSTOR pool-free wait, SC-fallback probe) under the branch's
  _lib/ rename with the Chainsaw finally/snapshot deltas re-applied
- resolve modify/delete on 10 drifted hack/e2e-apps/*.bats as deleted;
  their post-branch assertion drift is re-ported into the Chainsaw
  suites in follow-up commits
- drop the 4 new main-side bats (kubernetes-oidc-*, securitygroup,
  serviceexposure) from _lib/ relocation; ported as Chainsaw suites in
  follow-up commits
- docs/agents/e2e-testing.md: merge main's plan-job TIA wording with
  the Chainsaw suite phrasing

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:04:08 +05:00
myasnikovdaniil cece5c23b5 feat(release): immutable tags & rc→stable promotion (#2677) (#3017) 
## What this PR does

Implements the immutable-tag + rc-promotion flow from #2677. A **stable
release becomes a renamed release-candidate**: the bytes shipped as
`vX.Y.Z` are bit-for-bit the bytes built and e2e-tested as
`vX.Y.Z-rc.N`. No tag is ever force-moved, and stable is never rebuilt —
it is *promoted* by retagging the rc's existing images.

Layered onto the build matrix from #2937/#2983 (this PR is **stacked on
`refactor/build-matrix-2937`** and must merge after it).

## The five force-retag sites, removed

| Site | Before | After |
|---|---|---|
| `tags.yaml` api/apps/v1alpha1 tag | `git tag -f` / `push -f` |
write-once (create-if-absent, fail if it would move) |
| `tags.yaml` release-X.Y.Z branch | `git branch -f` / `push -f` |
compare-before-force (no-op if unchanged; staging branch only) |
| `pull-requests-release.yaml` stable tag | `git tag -f` / `push -f` |
write-once at the PR merge commit (force impossible by construction) |
| `pull-requests-release.yaml` maintenance branch | `updateRef
force:true` | fast-forward-only |
| `auto-release.yaml` patch tags | delete-recreate (cron) | **workflow
deleted** — stable only via explicit promote |

## Version decoupling (the enabler)

The operator baked its version into the image at build time, so an rc
image self-reported the rc string — blocking retag-promotion. It now
reads `COZYSTACK_VERSION` from the environment (threaded via
`cozystackOperator.platformVersion` → Deployment env, stamped by `make
manifests`), falling back to the build-time value when unset. The same
image bits can report any release name. The only runtime reader is the
telemetry metric `cozy_cluster_info{cozystack_version=...}`.

## Promotion flow

`promote-rc.yaml` (`workflow_dispatch`, `rc_tag=vX.Y.Z-rc.N`):

1. Validate the rc release exists and the stable tag does not.
2. `hack/promote-retag.sh` reads the rc's digest-pinned image refs from
`packages/*/*/values.yaml` and `skopeo copy`s each — **by digest** — to
`:vX.Y.Z` and `:latest`, verifying with `skopeo inspect`.
3. Rewrite the cosmetic `-rc.N` substring in vendored tags to the stable
version (the `@sha256` wins regardless), restamp the version-stamped
assets (operator manifests, cozypkg, openapi; the heavy Talos assets are
copied verbatim from the rc draft), open a `release-X.Y.Z` PR.
4. Merging the PR reuses the existing release-PR e2e and
`pull-requests-release.yaml` finalize to cut the **write-once** stable
tag at the merge commit and publish the release. Squash is disallowed
(the tag needs a real merge commit).

`nightly.yaml` cuts write-once `*-nightly.<date>` tags (gated by
`NIGHTLY_ENABLED`); `retention.yaml` keeps the newest 14 per line
(dry-run by default).

## Validation

**Local end-to-end — the core claim is proven, not asserted.** The whole
design rests on "copy-by-digest to a new tag preserves the digest, so
stable == the e2e-tested rc, bit-for-bit." This was exercised against a
real registry (`ttl.sh`) with real `skopeo`:

- Pushed a multi-arch image under a `:v1.4.0-rc.2` tag, built a temp
tree exercising all three `values.yaml` digest shapes (single `image:`
string, split `repository`/`tag`/`digest` map, and the
`platformSourceRef` OCI artifact), then ran the actual
`hack/promote-retag.sh v1.4.0`.
- **Result: `:v1.4.0-rc.2`, `:v1.4.0`, and `:latest` all resolve to the
identical digest** (`sha256:fd8d9aa6…`), across all 17 platform
manifests. The script's own `skopeo inspect` post-check passed, and an
independent re-inspection confirmed it.

Local validation caught (and this PR fixes) two bugs that only surface
against a real registry — never in static lint:
1. `skopeo copy --multi-arch all --all` — mutually exclusive flags,
fatal error; every retag would have failed on CI. Now `--multi-arch
all`.
2. The same digest was retagged twice when an image appeared in two
value shapes; the ref set is now deduped on the canonical `repo@digest`.

**Other local checks:** `go test ./pkg/version` + `go build`/`vet`;
`helm template` renders `COZYSTACK_VERSION` in all 3 operator variants
(omitted when unset); `make manifests` stamps the version into the
install assets; `shellcheck` clean on `promote-retag.sh`; `actionlint`
clean and `act -l` resolves the job graph on all workflows; the rc-tag
parser and nightly version-math unit-tested (accept/reject +
minor-vs-patch bumps).

**Still CI-only (cannot be exercised offline):** the full workflow
runtime — github-script API calls, OCIR auth, the rc-release /
staging-branch preconditions, the nightly `base_ref` push mechanics, and
self-hosted runners. The *logic* inside the steps is unit-tested; the
*orchestration* is not. Treat it as unproven until the workflows run.

## Out of scope / required follow-up (needs repo admin — not doable from
a PR)

- Tag-protection rules: `v*` and `api/apps/v1alpha1/*` create-only by
the CI app, no delete/update; "limit branches/tags updated in a single
push" = 1.
- Enable `NIGHTLY_ENABLED` (and `RETENTION_APPLY` when ready) repo
variables.
- Registry-side pruning of `*-nightly.*` image tags (no OCIR delete
parity yet — tracked TODO in `retention.yaml`).

## Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added nightly publishing (mirror-by-digest, disk build, e2e
validation) and nightly retention pruning.
  * Added RC-to-stable promotion via digest retagging (no rebuild).
* Operator/installer and console now expose version/platform metadata,
including runtime override via `COZYSTACK_VERSION`.
* **Bug Fixes**
* Enforced write-once tag behavior and fast-forward-only maintenance
updates across release workflows.
* Made nightly mirroring/selection and retention pruning more selective
and safer.
* **Tests**
* Added Bats coverage for nightly mirroring and RC retagging, plus new
Helm/unit checks.
* **Documentation**
* Updated the release model to center RC promotion and tag immutability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-07 17:46:06 +05:00
Myasnikov DaniilandClaude Opus 4.8 6d3d4908de test(installer): cover the generic variant in the version test 
The COZYSTACK_VERSION env test pinned talos and hosted but not generic,
leaving the generic branch of the endpoint-env interleave (KUBERNETES_
SERVICE_HOST/PORT from cozystack.apiServerHost/Port) unguarded. Add a
generic + platformVersion case so a refactor can't silently drop
COZYSTACK_VERSION or mis-render the generic endpoint block.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-06 16:49:42 +05:00
Andrey Kolkov d0740ed8c7 test(etcd): gate S3 backup round-trip behind ETCD_E2E_S3_ROUNDTRIP 
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-06 14:51:48 +04:00
Andrey Kolkov 011d6d1639 fix FUs 
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-06 14:41:33 +04:00
Andrey Kolkov f2866edcf0 review fixes 
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-06 14:41:33 +04:00
Andrey Kolkov 087a8c4be9 chore(etcd-operator): bump to v0.5.2 
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-06 14:41:33 +04:00
Andrey Kolkov cc5d688e15 feat(platform): migrate to etcd-operator v1alpha2 
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-06 14:41:33 +04:00
Myasnikov Daniil 22a9437aa6 test(installer): cover COZYSTACK_VERSION env rendering 
The COZYSTACK_VERSION env is what lets a release-candidate image be promoted to
stable by retagging alone, and it is interleaved with the per-variant endpoint
env under one conditional. Add a helm-unittest (and the package test target so
`make unit-tests` runs it) pinning both axes: talos/hosted x platformVersion
set/unset, asserting COZYSTACK_VERSION presence and the variant endpoint block.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-06 10:03:48 +05:00
Myasnikov DaniilandClaude Opus 4.8 fd7fcbda4e refactor(operator): read platform version from env (decouple from build) 
The operator's platform version was baked into the image at build time
(`-ldflags -X pkg/version.Version` from `git describe`), so a
`vX.Y.Z-rc.N` image literally carried the rc string. That blocks promoting
a release-candidate to stable by retag: the bits would still self-report
the rc version.

Resolve the version from the COZYSTACK_VERSION env var at runtime, falling
back to the build-time value when unset (local builds still work). The
installer chart threads `cozystackOperator.platformVersion` into the
operator Deployment as that env var, and `make manifests` stamps it into
the kubectl-apply release artifacts. The same image bits can now report any
release name, so promotion needs no rebuild.

The only behavioral reader is the telemetry metric
`cozy_cluster_info{cozystack_version=...}`; cozypkg/check-readiness keep
their own build-time Version and are unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-06 10:03:46 +05:00
IvanHunters a1c779854b fix(vertical-pod-autoscaler): migrate admission-controller TLS to cert-manager to stop reconcile-driven ReplicaSet churn (#3178) 
## Summary

The VPA admission-controller webhook TLS Secret is currently rendered by
an in-chart `genCA` + `genSignedCert` template. Those functions are not
idempotent across `helm template` invocations — every Flux HelmRelease
reconcile produces a fresh CA + cert. A `checksum/tls-secret`
podAnnotation on the admission-controller Deployment then flips its
sha, materialising a new ReplicaSet on every reconcile.

Under load — the E2E install phase where cert-manager / CDI / LINSTOR
cascade races repeatedly refire the VPA HelmRelease — this produces
10+ admission-controller ReplicaSets in minutes and blocks the whole
cozystack-platform HR: the admission-controller pod livelocks in
startup (waits on its own election lease which its previous
ReplicaSet is still holding, hits livenessProbe timeout,
CrashLoopBackOffs), VPA never becomes Ready, PackageSources stay
InProgress, and cozystack-platform aborts at its 15-minute install
timeout.

Fix: replace the source of the TLS Secret with a namespace-scoped
cert-manager `Issuer` + `Certificate`. cert-manager only rewrites the
Secret when the certificate is close to expiry (`renewBefore`), so
reconcile-driven cert churn stops. Pod rotation is delegated to a
runtime-lookup-based `checksum/tls-cert` podAnnotation that only
changes when the Secret's `tls.crt` content actually changes — not on
every render, not on every reconcile.

## What

-
`charts/vertical-pod-autoscaler/templates/admission-controller/certificate.yaml`
  (new): namespace-scoped selfSigned Issuer + Certificate. The
  Certificate mirrors the legacy `genSignedCert` shape (single DNS SAN
  `vpa-webhook.<ns>.svc`, 365d validity, matching Secret name), so
  cert-manager on the first upgrade adopts the same Secret name the
  legacy path wrote — no rename cascade in the Deployment volume mount.
  Rendered only when `admissionController.tls.certManagerEnabled=true`
  AND no `existingSecret` is supplied.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/tls-secret.yaml`
  (modified): wrapped in an extra `certManagerEnabled=false` guard so
  the legacy genCA path remains available as an explicit opt-out.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/deployment.yaml`
  (modified): the legacy `checksum/tls-secret` annotation is gated
  behind `certManagerEnabled=false`. Under the cert-manager path the
  Deployment carries a `checksum/tls-cert` annotation computed from
  `lookup`-ing the runtime Secret's `tls.crt` — see the template
  comment for the full timing analysis and the day-350 renewal
  trade-off it documents.
- `charts/vertical-pod-autoscaler/values.yaml` (modified): new
  `admissionController.tls.certManagerEnabled` value defaulting to
  `true`. Values-block header documents the switch as effectively
  one-way (cert-manager writes `type: kubernetes.io/tls`;
  `Secret.type` is immutable, so falling back requires `kubectl
  delete` first).
- `patches/cert-manager-admission-tls.diff` (new) +
  `Makefile update` extended: the four subchart edits above live as a
  single reviewable diff that Makefile re-applies over the freshly
  vendored chart on every `make update`, matching the fluxcd-operator
  package convention. Prevents a bump from silently reverting the
  fix.
- `packages/core/platform/sources/vertical-pod-autoscaler.yaml`
  (modified): VPA PackageSource now depends on `cozystack.cert-manager`.
  Non-circular: cert-manager depends on `cozystack.networking` +
  `cozystack.gateway-api-crds`, neither reaches VPA.
- `tests/vpa_test.yaml` (modified): +11 helm-unittest cases (15 total,
  4 pre-existing) covering `certManagerEnabled={true,false}`,
  `existingSecret` bypass, disabled admission-controller, and
  the offline-render behaviour of the `checksum/tls-cert` lookup.

## Why (evidence)

Snapshot from CI run 28587656430 (this branch, pre-workaround-review):

```
admission-controller ReplicaSets after first-install churn window:
  vertical-pod-autoscaler-admission-controller-55d888d6d4
  vertical-pod-autoscaler-admission-controller-56f8d4c747
  vertical-pod-autoscaler-admission-controller-5775dbc9b7
  vertical-pod-autoscaler-admission-controller-585b7c56c9
  vertical-pod-autoscaler-admission-controller-6fc44944fb
  vertical-pod-autoscaler-admission-controller-74bccf4849
  vertical-pod-autoscaler-admission-controller-84df98f98b
  vertical-pod-autoscaler-admission-controller-8666dffb85
  vertical-pod-autoscaler-admission-controller-86b469457c
  vertical-pod-autoscaler-admission-controller-9b979f9d6
  → 10 ReplicaSets, one per Flux reconcile, each with a different
    genCA-rooted TLS Secret sha in checksum/tls-secret.
```

## Migration for existing clusters

On upgrade the legacy Opaque Secret disappears from the release
manifest, Helm garbage-collects it, and cert-manager immediately
writes a fresh `kubernetes.io/tls` Secret of the same name via the
new Certificate. `Secret.type` is immutable, so in-place adoption is
impossible — the delete/create pass is the correct path, and the
seconds-long window it produces is safe: the `vpa-webhook-config`
MutatingWebhookConfiguration ships with `failurePolicy: Ignore`
(verified on cozy-dev3), so Pod admissions in that window skip VPA
mutation but never get rejected — no cluster-wide disruption.

The Deployment rolls in the same upgrade because the new
`checksum/tls-cert` annotation differs from anything the pod had
before, so the admission-controller re-registers `vpa-webhook-config`
with the fresh caBundle on the same pass.

## Test plan

- [x] `helm unittest packages/system/vertical-pod-autoscaler` — 15/15
      pass.
- [x] `helm template` diff between main and this branch is scoped
      exactly to the cert-manager path (new Certificate + Issuer,
      new `checksum/tls-cert` annotation, guarded legacy Secret, new
      values key, patch file + Makefile step, PackageSource
      dependsOn). No unrelated lines change.
- [x] `make update`-style simulation (rm -rf charts && helm pull &&
      patch) produces a byte-identical subchart to what git tracks —
      the patch stays clean across future upstream bumps until it
      genuinely conflicts.
- [ ] Live migration on cozy-dev3: `helm upgrade` VPA release from
      pre-fix version → observe cert-manager write the new tls Secret
      → verify `checksum/tls-cert` on the new pod's podTemplate →
      verify MutatingWebhookConfiguration `vpa-webhook-config`
      caBundle updated in place → confirm no ReplicaSet churn on
      subsequent Flux reconciles. Blocking the draft's removal.
- [ ] Full E2E on this branch — verify only 1 admission-controller
      ReplicaSet after install, VPA HR reaches Ready inside
      `cozystack-platform`'s 15m install timeout, and
      `kubernetes-latest.bats` / `kubernetes-previous.bats` no longer
      time out on VPA-attributable resource churn.

## Review pass — kvaps' CHANGES_REQUESTED addressed

- ✅ Vendored chart mods now carried as `patches/*.diff` re-applied by
  `Makefile update` (matches fluxcd-operator convention).
- ✅ Retired the tls-migrator hook — kvaps correctly showed it was
  a no-op: Helm's delete decision is driven by the stored release
  manifest not by live labels (`ResourcePolicyAnno` is the only opt-
  out), and `Secret.type` is immutable so cert-manager could not have
  adopted the legacy Opaque Secret in place anyway. ~200 lines
  removed.
- ✅ PR description rewritten to match the actual diff (no more
  reference to the removed `tls-rollout.yaml` post-upgrade hook, real
  test count, dev3 verification honestly listed as pending).
- ✅ `checksum/tls-cert` timing comment corrected: `lookup` runs on
  Helm actions only, day-350 renewal trade-off spelled out.
- ✅ `certManagerEnabled=false` documented as effectively one-way
  (Secret.type immutable).
- ✅ Dropped the `secretTemplate` labels — `app.kubernetes.io/managed-
  by: Helm` on a cert-manager-owned Secret was misleading.
- ✅ Banner comment indentation restored in `vpa_test.yaml`.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* VPA admission-controller webhook TLS now uses cert-manager by default,
with options to disable cert-manager or supply an existing TLS secret.
* Webhook pod restarts are now tied to cert-manager Secret contents via
a content-based checksum.
* **Bug Fixes**
* Improved webhook TLS provisioning reliability when TLS readiness is
delayed.
* Corrected conditional rendering/rollout behavior to avoid mismatched
TLS ownership.
* **Tests**
* Added rendering-invariant tests covering cert-manager, legacy TLS,
existing-secret bypass, and admission-controller disabled behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-04 22:48:00 +03:00
IvanHunters 59964043e3 fix(platform): migration 49 backfills tenant ancestor labels 
The tenant chart fix makes new and re-reconciled tenant namespaces
self-compute their full tenant.cozystack.io/<ancestor> chain, but existing
namespaces whose HelmRelease does not promptly re-reconcile (suspended or
failed releases) keep the incomplete labels — and with them the broken
<ancestor>-egress reachability that surfaces as upstream timeouts to nested
tenants. Backfill the labels directly so clusters heal on upgrade regardless
of per-tenant HelmRelease state.

Bump targetVersion to 50 so run-migrations.sh iterates the new script
(seq CURRENT..49); the missing-file guard otherwise never fires. Best-effort
and idempotent: kubectl label --overwrite adds only the listed labels, and
per-namespace failures are tolerated so a transient apiserver error cannot
abort the platform upgrade.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-03 09:54:12 +03:00
Ivan Okhotnikov 9dc64b9278 feat(platform): VPA depends on cert-manager after admission-controller TLS migration 
The VPA admission-controller webhook TLS is now provisioned via a
cert-manager Certificate + Issuer instead of the in-chart genCA
Secret. On a fresh install the VPA package must not reconcile until
cert-manager is Ready; otherwise the Certificate resource sits
Pending forever, the admission-controller Secret is never written,
and the pod CrashLoops on missing TLS mount — cascading up through
the cozystack-platform PackageSource in exactly the way the old
genCA path did, only for a different reason.

Add cozystack.cert-manager to the variant's dependsOn list. Verified
non-circular against packages/core/platform/sources/cert-manager.yaml
(cert-manager depends on cozystack.networking + cozystack.gateway-api-crds,
neither of which reaches VPA).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
 2026-07-02 18:57:01 +03:00
Алексей Артамонов a2ae1621af fix(clickhouse): surface cleanup hook failures and backfill legacy keeper PVC labels 
Address review on #3072:

- Cleanup hook no longer masks kubectl delete failures: the command now
  fails (exit 1) when PVC deletion fails instead of swallowing the error
  with || echo. The hook-delete-policy drops hook-succeeded (keeps only
  before-hook-creation) so a failed cleanup Job survives for inspection.
- Add migration 45 (45 -> 46) that backfills app.kubernetes.io/instance
  onto keeper PVCs created before this change; the altinity keeper operator
  never re-stamps volumeClaimTemplate labels onto existing PVCs, so without
  this the cleanup selector misses pre-upgrade keeper PVCs and #3057
  persists for existing releases. Bump migrations.targetVersion to 46.
- Unify helm.sh/hook-weight key quoting across the four hook manifests.
- Extend the unittest to assert the new hook-delete-policy and exit-on-failure.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
 2026-07-01 18:31:05 +03:00
Aleksei Sviridkin c841c70866 feat(operator): replicate operator wildcard certificate to tenant namespaces 
The operator-provided wildcard MVP serves the root tenant only: the root
ingress controller and Gateway read a TLS Secret from their own namespace.
Child tenants run their own ingress controller / Gateway in a separate,
default-deny namespace and cannot read that Secret cross-namespace.

Add a controller that mirrors the operator wildcard Secret into every
tenant namespace that owns a TLS termination point (its
namespace.cozystack.io/ingress or .../gateway label equals its own name).
The source is identified by name from the platform values channel
(_cluster.wildcard-secret-name) with no extra operator input; replicas
carry a managed-by label and refresh on rotation. An individual replica is
removed when its namespace stops terminating TLS; every replica is torn
down only on an explicit disable (clearing wildcard-secret-name). A missing
or mistyped source, or a transiently-absent values channel, keeps existing
replicas, so tenant TLS never drops on a transient gap. The controller's
Secret cache is scoped to managed replicas and the values channel, not
every cluster Secret. Only the platform controller's ServiceAccount gains
Secret writes — no tenant RBAC is widened, and every consumer reads only
its own-namespace copy.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-01 10:47:14 +03:00
Aleksei Sviridkin 41ff94eca9 fix(flux): tune source-watcher concurrency and CPU under load 
source-watcher reconciles one ArtifactGenerator per PackageSource and
builds the resulting ExternalArtifacts. On the management cluster that is
~100 artifact builds during install, and the platform HelmRelease blocks
until all of them are Ready.

It was the only Flux controller left untuned. In the bootstrap all-in-one
pod it ran without an explicit --concurrent (binary default 10) and with
the shared 100m CPU request; in the production FluxInstance it was missing
from the --concurrent / CPU patch that source-, kustomize- and
helm-controller already receive. Under load its per-artifact builds finish
at staggered times, leaving a random tail of not-yet-Ready artifacts that
times out the platform install.

Raise source-watcher to --concurrent=20 in both deployments. In the
bootstrap all-in-one pod also give it a 500m CPU request; the pod has no
CPU limits, so the request is what drives its CPU share under contention.
In the production FluxInstance add it to the shared reconciler patch, so it
now receives the same --requeue-dependency=5s and 2000m/2048Mi limits the
other tuned controllers already have.

The flux-aio module exposes a resources override for the watcher (set via
CUE) but no args override, so the bootstrap --concurrent bump is a
post-render patch in the Makefile.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-01 07:42:33 +03:00
Aleksei Sviridkin cbc314a8c5 fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public 
The kubevirt-cdi default variant only depended on cozystack.networking,
which reconciles long before cozystack-basics provisions the cozy-public
namespace. kubevirt-cdi ships the cdi-clone-dv RoleBinding into cozy-public,
so on a fresh install the RoleBinding can race namespace creation and fail
with `namespaces "cozy-public" not found`. Flux retries normally mask it,
but under slow reconcile it surfaces as a hard UpgradeFailed.

Add a cozystack.cozystack-basics edge so the namespace exists first,
mirroring the backupstrategy-controller ordering. No cycle: cozystack-basics'
own dependencies (tenant-application, cozystack-engine, gateway-api-crds) do
not reach kubevirt-cdi. Pin the edge with a helm-unittest regression guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-30 19:51:46 +03:00
Aleksei Sviridkin bf14a73b39 feat(api): add SecurityGroup network policy resource (sdn.cozystack.io) (#2922) 
## What this PR does

Introduces a tenant-facing `SecurityGroup` resource in a new
`sdn.cozystack.io/v1alpha1` API group, served by the Cozystack
aggregated API server as a 1:1 projection of a `CiliumNetworkPolicy`. A
SecurityGroup attaches to one of the tenant's managed applications by
reference (`spec.targetRef: {apiGroup, kind, name}`) and declares
allow-list ingress/egress for that application. Tenants manage network
policy for their own apps entirely through
`securitygroups.sdn.cozystack.io`, without being granted any access to
the `cilium.io` API group.

The translation is synchronous and stateless — the API server converts
each SecurityGroup into a CiliumNetworkPolicy in the same namespace on
write and back on read; there is no controller. The backing policy's
`endpointSelector` is **derived** from `targetRef` (matched on the
application's lineage labels
`apps.cozystack.io/application.{group,kind,name}`), never copied from
tenant input, so the selector is machine-generated and can only ever
match the referenced application's own pods in the tenant's namespace.
Policies owned by this API carry an `sdn.cozystack.io/securitygroup`
marker label that the storage always re-asserts and hides from the view,
so it only ever surfaces or mutates its own policies and never touches
platform-managed network policies.

The value is self-service network policy on a Cozystack-native,
RBAC-scoped surface, without exposing Cilium's version-coupled CRDs to
tenants. Because the selector is derived from an authorized application
reference rather than free-form tenant input, the RBAC boundary is
structural, not selector-validated: a tenant cannot express network
policy against arbitrary or platform-owned pods. The full design —
selector derivation from lineage labels, the structural boundary, the
additive-policy reality (a SecurityGroup widens a managed application's
allowed traffic but cannot sever its platform-managed management plane,
and equally cannot tighten below the platform baseline), the in-tree
CiliumNetworkPolicy mirror that avoids a Cilium Go-module dependency,
RBAC, and deferred extensions (raw-pod targeting, cluster scope,
reusable groups) — is in `pkg/apis/sdn/DESIGN.md`.

Implements the attachment model from #1614.

### Release note

```release-note
feat(api): add the SecurityGroup resource (sdn.cozystack.io/v1alpha1) — a tenant-facing, namespace-scoped firewall that attaches to a managed application by reference and projects 1:1 onto a CiliumNetworkPolicy with a machine-derived endpoint selector, letting tenants manage their application's network policy without cilium.io access.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added the `SecurityGroup` API resource (`sdn.cozystack.io` v1alpha1)
with a namespaced projection backed by Cilium network policies.
* Enabled the SDN `APIService` and wired SDN routing, scheme, and REST
support.
* Expanded RBAC to include SecurityGroup access and required Cilium
policy permissions.
* **Documentation**
* Added a design document describing projection behavior and rule
semantics.
* **Bug Fixes**
  * Improved API rule validation coverage for missing list-type fields.
* **Tests**
* Added SDN round-trip, fuzzing, RBAC, and comprehensive REST
create/get/list/update/delete/watch/resource-version tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-30 14:30:23 +03:00
Aleksei Sviridkin f9d1138c6e feat(keycloak): configurable login theme via platform branding (#3124) 
## What this PR does

Makes a realm's Keycloak login theme configurable through platform
`branding` values, closing a gap: `branding` could set the realm display
name but not its login theme, and the keycloak chart's `themes:` list
was not reachable from platform values.

Two new keys under `branding` (propagated via `_cluster.branding`, the
same path that already feeds `displayName`):

- `branding.loginTheme` — keycloak-configure renders it as
`ClusterKeycloakRealm.spec.themes.loginTheme`, so the realm selects that
login theme.
- `branding.loginThemeImage` — when set together with `loginTheme`, the
keycloak chart appends `{name: loginTheme, image: loginThemeImage}` to
its effective theme list, so the existing init-container mounts the
theme image into `/opt/keycloak/themes/`. No StatefulSet patching or
per-package values plumbing.

`loginTheme` alone (no image) still selects the theme — for themes
already baked into the Keycloak image. Explicit chart `themes:` entries
keep working; the branding-derived entry is appended after them.

Flow: platform `branding` → `_cluster.branding` (cozystack-values
Secret) → keycloak-configure (realm loginTheme) + keycloak (theme init
container). Covered by helm-unittest in both charts.

### Screenshots

No bundled UI assets — this PR adds the mechanism only; the login theme
is operator-supplied via `loginThemeImage`, so the visual result depends
on that image.

### Release note

```release-note
feat(keycloak): select the realm login theme and ship its theme image through platform `branding` values (`branding.loginTheme`, `branding.loginThemeImage`)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for configuring a custom login theme from platform
branding, including automatic selection in the realm setup.
* Theme entries can now be combined with existing theme settings and
rendered in the expected order.

* **Documentation**
* Expanded configuration guidance for branding and theme-related
settings with clearer examples.

* **Tests**
* Added coverage for login theme rendering and fallback behavior when
theme settings are missing.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-30 12:29:52 +03:00
Aleksei Sviridkin 31c15b282f feat(network): mechanism-agnostic LoadBalancer exposure (ExposureClass + ServiceExposure) (#3081) 
## What this PR does

Lays the foundation for migrating tenant LoadBalancer exposure off the
deprecated `Service.spec.externalIPs` (KEP-5707) and, eventually, off
MetalLB onto Cilium-native announcement — the mechanism-agnostic shape
prescribed when the earlier chart-side approach (#2468) was reverted in
#2603.

It adds a new `network.cozystack.io/v1alpha1` API group and a controller
that translates intent into whichever LoadBalancer backend the cluster
admin configured:

- `ExposureClass` (cluster-scoped, admin): binds a logical class name to
a backend (`externalIPs` | `metallb` | `cilium` | `robotlb`) and its VIP
address scope — the StorageClass/IngressClass pattern.
- `ServiceExposure` (namespaced, tenant intent): "expose this Service
via class X"; status reports the resolved backend and assigned IPs.
- A `serviceexposure` controller renders the backend's pool + announcer
once per class (MetalLB `IPAddressPool` + `L2Advertisement`; Cilium
`CiliumLoadBalancerIPPool` + `CiliumL2AnnouncementPolicy`), scoped to
the namespaces that opted in. Backends are pluggable behind a small Go
interface, so adding BGP or another cloud LB is one implementation, not
an API change. The controller never mutates the target Service — the
chart still owns its shape.

Platform wiring: `publishing.exposureClass.*` renders the admin
`ExposureClass` and the host ingress publishes through it. The default
(no class set, `externalIPs`) leaves every existing cluster's behaviour
byte-for-byte unchanged; the MetalLB→Cilium announcer migration becomes
a single `ExposureClass.spec.backend` change.

This is the foundation only — Part of #2586. Converting per-app
exposure, flipping Cilium `l2announcements` on, changing the default,
and removing MetalLB are deferred to later phases.

### Release note

```release-note
feat(network): add ExposureClass and ServiceExposure for mechanism-agnostic LoadBalancer exposure. Opt-in foundation; the default externalIPs publishing path is unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added new `ExposureClass` and `ServiceExposure` custom resources with
support for multiple backends (externalIPs, MetalLB, Cilium, RobotLB),
including optional L2 and address configuration.
* Added Helm wiring and templates to enable service/ingress publishing
via `ServiceExposure`, selecting a configured (or default)
`ExposureClass`.
* **Bug Fixes**
* Improved reconciliation, finalizer-based cleanup, and status updates
for readiness and assigned IP reporting, including safer handling when
backends or managed resources change.
* **Tests**
* Added end-to-end and unit test coverage for backend rendering,
readiness, and cleanup flows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-30 12:29:48 +03:00
Myasnikov Daniil b9749cec80 fix(platform): md0 migration -- fail closed and preserve the no-delete label 
Migration 47 pins the implicit default md0 node group before the kubernetes
chart upgrade so the live md0 MachineDeployment is not pruned. Two correctness
fixes:

- Stamp the cozystack-version ConfigMap via the shared stamp_cozystack_version
  helper instead of `kubectl create configmap | kubectl apply`. The label-less
  apply by the same field manager stripped the platform.cozystack.io/no-delete
  label migration 42 added, dropping cozystack-version out of the
  cozystack-no-delete-guardrail policy; the helper re-emits the label and is
  enforced for every migration >= 42 by hack/cozystack-version-stamp.bats.
- Fail closed. A non-NotFound read error is now counted as a failure rather
  than silently skipped, and any failure exits 1 so the pre-upgrade hook aborts
  before Helm rolls out the new chart against clusters whose md0 is not yet
  pinned. The version stamp stays at 47 and the migration retries next upgrade.

Also narrow the merge patch to only md0 so a concurrent edit to a sibling node
group between the read and the patch cannot be reverted.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-30 12:08:17 +05:00
mattia-eleuteri 602b8d06fd fix(platform): add migration 47 to pin implicit md0 node group 
After the kubernetes chart stops force-merging a default md0 node group,
clusters that relied on the implicit md0 (defined other nodeGroups without
md0) would have their live md0 MachineDeployment pruned by Helm on the next
reconcile. Migration 47 makes md0 explicit on those clusters to preserve the
current node topology; it is idempotent, treats a missing CRD as a no-op, and
skips clusters that already define md0.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
 2026-06-30 12:08:17 +05:00
Aleksei Sviridkin 60c9b39578 docs(platform): document branding keys including login theme 
The branding map shipped as an undocumented empty map. Describe the keys it
actually supports (brandName, brandHtmlName) and the new login-theme keys
(loginTheme, loginThemeImage) consumed by the Keycloak charts, with an example.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-28 05:08:47 +03:00
Myasnikov Daniil 70320e0abf fix(platform): stamp migrations 45/46 via the cozystack-version helper 
Migrations 45 and 46 (added by #2931) stamp the cozystack-version ConfigMap
with an inline `kubectl apply` heredoc instead of sourcing
lib/cozystack-version.sh and calling stamp_cozystack_version, the way 42/43/44
do. The cozystack-version-stamp.bats guard forbids any migration >= 42 from
bypassing the helper, so `make bats-unit-tests` fails on migration 45.

This only surfaces in PR CI (make unit-tests); main-push CI does not run the
bats suite, so #2931 landed red on this test and now blocks every PR that
merges current main.

Route both stamps through the helper. The helper renders a byte-identical
labeled manifest (golden-pinned in the bats suite), so this is behaviour-
preserving — it just restores the no-label-drift guarantee the heredoc form
could silently lose.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-26 20:40:02 +05:00
myasnikovdaniil 1df8c7c557 feat(kubernetes): bootstrap tenant workers with Talos via CABPT (#2931) 
## What this PR does

> **Supersedes #2610.** Same work, rebased cleanly onto current `main`
so it now sits on top of #2867 (the `kube-controller-manager` VAP
type-checker fix). That fix is what was killing the base install in
#2610's E2E *before* the `kubernetes` app test ever ran, leaving the
Talos worker-bootstrap rewrite verified only by code reading. #2610's
67-commit branch is squashed here into 8 logical commits with original
authorship preserved (@kvaps and @IvanHunters as authors); the tree is
**byte-identical** to #2610's head — no code changes, just the rebase
onto `main` (to pick up #2867) plus history cleanup. The detailed review
history lives on #2610.

Phase 1 of the Kubernetes-app split design (see cozystack/community#8):
replace the Ubuntu+kubeadm worker bootstrap path with Talos, driven by
`cluster-api-bootstrap-provider-talos` (CABPT) and a
`clastix/talos-csr-signer` sidecar embedded in the Kamaji control-plane
pod. Existing tenants keep working — old machines roll out and get
replaced by Talos workers without manual intervention.

Highlights:

- Add CABPT (v0.6.12) as a second `BootstrapProvider` alongside the
existing kubeadm one in `capi-providers-bootstrap`.
- Bump the Kamaji control-plane provider with an upstream-bound patch
that exposes `KamajiControlPlane.spec.network.additionalServicePorts`,
used to publish trustd (50001/TCP) on the apiserver service.
- Generate Talos PKI (Ed25519 CA + trustd TLS) via cert-manager and
stable random Talos secrets (`token`, `clusterId`, `clusterSecret`,
`bootstrapToken`) using the helm lookup-and-reuse pattern.
- Materialise a kubeadm-format `bootstrap-token-<id>` Secret inside the
tenant `kube-system` via a Helm post-install/upgrade Job.
- Render a `TalosConfigTemplate` (worker machineconfig, `generateType:
none`) and switch the `MachineDeployment.bootstrap` reference from
`KubeadmConfigTemplate` to `TalosConfigTemplate`, gated on Talos secrets
and the Kamaji apiserver service being ready.
- Boot workers from the Talos openstack image via a CDI `DataVolume`
(`source.http.url` → `factory.talos.dev`), expose the system disk as
virtio-blk with `blockSize.custom: logical=512, physical=4096` so
4Ki-native block storage backends (LINSTOR/DRBD) play nicely with QEMU's
O_DIRECT writes and SeaBIOS still boots, drop the separate kubelet disk
(Talos lays out EPHEMERAL itself).
- Pin pod/service CIDRs in the worker machineconfig to the tenant ranges
(`10.243.0.0/16` / `10.95.0.0/16`) to avoid Talos's address-overlap
diagnostic against the host pod CIDR.
- Set `cluster.controlPlane.endpoint` and `cilium.k8sServiceHost` to the
Kamaji apiserver Service ClusterIP (looked up at render time), so
bootstrap survives the chicken-and-egg moment where tenant DNS does not
exist yet and the host CoreDNS does not serve the tenant zone.

End-to-end on a dev cluster: a fresh tenant brings up a Talos worker,
kubelet CSRs get approved (apiserver-client by Kamaji, kubelet-serving
by the talos-csr-signer sidecar), Cilium initialises with hostNetwork
against the apiserver ClusterIP, CoreDNS comes up, and the node
transitions to `Ready`.

### Release note

```release-note
feat(kubernetes): bootstrap tenant workers with Talos Linux instead of Ubuntu+kubeadm. Existing clusters roll over to Talos workers automatically; new tenants come up on Talos from the start. Powered by cluster-api-bootstrap-provider-talos and the talos-csr-signer sidecar in the Kamaji control-plane pod. Note: the tenant `kubernetes` HelmRelease now reports Ready as soon as Helm completes (Install/Upgrade DisableWait) and no longer blocks on worker/addon readiness — worker-rollout health is tracked by WorkloadMonitor, not the HelmRelease.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Release Notes

**New Features**
* Added Talos worker configuration for `talos.version` and
`talos.schematicID`
* Introduced `nodeHealthCheck` (`maxUnhealthy`, `nodeStartupTimeout`)
for worker remediation tuning
* Added image overrides for `kubectl` and `talosCsrSigner`
* Exposed additional tenant control-plane Service ports support

**Breaking Changes**
* Removed Kubernetes v1.30; supported versions start at v1.31
* Renamed `nodeGroups.ephemeralStorage` → `nodeGroups.diskSize` with
consolidated Talos system-disk semantics

**Removals**
* Removed Ubuntu container-disk images and related containerd patch
mechanisms
* Air-gapped tenant workers temporarily unsupported during the Talos
transition

**Documentation**
* Updated chart docs, parameters, and GitOps upgrade guidance for
version bumps
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-26 20:06:49 +05:00
Kirill Ilin 5cb6ef435f feat(keycloak): opt-in separate hostname and route for the admin console (#2954) 
## What this PR does

Adds an opt-in way to serve the Keycloak admin console and
Administration
REST API on a separate hostname and route from the public login
endpoints,
so the admin surface can be kept off the public internet.

- New `ingress.adminHost` in the keycloak chart. When set:
  - `KC_HOSTNAME_ADMIN` is configured so the admin console generates its
    URLs on that hostname;
  - the public (login) route is narrowed to the OIDC paths only
    (`/realms`, `/resources`, `/.well-known`), so the admin console and
    Administration REST API are no longer served on the login hostname;
- a dedicated route (Ingress or HTTPRoute, depending on
`gateway-enabled`)
    serves the admin hostname.
- New platform `publishing.ingressNameAdmin`, propagated as
  `_cluster.expose-ingress-admin`. The admin route attaches to the
Gateway/ingressClass it names, defaulting to `expose-ingress`. Pointing
it
at a separate (for example, private-IP) Gateway moves admin traffic off
the public one. The separate Gateway itself is provisioned out of band;
  this only selects where the admin route attaches.

Everything is empty by default, so existing deployments are unchanged.

Note: `KC_HOSTNAME_ADMIN` on its own is not a security boundary — the
Administration REST API stays reachable on any hostname routed to
Keycloak
(per the Keycloak docs). The path and network separation introduced here
is
what restricts access; full isolation requires attaching the admin route
to
a non-public Gateway.

Covered by helm-unittest suites for the keycloak chart and the platform
value wiring.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(keycloak): add opt-in `ingress.adminHost` to serve the admin console and Administration REST API on a separate hostname/route, plus `publishing.ingressNameAdmin` (`_cluster.expose-ingress-admin`) to attach that route to a separate (e.g. private) Gateway/ingressClass. Disabled by default; existing deployments are unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional configuration to serve the Keycloak admin console on a
dedicated hostname, with separate routing and ingress/gateway attachment
so admin UI/API can be isolated from public login/OIDC paths.
* Introduced a cluster-level setting to optionally expose admin routes
via a separate ingress/gateway class/namespace.

* **Tests**
* Added new YAML test coverage to validate defaulting, hostname split
behavior, and correct routing for both gateway-enabled and
gateway-disabled modes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-26 14:08:23 +05:00
Aleksei Sviridkin 67692b4544 feat(platform,ingress): publish host ingress via ServiceExposure 
Add the operator-facing knob that drives the new exposure model. When
publishing.exposureClass.name is set, the platform renders a cluster-scoped
ExposureClass binding the chosen backend (externalIPs, metallb, cilium,
robotlb) and VIP address scope, and carries the class name through the
cozystack-values Secret. The host ingress then renders a ServiceExposure
for its ingress-nginx Service, which the cozystack-controller translates
into the backend's pool and announcer.

The default (empty name) renders neither the ExposureClass nor the
ServiceExposure, so clusters on the externalIPs path are unchanged on
upgrade. The ingress Service shape is untouched: it is already published
as type LoadBalancer whenever externalIPs is empty.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-25 20:47:19 +03:00
IvanHunters e0597e3d99 fix(platform): make *-rd sources depend on cozystack-engine (#3041) 
## Problem

On an unlucky Flux reconcile order, a `*-application` PackageSource that
ships a `system/*-rd` component can install before the
`applicationdefinitions.cozystack.io` CRD exists. Every `*-rd` chart
renders an `ApplicationDefinition` (`cozystack.io/v1alpha1`); that CRD
is installed by the `application-definition-crd` component of
`cozystack.cozystack-engine`. When an `*-rd` HelmRelease renders first,
it hard-fails with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, and that cascades into a broad install
timeout — surfacing even on changes that have nothing to do with
ordering, because it is a latent race.

## Fix

Add `- cozystack.cozystack-engine` to the `dependsOn` of every
`*-application` source that ships a `system/*-rd` component (29
sources). `gateway-application` already carried this edge — it was added
earlier for the TenantGateway CRD, the same bug class, but only for that
one source. A new invariant test fails if any source with a
`system/*-rd` component is missing the edge.

## Why

The ordering was previously implicit, relying on Flux happening to
reconcile `cozystack-engine` before the `*-application` sources. Making
the edge explicit removes the race deterministically. No dependency
cycle is introduced: `cozystack-engine` depends only on `networking` and
`cert-manager` (plus keycloak in its oidc variant), none of which reach
back to any `*-application`.

## Test

- New generic guard `hack/check-applicationdefinition-crd-ordering.bats`
(auto-discovered by `make bats-unit-tests`, runs in CI via `make
unit-tests`): parses every source and fails if any PackageSource variant
with a `system/*-rd` component lacks the `cozystack.cozystack-engine`
edge — so future sources are covered, not just today's set.
- `helm unittest packages/core/platform`: 13 suites / 66 tests pass; the
source edits do not disturb existing assertions.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved dependency ordering across several platform packages to
ensure related components install in the correct sequence.
* Prevented end-to-end test selection from being overly broad when only
ordering-related dependencies change.

* **Tests**
* Added checks to catch missing dependency requirements and to verify
test selection behavior stays accurate.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-25 19:21:50 +03:00
IvanHunters f6d5086190 fix(platform): order backupstrategy-controller after cozystack-basics (#3037) 
## Problem

On a fresh install, the install step can fail with:

```text
Error creating resource via patch ... apps.cozystack.io/v1alpha1 Kind=Bucket
  name=cozy-backups namespace=tenant-root: namespaces "tenant-root" not found
```

`backupstrategy-controller` renders the `cozy-backups` Bucket **into**
the `tenant-root` namespace, but its `dependsOn` only guards the CRDs it
needs (backup-controller / bucket-application / objectstorage-controller
/ velero) — not the existence of the target namespace. `tenant-root` is
provisioned by `cozystack-basics` (the static tenant-root release), and
there is no ordering edge to it. On an unlucky Flux reconcile order the
Bucket patch races namespace creation, fails, and that transient failure
races the install script's `kubectl wait` deadline → the run fails.

This is latent and not introduced by any particular change — the
Bucket-into-tenant-root predates recent branches — so it flakes installs
on changes that have nothing to do with backups.

## Fix

Add `cozystack.cozystack-basics` to `backupstrategy-controller`'s
`dependsOn` so the `tenant-root` namespace exists before the Bucket is
applied. `cozystack-basics` is a dependency-graph leaf (nothing depends
on it), so no cycle is introduced.

Pinned with a helm-unittest regression guard
(`tests/sources_backupstrategy_dependson_test.yaml`) asserting the edge
— mirroring the existing `sources_basics_dependson_test.yaml` pattern.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved install ordering so the backup strategy component waits for
required foundational services before deploying, reducing race
conditions during fresh installs.
* Clarified dependency relationships for cluster-wide platform
components to better reflect safe upgrade and installation behavior.

* **Tests**
* Added regression coverage to verify the backup strategy package
declares all required dependencies, including namespace and CRD
providers.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-25 19:21:25 +03:00
Aleksei Sviridkin 69247ba80c feat(platform): deploy the securitygroup-controller 
Add the securitygroup-controller Helm package (Deployment, ServiceAccount,
ClusterRole and binding, image build) and register it as a default system
package so it ships with the platform.

The ClusterRole grants cluster-wide pod patch — the controller stamps the
SecurityGroup membership label on managed-app pods across dynamically-created
tenant namespaces, so the grant cannot be namespace-scoped — plus read and
finalizer patch on the backing CiliumNetworkPolicies, and leader-election
leases. Tenants gain no new permissions; the membership label is written
exclusively by this platform controller. helm-unittest pins the pod-patch and
ciliumnetworkpolicies grants so a dropped verb fails the suite rather than
silently failing closed at runtime.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-25 17:50:33 +03:00
Aleksei Sviridkin 4593b72cd1 feat(operator): support private OCI registry credentials for platform source (#2472) 
## Summary

Implements support for private OCI registry and Git repository
credentials in platform sources. Adds `--platform-source-secret` flag to
the operator and corresponding Helm chart parameter.

Closes #2455

## Changes

### Operator (`cmd/cozystack-operator/main.go`)
- Added `--platform-source-secret` flag for specifying Secret name
- Updated `generateOCIRepository()` to set `spec.secretRef` when secret
is provided
- Updated `generateGitRepository()` to set `spec.secretRef` when secret
is provided
- Secret is expected to be in `cozy-system` namespace

### Helm Chart
- **`packages/core/installer/values.yaml`**: Added
`cozystackOperator.platformSourceSecret` parameter with documentation
- **`packages/core/installer/templates/cozystack-operator.yaml`**: Pass
`--platform-source-secret` flag to operator container

## Usage Example

### For private OCI registry (e.g., GitLab Container Registry):

```bash
# Create registry credentials secret
kubectl create secret docker-registry gitlab-registry-creds \
  --namespace cozy-system \
  --docker-server=registry.gitlab.example.com \
  --docker-username=<username> \
  --docker-password=<token>
```

```yaml
# Helm values
cozystackOperator:
  platformSourceUrl: 'oci://registry.gitlab.example.com/mygroup/cozystack-packages'
  platformSourceRef: 'tag=v1.0.0'
  platformSourceSecret: 'gitlab-registry-creds'
```

### For private Git repository:

```bash
# Create Git credentials secret (SSH key or basic auth)
kubectl create secret generic git-creds \
  --namespace cozy-system \
  --from-literal=username=<username> \
  --from-literal=password=<token>
```

```yaml
# Helm values
cozystackOperator:
  platformSourceUrl: 'https://github.com/private-org/cozystack-packages'
  platformSourceRef: 'branch=main'
  platformSourceSecret: 'git-creds'
```

## Testing

Manually verified:
- OCIRepository resource is created with `spec.secretRef` when secret
name is provided
- GitRepository resource is created with `spec.secretRef` when secret
name is provided
- Resources are created without `spec.secretRef` when secret name is
empty (backward compatible)
- Helm chart renders correctly with and without `platformSourceSecret`
parameter

## Checklist

- [x] Code follows project style and conventions
- [x] Operator supports both OCI and Git source types
- [x] Helm chart parameter documented with comments
- [x] Backward compatible (empty secret name is handled correctly)
- [x] Closes referenced issue (#2455)


## Release note

```release-note
feat(operator): add `--platform-source-secret` flag to authenticate the platform package source against private OCI registries (kubernetes.io/dockerconfigjson) and private Git repositories (basic-auth, bearer token, or SSH key). Empty value preserves existing public-registry behaviour.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for configuring a Kubernetes Secret for the platform
source.
* Installer settings and startup options now let you provide a secret
name for Flux source credentials.

* **Bug Fixes**
* Generated platform source resources can now reference the provided
Secret, enabling authenticated access for OCI and Git sources.

* **Documentation**
* Updated configuration docs to explain the new secret setting and
required secret format.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-25 14:41:37 +03:00
Kirill Ilin caeb2c466f feat(platform): add publishing.ingressNameAdmin for a separate admin gateway 
Expose a new opt-in publishing.ingressNameAdmin that is propagated to
_cluster.expose-ingress-admin. System components (currently Keycloak)
use it to attach their admin routes to a separate Gateway/ingressClass,
e.g. one bound to a private IP and unreachable from the public internet.
Empty by default, so admin endpoints keep sharing expose-ingress. The
private Gateway itself is provisioned out of band; this only selects
where the admin route attaches.

Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
 2026-06-25 15:28:47 +05:00
Myasnikov DaniilandClaude Opus 4.8 2940093102 test(e2e): capture per-failed-test crust-gather snapshots in Chainsaw harness 
The Chainsaw app suite runs via `chainsaw test` directly, bypassing
hack/cozytest.sh — whose EXIT trap captures a crust-gather host snapshot
per failed BATS test. As a result a failing Chainsaw run uploaded a
cozyreport.tgz with zero per-test cluster snapshots, the richest diagnostic
for failures like the gateway/etcd/bucket/openbao ones on this branch.

Add the Chainsaw analog: a global spec.error.catch in .chainsaw.yaml that
collects a bounded host snapshot into _out/cozyreport/snapshots/<test>/host
on any failure (same --exclude-kind Secret flags and timeout -k 30 300 as
cozytest.sh), keyed by the $test.metadata.name binding with a mkdir-guard so
repeat catch invocations don't re-collect. Wire COZY_SNAPSHOT_NAME into the
kubernetes-latest/previous suites so the existing tenant snapshot from
run-kubernetes.sh co-locates under the same snapshots/<test>/ dir. cozyreport.sh
already folds snapshots/ into the uploaded artifact.

The e2e-testing guide and the sandbox Dockerfile comment are updated to record
the convention.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-25 13:43:50 +05:00
Myasnikov DaniilandClaude Opus 4.8 e22d84f2c8 test(e2e): migrate app suite from BATS to Chainsaw and wire into CI 
Replaces the per-app BATS E2E suite (hack/e2e-apps/*.bats) with declarative
Kyverno Chainsaw tests under hack/e2e-chainsaw/, extending this branch's pilot
(postgres + bucket) to all 18 app suites.

- Declarative ports assert on status/conditions and concrete fields; the
  repeated "timeout ... until kubectl get" + "kubectl wait" boilerplate
  collapses into a single polling assert with structured failure diffs and
  automatic events/describe/podLogs capture on failure.
- Inherently imperative suites keep their logic in script steps: openbao
  init/unseal, kuberture split-horizon probes, vminstance, the gateway
  admission/impersonation cases, and kubernetes-latest/previous, which wrap
  the relocated hack/e2e-chainsaw/_lib/run-kubernetes.sh verbatim.
- Gateway tests derive the tenant apex from the namespace host label at
  runtime, so they are host-independent.

CI: the e2e job now runs "chainsaw test hack/e2e-chainsaw/" via a new
test-chainsaw target and uploads the JUnit chainsaw-report.xml; the chainsaw
binary is added to the e2e-sandbox image. install-cozystack and test-openapi
stay BATS. The per-app 3-retry loop is dropped, since assertion polling
replaces the fixed-timeout flakiness it papered over.

Verified against a dev cluster (DB/app, storage and VM suites pass). Service
port assertions use the ports[*].port projection form because Chainsaw
v0.2.15 mis-evaluates JMESPath number-literal port filters.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-25 13:42:34 +05:00
Ivan Okhotnikov fff9e827d1 fix(operator): address review feedback on private platform source 
- Run `go mod tidy` to move github.com/fluxcd/pkg/apis/meta from
  indirect to direct require: it became a direct import after the
  SecretRef type fix.
- Refine the --platform-source-secret flag docstring and the matching
  values.yaml comment: drop identity.pub from the SSH required-keys
  list (Flux source-controller only consumes identity + known_hosts)
  and mention bearerToken as an alternative to username+password for
  Git over HTTPS.
- Note in values.yaml that the Secret must exist in cozy-system before
  the operator reconciles; otherwise source-controller reports a
  failed condition until the Secret is created.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
 2026-06-24 23:22:54 +03:00
IvanHunters 0427b184fa feat(operator): support private OCI registry credentials for platform source 
Added --platform-source-secret flag to cozystack-operator that allows
specifying a Secret name for registry authentication. This enables using
private OCI registries and Git repositories as platform sources.

Changes:
- cmd/cozystack-operator/main.go:
  - Add --platform-source-secret flag
  - Pass secretName to generateOCIRepository() and generateGitRepository()
  - Set spec.secretRef in generated resources when secret is specified

- packages/core/installer/values.yaml:
  - Add cozystackOperator.platformSourceSecret parameter

- packages/core/installer/templates/cozystack-operator.yaml:
  - Pass --platform-source-secret flag to operator container

Usage:
1. Create registry credentials secret in cozy-system namespace:
   kubectl create secret docker-registry my-registry-creds \
     --namespace cozy-system \
     --docker-server=registry.example.com \
     --docker-username=user \
     --docker-password=token

2. Configure Helm values:
   cozystackOperator:
     platformSourceUrl: 'oci://registry.example.com/repo/packages'
     platformSourceSecret: 'my-registry-creds'

Closes #2455

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-06-24 22:54:06 +03:00