admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-10-30 16:47:20 +09:00
Code Issues Packages Projects Releases Wiki Activity
4,971 Commits 539 Branches 294 Tags
2642965026bec3248c83f14bd4e89dec552ce594
Commit Graph
609 Commits
Author SHA1 Message Date
Aleksei Sviridkin 1e4a152291 test(mariadb): give first boot the readiness budget the probe grants 
The suite allowed 3 minutes for a MariaDB endpoint address to appear,
which was set when a replica had 40 seconds to finish its first boot.
The startup probe now grants 310 seconds, so a healthy install can
legitimately take longer than the gate allows and would be failed while
the probe is still willing to wait.

The endpoint assert is the gate that matters here. The chart hands the
operator a MariaDB custom resource and ships no built-in workload, so
helm has nothing to wait on and the HelmRelease reports Ready as soon as
the manifests are applied, long before a server exists. An endpoint
address, by contrast, appears only once a replica has passed its startup
and then its readiness probe, so that assert is what actually covers
bootstrap.

Raise it to 10m in both the replicated and the single-replica test. One
ready address satisfies the assert, so it has to clear a single startup
budget rather than one per replica, plus the operator reconcile, PVC
bind and image pull that precede it. The HelmRelease ceilings stay at
6m, since nothing about them gates on the server coming up.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-19 01:03:05 +03:00
Andrey KolkovandClaude Fable 5 8282c17d08 test(sdn): scope the new validate test's comment to what it proves 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-17 14:45:58 +04:00
Andrey KolkovandClaude Fable 5 bbc0103949 test(sdn): pin the fail-closed suite-mapping contract in validate 
Scrub the last "-" mention from the --validate usage doc (the escape no
longer exists, so the header asserted the opposite of the behavior), and
add the negative bats case the tightened branch was missing: a suite
whose hardcoded mapping names a PackageSource absent from the graph must
fail validation with the "maps to ..., not a PackageSource" error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-17 14:41:13 +04:00
Andrey KolkovandClaude Fable 5 4e9a40d811 test(sdn): drop the now-unreachable "-" mapping from suite_to_source 
With the serviceexposure case gone no suite returns "-", and the only
executable coverage of that branch was the bats test removed alongside
it. Keeping an untestable dead branch for a hypothetical future suite is
not worth it — re-adding the three lines with the next core-platform
suite is cheaper, and validate/closure both simplify.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-17 14:30:21 +04:00
Andrey KolkovandClaude Fable 5 bb1127c767 test(sdn): drop the serviceexposure chainsaw suite left after the feature removal 
Commit 3dbc7d366 removed ExposureClass/ServiceExposure (api/network, the
controller, and the CRDs shipped in cozystack-controller/definitions) in
favor of the native LoadBalancer path, but the chainsaw port that landed
the next day (dd2f89d20) brought the serviceexposure suite along. The
suite creates network.cozystack.io/v1alpha1 fixtures against an API that
no longer exists, so every full-suite run (any PR touching core paths,
and every release cut) fails its two tests with "no matches for
network.cozystack.io/v1alpha1".

Remove the dead suite and its wiring: the suite_to_source case and its
core-platform bats test in select-install (the "-" mechanism stays for
the next core-platform-owned suite), and the README suite list.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-17 14:20:56 +04:00
Andrei Kvapil fbbe4af7df refactor(network): drop ExposureClass/ServiceExposure for native loadBalancerClass (#3218) 
## What

Removes the `network.cozystack.io` API group — `ExposureClass` +
`ServiceExposure` (#3081) — and its reconciler, in favor of native
`Service` `type: LoadBalancer` + `loadBalancerClass`.

## Why

Full rationale and discussion in #3164. For **managed applications** the
chart already owns the Service, so:

- `type: LoadBalancer` + `loadBalancerClass` + an admin-provisioned pool
covers vendor-neutral external exposure without a dedicated API group
and controller;
- the `externalIPs` (node-IP) default never went through
`ServiceExposure` (its backend is a no-op) and is left unchanged;
- the only current consumer, the host ingress, keeps rendering `type:
LoadBalancer` and just drops the `ServiceExposure`; the admin provisions
the address pool / announcer out of band, as the install docs already
assume.

`ServiceExposure`'s one unique feature — dynamic per-namespace scoping
of a shared VIP pool — is a convenience the managed-app path does not
need, and it does not justify a second tenant-networking API group next
to `sdn.cozystack.io`.

## Changes

- Delete `api/network/v1alpha1` and
`internal/controller/serviceexposure` (+ CRD definitions, e2e bats,
controller registration, RBAC).
- Host ingress: drop the `ServiceExposure` template; add an optional
`publishing.loadBalancerClass` that sets `spec.loadBalancerClass` on the
host-ingress LoadBalancer Service.
- The `externalIPs` path and default behavior are unchanged.

## Verification

- `go build ./...` — green.
- `helm unittest` — `core/platform` 72/72, `extra/ingress` 25/25 (incl.
host-vs-tenant `loadBalancerClass` scoping and the immutable-field
migration guard).

## Migration

None required. `network.cozystack.io` (#3081) has never shipped in a
release — it is on `main` only (not backported to `release-1.5`; absent
from v1.4.x / v1.5.x), so no upgrading cluster has these CRDs and this
removes it before it is ever released.

Clusters that track `main` and explicitly set `publishing.exposureClass`
(opt-in, only since 2026-06-30) should provision their MetalLB / Cilium
address pool themselves and set `publishing.loadBalancerClass`; any
orphaned `cozystack-<class>` pool can be deleted by hand. Default
clusters (externalIPs path) are unaffected.

## Status

**Draft / RFC for the #3164 discussion** — this removes an API group
that landed only days ago, so nothing here should merge before the
design call. cc @lexfrei (author of #3081) and @lllamnyp (#3164).

Deferred: when Kubernetes actually removes `Service.spec.externalIPs`,
the node-IP default migrates via a MetalLB `IPAddressPool` with no
`L2Advertisement` (allocate-but-don't-announce) — a separate change.



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for configuring the ingress host Service with a native
`loadBalancerClass` value.
* Expanded chart tests to verify class wiring and immutability behavior
across install and upgrade scenarios.

* **Bug Fixes**
* Removed the legacy exposure-class/service-exposure flow and related
controller behavior.
* Updated templates and CRD packaging so the platform no longer renders
or depends on those deprecated resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-16 17:44:28 +02:00
myasnikovdaniil f6a03e93f8 fix(etcd): complete v1alpha2 transition for in-cluster 1.5→1.6 upgrades (#3270) 
Consolidates the etcd `v1alpha2` transition fix for **in-cluster 1.5 →
1.6 upgrades** into a single PR, rebased on current `main`. Supersedes
#3265 and #3261 — their commits are carried here (authorship preserved),
so those PRs can be closed once this lands. Fresh installs use the new
shapes directly and are unaffected, so fresh-install CI does not catch
these; the 1.5 → 1.6 e2e upgrade path is the authoritative regression
guard.

## What this PR does

### 1. Keep the legacy `etcd-headless` Service alive during adoption —
`packages/extra/etcd`
`etcd-migrate` adopts legacy clusters **in place**: the Pods keep their
original `spec.subdomain: etcd-headless` and are dialed at
`etcd-<i>.etcd-headless.<ns>.svc` until they roll onto the operator's
native `<member>.etcd.<ns>.svc` domain. The v1alpha2 operator only
creates the native `etcd` Service and the legacy `etcd-headless` Service
is pruned, so those per-pod names stop resolving (`no such host`),
`MemberList` fails, and `status.readyMembers` never populates — the
`EtcdCluster` never goes `Ready` even though etcd is healthy and in
quorum. We ship a chart-managed transitional headless `etcd-headless`
Service (selector mirrors the operator's native `etcd` Service via
`etcd-operator.cozystack.io/cluster`, `publishNotReadyAddresses: true`)
— the DNS counterpart of the legacy `*.etcd-headless.<ns>.svc` SAN
already kept for this window. Removable together with that SAN once
members roll onto the native subdomain.

### 2. Survive the immutable controller-Deployment selector on upgrade —
`packages/system/etcd-operator` (#3242)
#2859 replaced the upstream etcd-operator chart with the
cozystack-authored one, changing `Deployment.spec.selector.matchLabels`.
`spec.selector` is immutable, so `helm upgrade` cannot patch the
existing Deployment and the whole HelmRelease upgrade fails (`field is
immutable`). A **pre-upgrade hook**
(`templates/pre-upgrade-selector-fix.yaml`: ServiceAccount + Role +
RoleBinding + Job) deletes the Deployment **only** when its live
selector is the pre-1.6 one, so Helm recreates it cleanly. No-op when
the selector already matches, never runs on fresh install.

### 3. Raise the operator's memory cold-start floor —
`packages/system/etcd-operator`
Steady-state working set is ~250Mi; the static `limits.memory: 128Mi`
OOMKills a Pod that starts before the VPA admission webhook rewrites it.
Raise the floor to `256Mi` (and VPA `minAllowed` to match) as defense in
depth so the operator never depends on VPA timing to avoid crashing.

### 4. Make migration 50 (etcd adoption) robust in-cluster —
`packages/core/platform/images/migrations/migrations/50` (was #3261)
Exact server/peer cert-SAN match, Secret-gated wait on the adoption
Secret, and in-cluster (IPv6-safe) kubeconfig handling, so the migration
script drives the adoption reliably from inside the cluster. Covered by
`hack/migration-50-etcd-adopt.bats`.

### 5. Hardening / review fixes (this PR's original scope)
- **Hook `runAsUser: 65532`.** `pre-upgrade-selector-fix.yaml` set
`runAsNonRoot: true` but no numeric `runAsUser`; `clastix/kubectl`'s
image user is the non-numeric name `nonroot`, which the kubelet cannot
verify against `runAsNonRoot` — so the hook Pod fails admission and
silently blocks the very upgrade it exists to unblock. Adds `runAsUser:
65532`, matching the postgres-operator webhook-ready hook that runs the
same image.
- **Digest-pin the kubectl image.** The values comment claimed
digest-pinning but shipped a floating `v1.32` tag. Reuses the digest
postgres-operator vendors, adds the `renovate` annotation, and templates
`repo:tag@digest`.
- **Tests.** `etcd-operator/tests/selector-fix-hook_test.yaml` (hook
wiring, weight ordering, namespaced least-privilege RBAC,
numeric-non-root security context, digest-pinned image),
`etcd-operator/tests/deployment_test.yaml` (256Mi cold-start floor),
`extra/etcd/tests/etcd-cluster_test.yaml` (transitional `etcd-headless`
Service). Each assertion was mutation-tested.

### 6. Derive the default S3 endpoint from the provisioned bucket —
`packages/system/backupstrategy-controller`
Derive the default S3 endpoint (and per-driver scheme / TLS /
`secure_connection`) from the provisioned bucket Secret instead of
requiring it to be hand-set, so etcd (and other) backup strategies get a
working endpoint by default. Docs in
`docs/operations/backup-classes.md`; covered by
`tests/endpoint_form_test.yaml`.

### Verification
- `helm unittest` green on current `main`: etcd-operator **18/18**,
extra/etcd **18/18**, backupstrategy-controller **11/11**.
- Behaviours 1 & 2 were reproduced and confirmed live on a 1.5.2 →
1.6.0-rc.1 adoption (3-node cluster) — recreating the `etcd-headless`
Service took the adopted cluster to `readyMembers=3 / Available=True`.

```release-note
fix(etcd): complete the v1alpha2 transition on in-cluster 1.5→1.6 upgrades — keep the legacy etcd-headless Service alive so adopted members stay resolvable, delete the pre-1.6 operator Deployment via a pre-upgrade hook to get past the immutable selector, raise the operator's memory floor so it does not OOM before the VPA scales it, and make the etcd adoption migration robust in-cluster.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Improved backup storage endpoint handling across supported backup
drivers, including provisioned and external S3 storage.
* Added compatibility support for legacy etcd pod discovery during
migration.
  * Added an automated upgrade safeguard for etcd operator deployments.

* **Bug Fixes**
* Improved certificate SAN detection and etcd migration authentication.
* Increased the etcd operator’s minimum startup memory to prevent early
restarts.

* **Documentation**
  * Clarified backup endpoint, TLS, and driver-specific behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-16 12:37:00 +05:00
IvanHunters 3fb8ec5b0e feat(monitoring): per-instance OIDC selector for the Grafana instance (Phase 1) (#3176) 
<!-- Follow-up to cozystack/cozystack#3044 — same design family,
different subject. -->

## What this PR does

Adds a Phase-1 OIDC selector to the Grafana instance shipped by
`packages/system/monitoring/`, mirroring the shape of the tenant
kube-apiserver PR
([cozystack/cozystack#3044](https://github.com/cozystack/cozystack/pull/3044))
so that operators can use one mental model for both. Rationale in
[cozystack/community#24](https://github.com/cozystack/community/pull/24)
— per-cluster (here: per-Monitoring-release) audience isolation on the
flat `cozy` realm, no per-tenant realm.

Authorization is **app-side** via a chart-owned `users:` map reconciled
by a post-install/post-upgrade Job into Grafana org membership. The
chart does NOT own Keycloak directory objects (no `KeycloakRealmGroup`s)
and does NOT rely on Grafana's `role_attribute_path` — same design shape
as #3044's ClusterRoleBinding reconcile, adapted for Grafana orgs. See
`docs/oidc-grafana.md` for the full rationale.

The `Monitoring` CR grows a new field:

```yaml
spec:
  oidc:
    mode: None | System | CustomConfig     # default None
    customConfig:                          # only for CustomConfig
      config: {}                           # inline grafana.ini keys, or...
      secretRef:
        name: ""                           # ...an operator-managed Secret
    users:                                 # optional; ignored in mode=None
      - email: alice@example.com
        role: Admin | Editor | Viewer
```

### Modes

- **`None`** (default) — no OIDC; `admin_user` / `admin_password` Secret
is the only auth path. Existing releases render byte-identical.
- **`System`** — chart provisions, in the release namespace of the
Monitoring CR:
- `KeycloakClient` (`public: false`, confidential, `secret` sourced from
a chart-owned Secret via the EDP `$<secret>:<key>` reference;
`redirectUris` locked to `https://grafana.<host>/login/generic_oauth`).
- `KeycloakClientScope` with an `oidc-audience-mapper` pinning
`id_token.aud` to the per-release clientId — the isolation primitive.
- Persistent Kubernetes Secret carrying the confidential `client-secret`
(random on first install, preserved via `lookup`+fallback, same pattern
as `packages/system/dashboard`).
- Grafana CR's `spec.config.auth.generic_oauth` section wired to the
`cozy` realm issuer + per-release audience scope, with two chart-forced
settings the users-Job depends on: `skip_org_role_sync: "true"` (login
never overwrites the Job's org-role assignments) and
`oauth_allow_insecure_email_lookup: "true"` (OIDC identity binds to the
pre-provisioned local account by email).
- `GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET` env on the Grafana Deployment
sourced from the Secret.
- Post-install / post-upgrade **users-Job** reconciling
`spec.oidc.users` into Grafana's Main Org.: pre-provisions a local
account per listed email, `POST /api/orgs/1/users` + `PATCH
/api/orgs/1/users/{userId}` to converge role changes, prunes every
non-admin Main-Org member whose email is not in the list.
- Server-level `GrafanaAdmin` promotion (`allow_assign_grafana_admin`)
is out of scope for Phase 1. Every Grafana instance (platform and
tenant) caps at org-level `Admin`.
- **`CustomConfig`** — tenant supplies the whole `[auth.generic_oauth]`
payload; `cozy` is not in the path. Two mutually exclusive paths:
- `customConfig.config: {...}` — inline map of `grafana.ini` keys. The
chart **merges** the two chart-forced settings (`skip_org_role_sync`,
`oauth_allow_insecure_email_lookup`) on top so the users-Job contract
holds. The operator can NOT override those two (merge wins).
- `customConfig.secretRef.name: ...` — operator-owned Secret with an
`auth.ini` fragment; the chart mounts it under `/etc/grafana/oidc` and
wires `GF_PATHS_CUSTOM_INI`, and does NOT emit `auth.generic_oauth`
under `spec.config` (the ini fragment is authoritative). Because the
chart cannot inject the two settings into a mounted ini,
`spec.oidc.users` is NOT supported in this branch — the chart hard-fails
render on that combination.

### Fail-close guards

- `mode: System` without `_cluster.oidc-enabled == "true"` (the
platform-level `authentication.oidc.enabled` flag) → hard-fail with an
actionable message pointing at the platform values.
- `mode: System` on a cluster where the EDP Keycloak operator CRDs
(`v1.edp.epam.com/v1`) are not registered → hard-fail (symmetric on
`oidc-keycloak.yaml` and `grafana.yaml`).
- `mode: CustomConfig` with both or neither of `config` /
`secretRef.name` → hard-fail.
- `mode: CustomConfig` with `secretRef.name` AND non-empty `users` →
hard-fail (chart cannot inject the two forced settings into an
operator-mounted ini; forcing an explicit choice avoids silent contract
loss).
- `spec.oidc.users[]` entry with missing / unknown `role` or malformed
`email` → hard-fail at render time so the users-Job never fires with a
body that would 400 on Grafana's admin API.

### Break-glass posture

`admin_user` / `admin_password` and `disable_login_form: false` are
unchanged in every mode. Locking the form off under `mode: System` is a
documented follow-up hardening (see `docs/oidc-grafana.md`).

### What's NOT in this PR (documented as out of scope)

- Per-tenant Keycloak realms (Phase 2 territory, tracked in
community#24).
- Full-logout through Keycloak's end-session endpoint
(`backend-logout-url` and matching Keycloak client attribute).
- CEL `claimValidationRules` gate on `email_verified` — layered
guarantees are described in `docs/oidc-grafana.md`.
- Multi-issuer composition on one Monitoring release (mutually exclusive
modes).
- Server-level `GrafanaAdmin` promotion.

### Docs

- New `docs/oidc-grafana.md` — full operator guide (parallel to
`docs/oidc-tenant.md`).
- Website page follows in a paired PR against cozystack/website
(`docs/monitoring-oidc-authentication` branch, under
`content/en/docs/next/operations/services/monitoring/oidc-authentication.md`).

### Tests

- 33 `helm-unittest` cases in `tests/oidc_test.yaml` +
`tests/oidc_crd_missing_test.yaml` cover: all three modes; defensive nil
paths (`oidc: ~`); System-mode Keycloak client + audience scope shape;
CustomConfig chart-forced merge (positive assertion) AND
`mode=CustomConfig inline — operator cannot override chart-forced OIDC
settings` (regression guard); CustomConfig secretRef + `users` incompat
fail-fast; `spec.oidc.users[].role` missing/unknown fail-fast; users-Job
shape (hook annotations, RBAC-free, activeDeadlineSeconds,
`ttlSecondsAfterFinished: 3600`, admin credential wiring); no
chart-owned `KeycloakRealmGroup`s in any mode; no `role_attribute_path`
in Grafana config.
- Render-side bats in
`hack/e2e-apps/monitoring-oidc-{system,customconfig}.bats` — mirror the
tenant kube-apiserver's e2e shape (no live browser flow — deferred to a
follow-up integration suite, same as #3044). Asserts on live cluster:
users-Job appears with `DESIRED_USERS_JSON` populated, Grafana config
carries `skip_org_role_sync=true` and no `role_attribute_path`, no
chart-owned `KeycloakRealmGroup`s land in `cozy`.

### Screenshots

N/A — no UI change on default values; the visual delta is Grafana's own
"Sign in with Keycloak" button appearing under the login form when a
release opts in.

### Release note

```release-note
feat(monitoring): Each `Monitoring` CR can now opt its Grafana instance into OIDC via a flat selector — `spec.oidc.mode: System | CustomConfig | None` (default `None`). `System` trusts the platform `cozy` realm via a per-instance confidential Keycloak client + audience binding. Authorization is app-side: `spec.oidc.users: [{email, role: Admin|Editor|Viewer}]` is reconciled into Grafana's Main Org. by a chart-owned post-install/post-upgrade Job (pre-provision → org add → PATCH role → prune stale). `CustomConfig` accepts a tenant-supplied `[auth.generic_oauth]` payload — inline as a map (chart merges the two contract-critical settings `skip_org_role_sync` + `oauth_allow_insecure_email_lookup` on top) or as a Secret with an `auth.ini` key (in which case `spec.oidc.users` is not supported and the chart fails render on that combination). Server-level `GrafanaAdmin` promotion is out of scope for Phase 1. The `admin_user`/`admin_password` Secret stays a documented break-glass path in every mode.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added OIDC authentication for Monitoring Grafana with `None`,
platform-managed `System`, and tenant-controlled `CustomConfig` modes.
* System mode provisions platform-backed OIDC for each instance
(including persistent client credentials) and can optionally promote
matching users to server-level `GrafanaAdmin`.
* CustomConfig supports inline OAuth settings or Secret-backed
`auth.ini` (with mutual exclusivity validation).
 2026-07-15 18:17:56 +03:00
Andrey Kolkov 47d239c519 fix(platform): address review on migration 50 (exact SAN match, Secret-gated wait, IPv6 kubeconfig) 
Review feedback on #3261 (gemini-code-assist, coderabbitai, myasnikovdaniil):

- Exact SAN match. _san_present read the Certificate SANs as {.spec.dnsNames}
  (a bracketed JSON blob that `tr ','` cannot split) and matched with a
  substring `grep -qF`, so a wildcard that is a substring of a longer SAN
  (e.g. *.etcd.<ns>.svc inside *.etcd.<ns>.svc.cluster.local) could
  false-positive and skip the re-issue. Read {.spec.dnsNames[*]} (space
  separated) and match exactly with `grep -qxF`, matching the Secret-annotation
  branch.

- Wait on the re-issued Secret, not the patched spec. The post-patch wait loop
  called _san_present, which returns as soon as the Certificate spec.dnsNames
  contains the wildcard -- but the patch just added it there, so the loop broke
  on the first iteration before cert-manager re-issued the Secret, and the
  "re-issued" log printed unconfirmed. Add _secret_has_san (Secret alt-names
  annotation only, same empty-read retry + exact match) and gate the wait on it.

- IPv6-safe kubeconfig server. The synthesized kubeconfig used
  https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT, which is an invalid
  URL on an IPv6-only cluster (a bare IPv6 host must be bracketed). Use
  https://kubernetes.default.svc -- IP-family-agnostic, validated by the mounted
  SA CA, and the same server URL kubectl synthesizes in-cluster.

Tests (the review blocker): extend the bats suite (and its fake kubectl) with
two cases pinning these contracts -- a superstring dnsName must NOT skip the
re-issue patch, and etcd-migrate must be invoked with a synthesized in-cluster
kubeconfig whose server is kubernetes.default.svc (not the bare IPv6 host),
authenticating via the SA token file + CA. Both the kubeconfig path and the SA
directory are now env-overridable (ETCD_MIGRATE_KUBECONFIG, ETCD_ADOPT_SA_DIR)
so the synthesis is exercisable off-cluster; production defaults are unchanged.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-15 19:11:35 +05:00
myasnikovdaniil 215bd430a9 test(e2e): diff-driven install selector select-install.sh (#3279) 
## What this PR does

Adds `hack/select-install.sh` — the install-side companion to
`hack/select-e2e.sh`. Where `select-e2e.sh` picks *which* Chainsaw
suites run for a diff (a reverse dependency walk), `select-install.sh`
picks *what must be installed* for those suites: the forward `dependsOn`
closure over the `PackageSource` graph.

Given the suite names `select-e2e.sh` emits, it maps each back to its
`*-application` `PackageSource` and walks `spec.variants[].dependsOn`
forward to the transitive closure — the minimal set of packages to
enable instead of provisioning the full `isp-full` platform. A
`postgres` chart change, for example, resolves to 9 of 97 sources.

Design notes:

- Reuses the real `PackageSource` graph in
`packages/core/platform/sources/` — no parallel dependency-declaration
files to keep in sync.
- Unlike `select-e2e.sh`'s reverse walk, the forward walk **keeps** the
`cozystack.cozystack-engine` edge: for test *selection* it is noise (it
would fan every app out to every other), but for *install* it is a
genuine prerequisite (the app's `*-rd` HelmRelease needs the engine to
register the ApplicationDefinition CRD). Because that edge lives only on
`*-application` sources, a suite backed by a **system** package (e.g.
`securitygroup`, whose `SecurityGroup` in `sdn.cozystack.io` is served
by the aggregated `cozystack-api` — a `cozystack-engine` component)
can't reach the engine via the walk, so the engine is **seeded
explicitly for any non-empty closure** (and a graph missing it is a hard
error).
- `--validate` mode checks the whole graph — every `dependsOn` target
resolves to a real `PackageSource` (reachability) and there are no
dependency cycles — **and** the suite mapping: every
`hack/e2e-chainsaw/*/` suite resolves to a real `PackageSource` or to
`-` (a missing suites dir is itself a failure). The graph lives next to
the packages and can't drift from the suite dirs; the hand-maintained
mapping is the only part that can, so this is the cheap CI guard.
- Suite→source mapping follows the `<suite>-application` convention with
a bare-`<suite>` fallback, plus explicit cases for the irregular names
(`kubernetes`/`vminstance`/`securitygroup`) and for core-platform
features that have no separately-enabled package (`serviceexposure`,
whose `ExposureClass`/`ServiceExposure` CRDs live in
`packages/core/platform`). A suite may reference other app kinds by
selector without standing them up (e.g. `securitygroup`'s
`SecurityGroup` names `Postgres`/`Kubernetes` as selectors only) — only
the primary package is mapped here; the forward closure and the seeded
engine cover its prerequisites, and full multi-package modelling is left
to the `test-minimal` follow-up. A suite with no known mapping is a
**hard error** (fail closed), not a silent skip.

Includes a 17-case unit suite (`hack/select-install_test.bats`) covering
single-app closure, transitivity, multi-suite union, stdin input, the
kubernetes/irregular/core-platform suite mappings, the `securitygroup`
engine-seed regression, empty input, fail-closed unmapped suites (all
reported, no partial output), the missing-engine hard error, and the
`--validate` failure modes (dangling dep, cycle, unmapped suite dir,
missing suites dir). It is auto-discovered by `make unit-tests`
(`BATS_UNIT_FILES` globs `hack/*.bats`).

Verified end-to-end against real git diffs across six path types (app
chart, shared operator, docs-only, per-suite Chainsaw edit, library,
shared e2e helper): a `postgres` chart change scopes the install to 9 of
97 sources; the full-suite fallback to 39 of 97.

This is Track A of #2847 (diff-driven & scenario-based E2E); tracked in
#3278. Consuming the output — a `test-minimal` platform variant/bundle
plus wiring into the `pull-requests.yaml` install step — is a deliberate
follow-up, since it depends on a platform-variant design decision.

### Screenshots

Not applicable — CI/tooling-only change.

### Release note

```release-note
test(e2e): add hack/select-install.sh to compute the minimal package install set for a diff-scoped E2E run (forward PackageSource dependency closure), with a --validate graph-consistency mode
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added a tool to select the minimal package set required for one or
more end-to-end test suites.
* Supports combining multiple suites, resolving transitive dependencies,
and handling unmapped or empty suite selections.
* Added validation for missing dependencies and circular dependency
definitions.

* **Tests**
* Added coverage for package selection, dependency resolution, suite
mappings, empty selections, and validation failures.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-15 18:28:41 +05:00
IvanHunters 3776c447b9 test(e2e): poll users-Job DESIRED_USERS_JSON for target email, not just Job existence 
CI E2E showed the users-Job assertion returning `DESIRED_USERS_JSON:
[]` — but the CR body had `users: [{email: e2e-admin@example.com,
role: Admin}]`. Root cause: `hook-delete-policy: before-hook-creation`
keeps the previous hook Job around across helm upgrades, so `until
kubectl get job` matches a stale Job from the previous customconfig-
mode test where users may have been unset (the secretRef swap CR had
no `users:` field, so the schema default `[]` propagated to the
prior Job's DESIRED_USERS_JSON). The System test then grabbed that
stale Job before the helm upgrade for the new CR completed and swapped
in the fresh Job.

Fix: poll on the target env value itself — the new Job with the
correct users list is the first Job whose DESIRED_USERS_JSON string
contains `e2e-admin@example.com`. 600s window matches the inner HR's
15m timeout with room for a slow reconcile.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-15 15:31:45 +03:00
IvanHunters 7f9e1a8bf6 test(e2e): poll the reconciled Grafana CR field instead of racing on CR existence 
CI run 28794778739 caught two race conditions in the render-side
Monitoring OIDC bats:

- `Grafana CR carries auth.generic_oauth pointing at the cozy realm`
  (monitoring-oidc-system.bats:86) failed with `enabled=` empty after
  a 0-second wait. The prior `until kubectl get grafana grafana`
  only proved the CR EXISTS — it did not prove grafana-operator had
  reconciled the mode=System values onto it. When the CR from the
  previous `Monitoring CR accepts spec.oidc.mode=System` test was
  already there but the operator hadn't yet flipped `.spec.config.auth
  .generic_oauth.enabled` to "true", the jsonpath returned empty and
  `[ "" = "true" ]` short-circuited the whole test.
- `secretRef variant mounts operator Secret under /etc/grafana/oidc`
  (monitoring-oidc-customconfig.bats:119) timed out at 180s waiting
  for the volumeMount swap from inline → ini. On a warm sandbox the
  full inner-HR upgrade + grafana-operator reconcile cycle can
  legitimately take >3m — the timeout was too tight.

Fix: replace `until kubectl get grafana` with `until [ "$(kubectl
... jsonpath=<target-field>)" = <expected> ]` on every subsequent
assertion, and bump the secretRef-swap window to 600s (well inside
the inner HR's 15m timeout). Same pattern used for the "no
role_attribute_path" and "operator payload verbatim" assertions so
the whole suite is race-free.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-15 15:31:45 +03:00
IvanHunters 123616a1d0 test(e2e): swap KeycloakRealmGroup assertions for the users-Job in bats 
Aligns the render-side e2e bats with the users-map contract:

- Feed a `spec.oidc.users:` entry to each mode's Monitoring CR so
  the reconcile Job has real input to act on.
- Assert the users-Job appears in the release namespace with the
  desired `DESIRED_USERS_JSON` env and the expected `OIDC_MODE`.
- Assert the Grafana CR renders with `skip_org_role_sync = "true"`
  and `oauth_allow_insecure_email_lookup = "true"` and NO
  `role_attribute_path`.
- Flip the three KeycloakRealmGroup existence assertions to
  negative checks: chart-owned realm groups must NOT be present
  in either System or CustomConfig mode.

Does not add a full OIDC login flow through Keycloak — that stays
deferred to a follow-up integration suite, same posture as the
tenant kube-apiserver's kubernetes-oidc-system.bats.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-15 15:31:45 +03:00
Ivan Okhotnikov 5f7ed89799 fix(e2e): work around EDP schema-default strip + kubectl delete/apply race 
Two follow-up bats fixes uncovered by the first live e2e pass on the
singleton-aligned monitoring OIDC tests:

1. monitoring-oidc-system, test 4 (KeycloakClient assertion). The EDP
   Keycloak operator's KeycloakClient CRD strips `spec.public: false`
   on write because false is the schema default. The chart emits
   `public: false` explicitly (confidentiality guard), but neither
   managedFields nor the resulting spec ever carries the field, so
   the assertion `[ "${public}" = "false" ]` grepped an empty string
   and tripped. Assert `spec.clientAuthenticatorType == "client-secret"`
   instead — the operator persists it, and it is the direct signal
   for "confidential client" (which is what the `public: false`
   assertion was testing for).

2. monitoring-oidc-customconfig, test 4 (secretRef variant). The
   original flow was delete-then-recreate the Monitoring CR to swap
   inline `customConfig.config` for `customConfig.secretRef.name`.
   `kubectl apply` immediately after delete raced the API server's
   grace-period cache: apply's GET-then-decide phase saw the CR
   during grace, chose the patch path, and the patch hit NotFound
   once the delete propagated. Drop the delete entirely — cozystack-
   operator reconciles the CR mode change on the same object and
   Flux re-renders the chart with the new values.

   Add an explicit wait for the target state (Grafana CR's
   `oidc-custom-ini` volumeMount present AND `spec.config.auth.generic_oauth`
   absent) before the assertions, since the CR update triggers an
   asynchronous chart re-render and the immediate get would still
   see the inline auth.generic_oauth block from @test 1.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:45 +03:00
Ivan Okhotnikov cb04caecee fix(e2e): align monitoring OIDC bats with chart singleton naming 
The extra/monitoring chart enforces `.Release.Name == "monitoring"`
in templates/check-release-name.yaml (the tenant Monitoring is a
per-namespace singleton). Naming the Monitoring CR anything else
makes the operator materialise an outer HelmRelease with that name
and the chart hard-fails at render with

  execution error at (monitoring/templates/check-release-name.yaml:2:4):
    The name of the release MUST BE monitoring

Rename the CR to `monitoring` in both files. Rework every derived
name to match the actual reality of the chart pipeline instead of
the invented `monitoring-<TEST_NAME>` prefix the tests polled for:

  * outer HR  = ${CR_NAME}                        = monitoring
  * inner HR  = ${CR_NAME}-system                 = monitoring-system
    (rendered by extra/monitoring/templates/helmrelease.yaml as
     `metadata.name: {{ .Release.Name }}-system`)
  * clientId  = ${namespace}-${inner Release.Name}
              = tenant-test-monitoring-system
  * scope     = ${clientId}-audience
  * groups    = ${clientId}-{admin,editor,viewer}
  * client-secret Secret = ${inner Release.Name}-oidc-client
                         = monitoring-system-oidc-client

The BYO Secret name in the customconfig test is chart-independent
(the tenant provides it via customConfig.secretRef.name), so it is
kept short and no longer tied to CR_NAME.

The wait sequence in the first @test of each file is also split
into two probes: outer HR appears first (from the operator), then
inner HR (from the extra/monitoring chart's helmrelease.yaml), then
the Grafana CR. Under the previous incorrect single-probe wait
(`hr monitoring-${TEST_NAME}`) the timeout tripped after 60s
because that HR never existed under any name.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:45 +03:00
Ivan Okhotnikov 9ff05b2f32 test(e2e): port monitoring OIDC bats to cozytest lifecycle conventions 
Mirrors the same port applied to kubernetes-oidc-*.bats in the tenant
kube-apiserver OIDC series: cozytest.sh (the e2e runner) is not real
bats, so setup()/teardown() are never invoked and any state they
established (TEST_NAME) is unset by the time an @test body runs,
tripping cozytest.sh's `set -eu` at the first "${TEST_NAME}"
expansion with `TEST_NAME: parameter not set`.

Lift TEST_NAME to file scope as a stable constant, register cleanup
via cozy_cleanup() so cozytest.sh runs it at suite exit and on the
first failing test, and call cleanup_mon() inline at the top of the
first @test for per-run isolation. Fixes the two E2E failures on
this PR: `Monitoring CR accepts spec.oidc.mode=CustomConfig with
inline config` and `Monitoring CR accepts spec.oidc.mode=System`.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:45 +03:00
Ivan Okhotnikov fe2c606f79 test(e2e): add Phase-1 OIDC render-side bats for Monitoring 
Two bats files mirroring the shape of the tenant kube-apiserver PR's
kubernetes-oidc-{system,customconfig}.bats:

- monitoring-oidc-system.bats — asserts the Monitoring CR admits the
  new spec.oidc.mode: System field; the chart renders the Grafana CR
  with an auth.generic_oauth section pointing at the cozy realm; a
  32-char client-secret Secret is persisted; the KeycloakClient (with
  public: false), KeycloakClientScope (audience mapper), and three
  KeycloakRealmGroups (-admin, -editor, -viewer) land in cozy; and
  the Grafana Deployment injects GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET
  from the Secret.

- monitoring-oidc-customconfig.bats — asserts the CR admits inline
  CustomConfig; the operator's payload lands on spec.config verbatim
  with no cozy realm in the auth_url; NO KeycloakClient / Scope /
  Groups are created in cozy; and the secretRef variant mounts the
  operator Secret under /etc/grafana/oidc without emitting
  auth.generic_oauth under spec.config.

Neither drives a real browser login through Keycloak — that is
deferred to the follow-up integration suite, same posture as the
tenant kube-apiserver's Phase-1 tests.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
 2026-07-15 15:31:44 +03:00
Myasnikov Daniil 1f084cecc8 test(e2e): harden select-install.sh per review 
Address the review blockers on the diff-driven install selector:

- B1: seed cozystack.cozystack-engine for any non-empty closure. Suites
  backed by a system package (e.g. securitygroup, which applies a
  sdn.cozystack.io resource served by the aggregated cozystack-api) carry
  no dependsOn edge to the engine, so the forward walk could never reach
  it and the emitted set could not run the suite. Hard-error if the engine
  source is absent from the graph.
- B2: an unmapped suite is now a hard error in closure mode (every bad
  suite reported in one run, no partial output) instead of failing open.
  --validate additionally asserts every hack/e2e-chainsaw suite resolves
  to a real PackageSource or "-", and fails when the suites dir is missing.

Also correct the inaccurate "securitygroup also stands up a Kubernetes
app" comment (its fixtures reference Postgres/Kubernetes as selectors
only), note the variant-union over-selection in build_forward_deps, drop
the unnecessary source copies from the tests, and add coverage for stdin
input, the engine seed, fail-closed mapping, the --validate suite guard,
and the missing-suites-dir / missing-engine failure modes.

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-15 14:48:30 +05:00
Aleksei Sviridkin 6b1ad96093 fix(e2e): gate LINSTOR node-list probe on a ready controller endpoint (#3250) 
## What this PR does

The E2E `Install Cozystack` step runs `hack/e2e-post-install-prep.sh` in
the background, which waits for 3 LINSTOR satellites to come Online by
dialing the `linstor-controller` Service
(`linstor+ssl://linstor-controller:3371`) from the controller pod. That
probe was gated only by a one-shot `kubectl wait
deployment/linstor-controller --for=condition=available`, which goes
stale. The controller carries `reloader.stakater.com/auto=true` and
mounts the cert-manager-issued API-TLS Secret
(`packages/system/linstor/templates/cluster.yaml`), so when cert-manager
(re)writes that Secret during bring-up, reloader rolls the
single-replica controller. The roll drops the Service to zero ready
endpoints *after* `Available` already passed, and a dial into that
window fails with `[Errno 113] No route to host` (EHOSTUNREACH — the
state of a ClusterIP whose Service has no ready backend). The blind
`until` loop then churned the CLI on that error, tolerating a real
un-Ready-controller race instead of gating on it.

This gates every probe on the `linstor-controller` Service actually
having a ready endpoint (a non-empty `Endpoints .subsets[].addresses`),
so the CLI only dials a routable controller and a mid-loop reload
re-waits for the endpoint instead of erroring. The `Online == 3`
satellite-convergence check stays the real readiness assertion, and the
dedicated 300s budget is preserved from the previous `timeout 300`; on
timeout the loop now dumps the controller endpoints and pods for
diagnostics before failing, per the E2E diagnostics convention.

Evidence for the mechanism: the same single `Error: Unable to connect to
linstor+ssl://linstor-controller:3371: [Errno 113] No route to host`
line appears exactly once per install across multiple runs and is
immediately recovered by the retry — consistent with one reloader-driven
controller roll during initial bring-up, not a persistent outage.

Scoped to the LINSTOR readiness path in `hack/e2e-post-install-prep.sh`
only; no change to install ordering, dependencies, or timeouts.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved the LINSTOR startup readiness check to wait more reliably for
the service to become reachable before verifying node status.
* Made the node-online verification more resilient by handling transient
connectivity errors and enforcing a clearer timeout.
* Added extra diagnostic output when readiness does not complete in
time, helping with troubleshooting failed installs.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-15 11:32:05 +03:00
myasnikovdaniil 6c9edf5038 fix(platform): route OIDC backend calls through the in-cluster Keycloak by default (#3289) 
## What this PR does

Defaults `authentication.oidc.keycloakInternalUrl` to
`http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy` instead of an
empty string.

With the empty default, oauth2-proxy performed OIDC discovery against
the external Keycloak hostname. That request resolves through public DNS
and lands on the root ingress controller, which does not serve yet on a
fresh install: the root tenant's IngressClass only exists once
`tenant-root` is ready, and `tenant-root` is still reconciling while the
dashboard starts. Discovery has nothing to talk to, the gatekeeper
fails, and the dashboard never comes up — with no obvious diagnosis for
a first-time installer.

The bundled Keycloak is the only supported provider (the external
hostname is templated as `keycloak.<host>` and the realm is fixed), so
backend-to-backend calls have no reason to leave the cluster. The new
default is the same Service address `keycloak-configure` already uses
for its own backend calls. The dashboard and linstor-gui gatekeepers
then run with `--skip-oidc-discovery` and send token, jwks, userinfo and
logout requests straight to the Service, while browser-facing redirects
stay on the external host.

Both gatekeeper templates already branch on the value being non-empty,
so neither needed a change. The value stays overridable for a Keycloak
that does not live at the standard in-cluster address, and setting it
back to `""` restores the previous external-discovery behaviour.

Token validation is unaffected: Keycloak pins `KC_HOSTNAME` to the
external URL, so tokens redeemed over the internal Service still carry
the external `iss` and match the unchanged `--oidc-issuer-url`.

### Upgrade behaviour

Existing OIDC clusters that never set `keycloakInternalUrl` pick up the
new default on upgrade: their gatekeepers roll once and move to the
internal path. Clusters that set the value explicitly — including to
`""` — keep exactly what they configured.

### Security posture

The backend leg (token redemption, which carries the gatekeeper's
Keycloak client secret, plus jwks/userinfo/logout) now travels as
plaintext HTTP between pods rather than HTTPS to the ingress. This is
stated in the `values.yaml` comment rather than left implicit. The
`keycloak-http` Service has no TLS listener, so in-cluster HTTPS is not
an available alternative, and `keycloak-configure` already sends
Keycloak *admin* credentials over that identical address — anyone
positioned to read this traffic can already reach a strictly more
powerful secret. The externally reachable front door (browser to
oauth2-proxy) is unchanged.

### Testing

helm-unittest coverage across all three affected charts: the default
landing on the `_cluster` channel, both gatekeepers rendering the
in-cluster backend URLs while keeping browser URLs external, an explicit
override winning, the empty-string escape hatch still rendering the
external-discovery shape, and OIDC-disabled clusters (now the common
case of "OIDC off, internal URL set") ignoring the value entirely.

The E2E OIDC case previously applied this exact URL as a manual
workaround. It now sets only `oidc.enabled` and asserts the consequence:
it waits for the `auth-proxy` container to appear in the gatekeeper
Deployment, then gates on `kubectl rollout status`. Readiness alone
would prove nothing there — the HelmRelease is still `Ready` from the
token-proxy install, and with `replicas: 1` and `maxUnavailable: 25%`
(which rounds down to 0) the Deployment keeps the old ReplicaSet
`Available` while a new pod crashloops. Only the rollout catches a
regression back to external discovery.

Follow-up: the OIDC documentation in the website repo still frames
`keycloakInternalUrl` as an optional tweak "for self-signed certificates
or restricted external access". It is now the default path and needs a
separate docs PR.

Closes #3288

### Screenshots

No UI changes.

### Release note

```release-note
fix(platform): `authentication.oidc.keycloakInternalUrl` now defaults to the in-cluster Keycloak Service (`http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy`). OIDC backend calls from the dashboard and linstor-gui gatekeepers no longer leave the cluster, so enabling OIDC on a fresh install no longer leaves the dashboard down while the root ingress is not yet serving. Existing clusters that never set the value pick up the new default and roll their gatekeepers once. Set the value explicitly to point at a non-standard Keycloak placement, or to an empty string to restore external OIDC discovery.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- OIDC-enabled applications now use the internal Keycloak service for
backend authentication by default, improving reliability and avoiding
unnecessary external discovery.
- Browser redirects and login flows continue using the external Keycloak
address.
- Operators can provide a custom internal Keycloak URL or set it to an
empty value to restore external OIDC discovery.

- **Bug Fixes**
- Dashboard and Linstor GUI authentication now correctly apply internal
Keycloak settings only when OIDC is enabled.
- Dashboard upgrades now wait for authentication components to become
ready before completing.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-15 13:08:56 +05:00
Aleksei Sviridkin 44507d2cd0 docs(agents): add downstream repository checklist to the PR template 
Cozystack is upstream for repositories that nothing keeps in sync with
it, and no CI job compares the two sides, so a change here breaks them
in silence. Only two narrow paths are automated, and both fire on a
release rather than on a PR.

It has already happened. The website docs generator works from an app
list hardcoded in its own Makefile, and packages/apps/opensearch was
never added to it, so an application we ship has no reference page at
all. The Terraform provider still offers a Kubernetes version this repo
removed. The plugin repo still sends operators to hack/ scripts that
were deleted, one of them rewritten as a Go command.

Add a Downstream Repositories trigger map to the contributing guide,
naming per repository which change here forces a change over there and
which file to touch, and a checklist in the PR template that links to
it. The template addresses AI agents directly: they author a large share
of PRs and will otherwise tick a box without opening the follow-up it
claims.

The coupling is wider than the obvious ones, and mostly invisible. talm
restates the node contract, overlapping ours but already diverged, so it
must be diffed rather than overwritten. cozyhr reads an annotation the
operator writes and silently merges the wrong values if it is renamed.
cozy-proxy is vendored in from its own repo, yet consumes labels
produced here and quietly stops handling external VM services if they
move. The telemetry server queries metric names emitted here, and a
rename just empties its dashboards. None of these fail loudly.

Guard the map with a unit test, because a map about silent drift must
not drift silently itself. It pins every in-repo path the map cites to a
file that exists, pins the map and the checklist to the same repository
list, and pins the workflow carve-out below to the map's location, to
the job that reads the flag, and to the output that carries it: cutting
any one of those three leaves the other two looking wired up. The header
states plainly what the suite cannot check, so a green tick is not
mistaken for a correct map: whether a coupling is real, and whether it
still holds over there, stays a human's job.

Editing the map alone is a docs-only PR, which skips the unit tests and
would exempt the map from its own guard. Rather than widen the docs-only
rule, which also gates the Talos image build, the plan step exports a
separate flag that only the unit-test job reads: a docs PR now runs the
tests and still builds nothing. It detects the map by its old path too,
since a rename is reported as the new path alone.

Document that gh pr create --body and --body-file replace the body
wholesale, which would otherwise drop the checklist from every
CLI-authored PR.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-14 17:07:47 +03:00
Aleksei Sviridkin 6f5889816b feat(ci): guard migrations.targetVersion against silent skip (#2771) 
## Summary

- Add `hack/check-migrations-target.sh` that fails when the latest
numbered file under
`packages/core/platform/images/migrations/migrations/` is not matched by
a bumped `migrations.targetVersion` in
`packages/core/platform/values.yaml`.
- Wire it into `unit-tests` via a new `migrations-target-check` Makefile
target so PR CI catches the regression mechanically.

`run-migrations.sh:34` iterates `seq $CURRENT_VERSION $((TARGET_VERSION
- 1))`, so a migration whose number equals the configured
`targetVersion` is silently skipped. The cost is state corruption on
upgrade — a class of bug worth eliminating with a 10-line check rather
than relying on reviewer vigilance.

## Test plan

- [x] `make migrations-target-check` passes on `main` HEAD
(`targetVersion=43`, latest migration `42`, `43 >= 43`).
- [x] Synthetic failure: `touch
packages/core/platform/images/migrations/migrations/99` makes the script
exit 1 with a clear error pointing at
`packages/core/platform/values.yaml` and the required bump.
- [x] Tested on macOS (BSD `ls` + mikefarah `yq`).

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Added a CI validation step to unit tests to ensure
`migrations.targetVersion` is kept in sync with newly added database
migrations, preventing upgrade scripts from skipping new migration
steps.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-14 14:55:01 +03:00
Aleksei Sviridkin a4bbc74b61 feat(agents): add md-no-hardwrap hook enforcing the prose rule 
An ~80-column wrap is correct in Go comments and commit bodies, where
coding agents spend most of their time, and the reflex leaks from there
into markdown files and GitHub bodies. A documented rule alone does not
survive that reflex, so check it mechanically at the moment of writing.

The hook refuses a Write/Edit/MultiEdit that would add a hardwrapped
paragraph, list item or blockquote to a markdown file, and a gh command
that would publish one to GitHub via --body, --body-file, a heredoc, or
an api body field.

Judging what an edit ADDS is the whole difficulty. Comparing the text of
each wrapped block refuses a typo fix inside a paragraph that was
already wrapped; counting the blocks lets a file be rewritten as
entirely fresh hardwrapped prose so long as the total does not rise. So
the hook asks about the break rather than the block: a soft break is the
edit's doing only when both lines it joins are lines the edit wrote.

Whatever it cannot read with certainty, it allows -- a body the shell
assembles at runtime, an unsplittable command line, a path that is not a
regular file, a --body-file the same command has yet to write. A guard
that blocks legitimate work gets switched off, and then it guards
nothing. Vendored charts, published changelogs and Go templates are out
of scope entirely, so hack/upload-releasenotes.sh keeps publishing its
historical release notes.

Line breaks are only rejected inside prose. Code blocks, tables, HTML,
front matter, link reference definitions, footnotes, badge stacks and
explicit hard breaks -- two trailing spaces or a trailing backslash,
which render as <br> -- keep their breaks, because reflowing those would
change the rendered output. A heredoc is attributed to the command that
opened it, so a YAML manifest piped to kubectl is not read as a PR body.

The hook and its registration are tracked by name so contributors get
them on clone, while personal agent state stays ignored; personal
overrides belong in .claude/settings.local.json. hack/md-no-hardwrap.bats
covers both surfaces with 60 cases and make unit-tests picks it up
automatically.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-14 14:37:52 +03:00
Aleksei Sviridkin f24b56327b fix(platform): route OIDC backend calls through the in-cluster Keycloak by default 
authentication.oidc.keycloakInternalUrl defaulted to an empty string, which
left oauth2-proxy performing OIDC discovery against the external Keycloak
hostname. That request resolves through public DNS and lands on the root
ingress controller, which does not serve yet on a fresh install: the root
tenant's IngressClass only exists once tenant-root is ready, and tenant-root
is still reconciling while the dashboard starts. Discovery has nothing to talk
to, the gatekeeper fails, and the dashboard never comes up.

The bundled Keycloak is the only supported provider — the external hostname is
templated as keycloak.<host> and the realm is fixed — so backend-to-backend
calls have no reason to leave the cluster. Default the value to
http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy, the same Service
address keycloak-configure already uses. The dashboard and linstor-gui
gatekeepers then run with --skip-oidc-discovery and send token, jwks, userinfo
and logout requests to the Service, while browser redirects stay on the
external host. Those calls travel as plaintext HTTP between pods, as
keycloak-configure's admin calls to the same address already do.

The value remains overridable for a Keycloak that does not live at the standard
in-cluster address, and setting it back to an empty string restores external
discovery. Existing OIDC clusters that never set it get a gatekeeper rollout
onto the internal path on upgrade.

Enabling OIDC swaps the dashboard's token-proxy container for oauth2-proxy, so
the E2E install now drives that swap to completion: it waits for the auth-proxy
container to appear, then gates on the gatekeeper rollout. Readiness alone
proves nothing there — the HelmRelease is still Ready from the token-proxy
install, and the Deployment keeps the old ReplicaSet Available while a new pod
crashloops, so only the rollout catches a regression back to external discovery.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-14 14:22:46 +03:00
IvanHunters 3380bb4ab6 refactor(hack): replace ls|grep with pure-bash glob (shellcheck SC2010) 
Filenames in packages/core/platform/images/migrations/migrations/ are
numeric-only by convention so the original `ls -1 | grep -E` filter was
safe in practice, but the cleaner pure-bash form is equivalent and
silences SC2010 without an annotation.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-14 13:08:54 +03:00
IvanHunters c3d3e2596e feat(ci): guard migrations.targetVersion against silent skip 
Adds hack/check-migrations-target.sh and wires it into unit-tests via the
migrations-target-check Makefile target. The check fails when the highest
numbered file under packages/core/platform/images/migrations/migrations/
is not matched by a bumped migrations.targetVersion in
packages/core/platform/values.yaml.

run-migrations.sh:34 loops `seq $CURRENT_VERSION $((TARGET_VERSION - 1))`,
so a migration whose number equals targetVersion never runs and the
intended state change is silently skipped on cluster upgrade.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-14 13:08:54 +03:00
Myasnikov Daniil 103b8e16ee test(e2e): diff-driven install selector select-install.sh (#2847) 
Add hack/select-install.sh: given the E2E suites select-e2e.sh picked,
emit the forward dependsOn closure over the PackageSource graph — the
minimal set of packages to install for those suites. Companion to the
reverse (test-selection) walk in select-e2e.sh; keeps the cozystack-engine
edge since install needs it.

Suite->source mapping follows the <suite>-application convention with a
bare-<suite> fallback plus explicit cases for irregular names
(kubernetes/vminstance/securitygroup) and core-platform features that have
no separately-enabled package (serviceexposure). Includes a --validate mode
(reachability + cycle check) and a 13-case unit suite auto-discovered by
make unit-tests.

Track A of #2847; tracked in #3278.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-13 17:47:26 +05:00
Myasnikov DaniilandClaude Opus 4.8 8af4ed543e test(e2e): address review nits — shellcheck directive + ports-workaround TODO 
Non-blocking review notes from #2826:

- Add `# shellcheck shell=bash` to the four sourced `_lib/*.sh` helpers
  (etcd-cleanup, remediation-guard, run-kubernetes, talos-image-cache) so
  shellcheck stops flagging SC2148 (no shell directive) — they are only ever
  `. sourced`, so there is no runtime interpreter to infer from a shebang.
- Add a `# TODO(chainsaw>0.2.15)` marker at the postgres Service asserts
  explaining the `(ports[0].port)`/`(ports[*].port)` projection is a v0.2.15
  number-literal-filter workaround (and that `ports[0]` checks only the first
  port), so it gets reverted on a chainsaw bump; cross-references the same
  workaround in kafka/mariadb/harbor and the e2e-testing.md gotchas section.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-10 17:02:46 +05:00
Myasnikov DaniilandClaude Opus 4.8 ff0d0d77a7 test(e2e): make the talos image cache reachable by tenant workers (#3254) 
The in-sandbox Talos image mirror (#3244) is unreachable by the tenant
workers it serves, so the kubernetes-* e2e tests fail deterministically at
the 12m node-join wait instead of relieving the public-factory flake (#3231).

Root cause (from #3254 by @lexfrei, verified against the failing run's
crust-gather snapshot): tenant namespaces run a default-deny Cilium egress
(allow-internal/external-communication + the <tenant>-egress CCNP) that
permits the world entity and the tenant tree but NOT an arbitrary kube-system
Service, so a worker's CDI importer resolves talos-image-cache.kube-system.svc
yet its TCP connect to the ClusterIP is silently dropped and no disk imports.
#3244's gate never caught this because it probed the server over localhost
inside the serve pod, which passes regardless of cross-pod reachability.

Port of #3254 to the Chainsaw layout (hack/e2e-chainsaw/_lib/):
- ship a tightly-scoped CiliumClusterwideNetworkPolicy (importer pods in
  tenant-test -> mirror pod in kube-system), applied at point-of-use once
  Cilium CRDs exist; excluded from the pre-Cilium install apply via yq.
- replace the readiness probe with a throwaway pod in tenant-test labelled
  cdi.kubevirt.io=importer that curls the ClusterIP with a range request and
  must get 206 - the exact egress path a real importer faces; tenants use the
  mirror only when that end-to-end check passes, else fall back to the public
  factory, so it can only help, never make CI worse.
- unit test pinning the manifest split, probe/policy label agreement, the
  strict 206 gate, and the --overrides JSON builder.

Also adds (a2) DataVolume import-stage diagnostics to the node-join failure
block in run-kubernetes.sh (importer logs + a ClusterIP re-probe via
talos_image_cache_diagnose) so the import-stage failure mode is legible.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-09 18:21:07 +05:00
Myasnikov DaniilandClaude Fable 5 1f84d2d49e Merge origin/main into feat/e2e-chainsaw-example (post-#3244) 
Fold the in-sandbox Talos image cache work into the Chainsaw layout:
the new shared helper relocates to hack/e2e-chainsaw/_lib/
talos-image-cache.sh and run-kubernetes.sh sources it from there (the
ouroboros DNS-check no-retry rework merges clean). The OIDC bats are
resolved as deleted again; their Chainsaw fixtures deliberately skip
the talos.imageFactoryURL injection — the helper only emits it when
the cache is up, and the zero-worker render-side tests instantiate no
DataVolume imports for it to redirect.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-09 10:38:35 +05:00
Aleksei Sviridkin 336f873b07 fix(e2e): serve the Talos image cache with a range-capable HTTP server 
The talos-image-cache serve container ran `busybox httpd`, but the pinned alpine/k8s image ships no `httpd` applet and no standalone httpd, so the container exited 127 and crash-looped and the Deployment never became Available. The resolver then burned its full 12-minute rollout wait before falling back to the public factory — a fixed latency tax on the first kubernetes-* test while the flake the mirror targets stayed live.

Serve the seeded image with a small range-capable server on the image's bundled python3 instead (stock `python3 -m http.server` ignores Range headers and answers 200, not 206). It needs no extra package pull, so the mirror comes up deterministically and keeps CDI's fast range import. The seed curl now also aborts and retries a stalled transfer (--speed-time/--speed-limit), and the resolver verifies a 206 range response before pointing tenants at the mirror.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-09 00:52:58 +03:00
Aleksei Sviridkin f60fe2270b Merge remote-tracking branch 'origin/main' into feat/kubernetes-talos-image-factory-override 
Signed-off-by: Aleksei Sviridkin <f@lex.la>

# Conflicts:
#	packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
 2026-07-09 00:46:33 +03:00
Aleksei Sviridkin 41eb6f912b fix(e2e): gate LINSTOR node-list probe on a ready controller endpoint 
The post-install-prep node-list loop dialed the linstor-controller Service
(linstor+ssl://linstor-controller:3371) as soon as the controller Deployment
reported Available once. That gate goes stale: the controller carries
reloader.stakater.com/auto=true and mounts the cert-manager-issued API-TLS
Secret, so when cert-manager (re)writes that Secret during bring-up, reloader
rolls the single-replica controller. The roll drops the Service to zero ready
endpoints after Available already passed, and a dial into that window fails
with "[Errno 113] No route to host" (EHOSTUNREACH). The blind until-loop then
churned the CLI on that error instead of waiting for a routable controller.

Gate every probe on the linstor-controller Service having a ready endpoint, so
the CLI only dials a reachable controller and a mid-loop reload re-waits for
the endpoint instead of erroring. The "Online == 3" satellite-convergence
assertion and the dedicated 300s budget are preserved; a timeout now dumps
controller endpoints and pods for diagnostics before failing.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-09 00:44:34 +03:00
scooby87 6c69c0055e fix(monitoring)!: clean up orphaned storage PVCs and TLS secrets on tenant module delete (#3091) (#3094) 
## What this PR does

When the `monitoring` tenant module is disabled, the
VictoriaMetrics/VictoriaLogs storage PVCs (created by vm-operator) and
the `alerta-tls` / `grafana-ingress-tls` cert-manager secrets were left
orphaned in the tenant namespace.

This adds a `post-delete` cleanup hook
(`packages/extra/monitoring/templates/hooks/cleanup.yaml`) that:
- deletes the VM/VL storage PVCs scoped to this release via `-l
apps.cozystack.io/application.name={{ .Release.Name }}-system`. That
label is stamped on the `storage.volumeClaimTemplate.metadata.labels` of
the VMCluster/VLCluster CRs (`packages/system/monitoring`); vm-operator
copies claim-template labels onto the PVCs (`IntoSTSVolume`), so the
selector matches and stays scoped to this monitoring release.
`managed-by=vm-operator` is intentionally **not** used — vm-operator
does not reliably stamp it onto the storage PVCs;
- deletes the `alerta-tls` and `grafana-ingress-tls` secrets by name
(cert-manager TLS secrets carry only the cluster-wide
`controller.cert-manager.io/fao=true` label, so a label sweep would be
unsafe — they are removed by name).

CNPG PostgreSQL PVCs (`alerta-db-*`, `grafana-db-*`) are intentionally
**not** targeted — CNPG cleans those up itself, and they do not carry
`apps.cozystack.io/application.name`, so the selector leaves them
untouched.

**Upgrade path:** the claim-template label only lands on PVCs created
after the change, so **migration 50**
(`packages/core/platform/images/migrations/migrations/50`) backfills
`apps.cozystack.io/application.name` onto the pre-existing VM/VL storage
PVCs of every cozystack-managed monitoring cluster, so the cleanup
reclaims storage on already-deployed clusters too.

A `test` Makefile target and helm-unittest suites are added for both the
cleanup hook and the claim-template label. The cleanup Job is hardened:
non-root, read-only root fs, dropped capabilities, `seccompProfile:
RuntimeDefault`, resource requests/limits, `activeDeadlineSeconds: 120`,
`backoffLimit: 0`, `--wait=false` on both deletes; RBAC scoped to PVC
get/list/delete and the two named secrets. Failures are logged but never
block teardown.

> [!WARNING]
> **Disabling the monitoring module now permanently deletes its
VictoriaMetrics/VictoriaLogs storage PVCs and the metrics/logs they
hold.** This is intentional and fixes the storage leak. Back up before
disabling if the data matters.

Fixes #3091.

### Release note

```release-note
fix(monitoring): garbage-collect the orphaned VictoriaMetrics/VictoriaLogs storage PVCs and the alerta/grafana TLS secrets left behind when a tenant's monitoring module is disabled; a migration backfills the release label onto pre-existing storage PVCs so already-deployed clusters are covered too. WARNING: disabling monitoring now permanently removes its storage PVCs and the data they hold.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a Helm **post-delete** monitoring cleanup job to best-effort
remove managed PVCs and two specific TLS secrets.
* Included dedicated cleanup RBAC and hardened pod security settings for
the cleanup job.
* **Tests**
* Added Helm chart tests validating hook weight/policies, job settings,
exact deletion commands, and negative cases.
  * Added a `make test` target to run Helm unit tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-09 00:22:23 +03:00
Aleksei Sviridkin a28ef9ba13 test(e2e): make the hairpin DNS check tolerate a tenant node recycle (#3241) 
## What this PR does

Makes the ouroboros hairpin DNS assertion in the `kubernetes-latest` e2e
deterministic against a tenant worker-node recycle. This is a widespread
pre-existing flake: it fails PRs that touch nothing virt-related and
passes on re-run.

### Root cause (mechanism, from a failed run's own event stream)

The assertion resolves `hairpin-*.example.invalid` from inside the
tenant cluster using a single `--restart=Never` netshoot Pod, then waits
for that Pod to reach `Succeeded`. The tenant crust-gather snapshot of a
failed run shows the pod never ran the DNS loop at all:

- `09:00:02` scheduler: `Successfully assigned default/dnscheck to
...-md0-vqcs7-hhfxx`
- `09:00:06` kubelet on `hhfxx`: `Pulling image
"nicolaka/netshoot:v0.13"` (no subsequent `Pulled`/`Started`)
- node `hhfxx` then disappears from the tenant node list; on the
management cluster its Machine is gone and a replacement Machine is
created at `09:02:01` (`Provisioning`), while the other worker (created
`08:43:39`) survives
- `09:02:47` `taint-eviction-controller`: `Cancelling deletion of Pod
default/dnscheck`

The tenant's CAPI `MachineHealthCheck` remediates a node whose `Ready`
goes `Unknown` for just 30s by deleting its Machine, KubeVirt VM and
Node and provisioning a replacement. On the memory-constrained
single-node CI sandbox a worker VM losing its kubelet heartbeat briefly
is routine. A bare Pod bound to the deleted node is removed by the node
controller and, with `restartPolicy: Never`, is never recreated — so the
probe silently vanishes while its (large) image is still pulling and it
has never left `Pending`. The single-shot wait then times out and
reports `dnscheck pod did not reach Succeeded phase (last seen:
<empty>)`, which is indistinguishable from a real DNS-resolution
failure.

### The fix

Wrap the probe create-and-wait in a bounded outer deadline (7m) that
recreates the Pod when it is confirmed `NotFound` — the node-recycle
signature — so the check is retried on a surviving node instead of
failing the whole test. An empty phase is treated as ambiguous (a
transient API blip is not misread as a lost Pod); only a definitive
`NotFound` triggers a recreate. A Pod that actually runs to a
non-`Succeeded` terminal state, or the deadline elapsing with no
successful run, still fails the test. The in-Pod DNS resolution loop
that the assertion exists to exercise is unchanged, so the
reconciliation coverage is preserved.

This fixes the mechanism at the right layer: the check tolerates a
transient node recycle of a throwaway diagnostic pod rather than
reporting it as a DNS regression. It is deliberately not a timeout bump
— the pod was actively deleted, not merely slow, so more time alone
would not help.

### Out of scope

The `kubernetes-previous` failure seen in the same run is a different
mechanism: both tenant worker VMs stayed `Provisioning` for 12m
(DataVolume/CDI storage-import stall on the management cluster), so no
tenant node ever joined and the run fails at the existing node-join
diagnostic block, well before this assertion. Both smell like the same
underlying single-node-sandbox resource/storage pressure, but that is an
infrastructure-capacity concern, not a harness race this change can fix.
Left for a separate follow-up.

### Screenshots

N/A — CI/test-harness change, no UI impact.

### Release note

```release-note
test(e2e): recreate the tenant hairpin DNS probe pod on a bounded deadline so a worker-node recycle mid-check is retried instead of reported as a spurious DNS failure
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Improved reliability of the Kubernetes end-to-end DNS check.
* The DNS validation now retries more safely during
infrastructure-related pod restarts or node churn.
* Failures now surface faster with clearer diagnostic output when the
check does not pass.


<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-09 00:06:15 +03:00
Алексей Артамонов ff037c04d3 test(monitoring): reproduce the real vmselect cache PVC name in the backfill test 
/branch-review flagged that the migration comment and its bats test used the
simplified vmselect cache PVC name cache-vmselect-<cr>-<ordinal>, while the real
name is vmselect-cachedir-vmselect-<cr>-<ordinal>. The anchored regex matches
both (it suffix-matches <component>-<cr>-<ordinal>), so behaviour was correct,
but the test did not exercise the production name. Feed the realistic name in
the fake kubectl so the regex is validated against what vm-operator actually
provisions, and reword the migration comment to describe the suffix match
generically instead of asserting a specific claim-template prefix.

Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-08 22:27:42 +03:00
Алексей Артамонов 09c422562a test(monitoring): unit-test the VM/VL PVC backfill migration behaviour 
helm-unittest only inspects rendered manifests, not the operator's runtime PVC
labels, so it cannot prove migration 50 actually relabels the right PVCs — the
exact no-op/mis-scope class of bug this PR iterated on. Add a bats test (fake
kubectl on PATH, mirroring hack/kubernetes-md0-migration.bats) that pins the
runtime behaviour: it relabels the monitoring cluster's vmstorage/vmselect/
vlstorage PVCs with application.name=monitoring-system, never touches the CNPG
db PVCs or a user's own VMCluster, and stays a safe version-stamping no-op when
the VM/VL CRDs are absent.

Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Assisted-By: Claude <noreply@anthropic.com>
 2026-07-08 22:27:42 +03:00
Myasnikov DaniilandClaude Fable 5 dea35f7432 Merge origin/main into feat/e2e-chainsaw-example (post-#3233) 
Main's #3233 landed mid-reconciliation and touched the two OIDC bats
again: nodeGroups (and storageClass) dropped from the render-side test
CRs — the schema default {} now renders one zero-replica
MachineDeployment via CAPI defaulting, so the tests provision zero
worker DataVolumes (#3231). Resolve the modify/delete as deleted and
mirror the fixture change (and its rationale) into the ported
kubernetes-oidc-system / kubernetes-oidc-customconfig Chainsaw suites.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 21:33:08 +05:00
IvanHunters 0c7b598304 fix(apps/kubernetes): honour nodeGroup minReplicas in MachineDeployment.spec.replicas (#3233) 
Refs #3231 (parts 1 + 2 addressed here; part 3 — intra-file
`cozy_wait_linstor_pool_free` gate + orphan
`default-chk-clickhouse-test-keeper-cluster1-0-2-0` PVC cleanup —
deferred to a separate follow-up PR against
`hack/e2e-apps/run-kubernetes.sh`).

**Do not auto-close the issue on merge** — the hardening work is a
distinct concern and belongs on its own tracker entry.

## Root cause chain (recap from the issue)

`packages/apps/kubernetes/templates/cluster.yaml` hardcoded
`MachineDeployment.spec.replicas: 2` regardless of the nodeGroup's
`minReplicas`. The two render-side
`kubernetes-oidc-{system,customconfig}` bats set `minReplicas: 0` and
delete their `Kubernetes` CR without waiting for readiness — but the
chart still spun up 2 × 20 GiB `replicated` (DRBD) worker `DataVolume`s
each. Their create/delete churn overlapped `kubernetes-previous`, which
then saw slow LINSTOR provisioning (my cozyreport from run 28843794670
shows a ~6-minute gap between the `Kubernetes` CR being created and the
CDI `ExternalProvisioning` event actually firing for the worker root
disk); the tenant node registered late; Cilium's CNI init missed the
12-minute node-join deadline; and the test failed with `cni plugin not
initialized`.

## Changes

Two edits, applied together — the chart edit is a no-op without the test
edit, and vice-versa (dropping `nodeGroups` falls back to the chart
helper's default `md0`, which itself has `minReplicas: 0`, so both
pieces are needed for zero workers to actually render).

### 1. `packages/apps/kubernetes/templates/cluster.yaml`

```diff
 spec:
   clusterName: {{ $.Release.Name }}
-  replicas: 2
```

The field is **omitted entirely** so the cluster-autoscaler owns it
alone. CAPI's MachineDeployment defaulting webhook seeds `spec.replicas`
from the `cluster-api-autoscaler-node-group-min-size` annotation on
Create (documented at
https://cluster-api.sigs.k8s.io/tasks/automated-machine-management/autoscaling
— *"if it's a new MachineDeployment or MachineSet, use min size"*), and
Server-Side-Apply field-manager ownership then keeps chart upgrades from
overwriting whatever value the autoscaler patched in the meantime.

Emitting the field from the chart at all — even with the
semantically-correct `{{ $group.minReplicas }}` — would keep the
helm-controller ↔ cluster-autoscaler field-ownership fight alive:
helm-controller applies with SSA + force-conflicts
(https://fluxcd.io/flux/components/helm/helmreleases/), so every routine
`helm upgrade` re-asserts the chart's chosen value and drains whatever
the autoscaler had patched in the meantime.

### 2. `hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats`

Drop the explicit `nodeGroups:` block and the paired `storageClass:`
from the render-side CR bodies. The tests only assert on the
KamajiControlPlane and the cozy-realm Keycloak objects — worker plumbing
was never part of their coverage.

After the chart fix, the chart helper's default `md0` (with
`minReplicas: 0`) renders a MachineDeployment with `spec.replicas: 0` —
the resource is still emitted for API-surface completeness but never
creates a KubevirtMachine, never provisions a DataVolume, and never
touches DRBD.

## Verification

### helm-unittest

```
Charts:      1 passed, 1 total
Test Suites: 16 passed, 16 total
Tests:       171 passed, 171 total
```

No suite asserted on `MachineDeployment.spec.replicas` directly, so
nothing needed updating.

### helm template — verify the omit

With `tests/values/common.yaml` (which has `minReplicas: 0`), the
rendered MD carries the `cluster-api-autoscaler-node-group-min-size:
"0"` annotation and **no `spec.replicas` field**. CAPI's defaulting
webhook will fill in `spec.replicas: 0` on Create — no worker
DataVolumes provisioned until scale-out.

With a user CR that sets `minReplicas: 2` (the shape the
`kubernetes-latest` / `kubernetes-previous` bats CRs use), the rendered
MD carries the `cluster-api-autoscaler-node-group-min-size: "2"`
annotation and again **no `spec.replicas` field**. CAPI's defaulting
webhook will fill in `spec.replicas: 2` on Create — byte-identical
topology to the pre-PR chart's fresh-install state (2 workers), but
without the recurring drain-on-upgrade cycle.

### Render-side render walk

`helm template` of the OIDC bats CR bodies (after this PR): 1
MachineDeployment with **no chart-managed `spec.replicas`** and
`min-size: "0"` on the autoscaler annotation. CAPI seeds `replicas: 0`
on Create, 0 KubevirtMachines emitted, 0 DataVolume provisioning
triggered. Test assertions on KamajiControlPlane / KeycloakClient /
KeycloakClientScope / AuthenticationConfiguration Secret are unchanged.

## Upgrade impact

Existing customers stop losing workers on `helm upgrade`. Prior to this
PR the chart re-asserted `MachineDeployment.spec.replicas: 2` on every
Helm action, so a customer running with the autoscaler at, say,
`replicas: 8` would see 6 workers drained on every routine Cozystack
platform bump and then wait for the autoscaler to scale back up in
response to unschedulable Pods. With `spec.replicas` no longer emitted
by the chart, SSA field-manager ownership preserves the autoscaler's
value across upgrades unconditionally.

Fresh installs with `nodeGroups: {}` (the schema default) transition
from "2 workers by default" to "0 workers until scale-out". The chart
helper's default md0 carries `minReplicas: 0`, which is what CAPI's
defaulting webhook seeds `spec.replicas` from. The `values.yaml` /
README / helper docstring / regenerated Go type doc + openAPISchema call
this out explicitly; the release-note repeats it below. See finding 5 in
the paired self-review for the specific `addons.ingressNginx.enabled:
true` interaction operators need to be aware of on fresh installs.

## Test plan

- [x] `helm unittest packages/apps/kubernetes/` — 171/171 passing.
- [x] `helm template` with `tests/values/common.yaml` (minReplicas: 0) —
MD.replicas: 0.
- [x] `helm template` with a `minReplicas: 2` override — MD.replicas: 2
(byte-identical to pre-PR).
- [ ] E2E: `kubernetes-latest` unchanged (spawns its two workers as
before), `kubernetes-oidc-{system,customconfig}` render zero workers,
`kubernetes-previous` no longer flakes on LINSTOR pool pressure
originating from the earlier OIDC suite.

## Notes

- Part 3 of #3231 (intra-file `cozy_wait_linstor_pool_free` gate between
kube sub-tests and the orphan clickhouse keeper PVC cleanup) is **not**
in this PR — worth a separate targeted change to
`hack/e2e-apps/run-kubernetes.sh`, and it's a hardening-only improvement
rather than a root-cause fix. Happy to send it as a follow-up if wanted.

## Release note

```release-note
fix(apps/kubernetes): stop the chart from managing `MachineDeployment.spec.replicas`; the cluster-autoscaler owns the field alone. CAPI's defaulting webhook now seeds `spec.replicas` from the `cluster-api-autoscaler-node-group-min-size` annotation on Create, and SSA field-manager ownership keeps subsequent `helm upgrade` actions from overwriting whatever value the autoscaler patched. Two behavioural changes worth calling out:

- Existing customers: `helm upgrade` no longer drains workers on every platform bump — the autoscaler-held `spec.replicas` is preserved unconditionally. Prior to this change the chart hardcoded `replicas: 2`, which drained back to 2 on every reconcile action.
- Fresh installs with `nodeGroups: {}` (the schema default) no longer come up with two workers by default. The chart helper's default md0 carries `minReplicas: 0`, and CAPI's defaulting webhook seeds `spec.replicas` from that. Operators enabling `addons.ingressNginx.enabled: true` on such a CR must either supply an explicit nodeGroup with `roles: [ingress-nginx]` and `minReplicas >= 1`, or accept that the cluster-autoscaler will bring the default md0 up in response to the ingress-nginx controller Pods becoming Pending on install.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Rendered `MachineDeployment` manifests no longer set `spec.replicas`,
avoiding chart upgrades overwriting autoscaler-managed sizing.
* **Documentation**
* Clarified `nodeGroups: {}` behavior: a default `md0` is emitted with
`minReplicas: 0`/`ingress-nginx` role, and workers are provisioned only
after an autoscaler-triggering unschedulable Pod or manual scaling;
updated related storage and ingress guidance.
* **Tests**
* Strengthened default node group rendering assertions and adjusted OIDC
e2e manifests to reduce CI provisioning churn.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-08 18:58:32 +03:00
Myasnikov DaniilandClaude Fable 5 1f9555a3c6 test(e2e): resolve outstanding bot review findings 
- redis: assert both StatefulSet-templated PVCs Bound (replicas: 2
  provisions two, only ordinal 0 was checked)
- gateway: || true each delete inside the three cleanup() traps — they
  run under set -eu, so one transient delete failure skipped the
  remaining cleanup commands
- bucket, backup(.disabled): guard the trap's kill against an unset
  pf_pid (script can exit before the port-forward starts)

Not addressed on purpose: the sort(ports[*].port) suggestions on
clickhouse/kafka — sort() is among the JMESPath forms chainsaw v0.2.15
(still the latest release) mis-evaluates, which is why the suites use
the ordered (ports[*].port) projection; services keep spec order.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 19:13:01 +05:00
Myasnikov DaniilandClaude Fable 5 dd2f89d208 test(sdn): port the securitygroup and serviceexposure suites to Chainsaw 
Main added securitygroup.bats (#2922 membership model) and
serviceexposure.bats (#3081 ExposureClass/ServiceExposure) after the
Chainsaw migration branch was cut.

- securitygroup: two Tests — the aggregated-API projection round-trip
  (backing CNP marker label, empty-value membership endpointSelector,
  lineage fromApp + membership fromSG peers, 1:1 rule translation, the
  view hiding storage internals, unmarked-CNP invisibility via an error
  op, delete propagation) and the controller membership-label
  stamp/clear lifecycle. Only the attachments annotation stays a jq
  script (it is a JSON string). Chainsaw auto-cleanup covers every
  fixture even on mid-test failure — the bats had no teardown hook and
  leaked its fixtures.
- serviceexposure: the externalIPs backend resolve/Ready/assignedIPs
  path with the no-MetalLB-pool negative as an error op (ordered after
  resolvedBackend so it cannot false-pass), and the finalizer-removal
  test collapsed onto the delete op itself, which waits for full
  disappearance.
- selection: kubernetes-application now also maps to the two OIDC
  render-side suites (they exercise the same app chart — a gap main's
  bats selection had too); the new dirs enter all_apps discovery
  automatically. Selector unit tests extended and green (13/13 under
  cozytest).
- README: suites list updated.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:24:24 +05:00
Myasnikov DaniilandClaude Fable 5 cac07db8fc test(kubernetes): port the Phase-1 OIDC render-side suites to Chainsaw 
Main added kubernetes-oidc-system.bats and
kubernetes-oidc-customconfig.bats after the Chainsaw migration branch
was cut; without a port the 260 lines of coverage would vanish with
the hack/e2e-apps/ deletion.

Both are light cluster e2e (apply CR -> assert rendered objects; no
Kamaji bringup, minReplicas: 0), exactly Chainsaw's declarative shape:
- System mode: KamajiControlPlane carries --authentication-config +
  the chart-owned volume/mount; the AuthenticationConfiguration Secret
  carries the cozy-realm issuer (regex, script step) + per-cluster
  audience; per-cluster KeycloakClient/KeycloakClientScope land in the
  cozy realm
- CustomConfig: the flag is still wired; the Secret carries the
  tenant-supplied issuer and NOT the cozy realm (fail-loud negation);
  no Keycloak objects and no pre-rendered OIDC kubeconfig Secret,
  expressed as error ops ordered after the positive asserts so an
  unreconciled HR cannot false-pass them

The port also sheds two latent bats bugs: cozytest defines no skip()
(the Keycloak guard would have died with exit 127 had it ever fired)
and the api-resources group probe exits 0 even when the group is
absent — the chart capability-gates Keycloak rendering and CI always
ships the operator, so the asserts are unconditional here.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:20:20 +05:00
Myasnikov DaniilandClaude Fable 5 6c76a462a0 test(e2e): fold remaining post-branch bats drift into the Chainsaw suites 
- qdrant: add the post-delete PVC-reclaim guard (#3059) — delete the CR
  and poll the ordinal-anchored data PVC to absence via the error op
  (retries transient API errors internally), with a PVC/jobs/pods dump
  on failure; Chainsaw auto-cleanup alone would never notice a leaked
  StatefulSet-templated PVC
- kuberture: make the split-horizon negative log assertions fail loud —
  under set -e a pipeline beginning with ! suppresses errexit, so both
  leak checks silently passed (the same vacuous-negation bug main fixed
  in 12fe3256f)
- foundationdb: retry the one-shot generations.reconciled read for up
  to 2m — the operator stamps it after the post-bounce uptime gate, so
  it can trail the fdbcli-derived health fields the previous assert
  converged on (a79f8009f)
- kubernetes-latest: add the version-independent SC-fallback-default
  Test (#2872 B1) calling verify_storageclass_fallback_default from the
  shared lib; a separate Test doc so it runs regardless of the heavy
  bringup Test's outcome, safe under the suite-wide parallel: 1 while
  it temporarily mutates cluster-scoped StorageClasses

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:17:19 +05:00
Myasnikov DaniilandClaude Fable 5 d6ac62a6d9 test(bucket,harbor): fail fast on frozen BucketClaims; enforce readonly denial 
Fold main's post-branch bucket/harbor bats drift into the Chainsaw
suites:

- BucketClaim bucketReady budgets tightened to 2m (bucket had the 5m
  default, harbor a stale 10m with the reversed 'can lag on a loaded
  runner' rationale): the objectstorage-controller requeues the claim
  until the backend Bucket is ready (0f1cc548b), so readiness converges
  in tens of seconds and a claim not ready after 2m is frozen — the
  tight bound surfaces the propagation regression instead of masking
  it (63d14e748)
- the bucket viewer-upload soft-xfail is promoted back to a hard
  failure: accessPolicy=readonly enforcement landed with
  seaweedfs-cosi-driver v0.3.1 (#1921, eab9b034f), so a successful
  readonly upload is now a regression, exactly what the suite's own
  TODO called for
- both catch blocks gain the backend view (cluster-scoped Buckets +
  cozy-objectstorage-controller pods) where a frozen claim actually
  surfaces

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:15:56 +05:00
Myasnikov DaniilandClaude Fable 5 21e21a9234 test(kafka): raise fixture presets off the 128Mi OOM threshold 
Main raised kafka/zookeeper resourcesPreset nano -> c1.small
(de9c81ce6, #3052) after the Chainsaw port was cut; the fixture still
pinned both to nano.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:14:04 +05:00
Myasnikov DaniilandClaude Fable 5 7b557db59f test(mariadb): port the single-replica webhook-regression guard to Chainsaw 
Main's mariadb.bats gained "Create single-replica MariaDB" after the
Chainsaw port was cut: replication is guarded on replicas>1
(c7e8f4d97), so replicas=1 must render a CR the operator webhook
admits and provision the bare <name> Service without the
-primary/-secondary split. Mirrors the bats assertions (svc :3306 +
endpoints, StatefulSet replicas=1, metrics svc/endpoints/deployment)
and dumps the rendered k8s.mariadb.com CR in catch so a webhook
rejection is legible.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:14:04 +05:00
Myasnikov DaniilandClaude Fable 5 c8e775d2d0 test(gateway): port the 12 ingress-hostname-policy cases to Chainsaw 
Main added cozystack-ingress-hostname-policy (d89ad7069) with 12 bats
cases after the Chainsaw port was cut; this appends them as one Test.

- VAP/binding installation asserted declaratively (binding must carry
  Deny)
- rejects and accepts grouped by derivation needs: apex-derived rejects
  (foreign-apex host, dot-boundary prefix-adjacent, foreign tls host,
  wildcard matching the root apex), static catch-all rejects (hostless
  rule, empty-string host, defaultBackend), own-apex accepts (in-apex
  rule+tls, own-apex wildcard), and static external-domain accepts as a
  declarative apply with auto-cleanup
- hostnames derived at runtime: the tenant apex from the test
  namespace's namespace.cozystack.io/host label (suite convention) and
  the platform ROOT apex from the same label on tenant-root, replacing
  the bats' hardcoded test.example.org/example.org; evil${apex}
  preserves the dot-boundary semantics of eviltest.example.org, and the
  external-domain probes use the RFC 2606 .invalid TLD so they can
  never collide with a real cluster apex

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:12:42 +05:00
Myasnikov DaniilandClaude Opus 4.8 47e7d224d8 test(e2e): serve Talos worker OS image from an in-sandbox cache 
The kubernetes-* e2e tests fail intermittently and run 18-22 min each
because every tenant worker VM imports its Talos OS disk directly from the
public factory.talos.dev: that endpoint has no byte-range support (forcing
CDI into a full copy-to-scratch) and its egress stream-resets / stalls
mid-transfer from the CI runner, so a worker DataVolume import hangs past
the chart's 12-minute node-join deadline and the test fails with zero
tenant nodes joined (observed on run 28924455578, and on the unrelated
branch in run 28929095889 - i.e. not specific to any one PR).

Deploy a best-effort in-sandbox mirror (hack/e2e-talos-image-cache.yaml)
during install: an initContainer fetches the image once with an aggressive
curl retry loop, then busybox httpd serves it locally with range support.
Tenant Kubernetes CRs point spec.talos.imageFactoryURL at its Service via
a shared helper (hack/e2e-apps/talos-image-cache.sh) that gates on the
mirror being Available and falls back to the public factory otherwise, so
the mirror can only help, never make CI worse. Deployed early so the seed
overlaps the Cozystack install; readiness is resolved once and cached
across bats files. Uses the overridable chart value added in the previous
commit.

Refs #3231

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:11:40 +05:00
Myasnikov DaniilandClaude Fable 5 ece64ee0a9 test(etcd): re-port the Chainsaw suite to the v1alpha2 operator contracts 
The suite was ported from the pre-#2859 etcd.bats and asserted
etcd.aenix.io/v1alpha1 objects that no longer exist on main — every
Test hard-failed at the first assert on an unknown GVK.

- drop the etcd-empty-backup and etcd-backup-schedule scenarios with
  their fixtures: the chart no longer renders EtcdBackupSchedule or
  etcd-s3-creds (backups moved to the generic backups API), and
  legacy-values tolerance is pinned by helm-unittest, not e2e
- rewrite the remaining scenario into etcd-1-contracts, mirroring
  main's contracts test: gate on status.readyMembers=3 (Available=True
  flips at seed-member discovery and races replica-count contracts),
  assert the operator-owned member Pod shape (strict selector,
  container name, named metrics port), WorkloadMonitor
  operational/availableReplicas, and script the three contracts that
  need imperative probes: the /scale subresource, the plaintext-HTTP
  metrics probe against the Pod IP, and the on-demand defrag Job
- add etcd-2-backup-roundtrip driving examples/backups/etcd/run-all.sh,
  gated on ETCD_E2E_S3_ROUNDTRIP=1 exactly like main (kind CI has no
  publicly-trusted S3 endpoint the snapshot agent could verify)
- upgrade catch blocks to the v1alpha2 GVKs plus EtcdMember/
  EtcdSnapshot/BackupJob describes and the snapshot-Job Pod logs
- strengthen _lib/etcd-cleanup.sh with main's etcd_drain deltas: probe
  the Etcd CR itself (it can linger Terminating and block the next
  singleton apply) and use the err-sentinel pattern so a transient API
  error is never misread as "deleted"; add etcd_backup_cleanup wrapping
  examples/backups/etcd/cleanup.sh for the round-trip's teardown

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:10:20 +05:00
Myasnikov DaniilandClaude Fable 5 efbf372d25 test(e2e): fix post-merge seams the conflict markers missed 
- nightly.yaml still looped over hack/e2e-apps/*.bats with the removed
  test-apps-% target; switch it to the single chainsaw invocation
  (empty CHAINSAW_SUITES = full suite) and upload the JUnit report
- pin the chainsaw-report upload-artifact step to the same SHA the
  rest of the workflows use (scorecard/zizmor gate)
- repoint hack/run-kubernetes-drain_test.bats at the relocated
  hack/e2e-chainsaw/_lib/run-kubernetes.sh
- replicate cozytest.sh's NotReady-pod data-plane capture
  (hack/e2e-capture-dataplane.sh) in the Chainsaw global error.catch so
  chainsaw-suite failures keep diagnostic parity with the BATS lanes
- refresh stale bats path comments

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-08 18:06:22 +05:00