admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-06 12:03:39 +09:00
Code Issues Packages Projects Releases Wiki Activity
5,280 Commits 571 Branches 294 Tags
ci/shellcheck-gate
Commit Graph
45 Commits
Author SHA1 Message Date
Aleksei Sviridkin 7739d1e294 feat(multus)!: stage reference CNI plugins from the multus image 
Multus-backed NetworkAttachmentDefinitions delegate to the upstream
containernetworking reference plugins (bridge, macvlan, ipvlan, ...),
and nothing guarantees a node has them in the directory Multus
searches.  On k3s they live under /var/lib/rancher/k3s/data/cni while
Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails
there with `failed to find plugin "bridge" in path [/opt/cni/bin]`.

Bundle 14 of the release's plugins into the multus image and add an
install-cni-plugins init container that installs them into the host
/opt/cni/bin, mirroring the existing install-multus-binary container.
loopback, dhcp, tap and dummy are left out; the Dockerfile records why.

Gate it on a new chart value, stageCniPlugins, resolved per bundle and
on for both.  Talos bundles only bridge, firewall, flannel, host-local,
loopback and portmap, so the other ten are missing there exactly as on
generic Linux; the four that overlap are rewritten with the same
upstream version, v1.9.1 on both sides, rather than an older one.  A
marker in the image Dockerfile records the Talos release that pin was
compared against, and a test holds it to the release the platform
ships.  multus therefore moves out of system.common-packages into the
two variant branches, the way linstor already does for talos.enabled.
networking.stageCniPlugins overrides the variant default in either
direction.

Where staging is on the install is unconditional: a plugin is replaced
on every pod recreation, not only when the destination is missing.  A
destination that is already a directory is reported and left alone,
since replacing it is not something this can do safely.  Turning the
value off afterwards does not undo it; nothing here removes or
restores a plugin.

The container is not privileged and takes no mount propagation.  It
runs in the spc_t SELinux domain, without which an enforcing node
refuses the write.  Each plugin is installed under a temporary name and
swapped in with a rename: /opt/cni/bin is shared with cilium and
kube-ovn.  The plugin tarball is checksummed before extraction.

A plugin that cannot be installed is reported on stderr and to
/dev/termination-log, and the container still exits 0: Multus stays the
node's primary CNI and nothing removes its config, so an init container
that never completes would leave the shim with no daemon behind it.

The manifest change lives in customize-deployment.patch so it survives
make update.

BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI
plugins in /opt/cni/bin on existing nodes, Talos included -- four of the
names it writes are ones the Talos node image also carries.  Opt out
before upgrading: declining afterwards does not restore what was
replaced.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-08-05 14:19:47 +03:00
myasnikovdaniil 3f967ec7e0 fix(platform): restore the isp-hosted dependency closure (#3391) 
## Problem

Fixes #3376. The `isp-hosted` preset shipped `bundles.system.enabled:
false` while `paas` and `naas` were enabled, so it enabled an app
catalog whose dependency closure it simultaneously disabled. Every
emitted Package sat in `DependenciesNotReady`, the `*-rd` HelmReleases
failed with `no matches for kind "ApplicationDefinition" in version
"cozystack.io/v1alpha1"`, and the `cozystack-platform` HelmRelease never
went Ready.

Rendering the preset at `origin/main` shows the scale: **24 emitted
Packages, 24 of them blocked**, by 6 missing dependencies — two more
than the issue lists (`cert-manager` and `prometheus-operator-crds` are
also absent). `isp-full` renders 75 Packages with a fully satisfied
closure, which is a clean control.

## Root cause

`templates/bundles/system.yaml` is gated end to end on
`bundles.system.enabled`, but it carries two separable things: a
variant-selected **data-plane head** (kubeovn/cilium/linstor on
`isp-full`, a noop networking Package on `isp-hosted`) and a
variant-agnostic **control-plane body** — `cozystack-engine`, the sole
source of the `application-definition-crd` component, plus
`cert-manager`, `prometheus-operator-crds`, `postgres-operator` and
`objectstorage-controller`.

"Hosted" means the host already provides CNI, storage and
virtualisation, so only the data-plane head should be dropped. Disabling
the whole bundle also deleted the control-plane body that every variant
needs.

Three things show the disable was the mistake rather than the intent:
the preset still sets `variant: "isp-hosted"`; `system.yaml` contains an
`isp-hosted` branch (noop networking plus default scheduler) that could
never render; and `iaas.yaml` already refuses to render on this variant,
encoding the same "system bundle minus iaas" contract. The preset has
never worked — it was born this way in `2d022e38e` (January 2026), the
refactor that replaced `bundles/paas-hosted.yaml`, whose contents
confirm hosted was always meant to ship the engine, `cert-manager` and
`prometheus-operator-crds`.

## Changes

1. **`values-isp-hosted.yaml`** — enable the system bundle. Hosted now
emits 53 Packages with every `dependsOn` satisfied, installing 59
components against the 60 of the pre-refactor `paas-hosted` bundle, with
no LINSTOR, KubeVirt, CAPI or Kamaji leakage.
2. **Move `cozystack.bucket-application`** from the iaas bundle into the
system bundle. Its own `dependsOn` closure is `networking` +
`objectstorage-controller` + `cozystack-engine`, none of them iaas,
while `cozystack.backupstrategy-controller` — a *default* package of the
system bundle — hard-depends on it. Any system-on/iaas-off configuration
therefore wedged `backupstrategy-controller`; both `isp-full` presets
masked this by enabling iaas. This is a second, independent bug not
reported in the issue.
3. **Guards** — `paas`, `naas` and `iaas` now also require
`bundles.system.enabled`. The pre-existing checks looked only at
`bundles.system.variant`, which keeps its value when the bundle is
disabled, so they passed on exactly this configuration. `iaas` had the
identical hole (12 Packages, 6 unsatisfied deps). This turns a silent
multi-hour `DependenciesNotReady` hang into a render-time failure that
names the remedy.
4. **Two helm-unittest suites** — one pinning the dependency closure the
catalog needs, one pinning the guards.

## Verification

- All four operator-registered variants render dependency-closed:
`default` 0 Packages, `isp-hosted` 53, `isp-full` 75, `isp-full-generic`
75. The two `isp-full` variants are byte-identical before and after, so
the bucket move is a no-op for them.
- Hosted is a strict subset of full with zero data-plane leakage, and
its exclusions match the pre-refactor hosted bundle.
- 23 suites / 97 tests pass in `packages/core/platform`; repo-wide
`hack/helm-unit-tests.sh` is green.
- Every change was **mutation-tested** — reverting each one in turn
makes the suite fail, so the tests are not vacuous.
- The workaround documented in #3376 (`paas`/`naas` disabled) still
renders cleanly; the new guards do not fire on it.
- Hosted runtime spot-checks: `cozystack-api` is a Deployment with only
a *preferred* control-plane affinity plus blanket tolerations, and
`cozystack-scheduler` uses its `default` variant (plain Deployment, no
LINSTOR extender) — both schedule fine on a cluster with no
control-plane nodes.

## What this does not fix

This restores the dependency closure. It is **not** by itself sufficient
for a fully converged hosted install. Review surfaced further
hosted-unaware defaults that the dead bundle had been masking, filed
separately:

- #3387 — `apps/tenant` renders 14 `cilium.io/v2` policies with no
`.Capabilities` guard, so `tenant-root` fails on a non-Cilium cluster.
This is the blocker for actual hosted convergence, and it needs a
deliberate decision rather than a reflexive capability guard, because
those policies implement tenant network isolation.
- #3388 — `extra/monitoring` defaults `logsStorages` to the LINSTOR-only
`replicated` StorageClass.
- #3389 — `bundles.disabledPackages` can still break the closure with no
render-time error; the guards here cover the bundle flags only.
- #3390 — nothing in CI exercises `isp-hosted`, which is why all of the
above accumulated undetected.

## Upgrade note

Existing `*-application` Package CRs carry `helm.sh/resource-policy:
keep`. An operator who applied the #3376 workaround but did not also
delete the retained Package CRs (as that issue instructs) will find them
install once this change makes their dependencies Ready, since the
reconciler re-enqueues dependents on readiness. Deleting the unwanted
Packages before upgrading avoids that.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added fail-fast validation that requires the **System bundle** when
enabling **IaaS**, **NaaS**, or **PaaS** to prevent incomplete catalogs.
* Updated bundle composition so the **bucket application** is provided
by the **System bundle**.
* Enabled the **System bundle** by default for the `isp-hosted` variant.
* **Bug Fixes**
* Improved error messaging to clearly indicate the required
bundle/variant configuration.
* **Tests**
* Added Helm unittest coverage for rendering-gate failures,
dependency/package expectations, and hosted data-plane exclusions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-03 21:27:56 +05:00
Andrei Kvapil 4c3262a4ee feat(computeplane): deliver ComputePlane as a tenant module wrapping the kubernetes app 
Add the operator-owned computeplane tenant module per the revised design
(cozystack/community design-proposals/compute-plane): an extra/computeplane
chart that provisions a single-tenant, Cozystack-managed cluster for
untrusted-code apps by deploying the ordinary apps/kubernetes chart with
operator-fixed values (fluxcd, cert-manager, ingress-nginx Proxied). The
package re-declares apps/kubernetes as its own source-only component, so the
module is self-contained; the cluster release is named computeplane so Kamaji
writes the admin kubeconfig to computeplane-admin-kubeconfig — the contract
consumed by placement: ComputePlane applications. The tenant holds no admin
kubeconfig and can set cluster shape (nodeGroups) only.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-07-28 18:08:02 +02:00
Aleksei Sviridkin 8bffa34b54 feat(platform): add opt-in in-cluster east-west encryption toggle (#2984) 
## What this PR does

Adds `networking.encryption.enabled` to the platform chart — a single
operator-facing opt-in, off by default, that turns on in-cluster
(east-west) transport encryption for the active datapath. When enabled,
pod-to-pod traffic across nodes is encrypted at the network layer
regardless of whether the workload implements TLS: defense-in-depth for
the whole mesh without per-application changes.

The default `isp-full` bundle selects the `kubeovn-cilium` datapath (and
`isp-full-generic` selects `kubeovn-cilium-generic`), where KubeOVN owns
the datapath and the overlay is a Geneve tunnel. The toggle therefore
maps to KubeOVN's overlay IPsec (`kube-ovn` `ENABLE_OVN_IPSEC`).
KubeOVN's own controller provisions the IPsec CA and per-node
certificates through the Kubernetes CSR API, so no cert-manager wiring
is needed.

This complements, and does not replace, application-level / end-to-end
TLS: in-cluster encryption never reaches an external client and gives a
tenant no verifiable endpoint certificate. The values comments document
the per-datapath scope — including that Kilo's WireGuard mesh
(`meshGranularity: location`) covers only cross-location traffic and
leaves the intra-location gap this toggle targets, and that the
Cilium-native WireGuard/IPsec path on the `cilium` / `cilium-generic`
variants is not reached because the system bundle does not select those
variants — plus the MTU/throughput overhead.

Scope: only the default KubeOVN datapath is wired through the platform
toggle, which is what the system bundle actually selects. Wiring the
Cilium-native path would require the bundle to expose cilium-only
variant selection; that is out of scope here and called out in the
values comments.

Validation: `helm unittest` on the platform chart passes (66 tests, 7
new) covering on / off / absent / null and verifying the IPsec flag
deep-merges without dropping the existing `kube-ovn` overrides, for both
`isp-full` and `isp-full-generic`.

Closes #2977

Part of #2811

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(platform): add opt-in `networking.encryption.enabled` (off by default) that enables in-cluster east-west transport encryption for the default KubeOVN datapath via overlay IPsec; complements, and does not replace, application-level TLS.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Introduced opt-in in-cluster network encryption using KubeOVN IPsec
for east-west traffic protection.
* Added `networking.encryption` configuration toggle (disabled by
default) with full backward compatibility.

* **Tests**
* Added comprehensive test coverage for network encryption
configurations across system variants and edge cases.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-27 22:47:38 +03:00
Timofei Larkin 2823b76802 feat(gateway): advertise HTTP/2 via ALPN on Gateway API listeners (#3398) 
## What this PR does

Browsers negotiate HTTP/2 exclusively through TLS ALPN: if the
terminating listener does not advertise `h2`, every client silently
falls back to HTTP/1.1, with no error surfaced anywhere. Cozystack's
Gateway API path is served by Cilium's embedded Envoy, and Cilium ships
with `enable-gateway-api-alpn=false` — so every Gateway listener
advertised no ALPN protocols and all browser traffic through a Gateway
ran over HTTP/1.1 (observable in browser dev tools as `http/1.1` on
every request). The ingress-nginx path this replaces advertised `h2` out
of the box, so the migration to Gateway API silently downgraded clients:
head-of-line blocking and the ~6-connections-per-origin limit are back
for anything served through a Gateway.

This PR enables ALPN (`h2`, then `http/1.1`) on Gateway API listeners,
default-on, with an opt-out at each management layer:

- `packages/system/cilium/values.yaml` now defaults
`gatewayAPI.enableAlpn: true`, which renders `enable-gateway-api-alpn:
"true"` into the `cilium-config` ConfigMap.
- The platform chart gains `gateway.http2` (default `true`) next to the
existing `gateway.enabled`/`gateway.attachedNamespaces`. The `isp-full`
and `isp-full-generic` bundles thread it as inline component values into
the `cozystack.networking` Package CR
(`components.cilium.values.cilium.gatewayAPI.enableAlpn`); inline
HelmRelease values override the package `valuesFiles`, so an explicit
`gateway.http2: false` actually reaches the dataplane rather than being
shadowed by the new chart default. On `isp-hosted` the hosting-side CNI
owns this setting and the flag is a no-op.
- Tenant Kubernetes clusters (`packages/apps/kubernetes`) get the same
fix: when the `gatewayAPI` addon is enabled, the tenant cilium
HelmRelease now sets `enableAlpn: true`; individual clusters opt out via
`addons.cilium.valuesOverride.cilium.gatewayAPI.enableAlpn=false`.

### Why default-on rather than opt-in

HTTP/2 for browsers was the platform's behavior before the Gateway API
migration, so keeping ALPN off would preserve a regression, not a status
quo. Defaulting on is safe because ALPN is a negotiation: clients that
cannot speak HTTP/2 simply keep negotiating `http/1.1`, exactly as
before. Operators who need the old wire behavior can set `gateway.http2:
false` platform-wide.

### Frontend vs backend hop

This change affects the client↔gateway hop only. Gateway↔backend
connections stay HTTP/1.1 unless a Service explicitly opts in per
GEP-1911 by declaring `appProtocol: kubernetes.io/h2c` on its port.
Cilium couples that backend-protocol support to the ALPN flag
(`enable-gateway-api-app-protocol` flips to true together with it), but
it only acts on Services that declare a recognized `appProtocol` value —
and no cozystack-shipped Service does today (only plain `http`/`https`
values exist in-tree, which Cilium ignores for protocol selection). So
upstreams see no change from this PR alone; h2c/gRPC backends become
possible to configure, not configured.

The setting is cluster-wide Cilium agent config, so it covers the root
Gateway and all tenant Gateways at once — Cilium offers no per-Gateway
granularity here. Flipping it re-rolls the cilium DaemonSet on upgrade
(`rollOutCiliumPods: true`), the same disruption profile as any other
cilium config change.

### How it was verified

- New helm-unittest suites:
`packages/core/platform/tests/bundles_gateway_http2_wiring_test.yaml`
asserts the Package CR carries `enableAlpn` true by default and false
with `gateway.http2: false` for both cilium bundles;
`packages/apps/kubernetes/tests/cilium_gateway_alpn_test.yaml` asserts
the tenant HelmRelease default, the addon-off case, and the
`valuesOverride` opt-out. Full suites for both charts pass (93 and 184
tests).
- `helm template` of `packages/system/cilium` confirms the value is
consumed, not just rendered: default produces `enable-gateway-api-alpn:
"true"` (and `enable-gateway-api-app-protocol: "true"`) in
`cilium-config`; the threaded `false` produces `"false"` for both.

### Screenshots

Not a UI change.

### Downstream repositories

The website's hand-written platform values table
(`content/en/docs/next/operations/configuration/platform-package.md`)
needs a row for `gateway.http2` — filed as cozystack/website#625. No
other downstream repository restates anything in this diff (no app
`values.schema.json`, ansible-consumed platform key, or hack/ tooling
changed).

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/625
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(gateway): Gateway API listeners now advertise HTTP/2 via TLS ALPN (h2, then http/1.1) by default, restoring the browser HTTP/2 support lost in the migration from ingress-nginx. Opt out platform-wide with gateway.http2=false, or per tenant Kubernetes cluster via addons.cilium.valuesOverride.cilium.gatewayAPI.enableAlpn=false. Backend (gateway-to-upstream) connections are unchanged and stay HTTP/1.1 unless a Service declares appProtocol kubernetes.io/h2c.
```
 2026-07-23 16:05:31 +04:00
Timofei Larkin ebe111c439 feat(gateway): advertise HTTP/2 via ALPN on Gateway API listeners 
Browsers negotiate HTTP/2 exclusively through TLS ALPN "h2". Cilium's
Gateway API implementation ships with enable-gateway-api-alpn=false, so
every Gateway listener advertised no ALPN protocols and every browser
silently fell back to HTTP/1.1 — a regression against the ingress-nginx
path, which advertised h2 out of the box.

Enable ALPN (h2, then http/1.1) by default on the client-facing hop:

- packages/system/cilium: default gatewayAPI.enableAlpn=true.
- platform chart: new gateway.http2 value (default true), threaded as
  inline component values into the cozystack.networking Package for the
  isp-full and isp-full-generic bundles, so an explicit false overrides
  the system/cilium valuesFiles default either way.
- apps/kubernetes: tenant-cluster cilium gets enableAlpn=true whenever
  the gatewayAPI addon is enabled; opt out per cluster via
  addons.cilium.valuesOverride.

The backend hop is unchanged: gateway-to-upstream stays HTTP/1.1 unless
a Service opts in per GEP-1911 via appProtocol kubernetes.io/h2c (that
support is switched on together with ALPN; no cozystack-shipped Service
declares such a value today).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
 2026-07-21 14:40:30 +03:00
Andrey KolkovandClaude Fable 5 7d117679d4 docs(backups): document bucketName's two modes, cover isp-hosted in the wiring test 
Branch-review on this PR: for provisionBucket: false the
backupstrategy-controller.bucketName helper returns
.Values.backupStorage.bucketName verbatim as the real S3 bucket name
and every strategy CR plus the Velero BSL bakes it in, so the
external-S3 section must require updating bucketName (the Secret's
bucketName key alone is not read by the strategies), and the knob
table must state the two-mode semantics instead of the COSI-only one.
Also route the external-S3 instruction through the platform component
override instead of naming raw chart values, and add the isp-hosted
variant case to the forwarding test (92 tests green).

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-21 14:49:02 +04:00
Andrey KolkovandClaude Fable 5 1422f8b78a fix(platform): forward backupStorage overrides to the backupstrategy-controller Package (#3245) 
The documented admin override for the cozy-default backup S3
coordinates (spec.components.backupstrategy-controller.values on the
cozystack.cozystack-platform Package) had no effect: that PackageSource
exposes only the platform component, so the operator silently ignored
the block, and nothing in the platform chart forwarded backupStorage to
the child Package.

Wire it the same way bundles/iaas.yaml forwards kubevirt / gpu-operator
values: a new top-level backupStorage value on the platform chart is
rendered into the emitted cozystack.backupstrategy-controller Package
CR as components.backupstrategy-controller.values.backupStorage, from
where the cozystack operator merges it into the
backupstrategy-controller HelmRelease over the chart defaults. When
backupStorage is unset the emitted Package is unchanged.

Update docs/operations/backup-classes.md to the working override path
(spec.components.platform.values.backupStorage) and call out the two
plausible-looking paths that do not work. Add a helm-unittest suite
covering the no-op default and the forwarding on isp-full and
isp-full-generic.

Fixes #3245

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-21 14:47:08 +04:00
Myasnikov Daniil 0bfbff26ff fix(platform): restore the isp-hosted dependency closure 
The isp-hosted preset shipped bundles.system.enabled: false with paas and
naas enabled. templates/bundles/system.yaml is gated end to end on that
flag, but it carries two separable things: a variant-selected data-plane
head (kubeovn/cilium/linstor on isp-full, a noop networking Package on
isp-hosted) and a variant-agnostic control-plane body — cozystack-engine,
the sole source of the application-definition-crd component, plus
cert-manager, prometheus-operator-crds, postgres-operator and
objectstorage-controller.

"Hosted" means the host already provides CNI, storage and virtualisation,
so only the data-plane head should be dropped. Disabling the whole bundle
also deleted the control-plane body, leaving an app catalog whose entire
dependency closure was absent: all 24 emitted Packages sat in
DependenciesNotReady, the *-rd HelmReleases failed with `no matches for
kind "ApplicationDefinition" in version "cozystack.io/v1alpha1"`, and the
cozystack-platform HelmRelease never went Ready. That the variant field
was still set to isp-hosted — and that system.yaml has an isp-hosted
branch which could never render — shows the disable was the mistake, not
the intent; iaas.yaml already refuses to render on this variant, encoding
the same "system bundle minus iaas" contract. The preset has never worked
since it was introduced in 2d022e38e.

Enable the system bundle for the preset. isp-hosted now emits 53 Packages
with every dependsOn satisfied, and installs 59 components against the 60
of the pre-refactor bundles/paas-hosted.yaml, with no LINSTOR, KubeVirt,
CAPI or Kamaji leakage.

Also move cozystack.bucket-application from the iaas bundle into the
system bundle. Its own dependsOn closure is networking,
objectstorage-controller and cozystack-engine, none of which are iaas, and
cozystack.backupstrategy-controller — a default package of the system
bundle — hard-depends on it. Any system-on/iaas-off configuration
therefore wedged backupstrategy-controller; both isp-full presets masked
this by enabling iaas.

Guard the class of bug rather than only these two instances. The existing
paas/naas/iaas guards check bundles.system.variant, which keeps its value
when the bundle is disabled, so they passed on exactly this
configuration. Make all three require bundles.system.enabled too and point
the operator at the supported topology, turning a silent multi-hour
DependenciesNotReady hang into a render-time failure.

Covered by two helm-unittest suites: one pinning the dependency closure
the catalog needs — including a case that loads values-isp-hosted.yaml
unmodified, so a regression in the preset itself is caught — and one
pinning the guards. The fix changes and the guards were mutation-tested by
reverting each in turn and confirming the suite fails.

This restores the dependency closure; it is not by itself sufficient for a
fully converged hosted install. Review surfaced further hosted-unaware
defaults that the dead bundle had been masking — notably apps/tenant
rendering cilium.io/v2 policies with no .Capabilities guard, and
extra/monitoring defaulting logsStorages to the LINSTOR-only "replicated"
StorageClass. Those are tracked separately. bundles.disabledPackages can
also still remove a dependency while its dependents stay emitted; the
guards do not cover that path.

Refs #3376

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-21 11:51:52 +05:00
Myasnikov Daniil efa85a4f61 feat(platform): make the etcd-adoption skip-backup hatch reachable 
Migration 50 has honoured ETCD_ADOPT_SKIP_BACKUP since it was written, and
its failure paths tell operators to "re-run with ETCD_ADOPT_SKIP_BACKUP=1".
But migration-hook.yaml passes only NAMESPACE/CURRENT_VERSION/
TARGET_VERSION, and the Job is a Helm hook the chart re-renders, so there
has never been a way to set it. The script advertised an escape the
platform made impossible. The endpoint-classification work landing
alongside this makes that worse: it adds refuse-rather-than-guess paths,
so more clusters can now legitimately stop here.

Plumb it through migrations.etcdAdoptSkipBackup, default false. Skipping
the pre-adoption snapshot rewires ownership of LIVE etcd storage with no
way back, so it stays opt-in and deliberate — the flag is the last resort
for a cluster that has nothing to fix, not the first response to a
refusal.

Render "1"/"0" rather than the bare bool. The script matches the literal
string "1", so a bool would render "true", sail through the Job spec
looking correct, and be ignored — a safety valve that is set but silently
disregarded is worse than one that was never offered, because the operator
believes they opted in. The value is normalised through a string, so a
bool, a quoted "true" out of the Package CR's Values JSON, or a numeric 1
all land on the same answer, while nil or garbage resolves to "0": taking
this hatch must require an affirmative spelling, never a typo. Resolved
with `dig` so a Platform Package predating the key renders "0" instead of
failing the render of the entire platform chart and bricking every
upgrade.

Widen the script's own check to accept 1/true/yes as well. The template
guarantees the GitOps path, but an operator running this image by hand
will reasonably type "true", and that is exactly the moment they are
already stuck. The two layers cover independent failure modes.

Emit the var unconditionally, including the "0" default, so the rendered
Job states the cluster's safety posture outright rather than leaving an
operator to diff values to find out whether a snapshot was skipped.

Point the failure messages at the values path that actually exists, and
describe it honestly: editing the Platform Package and letting Flux
re-render is a real operation, not a one-liner, and the flag has to be
reverted afterwards or the next etcd migration skips its snapshot too.
Fixing the condition the migration names remains the recommended path —
the migration is idempotent, so re-running after a fix is safe.

templates/migration-hook.yaml had no coverage at all, because the Job only
renders when a lookup of the cozystack-version ConfigMap reports a version
below targetVersion, and lookup is nil under helm template. The new suite
mocks that ConfigMap and asserts the RENDERED VALUE — enabled, disabled,
key absent, stringified true, garbage, and the pre-existing three env vars
— rather than merely that the variable exists, which is what let the trap
above go unnoticed in the first place.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-07-20 11:29:09 +05:00
Andrei Kvapil fbbe4af7df refactor(network): drop ExposureClass/ServiceExposure for native loadBalancerClass (#3218) 
## What

Removes the `network.cozystack.io` API group — `ExposureClass` +
`ServiceExposure` (#3081) — and its reconciler, in favor of native
`Service` `type: LoadBalancer` + `loadBalancerClass`.

## Why

Full rationale and discussion in #3164. For **managed applications** the
chart already owns the Service, so:

- `type: LoadBalancer` + `loadBalancerClass` + an admin-provisioned pool
covers vendor-neutral external exposure without a dedicated API group
and controller;
- the `externalIPs` (node-IP) default never went through
`ServiceExposure` (its backend is a no-op) and is left unchanged;
- the only current consumer, the host ingress, keeps rendering `type:
LoadBalancer` and just drops the `ServiceExposure`; the admin provisions
the address pool / announcer out of band, as the install docs already
assume.

`ServiceExposure`'s one unique feature — dynamic per-namespace scoping
of a shared VIP pool — is a convenience the managed-app path does not
need, and it does not justify a second tenant-networking API group next
to `sdn.cozystack.io`.

## Changes

- Delete `api/network/v1alpha1` and
`internal/controller/serviceexposure` (+ CRD definitions, e2e bats,
controller registration, RBAC).
- Host ingress: drop the `ServiceExposure` template; add an optional
`publishing.loadBalancerClass` that sets `spec.loadBalancerClass` on the
host-ingress LoadBalancer Service.
- The `externalIPs` path and default behavior are unchanged.

## Verification

- `go build ./...` — green.
- `helm unittest` — `core/platform` 72/72, `extra/ingress` 25/25 (incl.
host-vs-tenant `loadBalancerClass` scoping and the immutable-field
migration guard).

## Migration

None required. `network.cozystack.io` (#3081) has never shipped in a
release — it is on `main` only (not backported to `release-1.5`; absent
from v1.4.x / v1.5.x), so no upgrading cluster has these CRDs and this
removes it before it is ever released.

Clusters that track `main` and explicitly set `publishing.exposureClass`
(opt-in, only since 2026-06-30) should provision their MetalLB / Cilium
address pool themselves and set `publishing.loadBalancerClass`; any
orphaned `cozystack-<class>` pool can be deleted by hand. Default
clusters (externalIPs path) are unaffected.

## Status

**Draft / RFC for the #3164 discussion** — this removes an API group
that landed only days ago, so nothing here should merge before the
design call. cc @lexfrei (author of #3081) and @lllamnyp (#3164).

Deferred: when Kubernetes actually removes `Service.spec.externalIPs`,
the node-IP default migrates via a MetalLB `IPAddressPool` with no
`L2Advertisement` (allocate-but-don't-announce) — a separate change.



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for configuring the ingress host Service with a native
`loadBalancerClass` value.
* Expanded chart tests to verify class wiring and immutability behavior
across install and upgrade scenarios.

* **Bug Fixes**
* Removed the legacy exposure-class/service-exposure flow and related
controller behavior.
* Updated templates and CRD packaging so the platform no longer renders
or depends on those deprecated resources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-07-16 17:44:28 +02:00
Ivan Okhotnikov b46543e5f2 fix(monitoring): defer platform GrafanaAdmin lever to a future phase 
The MAJOR finding from the cozy-review agent: the platform-side
`oidc.grafanaAdmin: true` documented in the PR body and in
docs/oidc-grafana.md never actually reached the platform Grafana's
HelmRelease. `Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin`
was set in packages/core/platform/templates/bundles/system.yaml, but
`monitoring-system` (and `monitoring`) have no `install:` block in
packages/core/platform/sources/monitoring-application.yaml — only
`monitoring-rd` does. PackageReconciler skips components without an
`install:` block before ever reaching the assignment that copies
`pkg.Spec.Components[].Values` onto `hr.Spec.Values`
(internal/operator/package_reconciler.go around lines 197-201 and
292-293). The values-override dictionary was silently dropped, no
HR carried `oidc.grafanaAdmin: true`, and the platform Grafana
rendered with the chart default (`false`). The
`bundles_monitoring_grafana_admin_test.yaml` unit test only asserted
against the emitted Package CR spec — exactly the layer where the
value still existed but was about to be discarded — so the broken
propagation slipped past.

Trim the broken promise from Phase 1. The chart-side helper
(`monitoring.oidc.allowAssignGrafanaAdmin`) stays as-is so a future
phase that routes the override through the tenant-root chart's
monitoring HR values (or adds an `install:` block on the
`monitoring-system` component) lands on a green chart-side contract.
The helm-unittest that verifies chart-side emission stays too, but
with a docstring calling out that end-to-end wiring is deferred.
Everything else that promised the lever works — bundle line, docs,
platform bundle test — is either removed or rewritten to match
reality.

Also picks up two MINOR findings from the same review:

* Drop the misleading `secret-hash` annotation on the OIDC
  client-secret Secret. The comment claimed it "rotates the
  KeycloakClient CR and therefore the Grafana Deployment env when
  the client-secret changes", but nothing downstream watches an
  annotation on the Secret itself — no rotation is triggered. The
  `lookup`-based fallback already keeps the value stable across
  reconciles, so no rotation is even needed. Removing the dead
  annotation avoids planting a broken assumption for future readers.

* Add `release.cozystack.io/helm-install-timeout: "15m"` on the
  monitoring cozyrds ApplicationDefinition. Under `mode: System` the
  chart now emits, on top of the existing Grafana + Alerta +
  VMCluster + CNPG stack, four additional Keycloak CRs whose
  reconciliation is asynchronous against an operator in
  `cozy-keycloak` (not the tenant namespace). Flux's default 5m
  install window was already tight for the pre-OIDC stack; 15m
  matches the precedent set by tenant-rd/kubernetes-rd cozyrds.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:45 +03:00
Ivan Okhotnikov be52e80289 fix(monitoring): split monitoring-application PackageSource into default/oidc variants 
Follow-up to the previous self-review-fix commit: the unconditional
`- cozystack.keycloak-operator` in dependsOn was itself a
CRITICAL cold-install regression on any cluster where
`authentication.oidc.enabled: false` (the default). The
keycloak-operator Package is only emitted by
`packages/core/platform/templates/bundles/system.yaml:112` when
that flag is on — a hard dep against it from the default variant
would leave `cozystack.monitoring-application` stuck on
DependenciesNotReady forever, cascading a
`cozystack-platform` HR timeout that hits every OIDC-off customer.

Fix by mirroring the cozystack-engine variant split:

- `default` variant keeps the baseline reconciliation edges
  (no keycloak-operator dep).
- `oidc` variant additionally waits for
  `cozystack.keycloak-operator` so the v1.edp.epam.com CRDs are
  registered before the monitoring chart reconciles and the
  `.Capabilities.APIVersions.Has` guard in
  templates/grafana/oidc-keycloak.yaml can safely emit
  KeycloakClient/Scope/Groups.

The bundle template swings between variants on
`.Values.authentication.oidc.enabled`, mirroring the
cozystack-engine invocation two lines above. The
`monitoring-system` component override
(oidc.grafanaAdmin: true) is preserved across both branches — the
values field is a no-op when mode: None (no auth.generic_oauth
block is emitted at all), so keeping it identical avoids two
divergent code paths.

Also widened the platform helm-unittest coverage to lock:

- default variant does NOT depend on keycloak-operator (permanent
  DependenciesNotReady guard).
- oidc variant DOES.
- Bundle emits the right variant based on
  authentication.oidc.enabled AND keeps the GrafanaAdmin uplift
  on the monitoring-system component in either branch.

83 platform tests + 23 monitoring tests still green.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:44 +03:00
Ivan Okhotnikov ffeb8e5e4f fix(monitoring): close self-review blockers for the OIDC PR 
Address the two blocking-class findings from the cozy-review pass on
this PR:

(1) Missing PackageSource dependency on cozystack.keycloak-operator
    (MAJOR cold-install race).

    Without this edge, the monitoring chart can reconcile before the
    v1.edp.epam.com CRDs are registered on a fresh cluster — the
    `.Capabilities.APIVersions.Has` guard in
    templates/grafana/oidc-keycloak.yaml then silently drops all
    Keycloak-side objects for mode: System, and Grafana's
    auth.generic_oauth block points at a client Keycloak never
    provisions. Add the ordering edge, plus a
    sources_monitoring_application_dependson_test.yaml regression
    guard so a future edit cannot drop it silently.

(2) allow_assign_grafana_admin driven by `.Release.Name ==
    monitoring-system` (MINOR fragile coupling).

    Replace the release-name sniff with an explicit
    `oidc.grafanaAdmin: false` values field, and flip it on for the
    platform Grafana release via
    Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin
    in packages/core/platform/templates/bundles/system.yaml —
    mirroring the pattern monitoring-agents already uses for
    per-component overrides. Tenant Grafana releases inherit the
    chart default false and stay at org-level Admin.

    Two new helm-unittest cases in packages/system/monitoring/tests/
    lock the values-driven behaviour on both sides
    (grafanaAdmin=true → Admin uplift, default → tenant stays at org
    Admin), plus bundles_monitoring_grafana_admin_test.yaml in
    packages/core/platform/tests/ regression-guards the Package's
    component override so the platform release keeps its uplift on
    future edits.

Also document the mode-toggle group-membership loss in
docs/oidc-grafana.md — mode: System → CustomConfig/None deletes the
three chart-owned KeycloakRealmGroups; membership is not restored on
toggle-back. Not a bug, but the operator UX cost of the toggle
deserves a bullet in Failure modes.

Regenerated artifacts (cozyvalues-gen + hack/update-crd.sh):
  packages/extra/monitoring/values.schema.json
  packages/extra/monitoring/README.md
  packages/system/monitoring-rd/cozyrds/monitoring.yaml

Test suites still green: packages/system/monitoring 23/23,
packages/core/platform 81/81.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:44 +03:00
Aleksei Sviridkin f24b56327b fix(platform): route OIDC backend calls through the in-cluster Keycloak by default 
authentication.oidc.keycloakInternalUrl defaulted to an empty string, which
left oauth2-proxy performing OIDC discovery against the external Keycloak
hostname. That request resolves through public DNS and lands on the root
ingress controller, which does not serve yet on a fresh install: the root
tenant's IngressClass only exists once tenant-root is ready, and tenant-root
is still reconciling while the dashboard starts. Discovery has nothing to talk
to, the gatekeeper fails, and the dashboard never comes up.

The bundled Keycloak is the only supported provider — the external hostname is
templated as keycloak.<host> and the realm is fixed — so backend-to-backend
calls have no reason to leave the cluster. Default the value to
http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy, the same Service
address keycloak-configure already uses. The dashboard and linstor-gui
gatekeepers then run with --skip-oidc-discovery and send token, jwks, userinfo
and logout requests to the Service, while browser redirects stay on the
external host. Those calls travel as plaintext HTTP between pods, as
keycloak-configure's admin calls to the same address already do.

The value remains overridable for a Keycloak that does not live at the standard
in-cluster address, and setting it back to an empty string restores external
discovery. Existing OIDC clusters that never set it get a gatekeeper rollout
onto the internal path on upgrade.

Enabling OIDC swaps the dashboard's token-proxy container for oauth2-proxy, so
the E2E install now drives that swap to completion: it waits for the auth-proxy
container to appear, then gates on the gatekeeper rollout. Readiness alone
proves nothing there — the HelmRelease is still Ready from the token-proxy
install, and the Deployment keeps the old ReplicaSet Available while a new pod
crashloops, so only the rollout catches a regression back to external discovery.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-14 14:22:46 +03:00
Alexey Artamonov f9afc73096 fix(monitoring): surface missing VictoriaLogs and gate dead-end vlinsert ExternalName (#3181) 
Platform log ingestion could fail silently: fluent-bit shipped to
vlinsert-generic while the monitoring HelmRelease reported Ready, and a
vlinsert-generic ExternalName in cozy-monitoring dead-ended at a
tenant-root VictoriaLogs that did not exist when the root tenant had
monitoring disabled. Logs were dropped with no failing signal.

Three complementary changes:

- Readiness (do not stay silently Ready): the monitoring-system
  HelmRelease now uses a `poller` waitStrategy and a healthCheckExpr on
  its VLCluster (status.updateStatus), so a missing/unhealthy log store
  makes the release not-ready instead of flipping Ready as soon as the
  CR is applied. Mirrors the seaweedfs CNPG pattern.

- No empty-log-store hole: system/monitoring rejects an empty
  logsStorages at render time (helm fail). With no entry no VLCluster is
  created, the readiness gate has nothing to match, and fluent-bit would
  silently drop logs — so this is made a loud misconfiguration rather
  than a documented footgun.

- No silent fallback: the cozy-monitoring -> tenant-root
  vlinsert/vminsert ExternalName redirects in cozystack-basics are now
  created only when the root tenant hosts monitoring, gated on
  _cluster.monitoring-enabled — fed from a new platform value
  monitoring.rootEnabled (default true, preserving the historical
  always-create behaviour). monitoring.rootEnabled is the single
  render-time source of truth: changing it re-renders cozystack-basics
  (helm-controller watches the cozystack-values Secret it flows into). A
  live `lookup` of Tenant/root spec.monitoring is deliberately NOT used:
  cozystack-basics is an operator-generated HelmRelease with drift
  detection off, so it re-renders only on install/upgrade/values-change
  — never on a reconcile tick or when a lookup result changes — and a
  lookup would go stale in both directions. Operators must keep
  monitoring.rootEnabled in sync with Tenant/root spec.monitoring.

Covered by helm-unittest across all four charts: readiness gate fields,
empty-vs-populated logsStorages contract, ExternalName present/absent
and null-guard contract, and the platform wiring into the
cozystack-values Secret. The CEL health predicates are exercised live,
not in helm-unittest (which cannot evaluate CEL). Adds the missing
`test:` target to packages/extra/monitoring/Makefile so its suite runs
in CI.

Known limitation (follow-up): the legacy cozystack.monitoring
PackageSource that installs system/monitoring directly into
cozy-monitoring cannot carry the readiness gate, because the engine's
ComponentInstall/HelmRelease builder and the vendored helm-controller
API v1.4.3 lack waitStrategy/healthCheckExprs. That source is not
referenced by any active bundle; full coverage needs a helm-controller
bump plus engine support.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-07-08 22:27:22 +03:00
Aleksei Sviridkin a1748009ac fix(ingress): scope host loadBalancerClass to the external-facing ingress 
publishing.loadBalancerClass is a host-publishing setting, but its
_cluster.load-balancer-class block was emitted on any ingress taking the
LoadBalancer branch — including tenant ingresses — unlike expose-external-ips
and proxy-protocol, which gate on the host namespace. A tenant ingress would
then be claimed by an LB controller the tenant never selected. Gate it the
same way, and add helm-unittest coverage for the host/tenant split and the
platform to _cluster wiring.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-07-07 12:39:30 +03:00
Andrei Kvapil 3dbc7d3662 refactor(network): drop ExposureClass/ServiceExposure for native loadBalancerClass 
Remove the network.cozystack.io API group (ExposureClass + ServiceExposure)
and its reconciler. Managed-application charts own their Service, so native
type: LoadBalancer + loadBalancerClass + an admin-provisioned pool covers
external exposure without a dedicated API group and controller. The host
ingress -- the only consumer -- keeps rendering type: LoadBalancer and gains
an optional publishing.loadBalancerClass to pick the LB controller; the
externalIPs (node-IP) default path is unchanged.

Rationale in cozystack#3164.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-07-07 12:39:30 +03:00
Aleksei Sviridkin cbc314a8c5 fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public 
The kubevirt-cdi default variant only depended on cozystack.networking,
which reconciles long before cozystack-basics provisions the cozy-public
namespace. kubevirt-cdi ships the cdi-clone-dv RoleBinding into cozy-public,
so on a fresh install the RoleBinding can race namespace creation and fail
with `namespaces "cozy-public" not found`. Flux retries normally mask it,
but under slow reconcile it surfaces as a hard UpgradeFailed.

Add a cozystack.cozystack-basics edge so the namespace exists first,
mirroring the backupstrategy-controller ordering. No cycle: cozystack-basics'
own dependencies (tenant-application, cozystack-engine, gateway-api-crds) do
not reach kubevirt-cdi. Pin the edge with a helm-unittest regression guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-30 19:51:46 +03:00
Aleksei Sviridkin 31c15b282f feat(network): mechanism-agnostic LoadBalancer exposure (ExposureClass + ServiceExposure) (#3081) 
## What this PR does

Lays the foundation for migrating tenant LoadBalancer exposure off the
deprecated `Service.spec.externalIPs` (KEP-5707) and, eventually, off
MetalLB onto Cilium-native announcement — the mechanism-agnostic shape
prescribed when the earlier chart-side approach (#2468) was reverted in
#2603.

It adds a new `network.cozystack.io/v1alpha1` API group and a controller
that translates intent into whichever LoadBalancer backend the cluster
admin configured:

- `ExposureClass` (cluster-scoped, admin): binds a logical class name to
a backend (`externalIPs` | `metallb` | `cilium` | `robotlb`) and its VIP
address scope — the StorageClass/IngressClass pattern.
- `ServiceExposure` (namespaced, tenant intent): "expose this Service
via class X"; status reports the resolved backend and assigned IPs.
- A `serviceexposure` controller renders the backend's pool + announcer
once per class (MetalLB `IPAddressPool` + `L2Advertisement`; Cilium
`CiliumLoadBalancerIPPool` + `CiliumL2AnnouncementPolicy`), scoped to
the namespaces that opted in. Backends are pluggable behind a small Go
interface, so adding BGP or another cloud LB is one implementation, not
an API change. The controller never mutates the target Service — the
chart still owns its shape.

Platform wiring: `publishing.exposureClass.*` renders the admin
`ExposureClass` and the host ingress publishes through it. The default
(no class set, `externalIPs`) leaves every existing cluster's behaviour
byte-for-byte unchanged; the MetalLB→Cilium announcer migration becomes
a single `ExposureClass.spec.backend` change.

This is the foundation only — Part of #2586. Converting per-app
exposure, flipping Cilium `l2announcements` on, changing the default,
and removing MetalLB are deferred to later phases.

### Release note

```release-note
feat(network): add ExposureClass and ServiceExposure for mechanism-agnostic LoadBalancer exposure. Opt-in foundation; the default externalIPs publishing path is unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added new `ExposureClass` and `ServiceExposure` custom resources with
support for multiple backends (externalIPs, MetalLB, Cilium, RobotLB),
including optional L2 and address configuration.
* Added Helm wiring and templates to enable service/ingress publishing
via `ServiceExposure`, selecting a configured (or default)
`ExposureClass`.
* **Bug Fixes**
* Improved reconciliation, finalizer-based cleanup, and status updates
for readiness and assigned IP reporting, including safer handling when
backends or managed resources change.
* **Tests**
* Added end-to-end and unit test coverage for backend rendering,
readiness, and cleanup flows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-30 12:29:48 +03:00
Kirill Ilin 5cb6ef435f feat(keycloak): opt-in separate hostname and route for the admin console (#2954) 
## What this PR does

Adds an opt-in way to serve the Keycloak admin console and
Administration
REST API on a separate hostname and route from the public login
endpoints,
so the admin surface can be kept off the public internet.

- New `ingress.adminHost` in the keycloak chart. When set:
  - `KC_HOSTNAME_ADMIN` is configured so the admin console generates its
    URLs on that hostname;
  - the public (login) route is narrowed to the OIDC paths only
    (`/realms`, `/resources`, `/.well-known`), so the admin console and
    Administration REST API are no longer served on the login hostname;
- a dedicated route (Ingress or HTTPRoute, depending on
`gateway-enabled`)
    serves the admin hostname.
- New platform `publishing.ingressNameAdmin`, propagated as
  `_cluster.expose-ingress-admin`. The admin route attaches to the
Gateway/ingressClass it names, defaulting to `expose-ingress`. Pointing
it
at a separate (for example, private-IP) Gateway moves admin traffic off
the public one. The separate Gateway itself is provisioned out of band;
  this only selects where the admin route attaches.

Everything is empty by default, so existing deployments are unchanged.

Note: `KC_HOSTNAME_ADMIN` on its own is not a security boundary — the
Administration REST API stays reachable on any hostname routed to
Keycloak
(per the Keycloak docs). The path and network separation introduced here
is
what restricts access; full isolation requires attaching the admin route
to
a non-public Gateway.

Covered by helm-unittest suites for the keycloak chart and the platform
value wiring.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(keycloak): add opt-in `ingress.adminHost` to serve the admin console and Administration REST API on a separate hostname/route, plus `publishing.ingressNameAdmin` (`_cluster.expose-ingress-admin`) to attach that route to a separate (e.g. private) Gateway/ingressClass. Disabled by default; existing deployments are unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional configuration to serve the Keycloak admin console on a
dedicated hostname, with separate routing and ingress/gateway attachment
so admin UI/API can be isolated from public login/OIDC paths.
* Introduced a cluster-level setting to optionally expose admin routes
via a separate ingress/gateway class/namespace.

* **Tests**
* Added new YAML test coverage to validate defaulting, hostname split
behavior, and correct routing for both gateway-enabled and
gateway-disabled modes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-26 14:08:23 +05:00
Aleksei Sviridkin 67692b4544 feat(platform,ingress): publish host ingress via ServiceExposure 
Add the operator-facing knob that drives the new exposure model. When
publishing.exposureClass.name is set, the platform renders a cluster-scoped
ExposureClass binding the chosen backend (externalIPs, metallb, cilium,
robotlb) and VIP address scope, and carries the class name through the
cozystack-values Secret. The host ingress then renders a ServiceExposure
for its ingress-nginx Service, which the cozystack-controller translates
into the backend's pool and announcer.

The default (empty name) renders neither the ExposureClass nor the
ServiceExposure, so clusters on the externalIPs path are unchanged on
upgrade. The ingress Service shape is untouched: it is already published
as type LoadBalancer whenever externalIPs is empty.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-25 20:47:19 +03:00
Kirill Ilin caeb2c466f feat(platform): add publishing.ingressNameAdmin for a separate admin gateway 
Expose a new opt-in publishing.ingressNameAdmin that is propagated to
_cluster.expose-ingress-admin. System components (currently Keycloak)
use it to attach their admin routes to a separate Gateway/ingressClass,
e.g. one bound to a private IP and unreachable from the public internet.
Empty by default, so admin endpoints keep sharing expose-ingress. The
private Gateway itself is provisioned out of band; this only selects
where the admin route attaches.

Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
 2026-06-25 15:28:47 +05:00
Aleksei Sviridkin ee853b00bd fix(platform): order backupstrategy-controller after cozystack-basics 
backupstrategy-controller creates the cozy-backups Bucket into the
tenant-root namespace, which cozystack-basics provisions. On a fresh
install the Bucket patch can race namespace creation and fail with
`namespaces "tenant-root" not found`, which then races the install
wait deadline and fails the run — on changes that have nothing to do
with backup ordering. Add the cozystack-basics dependsOn edge so the
target namespace exists first, pinned with a helm-unittest regression
guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-24 22:34:09 +03:00
Aleksei Sviridkin 51c5273d0b feat(platform): ACME wildcard certificate on the default ingress-nginx path (#2988) 
## What this PR does

On the default ingress-nginx path (`gateway.enabled=false`) every
published hostname still minted its own per-host ACME certificate via
ingress-shim, even with a DNS-01 solver configured. That hits the Let's
Encrypt rate limit (50 certs per registered domain per week) once a
deployment exceeds ~50 endpoints. The Gateway API path already issues a
single per-apex wildcard in DNS-01 mode; this brings the ingress-nginx
path to parity. It implements the remaining ingress-nginx gap that #2400
was narrowed to in its triage comment (DNS-01 is already multi-provider,
and the Gateway path already issues wildcards).

The new `publishing.certificates.wildcard` toggle is opt-in and OFF by
default. When set to `true` with `solver=dns01` and no operator-provided
wildcard Secret, the platform issues one shared wildcard `Certificate`
for `<root-host>` + `*.<root-host>` and serves the resulting Secret as
the publishing controller's `--default-ssl-certificate`. The system
service Ingresses (dashboard, grafana, keycloak, harbor, …) then stop
requesting a per-host cert.

How it works:

- `core/platform` `apps.yaml` computes the effective wildcard Secret
name and a new `_cluster.wildcard-issue` signal. An operator-provided
`wildcardSecretName` (the BYO path) always wins and never triggers
issuance; HTTP-01 cannot issue wildcards; and when Gateway API is
enabled the TenantGateway controller issues the wildcard instead — so
all three are excluded from auto-issuance.
- `extra/ingress` renders the wildcard `Certificate` only on the
publishing controller (`Release.Namespace == expose-ingress`), because
the Secret must be same-namespace for ingress-nginx to read it. The
Certificate references the DNS-01 `ClusterIssuer` that
`cert-manager-issuers` already renders in `dns01` mode, so no per-tenant
Issuer is minted.

This reuses the existing wildcard-secret consumption path, so no
per-service Ingress template changes were needed — only the issuance
side is new.

Why opt-in (default off): the chosen wildcard Secret name rides the
`_cluster` channel, which every child tenant inherits verbatim, so this
is not root-tenant scoped. A single-label wildcard `*.<root-host>`
covers `<service>.<root-host>` but not a custom service host outside it
(a keycloak `ingress.host`, harbor `host`, or grafana `host` pointed at
another domain) nor a child tenant's nested host
(`<service>.<tenant>.<root-host>`). Enabling issuance makes every such
service drop its per-host ACME cert and fall back to the default
certificate, which does not cover it. Leaving it off by default means a
dns01 cluster is never silently switched on upgrade; an operator who
enables it accepts the same coverage responsibility as the
operator-provided wildcard path, which propagates identically. The
tradeoff is pinned by contract tests (keycloak custom host, bucket
nested child-tenant host). Per-tenant wildcards remain on the Gateway
path and are tracked separately.

Docs: cozystack/website#588

Closes #2400. Part of #2811.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(platform): add opt-in shared wildcard certificate issuance on the default ingress-nginx path via publishing.certificates.wildcard (default false). When enabled with a DNS-01 solver, the platform issues one *.<root-host> wildcard for system services instead of a per-host ACME certificate, avoiding Let's Encrypt rate limits at scale.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added automatic wildcard certificate issuance support for DNS-01 ACME
solver.
* New `certificates.wildcard` configuration option to enable shared
wildcard TLS certificates.
* Ingress rules now properly utilize wildcard certificates when enabled.

* **Tests**
* Added comprehensive test suites validating wildcard certificate
generation and ingress integration across multiple scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-23 13:13:48 +03:00
Aleksei Sviridkin 34563ce8c7 feat(platform): add opt-in shared wildcard issuance for ingress path 
On the default ingress-nginx path (gateway.enabled=false) a DNS-01 solver
still minted a per-host ACME certificate for every system service, hitting
Let's Encrypt rate limits once a deployment exceeds ~50 endpoints. Add an
opt-in publishing.certificates.wildcard toggle (default false) and compute,
in apps.yaml, an effective wildcard secret name plus a wildcard-issue
signal: when enabled with solver=dns01 and no operator-provided
wildcardSecretName, the platform targets one shared wildcard cert
(*.<root-host> + <root-host>) instead of per-host certs.

Issuance is off by default because a single-label wildcard does not cover
a custom service host outside *.<root-host> (e.g. a keycloak ingress.host,
harbor host, or grafana host pointed at another domain); enabling it would
otherwise switch such a host to the controller's default certificate and
serve the wrong certificate. The operator-provided (BYO) name always wins
and never triggers issuance; http01 cannot issue wildcards; and the
Gateway API path issues its own wildcard via the controller, so all three
are excluded. This reuses the existing wildcard-secret-name consumption
path, so no per-service ingress template changes are needed.

Part of #2811.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-22 20:01:40 +03:00
Aleksei Sviridkin e9aaee286e fix(mongodb-operator): apply CRD updates on upgrade via CreateReplace 
psmdb-operator evolves its CRD schema between minor versions; 1.22.0 adds
spec.storageScaling/storageAutoscaling, vault and other fields. Helm never
upgrades CRDs shipped in crds/, so without an explicit policy the operator
binary upgrades on an existing cluster while the live CRD stays behind and
the apiserver rejects the new fields. Opt the mongodb-operator PackageSource
into CreateReplace, consistent with etcd-operator, and add a platform test
pinning the policy.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-22 19:05:03 +03:00
Aleksei Sviridkin aced011ab7 feat(platform): add opt-in in-cluster east-west encryption toggle 
Add networking.encryption.enabled (off by default), a platform-level
opt-in that turns on in-cluster transport encryption for the active
datapath. On the default kubeovn-cilium datapath (and its generic
variant) KubeOVN owns the overlay, so the toggle maps to KubeOVN's
Geneve-overlay IPsec (kube-ovn ENABLE_OVN_IPSEC). KubeOVN's controller
provisions the IPsec CA and per-node certificates through the
Kubernetes CSR API, so no cert-manager wiring is required.

This is defense-in-depth for the whole mesh without per-application
changes; it complements, and does not replace, application-level /
end-to-end TLS. Values comments document the per-datapath scope
(including Kilo's cross-location-only mesh and the unreached
Cilium-native path on cilium/cilium-generic variants) and the
MTU/throughput overhead.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-22 18:16:15 +03:00
Aleksei Sviridkin 6e219c241b feat(platform,gateway): distribute operator wildcard secret name 
Add publishing.certificates.wildcardSecretName to the platform chart
and propagate it through the cozystack-values channel as
_cluster.wildcard-secret-name. Only the Secret NAME travels — the
cert/key material is never broadcast, unlike the public kube-root-ca
that already rides this channel.

When the key is set, the gateway chart switches the rendered
TenantGateway to certMode=existingSecret with a wildcardSecretRef
pointing at the operator-supplied Secret, and skips the ACME
solver/provider/issuer validations that no longer apply. Wildcard mode
takes precedence over solver=dns01. TLS-passthrough services and
attached namespaces continue to render unchanged.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-10 03:08:07 +03:00
Aleksei Sviridkin 1d747b3d8d feat(platform): auto-wire KubeVirt permittedHostDevices and HostDevices feature gate when gpu-operator is enabled (#2768) 
Closes #2765.

## What this PR does

Makes the passthrough and vGPU variants of `cozystack.gpu-operator`
produce a KubeVirt CR that is immediately usable, without an out-of-band
`kubectl patch`. Today the gpu-operator stack reports green,
`nvidia.com/<model>` shows up under node `.status.allocatable`, but
every `VirtualMachine` referencing the resource via
`domain.devices.hostDevices` is rejected by the admission webhook
because `HostDevices` is not in `developerConfiguration.featureGates`
and the resource is not listed under `permittedHostDevices`. That manual
reconciliation is now driven by the platform.

The change has three parts.

`packages/system/kubevirt` now parameterizes the KubeVirt CR template.
`developerConfiguration.featureGates` keeps its existing static defaults
and appends `.Values.extraFeatureGates`;
`spec.configuration.permittedHostDevices` renders verbatim from
`.Values.permittedHostDevices` when set;
`spec.configuration.mediatedDevicesConfiguration` does the same.
Defaults stay empty so the rendered CR is byte-identical to the previous
template when no overrides are supplied. The chart's `make update`
target re-applies the four sed transforms after fetching the upstream
CR, so refreshing the KubeVirt release version stays a single command.

`packages/core/platform` adds two defaults tables under `files/`.
`gpu-passthrough-defaults.yaml` covers Hopper (H100/H200), Ada Lovelace
(L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10), Turing (T4) and
Volta (V100/V100S); PCI vendor:device pairs are stable, `resourceName`
slugs follow the `<arch>_<model>_<form>_<mem>` convention
`nvidia-sandbox-device-plugin` generates as of v25.x, and
`externalResourceProvider: true` is set everywhere because those
resources are advertised by the sandbox plugin rather than KubeVirt's
in-tree device plugin. `gpu-vgpu-defaults.yaml` covers the Pascal–Ampere
mdev profile range (A100-40C/80C, A40-24Q/48Q, A30-24C, A10-24Q,
V100D-32C, T4-16Q) — the same family span the upstream
`vgpu-device-manager` walks `/sys/class/mdev_bus/` for. Ada Lovelace and
Blackwell SR-IOV vGPU is intentionally out of scope for the default
table and is expected to be advertised via the user-override hook
described below.

`templates/bundles/iaas.yaml` now selects between the two tables via a
new `bundles.iaas.gpuOperatorVariant` knob (`default` for passthrough,
`vgpu` for mediated; default is passthrough). The same variant flows
through to the gpu-operator Package CR, replacing the previous
always-default call. When the package is enabled, the platform injects
`HostDevices` into `extraFeatureGates` plus the chosen defaults into
`permittedHostDevices` for the kubevirt component's values. Operators
extend or replace via top-level platform values:

```yaml
gpu:
  replaceDefaults: false              # append (default) vs. wipe-and-replace
  permittedHostDevices:
    pciHostDevices: []                # cluster-specific PCI passthrough entries
    mediatedDevices: []               # cluster-specific vGPU entries
  mediatedDevicesConfiguration: {}    # extra mediated-device types / per-node mappings
```

`docs/gpu-vgpu.md` is updated to drop the manual `kubectl patch` step
and describe the auto-wiring contract, the per-variant defaults, the
escape hatches, and the sandbox-device-plugin `resourceName` slug
convention so an operator whose node publishes a different name knows
where to override.

## Test coverage

`helm-unittest` covers eight branches in
`packages/core/platform/tests/bundles_gpu_kubevirt_wiring_test.yaml`:
gpu-operator disabled emits no extras; gpu-operator enabled at default
variant emits the HostDevices gate plus `pciHostDevices`; vgpu variant
switches to `mediatedDevices` plus `mediatedDevicesConfiguration`; user
overrides append by default; `replaceDefaults: true` wipes the platform
table; an unknown variant fails fast with a descriptive error.
`packages/system/kubevirt/tests/kubevirt_cr_test.yaml` pins the
empty-values render and the three populated shapes at the CR template
level. `make helm-unit-tests` passes locally on the full repo.

### Release note

```release-note
feat(platform): the KubeVirt CR is now auto-configured for GPU host-device passthrough (HostDevices feature gate plus permittedHostDevices populated from NVIDIA defaults) whenever cozystack.gpu-operator is in bundles.enabledPackages. Operators no longer need a manual kubectl patch to make VMs schedule against a GPU. The variant is selected via bundles.iaas.gpuOperatorVariant (default: passthrough; vgpu: mediated). Extend or replace the NVIDIA defaults via .gpu.permittedHostDevices and .gpu.replaceDefaults.

**Upgrade note:** the bundle now OWNS spec.configuration.permittedHostDevices on the KubeVirt CR. If you previously hand-edited it (kubectl edit kubevirt), the first reconcile after upgrade overwrites your manual entries with the rendered NVIDIA default table. Before upgrading, move any custom entries into .gpu.permittedHostDevices (set .gpu.replaceDefaults: true to keep only your list) and verify each resourceName against what your nodes advertise (kubectl describe node <node> | grep nvidia.com/) — the default table now carries the exact slug the sandbox device plugin generates, which upper-cases the card's PCI-IDs name and keeps the Tesla/GL tokens (e.g. a Tesla T4 is nvidia.com/TU104GL_TESLA_T4), but a different plugin build can publish a different string; a mismatch makes the admission webhook reject GPU VMs on their next restart or migration.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Bundle-managed GPU Operator with selectable passthrough vs vGPU
variants; platform now injects GPU settings into KubeVirt and forwards
vGPU manager overrides when enabled.
* Default platform mappings for PCI passthrough and vGPU mediated-device
resources added.

* **Documentation**
* Expanded GPU/vGPU deployment guide with bundle-managed vs manual
guidance, wiring/precedence rules, resource-name conventions, and
verification steps.

* **Tests**
* New and expanded tests for bundle→KubeVirt wiring, variant behaviors,
default/override merging, and idempotent template parameterization.

* **Chores**
* Added Helm test target and hardened KubeVirt template
update/idempotency checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-10 02:02:01 +03:00
Aleksei Sviridkin 931f286b9f fix(ingress,platform): deliver publishing.proxyProtocol to host ingress-nginx (#2799) 
## What this PR does

`publishing.proxyProtocol: true` is documented to enable PROXY-protocol
on the host ingress-nginx, but the three nginx config keys
(`use-proxy-protocol`, `real-ip-header`, `enable-real-ip`) never reached
the controller. They were written into the `ingress-system` component
values of the `cozystack.ingress-application` Package, but that
component has no `install:` block, so the package controller never
builds a HelmRelease for it and the values are dropped. External traffic
behind a PROXY-protocol L4 load balancer kept breaking.

This routes the switch through `_cluster.proxy-protocol` in the
`cozystack-values` Secret — the same path `expose-external-ips` already
travels to reach the host ingress release — and the `extra/ingress`
chart applies it to the external-facing ingress (the namespace named by
`expose-ingress`). The dead component-values injection is removed.

The `extra/ingress` chart also gains a first-class `proxyProtocol`
value, so PROXY-protocol can be enabled per Ingress application (a
per-app boolean knob like `cloudflareProxy`).

Two `fail` guards keep the new knobs safe: the per-app `proxyProtocol`
value is rejected on the external-facing (host) ingress — that path must
go through `publishing.proxyProtocol`, which also installs ouroboros
(hairpin handling) and arms the disable-hazard guard — and
`cloudflareProxy` + `proxyProtocol` together are refused, since
Cloudflare's HTTP proxy does not speak PROXY-protocol to origins and its
`real_ip_header` server-snippet would override `real-ip-header:
proxy_protocol`.

`use-forwarded-headers` / `compute-full-forwarded-for` are deliberately
not set: without a paired `proxy-real-ip-cidr` they would let clients
spoof `X-Forwarded-For`, and the PROXY-protocol frame is already the
source of truth for the client address.

helm-unittest coverage pins runtime delivery on the host path, the
per-app path, and that non-host tenants stay untouched; the previous
test that only checked template emission into the (non-consumed) Package
component is replaced.

Affected versions: v1.4.0–v1.4.2 and current main (introduced in #2561).

**Upgrade note (action required).** Because the flag was a silent no-op
before this fix, a v1.4.x cluster that set `publishing.proxyProtocol:
true` aspirationally — without its L4 load balancer actually injecting
PROXY frames — kept working by accident. After this fix the flag really
enables `use-proxy-protocol`, which makes ingress-nginx require a PROXY
frame on every connection; if the LB does not send one, all external
ingress breaks. Before upgrading, confirm the upstream LB injects
PROXY-protocol v1 frames, or unset `publishing.proxyProtocol`.

Closes #2796

### Release note

```release-note
fix(ingress): publishing.proxyProtocol now enables PROXY-protocol on the host ingress-nginx (the config was silently dropped before reaching the controller); the Ingress application also gains a proxyProtocol value to enable PROXY-protocol per-app. ACTION REQUIRED on upgrade: the flag was previously a no-op, so a cluster that set it without the upstream LB injecting PROXY-protocol v1 frames will lose all external ingress once it takes effect — verify the LB, or unset publishing.proxyProtocol before upgrading.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Tenant ingress support for PROXY-protocol via a new proxyProtocol
option (default: false).
* Platform-level publishing.proxyProtocol now controls external/host
ingress behavior.

* **Bug Fixes / Validation**
* Rejects enabling per-app proxyProtocol on external-facing host
ingresses.
* Rejects using proxyProtocol together with cloudflareProxy to prevent
conflicting real‑IP handling.

* **Documentation**
* Updated docs describing proxyProtocol usage, defaults, and safety
requirements.

* **Tests**
* Added tests covering proxyProtocol wiring, config emission, and
failure cases.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-09 17:04:14 +03:00
Aleksei Sviridkin 479929ebfb ci(e2e): drop 3x retry on Run E2E + Install Cozystack, wait out gateway.bats tenant teardown (#2558) 
## What this PR does

Drops the 3× retry loop on `Run E2E tests` and `Install Cozystack into
sandbox`. `Prepare environment` keeps its 3× retry — that step is pure
infrastructure (Talos image download, sandbox VM boot, network) where
transient runner hiccups warrant a retry.

On failure, the test step now captures `kubectl get hr -A -o wide` and
`kubectl get events -A` under a collapsible group so triage starts with
the actual broken-state snapshot.

> [!NOTE]
> An earlier revision of this PR also doubled every bats timeout. That
commit was dropped in a rebase and is intentionally **not restored**:
the timeout class that actually matters (per-app HR-Ready waits) has
since been standardized at 5m on `main` (7b9f286fa), making a blanket 2×
redundant.

**Fixes gateway.bats teardown leakage.** The nested-tenant tests deleted
tenants fire-and-forget, parent and child back-to-back. The leftover
uninstalls (each blocked on a cleanup Job, parents wedged on
still-terminating child namespaces) plus one mid-install child HR
occupied exactly 5 workers on the `--concurrent=5` tenants
helm-controller shard, starving whichever app test ran next — observed
as the harbor HR sitting unreconciled for its whole 5m HR-Ready budget
in [run
27020081550](https://github.com/cozystack/cozystack/actions/runs/27020081550),
surfaced by this PR's own retry removal + diagnostics dump. Teardown now
deletes child→parent with hard `wait hr --for=delete` between, so a
wedged tenant uninstall fails gateway.bats itself, not an innocent
neighbor.

## Why

Audit of 30 successful PR runs found that across 5 sampled failure
attempts, **25/25 retries** on `Run E2E tests` failed — the retry loop
never recovered a flake, only stretched deterministic failures and
tripled diagnostic wall-time. Same data shape on `Install Cozystack`.

Beyond wasted CI time, the retry was hiding ~10 deterministic bugs (Helm
namespace-ownership conflict, seaweedfs HR timeout, harbor BucketInfo
wiring, vminstance disk race, etc.). Each failure looked like a "flake"
because the retry sometimes coincided with whatever transient state had
cleared — the retry never fixed the bug, just delayed surfacing.

## Dependencies

The deterministic bugs the retry was masking are now fixed on `main`:
- ✅ **#2508** — installer namespace bootstrap (Helm namespace-ownership
conflict on cold install) — merged
- ✅ **#2509** — operator HelmRelease config knobs (`seaweedfs-system`
2-min wait race within Flux's 5-min reconcile windows) — merged
- ✅ **#2528** — harbor bucket-secret + BucketInfo gating (harbor
`ValuesError` on first install) — merged
- ✅ **#2529** — objectstorage-controller BucketAccess conflict retry —
merged

Companion PRs in the #2619 split (independent of this PR,
ordering-wise):
- **#2602** — Flux v2.8.0 + chart fixes
- **#2601** — seaweedfs-system split

This PR does NOT depend on #2602/#2601 — it now touches only the
workflow file and gateway.bats teardown, both on top of fresh `main`.

Surfaced from #2500.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* CI prepare-environment step now reports plain attempt counts with
clear success/failure messages.
* Install and per-app test steps no longer retry; each runs once and
fails immediately on error. Failed apps log diagnostics and job proceeds
to remaining apps while overall job fails.
* **Tests**
* End-to-end tests and install/prepare flows use longer, more tolerant
timeouts and added existence polling to reduce flakiness and improve
diagnostics.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2558?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-09 16:21:15 +03:00
Aleksei Sviridkin c32812ccf9 fix(platform): deliver publishing.proxyProtocol to host ingress-nginx 
The host PROXY-protocol switch wrote the nginx config keys into the
ingress-system component values of the cozystack.ingress-application
Package. That component has no install block, so the package controller
never built a HelmRelease for it and the values were dropped — the host
ingress-nginx never received use-proxy-protocol and external traffic
behind a PROXY-protocol L4 load balancer kept breaking.

Route the switch through _cluster.proxy-protocol in the cozystack-values
Secret instead, the same path expose-external-ips travels to reach the
host ingress release, and drop the dead component-values injection.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-09 12:50:53 +03:00
Aleksei Sviridkin 117d317fd8 fix(platform): drop fragile numeric vGPU mediatedDeviceTypes defaults 
The vgpu defaults shipped a mediatedDevicesConfiguration block whose
mediatedDeviceTypes were hardcoded nvidia-NNN ids. Those ids are not
stable identifiers — they are the per-(SKU, driver) indices under which a
host enumerates mdev types in /sys/.../mdev_supported_types/, so the same
profile gets a different number on a different card or driver branch.
Auto-injecting them made KubeVirt try to create mdev types that usually
do not exist on the operator's nodes, and duplicated work the GPU
Operator's vGPU Device Manager already does.

Delegate mdev creation to the vGPU Device Manager, which is name-based and
stable, and keep only the name-based permittedHostDevices.mediatedDevices
exposure list (mdevNameSelector) in the defaults. KubeVirt-driven creation
stays available through the .gpu.mediatedDevicesConfiguration override for
operators who supply host-verified type ids; document the lookup recipe.

Update the bundle wiring, the gpu/vgpu values comments, the vGPU guide and
the tests accordingly (the default vgpu path now emits no
mediatedDevicesConfiguration; the override path is covered explicitly).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-08 18:14:59 +03:00
Aleksei Sviridkin 919073601d fix(platform): match NVIDIA passthrough defaults to the resourceName the plugin advertises 
The auto-injected permittedHostDevices only works if each resourceName
equals what nvidia-sandbox-device-plugin advertises on the node; otherwise
the admission webhook rejects the VM — the very manual step this bundle
removes. Several defaults did not match the slug the plugin
(nvcr.io/nvidia/kubevirt-gpu-device-plugin) derives from the PCI-IDs
database name, which uppercases the name and keeps the die suffix, the
Tesla brand and the form-factor/memory tokens:

  10DE:1EB8  TU104GL_T4          -> TU104GL_TESLA_T4
  10DE:20B7  GA100_A30           -> GA100GL_A30_PCIE
  10DE:2331  GH100_H100_PCIE_80GB-> GH100_H100_PCIE
  10DE:2321  GH100_H100_NVL_94GB -> GH100_H100L_94GB
  10DE:1DB1/1DB4/1DF6 + new 1DB5 -> GV100GL_TESLA_V100_* family

Device 10DE:1DB1 was also mislabelled: it is the SXM2 16GB part, not the
32GB one — the 32GB SXM2 V100 is a distinct device (10DE:1DB5) that was
missing entirely. Relabel 1DB1 as 16GB and add 1DB5.

Restate the header comment as the actual generation rule and pin the
corrected slugs (T4, A30, both V100 SXM2 parts) with a regression test
that also asserts the tidy/short forms never reappear.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-08 18:06:31 +03:00
Aleksei Sviridkin da0dcf9edf test(platform): pin nil-safe GPU bundle wiring against null gpu values 
The .Values.gpu accesses in the iaas bundle are fully parenthesized
((( .Values.gpu ).x).y) so the field walk stays nil-safe when an operator
nulls the whole gpu block or leaves a sub-key blank (an empty YAML key
parses to null, which Helm then drops from .Values). A bare dotted walk
would panic with "nil pointer evaluating interface {}.<field>" on the
first nil hop.

Add regression cases covering gpu: null in both passthrough and vgpu
variants plus blank sub-keys, and a comment warning against flattening
the parenthesized form. A mutation check confirms the cases fail with the
nil-pointer panic the moment the parentheses are removed.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-08 17:08:21 +03:00
Aleksei Sviridkin 91efa1f4eb feat(platform): auto-wire KubeVirt host-device support when gpu-operator is enabled 
When cozystack.gpu-operator is in bundles.enabledPackages, the platform
now mirrors the chosen GPU variant into the KubeVirt CR — the HostDevices
feature gate is appended to developerConfiguration.featureGates, and a
starter permittedHostDevices block is filled from a per-variant defaults
table. Without this, a passthrough-variant GPU shows up under
.status.allocatable on the node but the KubeVirt admission webhook
rejects every VM that references it as a hostDevices.deviceName.

Two defaults tables ship under packages/core/platform/files/:

  - gpu-passthrough-defaults.yaml covers Hopper (H100/H200), Ada
    Lovelace (L4/L40/L40S), Ampere (A100 PCIe/SXM, A40, A30, A10),
    Turing (T4), Volta (V100/V100S). PCI vendor:device pairs are
    stable; resourceName slugs mirror nvidia-sandbox-device-plugin's
    <arch>_<model>_<form>_<mem> convention. externalResourceProvider
    is set on every entry because the resources are advertised by
    the sandbox-device-plugin DaemonSet, not by KubeVirt itself.

  - gpu-vgpu-defaults.yaml covers the Pascal-Ampere mdev profile
    range the upstream vgpu-device-manager walks /sys/class/mdev_bus/
    for. Ada Lovelace and Blackwell SR-IOV vGPU is out of scope for
    the chart's default list (NVIDIA driver path differs) and is
    expected to be advertised via the user-override hook below.

bundles.iaas.gpuOperatorVariant picks which table applies (default =
passthrough, vgpu = mediated). The same variant flows through to the
gpu-operator Package CR, replacing the previous always-default call.

Operators extend or replace the defaults via .gpu in platform values:

  gpu:
    replaceDefaults: false
    permittedHostDevices:
      pciHostDevices: [...]
      mediatedDevices: [...]
    mediatedDevicesConfiguration: {...}

replaceDefaults: false (the default) appends the user list to the
NVIDIA table; true drops the table entirely for clusters with only
non-NVIDIA accelerators or a strict allowlist requirement.

Companion helm-unittest suite covers the eight branches: no gpu
package emits no extras; gpu-operator at default variant emits the
HostDevices gate plus pciHostDevices; vgpu variant emits the
HostDevices gate plus mediatedDevices and mediatedDevicesConfiguration;
user overrides append by default; replaceDefaults wipes; unknown
variant fails fast.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-08 17:01:51 +03:00
Aleksei Sviridkin 80fbfa1ed8 fix(networking): point host ouroboros proxy at the root-tenant ingress (#2800) 
## What this PR does

With `publishing.proxyProtocol: true`, the host emits a
`cozystack.ouroboros` Package using the no-override Package macro, so
the proxy inherits the wrapper chart default backend
`ingress-nginx-controller.cozy-ingress-nginx`. That layout describes a
managed Kubernetes tenant cluster, not the host: on the host,
`extra/ingress` deploys ingress-nginx into the root tenant namespace as
`root-ingress-controller` in `tenant-root`. The composed backend FQDN
never resolves, so `ouroboros-proxy` never reaches Ready.

This emits the host ouroboros Package with a `proxy.target` override
derived from `publishing.ingressName` — namespace plus
`<short>-ingress-controller` service name — mirroring the `trimPrefix
"tenant-"` transform `extra/ingress` applies to its release namespace.
The derivation reuses the existing platform invariant that
`publishing.ingressName` is the namespace where the host ingress
controller runs.

Adds bundle helm-unittest coverage for the default ingress name and a
non-default one, and corrects the misleading wrapper `values.yaml`
comment that claimed the host uses `cozy-ingress-nginx`.

Scope: this fixes the ouroboros readiness defect only. The separate
defect where the injected nginx PROXY-protocol config keys are not
consumed by the host ingress is tracked independently and not touched
here.

Closes #2797

### Release note

```release-note
fix(networking): point the host ouroboros PROXY-protocol proxy at the root-tenant ingress so it becomes Ready when publishing.proxyProtocol is enabled
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Enhanced proxy protocol configuration to automatically route through
the appropriate ingress-nginx service based on your ingress name
settings.

* **Tests**
* Added test coverage for proxy protocol bundle wiring and ingress
service routing.

* **Documentation**
* Clarified proxy target configuration behavior and fallback mechanisms.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-08 16:41:15 +03:00
Myasnikov Daniil 56014f1fdc fix(platform): order cozystack-basics after the APIs its admission policies type-check 
cozystack-basics ships cluster-wide ValidatingAdmissionPolicies matching
tenants.apps.cozystack.io and the Gateway API types. KCM's VAP status
type-checker panics on a nil schema when it type-checks a policy against a
matched type that isn't resolvable yet; during a cold install this crashes
KCM, which stalls cert/token issuance, so cozystack-api never starts and the
tenants schema never resolves — a bootstrap deadlock.

Add dependsOn edges so the package installs only after cozystack-engine
(cozystack-api) and gateway-api-crds are Ready. Drift detection is off on
operator-generated HelmReleases, so a template capability gate would drop the
policy at first install and never re-add it; ordering is the reliable fix.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 03f1e7a55a)
 2026-06-08 15:15:32 +05:00
Myasnikov Daniil 2e23198b44 fix(platform): install velero by default 
backupstrategy-controller is installed by default since 494144fb9 and
its PackageSource declares a hard dependsOn on cozystack.velero
(f17ead3d2). With velero still optional, any bundle that does not
explicitly enable it leaves the backupstrategy-controller Package
permanently DependenciesNotReady, so the cozystack-platform HelmRelease
never becomes Ready. Since 67b1a960d made the e2e install test gate on
HelmRelease readiness, this deterministically fails CI on every PR.

Promote cozystack.velero from optional to default in the system bundle
so the dependency is always satisfiable. The bundles.disabledPackages
opt-out keeps working; drop the now-redundant velero entry from the
isp-full-generic enabledPackages list.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-05 15:35:41 +05:00
Aleksei Sviridkin f90411da7a fix(networking): point host ouroboros proxy at the root-tenant ingress 
With publishing.proxyProtocol enabled the host emits a cozystack.ouroboros
Package via the no-override Package macro, so the proxy inherits the wrapper
chart default backend ingress-nginx-controller.cozy-ingress-nginx. That
layout describes a managed Kubernetes tenant cluster; on the host,
extra/ingress deploys ingress-nginx into the root tenant namespace as
root-ingress-controller in tenant-root. The composed backend FQDN never
resolves, so ouroboros-proxy never reaches Ready.

Emit the host ouroboros Package with a proxy.target override derived from
publishing.ingressName (namespace plus <short>-ingress-controller service
name), mirroring the trimPrefix transform extra/ingress applies. Add bundle
helm-unittest coverage for the default and a non-default ingress name, and
correct the misleading wrapper values.yaml comment.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-03 11:57:01 +03:00
Myasnikov Daniil 91aae36da3 feat(platform): NOTES.txt warns when no-delete VAP is silently skipped 
Address review feedback from @lexfrei on packages/core/platform/templates/
deletion-protection.yaml: on a pre-1.30 cluster the capability gate
(`.Capabilities.APIVersions.Has "admissionregistration.k8s.io/v1/
ValidatingAdmissionPolicy"`) evaluates to false and the policy + Binding
render to nothing. There was no signal in `helm install`/`helm upgrade`
output that the guardrail had been silently skipped, so operators on
older clusters lost the protection without any way to discover that from
the install log. Compounded the integration-test gap @lllamnyp called
out in blocker 3.

Add packages/core/platform/templates/NOTES.txt that renders a multi-line
WARNING whenever the capability is absent: states the guardrail is NOT
active, explains why (1.30+ required), notes that the label is still
stamped on every protected object so the guardrail becomes active
automatically on the next upgrade after a cluster bump, no migration
step required.

Add packages/core/platform/tests/notes_capability_gate_test.yaml: positive
case asserts the warning text + the 1.30 hint + the label string render
on a chart-level `capabilities.apiVersions: []`. Negative case asserts
the warning is absent when admissionregistration.k8s.io/v1/
ValidatingAdmissionPolicy is in the capability list. Verified the
positive case fails when the warning text is removed from NOTES.txt.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-26 15:12:53 +05:00
Myasnikov Daniil e34eeb0ee7 fix(platform): document --namespace bypass and add VAP unit tests 
Address review feedback from @lllamnyp on packages/core/platform/templates/
deletion-protection.yaml:

The bypass instructions in both the template-author-facing comment and the
user-facing CEL message did `kubectl label <kind> <name>` with no
--namespace flag. Several protected objects are namespaced (cozystack-version
ConfigMap, cozystack-packages Repository, tenant-root HelmRelease), so an
operator who copies the documented command verbatim against their default
context gets a "not found" error and concludes that the bypass mechanism is
broken. Updated both call sites to show --namespace <ns> with an explicit
"(omit for cluster-scoped resources)" caveat.

The reviewer also flagged that the VAP contract is otherwise unasserted by
unit tests. Add packages/core/platform/tests/deletion_protection_test.yaml
covering:
  - both the VAP and the Binding carry platform.cozystack.io/no-delete=true
    on themselves (the self-protection contract)
  - failurePolicy is intentionally absent on the VAP
  - the deny message contains the bypass command WITH --namespace and the
    cluster-scoped caveat (regression catch for this fix; verified failing
    when --namespace is dropped from the template)
  - the Binding's objectSelector + validationActions: [Deny] still match
    the no-delete label

Wire packages/core/platform/Makefile's test target to helm unittest so
hack/helm-unit-tests.sh actually runs the new suite in CI (the existing
tests/ files were orphaned without it).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-26 15:12:53 +05:00
Aleksei Sviridkin d2e851f63b feat: Gateway API support via Cilium 
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.

The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.

publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.

Security model -- seven independent layers, all admission-time gates
fail-closed:

  1. Listener allowedRoutes namespace whitelist via the
     kube-apiserver-written kubernetes.io/metadata.name label;
     plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
     passthrough listeners restrict allowedRoutes.kinds to
     HTTPRoute / TLSRoute respectively.
  2. cozystack-gateway-hostname-policy VAP on Gateway.
  3. cozystack-gateway-attached-namespaces-policy VAP on Package --
     blocks tenant-* in gateway.attachedNamespaces.
  4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
     writes from non-trusted callers.
  5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
     writes to namespace.cozystack.io/host from non-trusted callers.
  6. Render-time fail in cozystack-basics for tenant-* in
     gateway.attachedNamespaces.
  7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
     scoped to tenant-* namespaces.

Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).

Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.

Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.

Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.

Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.

Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.

Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-25 17:07:02 +03:00
Aleksei Sviridkin 7f01c7eb32 feat(platform): single switch for host PROXY-protocol + ouroboros 
Adds publishing.proxyProtocol on the platform values: a single flag
that both turns PROXY-protocol on at the host ingress-nginx (via the
ingress-application Package CR's component values dict) and emits the
new cozystack.ouroboros Package CR for the host install. The two
pieces are coupled by design — turning PROXY-protocol on at
ingress-nginx without ouroboros breaks intra-cluster traffic to the
cluster's own public hostnames (kube-proxy / Cilium-KPR short-circuits
the LB IP, the connection arrives at ingress-nginx without the PROXY
header it now requires, and ingress-nginx closes the connection).

The new sources/ouroboros.yaml PackageSource declares the host chart
with dependsOn: cozystack.networking + cozystack.ingress-application
so install ordering is deterministic. The bundle template emits the
host ouroboros Package via the standard helper — the parent chart's
Gateway-API watching default of false stays in place, because cozystack
does not install Gateway-API CRDs cluster-wide on the host (system/
gateway-api-crds is referenced only from the kubernetes-application
PackageSource, tenant scope). Forcing --gateway-api on a host without
the CRDs would CrashLoop the controller on WaitForCacheSync against
missing GVRs.

The flag default is false. Clusters that do not run PROXY-protocol get
zero new resources, zero RBAC delta, zero behaviour change.

Disabling: flipping publishing.proxyProtocol back to false stops
emitting the cozystack.ouroboros Package CR, but every Package CR
carries helm.sh/resource-policy: keep — Helm leaves the existing
Package on the cluster so ouroboros stays installed and the live
Corefile rewrite block keeps pointing at the still-running
ouroboros-proxy Service. Flag-flip alone is idempotent and safe. To
actually uninstall ouroboros the operator removes the Package CR
explicitly (or adds the package name to bundles.disabledPackages) and
runs the documented Corefile cleanup recipe in the same window —
otherwise stale rewrites point at a deleted Service and in-cluster
DNS for hairpinned hostnames returns connection-refused. The values
comment ships the cleanup recipe (a JSON merge-patch on data.Corefile
that strips the BEGIN/END block while preserving labels, annotations,
and other data keys including any Talos-managed metadata) and
documents the upstream-LB precondition (the L4 LB in front of
ingress-nginx must already be injecting PROXY-v1 headers before the
flag flips on), the host-vs-tenant fan-out (the flag does not cascade
into tenant clusters; tenants opt in per-Kubernetes-CR via
addons.ouroboros.enabled), and the Talos re-render flap window
(machine-config pushes wipe the BEGIN/END markers; the controller
re-applies on next reconcile, ≤10 minutes by default).

Also drops the stale "distro-full" entry from the bundles.system.variant
options comment in core/platform/values.yaml — the variant was removed
in v1.0.1 and was never restored.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-06 06:48:07 +03:00