admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-09 05:25:14 +09:00
Code Issues Packages Projects Releases Wiki Activity
6,613 Commits 569 Branches 294 Tags
11bac54f931682dd94de4081bb9e8a7eef60bef1
Commit Graph
113 Commits
Author SHA1 Message Date
Aleksei Sviridkin 08bbed3fb5 feat(talos-log-collector): add talos-log-collector package (#3260) 
## What this PR does

Adds `talos-log-collector`, an optional system package: a node-local
Vector
DaemonSet that receives Talos machine and kernel logs and forwards them
to the
tenant VictoriaLogs (vlinsert), alongside the container/audit/event logs
already collected by monitoring-agents.

Talos keeps system and kernel (kmsg) logs inside the OS and does not
write them
to a host file the Fluent Bit `tail` input could read, so these logs are
currently never collected. Vector receives them over the node loopback
(Talos pushes to 127.0.0.1:5170). Talos exposes the two streams through
independent config paths, and both must point at the loopback socket:
`machine.logging.destinations` forwards the JSON-lines service logs, and
a
`KmsgLogConfig` document forwards the kernel (kmsg) stream (including
DRBD
kmsg). The node-side setup for both is in the README.

Design notes:
- The pod stays on the pod network so cluster DNS resolves vlinsert with
no
  extra config. Talos pushes to 127.0.0.1:5170; a `hostPort` bound to
`hostIP: 127.0.0.1` publishes that loopback socket into the pod via
portmap,
  keeping the receiver off the host network (no node-wide pod exposure).
- The receiver binds inside the pod netns, so a NetworkPolicy is
load-bearing:
a CiliumNetworkPolicy restricts ingress to the host entity when Cilium
is
present, otherwise a default-deny-ingress `networking.k8s.io/v1`
fallback
  keeps the socket protected on non-Cilium variants.
- Runs non-root, read-only root filesystem, all capabilities dropped.
- Optional package (opt-in via `bundles.enabledPackages`): inert until
each
Talos node is configured to push logs. Node-side setup is in the README.

Verified end-to-end on a Talos v1.12 dev cluster: service and kernel
logs
(including DRBD kmsg) reach VictoriaLogs; pods run non-root with 0
restarts.

### Release note

```release-note
feat(talos-log-collector): add talos-log-collector package to collect Talos machine and kernel logs into VictoriaLogs
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an optional node-local Talos log collector (via system bundle
packages) that forwards Talos service and kernel logs over loopback to
the platform log ingestion endpoint.
* Introduced the collector Helm chart (Vector-based) with configurable
log level, loopback TCP receive port, digest-pinned image, default
resources, and routing target.
* Added conditional Cilium network policy support with a safe
deny-by-default fallback.
* **Security**
* Hardened the collector to run non-root with reduced privileges,
read-only filesystem, and dropped Linux capabilities (and disabled
service account token automount).
* **Documentation**
* Documented configuration requirements (including kernel log
enablement), operational behavior, and teardown steps.
* **Build & Release**
  * Extended the build to include building the collector’s image.
* **Tests**
* Added Helm rendering/contract tests covering daemonset, configmap,
service account, and network policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-22 23:49:33 +04:00
Andrei Kvapil 06bd17621f chore(platform): stop shipping the http-cache package 
Drops the cozystack.http-cache-application PackageSource and its entry
in the naas bundle.

No migration accompanies this. Platform Package objects carry
helm.sh/resource-policy: keep, so an existing cluster keeps its
http-cache Package and its http-cache-rd HelmRelease, and with them the
ApplicationDefinition that makes the kind resolvable. Instances that are
already deployed therefore keep running and stay addressable; the
platform simply stops delivering and updating the application. Removing
those objects would be the more invasive choice, not the safer one,
because there is no successor in the catalog to migrate an instance onto.

The httpcaches grant in cozy:tenant:admin:base stays for now. It is the
only rule in the tenant admin chain that can delete an instance, and no
tenant role carries any verb on helm.toolkit.fluxcd.io, so without it
the owner of a surviving HTTPCache could not remove it at all.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
 2026-09-14 13:11:00 +02:00
scooby87 afc003c464 feat(postgres): database read-replica autoscaling via KEDA (#3954) 
## What this PR does

Adds automatic horizontal autoscaling of a PostgreSQL database's **read
replicas** in response to load, implementing the merged design proposal
[Database Horizontal
Autoscaler](https://github.com/cozystack/community/tree/main/design-proposals/database-horizontal-autoscaling)
(community#44). The mechanism is entirely stock — no bespoke operator,
no new CRD:

- **KEDA as an optional platform component** (`packages/system/keda`,
wired into `core/platform` as a source + system-bundle entry). It claims
the `external.metrics.k8s.io` APIService that nothing served before.
- **`cozy-lib.keda.scaledObject` helper**
(`packages/library/cozy-lib/templates/_keda.tpl`) renders a KEDA
`ScaledObject`, fail-closed on missing/invalid arguments.
- **`packages/apps/postgres` autoscaling** — an `autoscaling` values
block (`enabled`, `minReplicas`, `maxReplicas`, `metric`, `target`,
`maxReplicationLagSeconds`, `dryRun`, `transition`) validated by
`values.schema.json`; under active autoscaling the chart renders a
`ScaledObject` and seeds `spec.instances` to a constant floor
(`max(replicas, effectiveMin)`), while the postgres release is forced to
client-side (two-way) Helm apply so the seed is a merge no-op on upgrade
and the KEDA-managed HPA — the runtime writer via the CNPG `Cluster`
scale subresource — is not reverted.
- **Single-value metric (§1)** — one PromQL query returns `Σ(read load
over standbys) + target`, so a stock HPA computes `desired =
ceil((Σ+target)/target) = 1 + ceil(Σ/target) = primary + desiredRead`,
with no per-pod emission.
- **The four static brakes (§5)** — quorum floor `max(minReplicas,
maxSyncReplicas+1, 2)` (quorum wins over `maxReplicas`), scale-down
pacing (`Pods/1/600s`), a replication-lag clamp folded into the query
(freeze both directions while lag exceeds the threshold and the primary
is writing, with `max_over_time` hysteresis), and dry-run (render
paused).
- **Two-phase enable/disable migration** via a `transition` sub-flag:
phase 1 stages `spec.instances` to the live count and brings KEDA up
with its HPA paused in both directions; flipping to phase 2 hands the
runtime count to KEDA, and because the seed re-renders the same count
the flip is a merge no-op with no dip.
- **Observability** — a database-autoscaling Grafana dashboard and
PrometheusRule alerts.

**Live validation (dev cluster, CNPG 1.30.0).** The full path works
end-to-end: the CNPG `Cluster` scale subresource serves
`status.selector`, KEDA renders the HPA, the metric reaches it
(`ScalingActive=ValidMetricFound`, `TARGETS 151/150`, `desiredReplicas=2
= 1 primary + 1 replica` — the §1 arithmetic), and the lag-clamp query
returns the base value on live VictoriaMetrics. Two packaging fixes were
needed and are included: KEDA must run on the cozystack `cozy.local`
cluster domain (the upstream `cluster.local` default makes the metrics
gRPC target NXDOMAIN and every HPA hangs on `FailedGetExternalMetric`),
and the operator's client-go throttle is raised so a single-replica
operator does not lose its leader lease under scale-subresource polling.

Implements #3953.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. Two repositories are affected; both are follow-ups to open once
this lands (and after its dependency #3951 merges and the values surface
is final), left unticked here for a maintainer to confirm rather than
claimed with a speculative link:

- **cozystack/terraform-provider-cozystack** — this adds the nested
`autoscaling` block to `packages/apps/postgres/values.schema.json`. The
provider hand-models each app's fields (schema + model +
expand/flatten), so it needs the new block to expose it in Terraform.
- **cozystack/website** — KEDA is a new platform component
(`guides/platform-stack`), and the postgres reference page gains the
`autoscaling` params. The managed-app reference is regenerated from
`README.md` by the release bot; the platform-stack component list is
hand-maintained.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: to open (new platform component + postgres autoscaling
params)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: to open (new `autoscaling` block in postgres
values.schema.json)

### Depends on

- #3951 — bump CloudNativePG to 1.30.0. The HPA can only drive the CNPG
`Cluster` scale subresource on CNPG ≥ 1.28.4, where it serves
`status.selector`; without it the HPA fails with `InvalidSelector`. This
PR should not merge before #3951.

### Release note

```release-note
feat(postgres): add optional horizontal autoscaling of read replicas via KEDA, configured through the new `autoscaling` values block; KEDA ships as a new optional platform component. Requires CloudNativePG >= 1.28.4 for the Cluster scale subresource selector. Note: every Postgres release now applies client-side (`release.cozystack.io/helm-server-side-apply: "false"`) so the autoscaler's live `/scale` count is not reverted by Helm; a kind-wide side effect is that chart-rendered fields on Postgres releases are no longer re-asserted by server-side-apply drift correction (driftDetection is unset), a documented trade-off with a per-instance apply strategy as follow-up.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * Added configurable PostgreSQL read-replica autoscaling with KEDA.
* Added CPU and read-connection metrics, replica bounds, replication-lag
safeguards, dry-run mode, and transition handling.
* Added optional KEDA platform support, Grafana autoscaling dashboard,
and Prometheus alerts.

* **Documentation**
* Clarified integer thresholds, prerequisites, configuration, and
transition guidance.

* **Bug Fixes**
* Improved validation and actionable errors for unavailable KEDA or
invalid settings.

* **Tests**
* Expanded coverage for autoscaling, observability, package selection,
and validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-09-09 21:28:34 +03:00
Aleksei Sviridkin a3ddba441f fix(opensearch-operator): set vm.max_map_count from a node DaemonSet 
OpenSearch refuses to start when vm.max_map_count is below 262144 and
the store may use mmap. The kernel default is 65530 and nothing in the
platform's node configuration raises it: the Talos profiles declare no
sysctls, and neither does any documented node-preparation path.

The value cannot come from the OpenSearch pod. The kernel does not
namespace vm.*, so kubelet refuses it in a pod's securityContext.sysctls
and in --allowed-unsafe-sysctls alike, both of which admit namespaced
sysctls only. The operator's own answer, spec.general.setVMMaxMapCount,
appends a privileged init container to every OpenSearch pod. Tenant
namespaces carry no PodSecurity label of their own and take the cluster
default, which Talos ships as baseline through the apiserver's
PodSecurityConfiguration; that default rejects the container, so the
StatefulSet is created and never produces a pod.

Set the value once per node instead, from a DaemonSet in the operator's
own namespace. The init container raises the limit to a floor rather
than assigning it: the tunable is node-global, so a node already tuned
higher for another mmap-heavy workload keeps its own value. The pod runs
at system-node-critical: it is a precondition for every OpenSearch pod on
the node, so it must not lose a scheduling race to the workload it
enables, though priority is not start ordering: a pod that starts first
just fails its own bootstrap check and retries. The idle container runs
the sleep under a shell with a TERM handler, because PID 1 of a
namespace receives only the signals it installed a handler for and a
bare sleep would be SIGKILLed after the full grace period on every
rollout and drain. The write
goes to /proc/sys only, with no hostPath and no edit to the node's
on-disk configuration; the pod restarts with the node and reapplies the
value. The component is declared privileged so the platform labels that
namespace enforce=privileged, which is what this DaemonSet needs and
what tenant namespaces deliberately do not get.

The file returns to a path bb660b57c emptied. That earlier revision
bind-mounted the host's /etc to append to sysctl.conf and ran an
unpinned image; this one writes no host file and pins the image by
digest.

This removes the reason for an OpenSearch pod to ask the operator for
the sysctl, but the app chart still asks. Dropping that request is a
separate change and is what stops the pods being rejected.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-09-08 00:35:06 +03:00
Alexey Artamonov 7f8f2c1d86 feat(keda): add KEDA as an optional platform component 
Vendor the upstream KEDA chart (kedacore/keda 2.20.2) as a new optional
system package and wire it into the platform, so database read-replica
autoscaling (which renders KEDA ScaledObjects) has a backend.

- packages/system/keda: umbrella cozy-keda + vendored chart (Makefile
  pins KEDA_CHART_VERSION with a lockstep assert), shim values enabling
  the operator / metrics-server / webhook ServiceMonitors under the
  platform Prometheus.
- core/platform: PackageSource cozystack.keda (namespace cozy-keda,
  dependsOn networking + prometheus-operator-crds; cert-manager is not
  needed, KEDA self-generates its webhook TLS) plus an optional bundle
  entry, opt-in via bundles.enabledPackages. It is not a dependsOn of any
  default package, so the platform HelmRelease never blocks on it.
- KEDA claims the cluster-wide external.metrics.k8s.io APIService
  (distinct from metrics-server's metrics.k8s.io — no conflict).

Covered by helm unittest: platform bundle enable/disable/opt-out, the
PackageSource dependsOn, and the vendored chart's external-metrics
APIService + shim-enabled ServiceMonitor.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
 2026-09-04 23:18:47 +03:00
Andrei Kvapil 53412c3033 fix(valkey): add cozystack-engine dependsOn edge to the PackageSource 
The valkey-application PackageSource ships system/valkey-rd (an
ApplicationDefinition); it must dependOn cozystack.cozystack-engine so the
ApplicationDefinition CRD is registered before the -rd HelmRelease reconciles.
Mirrors redis-application.yaml and satisfies check-applicationdefinition-crd-ordering.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-09-01 11:44:29 +04:00
Andrei Kvapil 4e6313b608 feat(valkey): register Valkey application in the platform 
Wire the managed Valkey app into the platform: add the valkey-rd package
carrying the Valkey ApplicationDefinition, the valkey-application PackageSource
(reusing the existing redis-operator), enable it in the paas bundle, and grant
tenant roles access to the valkeys resource.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-09-01 11:43:10 +04:00
Timofei Larkin f5b957ee27 feat(kubernetes)!: manage worker pools as KubernetesNodes, adopt on upgrade (Phase 2, breaking) (#3315) 
## What this PR does

Phase 2 of the Kubernetes-app split
([cozystack/community#8](https://github.com/cozystack/community/pull/8))
— the **breaking** half. Stacked on #3314 (review that first).

Removes worker node pools from the `kubernetes` chart (they moved to
`kubernetes-nodes` in #3314) and adopts existing pools on upgrade:

- `kubernetes` chart is now **control-plane only**: drops the pool
`range` block, the `talos-reconcile` Job + RBAC (deleted; moved to
`kubernetes-nodes`), the preserve-old-KMT block, the
`kubernetes.nodeGroups` helper, and the per-group dashboard RBAC.
- **`spec.nodeGroups` is removed from the `Kubernetes` CR** (BREAKING).
Worker pools are now separate `KubernetesNodes` resources.
- **Migration 54** (bumps `targetVersion` to 55): pre-upgrade hook that,
for every `kubernetes-<cluster>` release, creates one
`kubernetes-nodes-<cluster>-<pool>` HelmRelease and re-annotates the
existing worker objects onto it with `helm.sh/resource-policy=keep`, so
the now control-plane-only parent upgrade does not prune running
workers. The default `md0` (implicit when `nodeGroups` was empty) is
materialised explicitly. Idempotent. Genuine infrastructure/read
failures fail closed (`exit 1`, retried on the next upgrade); hostile
stored shapes (a pool name over Helm's 53-char release-name cap, an
RFC-1123-invalid or whitespace/slash map key, a non-object `nodeGroups`)
are warned, pinned, and skipped rather than deadlocking the entire
fleet's upgrade.
- The `ingressNginx` addon no longer auto-provisions a node (the
implicit `md0` default is gone). If enabled without a pool carrying
`roles: [ingress-nginx]`, the controller DaemonSet simply schedules
nothing (degraded, not failed) — the requirement is documented in the
chart NOTES.txt rather than enforced at render time.

Because the child chart renders the pool objects **byte-identically** to
the old parent render (golden parity in #3314; the
KubevirtMachineTemplate content-hash name is preserved), adoption does
not churn live worker VMs.

### Migration number

`54` is contiguous with main (latest is `53`) and `targetVersion` is
`55`. The `52`/#3201 collision noted in earlier revisions has been
resolved: this branch was rebased and merged past it, so the numbering
is contiguous and `run-migrations.sh` has no gap to hard-fail on.

### Not yet done (blocking draft → ready)

- [ ] **dev e2e upgrade validation** (`cozystack-pr-test`): upgrade a
custom-`nodeGroups` cluster and a default-`md0` cluster; assert zero
worker-VM churn (KMT hash, Machine uid/creationTimestamp stable) and
that the adopted child HRs reconcile.
- [ ] downstream trigger-map walk (website docs for the `nodeGroups` →
`KubernetesNodes` change; terraform-provider for the removed API field).

### Downstream repositories

Deliberately unticked (draft) — the breaking API change affects docs and
the terraform provider; follow-ups to be filed before ready.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:

```release-note
Worker node pools of a managed Kubernetes cluster are now managed as separate `KubernetesNodes` resources. `spec.nodeGroups` on the `Kubernetes` CR is removed; existing pools are adopted automatically on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Worker pools are now separate `KubernetesNodes` resources with their
own lifecycle.
* Added a render-time compatibility guard to prevent worker pools from
running ahead of the parent control-plane minor version.
* Improved upgrade/uninstall behavior to better preserve already-adopted
worker objects.

* **Breaking Changes**
* Removed per-pool `nodeGroups` and worker node health settings from the
`Kubernetes` chart/CR; pool configuration now lives in
`KubernetesNodes`.
* Ingress requires at least one `KubernetesNodes` pool with `roles:
[ingress-nginx]`.
* `cluster` and `storageClass` are immutable after creation; pool
`version` may lag but must not be ahead of the parent minor.

* **Documentation**
* Updated READMEs, notes, and schemas to reflect the `KubernetesNodes`
split and the immutability/version rules.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-26 12:06:18 +04:00
IvanHunters d93e136e23 fix(computeplane): provision worker pools via kubernetes-nodes after Phase 2 split 
Phase 2 moved worker nodeGroups out of apps/kubernetes into apps/kubernetes-nodes,
but the ComputePlane module still passed nodeGroups to the wrapped kubernetes
chart, which now ignores it: a fresh ComputePlane cluster came up with zero
workers (and the pinned ingress-nginx addon stuck Pending), while existing ones
were unreachable by migration 56 (it only adopts kubernetes-* releases, not
computeplane-cluster).

The module now emits one kubernetes-nodes child HelmRelease per nodeGroup,
shaped like a natively created pool, and materialises the pre-split default md0
(roles: [ingress-nginx]) when nodeGroups is empty. The pool chart's clusterName
gains an internal clusterReleaseName override so pools attach to the module's
fixed computeplane-cluster release, which is off the aggregated API's
kubernetes-<cluster> convention; the default keeps every API-created pool
byte-identical.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-08-24 11:19:54 +03:00
Aleksei Sviridkin 9ccb133b66 fix(ci): de-escalate select-e2e and name every escalation (#3817) 
## Why

I replayed the 150 most recently merged pull requests through
`hack/select-e2e.sh` with every escalation branch instrumented. 118 of
them (78.7%) ran the full 21-suite Chainsaw run, and the causes, counted
once per escalating PR with the count where that cause was the only one
in brackets, were `FULL_PATTERN` 88 [68], `NO_SUITE_FOR_GROUP` 24 [24],
`CHAINSAW_SHARED` 18 [1] and `UNCLASSIFIED` 7 [4]. `YQ_BROKEN`,
`NO_GRAPH_OWNER` and `BACKSTOP_EMPTY` never fired.

Two problems come out of that. Most of the escalation is real but
unnecessary. Whole classes of path that provably cannot affect a
Chainsaw suite were escalating because no rule claimed them. And the
commonest cause of a full run was absent from the log, so "why did this
pull request run everything" could only be answered by re-deriving the
selection by hand.

## What changes

**Every escalation now names its cause on stderr.** Seven branches could
reach the full suite and four did it in silence, `full_suite_pattern`
among them, the commonest cause by a wide margin, so the usual answer
was the one the log never carried. The lines go to stderr and must stay
there: stdout is the suite list and both e2e lanes parse it, so a reason
line on stdout would be read as a suite name. A unit test asserts every
message, verified by muting each of the eleven in turn and confirming
the file goes red for all of them. A reason that regresses to silence
changes no selection and is otherwise invisible.

**`cozystack.etcd-application` gains its missing `dependsOn:
cozystack.etcd-operator`.** `packages/extra/etcd` renders `kind:
EtcdCluster` from `etcd-operator.cozystack.io/v1alpha2`, and this was
the only operator-backed application source in the tree with no edge to
its operator. It is a latent install bug as much as a selection gap:
without the edge `etcd-rd` registers the ApplicationDefinition as soon
as the engine is up, so a tenant can create an `Etcd` before the
operator exists and its HelmRelease fails on `no matches for kind
"EtcdCluster"`, and `hack/select-install.sh`'s forward closure for the
`etcd` suite omitted the operator for the same reason. On the selection
side `cozystack.etcd-operator` reached no runnable suite, so every
change to the operator ran all 21.

**`vm-disk-application` maps to the `vminstance` suite.** It resolved to
a `vm-disk` suite that does not exist, `intersect_suites()` dropped the
name, and the source reached nothing runnable. Neither did
`cozystack.kubevirt-cdi` above it, whose only other dependents are
`vm-default-images` and `vm-disk-application` itself, so every CDI
change ran all 21. The coverage was there and the table did not know it:
the `vminstance` suite creates a `VMDisk` and asserts the `DataVolume`
behind it (`hack/e2e-chainsaw/vminstance/vmdisk.yaml`,
`vmdisk-vmi.yaml`). Same defect class as the etcd edge above, on the
mapping side rather than the graph side, and with no production blast
radius of its own, since `src_to_suites` is read by the selector and by
nothing else.

**`cozystack.cozystack-basics` joins `cozystack.cozystack-engine` as a
propagation hub excluded from the reverse-dependency walk.** Every edge
into it exists so a namespace or a platform-wide policy is in place
before the dependent installs, which `kubevirt-cdi` states in its own
source ("Depend on cozystack-basics so the target namespace exists
first"), and that is install ordering rather than behaviour. The damage
runs opposite to the engine's: the engine fans one change out to every
app, basics narrows instead. It reaches no suite today so it escalates,
but it sits upstream of `kubevirt-cdi`, so the first suite to land under
CDI silently converts the platform's namespace-and-policy package from
the full run to that one suite. Not hypothetical — on #3426 enabling the
site-router suite takes `cozystack-basics` from 22 suites to
`site-router` alone, and nothing in the output says coverage was lost.
Dropping the reverse edges keeps the package reachable, stops it
propagating, and leaves a change to it running everything through the
per-path escalation.

**`hack/e2e-apps/<name>.bats` maps to the `<name>` suite**, off the
basename, exactly as the per-suite rule takes the name off a
`hack/e2e-chainsaw/<app>/` directory. Deliberately mapped rather than
marked inert: what remains there is wired to nothing after the Chainsaw
migration, and inert would bake that orphan status into the rule and go
quietly wrong the day a lane runs those files again. The basename is
membership-tested against the suite list on the spot, so an unmatched
one escalates there and then rather than having the verdict deferred to
the final intersection, where the rest of the diff would decide it.

**`packages/tests/` and three `.gitattributes` join
`inert_config_pattern`,** which is what the script's own header says to
do with a genuinely inert path instead of widening the fall-through.
`packages/tests/` is a helm-unittest fixture chart, and changing a test
*of* `cozy-lib` does not change `cozy-lib`, no PackageSource lists those
paths as a component, and nothing installs them, while a change to the
library itself still escalates through `packages/library/`. The
`.gitattributes` entries are enumerated rather than matched by filename,
because the justification is what those files contain (only
`linguist-generated` markers) and the name does not carry it:
`.gitattributes` can also set `filter`, `eol`, `working-tree-encoding`
and `export-subst`, each of which changes what lands in the working tree
and therefore what gets built.

**`full_suite_pattern` escalates only `hack/e2e-*.bats`, not every
`hack/*.bats`.** The root `Makefile` is the authority on the split and
draws it at exactly that prefix, `BATS_UNIT_FILES := $(filter-out
hack/e2e-%.bats,$(wildcard hack/*.bats))`, so the 60 files it keeps are
the unit lane and the e2e sandbox runs none of them, while the three it
filters out are what `packages/core/testing`'s recipes execute and those
still escalate. Being inert here does not leave them untested: `make
unit-tests` is gated on the `plan` job's `code` output, which
`pull-requests.yaml` computes as "any changed path outside `docs/`" and
never from `select-e2e.sh`, so a bats-only pull request still runs the
whole unit lane, plus install and the OpenAPI tests, since the `e2e` job
reads the same output rather than the selection.

## Measured effect

The before-and-after comparison below is a second measurement over a
slightly different population, stated separately because the two are not
interchangeable: it replays the last 150 first-parent merge commits on
`main` (`git diff --name-only $sha^1 $sha`) through the base selector
with base sources versus this branch with its own, where the histogram
above walks the 150 merged pull requests as GitHub lists them. The
populations overlap heavily and the direction is the same, and the
totals differ by two commits.

| | full suite | scoped | nothing |
|---|---|---|---|
| before | 120 | 20 | 10 |
| after | 106 | 25 | 19 |

Full-suite rate 80.0% → 70.7%, verdict changed on 15 commits. Fourteen
de-escalated and I read every one of their file lists: they are
unit-lane bats files, non-e2e workflows, `docs/`, `CODEOWNERS`,
helm-unittest fixtures and linguist markers. The fifteenth gets one
suite *wider*: a `cert-manager` change now also selects `etcd`, which is
correct, since `cert-manager` is a dependency of `etcd-operator` and the
etcd app now genuinely reaches it through the new edge.

That replay predates the two reachability rules, and replaying the same
150 first-parent merges across those two on their own — the selector at
`73bc4c0ae` against the selector at `adf231513`, same sources on both
sides — changes no verdict at all. Two of the 150 touch
`cozystack-basics` and both ran the full suite before and after, which
is the hub exclusion keeping something true rather than failing to do
anything, and none of the 150 touch `kubevirt-cdi` or `vm-disk`, so the
mapping shows up only in a direct selection.

A few individual selections, before → after:

| changed files | before | after |
|---|---|---|
| `hack/select-e2e_test.bats` | 21 suites | *nothing* |
| `hack/select-e2e_test.bats` +
`packages/apps/redis/templates/redis.yaml` | 21 suites | `redis` |
| `hack/e2e-install-cozystack.bats` | 21 suites | 21 suites |
| `packages/tests/cozy-lib-tests/**` | 21 suites | *nothing* |
| `packages/system/.gitattributes` | 21 suites | *nothing* |
| `hack/e2e-apps/postgres.bats` | 21 suites | `postgres` |
| `packages/system/etcd-operator/values.yaml` | 21 suites | `etcd` |
| `packages/system/kubevirt-cdi/values.yaml` | 21 suites | `vminstance`
|
| `packages/apps/vm-disk/values.yaml` | 21 suites | `vminstance` |
| `packages/system/cozystack-basics/values.yaml` | 21 suites | 21 suites
|

## Blast radius of the `dependsOn` edge

A `dependsOn` on a PackageSource is an install-ordering edge in
production, not only a test-selection hint, so this is the part worth
reviewing hardest. Both Packages are emitted by
`packages/core/platform/templates/bundles/system.yaml` under the single
`bundles.system.enabled` guard with no intervening conditional between
them, in every variant, so no supported configuration has the app
without the operator. `hack/select-install.sh --validate` reports the
graph still has no cycle and nothing dangling. The one new exposure is
an admin who lists `cozystack.etcd-operator` in
`bundles.disabledPackages` while keeping the app, which now leaves it
`DependenciesNotReady`. That is the same exposure postgres, mariadb,
kafka and redis already carry, with `Package.spec.ignoreDependencies` as
the escape hatch. On upgrade `etcd-rd` waits for the operator's two
HelmReleases, and an unhealthy `etcd-operator` already means broken etcd
apps.

## Testing

Every rule added or changed has a test in `hack/select-e2e_test.bats`,
the install-side change has one in `hack/select-install_test.bats`, and
the new graph edge has a helm-unittest guard at
`packages/core/platform/tests/sources_etcd_application_dependson_test.yaml`.
The round-trip test that walks every suite through both mapping tables
stays green, `vminstance` included now that two sources map onto it.

The two reachability rules were each checked by removing the rule and
confirming its own test reddens: without the mapping the CDI test falls
back to the full-suite escalation, and without the hub exclusion
`cozystack-basics` selects `redis vminstance` instead of everything. The
hub test has to seed its own hazard, since no suite sits downstream of
basics in this tree yet, so it gives `redis-application` an edge onto
basics in a copied sources directory and pins that a redis change still
selects redis, or the assertion would pass against a graph that resolved
nothing.

`make bats-unit-tests` aborts on the first failing file (`for f in …; do
… || exit 1; done`), and `hack/ghcr-mirror_test.bats` fails on `main`
today and sorts before `select-e2e_test.bats`, so the aggregate target
never reaches these tests. I ran the 60 unit files individually instead:
59 pass, and the one failure is that pre-existing one, byte-identical to
the base commit and referencing nothing this branch touches. Also green:
`make helm-unit-tests`, `make rd-presets-check migrations-target-check
test-check-readiness`, `hack/select-install.sh --validate`, and
`pre-commit run --all-files` leaving a clean tree.

## Two things a reviewer should know, neither addressed here

`assert_selection` in `hack/select-e2e_test.bats` aborts the whole file
with `exit 1` rather than failing a single test. That is the only thing
that works under `hack/cozytest.sh`, which provides no real `run`,
`$status` or `skip`, and it is the idiom the existing tests in that file
already use. It becomes wrong when #3497 and #3498 flatten these onto
real `bats(1)`, where a failing assert should end one test and let the
rest report. Worth converting with that work rather than ahead of it.

`cozystack.mongodb-application` carries the identical missing operator
edge: `packages/apps/mongodb` renders `psmdb.percona.com/v1` while the
source depends only on `cozystack.networking` and
`cozystack.cozystack-engine`, so `cozystack.mongodb-operator` still
reaches no runnable suite and every change to it escalates to all 21.
Same defect class and same fix shape as the etcd edge, but it is a
second production install-ordering change and belongs in its own pull
request.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- The default etcd application now includes its operator and required
platform dependencies, ensuring the EtcdCluster provider is available.

- **Bug Fixes**
- Improved end-to-end test selection for application-specific changes,
shared components, inert paths, and unresolved or unclassified files.
- Full-suite escalations now provide clear reasons through standard
error.

- **Tests**
- Expanded coverage for test selection, dependency reachability, and
etcd application dependency declarations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-08-20 14:12:56 +03:00
Andrei Kvapil 7cca3d75e7 feat(kubernetes)!: remove the FluxCD tenant addon 
The optional `addons.fluxcd` toggle of the Kubernetes app deployed a
Flux Operator + FluxInstance into a tenant cluster. It was served by two
vendored charts, packages/system/fluxcd-operator and packages/system/
fluxcd, which were the last AGPL-3.0 content in this Apache-2.0
repository and a governance blocker for CNCF Incubation.

The management cluster does not use these charts (it runs classic Flux
via internal/fluxinstall; migration 21 already removed the operator from
it), and the addon defaulted to disabled. Rather than maintain an AGPL
payload for an unused feature, drop it entirely:

- delete both system packages and their platform source entries
- remove the addons.fluxcd HelmReleases, values and generated schema
  from the Kubernetes app
- drop the addons.fluxcd block still submitted by the e2e harness
- drop the removed fluxcd package entries from CODEOWNERS

BREAKING CHANGE: the addons.fluxcd option of the Kubernetes app is
removed. Tenant clusters that had it enabled keep their running Flux
release; see the companion migration for the upgrade path.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-08-20 13:34:32 +03:00
Andrei Kvapil caf9511365 fix(platform): CreateReplace the redis-operator CRDs on upgrade 
helm-controller skips CRD upgrades by default, so an upgraded cluster would
keep the old spotahome schema and prune the fork's spec.engine field on
admission. The schema diff is additive, so CreateReplace is safe. Also refresh
the harbor e2e wait comment for the freshworks operator.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-08-19 10:15:58 +02:00
Myasnikov Daniil 42bd45a87c fix(platform): make etcd-application depend on etcd-operator 
extra/etcd renders kind: EtcdCluster from
etcd-operator.cozystack.io/v1alpha2, but cozystack.etcd-application
declared no dependsOn edge to cozystack.etcd-operator -- the only
operator-backed application source in the tree that did not. The
edge has been absent since the sources were first added.

Two consequences, one per side of the graph.

On install this is a latent ordering bug. Without the edge the
etcd-rd HelmRelease registers the ApplicationDefinition as soon as
cozystack-engine is up, so a tenant can create an Etcd before the
operator exists and its HelmRelease then fails with `no matches for
kind "EtcdCluster"`. hack/select-install.sh's forward closure for the
etcd suite omitted the operator for the same reason, which would
install the suite against a cluster with no CRD to apply to.

On test selection cozystack.etcd-operator reached no runnable suite,
so every change to the operator escalated to all 21 suites -- 5 of
the last 150 merged pull requests. The etcd suite exists; only the
edge that makes it reachable from the operator was missing.

Scope of the ordering change: both Packages are emitted
unconditionally by the same bundle (templates/bundles/system.yaml,
under the single bundles.system.enabled guard, in every variant), so
no supported configuration has one without the other. An admin who
lists cozystack.etcd-operator in bundles.disabledPackages while
keeping the app now leaves it DependenciesNotReady -- the same
exposure postgres, mariadb, kafka and redis already carry, with
Package.spec.ignoreDependencies as the escape hatch.
select-install.sh --validate confirms the edge adds no cycle and
nothing dangling.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-08-15 17:58:07 +05:00
Aleksei Sviridkin f4d812cfb9 fix(velero): apply CRD updates on upgrade via CreateReplace 
velero evolves its CRD schema between versions; v1.18 adds the Queued
and ReadyToStart backup phases. Helm never upgrades CRDs shipped in
crds/, the generated HelmRelease left spec.upgrade.crds at the Skip
default, and the package disables the chart's upgrade-crds Job, so live
CRDs stayed frozen at first-install state: the server upgrades, the
apiserver rejects the new phases, and backups sit in New while the
HelmRelease stays green. Opt the velero PackageSource into
CreateReplace, consistent with mongodb-operator, add a platform test
pinning the policy, and correct the stale Job-disable rationale in the
package.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
 2026-08-10 12:23:16 +03:00
Andrei Kvapil ebeeab77aa docs(platform): correct the computeplane dependsOn and variant comments 
The dependsOn comment claimed the kubernetes-application edge orders the
addon artifacts the wrapped chart references; variant dependsOn only gates
install: components — the operative effect is ordering the computeplane-rd
install so the ComputePlane kind is not registered before the kubernetes
machinery exists. Also document why the variant is named kubevirt.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-07-28 18:08:02 +02:00
Andrei Kvapil aa5198b758 fix(platform): add cozystack-engine dependency to the computeplane package 
The computeplane-rd component installs an ApplicationDefinition, whose CRD is
served by cozystack-engine — hack/check-applicationdefinition-crd-ordering.bats
enforces the edge for every package with a system/*-rd component.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-07-28 18:08:02 +02:00
Andrei Kvapil 4c3262a4ee feat(computeplane): deliver ComputePlane as a tenant module wrapping the kubernetes app 
Add the operator-owned computeplane tenant module per the revised design
(cozystack/community design-proposals/compute-plane): an extra/computeplane
chart that provisions a single-tenant, Cozystack-managed cluster for
untrusted-code apps by deploying the ordinary apps/kubernetes chart with
operator-fixed values (fluxcd, cert-manager, ingress-nginx Proxied). The
package re-declares apps/kubernetes as its own source-only component, so the
module is self-contained; the cluster release is named computeplane so Kamaji
writes the admin kubeconfig to computeplane-admin-kubeconfig — the contract
consumed by placement: ComputePlane applications. The tenant holds no admin
kubeconfig and can set cluster shape (nodeGroups) only.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-07-28 18:08:02 +02:00
Ivan Okhotnikov e7947dd0e7 feat(platform): register kubernetes-nodes application 
Add the kubernetes-nodes-rd package (ApplicationDefinition), the
cozystack.kubernetes-nodes-application PackageSource (kubevirt variant,
kubernetes-nodes + kubernetes-nodes-rd components), and wire it into the
iaas bundle. Mirrors the vm-instance/vm-disk registration precedent.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-21 22:35:10 +03:00
IvanHunters d4f59af624 chore(monitoring): sync PackageSource comment with removed Keycloak groups 
Update the oidc-variant comment on
`packages/core/platform/sources/monitoring-application.yaml` so it
no longer claims `spec.oidc.mode: System` renders three
KeycloakRealmGroups. The dependsOn edge and the fail-close guard
in `templates/grafana/oidc-keycloak.yaml` stay unchanged: the
KeycloakClient + KeycloakClientScope path still needs the
`v1.edp.epam.com` CRDs from `cozystack.keycloak-operator` to be
registered before the monitoring chart reconciles.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-15 15:31:45 +03:00
Ivan Okhotnikov be52e80289 fix(monitoring): split monitoring-application PackageSource into default/oidc variants 
Follow-up to the previous self-review-fix commit: the unconditional
`- cozystack.keycloak-operator` in dependsOn was itself a
CRITICAL cold-install regression on any cluster where
`authentication.oidc.enabled: false` (the default). The
keycloak-operator Package is only emitted by
`packages/core/platform/templates/bundles/system.yaml:112` when
that flag is on — a hard dep against it from the default variant
would leave `cozystack.monitoring-application` stuck on
DependenciesNotReady forever, cascading a
`cozystack-platform` HR timeout that hits every OIDC-off customer.

Fix by mirroring the cozystack-engine variant split:

- `default` variant keeps the baseline reconciliation edges
  (no keycloak-operator dep).
- `oidc` variant additionally waits for
  `cozystack.keycloak-operator` so the v1.edp.epam.com CRDs are
  registered before the monitoring chart reconciles and the
  `.Capabilities.APIVersions.Has` guard in
  templates/grafana/oidc-keycloak.yaml can safely emit
  KeycloakClient/Scope/Groups.

The bundle template swings between variants on
`.Values.authentication.oidc.enabled`, mirroring the
cozystack-engine invocation two lines above. The
`monitoring-system` component override
(oidc.grafanaAdmin: true) is preserved across both branches — the
values field is a no-op when mode: None (no auth.generic_oauth
block is emitted at all), so keeping it identical avoids two
divergent code paths.

Also widened the platform helm-unittest coverage to lock:

- default variant does NOT depend on keycloak-operator (permanent
  DependenciesNotReady guard).
- oidc variant DOES.
- Bundle emits the right variant based on
  authentication.oidc.enabled AND keeps the GrafanaAdmin uplift
  on the monitoring-system component in either branch.

83 platform tests + 23 monitoring tests still green.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:44 +03:00
Ivan Okhotnikov ffeb8e5e4f fix(monitoring): close self-review blockers for the OIDC PR 
Address the two blocking-class findings from the cozy-review pass on
this PR:

(1) Missing PackageSource dependency on cozystack.keycloak-operator
    (MAJOR cold-install race).

    Without this edge, the monitoring chart can reconcile before the
    v1.edp.epam.com CRDs are registered on a fresh cluster — the
    `.Capabilities.APIVersions.Has` guard in
    templates/grafana/oidc-keycloak.yaml then silently drops all
    Keycloak-side objects for mode: System, and Grafana's
    auth.generic_oauth block points at a client Keycloak never
    provisions. Add the ordering edge, plus a
    sources_monitoring_application_dependson_test.yaml regression
    guard so a future edit cannot drop it silently.

(2) allow_assign_grafana_admin driven by `.Release.Name ==
    monitoring-system` (MINOR fragile coupling).

    Replace the release-name sniff with an explicit
    `oidc.grafanaAdmin: false` values field, and flip it on for the
    platform Grafana release via
    Package.spec.components["monitoring-system"].values.oidc.grafanaAdmin
    in packages/core/platform/templates/bundles/system.yaml —
    mirroring the pattern monitoring-agents already uses for
    per-component overrides. Tenant Grafana releases inherit the
    chart default false and stay at org-level Admin.

    Two new helm-unittest cases in packages/system/monitoring/tests/
    lock the values-driven behaviour on both sides
    (grafanaAdmin=true → Admin uplift, default → tenant stays at org
    Admin), plus bundles_monitoring_grafana_admin_test.yaml in
    packages/core/platform/tests/ regression-guards the Package's
    component override so the platform release keeps its uplift on
    future edits.

Also document the mode-toggle group-membership loss in
docs/oidc-grafana.md — mode: System → CustomConfig/None deletes the
three chart-owned KeycloakRealmGroups; membership is not restored on
toggle-back. Not a bug, but the operator UX cost of the toggle
deserves a bullet in Failure modes.

Regenerated artifacts (cozyvalues-gen + hack/update-crd.sh):
  packages/extra/monitoring/values.schema.json
  packages/extra/monitoring/README.md
  packages/system/monitoring-rd/cozyrds/monitoring.yaml

Test suites still green: packages/system/monitoring 23/23,
packages/core/platform 81/81.

Signed-off-by: Ivan Okhotnikov <xorokhotnikov@gmail.com>
 2026-07-15 15:31:44 +03:00
IvanHunters 1fb4bcf703 feat(system): add talos-log-collector package 
Add a node-local Vector DaemonSet that receives Talos machine and kernel
logs (pushed via machine.logging over the node loopback) and forwards them
to the tenant VictoriaLogs. Talos keeps its system and kernel logs inside
the OS and does not expose them as host files, so the tail-based Fluent Bit
in monitoring-agents never collects them.

The receiver stays on the pod network so cluster DNS resolves the vlinsert
service, and accepts Talos logs via a hostPort bound to hostIP 127.0.0.1,
avoiding the host-network DNS breakage a hostNetwork receiver would hit.
Shipped as an optional system package.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-07-10 15:34:08 +03:00
Andrey Kolkov cc5d688e15 feat(platform): migrate to etcd-operator v1alpha2 
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-07-06 14:41:33 +04:00
Ivan Okhotnikov 9dc64b9278 feat(platform): VPA depends on cert-manager after admission-controller TLS migration 
The VPA admission-controller webhook TLS is now provisioned via a
cert-manager Certificate + Issuer instead of the in-chart genCA
Secret. On a fresh install the VPA package must not reconcile until
cert-manager is Ready; otherwise the Certificate resource sits
Pending forever, the admission-controller Secret is never written,
and the pod CrashLoops on missing TLS mount — cascading up through
the cozystack-platform PackageSource in exactly the way the old
genCA path did, only for a different reason.

Add cozystack.cert-manager to the variant's dependsOn list. Verified
non-circular against packages/core/platform/sources/cert-manager.yaml
(cert-manager depends on cozystack.networking + cozystack.gateway-api-crds,
neither of which reaches VPA).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
 2026-07-02 18:57:01 +03:00
Aleksei Sviridkin cbc314a8c5 fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public 
The kubevirt-cdi default variant only depended on cozystack.networking,
which reconciles long before cozystack-basics provisions the cozy-public
namespace. kubevirt-cdi ships the cdi-clone-dv RoleBinding into cozy-public,
so on a fresh install the RoleBinding can race namespace creation and fail
with `namespaces "cozy-public" not found`. Flux retries normally mask it,
but under slow reconcile it surfaces as a hard UpgradeFailed.

Add a cozystack.cozystack-basics edge so the namespace exists first,
mirroring the backupstrategy-controller ordering. No cycle: cozystack-basics'
own dependencies (tenant-application, cozystack-engine, gateway-api-crds) do
not reach kubevirt-cdi. Pin the edge with a helm-unittest regression guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-30 19:51:46 +03:00
Aleksei Sviridkin bf14a73b39 feat(api): add SecurityGroup network policy resource (sdn.cozystack.io) (#2922) 
## What this PR does

Introduces a tenant-facing `SecurityGroup` resource in a new
`sdn.cozystack.io/v1alpha1` API group, served by the Cozystack
aggregated API server as a 1:1 projection of a `CiliumNetworkPolicy`. A
SecurityGroup attaches to one of the tenant's managed applications by
reference (`spec.targetRef: {apiGroup, kind, name}`) and declares
allow-list ingress/egress for that application. Tenants manage network
policy for their own apps entirely through
`securitygroups.sdn.cozystack.io`, without being granted any access to
the `cilium.io` API group.

The translation is synchronous and stateless — the API server converts
each SecurityGroup into a CiliumNetworkPolicy in the same namespace on
write and back on read; there is no controller. The backing policy's
`endpointSelector` is **derived** from `targetRef` (matched on the
application's lineage labels
`apps.cozystack.io/application.{group,kind,name}`), never copied from
tenant input, so the selector is machine-generated and can only ever
match the referenced application's own pods in the tenant's namespace.
Policies owned by this API carry an `sdn.cozystack.io/securitygroup`
marker label that the storage always re-asserts and hides from the view,
so it only ever surfaces or mutates its own policies and never touches
platform-managed network policies.

The value is self-service network policy on a Cozystack-native,
RBAC-scoped surface, without exposing Cilium's version-coupled CRDs to
tenants. Because the selector is derived from an authorized application
reference rather than free-form tenant input, the RBAC boundary is
structural, not selector-validated: a tenant cannot express network
policy against arbitrary or platform-owned pods. The full design —
selector derivation from lineage labels, the structural boundary, the
additive-policy reality (a SecurityGroup widens a managed application's
allowed traffic but cannot sever its platform-managed management plane,
and equally cannot tighten below the platform baseline), the in-tree
CiliumNetworkPolicy mirror that avoids a Cilium Go-module dependency,
RBAC, and deferred extensions (raw-pod targeting, cluster scope,
reusable groups) — is in `pkg/apis/sdn/DESIGN.md`.

Implements the attachment model from #1614.

### Release note

```release-note
feat(api): add the SecurityGroup resource (sdn.cozystack.io/v1alpha1) — a tenant-facing, namespace-scoped firewall that attaches to a managed application by reference and projects 1:1 onto a CiliumNetworkPolicy with a machine-derived endpoint selector, letting tenants manage their application's network policy without cilium.io access.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added the `SecurityGroup` API resource (`sdn.cozystack.io` v1alpha1)
with a namespaced projection backed by Cilium network policies.
* Enabled the SDN `APIService` and wired SDN routing, scheme, and REST
support.
* Expanded RBAC to include SecurityGroup access and required Cilium
policy permissions.
* **Documentation**
* Added a design document describing projection behavior and rule
semantics.
* **Bug Fixes**
  * Improved API rule validation coverage for missing list-type fields.
* **Tests**
* Added SDN round-trip, fuzzing, RBAC, and comprehensive REST
create/get/list/update/delete/watch/resource-version tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-30 14:30:23 +03:00
IvanHunters e0597e3d99 fix(platform): make *-rd sources depend on cozystack-engine (#3041) 
## Problem

On an unlucky Flux reconcile order, a `*-application` PackageSource that
ships a `system/*-rd` component can install before the
`applicationdefinitions.cozystack.io` CRD exists. Every `*-rd` chart
renders an `ApplicationDefinition` (`cozystack.io/v1alpha1`); that CRD
is installed by the `application-definition-crd` component of
`cozystack.cozystack-engine`. When an `*-rd` HelmRelease renders first,
it hard-fails with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, and that cascades into a broad install
timeout — surfacing even on changes that have nothing to do with
ordering, because it is a latent race.

## Fix

Add `- cozystack.cozystack-engine` to the `dependsOn` of every
`*-application` source that ships a `system/*-rd` component (29
sources). `gateway-application` already carried this edge — it was added
earlier for the TenantGateway CRD, the same bug class, but only for that
one source. A new invariant test fails if any source with a
`system/*-rd` component is missing the edge.

## Why

The ordering was previously implicit, relying on Flux happening to
reconcile `cozystack-engine` before the `*-application` sources. Making
the edge explicit removes the race deterministically. No dependency
cycle is introduced: `cozystack-engine` depends only on `networking` and
`cert-manager` (plus keycloak in its oidc variant), none of which reach
back to any `*-application`.

## Test

- New generic guard `hack/check-applicationdefinition-crd-ordering.bats`
(auto-discovered by `make bats-unit-tests`, runs in CI via `make
unit-tests`): parses every source and fails if any PackageSource variant
with a `system/*-rd` component lacks the `cozystack.cozystack-engine`
edge — so future sources are covered, not just today's set.
- `helm unittest packages/core/platform`: 13 suites / 66 tests pass; the
source edits do not disturb existing assertions.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved dependency ordering across several platform packages to
ensure related components install in the correct sequence.
* Prevented end-to-end test selection from being overly broad when only
ordering-related dependencies change.

* **Tests**
* Added checks to catch missing dependency requirements and to verify
test selection behavior stays accurate.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-25 19:21:50 +03:00
Aleksei Sviridkin 69247ba80c feat(platform): deploy the securitygroup-controller 
Add the securitygroup-controller Helm package (Deployment, ServiceAccount,
ClusterRole and binding, image build) and register it as a default system
package so it ships with the platform.

The ClusterRole grants cluster-wide pod patch — the controller stamps the
SecurityGroup membership label on managed-app pods across dynamically-created
tenant namespaces, so the grant cannot be namespace-scoped — plus read and
finalizer patch on the backing CiliumNetworkPolicies, and leader-election
leases. Tenants gain no new permissions; the membership label is written
exclusively by this platform controller. helm-unittest pins the pod-patch and
ciliumnetworkpolicies grants so a dropped verb fails the suite rather than
silently failing closed at runtime.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-25 17:50:33 +03:00
Aleksei Sviridkin fd6e121b28 fix(platform): make *-rd sources depend on cozystack-engine 
Every *-application PackageSource that ships a system/*-rd component
renders an ApplicationDefinition (cozystack.io/v1alpha1). That CRD is
installed by the application-definition-crd component of
cozystack-engine. Without an explicit dependsOn edge, a fresh install
where Flux reconciles a *-application source before cozystack-engine
fails to render with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, which cascades into a broad install
timeout unrelated to whatever change triggered the reconcile.

Add `- cozystack.cozystack-engine` to the dependsOn of every source that
ships a system/*-rd component; gateway-application already carried the
edge for the TenantGateway CRD. Add a generic invariant test that fails
if any such source is missing the edge, so future sources cannot
reintroduce the race.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-24 22:43:56 +03:00
Aleksei Sviridkin ee853b00bd fix(platform): order backupstrategy-controller after cozystack-basics 
backupstrategy-controller creates the cozy-backups Bucket into the
tenant-root namespace, which cozystack-basics provisions. On a fresh
install the Bucket patch can race namespace creation and fail with
`namespaces "tenant-root" not found`, which then races the install
wait deadline and fails the run — on changes that have nothing to do
with backup ordering. Add the cozystack-basics dependsOn edge so the
target namespace exists first, pinned with a helm-unittest regression
guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-24 22:34:09 +03:00
Aleksei Sviridkin e9aaee286e fix(mongodb-operator): apply CRD updates on upgrade via CreateReplace 
psmdb-operator evolves its CRD schema between minor versions; 1.22.0 adds
spec.storageScaling/storageAutoscaling, vault and other fields. Helm never
upgrades CRDs shipped in crds/, so without an explicit policy the operator
binary upgrades on an existing cluster while the live CRD stays behind and
the apiserver rejects the new fields. Opt the mongodb-operator PackageSource
into CreateReplace, consistent with etcd-operator, and add a platform test
pinning the policy.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-22 19:05:03 +03:00
Myasnikov DaniilandClaude dc9427285d Merge branch 'main' into feat/flux-shard-operator-impl 
Resolve go.mod conflict: keep main's go 1.26.4 and golang.org/x/* bumps, and keep gomodules.xyz/jsonpatch/v2 v2.4.0 as a direct dependency (imported by the flux-shard-operator admission webhook) instead of main's indirect entry. go mod tidy leaves go.sum consistent.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-06-22 10:19:07 +05:00
Aleksei Sviridkin b6108039b7 feat(gpu-operator): add container variant for preinstalled host driver (#2766) 
## What this PR does

Add a third architectural variant to `cozystack.gpu-operator` for hosts
where the NVIDIA driver and `nvidia-container-toolkit` are already
installed by the distro package manager (the typical Ubuntu/Debian GPU
node shape: apt-installed driver + apt-installed CT).

The existing variants do not produce a working setup in that shape.
`default` (passthrough) runs `vfioManager`, which unbinds the host
driver to bind `vfio-pci` — with the host driver active,
`k8s-driver-manager` early-exits and the bind never happens, so the GPU
never reaches a pod. `vgpu` requires the proprietary NVIDIA vGPU host
driver and a license server; the user-installed open or proprietary
driver does not satisfy it.

The new `container` variant maps onto upstream gpu-operator's third mode
(`sandboxWorkloads.enabled=false` + `devicePlugin.enabled=true`) with
every component that would conflict with the host install switched off:

- `driver.enabled=false` — use the host driver.
- `toolkit.enabled=false` — host already has `nvidia-container-toolkit`.
Enabling the operator's toolkit DaemonSet would overwrite the
`/etc/containerd/config.toml` the host package configured (via
`nvidia-ctk runtime configure`), breaking the host runtime wiring.
Matches NVIDIA's documented guidance and the existing
`examples/values-native-talos.yaml` reference, which pins the same flag
for the same reason.
- `devicePlugin.enabled=true` — publish `nvidia.com/gpu` to the kubelet.
- `vfioManager.enabled=false` — do not unbind the host driver. Upstream
renders this `true` unconditionally; it only stays at zero pods because
no node carries a `vm-passthrough` sandbox assignment when
`sandboxWorkloads` is off — pinned off so the rendered policy matches
intent.
- `cdi.enabled=false` — upstream defaults this `true`, but CDI spec
generation is serviced by the operator-managed toolkit DaemonSet this
variant disables. An apt host doesn't auto-generate CDI specs
(`nvidia-ctk cdi generate` is a manual step) and uses the classic
`nvidia-container-runtime` hook, so leaving CDI on would point the
device plugin at CDI injection with no specs on the host and silently
break allocation.
- `sandboxWorkloads.enabled=false` — regular pod workloads, not VMs.
With this off, upstream `defaultWorkload` is already `container`, so no
override is needed.

Host prerequisite: because the operator's toolkit component is disabled,
the node's containerd must already have the `nvidia` runtime registered
(`nvidia-ctk runtime configure --runtime=containerd`); `apt install
nvidia-container-toolkit` installs binaries only and does not wire
containerd. With `driver.enabled=false` the operator uses the
pre-installed host driver at its standard location, so a stock apt
install needs no `hostPaths.driverInstallDir` override (Talos lands the
driver under a non-standard prefix — see
`examples/values-native-talos.yaml`).

The bundle wiring (`packages/core/platform/templates/bundles/iaas.yaml`)
already enables `cozystack.gpu-operator` as an optional package — no
bundle changes required. Users opt into this variant by setting
`variant: container` in their `Package` CR.

Verified locally with `helm template` against `values.yaml` +
`values-container.yaml`: ClusterPolicy renders with
`driver.enabled=false`, `toolkit.enabled=false`,
`devicePlugin.enabled=true`, `vfioManager.enabled=false`,
`cdi.enabled=false`, `sandboxWorkloads.enabled=false`. Platform `helm
unittest` and the `gpu-operator` variant bats suite are green. Not yet
smoke-tested on a real Ubuntu GPU node — the config matches the
documented apt-host wiring (legacy `nvidia-container-runtime` hook, CDI
off), but an end-to-end CUDA-pod run on hardware is still owed.

Closes #2764

### Release note

```release-note
feat(gpu-operator): add `container` variant for hosts with a pre-installed NVIDIA driver and nvidia-container-toolkit, exposing containerized GPU workloads via the standard NVIDIA device plugin without VFIO passthrough or vGPU.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a "container" GPU operator variant to support containerized GPU
workloads on hosts with the NVIDIA driver and nvidia-container-toolkit.

* **Documentation**
* Updated variants documentation and OS support table to include the
container variant and per-host requirements (including Ubuntu and Talos
notes).
  * Clarified Talos native pod example wording.

* **Tests**
* Added regression tests to verify container-variant configuration and
component enablement/disablement.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-10 02:00:59 +03:00
Aleksei Sviridkin 9352d3ee0e feat(gpu-operator): add container variant for preinstalled host driver 
The two existing variants assume the host has neither the NVIDIA driver
nor the container toolkit: default unbinds the host driver to bind
vfio-pci for KubeVirt VMs, vgpu requires the proprietary vGPU host
driver and a license server. Neither path produces a working setup on a
node that already ships the driver and nvidia-container-toolkit via the
distro package manager — the most common shape of a Linux GPU node.

The new container variant surfaces the third architectural mode that
upstream NVIDIA gpu-operator already supports: sandboxWorkloads
disabled, devicePlugin enabled, with every component that would
conflict with the host install switched off.

- driver.enabled=false        use the host driver
- toolkit.enabled=false       host already has nvidia-container-toolkit;
                              enabling overwrites the containerd
                              config.toml the host configured via
                              nvidia-ctk runtime configure
- devicePlugin.enabled=true   publish nvidia.com/gpu to the kubelet
- vfioManager.enabled=false   do not unbind the host driver
- cdi.enabled=false           CDI spec generation is serviced by the
                              disabled toolkit; an apt host uses the
                              legacy nvidia-container-runtime hook, so
                              pin off to match the host runtime wiring
- sandboxWorkloads.enabled=false   regular pod workloads, not VMs

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-06-08 16:58:51 +03:00
Myasnikov Daniil 56014f1fdc fix(platform): order cozystack-basics after the APIs its admission policies type-check 
cozystack-basics ships cluster-wide ValidatingAdmissionPolicies matching
tenants.apps.cozystack.io and the Gateway API types. KCM's VAP status
type-checker panics on a nil schema when it type-checks a policy against a
matched type that isn't resolvable yet; during a cold install this crashes
KCM, which stalls cert/token issuance, so cozystack-api never starts and the
tenants schema never resolves — a bootstrap deadlock.

Add dependsOn edges so the package installs only after cozystack-engine
(cozystack-api) and gateway-api-crds are Ready. Drift detection is off on
operator-generated HelmReleases, so a template capability gate would drop the
policy at first install and never re-add it; ordering is the reliable fix.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 03f1e7a55a)
 2026-06-08 15:15:32 +05:00
Andrei KvapilandClaude d75fac0096 feat(flux-shard-operator): add system package 
New cozy-fluxcd system package (Deployment with 2 replicas, webhook
cert via cert-manager, CREATE-only MutatingWebhookConfiguration with
failurePolicy Ignore scoped to HelmReleases born on the legacy tenants
bucket), registered in the platform system bundle and the root image
build.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
 2026-06-03 19:47:19 +02:00
myasnikovdaniil c435efa199 feat(flux): upgrade to v2.8.0 + chart fixes for strict SSA & kstatus (folds #2612) (#2602) 
## What this PR does

Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and
the embedded management-cluster manifests, and folds in the chart fixes
that become hard errors under the new strict helm-controller v1.5.

Flux v2.8's helm-controller v1.5.0 ships:
- **Server-Side Apply with `--force-conflicts`** — strict CRD schema
enforcement; misplaced fields (silently dropped on v2.7) now fail the
apply.
- **kstatus-based health checking** by default — parent HR waits for
every applied resource (including child HRs) to be Ready before flipping
its own Ready, surfacing latent ordering deadlocks.
- **Helm v4 embedded** — `upgrade.force: true` is deprecated and now
conflicts with SSA.
- **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper
readiness gating (used in PR #2601 split).

Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same
Flux upgrade triggers it.

## Commits

**Flux upgrade itself:**
- `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts
to v0.48.0; web UI opt-in.
- `feat(flux)`: regenerate embedded management-cluster manifests via
`make update` in `packages/core/flux-aio` (timoni bundle build).

**Chart fixes for strict SSA — fields the chart sent that v2.7 silently
dropped, v2.8 rejects:**
- `fix(kubevirt-instancetypes)`: drop persistent strip that produced
null `preferredTPM` (folds #2612).
- `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`,
`minimumUptimeSecondsForBounce` from inside `automationOptions` to
direct children of `spec`.
- `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a
phantom `template.spec`.
- `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is
set.
- `fix(platform)`: drop deprecated `upgrade.force: true` from
HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths.

**Ordering / deadlock fixes under v2.8 kstatus:**
- `fix(vpa)`: break circular wait between parent install and nested
`vpa-for-vpa` HR.
- `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant
addon child HRs (parent waits on child via kstatus, child waited on
parent — deadlock).

**E2E waits for v2.8 kstatus timing:**
- `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's
faster dispatch).
- `test(e2e)`: wait for parent HR Ready before downstream asserts in
`run-kubernetes.sh` and `vminstance.bats`.

## Scope discipline

This PR is part of the split of #2619 (the consolidated CI fixes branch)
into review-friendly pieces. Companion PRs:
- **PR #2601** (seaweedfs split) — folded into this PR (commits
`29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the
seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption
migration 43 (targetVersion 44), and the configurable db resources all
land here, because the strict-SSA `upgrade.force` removal and the
kstatus parent-HR timeout bump only make sense together with the split.
#2601 is superseded.
- **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) —
independent, lands separately.
- Several smaller standalone fixes lifted out of #2619 (startup probes,
cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC
test improvements) — opened as separate PRs.

## Verification

- `helm template` renders cleanly for both `fluxcd` and
`fluxcd-operator` packages with `web.enabled=false` (default) and
`web.enabled=true`.
- Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0
/ v2.1.0 controller image strings.
- No references to the v0.39-removed `--disable-wait-interruption` flag
anywhere in `packages/` or `internal/`.

### Release note

```release-note
Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking).

When upgrading existing clusters:
- Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too.
- HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it.
- KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass.
- KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained.
- FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Flux Status web UI: optional web server, config secret, service port,
HTTPRoute/Ingress, network policy, and RBAC roles.
  * New SeaweedFS DB Helm chart and optional managed DB release.

* **Improvements**
* CRD/schema enhancements: new provider kinds, validations, variant
option, and external checksum refs.
* Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for
reliability.

* **Chores**
* Bumped Flux Operator and Flux versions; documentation links updated to
fluxoperator.dev.
* **Bug Fixes**
  * Removed aggressive HelmRelease force-upgrade/install flags.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-06-02 14:08:39 +05:00
Aleksei Sviridkin 2d58479f06 feat(platform): add kuberture as optional system package (#2647) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Adds [kuberture](https://github.com/lexfrei/kuberture) as an optional
system package. Kuberture is a small controller that translates the
`default/kubernetes` EndpointSlice into headless Services with
external-dns-style annotations, closing the gap that external-dns cannot
use EndpointSlice as a source directly:

```text
EndpointSlice (kubernetes) → kuberture → headless Service(s) with annotations → external-dns → DNS
```

The package is opt-in via `bundles.enabledPackages:
[cozystack.kuberture]`. Operators populate `config.outputs` per
HelmRelease — each output targets a specific external-dns instance by
its `annotationPrefix`, so a single kuberture install can serve multiple
external-dns instances (e.g. one publishing to the public zone, another
to an internal corporate prefix) simultaneously.

The vendored chart at `packages/system/kuberture/charts/kuberture/` is
pulled from `oci://ghcr.io/lexfrei/kuberture/charts/kuberture` and
pinned by both version AND OCI manifest digest in the Makefile
(mirroring the ouroboros lockstep pattern). The controller image at
`ghcr.io/lexfrei/kuberture` is pinned by tag+sha256 in the shim
values.yaml. Both pins advance in lockstep with each upstream release;
the procedure is documented inline.

Install-order dependencies declared in `PackageSource`:
`cozystack.networking` (CNI before pod scheduling) and
`cozystack.prometheus-operator-crds` (ServiceMonitor CRD before the
chart's enabled-by-default ServiceMonitor manifest). External-dns is
deliberately NOT a dependency — kuberture creates annotated Services
regardless of whether external-dns is installed; external-dns is a
downstream consumer, not a runtime prerequisite.

Coverage:

- **helm-unittest**
(`packages/system/kuberture/tests/kuberture_test.yaml`, 10 cases):
fail-fast on empty outputs, ServiceMonitor enabled by default, source
pinned to `default/kubernetes`, multi-output prefix isolation,
empty-string `annotationPrefix` schema rejection, omitted-prefix
fallback to controller default, ConfigMap → Deployment volume mount,
controller image pinned by tag+sha256 digest, Role permits full Service
lifecycle, metrics containerPort derived from `metricsBindAddress`.
- **bats e2e** (`hack/e2e-apps/kuberture.bats`): applies a `Package
cozystack.kuberture` with two outputs (different `annotationPrefix`
values), waits for HelmRelease and controller readiness, asserts both
output Services are headless and carry the right
`hostname`/`target`/`ttl` annotations under their configured prefix,
cross-checks per-output isolation (each Service only carries its own
prefix's annotations).

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not applicable — no UI changes.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(platform): add kuberture as an optional system package — translates the default/kubernetes EndpointSlice into annotated headless Services so external-dns can publish the Kubernetes API endpoint to DNS. Opt-in via bundles.enabledPackages: [cozystack.kuberture]; populate config.outputs to target one or more external-dns instances by annotationPrefix.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Adds kuberture as an optional system package to publish the Kubernetes
API endpoint to DNS via configurable external-dns outputs; includes Helm
chart, RBAC, NetworkPolicy, PDB, ServiceMonitor, and validated chart
values/schema.

* **Tests**
* Adds Helm unit tests and an end-to-end smoke test validating
controller readiness, headless Service annotations, and external-dns
routing/probe behavior.

* **Documentation**
* Adds a full README with usage, configuration examples, and
vendoring/pinning guidance.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2647)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---

Documentation companion:
[cozystack/website#539](https://github.com/cozystack/website/pull/539).
 2026-06-02 09:40:26 +03:00
Andrey Kolkov f17ead3d26 feat(platform): add default backupclass 
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
 2026-05-28 13:49:59 +04:00
Myasnikov DaniilandClaude Opus 4.7 29c6afc84d refactor(seaweedfs): split seaweedfs-system into seaweedfs-db + seaweedfs-system 
Splits the seaweedfs-system HelmRelease in two so the CNPG
Cluster/seaweedfs-db lives in its own HR (seaweedfs-db), and the
application HR (seaweedfs-system) dependsOn it. The new HR uses Flux v2
HelmRelease.spec.healthCheckExprs with a CEL expression on
Cluster.status.conditions[type=Ready] plus waitStrategy.name: poller, so
its Ready=True only flips after the postgres primary is actually serving
connections — not just after helm install applied the Cluster CR.

This eliminates the seaweedfs-filer CrashLoopBackOff race on a fresh
tenant install. With Cilium kubeProxyReplacement: true, socket-LB
returns EPERM from connect(2) to ClusterIPs with no Ready endpoints.
Pre-split, the filer StatefulSet scheduled concurrently with the CNPG
bootstrap (~55–70s of unavailable postgres), each connect() failed
EPERM, kubelet exponential restart backoff pushed past the e2e bats
'kubectl wait hr/seaweedfs-system --timeout=2m' window, and the
'Configure Tenant and wait for applications' test failed.

healthCheckExprs uses Flux's three-predicate form: route
ClusterIsNotReady (the CNPG bootstrap condition) to `inProgress` so the
HR keeps polling, keep `failed` for other Ready=False reasons, and rely
on the HR `timeout: 10m` as the real backstop for a genuinely stuck
cluster. A plain Ready=False predicate would flag the HR failed within
seconds of creating the Cluster CR.

Migration 42 adopts existing Cluster/seaweedfs-db resources into the new
release on upgrade by rewriting meta.helm.sh/release-name and stamping
helm.sh/resource-policy: keep so the seaweedfs-system upgrade (which no
longer renders the Cluster) does not delete it during the transition.

The application HR's dependsOn collapses the db dependency (unconditional)
with the existing ingress dependency (guarded by `if eq $ingress
.Release.Namespace` so sub-tenants that inherit ingress from a parent
namespace don't deadlock on a non-existent local HR) into one list.

Requires Flux v2.8.x (helm-controller v1.5.0+) for healthCheckExprs.
Provided by the parent PR #2602 in this chain.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-28 12:46:56 +05:00
Myasnikov Daniil 535f881160 fix(platform): include cert-manager in cozystack-engine oidc variant 
The oidc variant installs system/cozystack-api and
system/lineage-controller-webhook, both of which render
cert-manager.io/v1 Certificate resources. Add cozystack.cert-manager to
the oidc variant's dependsOn so it matches the default variant and
webhook-cert consumers do not race the cert-manager controller on cold
installs.

Address review feedback from gemini-code-assist and coderabbitai on
packages/core/platform/sources/cozystack-engine.yaml.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-05-27 20:35:49 +05:00
Myasnikov Daniil 69cb7b5f18 fix(platform): add cert-manager dependency to webhook-cert consumers 
Five PackageSources whose charts apply cert-manager Certificate CRs and
mount the resulting Secret on a webhook Deployment lacked an explicit
dependsOn: cozystack.cert-manager. Under Flux v2.8 helm-controller v1.5.0
kstatus, the dependent Deployment trips progressDeadlineSeconds (10m
default) when cert-manager controller isn't running yet, and the HR is
failed early. Repeatedly observed on PR #2619 with capi-operator,
kamaji, rabbitmq-operator, cozystack-api (cozystack-engine), and
ingress-nginx all cascading to InstallFailed at ~12-15m into a fresh
install.

The mariadb-operator, kubeovn-webhook, snapshot-controller, etcd-operator,
postgres-operator, mongodb-operator, victoria-metrics-operator, linstor,
and linstor-scheduler PackageSources already had the dependency, which
is why they weren't in the failure set.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 0fc0796b19)
 2026-05-27 20:35:49 +05:00
Aleksei Sviridkin d2e851f63b feat: Gateway API support via Cilium 
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.

The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.

publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.

Security model -- seven independent layers, all admission-time gates
fail-closed:

  1. Listener allowedRoutes namespace whitelist via the
     kube-apiserver-written kubernetes.io/metadata.name label;
     plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
     passthrough listeners restrict allowedRoutes.kinds to
     HTTPRoute / TLSRoute respectively.
  2. cozystack-gateway-hostname-policy VAP on Gateway.
  3. cozystack-gateway-attached-namespaces-policy VAP on Package --
     blocks tenant-* in gateway.attachedNamespaces.
  4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
     writes from non-trusted callers.
  5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
     writes to namespace.cozystack.io/host from non-trusted callers.
  6. Render-time fail in cozystack-basics for tenant-* in
     gateway.attachedNamespaces.
  7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
     scoped to tenant-* namespaces.

Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).

Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.

Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.

Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.

Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.

Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.

Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-25 17:07:02 +03:00
Aleksei Sviridkin 68a7624079 feat(platform): expose kuberture as optional system package 
Register cozystack.kuberture PackageSource and include it in the system
bundle's optional set. The variant declares install-order dependencies
on cozystack.networking and cozystack.prometheus-operator-crds (CNI
must be ready before the controller pod schedules; the CRD must exist
before the chart's enabled-by-default ServiceMonitor is applied). It
deliberately does NOT depend on external-dns: kuberture creates
annotated headless Services regardless of whether external-dns is
installed, since external-dns is a downstream consumer of those
annotations rather than a runtime prerequisite. Operators who want DNS
records published enable an external-dns instance separately, matching
annotationPrefix per output to target a specific instance.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-14 16:49:00 +03:00
Andrei Kvapil 45e21a330f feat(networking): add ouroboros for hairpin-NAT under PROXY-protocol (#2561) 
## What this PR does

Adds
[`lexfrei/ouroboros@v0.7.0`](https://github.com/lexfrei/ouroboros/releases/tag/v0.7.0)
— a Go reimplementation of
[`compumike/hairpin-proxy`](https://github.com/compumike/hairpin-proxy)
— and wires it into cozystack as a *conditional default*: a single
`publishing.proxyProtocol: true` switch turns PROXY-protocol on at the
host ingress-nginx **and** auto-deploys ouroboros to fix the resulting
hairpin-NAT problem (intra-cluster traffic to the cluster's own public
hostnames arriving at ingress-nginx without the PROXY header it now
requires). Default stays `false` — clusters that do not run
PROXY-protocol get zero new resources, zero RBAC delta, zero behaviour
change.

The chart is platform-agnostic at v0.7.0: the controller and proxy
binaries auto-detect the cluster's DNS domain from `/etc/resolv.conf`
inside the pod and compose the rewrite target FQDN and the backend host
at runtime as `<svc>.<ns>.svc.<cluster-domain>`. Cozystack tenants on
`cozy.local`, kubeadm-bootstrapped clusters on `cluster.local`,
federations on `k8s.example.com`, custom-domain RKE2/k3s — all work
without operator override. Verified end-to-end on a `cozy.local`
cluster: controller logs `cluster-domain=cozy.local` at startup, chart
emits `--proxy-service-name + --proxy-service-namespace` (no chart-baked
`--proxy-fqdn`), Corefile picks up the rewrite block between BEGIN/END
markers, in-pod `dig <ingress-host>` returns the ouroboros-proxy
ClusterIP. Operators that prefer a chart-baked FQDN can still set
`controller.clusterDomain` explicitly — the explicit-override path is
preserved for non-Service backends and for split-horizon DNS topologies.

The chart runs in `coredns` mode for the host (the host CoreDNS is
Talos-managed and does not carry an `import` directive, so the
alternative would silently no-op there) and in `coredns-import` mode for
tenant deployments (tenants run cozystack-coredns, which carries the
import directive and ships an empty `kube-system/coredns-custom`
ConfigMap; ouroboros writes plugin-only `rewrite name` snippets into the
`ouroboros.override` data key without ever touching the chart-rendered
Corefile, so it does not race the chart that re-templates
`data.Corefile` on every Flux reconcile).

### Layout

* `packages/system/ouroboros/` — vendored chart from
`oci://ghcr.io/lexfrei/charts/ouroboros@0.7.0`. Cozystack-side defaults
pinned in `values.yaml`: `controller.mode=coredns` (host scenario),
`proxy.target.namespace=cozy-ingress-nginx` (override the chart-default
`ingress-nginx` namespace; chart-default
`serviceName=ingress-nginx-controller` matches as-is), Gateway-API
watching left at upstream's default of `false`. `proxy.target.host` is
left empty so the proxy composes the backend host at runtime. Image is
digest-pinned at
`0.7.0@sha256:478665e05dd0ffc4c1f7764320ea24b52251781884ddfd668d93a03e39d9094c`
and pulled directly from `ghcr.io/lexfrei/ouroboros` — this package is
intentionally not mirrored under `ghcr.io/cozystack/*`. The chart
manifest itself is also digest-pinned via
`OUROBOROS_CHART_DIGEST=sha256:f28bfac9fd7070b1f3357983ff5aad28c16115e7de1cfc16ee568a3b4cfc9d7e`
in the package Makefile, with a Chart.yaml-version vs. Makefile-version
assert that aborts the update target on lockstep drift. Helm-unittest in
`tests/` covers controller args, RBAC narrowing per mode, Gateway-API
gating, runtime-composition flag emission in both `coredns` and
`coredns-import` modes, image pin drift.
* `packages/system/coredns/templates/coredns-custom.yaml` +
`values.yaml` — empty `coredns-custom` ConfigMap as the chart-managed
`import` prerequisite, plus the `import /etc/coredns/custom/*.override`
plugin in the `.:53` server block + `extraVolumes`/`extraVolumeMounts`
mount at `/etc/coredns/custom/`. Same package serves host and tenant
CoreDNS, so this covers both layers without duplication. Helm-unittest
pins the rendered Corefile via a snapshot so future `make update` does
not silently drop the import directive. The empty rendered ConfigMap
deliberately omits `data:` — Helm three-way merge has no template-side
keys to compare against, so apiserver-side runtime writes (the
`ouroboros.override` data key) survive every chart upgrade. The
`notExists: data` assertion in `tests/coredns_custom_test.yaml` pins the
invariant.
* `packages/core/platform/sources/ouroboros.yaml` — new
`cozystack.ouroboros` PackageSource for the host install (`dependsOn:
cozystack.networking, cozystack.ingress-application`).
* `packages/core/platform/sources/kubernetes-application.yaml` —
`kubernetes-ouroboros` component registered against `system/ouroboros`
so the tenant HelmRelease can reference it.
* `packages/apps/kubernetes/templates/helmreleases/ouroboros.yaml` —
tenant-side HelmRelease deployed via the tenant admin-kubeconfig,
mirroring the cert-manager pattern. A `cozystack.defaultOuroborosValues`
helper flips `mode` back to `coredns-import` for tenants and mirrors
per-tenant `addons.gatewayAPI.enabled` into the controller (forcing
`--gateway-api` on a tenant without Gateway-API CRDs CrashLoops the
controller on `WaitForCacheSync`). `controller.clusterDomain` is
intentionally NOT pre-set — chart 0.7.0 auto-detects per-pod from
`/etc/resolv.conf` and composes the rewrite target at runtime, so the
same wrapper renders correctly on `cluster.local`, `cozy.local`, or any
custom tenant cluster-domain. Operators on a non-default cluster-domain
can still override via
`valuesOverride.ouroboros.controller.clusterDomain`. `dependsOn` grows a
conditional `gateway-api-crds` entry on the same gate. If
`addons.ouroboros.enabled` is true while `addons.ingressNginx.enabled`
is false, the HelmRelease fails the render with a clear error rather
than silently skipping — the hairpin fix has nothing to fix against
without ingress-nginx, and a no-op addon that quietly does nothing is
harder to debug than an explicit render-time error. The valuesOverride
gateway-api guard accepts every truthy spelling Helm/YAML emit (bool
`true`, string `"true"`/`"True"`/`"TRUE"`, numeric `1`, YAML
`yes`/`Yes`/`YES`/`on`/`On`/`ON`/`y`/`Y`) via
`toString`-then-`lower`-then-string-compare; direct bool `eq` would
error on type mismatch when an operator passes the value as a string
from `--set`, and case-insensitive folding catches `--set ...=On` /
`--set ...=True` that pure-lowercase comparison would miss. Tenant
disable is single-step: flipping `addons.ouroboros.enabled: false`
deletes the rendered HelmRelease, helm-controller runs `helm uninstall`
on the tenant, and the chart's pre-delete cleanup hook
(`coredns-cleanup-hook.yaml`, vendored at 0.7.0) nulls the
`ouroboros.override` key on the tenant's `kube-system/coredns-custom`
ConfigMap automatically. No render-time guard is wired on the tenant
side — gating on a `lookup` for the leftover HelmRelease would deadlock
the routine disable (the parent render runs before helm-controller
applies the missing-child diff, so the lookup would always find the
leftover at the moment of the flag flip). If the chart's hook fails to
land (controller pod stuck CrashLooping, ConfigMap RBAC drift, Job
timeout), operators recover manually via the documented tenant cleanup
recipe.
* `packages/apps/kubernetes/values.yaml` — new `OuroborosAddon` typedef
with `enabled` + `valuesOverride`. `make generate` re-emitted README,
JSON schema, the Go struct + deepcopy, and the kubernetes-rd
ApplicationDefinition openAPISchema in lockstep.
* `packages/core/platform/values.yaml` — new `publishing.proxyProtocol`
flag with extensive comment covering the upstream-LB precondition (the
L4 LB in front of ingress-nginx must already be injecting PROXY-v1
headers before the flag flips on, otherwise external traffic breaks),
the host-vs-tenant fan-out (the flag does not cascade into tenants), the
Talos re-render flap window, and the asymmetric disable path. The host
disable path: flipping back to `false` reverts ingress-nginx
PROXY-protocol config (flip the LB off first), but the host ouroboros
itself stays installed because its Package CR carries
`helm.sh/resource-policy: keep` — an explicit `kubectl delete
package.cozystack.io cozystack.ouroboros` (or `bundles.disabledPackages`
entry) actually triggers helm uninstall, at which point the chart's
pre-delete cleanup hook fires and patches the host Corefile
automatically. The values comment ships a manual `sed`-based cleanup
recipe as a fallback for the rare case where the chart's hook fails to
land (controller stuck in CrashLoop, ConfigMap RBAC drift, hook
timeout); the `publishing.proxyProtocolAcknowledgeUnclean` gate keys off
a `lookup` against the leftover Package CR to catch operators who flip
the flag back without confirming the cleanup landed.
* `packages/core/platform/templates/bundles/system.yaml` — flips between
the default-form and explicit-form `cozystack.ingress-application`
Package emission so the host ingress-nginx gets `use-proxy-protocol` +
`real-ip-header=proxy_protocol` injected only when the flag is true
(`use-forwarded-headers` and `compute-full-forwarded-for` are
deliberately *not* on the list — they would let any upstream proxy spoof
`X-Forwarded-For`); the same conditional emits `cozystack.ouroboros`.
Host Gateway-API watching is left at the upstream chart's default of
`false` because cozystack does not install Gateway-API CRDs cluster-wide
on the host.
* `hack/e2e-apps/ouroboros.bats` — auto-discovered smoke test for the
tenant addon (HelmRelease ready, `coredns-custom` ConfigMap present in
tenant, controller pod running, and after creating a TLS Ingress the
rewrite snippet shows up in the import ConfigMap; an in-tenant `dig`
from a netshoot pod also confirms CoreDNS actually serves the rewrite —
pod-readiness alone passes vacuously since the upstream chart ships no
readiness probes).

### Why not KEP-1860 `ipMode=Proxy`

Cilium's `kubeProxyReplacement: true` (cozystack default) drops the LB
frontend entirely when `ipMode != VIP`, breaking Service ingress for
L2/BGP-announced IPs. KEP-1860's Proxy mode is reserved for clusters
fronted by an external CCM-managed proxy LB. ouroboros is the correct
fix for L2/BGP-announce topologies that cozystack ships by default.

### Coverage

Covered by helm-unittest (host chart, coredns wrapper, tenant addon —
including the `kubernetesProvider` mock that exercises the lookup-driven
`proxyProtocolAcknowledgeUnclean` gate, the lookup-returns-nil
CI/dry-run path, the gateway-api truthy-spelling acceptance across
`true`/`"true"`/`"True"`/`"TRUE"`/`1`/`yes`/`on`, the assertion that
`--proxy-service-name` + `--proxy-service-namespace` are emitted instead
of a chart-baked `--proxy-fqdn` in both `coredns` and `coredns-import`
modes, the assertion that `--target-service-name` +
`--target-service-namespace` are emitted for the proxy backend, and the
`_namespace.etcd`-empty render gate) and a tenant e2e bats test that
creates a real Ingress, asserts the rewrite snippet lands in the import
ConfigMap, and confirms in-tenant DNS serves it (with an in-pod retry
loop tolerating CoreDNS reload latency). The host-level
`publishing.proxyProtocol: true` path is not e2e-tested in this PR — it
requires a PROXY-protocol-aware upstream LB, which the cozystack e2e
harness does not stand up. That path is exercised manually and by docs
review.

End-to-end runtime auto-detect was verified manually on a `cozy.local`
cluster (dev17): chart deployed at v0.7.0, controller logged
`cluster-domain=cozy.local`, the rewrite block landed in
`kube-system/coredns` Corefile pointing at
`test-ouroboros-proxy.test-ouroboros.svc.cozy.local.`, an in-pod `dig
<ingress-host>` resolved to the ouroboros-proxy ClusterIP on the first
attempt, and the chart's pre-delete hook restored the Corefile cleanly
on `helm uninstall`.

### Supply-chain note

Image and chart are pulled from a contributor's personal namespace
(`ghcr.io/lexfrei/...`). Existing precedent
(`packages/system/etcd-operator/Makefile` pulls from
`ghcr.io/aenix-io/...`) is similar but `aenix-io` is the
cozystack-adjacent org while `lexfrei` is one human. Worth a maintainer
call.

Pairs with cozystack/website#527 (operator docs).

### Release note

```release-note
feat(networking): add `publishing.proxyProtocol: true` to enable PROXY-protocol on host ingress-nginx and auto-deploy ouroboros (a Go reimplementation of compumike/hairpin-proxy) to solve the resulting hairpin-NAT problem; tenants can opt in per-cluster via `addons.ouroboros.enabled` (requires `addons.ingressNginx.enabled`). Default behaviour unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Ouroboros addon to address hairpin‑NAT for ingress‑nginx with
configurable enablement and tenant overrides.
* Added a packaged Ouroboros chart providing proxy, controller, CoreDNS
integration, external‑DNS and etc‑hosts modes.
* Introduced PROXY‑protocol gating with an explicit acknowledgement
flag.

* **Documentation**
* Expanded addon reference and platform docs with detailed
Ouroboros/PROXY‑protocol configuration and defaults.

* **Tests**
* Added extensive unit and e2e tests covering rendering, RBAC, uninstall
cleanup, and DNS behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-07 18:41:13 +02:00
Aleksei Sviridkin a9c1298b42 [gpu-operator] Update to v26.3.1 and add experimental vGPU variant (#2323) 
## What this PR does

Updates the NVIDIA GPU Operator from v25.3.0 to v26.3.1 and adds an
**experimental** `vgpu` variant for SR-IOV vGPU on Ada Lovelace and
newer GPUs.

Changes:
- Update vendored upstream chart to v26.3.1 (CDI enabled by default,
component version bumps).
- Pin chart version in Makefile for reproducibility.
- Add `values-vgpu.yaml` for the vGPU variant with `defaultWorkload:
vm-vgpu` so the chart activates the vGPU Manager DaemonSet without
per-node labelling.
- Add `vgpu` variant to PackageSource alongside the existing `default`
(passthrough).
- Rename `values-talos.yaml` → `values-passthrough.yaml` and pin
`defaultWorkload: vm-passthrough` for symmetry with the vgpu variant.
The overrides apply to any host OS using vfio-pci, not Talos
specifically.
- Refresh `hack/dcgm-default-counters.csv` to match the new
dcgm-exporter image tag (`4.5.1-4.8.0`) — adds `DCGM_FI_DEV_FB_RESERVED`
and other fields introduced upstream.
`hack/check-gpu-recording-rules.bats` passes against the new fixture.
- Add `hack/check-gpu-operator-variants.bats` that renders both variants
via `helm template` and asserts the pinned ClusterPolicy values, so the
next chart bump cannot silently un-pin them.
- Add internal documentation for the vGPU bring-up.

The `vgpu` variant assumes the user supplies a proprietary vGPU Manager
image via Package CR component values — the .run cannot be
redistributed.

### `vgpu` variant is experimental

NVIDIA's `vgpu-device-manager` walks `/sys/class/mdev_bus/`, which does
not exist on Ada Lovelace and Blackwell — the DaemonSet errors with "no
parent devices found" and is disabled by default. Until an SR-IOV-aware
controller is shipped, profile assignment is an out-of-band step (`echo
<id> > /sys/.../current_vgpu_type` per VF) that must be re-applied after
every node reboot. Without it, `permittedHostDevices.pciHostDevices`
reports zero allocatable resources. The variant ships so the chart bump
and the underlying skeleton (DaemonSet + image plumbing + docs) land
together; production deployments need an automated profile-assignment
mechanism in addition to this PR.

### KubeVirt requirement

KubeVirt advertises VFs via `permittedHostDevices.pciHostDevices` rather
than `mediatedDevices`. This requires kubevirt/kubevirt#16890 ("vGPU:
SRIOV support", merged 2026-04-10) which is `main`-only — released tags
`v1.6.x` / `v1.7.x` / `v1.8.x` do not include it. The patch is targeted
at the next minor KubeVirt release (v1.9.0); track the PR for the actual
release tag. Backports to released branches are not planned.

### Behavior changes inherited from upstream chart v25 → v26

The upstream chart flipped two defaults that are pinned back to `false`
here, so `default` (passthrough) deployments do not silently change
footprint on upgrade:

- `ccManager.enabled` was flipped to `true` upstream — would auto-enable
Confidential Computing on Hopper hardware (H100/H200/B100) after
upgrade.
- `vgpuDeviceManager.enabled` was flipped to `true` upstream — DaemonSet
would render even in passthrough mode.

Both stay opt-in via Package CR overrides. The new bats test guards the
pin.

Other inherited changes that are user-visible and worth pinning down
before upgrade:

- `driver.licensingConfig.configMapName` is deprecated upstream in
favour of `driver.licensingConfig.secretName`. The deprecation marker
lives in the CRD's `description` field (visible via `kubectl explain`) —
there is no helm-render-time warning, the chart template silently
prefers `secretName` and falls back to `configMapName`. Operators with
existing licensing overrides should migrate the key on the next chart
upgrade.
- `driver.version` upstream default jumped from `570.x` to `580.x`.
Invisible on the `default` variant (which sets `driver.enabled: false`),
user-visible for anyone running the upstream Ubuntu driver path.
- `vfioManager.image` switched from
`nvcr.io/nvidia/cuda:12.8.1-base-ubi9` to
`nvcr.io/nvidia/cloud-native/k8s-driver-manager:v0.10.0`. Affects every
`default` (passthrough) deployment on upgrade — different image,
different layer, different image-pull surface. Worth pre-mirroring for
air-gapped clusters.
- `dcgmExporter.image` switched to the `-distroless` flavour. Smaller
surface, fewer base-image CVEs, but no shell — operators who used to
`kubectl exec` into the exporter for ad-hoc debugging will need to
switch to `kubectl debug` or container ephemeral images.
- `CDI` (Container Device Interface) is enabled by default in v26.3.1;
this is upstream's recommended setting and is left as-is.
- `node-feature-discovery` subchart source switched from the GitHub
Pages Helm repo to `oci://registry.k8s.io/nfd/charts`. Air-gapped
installations need to mirror the new OCI URL.

### Talos

Talos is **not recommended** for the vGPU variant. NVIDIA does not
publicly distribute the vGPU guest driver — it requires NVIDIA
Enterprise Portal access — and Sidero closed siderolabs/extensions#461
noting that they cannot support vGPU "unless NVIDIA changes their
licensing terms or provides us a way to obtain, test, and distribute the
software". Building a Talos system extension that bakes in the vGPU
driver is therefore not feasible without a private fork that violates
the EULA. Passthrough (`default` variant) on Talos is unaffected.

### Release note

```release-note
[gpu-operator] Update NVIDIA GPU Operator to v26.3.1 and add experimental `vgpu` variant for SR-IOV vGPU on Ada Lovelace / Blackwell (L4, L40, L40S, B100; Ubuntu host; requires KubeVirt v1.9.0+ for `pciHostDevices` advertising). The variant ships skeleton support — profile assignment (`current_vgpu_type` per VF) is currently out of band and must be automated by the operator before production use. On the existing `default` variant: `ccManager` and `vgpuDeviceManager` upstream defaults are pinned back to `false` to preserve prior behavior on chart upgrade; `defaultWorkload` is now pinned to `vm-passthrough` (was unset, falling back to `container`); `driver.licensingConfig.configMapName` is deprecated by upstream in favour of `secretName`; `driver.version` upstream default moves 570 → 580; `vfioManager.image` switches from CUDA UBI9 to k8s-driver-manager (relevant for air-gapped mirrors); `dcgmExporter` image switched to the `-distroless` flavour. NFD subchart source switched to `oci://registry.k8s.io/nfd/charts`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Introduced vGPU support variant with end-to-end configuration guide,
validation procedures, and troubleshooting documentation

* **Updates**
* Upgraded GPU Operator to v26.3.1 and Node Feature Discovery to v0.18.3
* Improved component health checks and added PodDisruptionBudget
availability controls

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-07 11:18:36 +03:00
Aleksei Sviridkin 7f01c7eb32 feat(platform): single switch for host PROXY-protocol + ouroboros 
Adds publishing.proxyProtocol on the platform values: a single flag
that both turns PROXY-protocol on at the host ingress-nginx (via the
ingress-application Package CR's component values dict) and emits the
new cozystack.ouroboros Package CR for the host install. The two
pieces are coupled by design — turning PROXY-protocol on at
ingress-nginx without ouroboros breaks intra-cluster traffic to the
cluster's own public hostnames (kube-proxy / Cilium-KPR short-circuits
the LB IP, the connection arrives at ingress-nginx without the PROXY
header it now requires, and ingress-nginx closes the connection).

The new sources/ouroboros.yaml PackageSource declares the host chart
with dependsOn: cozystack.networking + cozystack.ingress-application
so install ordering is deterministic. The bundle template emits the
host ouroboros Package via the standard helper — the parent chart's
Gateway-API watching default of false stays in place, because cozystack
does not install Gateway-API CRDs cluster-wide on the host (system/
gateway-api-crds is referenced only from the kubernetes-application
PackageSource, tenant scope). Forcing --gateway-api on a host without
the CRDs would CrashLoop the controller on WaitForCacheSync against
missing GVRs.

The flag default is false. Clusters that do not run PROXY-protocol get
zero new resources, zero RBAC delta, zero behaviour change.

Disabling: flipping publishing.proxyProtocol back to false stops
emitting the cozystack.ouroboros Package CR, but every Package CR
carries helm.sh/resource-policy: keep — Helm leaves the existing
Package on the cluster so ouroboros stays installed and the live
Corefile rewrite block keeps pointing at the still-running
ouroboros-proxy Service. Flag-flip alone is idempotent and safe. To
actually uninstall ouroboros the operator removes the Package CR
explicitly (or adds the package name to bundles.disabledPackages) and
runs the documented Corefile cleanup recipe in the same window —
otherwise stale rewrites point at a deleted Service and in-cluster
DNS for hairpinned hostnames returns connection-refused. The values
comment ships the cleanup recipe (a JSON merge-patch on data.Corefile
that strips the BEGIN/END block while preserving labels, annotations,
and other data keys including any Talos-managed metadata) and
documents the upstream-LB precondition (the L4 LB in front of
ingress-nginx must already be injecting PROXY-v1 headers before the
flag flips on), the host-vs-tenant fan-out (the flag does not cascade
into tenant clusters; tenants opt in per-Kubernetes-CR via
addons.ouroboros.enabled), and the Talos re-render flap window
(machine-config pushes wipe the BEGIN/END markers; the controller
re-applies on next reconcile, ≤10 minutes by default).

Also drops the stale "distro-full" entry from the bundles.system.variant
options comment in core/platform/values.yaml — the variant was removed
in v1.0.1 and was never restored.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-06 06:48:07 +03:00
Aleksei Sviridkin 7ab940f309 feat(kubernetes): expose ouroboros as a per-tenant addon 
Adds the OuroborosAddon typedef, addons.ouroboros knob, and tenant
HelmRelease that deploys the cozy-ouroboros chart inside a tenant
Kubernetes cluster via the host's flux-helm-controller and the tenant
admin-kubeconfig — same shape as the cert-manager addon.

The HelmRelease overrides controller.mode back to coredns-import
because tenants run the cozystack-coredns wrapper (which carries the
`import /etc/coredns/custom/*.override` directive); the parent chart's
default of mode=coredns is the right shape only for the Talos-managed
host CoreDNS. controller.gatewayApi.enabled is mirrored from the
per-tenant addons.gatewayAPI.enabled — forcing --gateway-api on a
tenant without Gateway-API CRDs CrashLoops the controller on
WaitForCacheSync against missing GVRs. dependsOn grows a conditional
gateway-api-crds entry on the same gate.

controller.clusterDomain is intentionally NOT pre-set by the wrapper.
The chart auto-detects it inside the controller pod by parsing
/etc/resolv.conf — the search list of which is `*.<ns>.svc.<domain>`
and reflects exactly the zone the tenant CoreDNS actually serves
(governed by KamajiControlPlane.spec.networkProfile.clusterDomain
defaulting to `cluster.local`, and the cozystack-coredns wrapper
kubernetes-plugin zone, also `cluster.local`). Pre-setting from a
host-side fact like `_cluster.cluster-domain` would override the
auto-detect with a value that does not match what the tenant kube-dns
serves, silently emitting NXDOMAIN-bound rewrites. Operators that run
a tenant on a non-default cluster-domain can still override via
addons.ouroboros.valuesOverride.ouroboros.controller.clusterDomain.

If addons.ouroboros.enabled is true but ingressNginx is off, the
HelmRelease fails the render with a clear error rather than silently
skipping — the hairpin fix has nothing to fix against without
ingress-nginx, and a no-op addon that quietly does nothing is harder
to debug than an explicit render-time error.

The valuesOverride gateway-api guard accepts every truthy spelling
Helm/YAML emit (bool true, string "true"/"True"/"TRUE", numeric 1,
YAML "yes"/"Yes"/"YES"/"on"/"On"/"ON"/"y"/"Y") via
toString-then-lower-then-string-compare; direct bool eq would error
on type mismatch when an operator passes the value as a string from
--set, and case-insensitive folding catches `--set ...=On` /
`--set ...=True` that pure-lowercase comparison would miss.

The acknowledgeUncleanRemoval gate is defence-in-depth for the rare
edge case where helm uninstall fails to land cleanly (tenant
unreachable, hook timeout, manual kubectl delete bypassing helm) —
in the normal path the upstream chart's pre-delete cleanup hook
(coredns-cleanup-hook.yaml, vendored at 0.6.0) nulls the
ouroboros.override key automatically when the HelmRelease is
uninstalled. The gate only blocks render when a leftover HelmRelease
on the host kube-apiserver shows the previous install is still
sitting on the cluster.

The component is also registered in the kubernetes-application
PackageSource (as kubernetes-ouroboros at path system/ouroboros) so
the tenant HelmRelease's ExternalArtifact resolves.

make generate re-emitted README.md, values.schema.json, the Go struct
+ deepcopy, and the kubernetes-rd ApplicationDefinition openAPISchema
in lockstep.

helm-unittest covers: render gating on enabled flag, hard-fail when
ingress-nginx is off, namespaces and labels, chartRef, dependsOn
cilium/coredns/ingress-nginx + conditional gateway-api-crds, default
coredns-import mode, valuesOverride precedence, gateway-api flag
mirroring in both directions, gateway-api truthy-spelling acceptance
(including case-insensitive on/On/True/TRUE), the lookup-driven
acknowledge gate (kubernetesProvider mock asserts the failedTemplate
path when a prior HelmRelease still exists on the host), the
lookup-returns-nil CI/dry-run path (render proceeds when the cluster
mock has no objects), the _namespace.etcd-empty render gate, and
that controller.clusterDomain is NOT pre-set in the rendered
HelmRelease (so chart auto-detect runs).

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-05-06 06:48:06 +03:00
Andrei Kvapil b5a1ae57de feat(etcd): update operator to v0.4.3 and expose S3 backup schedules (#2428) 
## What this PR does

Bumps the vendored `etcd-operator` chart from `v0.4.2` to `v0.4.3` and
wires the new backup machinery through the cozystack `Etcd` app API.

Upstream v0.4.3 ships two new CRDs — `EtcdBackup` (one-shot snapshot)
and `EtcdBackupSchedule` (periodic snapshots) — along with the
`EtcdCluster.spec.bootstrap` restore field, a new `OPERATOR_IMAGE` env
on the operator deployment, and a replacement of the deprecated
`gcr.io/kubebuilder/kube-rbac-proxy` image. The chart portion of this PR
is a clean re-vendor via `make update` — no local patches.

On top of the bump, `packages/extra/etcd` exposes a new optional
`backup.*` values block. When `backup.enabled=true`, the chart renders
an `EtcdBackupSchedule` referencing the platform `EtcdCluster/etcd` and
a `Secret/etcd-s3-creds` with the S3 credentials expected by the
backup-agent. The values field names (`destinationPath`, `endpointURL`,
`s3AccessKey`, `s3SecretKey`, `schedule`) mirror the existing `postgres`
backup pattern for consistency. The chart `fail`s at template render if
any of the four required S3 fields are empty or if `destinationPath` is
malformed (missing `s3://` prefix, `//` segments, no bucket).

Test coverage: 15 `helm-unittest` cases across
`tests/backup-secret_test.yaml` and
`tests/etcd-backup-schedule_test.yaml` (wired into `make test`, picked
up by `hack/helm-unit-tests.sh`), plus three bats cases in
`hack/e2e-apps/etcd.bats` covering the baseline `Etcd` app, the `backup:
{}` default-off path, and the full `backup.enabled=true` case that
verifies the `EtcdBackupSchedule` materializes and the operator
generates the downstream `CronJob`.

`packages/core/platform/sources/etcd-operator.yaml` now sets
`upgradeCRDs: CreateReplace` on the etcd-operator component, so Flux
applies the new CRDs on upgrade of the chart instead of silently
skipping them.

Depends on #2427.

### Scope deferred to follow-ups

- The one-shot `EtcdBackup` CRD is vendored but not exposed through this
chart — users can still hand-apply the CRD directly.
- `EtcdCluster.spec.bootstrap` (restore from an existing snapshot) is
likewise vendored but not exposed. Both gaps are called out in
`packages/extra/etcd/README.md`.

### Release note

```release-note
feat(etcd): update etcd-operator to v0.4.3 and expose S3 backup schedules on the Etcd app. Set `backup.enabled=true` along with `backup.s3AccessKey`, `backup.s3SecretKey`, `backup.destinationPath`, and `backup.endpointURL` to schedule periodic snapshots.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Scheduled S3 backups for etcd clusters (configurable cron, generated
S3 credentials) and restore/init from S3 or PVC via new backup
resources.

* **Tests**
* New end-to-end tests for etcd lifecycle and backup scenarios; added
Helm unit tests for backup templates and validations.

* **Documentation**
  * Backup configuration guide and parameter reference added.

* **Chores**
* Operator chart version bump; RBAC, webhooks, and deployment wiring
updated; added chart test target.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-01 13:16:08 +02:00
Aleksei Sviridkin e9bb00c074 fix(gpu-operator): set defaultWorkload=vm-vgpu, fix imagePullSecrets path, mark vgpu variant experimental 
Addresses second review pass. Three blockers + one polish:

* values-vgpu.yaml: set sandboxWorkloads.defaultWorkload=vm-vgpu.
  Without this the upstream default 'container' applies on any node
  without the per-node nvidia.com/gpu.workload.config label, the
  operator selects the container workload (driver/devicePlugin
  DaemonSets), and those are disabled in this variant — so nothing
  runs on the node and vgpu-manager never starts. The cluster-wide
  default makes the variant functional out of the box; per-node
  labels still override it for mixed clusters.

* docs/gpu-vgpu.md: imagePullSecrets is per-component on this chart
  (vgpuManager.imagePullSecrets, driver.imagePullSecrets, …). The
  Package CR example previously placed the field at gpu-operator: root
  where it would be silently ignored at render. Move it under
  vgpuManager: where the chart actually reads it.

* docs/gpu-vgpu.md, release note: mark the vgpu variant as
  experimental and call out the missing profile-assignment loop
  explicitly. Until an SR-IOV-aware controller writes
  current_vgpu_type per VF (and re-applies after every reboot —
  PCIe re-enumeration resets the value), the variant is not safe
  for production. The docs now lead with that warning instead of
  burying it in a caveat.

* Rename packages/system/gpu-operator/values-talos.yaml to
  values-passthrough.yaml. The overrides (sandboxWorkloads.enabled,
  driver and devicePlugin disabled) apply to any host OS using the
  vfio-pci passthrough path — not Talos specifically. Update
  PackageSource ref accordingly. Comment in the file documents the
  rename so future readers understand the prior name.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
 2026-04-30 14:52:56 +03:00