Commit Graph
3257 Commits
Author SHA1 Message Date
IvanHunters a537bb647b fix(kubeovn-webhook): harden certificate reloader
Address review of the reload path:
- Observe the cert file mtime before reading the key pair, so a Secret
  swap racing the read is retried on the next handshake instead of being
  cached under a newer mtime and missed.
- Record the load attempt regardless of outcome, so a persistently
  unreadable file is retried only once its mtime advances, not on every
  handshake.
- Detect change with mtime inequality instead of a strictly-newer
  comparison, catching equal-mtime replacements and backward clock steps.
- Log stat failures instead of swallowing them, so a broken mount is not
  invisible until expiry.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 15:24:17 +03:00
IvanHunters 791d99d8e5 fix(kubeovn-webhook): widen serving cert renewBefore to 720h
renewBefore: 24h on a one-year certificate leaves a one-day window for
cert-manager to rotate the Secret and for the webhook to pick it up.
Widen it to 720h (30 days) so renewal happens well ahead of expiry.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 15:04:43 +03:00
IvanHunters 44d9f534a9 fix(kubeovn-webhook): reload serving certificate on cert-manager renewal
The webhook loaded its TLS key pair once at startup and served it as a
static tls.Config.Certificates entry for the lifetime of the process. A
cert-manager renewal of the backing Secret was never picked up by a
running pod, so once the certificate expired (about a year after install)
the pod kept presenting the expired certificate. The kube-apiserver's TLS
call to the webhook then failed and, because the MutatingWebhookConfiguration
uses failurePolicy: Fail, every pod creation in tenant namespaces was
rejected (including virt-launcher pods, blocking all VMIs).

Serve the certificate through a reloading tls.Config.GetCertificate
callback that re-reads the key pair when the mounted files change and
keeps serving the last good certificate on a failed reload. Add a
regression test that fails against the static implementation and passes
once the certificate is reloaded.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-08-05 15:04:35 +03:00
myasnikovdaniil ee26a50f0e fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix (#3510)
## What this PR does

Raises the CloudNativePG operator image **and its CRDs together** to
1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 /
cloudnative-pg#9981). Symptom: after a simultaneous `resources` + `size`
change on a single-instance Postgres, the operator deletes the sole
primary Pod, classifies the PVC as `resizing`, and never recreates the
Pod — the cluster wedges with zero instances and the disk FS resize
never completes.

Upstream publishes a CloudNativePG chart only per minor (`.0`/`.1`), so
there is no 1.28.2 chart on the 1.28 line. Following the pattern of
#3526 / #3528, `patches/cloudnative-pg-1.28.2.patch` (applied by `make
update`) raises the vendored chart's `appVersion` and CRDs from 1.28.1
to 1.28.2 in lockstep. The operator image follows `appVersion`, so there
is no `image.tag` pin (keeping operator and CRDs aligned and avoiding
#3479).

Verified #3479-safe: there are no CRD `status` field changes between
1.28.1 and 1.28.2 — only the `extensions` spec grows. Backportable to
release-1.6 as-is (same 0.27.1 base chart).

### Release note

```release-note
fix(postgres-operator): align CloudNativePG operator and CRDs to 1.28.2, fixing the PVC resize-deadlock that wedged single-instance PostgreSQL clusters after a simultaneous resources+size change
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Added support for PostgreSQL extension configuration in catalog and
cluster schemas.
  - Made extension image settings optional where appropriate.

- **Bug Fixes**
- Updated CloudNativePG to version 1.28.2, including the PVC resize
deadlock fix.
  - Kept the operator image and CRDs aligned at the same version.

- **Documentation**
  - Clarified versioning and chart update guidance.

- **Tests**
  - Added checks confirming operator image and CRD version alignment.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-04 20:09:19 +05:00
Alexey Artamonov bcb36262ef fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix
Raises the CloudNativePG operator image and its CRDs together to 1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 / #9981): after a simultaneous resources+size change the operator deletes the sole primary Pod, leaves the PVC in the resizing class, and never recreates the Pod, wedging the cluster.

Upstream publishes a chart only per minor .0/.1, so there is no 1.28.2 chart on the 1.28 line. Following #3526/#3528, patches/cloudnative-pg-1.28.2.patch (applied by make update) raises the vendored chart's appVersion and CRDs from 1.28.1 to 1.28.2 in lockstep; the operator image follows appVersion, so no image.tag pin. Verified #3479-safe: no CRD status field changes between 1.28.1 and 1.28.2 (only the extensions spec grows). Backportable to release-1.6.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-08-04 17:31:05 +03:00
myasnikovdaniil a7149d11a3 chore(etcd-operator): bump etcd-operator to v0.5.4 (#3529)
## What this PR does

Bumps the cozystack etcd-operator packages from **v0.5.3 to v0.5.4**.

`v0.5.4` is a controller bug-fix release — no API, RBAC or values
changes:

- fix(controllers): derive `--initial-cluster-state` from phase, not
from the seed
([cozystack/etcd-operator#355](https://github.com/cozystack/etcd-operator/pull/355))
- fix(controllers): stop exempting the bootstrap seed from self-heal
([cozystack/etcd-operator#354](https://github.com/cozystack/etcd-operator/pull/354))
- fix(controllers): extend crash-loop self-heal to memory members
([cozystack/etcd-operator#352](https://github.com/cozystack/etcd-operator/pull/352))
- fix(controllers): switch the PDB from `maxUnavailable` to
`minAvailable`
([cozystack/etcd-operator#351](https://github.com/cozystack/etcd-operator/pull/351))

Changes in this repo:

- `packages/system/etcd-operator/Chart.yaml` — `appVersion: v0.5.3 →
v0.5.4`. The manager image tag defaults to `.Chart.AppVersion`
(`values.yaml` keeps `tag: ""`), so this reimages the controller to
`ghcr.io/cozystack/etcd-operator:v0.5.4`.
- `packages/system/etcd-operator/Makefile`,
`packages/system/etcd-operator-crds/Makefile` — `ETCD_OPERATOR_REF:
v0.5.3 → v0.5.4`.
- `packages/system/etcd-operator-crds/templates/etcdmembers.yaml` —
re-vendored at v0.5.4 via `make update`. Description-only change to the
`/scale` `replicas`/`selector` field docs, tracking the PDB
`minAvailable` fix. `etcdclusters` and `etcdsnapshots` are
byte-identical to v0.5.3.
- `templates/rbac.yaml` intentionally left as-is:
`manager-role-rules.yaml` is byte-identical between v0.5.3 and v0.5.4.

**Upgrade path (PDB switch, cozystack/etcd-operator#351).** This is the
one change that rewrites live objects: the operator moves each
EtcdCluster's PodDisruptionBudget from `maxUnavailable` to
`minAvailable`. Setting both fields is invalid, but upstream
`reconcilePDB` handles the migration — it treats a surviving
pre-migration `maxUnavailable` as divergence and explicitly clears it
(`MaxUnavailable = nil`) before writing `MinAvailable`, so existing
clusters are reconciled cleanly on upgrade rather than wedging their
PDB.

Verified locally: `helm lint` and `helm template` pass for both
packages; rendered manager image resolves to
`ghcr.io/cozystack/etcd-operator:v0.5.4`. These `packages/system/*`
packages have no `generate:` target and no `values.schema.json`, so
there is nothing for `make generate` to regenerate.

### Screenshots

Not applicable — no UI changes.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` file-by-file
against the diff:

- The diff touches only `packages/system/etcd-operator*` — no
`packages/apps/**` or `packages/extra/**` add/rename/remove, no
`packages/core/platform` or `installer` values, no Talos bump, no
asset-name or dev-tooling change → **website / ansible-cozystack** not
reached.
- The CRD edit is description-only, inside etcd-operator's own
`etcd-operator.cozystack.io` CRDs — not the provider's hand-typed
`Package`/`Plan`/`RestoreJob` types, and no new managed app →
**terraform-provider-cozystack** not reached.
- No `hack/` change, no `packages/system/<name>-rd/cozyrds/**`, no
`ApplicationDefinition` CRD / `chartRef.kind` enum change, no
`cozyhr`/`package.mk` contract change, no telemetry-metric or
proxy-label rename → **ccp / talm / cozyhr / cozy-proxy /
telemetry-server / examples / external-apps-example** not reached.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
chore(etcd-operator): bump etcd-operator to v0.5.4 (controller bug-fixes: PDB switched to minAvailable, crash-loop self-heal extended to memory members, --initial-cluster-state derived from phase, seed no longer exempt from self-heal)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Updates**
  - Updated the etcd operator to version v0.5.4.
- Clarified scale-related resource descriptions, including replica
counts, selectors, and disruption budget behavior.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-04 18:44:00 +05:00
Andrey Kolkov 22c1df8b94 test(etcd-operator): pin deployment unittest to v0.5.4
The manager image tag derives from .Chart.AppVersion, so bumping appVersion
to v0.5.4 without updating the two image literals in the deployment unittest
broke the suite (helm-unittest gates the PR via `make unit-tests`). Update
both asserts (manager container image and the agent OPERATOR_IMAGE env) to
ghcr.io/cozystack/etcd-operator:v0.5.4. Suite is green again (18/18).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-04 13:09:25 +04:00
Andrey Kolkov 19f969b2f8 chore(etcd-operator): bump etcd-operator to v0.5.4
Bump the cozystack etcd-operator packages from v0.5.3 to v0.5.4. The
release is a controller bug-fix set with no API, RBAC or values changes:

- fix(controllers): derive --initial-cluster-state from phase, not from
  the seed (cozystack/etcd-operator#355)
- fix(controllers): stop exempting the bootstrap seed from self-heal
  (cozystack/etcd-operator#354)
- fix(controllers): extend crash-loop self-heal to memory members
  (cozystack/etcd-operator#352)
- fix(controllers): switch the PDB from maxUnavailable to minAvailable
  (cozystack/etcd-operator#351)

Adaptations:
- etcd-operator/Chart.yaml: appVersion v0.5.3 -> v0.5.4 (the manager image
  tag defaults to .Chart.AppVersion, so this reimages the controller).
- etcd-operator/Makefile, etcd-operator-crds/Makefile: ETCD_OPERATOR_REF
  v0.5.3 -> v0.5.4.
- etcd-operator-crds/templates/etcdmembers.yaml: re-vendored at v0.5.4 via
  `make update`; description-only change to the /scale replicas/selector
  fields tracking the PDB minAvailable fix. etcdclusters/etcdsnapshots
  unchanged.
- templates/rbac.yaml left as-is: manager-role-rules.yaml is byte-identical
  between v0.5.3 and v0.5.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-08-04 12:41:43 +04:00
Myasnikov Daniil 4e9c5ce8ca fix(kubernetes): render the talos-reconcile Job for the default md0 group
The Job in templates/talos/talos-reconcile-job.yaml is the only
producer of the TalosConfigTemplate that each worker
MachineDeployment names in spec.template.spec.bootstrap.configRef;
cluster.yaml renders the reference but deliberately not the object.
Its loop read .Values.nodeGroups directly, while the
MachineDeployment loop reads the effective set through the
kubernetes.nodeGroups helper.

For a cluster that supplies no nodeGroups the two disagree: the
helper's else-branch emits the built-in md0 group, so the md0
MachineDeployment renders, but the raw map is empty and no Job
renders at all. Every Machine the cluster-autoscaler adds to md0
then blocks indefinitely on a TalosConfigTemplate that nothing will
ever create, and the KamajiControlPlane spec.network.certSANs patch
the same Job performs is skipped too.

Range over the helper so the Job set tracks the MachineDeployment
set exactly. No change for a cluster that declares its own groups,
and md0 stays removable: the helper's if-branch keeps a
user-supplied map authoritative.

The gap survived because every helm-unittest fixture and e2e suite
declares md0 explicitly. tests/nodegroups_default_test.yaml does
render the empty-nodeGroups case but lists only
templates/cluster.yaml, so it never looked at the Job. The new
suite pins both halves of the helper contract and fails without
this fix.

Fixes #3504

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-08-04 13:40:36 +05:00
myasnikovdaniil 3f967ec7e0 fix(platform): restore the isp-hosted dependency closure (#3391)
## Problem

Fixes #3376. The `isp-hosted` preset shipped `bundles.system.enabled:
false` while `paas` and `naas` were enabled, so it enabled an app
catalog whose dependency closure it simultaneously disabled. Every
emitted Package sat in `DependenciesNotReady`, the `*-rd` HelmReleases
failed with `no matches for kind "ApplicationDefinition" in version
"cozystack.io/v1alpha1"`, and the `cozystack-platform` HelmRelease never
went Ready.

Rendering the preset at `origin/main` shows the scale: **24 emitted
Packages, 24 of them blocked**, by 6 missing dependencies — two more
than the issue lists (`cert-manager` and `prometheus-operator-crds` are
also absent). `isp-full` renders 75 Packages with a fully satisfied
closure, which is a clean control.

## Root cause

`templates/bundles/system.yaml` is gated end to end on
`bundles.system.enabled`, but it carries two separable things: a
variant-selected **data-plane head** (kubeovn/cilium/linstor on
`isp-full`, a noop networking Package on `isp-hosted`) and a
variant-agnostic **control-plane body** — `cozystack-engine`, the sole
source of the `application-definition-crd` component, plus
`cert-manager`, `prometheus-operator-crds`, `postgres-operator` and
`objectstorage-controller`.

"Hosted" means the host already provides CNI, storage and
virtualisation, so only the data-plane head should be dropped. Disabling
the whole bundle also deleted the control-plane body that every variant
needs.

Three things show the disable was the mistake rather than the intent:
the preset still sets `variant: "isp-hosted"`; `system.yaml` contains an
`isp-hosted` branch (noop networking plus default scheduler) that could
never render; and `iaas.yaml` already refuses to render on this variant,
encoding the same "system bundle minus iaas" contract. The preset has
never worked — it was born this way in `2d022e38e` (January 2026), the
refactor that replaced `bundles/paas-hosted.yaml`, whose contents
confirm hosted was always meant to ship the engine, `cert-manager` and
`prometheus-operator-crds`.

## Changes

1. **`values-isp-hosted.yaml`** — enable the system bundle. Hosted now
emits 53 Packages with every `dependsOn` satisfied, installing 59
components against the 60 of the pre-refactor `paas-hosted` bundle, with
no LINSTOR, KubeVirt, CAPI or Kamaji leakage.
2. **Move `cozystack.bucket-application`** from the iaas bundle into the
system bundle. Its own `dependsOn` closure is `networking` +
`objectstorage-controller` + `cozystack-engine`, none of them iaas,
while `cozystack.backupstrategy-controller` — a *default* package of the
system bundle — hard-depends on it. Any system-on/iaas-off configuration
therefore wedged `backupstrategy-controller`; both `isp-full` presets
masked this by enabling iaas. This is a second, independent bug not
reported in the issue.
3. **Guards** — `paas`, `naas` and `iaas` now also require
`bundles.system.enabled`. The pre-existing checks looked only at
`bundles.system.variant`, which keeps its value when the bundle is
disabled, so they passed on exactly this configuration. `iaas` had the
identical hole (12 Packages, 6 unsatisfied deps). This turns a silent
multi-hour `DependenciesNotReady` hang into a render-time failure that
names the remedy.
4. **Two helm-unittest suites** — one pinning the dependency closure the
catalog needs, one pinning the guards.

## Verification

- All four operator-registered variants render dependency-closed:
`default` 0 Packages, `isp-hosted` 53, `isp-full` 75, `isp-full-generic`
75. The two `isp-full` variants are byte-identical before and after, so
the bucket move is a no-op for them.
- Hosted is a strict subset of full with zero data-plane leakage, and
its exclusions match the pre-refactor hosted bundle.
- 23 suites / 97 tests pass in `packages/core/platform`; repo-wide
`hack/helm-unit-tests.sh` is green.
- Every change was **mutation-tested** — reverting each one in turn
makes the suite fail, so the tests are not vacuous.
- The workaround documented in #3376 (`paas`/`naas` disabled) still
renders cleanly; the new guards do not fire on it.
- Hosted runtime spot-checks: `cozystack-api` is a Deployment with only
a *preferred* control-plane affinity plus blanket tolerations, and
`cozystack-scheduler` uses its `default` variant (plain Deployment, no
LINSTOR extender) — both schedule fine on a cluster with no
control-plane nodes.

## What this does not fix

This restores the dependency closure. It is **not** by itself sufficient
for a fully converged hosted install. Review surfaced further
hosted-unaware defaults that the dead bundle had been masking, filed
separately:

- #3387 — `apps/tenant` renders 14 `cilium.io/v2` policies with no
`.Capabilities` guard, so `tenant-root` fails on a non-Cilium cluster.
This is the blocker for actual hosted convergence, and it needs a
deliberate decision rather than a reflexive capability guard, because
those policies implement tenant network isolation.
- #3388 — `extra/monitoring` defaults `logsStorages` to the LINSTOR-only
`replicated` StorageClass.
- #3389 — `bundles.disabledPackages` can still break the closure with no
render-time error; the guards here cover the bundle flags only.
- #3390 — nothing in CI exercises `isp-hosted`, which is why all of the
above accumulated undetected.

## Upgrade note

Existing `*-application` Package CRs carry `helm.sh/resource-policy:
keep`. An operator who applied the #3376 workaround but did not also
delete the retained Package CRs (as that issue instructs) will find them
install once this change makes their dependencies Ready, since the
reconciler re-enqueues dependents on readiness. Deleting the unwanted
Packages before upgrading avoids that.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added fail-fast validation that requires the **System bundle** when
enabling **IaaS**, **NaaS**, or **PaaS** to prevent incomplete catalogs.
* Updated bundle composition so the **bucket application** is provided
by the **System bundle**.
* Enabled the **System bundle** by default for the `isp-hosted` variant.
* **Bug Fixes**
* Improved error messaging to clearly indicate the required
bundle/variant configuration.
* **Tests**
* Added Helm unittest coverage for rendering-gate failures,
dependency/package expectations, and hosted data-plane exclusions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-08-03 21:27:56 +05:00
Timofei Larkin e5d4940338 chore(codeowners): don't gate per-app generated files on architect review
api/apps/v1alpha1/*/types.go and packages/system/*/cozyrds/*.yaml are
entirely produced by `make generate` from each app's own values.yaml,
but the /api/ codeowner rule pulled every app's regenerated types.go
into an architect-only review, regardless of which area actually
changed. Correctness there is already enforced by the pre-commit
make-generate CI hooks, not by human review, so list no owner for
these generated paths and let that check carry the weight instead.

Also fixes a stale api/.gitattributes pattern that never matched any
real file (zz_generated_deepcopy.go vs the actual
zz_generated.deepcopy.go), and adds a matching linguist-generated
attribute for the cozyrds ApplicationDefinition manifests, so both
collapse in the PR diff view like the Go files already do.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-08-03 14:08:36 +03:00
myasnikovdaniil 10554f9dce [cozystack-scheduler] Publish SchedulingClass demo (#2252)
## What this PR does

Publish the scripts used for running the Cozystack scheduler demo at the
community meetup on 19.03.2026.

### Release note

```release-note
[cozystack-scheduler] Add demo scripts to the cozystack-scheduler system
packages.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added a complete set of interactive Bash demo scripts covering
scheduling classes: creating scheduling classes and tenants, colocating
pods on a single node, spreading replicas evenly, enforcing
one-pod-per-node anti-affinity, and pinning workloads to a specific
node.
* Included guided scaling steps plus cleanup and teardown scripts to
remove demo resources.

* **Documentation**
* Introduced a shared demo helper with colored, step-by-step terminal
logging and Kubernetes polling to wait for pods and Redis readiness,
improving clarity and consistency across demos.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-30 14:20:18 +05:00
Timofei Larkin 337bb4933e [cozystack-scheduler] Publish SchedulingClass demo
## What this PR does

Publish the scripts used for running the Cozystack scheduler demo at the
community meetup on 19.03.2026.

### Release note

```release-note
[cozystack-scheduler] Add demo scripts to the cozystack-scheduler system
packages.
```

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-07-30 11:03:21 +03:00
Andrei Kvapil 3dc0f56477 feat(kube-ovn): build the chart and image in-tree, retiring the wrapper repo (#3460)
## What this PR does

Brings the `kubeovn-chart` wrapper into the tree, so
`cozystack/kubeovn-chart` can be archived.

Until now `packages/system/kubeovn` pulled a chart tarball from the
latest `cozystack/kubeovn-chart` tag at `make update` time, and that
repository also built and pushed the `kubeovn` image. The wrapper
existed only to cache the upstream chart; with the current caching
approach that reason is gone, and the split cost a second repository, a
second release step, and a package whose provenance a reader cannot
follow without leaving the tree.

The package is now shaped like `packages/system/cilium`:

- `update` pulls the upstream `kubeovn/kube-ovn` chart directly and
applies `patches/mtu.diff`, then syncs the version into the Dockerfile.
The MTU patch already carried monorepo-relative paths, so it applies
unchanged.
- `image` builds `images/kubeovn` (carrying the existing
`fix-keepvmip-portgroup` patch) and pins the digest into `values.yaml`,
using the shared `image-tags` / `cache-args` helpers.

It also carries `cozystack/kubeovn-chart#5`, which never got a tag:
`OVN_IPSEC_KEY_DIR` is pinned under `/var/lib`. The kube-ovn default
sits in `/etc/origin`, which a Talos node cannot create, so enabling
IPsec left the node without ESP security associations.

### The nightly host rewrite had to be fixed first

Moving the image build here activates a gap recorded in
`docs/agents/image-refs.md`. The nightly host rewrite is a literal
`<src-registry>/` substring replace, and kubeovn keeps its host in
`global.registry.address` with the bare repository in a sibling key — no
trailing slash, no match. While the image came from a separate
repository the baked reference carried the public host and the ownership
filter skipped it; once CI builds the image here it stamps the
build-registry host into that key, and the result is an image mirrored
to the public registry while the published tree keeps pointing at the
private build registry.

The second commit adds an end-of-line-anchored expression that fires
only when the source registry is the complete scalar value, so a
contiguous reference — which continues past the host — is still handled
by the first expression and the two can never both fire on one
reference. `keycloak-operator` splits at a different boundary and stays
unfixed; that is documented, and dormant while no CI path rebuilds it.

Verified: `hack/nightly-mirror_test.bats` passes with the new case and
the case fails without the fix; `promote-retag` and
`promote-rewrite-tags` suites unaffected; `helm template` and `helm
unittest` on the package pass. The pre-existing `helm lint` complaint
about a missing `kube-ovn` dependency reproduces on `main` and is left
alone.

### Follow-ups this PR does not do

- `cozystack/kubeovn-chart` should be archived once this merges, with a
pointer here. Its open PR #5 is carried by this change and needs no
separate landing.
- `cozystack/website` links the Kube-OVN license at
`cozystack/kubeovn-chart` in `operations/configuration/licenses.md` and
`guides/platform-stack/_index.md` (all versions, including `next`).
Archived repositories stay readable so nothing breaks, but the link
should move upstream or in-tree. **No follow-up has been opened yet** —
flagging rather than ticking the box.

### Screenshots

N/A — build and packaging change, no user-facing or UI impact.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

Walked the trigger map against the diff. The website row matches — not
through the app-list or platform-component triggers, but because two of
its pages link the repository being archived. Left unticked because no
follow-up exists there yet; a human should decide whether the license
link moves to `kubeovn/kube-ovn` or to this package. No other row
matches: the change touches no `packages/apps/` or `packages/extra/`
package, no `values.schema.json`, no `ApplicationDefinition`, no
platform variant, and no node prerequisite.

### Release note

```release-note
feat(kube-ovn): the Kube-OVN chart and image are now built in this repository instead of cozystack/kubeovn-chart. OVN_IPSEC_KEY_DIR now defaults to /var/lib/ovs_ipsec_keys, so enabling IPsec on Talos nodes no longer leaves them without ESP security associations.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Updated Kube-OVN image build to use upstream chart metadata and keep
registry/version alignment during `update`.
  - Added conditional MTU support for the OVN CNI DS via chart values.
- Introduced `OVN_IPSEC_KEY_DIR` to standardize the OVN IPsec keys
directory.

- **Bug Fixes**
- Preserved VM networking port-group assignments during live migration
when an alive virt-launcher sibling pod exists.
- Improved nightly mirror host rewriting for split and standalone
registry host formats.

- **Tests**
- Added coverage for the end-of-line “whole scalar value” host rewrite
case.

- **Documentation**
- Refreshed “Known gaps” guidance for nightly mirror host rewriting
behavior and limitations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-29 18:19:41 +02:00
myasnikovdaniil 5d420475f4 refactor(ci): drop the vestigial /tmp workspace from e2e (both workflows) (#3454)
## What this PR does

The E2E job's `if: always()` cleanup ran `rm -rf /tmp/$SANDBOX_NAME`
with no check that `SANDBOX_NAME` was ever set. `Set sandbox ID` carries
no condition, so it inherits the implicit `success()` every
unconditional step gets: any failure ahead of it — a conflicting
`pr.patch`, a `download-artifact` miss, `Checkout code`, the
release-path `curl` — or a cancellation that reaches the job after it
has started (the job condition is `!cancelled()`) skips it, and
`SANDBOX_NAME` is never written to `$GITHUB_ENV`. The `if: always()`
steps then run anyway and interpolate the empty value; `Remove
workspace` is the destructive one, expanding to `rm -rf /tmp/` and
failing the cleanup step against root-owned entries in the runner's
`/tmp`. This is not a "re-run skips the setup" bug: a re-run restarts
the job at step one and re-runs the unconditional `Set sandbox ID` — the
trigger is incomplete initialization, by any of the routes above.

The `/tmp/$SANDBOX_NAME` workspace is a vestige of the retired multi-job
self-hosted-runner design, where several `runs-on: [self-hosted]` jobs
shared no state and rendezvoused through a `sha256(REPO:WORKFLOW:REF)`
key, with a per-run `cp -r` of the checkout into `/tmp`. Since CI moved
onto the ephemeral pool (`runs-on: oracle-vm-32cpu-128gb-x86-64`) it is
a single job: there is no cross-job rendezvous and the checkout is never
reused across runs, so copying it into `/tmp` and removing it afterwards
is dead weight. (An earlier review justified the change by "wiping the
entire `/tmp` of the self-hosted runner" — that describes the topology
we retired; the hazard is real, but not for that reason.)

So this drops the workspace rather than guarding the `rm`. Both the PR
(`pull-requests.yaml`) and nightly (`nightly.yaml`) E2E jobs now run in
`$GITHUB_WORKSPACE`: the `Prepare workspace` and `Remove workspace`
steps are gone, the `cd /tmp/$SANDBOX_NAME` prefixes are dropped, and
the upload paths point at `_out/…` instead of
`/tmp/${SANDBOX_NAME}/_out/…`. That takes the empty-value expansion out
of the `cd`s, the stray `mkdir`, the upload paths and the destructive
`rm` in one move, across both workflows rather than one step of many.
`Set sandbox ID` stays: the job still reads `$SANDBOX_NAME` directly
(failure diagnostics, the chainsaw `docker cp`, teardown, and every
`make SANDBOX_NAME=…` argument) — the container needs a name, just not a
directory.

It also removes the dead code the single-runner retirement left in
`packages/core/testing/Makefile`: the world-writable `mkdir -p
/tmp/${SANDBOX_NAME}` + `chmod 777` in the `apply` target (they backed a
`-v /tmp/${SANDBOX_NAME}:/workspace/hosttmp` bind mount that was later
removed — nothing under `images/e2e-sandbox/` references
`/workspace/hosttmp`), and the `-e SANDBOX_NAME=${SANDBOX_NAME}` passed
into the container (nothing under `images/e2e-sandbox/` reads it). Every
other use of `SANDBOX_NAME` — the container name and the `make`
arguments — is kept.

Note this cannot make a job with an empty `SANDBOX_NAME` go green: such
a job already failed initialization or was cancelled, so its conclusion
is fixed before cleanup runs. It removes a second, destructive failure
the old cleanup piled on top, and it does not change whether re-runs
work in general — they do.

The diagnosis of the real trigger, the corrected blast radius, and the
scope across both workflows plus the Makefile are credited to the
maintainer review on this PR.

### Evidence

Run [30067779342 attempt
3](https://github.com/cozystack/cozystack/actions/runs/30067779342/attempts/3)
exhibits the mechanism: `Download Talos image (regular PR)` failed on
`GetSignedArtifactURL` with a 404, which skipped `Set sandbox ID`, and
the `always()` steps then ran on an empty value — `Collect chainsaw
report` created `/tmp/_out`, the uploads resolved to `/tmp//_out/…`, and
`Remove workspace` did:

```
rm: cannot remove '/tmp/systemd-private-…-systemd-timedated.service-1oADZP': Operation not permitted
rm: cannot remove '/tmp/.X11-unix': Operation not permitted
##[error]Process completed with exit code 1.
```

The step ran unprivileged in `/home/ubuntu/_work/…`, so all it could hit
was `Operation not permitted` on root-owned entries in the sticky `/tmp`
— it deleted nothing. That run is a later, independent occurrence (it
started well after this PR was opened), not this change's motivating
case. The 404 that triggered it is a separate, non-deterministic problem
— the artifact listing succeeded and only the signed-URL call failed
~190 ms in, `download-artifact` treats 404 as non-retryable, and attempt
2 of the same run fetched the artifact fine — and it deserves its own
issue rather than being folded in here.

### Screenshots

Not applicable — CI workflow change only.

### Downstream repositories

The change is confined to this repo's E2E CI
(`.github/workflows/pull-requests.yaml`,
`.github/workflows/nightly.yaml`) and its testing Makefile
(`packages/core/testing/Makefile`). No downstream repository consumes
it.

- [x] No downstream repository is affected by this change

### Release note

```release-note
refactor(ci): drop the vestigial /tmp/$SANDBOX_NAME workspace from the E2E PR and nightly jobs — they now run in $GITHUB_WORKSPACE, removing an unguarded `rm -rf /tmp/$SANDBOX_NAME` cleanup that failed the job on cleanup when SANDBOX_NAME was empty
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Improved end-to-end test environment setup and sandbox cleanup.
* Updated test reporting and artifact collection for more reliable
results.
* Ensured OpenAPI and full-suite tests run from the correct prepared
environment.

* **Chores**
* Simplified container and workflow configuration for sandbox-based
testing.
  * Clarified checkout token handling in nightly workflows.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-29 15:48:39 +05:00
Andrei Kvapil 93338cc434 refactor(computeplane): drop the in-cluster Flux pin
The fluxcd addon pin was a holdover from the #3150-era model where app
charts rendered nested HelmReleases into the target cluster and needed an
in-cluster helm-controller. Consumers now keep their releases on the
management cluster and remote-apply via spec.kubeConfig, so the management
Flux resolves charts and applies manifests — nothing inside the ComputePlane
reconciles HelmReleases. This also decouples the module from the FluxCD
addon removal (#3379). certManager and ingressNginx stay pinned: Certificate
and Ingress objects land inside the cluster and need their controllers
there.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:03 +02:00
Andrei Kvapil 2ff9c30d04 docs(kubernetes): note the etcd gate in the Proxied Service comment
The Proxied ingress Service is gated on _namespace.etcd as well as the
addon toggle and exposeMethod, but the comment above it still described
only the addon+exposeMethod gate — comment drift from the etcd-gate
change. State both gates so the comment matches the condition it sits on.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:03 +02:00
Andrei Kvapil 8fc8d23dfd fix(computeplane): address review round three
Gate the Proxied ingress Service on _namespace.etcd like the addon
HelmRelease it fronts, so an etcd-less tenant with the addon pinned on does
not get a selector-less Service (+ test). Correct the secrets framing: the
tenant-visible surface is an allowlist (include: [] withholds everything),
so drop the never-load-bearing exclude entry, the bats cross-check and the
rename-coupling comments, and state the real mechanism. Fix the etcd
prerequisite docs (an ancestor's etcd satisfies the wrapped chart; the
awaiting-etcd beacon is the diagnostic), reword the tenant toggle to stop
advertising the not-yet-shipped placement routing, require maxReplicas >= 1
on the ingress-role node group, and fix comment drift (Service name after
the release rename, placement-hook label, package-guard lookup semantics).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:03 +02:00
Andrei Kvapil 366268d9f5 test(kubernetes): pin that a disabled ingress addon renders nothing with hosts set
hosts with the addon disabled used to render the Ingress; the addon gate
makes 'the addon is off' mean nothing renders. Pin the combination so the
behaviour change reads as a decision.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:03 +02:00
Andrei Kvapil 5a3d25b8f5 docs(computeplane): document the etcd prerequisite and unlisted-field pass-through
The tenant Kubernetes machinery — including the cluster-autoscaler that
brings up the default scale-from-zero md0 group — is backed by the tenant's
etcd module, so a ComputePlane without it provisions no workers and gives no
diagnostic; state the prerequisite in the README. Also state plainly that
unlisted NodeGroup fields pass through to the wrapped chart unvalidated (the
schema generator cannot close the object), with a test pinning that
behaviour.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:03 +02:00
Andrei Kvapil 6dcf5373e7 fix(computeplane): let Flux derive both release names
The aggregated API cannot represent spec.releaseName: the conversion
rebuilds HelmRelease specs without it, so the first tenant edit through the
API dropped the module's explicit computeplane-module release name, let it
default to the object name computeplane, and collided with the Helm storage
record the cluster release owned — with the tenant chart writing the old
name back every interval. Drop releaseName from both HelmReleases and derive
names from object names, like every other tenant module. Kamaji now writes
the admin kubeconfig to computeplane-cluster-admin-kubeconfig; the
computeplane-rd secrets.exclude entry follows in the same change, and a bats
test pins the exclusion to the name the cluster release actually produces so
a future rename cannot silently re-expose the credential. Also add the
check-release-name guard the other extra modules carry, and pin the fluxcd
addon's distribution artifact to the flux-operator version vendored in
system/fluxcd-operator instead of :latest.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:03 +02:00
Andrei Kvapil ebeeab77aa docs(platform): correct the computeplane dependsOn and variant comments
The dependsOn comment claimed the kubernetes-application edge orders the
addon artifacts the wrapped chart references; variant dependsOn only gates
install: components — the operative effect is ordering the computeplane-rd
install so the ComputePlane kind is not registered before the kubernetes
machinery exists. Also document why the variant is named kubevirt.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:02 +02:00
Andrei Kvapil 96842d4a94 fix(computeplane): validate nodeGroups and require an ingress-nginx role
nodeGroups was a bare object: typos surfaced as a stuck release deep in the
wrapped chart instead of an admission error, and groups without the
ingress-nginx role silently left the pinned ingress-nginx controller (and
with it the ComputePlane ingress) Pending forever, since supplied groups are
not merged with the wrapped chart's default md0. Reuse the kubernetes app's
nodeGroups typedefs for the schema and refuse rendering when no group
carries the role.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:02 +02:00
Andrei Kvapil 76374f6472 fix(tenant): fail loudly when computeplane is enabled without its package
The computeplane-application package ships in the iaas bundle while the
toggle arrives with tenant-rd in the system bundle, and bundles are
independent booleans. Refuse the render with a clear diagnostic instead of
producing a HelmRelease that retries a nonexistent artifact forever.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:02 +02:00
Andrei Kvapil bca5b3f9fb fix(kubernetes): render the Proxied ingress Service only when the addon is enabled
exposeMethod defaults to Proxied while the addon defaults to off, so gating
the Service on exposeMethod alone emitted a selector-less Service — surfaced
in the tenant dashboard — on every cluster that runs no ingress-nginx at all.
Gate on the addon instead; ComputePlane sets enabled: true explicitly, and
the Service still renders without a hosts list.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:02 +02:00
Andrei Kvapil aa5198b758 fix(platform): add cozystack-engine dependency to the computeplane package
The computeplane-rd component installs an ApplicationDefinition, whose CRD is
served by cozystack-engine — hack/check-applicationdefinition-crd-ordering.bats
enforces the edge for every package with a system/*-rd component.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:02 +02:00
Andrei Kvapil 9cb4f2922d fix(kubernetes): create the Proxied ingress Service independently of hosts
The proxy Service was only rendered when addons.ingressNginx.hosts was
non-empty, but external consumers — e.g. a ComputePlane app's per-app
management Ingress — need to route to the tenant cluster's ingress-nginx
without the module knowing every app host in advance. Render the Service
whenever exposeMethod is Proxied; the Ingress still requires hosts.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:02 +02:00
Andrei Kvapil b2232904b6 feat(tenant): add the computeplane module toggle
Wire the computeplane module into the tenant chart as a bool toggle, matching
the other tenant modules. The module's own Helm release takes the -module
suffix so the canonical release name computeplane (and with it the
computeplane-admin-kubeconfig Secret) stays reserved for the cluster
HelmRelease the module renders.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:02 +02:00
Andrei Kvapil 4c3262a4ee feat(computeplane): deliver ComputePlane as a tenant module wrapping the kubernetes app
Add the operator-owned computeplane tenant module per the revised design
(cozystack/community design-proposals/compute-plane): an extra/computeplane
chart that provisions a single-tenant, Cozystack-managed cluster for
untrusted-code apps by deploying the ordinary apps/kubernetes chart with
operator-fixed values (fluxcd, cert-manager, ingress-nginx Proxied). The
package re-declares apps/kubernetes as its own source-only component, so the
module is self-contained; the cluster release is named computeplane so Kamaji
writes the admin kubeconfig to computeplane-admin-kubeconfig — the contract
consumed by placement: ComputePlane applications. The tenant holds no admin
kubeconfig and can set cluster shape (nodeGroups) only.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:08:02 +02:00
7ebce2e942 feat(kubeovn): build the chart and image in-tree
kubeovn was the last package whose chart and image were produced by a
separate repository: packages/system/kubeovn pulled a tarball from the
latest cozystack/kubeovn-chart tag at update time, and that repository
built and pushed the image. The wrapper existed only to cache the
upstream chart; with the current caching that reason is gone, and the
split cost a second repository, a second release step, and a package
whose provenance a reader could not follow without leaving the tree.

Move the wrapper in, shaped like packages/system/cilium: update pulls
the upstream kube-ovn chart directly and applies patches/mtu.diff, and
image builds images/kubeovn and pins the digest into values.yaml. The
MTU patch already carried monorepo-relative paths, so it applies here
unchanged.

Also carries cozystack/kubeovn-chart#5, which never got a tag: pin
OVN_IPSEC_KEY_DIR under /var/lib. The kube-ovn default sits in
/etc/origin, which a Talos node cannot create, so enabling IPsec left
the node without ESP security associations.

Co-authored-by: Timofei Larkin <lllamnyp@gmail.com>
Co-authored-by: Kirill Ilin <stitch14@yandex.ru>
Co-authored-by: Aleksei Sviridkin <f@lex.la>
Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 18:07:33 +02:00
Andrei Kvapil 8b461f0c0b feat(monitoring): drop the Grafana rebuild for the upstream image + catalog plugins (#3378)
## What this PR does

Cozystack builds and publishes `ghcr.io/cozystack/cozystack/grafana` — a
rebuilt Grafana image — and uses it by default in the monitoring stack.
Grafana is AGPL-3.0, so publishing a rebuilt binary makes the project a
redistributor of an AGPL Grafana build. This is a governance concern for
CNCF Incubation (there is no precedent for a CNCF project redistributing
an AGPL binary).

The only reason the image was ever rebuilt was to bundle the
VictoriaLogs datasource plugin, which was not in the official Grafana
plugin catalog at the time. It is now published there as a signed plugin
(`victoriametrics-logs-datasource`), so the rebuild is no longer
necessary.

This PR moves the deployment to the upstream `grafana/grafana` image
(digest-pinned, routed through `cozy-lib.image` so mirrored-registry
installs keep working) and installs the plugins at container startup via
`GF_INSTALL_PLUGINS`, with versions pinned:

- `victoriametrics-logs-datasource 0.29.0` — the VictoriaLogs datasource
(signed; the reason the image was rebuilt)
- `natel-discrete-panel 0.1.1`, `grafana-worldmap-panel 1.0.6`,
`marcusolsson-dynamictext-panel 6.2.0` — reproduce the panel set the
removed Dockerfile vendored

**Air-gapped installs keep working.** The plugin zips are not downloaded
from the internet at pod startup: they are mirrored from the official
catalog into the existing `grafana-dashboards` image at build time and
served by that in-cluster static file server, next to the dashboards it
already serves. `GF_INSTALL_PLUGINS` points at
`http://grafana-dashboards.cozy-grafana-operator.svc/plugins/…`, so the
runtime dependency stays inside the cluster and the artifact is mirrored
into private registries together with every other first-party image. A
new `allow-to-grafana-dashboards` CiliumNetworkPolicy (mirroring
`allow-to-keycloak`) opens tenant egress to that server, since tenant
egress is default-deny.

As a result, `ghcr.io/cozystack/cozystack/grafana` is no longer built or
published: the Dockerfile, the pinned image tag, the `image` targets in
both monitoring Makefiles, and the build step in the root Makefile are
removed.

Notes:

- The AGPL-3.0 Grafana binary is no longer redistributed; the mirrored
plugin zips are Apache-2.0 licensed and are served unmodified from the
official catalog builds (signatures intact).
- `allow_loading_unsigned_plugins` and the custom
`/var/lib/grafana-plugins` path are dropped: the catalog datasource is
signed, and plugins install into the writable default `GF_PATHS_PLUGINS`
(`/var/lib/grafana/plugins`).
- The VictoriaLogs datasource provisioning (the `GrafanaDatasource` CRs
pointing at `vlselect`) is unchanged.
- Plugins are re-installed on every pod start from the in-cluster server
(no PVC involved, ~77 MB per start per replica over the cluster
network); a failed download fails the pod loudly instead of silently
dropping the datasource.
- `natel-discrete-panel` and `grafana-worldmap-panel` are Angular
plugins; Angular is disabled by default in Grafana 11, so they do not
load — this is unchanged from the current rebuilt image (same Grafana
11.6.15, same config). They are kept for parity and can be dropped as a
follow-up.

### How this was verified

- `helm unittest` across the repo passes, including new regression
tests: the monitoring chart asserts the exact upstream image reference
(with and without `images-registry` set) and the exact
`GF_INSTALL_PLUGINS` value; the tenant chart asserts the new egress
policy.
- Local end-to-end of the full chain in Docker: built the modified
`grafana-dashboards` image, served it on a shared network, and started
upstream `grafana/grafana:11.6.15` with the rendered
`GF_INSTALL_PLUGINS` value. All four plugins download from the
in-cluster-style URL (no internet egress from Grafana), and `GET
/api/plugins` reports `victoriametrics-logs-datasource` v0.29.0 as
`enabled: true` with `signature: valid`.

### Screenshots

Not applicable — no UI change. Grafana renders the same datasources and
panels; only the image source and plugin delivery mechanism change.

### Downstream repositories

Walked the trigger map against the diff. This is an internal change to
how the monitoring stack sources the Grafana image and its plugins
(`packages/system/monitoring`, `packages/system/grafana-operator`,
`packages/apps/tenant` network policy). No package is added, renamed or
removed; no `values.schema.json`, `ApplicationDefinition`, platform
values, release asset, shared `hack/` tooling or `package.mk` contract
changes; the added `grafana.image` key is an internal system-chart
value, not a user-facing app field. No downstream repository is
affected.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(monitoring): Grafana now runs on the upstream `grafana/grafana` image; the VictoriaLogs datasource and panel plugins are mirrored from the official Grafana catalog into the in-cluster `grafana-dashboards` server and installed at startup, so air-gapped installs keep working. Cozystack no longer builds or publishes `ghcr.io/cozystack/cozystack/grafana`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Grafana in monitoring is now based on a digest-pinned upstream image
for more consistent deployments.
* Required Grafana plugins are installed automatically at startup via
`GF_INSTALL_PLUGINS`; unsigned-plugin configuration was removed.
* Tenant networking now includes a new egress rule specifically for
Grafana dashboards.
* Grafana dashboards image now includes required plugin assets prepared
during image build.
* **Tests**
* Added/updated checks covering digest pinning, mirrored-registry
behavior, plugin-installation behavior, and the new Grafana dashboards
network policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-28 18:06:57 +02:00
Andrei Kvapil 78fc3f9178 refactor(ci): drop the vestigial /tmp workspace from e2e (both workflows)
The E2E job once ran as several state-sharing self-hosted-runner jobs
that rendezvoused through a sha256(REPO:WORKFLOW:REF) key and a per-run
/tmp/$SANDBOX_NAME copy of the checkout. Since the move to the ephemeral
pool it is a single job: there is no cross-job rendezvous and the
checkout is never reused, so copying it into /tmp and removing it
afterwards is dead weight.

Run every step in $GITHUB_WORKSPACE instead: drop the Prepare/Remove
workspace steps and the `cd /tmp/$SANDBOX_NAME` prefixes, and point the
upload paths at _out/. This deletes the unguarded `rm -rf
/tmp/$SANDBOX_NAME` rather than guarding it: when an upstream step fails
or the job is cancelled, the unconditional `Set sandbox ID` is skipped,
SANDBOX_NAME stays empty, and the `if: always()` steps interpolate it —
the removal expanding to `rm -rf /tmp/` and failing the cleanup step.
`Set sandbox ID` stays, since the container still needs a name.

Apply the same treatment to nightly.yaml, and drop the Makefile dead
code the single-runner retirement left behind: the world-writable
mkdir/chmod of /tmp/$SANDBOX_NAME (backed a bind mount removed long ago)
and the `-e SANDBOX_NAME` container env (nothing in the e2e-sandbox
image reads it).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-28 17:49:02 +02:00
Andrei Kvapil d598e672f2 fix(reloader): add startupProbe to survive slow first start under load (#3470)
## What this PR does

Adds a startup probe to the reloader deployment, rendered from package
values through a patch on the vendored chart.

Reloader went into CrashLoopBackOff on a loaded install. Events show
`Liveness probe failed: ... connect: connection refused`, repeating on
the liveness cadence until the kubelet gave up.

A refusal rather than a timeout closes two questions at once. It is an
RST from the target netns, so the packet arrived and the pod network is
fine. It also rules out the mode in
[stakater/Reloader#252](https://github.com/stakater/Reloader/issues/252),
where the listener is already up and `/metrics` just answers slowly;
that one produces probe timeouts. Nothing was listening on `:9090` yet.

`http.ListenAndServe` is the last statement of `startReloader`, so the
whole prologue runs before the socket opens. The slow part is not the
informers, those are lazy and do their List in a goroutine. It is
`IsOpenshift = isOpenshift()` in `pkg/kube`: a package-level var, so it
runs at init before `main()`, and it does a `GET
/apis/project.openshift.io` on a `context.TODO()` carrying no deadline.
The latency is the API server's, and the container waits it out under a
100m CPU limit.

Without a startup probe the only grace period is the liveness budget
itself, `initialDelaySeconds 10 + periodSeconds 10 * failureThreshold
5`. A start that overruns it gets killed mid-prologue, and the restart
re-runs the prologue from the top against an API server that is no less
busy, so the loop feeds itself.

This reaches past reloader.
`packages/core/platform/sources/linstor.yaml` lists `cozystack.reloader`
in the LINSTOR stack's `dependsOn`, and `dependsOn` fans out to every
component of that stack, so a reloader stuck in this loop blocks that
whole install.

The probe is `/live` at `periodSeconds 5` and `failureThreshold 45`,
about 225s, well inside the 10m install timeout that cozystack-operator
stamps on generated HelmReleases. Liveness is suspended while it runs
and gets its tight budget back as soon as the listener binds.
`timeoutSeconds` is 5 to match the other two probes instead of the
kubelet default of 1s, since a throttled process at 100m can hold the
socket open and still need more than a second to answer.

Both liveness knobs the chart already exposes would work without any
patch. Widening `failureThreshold` buys the grace permanently and
loosens hang detection for the life of the container.
`initialDelaySeconds` is start-up-only and would genuinely do, but it is
a fixed blind window on every start, including one that binds in three
seconds, and nothing restarts a container that wedges inside it.
Adaptive against fixed is the trade. The win is smaller than it sounds
while `reloader.enableHA` is unset, since `/live` is a constant 200 then
and liveness only catches the listener dying outright.

A chart bump does not fix this. 2.2.14 still renders no `startupProbe`
anywhere, and v1.4.19 added a second eager package-level call next to
`IsOpenshift`, so the prologue got longer instead of shorter.
`packages/system/velero` already carries a startupProbe patch on its
vendored chart for the same reason.

`make update` now drives the shared `%-update` rule with an exact
`CHART_VERSION`. It has to be exact. With a stale local repo cache `helm
pull --version 2.0.0` fails with version-not-found, while `--version
"^2.0.0"` quietly resolves to 2.2.14, whose `deployment.yaml` has the
ports block at line 179 instead of 161, so the hunk would fuzz or land
somewhere unintended.

The patch runs with `--forward` because `patch(1)` defaults diverge
between implementations over a tree that already carries the change. GNU
2.7.6 refuses and exits 1. Apple 2.0-12u11 takes its `Assume -R? [y]`
default, strips the hunk back out and exits 0. GNU under `--batch` does
the same, which is the trap, since `--batch` is what you reach for to
make a script non-interactive. `--forward` refuses in all three and
leaves a `.rej`.

Two things keep the probe from disappearing quietly, and they fire at
different times. `--forward` catches it at the moment someone re-runs
the patch by hand. The test only catches it at the next run of the
suite, late enough for a commit to travel in between.

`make test` also checks that the vendored `Chart.yaml` still records the
pinned version, so bumping `CHART_VERSION` without re-running `make
update` fails instead of leaving the pin and the tree disagreeing.
Checked by doing the violation and watching it go red.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. All five files sit inside one `packages/system` package. No
package is added, renamed or removed, nothing under `hack/` is touched,
and no CRD, namespace, `packages/core` values key, version enum or
release asset changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(reloader): add a startup probe so a slow first start is not killed by the liveness probe and left in CrashLoopBackOff, which also unblocks the LINSTOR stack that depends on reloader
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Added a configurable startup health probe for the Reloader service.
* The probe allows additional time for the service to start before
health checks begin.
* Documented the probe’s timing and behavior in the chart configuration.

* **Bug Fixes**
  * Preserved existing liveness and readiness probe behavior.

* **Tests**
  * Added coverage for enabled, disabled, and unaffected health probes.
* Added validation to ensure the chart version matches the pinned
release.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-28 17:45:52 +02:00
Andrei Kvapil 9fe6ae2ec1 fix(keycloak-configure): patch HelmRelease in release namespace on teardown (#3372)
## What this PR does

The `keycloak-configure` pre-delete teardown Job clears the Flux
`HelmRelease` finalizers as its final step so the release can be
uninstalled. That final `kubectl patch` targeted a hardcoded namespace
that does not match where the release actually installs. The release
installs into `cozy-keycloak`, and the teardown Job's own RBAC — a
`Role` + `RoleBinding` granting `patch` on the `HelmRelease` by release
name — is created in the release namespace, but the patch was aimed at
`cozy-system`.

The result is a deadlock: the Job's `ServiceAccount` is `Forbidden` to
patch the `HelmRelease` in the mismatched namespace, so the Job errors
and retries forever, the `HelmRelease` sticks in `Terminating`, and the
Helm release wedges in status `uninstalling`. This blocks any teardown
or reinstall of the component.

The fix templates both the release name and the namespace from `{{
.Release.Name }}` / `{{ .Release.Namespace }}`, so the patch targets the
namespace where the `HelmRelease` lives and where the teardown RBAC
grants access. I also added a helm-unittest that pins the rendered
target, guards against the hardcoded namespace regressing, and asserts
the patch tracks the release name and namespace.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
fix(keycloak-configure): fix the pre-delete teardown Job so uninstalling the release no longer wedges — it now patches the HelmRelease in the release namespace instead of a hardcoded one
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Improved Helm chart deletion cleanup to patch the correct HelmRelease
using the chart’s release name and namespace instead of fixed values.
- Reduced risk of teardown targeting resources in an unintended
namespace.
- **Tests**
- Added a new delete/teardown YAML test suite to verify correct
parameterization and that only the intended namespace is modified.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-28 17:45:47 +02:00
Andrei Kvapil 07cefee158 fix(kubernetes): raise kubevirt-csi API client rate limits (#3428)
## What this PR does

The kubevirt-csi-driver builds its infra and tenant cluster clients from
a `rest.Config` with QPS/Burst left unset, so client-go uses its default
5 QPS / 10 burst. The wrapper's NFS `ControllerPublishVolume` polls the
infra PVC once per second for up to two minutes. When a tenant scales
out and several volumes attach at once, that shared token bucket starves
and the poll returns `client rate limiter Wait returned an error:
context deadline exceeded`, which the tenant kubelet reports as
`FailedAttachVolume`. It looks like an intermittent E2E failure
attaching a tenant volume, with nothing wrong in the storage backend.

This sets QPS/Burst to 100/200 on both configs, before any client is
built. The new flags `--kube-api-qps` and `--kube-api-burst` make the
limits tunable. The compiled-in defaults are the real fix: the
deployment passes no client flags, so both configs resolve to the
in-cluster config and pick up the new defaults. Non-positive values are
rejected, since client-go would read `0` as the starving default and a
negative value as no rate limiting.

Includes a regression test: a burst of 200 waiters times out against the
default 5 QPS / 10 burst limiter and passes against the configured one.

### Screenshots

None. No UI changes.

### Downstream repositories

The diff touches only the driver's Go source in
`packages/apps/kubernetes/images/kubevirt-csi-driver/`. No package was
added, renamed or removed, and no `values.schema.json`, chart values,
`ApplicationDefinition`, node prerequisite, metric or label changed.
Nothing in the downstream trigger map is reached.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(kubernetes): raise the kubevirt-csi-driver API client rate limits (QPS/Burst, tunable via --kube-api-qps and --kube-api-burst) above the client-go defaults so a burst of concurrent volume attaches no longer times out as FailedAttachVolume.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added configurable Kubernetes API request rate limits for improved
handling of high-volume operations.
* Added validation to prevent invalid or non-positive rate-limit
settings.

* **Bug Fixes**
* Improved reliability during bursts of concurrent volume attachment
requests by reducing request throttling and timeout risk.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-28 17:45:42 +02:00
Aleksei Sviridkin 7f67d02fb7 fix(reloader): add startupProbe to survive slow first start under load
Reloader binds its :9090 listener as the last statement of
startReloader, so /live and /metrics answer connection refused until
everything before it has finished. The blocking part of that prologue
is the package-level IsOpenshift = isOpenshift() in pkg/kube, which
issues a GET /apis/project.openshift.io against the API server on a
context carrying no deadline, before main() is even entered. That
latency belongs to the API server, and the container waits it out at a
100m CPU limit.

The chart renders no startupProbe, so the only grace period is the
liveness budget: initialDelaySeconds 10 + periodSeconds 10 *
failureThreshold 5, about 60 seconds. A first start that needs longer
is killed mid-prologue, and the restart does not resume it, it runs it
again from the beginning, so the loop sustains itself. The blast
radius reaches past reloader: the LINSTOR stack lists
cozystack.reloader in dependsOn, which fans out to every component of
that stack, so a reloader stuck in this loop blocks the whole install.

Render a values-driven startupProbe on /live so liveness only begins
evaluating once the listener is up. Budget = periodSeconds 5 times
failureThreshold 45, so about 225 seconds. timeoutSeconds matches the
5s the chart already gives liveness rather than the kubelet default of
1s, which a throttled process can exceed while holding the socket
open.

Both liveness knobs the chart already exposes could buy the same grace
with no patch to carry. Widening failureThreshold buys it permanently,
loosening hang detection for the life of the container.
initialDelaySeconds is start-up-only and would genuinely do, but it is
a fixed blind window on every start, whereas a startup probe returns
liveness to its tight budget the moment the listener binds.

The chart is vendored, so the template change is carried as
patches/deployment-startup-probe.diff and reapplied by `make update`,
which now drives the shared %-update rule with an exact chart version
pin instead of pulling whatever is latest. The patch is applied with
--forward because patch(1) defaults diverge by implementation over a
tree that already carries the change: GNU 2.7.6 refuses and exits 1,
Apple 2.0-12u11 takes its "Assume -R? [y]" default and strips the hunk
back out with exit 0, and GNU under --batch does the same. --forward
refuses in all three.

A helm unittest pins the probe, its budget, the no-op path the
template promises for an unset value, and the liveness numbers the
budget is sized against, so the documented arithmetic cannot drift
away from the chart without a red test. The same target asserts that
the vendored Chart.yaml still records the pinned version, so raising
the pin without re-running `make update` fails instead of leaving the
tree and the pin quietly disagreeing.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-28 12:16:57 +03:00
Aleksei Sviridkin fd568dee87 fix(keycloak-configure): patch HelmRelease in release namespace on teardown
The pre-delete teardown Job clears the HelmRelease finalizers as its
final step, but targeted a hardcoded namespace that does not match the
namespace where the release and its RBAC live. The Job's Role and
RoleBinding are created in the release namespace and grant patch on the
HelmRelease by release name, so the ServiceAccount was Forbidden to patch
the HelmRelease in the hardcoded namespace. The Job errored and retried
forever, the HelmRelease stuck in Terminating, and the Helm release
wedged in "uninstalling", blocking any teardown or reinstall.

Target the release name and namespace so the patch matches the RBAC that
grants it. Add a helm-unittest pinning the rendered namespace.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-28 00:11:16 +03:00
Aleksei Sviridkin 2dfe380ac3 fix(kubernetes): raise kubevirt-csi API client rate limits
The kubevirt-csi controller built its infra- and tenant-cluster
Kubernetes clients on a rest.Config that left QPS/Burst unset, so
client-go applied its 5 QPS / 10 burst default. ControllerPublishVolume
polls the infra PVC once per second for up to two minutes; when a tenant
scales out and several volumes attach at once, the default limiter
starves and the poll fails with "client rate limiter Wait returned an
error: context deadline exceeded", which the tenant kubelet surfaces as
FailedAttachVolume.

Set QPS/Burst to 100/200, overridable via --kube-api-qps and
--kube-api-burst, on both the infra and tenant configs so a
concurrent-attach burst is admitted promptly. Reject non-positive flag
values, which client-go would otherwise turn into the starving default
(0) or unlimited (<0).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-27 22:59:56 +03:00
Aleksei Sviridkin 82e10878be fix(harbor): guarantee CPU for redis sentinels
The inline RedisFailover ran its three sentinels at 50m CPU request and
a 100m limit. Sentinels only become Ready after the redis-operator
elects a master and rewrites their monitored-master config, and on a
CPU-contended node CFS shares cycles proportionally to the request, so
underprovisioned sentinels converge slowly and delay harbor readiness
behind them. Raise the sentinels to Guaranteed QoS (requests equal
limits, 250m CPU / 128Mi) so election is never starved or throttled
under load. The redis data container is unchanged.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-27 22:48:24 +03:00
Aleksei Sviridkin 8bffa34b54 feat(platform): add opt-in in-cluster east-west encryption toggle (#2984)
## What this PR does

Adds `networking.encryption.enabled` to the platform chart — a single
operator-facing opt-in, off by default, that turns on in-cluster
(east-west) transport encryption for the active datapath. When enabled,
pod-to-pod traffic across nodes is encrypted at the network layer
regardless of whether the workload implements TLS: defense-in-depth for
the whole mesh without per-application changes.

The default `isp-full` bundle selects the `kubeovn-cilium` datapath (and
`isp-full-generic` selects `kubeovn-cilium-generic`), where KubeOVN owns
the datapath and the overlay is a Geneve tunnel. The toggle therefore
maps to KubeOVN's overlay IPsec (`kube-ovn` `ENABLE_OVN_IPSEC`).
KubeOVN's own controller provisions the IPsec CA and per-node
certificates through the Kubernetes CSR API, so no cert-manager wiring
is needed.

This complements, and does not replace, application-level / end-to-end
TLS: in-cluster encryption never reaches an external client and gives a
tenant no verifiable endpoint certificate. The values comments document
the per-datapath scope — including that Kilo's WireGuard mesh
(`meshGranularity: location`) covers only cross-location traffic and
leaves the intra-location gap this toggle targets, and that the
Cilium-native WireGuard/IPsec path on the `cilium` / `cilium-generic`
variants is not reached because the system bundle does not select those
variants — plus the MTU/throughput overhead.

Scope: only the default KubeOVN datapath is wired through the platform
toggle, which is what the system bundle actually selects. Wiring the
Cilium-native path would require the bundle to expose cilium-only
variant selection; that is out of scope here and called out in the
values comments.

Validation: `helm unittest` on the platform chart passes (66 tests, 7
new) covering on / off / absent / null and verifying the IPsec flag
deep-merges without dropping the existing `kube-ovn` overrides, for both
`isp-full` and `isp-full-generic`.

Closes #2977

Part of #2811

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(platform): add opt-in `networking.encryption.enabled` (off by default) that enables in-cluster east-west transport encryption for the default KubeOVN datapath via overlay IPsec; complements, and does not replace, application-level TLS.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Introduced opt-in in-cluster network encryption using KubeOVN IPsec
for east-west traffic protection.
* Added `networking.encryption` configuration toggle (disabled by
default) with full backward compatibility.

* **Tests**
* Added comprehensive test coverage for network encryption
configurations across system variants and edge cases.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-27 22:47:38 +03:00
Timofei Larkin 66c96b4329 fix(dashboard): stop requesting offline_access so logout can revoke sessions (#3447)
## What this PR does

The dashboard and linstor-gui auth-proxies (oauth2-proxy) requested the
`offline_access` scope, so Keycloak issued an **offline** refresh token.
oauth2-proxy has no server-side session store: it keeps the refresh
token in
the `kc-access` cookie and silently refreshes it (`--cookie-refresh`).
An
offline token is decoupled from the SSO session by design, so logout
(`end_session`) does not revoke it. As a result a copied `kc-access`
cookie
stayed usable for the whole offline-session lifetime (Keycloak default
~30
days) even after the user logged out — logout did not invalidate it.

This drops `offline_access` from the proxy `--scope` in both packages,
so
Keycloak issues an **online** refresh token bound to the SSO session,
which
logout revokes. The `offline_access` optional client scope is also
removed for
cleanliness (with a note on why).

Scope of the fix:

- `packages/system/dashboard` — proxy `--scope` + optional client scope
- `packages/system/linstor-gui` — proxy `--scope` + optional client
scope + ingress auth unittest

The public `kubernetes` OIDC clients (kubectl `oidc-login`) are
intentionally
left untouched: `offline_access` is legitimate there and those clients
do not
sit behind oauth2-proxy.

Notes:

- The keycloak-operator only *adds* client scopes, so removing the
optional
  scope does not strip the realm-default `offline_access` from an
already-created client. The proxy `--scope` change is what actually
prevents
  issuance.
- This prevents *new* offline tokens. Offline sessions already issued to
existing users remain valid until the realm's offline-session idle
timeout
(~30d) or explicit revocation via the Keycloak Admin REST API / Account
  Console.
- After this change dashboard session lifetime is governed by the online
SSO
session (`client.session.idle.timeout` / `client.session.max.lifespan`).

### Screenshots

Not applicable — no UI change (auth-proxy OIDC scope only).

### Downstream repositories

Walked the trigger map against the diff: the change touches only helm
templates
and a unittest under `packages/system/dashboard` and
`packages/system/linstor-gui`.
No downstream repository restates the auth-proxy OIDC scope.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(dashboard): stop requesting the offline_access scope in the dashboard and linstor-gui auth-proxies so that logout revokes the session; a copied kc-access cookie is no longer refreshable after logout. Offline sessions already issued to existing users must be revoked manually via Keycloak.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Security Improvements**
* Removed requests for offline access and persistent refresh tokens from
OIDC authentication.
* Sessions now rely on standard scopes, helping ensure logout
invalidates authentication as expected.

* **Tests**
* Updated authentication checks to verify the revised OIDC scope
configuration.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-25 08:17:13 +04:00
Andrei Kvapil 1c7eb22918 feat(tenant): allow egress to the grafana-dashboards plugin server
Tenant egress is default-deny plus explicit allow-to-* policies, so the
Grafana pods deployed per tenant could not reach the in-cluster plugin
server in cozy-grafana-operator. Mirror the allow-to-keycloak pattern
and cover the policy with a test.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-24 16:08:51 +02:00
Andrei Kvapil 705e39d3d7 feat(monitoring): install Grafana plugins from the in-cluster mirror
Point GF_INSTALL_PLUGINS at the grafana-dashboards static file server
instead of the public Grafana catalog, so plugin installation needs no
internet egress at pod startup and keeps working on air-gapped installs.
Route the upstream Grafana image reference through cozy-lib.image so
mirrored-registry installs keep working, and add regression tests for
the image reference and the plugin install path.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-24 16:08:51 +02:00
Andrei Kvapil 606ffd5b1b build(grafana-operator): mirror Grafana catalog plugins into the dashboards image
Download the pinned Grafana plugin zips (VictoriaLogs datasource + panels)
from the official catalog at image build time and serve them next to the
dashboards. Grafana pods can then install plugins from this in-cluster
server instead of reaching the public catalog at startup, which keeps
air-gapped and egress-restricted installs working.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-24 16:08:51 +02:00
Andrei Kvapil 9c63df7545 build(monitoring): stop building and publishing the Grafana rebuild
Remove the custom Grafana image build now that the deployment runs on the
upstream grafana/grafana image with catalog plugins. Drops the Dockerfile,
the pinned image tag, the image targets in both monitoring Makefiles, and
the build step from the root Makefile, so ghcr.io/cozystack/cozystack/grafana
is no longer produced or published.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-24 16:01:46 +02:00
Andrei Kvapil 44ff0fb286 feat(monitoring): run Grafana on the upstream image with catalog plugins
Point the Grafana deployment at the upstream grafana/grafana image and
install the VictoriaLogs datasource and panel plugins at startup via
GF_INSTALL_PLUGINS, pulled from the official Grafana plugin catalog
(https://grafana.com/grafana/plugins) instead of being baked into a
custom rebuilt image.

The VictoriaLogs datasource is now published and signed in the catalog
(victoriametrics-logs-datasource), so allow_loading_unsigned_plugins and
the custom /var/lib/grafana-plugins path are no longer needed; plugins
land in the writable default GF_PATHS_PLUGINS. Plugin versions are pinned
to releases compatible with the shipped Grafana version.

Assisted-By: Claude
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
2026-07-24 16:00:30 +02:00
Timofei Larkin 436743eb34 feat(workloadmonitor): bucket metrics endpoint override (#3443)
## What this PR does

The cozystack-controller meters S3 bucket sizes for billing by querying
SeaweedFS `SeaweedFS_s3_bucket_(size|physical_size)_bytes` metrics from
a VictoriaMetrics vmselect it locates via the
`namespace.cozystack.io/monitoring` label on the tenant namespace. That
discovery hardcodes the assumption that the monitoring stack scraping
SeaweedFS runs in the same cluster. Some deployments host SeaweedFS —
API and metrics both — in a separate cluster, so bucket sizes must be
fetched from a remote Prometheus-compatible endpoint that discovery can
never produce.

### Configuration surface

A single controller flag, `--seaweedfs-metrics-endpoint`, exposed as a
chart value:

```yaml
cozystackController:
  seaweedfsMetricsEndpoint: "https://vm.example.com/path/to/prometheus"
```

The value is an arbitrary base URL of a Prometheus-compatible query API
— scheme, host, optional port, and an optional path prefix — and the
controller appends `/api/v1/query` to it, normalizing slashes so a
trailing slash or a path prefix never produces a doubled or dropped
slash. Basic auth can be embedded as URL userinfo
(`https://user:pass@vm.example.com/...`); net/http turns it into an
`Authorization` header on every request. TLS uses the system trust
roots; a custom CA or bearer-token support would be a follow-up flag if
a deployment needs it. The flag is validated and normalized at startup
(absolute URL, http/https, no query string or fragment), so a malformed
value fails the rollout instead of failing queries at runtime.

Cluster-wide scope is deliberate: where SeaweedFS and its metrics live
is a property of the deployment, not of any tenant, and tenant
monitoring stacks keep working and keep being discovered via the
`namespace.cozystack.io/monitoring` label for everything else — the flag
only redirects the bucket-size queries. Being a deployment argument, it
is settable exclusively by platform administrators, so no new writable
surface is exposed to tenants. Rejected alternatives: a WorkloadMonitor
spec field (bucket monitors are rendered by the tenant-facing bucket
chart, so the field would have to travel through tenant-writable values
— an SSRF primitive — and it would be near-permanent CRD surface for a
deployment-level fact) and per-namespace annotations (per-tenant
granularity that nothing needs, plus a second admin-facing configuration
mechanism to document and validate).

### Backwards compatibility

Strictly opt-in: with the flag unset — the default — resolution walks
the exact same label-discovery path and produces the same vmselect URL.
No existing deployment changes behaviour.

### Billing safety: failures no longer collapse to zero

This also fixes a latent billing bug in the existing path: any metrics
query failure (endpoint down, non-200, malformed response) produced an
empty result map, and the reconciler overwrote
`Workload.status.resources` with it — silently wiping the recorded
bucket sizes, which billing reads as an empty bucket. Now:

- `queryAllBucketMetrics` returns errors instead of swallowing them into
an empty map, so "the bucket is empty" and "the endpoint could not be
queried" are distinguishable.
- On failure — and per bucket, when a query succeeds but reports no
series for a bucket that previously had a size — the reconciler retains
the last known `s3-storage-bytes` and `s3-physical-storage-bytes` values
instead of dropping them.
- The failure is surfaced three ways: a `BucketMetricsUnavailable`
warning event on the WorkloadMonitor, an error-level log line, and an
error returned from `Reconcile`, which lands in controller-runtime's
reconcile error metrics for alerting.

Unit tests cover URL validation and path joining, flag-over-discovery
precedence, userinfo-based basic auth, and both retention scenarios
(query failure and missing series).

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the diff
(controller Go code plus the cozystack-controller chart value and arg
plumbing): none of the listed triggers match — no package added or
renamed, no CRD, `ApplicationDefinition`, or platform values change, no
`hack/` or layout change.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(workloadmonitor): a new --seaweedfs-metrics-endpoint controller flag (chart value cozystackController.seaweedfsMetricsEndpoint) points bucket size metering at an explicit Prometheus-compatible endpoint for deployments where SeaweedFS and its metrics live in a separate cluster; metrics query failures now retain the last known bucket sizes and surface an event and error instead of silently zeroing billing data
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Added an optional SeaweedFS metrics endpoint configuration.
  * Metrics endpoint URLs are validated and normalized before use.
* Added support for endpoint overrides when discovering bucket metrics.
* **Bug Fixes**
* Preserved previously recorded bucket sizes when metrics are
unavailable or incomplete.
* Metrics failures now trigger retries and visible warning events
without blocking other reconciliation.
  * Improved handling of invalid or unsuccessful metrics responses.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-24 17:25:26 +04:00
IvanHunters 51c34c236e fix(linstor-gui): stop requesting offline_access so logout can revoke sessions
Same offline-token-survives-logout issue as the dashboard: the linstor-gui
auth-proxy requested offline_access, so a copied kc-access cookie stayed valid
for the whole offline-session lifetime after logout.

Drop offline_access from the proxy --scope and the optional client scope, and
update the ingress auth unittest accordingly.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-24 14:19:24 +03:00
IvanHunters 99936a92f8 fix(dashboard): stop requesting offline_access so logout can revoke sessions
The dashboard auth-proxy (oauth2-proxy) requested the offline_access scope,
so Keycloak issued an offline refresh token. oauth2-proxy stores that token
in the kc-access cookie (no server-side session store) and silently refreshes
it, and an offline token is decoupled from the SSO session: logout
(end_session) does not revoke it. A copied cookie therefore stayed usable for
the whole offline-session lifetime even after the user logged out.

Drop offline_access from the proxy --scope so Keycloak issues an online
refresh token bound to the SSO session, which logout revokes. Also remove the
offline_access optional client scope for cleanliness.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-24 14:19:18 +03:00
Aleksei Sviridkin a80f13405c fix(cozy-lib): converge the CA trust-anchor helper on the canonical name (#3408)
## What this PR does

Ships `cozy-lib.tls.caCertSecret`, a render-time helper that emits a
key-free trust-anchor Secret carrying only `ca.crt`. It converges the
object name on `<release>.tenant-ca`, requires the whole value to be
certificate blocks so a stray private-key header or trailing bytes are
rejected, and coerces numeric scalars before the guard runs so a numeric
value cannot slip past.

No chart calls it yet. It is the render-time producer half of the
trust-anchor contract; the controller half is #3407, now merged. The
doc-comment on the helper shows the one-line call a chart uses, and says
outright there is no caller yet, so nobody reads it as wired when it is
not.

Split out from #3299. Comments trimmed on review.

```release-note
fix(cozy-lib): the CA trust-anchor helper emits the canonical `<release>.tenant-ca` name and rejects non-certificate input.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Updated generated TLS CA Secret naming to use the
`<release>.tenant-ca` convention.
- Added/ensured a tenant CA label is set alongside the existing tenant
resource label.
  - Continued support for custom labels (caller labels still applied).

- **Bug Fixes**
- Strengthened CA certificate PEM validation to be fail-closed: requires
non-empty, complete `BEGIN/END CERTIFICATE` blocks only.
- Rejects private-key material and other malformed or improperly
structured PEM input, including unexpected extra content.

- **Tests**
- Updated TLS CA certificate fixtures and assertions to cover additional
negative cases and refined error expectations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-24 13:59:23 +03:00