Commit Graph
4483 Commits
Author SHA1 Message Date
Aleksei Sviridkin f76dedf9ca fix(kubernetes): keep NFS publish from hanging on a dead server
The kubevirt-csi node plugin mounts the NFS export root into a temp
dir to create /data and migrate old files. That mount was hard, so
with the server gone the mkdir, readdir and rename calls on it never
returned. kubelet gives up after 2m and retries, and every retry
blocked the same way, piling up goroutines and nfs-init-* dirs.

Mount the temp root soft, so its RPCs fail once the connection to the
server is declared dead instead of blocking forever; it only does
metadata work that a retried publish redoes. retry=0 keeps mount.nfs
from retrying a failed temp mount for 2m. timeo and retrans keep their
defaults, because the first mount of a server sets them for the
connection the pods' mounts of it share. The volume mount the pod gets
stays hard, since a soft mount can report a failed write the server
applied.

Over NFSv4 that soft failure takes minutes, and a stat of a target
that is already hard-mounted can block for good, so a publish for a
target path that already has one running now returns Aborted instead
of starting another.

A soft stat can fail with EIO on a file /data already holds, and a
rename issued once the server is back would replace it, so the
migration now fails the publish on any stat error other than a
missing destination. It checks the destination with lstat, so a
user's symlink in /data that loops or dangles counts as present
instead of failing every publish or being replaced by the rename.

The skip it makes for a destination already in /data leaves the root
entry where the pod cannot see it, so it is now logged as a warning.
/data is created with a plain mkdir, since with the server gone
MkdirAll fails its stat on the mount point and then reports the local
temp dir as already existing instead of the EIO. The temp mount
errors now name the volume, like the migration errors do.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 20:46:12 +03:00
Aleksei Sviridkin 0a2bb5ef98 fix(cert-manager): enable the Gateway API solver in the host cert-manager (#4039)
## What this PR does

With the platform Gateway enabled, the ClusterIssuers in
`cert-manager-issuers` and the per-tenant Issuers rendered for
`TenantGateway` carry an `http01.gatewayHTTPRoute` solver, but the
controller refuses it: the vendored chart never turns Gateway API
support on, so every HTTP-01 challenge behind the Gateway fails with
`gateway api is not enabled` and no certificate is issued.

This passes a `ControllerConfiguration` with `enableGatewayAPI: true` to
the host cert-manager through the component values of its platform
Package. The host ships the Gateway API CRDs on every variant and orders
cert-manager after them, so the flag is safe there.

It is deliberately not a default of the `system/cert-manager` chart. The
same chart is installed into tenant Kubernetes clusters, where the CRDs
are an opt-in addon, and cert-manager v1.20 exits at startup when
Gateway API support is on and the CRDs are missing. The tenant
HelmRelease in `packages/apps/kubernetes` already passes the same
`ControllerConfiguration` when its `gatewayAPI` addon is enabled, so
both clusters now opt in from the layer that knows whether the CRDs are
there, and the chart itself stays neutral.

The new tests pin both halves: the host Package carries the flag and the
preconditions that make it safe, and the chart wires `config` through to
the controller while shipping none by default.

Closes #3725

### Screenshots

Not applicable, no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(cert-manager): enable the Gateway API solver in the host cert-manager, so ACME HTTP-01 challenges succeed when the platform Gateway is enabled
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Supported system bundles can now use cert-manager’s Gateway API
HTTPRoute solver to issue certificates.
- Hosted deployments include the required Gateway API custom resource
definitions, installed before cert-manager.
- Gateway API solver support remains opt-in in environments that do not
provide the required definitions, avoiding unintended changes to
certificate management there.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 17:47:41 +03:00
Timofei Larkin 9b0e25b1a0 fix(etcd-operator): gate VMServiceScrape on VictoriaMetrics operator CRD (#3960)
## What this PR does

`packages/system/etcd-operator/templates/metrics.yaml` rendered a
`VMServiceScrape` (`operator.victoriametrics.com/v1beta1`)
unconditionally, so the `cozy-etcd-operator/etcd-operator` HelmRelease
fails to install on any cluster that does not have the VictoriaMetrics
operator CRDs, with `no matches for kind "VMServiceScrape" in version
"operator.victoriametrics.com/v1beta1"`. That made the VictoriaMetrics
operator a hard dependency of etcd-operator, which it should not be.

This wraps only the `VMServiceScrape` document in a
`.Capabilities.APIVersions.Has "operator.victoriametrics.com/v1beta1"`
gate, following the precedent already established in
`packages/system/linstor/templates/podscrape.yaml`. The
`etcd-operator-metrics` Service keeps rendering unconditionally, so any
scraper — a `VMServiceScrape` when the CRD is present, or something else
entirely — can still target the `metrics` port.

Note: this is a Cozystack-packaging-only issue. The upstream
etcd-operator repo's own chart is unaffected — it already gates its
scrape object.

Added `packages/system/etcd-operator/tests/metrics_test.yaml` (`helm
unittest`) pinning the new behavior: the Service always renders, and the
`VMServiceScrape` renders only when the capability is injected.

Kept this minimal — a pure capabilities gate with no new values key,
matching linstor's style, rather than adding a values toggle.

Follow-up (out of scope for this PR):
`packages/extra/etcd/templates/podscrape.yaml` (a `VMPodScrape`) is
unconditional the same way and would hit the same install failure on a
cluster without the VictoriaMetrics operator.
`packages/system/monitoring-agents` also ships several unconditional
`VMServiceScrape`/`VMPodScrape` objects, though that package may already
assume the VictoriaMetrics operator is present as part of the monitoring
stack it belongs to.

### Screenshots

Not applicable — no UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

### Release note

```release-note
fix(etcd-operator): stop requiring the VictoriaMetrics operator to install etcd-operator; the VMServiceScrape now only renders when the operator's CRD is present
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Metrics collection now works without the VictoriaMetrics operator API,
while scrape configuration is rendered when that API is available.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 16:54:10 +04:00
Aleksei Sviridkin 6192d5814b feat(local-storage): add opt-in OpenEBS LocalPV hostpath package (#3266)
## What this PR does

Adds `packages/system/local-storage`, an optional package that vendors
OpenEBS Dynamic LocalPV (chart 4.5.1) and registers it as the
`cozystack.local-storage` PackageSource. A cluster without the DRBD and
ZFS kernel modules can't run LINSTOR, so it has no dynamic PV
provisioning, hostpath volumes cover that. They also fit deliberately
node-local workloads.

- StorageClass is `cozystack-local-hostpath`, `WaitForFirstConsumer`,
`Delete` reclaim. It is not the default and doesn't take any existing
class name.
- Opt-in: a system bundle can list the package in
`bundles.enabledPackages`, no built-in variant enables it. Alone it
installs with `cozypkg add cozystack.local-storage`.
- Vendored RBAC is cut down by `patches/rbac.diff` (reapplied by `make
update`) to what volume reconciliation calls. Helper Pods, leader
election Leases and analytics state stay in the install namespace,
helpers are still privileged because they work on host directories.
- Images stay on docker.io, pinned by index digest with `make
update-image-pins`, nothing is rebuilt or mirrored.
- `local-storage` scope is mapped to `area/storage` in
`pr-labeler.yaml`, without it the title lands in `area/uncategorized` on
every run.

Tested with helm-unittest (StorageClass, PackageSource registration) and
`tests/rbac_test.py` (effective permissions across the bindings). Not
tried on a live cluster yet. Nothing installs the package by default,
the container cluster e2e lane that will use it is a follow-up
([#3238](https://github.com/cozystack/cozystack/issues/3238)).

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/726
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Website follow-up is an issue, the docs there describe a package that
isn't merged yet. The diff adds a platform component and touches no
platform values, ApplicationDefinition, tooling or `hack/` path, so
nothing else in the trigger map matches.

### Release note

```release-note
feat(local-storage): add opt-in cozystack.local-storage package that installs OpenEBS LocalPV hostpath and provides the cozystack-local-hostpath StorageClass, it is not enabled by any built-in variant and is not the default StorageClass.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional OpenEBS LocalPV Hostpath storage for provisioning
persistent volumes on node-local storage.
* Added a configurable `cozy-local-hostpath` StorageClass, disabled by
default as the cluster’s default class.
* Added controls for storage class creation, reclaim policy, hostpath
quotas, and node deployment.

* **Documentation**
* Added guidance on installation, configuration, storage behavior, and
uninstall considerations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 14:59:03 +03:00
Aleksei Sviridkin 76b9e55102 fix(cert-manager): enable the Gateway API solver in the controller
With the platform Gateway enabled, the ClusterIssuers in
cert-manager-issuers and the per-tenant Issuers rendered for
TenantGateway carry an http01.gatewayHTTPRoute solver, but the
controller only honours that solver when Gateway API support is
switched on, and the vendored chart leaves it off. Every HTTP-01
challenge behind the Gateway then fails with "gateway api is not
enabled" and no ACME certificate is issued.

Pass a ControllerConfiguration with enableGatewayAPI to the host
cert-manager through the component values of its platform Package.
The host always ships the Gateway API CRDs and orders cert-manager
after them, so the flag is safe on every variant.

It is not a default of the system/cert-manager chart: the same chart
is installed into tenant clusters, where the CRDs are an opt-in addon
and a controller started with Gateway API support and no CRDs exits
at startup. The tenant side keeps opting in on its own.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
2026-10-07 14:14:35 +03:00
Myasnikov Daniil d61fed46cc feat(local-storage): add opt-in OpenEBS LocalPV hostpath package
LINSTOR needs the DRBD and ZFS kernel modules, so a cluster without
them has no dynamic PV provisioning. OpenEBS LocalPV hostpath needs no
kernel modules and also serves deliberately node-local workloads.

The package is opt-in so it cannot change an existing cluster. Its
class is named cozystack-local-hostpath so it cannot collide with an
existing class name, and it is not the default so PVCs that omit
storageClassName keep their current binding. The vendored RBAC is
patched down to what volume reconciliation calls; helper Pods, Leases
and analytics state stay in the install namespace.

Images stay on docker.io, pinned by index digest.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 11:30:45 +03:00
Aleksei Sviridkin 6addef4d3c fix(kubevirt): route the export proxy to the Service KubeVirt creates (#4637)
## What this PR does

The VM export routes point at a Service that does not exist, and the
tenant network policy would block the Ingress even if it did.

KubeVirt creates the export proxy Service under the name
`virt-exportproxy` (`VirtExportProxyName` in
`pkg/virt-operator/resource/generate/components`, v1.9.0). The Ingress
and the TLSRoute that publish `vm-exportproxy.<host>` name a
`vm-exportproxy` Service, and nothing creates one. The backend was
`virt-exportproxy` when the Ingress was added in 0ddaff938. Then
3780925a6 renamed the hostname to `vm-exportproxy` and the backend along
with it. The hostname stays, only the backend goes back.

Renaming the backend is not enough for the Ingress, because it is served
by the ingress controller in tenant-root and tenant egress is
default-deny. Each allowed destination has its own `allow-to-*` policy.
The CDI upload proxy has `allow-to-cdi-upload-proxy`, the export proxy
had nothing. The new `allow-to-vm-exportproxy` policy renders for the
same tenants and the same source pods as the CDI one. Its destination is
narrower: only the `kubevirt.io=virt-exportproxy` pods in
`cozy-kubevirt`, on TCP 8443, the container port virt-operator gives
them.

The unit tests pin the backend name for the Ingress and the TLSRoute,
and the placement, selector and port of the new policy.

I checked this on a test cluster with KubeVirt v1.9.0 and ingress-nginx.
Without the fix the controller logs `Error obtaining Endpoints for
Service "cozy-kubevirt/vm-exportproxy"`. With the backend renamed and
the tenant allowed egress to virt-exportproxy, a passthrough request to
`vm-exportproxy.<host>` reaches the export proxy, and it answers with
its own serving certificate.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubevirt): requests to vm-exportproxy.<host> now reach the KubeVirt export proxy: the Ingress and TLSRoute route to the virt-exportproxy Service KubeVirt creates, and tenants are allowed egress to the export proxy.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Tenant workloads can now connect to the KubeVirt export proxy over TCP
port 8443.
* **Bug Fixes**
* Updated export-proxy Ingress and TLSRoute routing to use the
`virt-exportproxy` backend, keeping export-proxy traffic directed to the
correct service.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 11:19:32 +03:00
Aleksei Sviridkin 8c1a78dc9b feat(operator): default system namespace memory via LimitRange (#3494)
## What this PR does

System components now get a default container memory limit, so their pod
cgroups carry a `memory.max` and drop out of the Talos userspace OOM
handler's victim set. `cozystack-operator` maintains a LimitRange named
`cozystack-system-defaults` in every system namespace it reconciles.

The Talos handler (v1.12+) scores cgroups with `memory_max.hasValue() ?
0.0 : {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current`
and drops everything that scores zero. Only a memory limit sets
`memory.max`, so only a limit takes a pod out of the candidate set. A
request just moves a pod from BestEffort to Burstable, which under the
default `strictCgroupClassOrdering: true` still means killed second.
`PriorityClass` does nothing here, because the handler reads cgroupfs
and never talks to the apiserver. Victim selection is also unrelated to
the trigger, so a tenant workload thrashing against its own limit can
drive node memory pressure while a system component with no limit is
what gets killed.

An audit of a running cluster found around 70 workloads across ~30
packages with at least one container missing a memory limit. Per-chart
resources would not cover that set, and would not keep covering it as
packages are added. So the LimitRange is written by
`reconcileNamespaces`, which already creates these namespaces and
already tells system from tenant. It also covers init containers, which
#3495 can't reach: one of them without a limit leaves the pod cgroup
without `memory.max`, whatever the main containers declare.

Tenant namespaces are skipped. Where the tenant chart's own
`tenant-range-limits` exists (only when the operator configured
`resourceQuotas`), a second LimitRange next to it would leave the result
to the order LimitRanger iterates them. Where it doesn't exist, a tenant
namespace is not the place for the platform to impose a default.

### Why the default is 32Gi

The handler grants immunity for `memory.max` existing at all, whatever
its value. So the ceiling doesn't need to be fitted, and 32Gi is high
enough that nothing plausibly reaches it. A ceiling low enough to bind
would turn a rare pressure-driven kill into a deterministic one for any
component whose real usage nobody measured. This repo already has one
such case, recorded in the `vertical-pod-autoscaler` values, where a
110Mi limit OOM-killed the updater and failed HelmReleases with it.
Scheduling is unaffected, since pods are scheduled against requests and
the paired default request is 32Mi. An operator who wants the ceiling to
bind can lower it, per component in that component's chart or fleet-wide
with the knob, once real usage is measured.

### When the default is withheld

A default below a container's own request makes that pod impossible to
admit, so the operator checks before it applies. It scans Deployments,
StatefulSets, DaemonSets, CronJobs, Jobs and Pods in the namespace, init
containers included, for a container that requests more than the
configured limit and declares no limit of its own. If it finds one, it
withdraws the managed LimitRange and records a hold in the
`operator.cozystack.io/system-memory-state` annotation on the Namespace.

The hold does not clear by itself when the blocker disappears, because
an empty scan looks the same as a pod between deletion and recreation.
Lowering a previously applied limit also creates a hold, even with no
visible blocker, because LimitRanger writes the defaulted limit onto
admitted pods and the scan can no longer tell which containers declare
one. A cluster administrator releases the hold by copying the
`<hold>:<target>` token from the state into the
`operator.cozystack.io/system-memory-ack` annotation. The token is used
once and only for that hold and target. If a blocker is still visible,
the token is rejected and the hold stays. Package reconciles are
scheduled at least every five minutes, so an acknowledgement takes
effect on a quiet cluster too. The reason and the token also appear in
the operator log under `withholding the default container memory limit`.

### Namespace another LimitRange already governs

If a system namespace already has a LimitRange that says anything about
memory, at the Container or the Pod scope, the operator writes nothing
there and withdraws what it wrote before. A second default leaves the
effective value to iteration order. A foreign `max` below this default,
a `min` above the default request or a `maxLimitRequestRatio` rejects
pods outright, which takes out a whole namespace. At the Pod scope the
collision can't be decided in advance, because it depends on how many
containers a future pod has. A LimitRange that bounds only cpu or
storage coexists normally. A same-named LimitRange without the
operator's managed-by label is treated as foreign and left alone.

This matters most in `kube-system`, which is in scope because
`cozystack-scheduler` installs there, and where a distribution's or an
administrator's own LimitRange is likeliest to sit.

### Disabling

Setting `cozystackOperator.systemNamespaceMemoryLimit` to `0` disables
the feature. On the next reconcile the operator removes every LimitRange
it owns cluster-wide, found by the managed-by label. While the feature
is on, it also removes its LimitRange from namespaces no active Package
targets any more. The safety state on the Namespace stays in both cases.

### Docs

`docs/operations/system-memory-limits.md` landed in #3496 ahead of this
code, so main describes a `4Gi` default and installer values that do
nothing yet. The page is corrected to the 32Gi default and now describes
withholding, the acknowledgement, and the foreign LimitRange rule.

### Not in this PR

linstor policy. #3824 extends the controller's existing exemption from
limits to the satellite, and a linstor limit here would contradict it,
so linstor stays with that PR.

The scan reads six workload kinds through `APIReader`, uncached, on
every reconcile of every system namespace. That cost is tracked in
#3977, since the system namespace set isn't known when the cache is
built and the right fix depends on a reconcile-rate measurement nobody
has taken.

### Verification

`go vet` and `go test` pass for `cmd/cozystack-operator` and
`internal/operator`, and `helm unittest` passes on the installer chart
for the knobs unset, set, and the limit at `0`. The reconciler tests
cover withholding on an oversized request, the hold surviving the
blocker's removal, the hold on a lowered limit, acknowledgement and
token rejection, every memory field of a foreign LimitRange at both
scopes, ownership by label, the disabled sweep, and a deleting Package
not creating policy.

The withhold design has not been run on a live cluster yet. An earlier
design of this PR was, and that run does not carry over.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff.
`packages/core/installer/values.yaml` gains two keys, but the
`ansible-cozystack` trigger is for a required key or a rename of one the
role passes, and these are new, optional and default to `""`, so the
role is unaffected. Nothing else in the map is reached by the Go
changes, the operator template or the page.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(operator): system namespaces now get a default container memory limit through a LimitRange maintained by cozystack-operator, which keeps system components out of the Talos userspace OOM handler's victim set. Tune it with cozystackOperator.systemNamespaceMemoryLimit (default 32Gi) and cozystackOperator.systemNamespaceMemoryRequest (default 32Mi); set the limit to 0 to disable it. The default is far above any real working set on purpose, because the handler grants immunity for a memory.max existing at all, not for its value. A namespace with a container that requests more than the limit and declares no limit of its own gets no default, and stays without one until a cluster administrator acknowledges the hold recorded on the Namespace (see docs/operations/system-memory-limits.md); lowering a previously applied limit needs the same acknowledgement. A namespace where another LimitRange already sets anything about memory is left alone. A container that declares its own memory limit is never changed.
```
2026-10-07 11:19:26 +03:00
Aleksei Sviridkin 84e8fcd477 fix(dashboard): re-authenticate on 401 instead of failing silently (#3115)
## What this PR does

When the dashboard session expires, the console now sends the user back
to sign-in instead of sitting on a stale page. This re-applies
cozystack/cozystack-ui#34 by @IvanHunters onto the in-tree console,
since the UI was vendored into the monorepo (#2963) and the standalone
repo is being archived. It is part of the cozystack-ui retirement
tracked in #3097.

When the `kc-access` cookie expires, k8s API calls return 401.
`K8sClient` now treats a 401 as an auth boundary and makes one top-level
navigation to `/oauth2/sign_in?rd=<current-url>`. That path is the
sign-in entry both gatekeeper modes serve. With OIDC enabled,
oauth2-proxy goes straight to the provider (`--skip-provider-button`)
and returns the user to `rd`. Without OIDC, token-proxy renders its
token form there, but it does not honour `rd` yet, so the user lands on
`/` after sign-in.

The handler can be replaced through `K8sClientConfig.onUnauthorized` and
does nothing outside the browser. A latch keeps concurrent 401s from
redirecting more than once, and any 2xx from either fetch site re-arms
it, so a handler that re-authenticates in place still sees the next
expiry.

Related: #2788.

### Verification

`client.unauthorized.test.ts` covers the hook on the request and watch
paths, the latch across concurrent 401s, the re-arm after a success, and
the default handler's target and relative `rd`. `pnpm typecheck` and the
console test suite pass.

Original PR: cozystack/cozystack-ui#34, credited via `Co-authored-by`.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(dashboard): on a 401 (expired session) the console now sends the user to sign in again instead of silently showing a stale page. With OIDC enabled the user returns to the page they were on; without OIDC they land on the start page after entering a token.
```
2026-10-07 11:14:37 +03:00
IvanHunters 3cce629535 feat(redis,postgres): certify the per-engine external hostname (#4627)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Redis and postgres server certificates now carry the per-engine external
name `<release>.<engine>.<apex>` (for example
`redis-foo.redis.example.org`) next to the flat `<release>.<apex>` they
already had. Both names appear under the same condition as before: TLS
on and `external: true`, with a tenant apex present.

A TLS-passthrough listener on the tenant Gateway forwards the handshake
untouched, so the certificate the client checks is the engine's own. The
design in #2815 routes databases by SNI under `*.redis.<apex>` and
`*.postgres.<apex>`, and without the per-engine name in the SAN list a
verifying client would fail the hostname check. Issuing both names now
means a later move between the two hostname schemes reissues no
certificate and keeps clients on the same trust anchor. This is the
certificate step of #2815. Nothing renders a route yet.

The SAN list changes, so an existing external TLS release gets its
server certificate reissued once on upgrade, from the same CA. For redis
the operator then rolls the Redis and Sentinel pods one at a time, the
same way it handles a renewal.

The READMEs say who can reach the new name through a passthrough
listener. For postgres that is only clients using direct TLS negotiation
(server 17+, `sslnegotiation=direct`), because by default a plaintext
SSLRequest comes before the handshake. `redis-cli` needs `--sni`.

MongoDB is not in this PR. PSMDB issues the certificate itself and its
CRD has no field for extra SANs, so adding the mongos name means the
chart taking over the certificate. That needs its own change.

Covered by helm-unittest: the name is present with TLS and external on,
and absent with external off, with TLS off, and in a tenant with no
apex.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(redis,postgres): server certificates of external redis and postgres releases also carry the `<release>.<engine>.<apex>` name that the tenant Gateway's TLS-passthrough listeners are designed to route by. Existing external TLS releases get their server certificate reissued once on upgrade, from the same CA.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* PostgreSQL TLS certificates now cover both the tenant hostname and the
per-engine `postgres` hostname for external connections.
* Redis TLS certificates now also cover the per-engine `redis` hostname
when external services are enabled.

* **Documentation**
* Clarified connection requirements for per-engine hostnames: PostgreSQL
routing requires direct TLS negotiation, PostgreSQL 17 or later, and a
compatible client; Redis clients must specify SNI.
* Clarified that the hostnames may need to be mapped manually to the
LoadBalancer address.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 10:56:34 +03:00
Aleksei Sviridkin b1c84e9c06 feat(platform, tests): select E2E packages and declare controller dependencies (#4022)
## What this PR does

E2E installs the whole platform on a fresh cluster for every PR,
whatever the PR touches, so install time and flake surface go to
packages the selected suites never use. This PR makes the E2E install
take only what the selected suites need.

`hack/select-install.sh --disabled <suites>` generates
`bundles.disabledPackages` for fresh E2E installs using an ordinary
platform variant. It keeps suite owners, runtime requirements and the
install baseline, then follows their forward `PackageSource`
dependencies.

The baseline follows the unconditional waits in
`hack/e2e-install-cozystack.bats` and stays seeded directly even where
the graph reaches the same packages. It also keeps platform services
that no suite owns but the rest of the platform uses: `multus`,
`cozystack-scheduler`, `linstor-scheduler`, `flux-plunger`, `cozy-proxy`
and `ouroboros` (the Kubernetes suites assert its hairpin NAT). Backup
suites also keep `backupstrategy-controller` and its dependencies, and
the gateway suite keeps `kubeovn-webhook`.

The E2E job now selects the suites before it installs, and passes the
complement to the platform Package as `disabledPackages`. The `full-e2e`
label and an empty selection keep the full install, and the nightly and
tag lanes pass no list.

The platform sources also change: `mongodb-application` now depends on
`mongodb-operator`, and `seaweedfs-application` on
`objectstorage-controller`. These dependencies apply to regular platform
installs and upgrades too.

The new `mongodb-operator` edge also reaches reverse test selection: a
change under `packages/system/mongodb-operator` now selects the
`mongodb` suite instead of falling back to the full suite. Backup APIs
and the gateway webhook remain suite-specific install requirements, so
they add no edges.

The disable list governs fresh installs because existing Packages carry
`helm.sh/resource-policy: keep`. Optional packages still need
`bundles.enabledPackages`. Packages that no suite owns and the baseline
does not keep stay uninstalled on purpose, also when test selection
answers with every suite (monitoring exporters like `goldpinger` and
`kubeovn-plunger`, apps without a suite, provider-specific operators).
Nothing in the run looks at them, so installing them only adds a
readiness wait, and a change to such a package gets no install check
from this gate.

### Measured

One run per variant on a 32 vCPU / 128 GB VM, container lane, the CI
artifacts of this PR, `postgres` suite selected, 55 packages disabled:

- HelmReleases 111 to 77, pods 219 to 197
- pod requests 14.8 CPU / 37.7 GiB to 13.5 CPU / 35.7 GiB
- host memory peak during install 33.4 GB to 30.2 GB
- `install-cozystack` 26m33s in both runs
- `postgres` suite 25m50s to 25m17s, passes on the reduced install

Install time did not change. The removed packages are mostly operators
and apps without instances, and what the install waits on (SeaweedFS,
LINSTOR, CDI, the Cluster API providers, Keycloak) stays in the
baseline. With one run each, differences of a minute or two are noise.

The selection, the install closure and the workflow wiring are covered
by bats tests (`hack/select-install_test.bats`,
`hack/select-e2e_test.bats`, `hack/platform-packages_test.bats`) and by
a helm-unittest suite in the platform chart.

### Screenshots

Not a UI change.

### Downstream repositories

The new dependencies ship in the platform chart. No coordinated changes
are needed in downstream repositories.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(platform, tests): E2E installs only the packages the selected suites need. MongoDB now depends on its operator, and SeaweedFS on the object storage controller.
```
2026-10-07 09:36:24 +03:00
Aleksei Sviridkin 5486d4ffcf fix(networking): harden webhook policy assumptions (#4058)
## What this PR does

This is follow-up for
[#3997](https://github.com/cozystack/cozystack/pull/3997). It reads
ingress-nginx admission port from effective subchart values and refuses
empty value, and fixes `kubeProxyReplacement=false` status because
cozystack has no supported Service translator in that state. Gateway
test now requires one real ClusterIP and keeps same ready Pod UID, IP
and port for whole probe, so stale EndpointSlice, replaced Pod or wrong
Service can't report deny. Policy selectors and two-ended deny from
[#3997](https://github.com/cozystack/cozystack/pull/3997) stay
unchanged, and this does not claim root cause for
[#3983](https://github.com/cozystack/cozystack/issues/3983).

### Screenshots

Not applicable.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(networking): use effective ingress admission port, report unsupported Cilium service routing correctly, and reject stale backend evidence in gateway tests
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
- Admission webhook connectivity checks now validate the configured
Service and its exact backend endpoint throughout probing, improving
failure detection.
- Admission webhook policies now fail closed when the webhook port is
not explicitly configured.

- **Documentation**
- Clarified that supported service routing requires
`kubeProxyReplacement: true`.
- Updated guidance to explain that disabling this setting or webhook
policies does not provide a supported fallback.

- **Tests**
- Added coverage for missing webhook port configuration and strengthened
networking stability checks.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 08:37:53 +03:00
IvanHunters ac4ab1c4a6 chore: drop private cluster and host names from the tree (#4461)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

I remove internal cluster names, a real hostname with its IP addresses,
internal mirror hostnames and an internal product name from comments, an
example file, test fixtures and a dead commented-out config block. Where
a stand-in is needed I use neutral test-cluster wording, RFC 2606
domains or RFC 5737 documentation addresses. I also replace two real
Keycloak usernames and a reviewer's name in code comments with
placeholders, or drop them. Rendered output does not change. The test
fixture hostnames change on the input and the expected side together,
and the unit tests of every touched package still pass.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Documentation**
* Updated backup, OIDC, autoscaling, and platform references to describe
test environments without naming specific development clusters.
* Replaced development-specific endpoints in the platform example with
`example.org` and documentation-only IP addresses.
  * Updated OIDC username examples.
* **Tests**
  * Updated fixture hostnames and expected URLs to use `example.org`.
* **Chores**
* Removed commented-out registry mirror entries; the active mirror
configuration is unchanged.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 08:05:19 +03:00
Aleksei Sviridkin bc9a594e47 test(clickhouse): add backup API render assertions (#4011)
## What this PR does

This adds unit tests for how the ClickHouse chart wires up the backup
API that the default backup strategy calls. The Pod of the
`cozy-default-altinity` strategy reaches the clickhouse-backup sidecar
on the named port `ch-backup-api`. It authenticates with the credentials
from the `<release>-backup-api-auth` Secret.

The new suite is
`packages/apps/clickhouse/tests/backup_api_contract_test.yaml`. It
checks that:

- the sidecar exposes `ch-backup-api` on 7171/TCP;
- `API_USERNAME` and `API_PASSWORD` come from the `username` and
`password` keys of `<release>-backup-api-auth`;
- that Secret renders with `username: backup` and a non-empty password;
- no auth Secret renders when backups are disabled.

All of this is rendered by the chart, so the unit tests catch a
regression on every pull request. The e2e assertions on the same wiring
run only when test-impact analysis selects the ClickHouse suite. They
stay as they are in `hack/e2e-chainsaw/clickhouse`, because they check
the operator-created Pod and the admitted Secret, and rendering can't do
that.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```
2026-10-07 06:58:44 +03:00
Aleksei Sviridkin a1416655f4 fix(tenant): allow tenant egress to the KubeVirt export proxy
The vm-exportproxy Ingress is served by the ingress controller in
tenant-root, and tenant egress is default-deny. Nothing allowed traffic
to virt-exportproxy in cozy-kubevirt, so the Ingress stayed unreachable
even with the right backend Service. The new policy renders wherever
allow-to-cdi-upload-proxy does, but narrows the destination to the
export proxy pods and their container port.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 06:23:28 +03:00
Aleksei Sviridkin c069a191fc fix(monitoring-agents): give kube-state-metrics a sized memory limit (#3846)
`kube-state-metrics` ran BestEffort with no resources at all: the
subchart ships `resources: {}` and our wrapper never set any. So it put
no `memory.max` on its pod cgroup, and the Talos userspace OOM handler
only ever selects cgroups that lack one, which left it permanently in
the victim set. Any tenant workload driving node memory pressure could
evict the source of every kube-state metric we alert on. Background is
in `docs/operations/system-memory-limits.md`.

## The numbers are measured

On a three node cluster with 322 pods and roughly 25.6k series, which is
the smallest we run: 14 day peak working set 307Mi, peak RSS 280Mi, mean
176Mi.

So the request sits above the mean at 192Mi rather than at the peak, and
the limit carries wide headroom at 2Gi. The headroom is deliberate: this
component's memory scales with the number of objects in the cluster, and
the `--metric-labels-allowlist=pods=[*],deployments=[*]` already in this
file widens every pod and deployment series further, so a cluster larger
than the one measured lands well above what was seen here. `2Gi` is 6.7x
that peak, which read as linear in object count is roughly 2100 pods and
170k series, so the limit stays a ceiling rather than a bound well past
the largest cluster this sizing was drawn from.

No CPU limit, following the node DaemonSet sizing in `c9ba7a3cc`, which
set one zero times across nineteen files. CPU is irrelevant to the OOM
ranking, and a ceiling that binds on a scraper turns into scrape
timeouts rather than backpressure.

## This supersedes #2632

#2632 proposes 256Mi, which is below the 307Mi peak measured on the
smallest cluster we have, so it would convert an eviction risk into a
reliable OOM kill. It also adds a `cpu: 200m` limit, which is the half
that buys nothing. The diagnosis there was right and it sat since May;
the numbers are what needed replacing.

## Checks


`packages/system/monitoring-agents/tests/kube_state_metrics_resources_test.yaml`
pins the limit, both requests, and the absence of a CPU limit. Mutation
checked: dropping the limit to 256Mi turns it red. The chart's suites
are 3 files and 6 tests, all green.

### Release note

```release-note
fix(monitoring-agents): kube-state-metrics now carries a memory request and limit sized from a measured peak, so it is no longer a permanent candidate for the node OOM handler
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Configured kube-state-metrics with requests of 50m CPU and 192Mi
memory, plus a 2Gi memory limit. No CPU limit is configured, allowing
CPU usage to scale as needed. These settings help ensure the service has
the memory and CPU resources it needs while keeping memory use bounded.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 06:19:17 +03:00
Myasnikov Daniil 5b585d6eda fix(operator): persist holds for unsafe memory defaults
An admitted Pod loses the distinction between explicit and defaulted
limits. Its disappearance cannot prove a lower default is safe for
its replacement. Keep that decision on the Namespace until an
administrator acknowledges the particular target, without waiving
current blockers or foreign policy ownership.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:06:51 +03:00
Myasnikov Daniil 30ecc41070 fix(operator): close memory policy safety gaps
Preserve foreign same-name objects and keep create-time SSA non-force.

Only ignore failed pods controlled by terminal Jobs.

Make the failure-containment test inject Apply errors for real.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:06:51 +03:00
Myasnikov Daniil 6a521f0c68 fix(operator): sweep memory defaults when disabled
List LimitRanges by the ownership label once per Package reconcile when
the feature is disabled, and remove managed objects even in namespaces
outside the current Package target set. Skip the per-namespace path while
disabled and keep cleanup failures non-fatal to namespace reconciliation.

Update the flag help, installer guidance, and operations documentation to
describe the global cleanup boundary.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:06:49 +03:00
Myasnikov Daniil f6dcb69b61 refactor(operator): withhold instead of raising, and drop the grace period
The raise-and-grace machinery cannot work: LimitRanger writes the
raised limit onto the very container that justified the raise, so at
that container's next admission it carries a limit, the scan skips
containers that have one, and the raise has erased its own evidence.
The ceiling then drops on schedule and the workload is rejected
permanently. Any such machinery approximates admission logic ahead of
admission, and that is incomplete by construction: a
LimitRange default is one number for a whole namespace and cannot mean
"the configured limit, or this container's request, whichever is
larger". Only per-pod conditional defaulting expresses that, which is a
mutating webhook and not this.

The justification for the machinery had also expired. It was designed
when the default was 4Gi, where withholding was self-perpetuating for
VPA-managed namespaces. At 32Gi nothing in the tree approaches the
ceiling -- the largest static memory request is 2Gi -- and VPA sets
requests and limits together, so those containers carry their own limit
and were never blockers.

So a namespace holding a container that requests more than the
configured limit with no limit of its own is now left with no default at
all, and anything written there earlier is withdrawn. It fails in the
safe direction, which is the whole argument: a namespace with no
memory.max is a namespace behaving as it did in every release before
this, while a default below a request is a namespace whose workload
cannot be admitted. It needs no stored state and no timer, because
withholding is not a decision that has to survive between reconciles --
once the oversized request is gone the default is applied again on the
next pass.

Gone with it: the grace period, the clear-since annotation and its
clamp, the injected clock, containerMemoryDefault, raisedCeiling, and
fourteen tests. The scan stays, because deciding whether to apply or
withhold is what it was always for. The foreign-LimitRange ownership
guard, the disable path and the 32Gi default are untouched.

The residual sharp edge is stated rather than machinery: a workload
whose only evidence is a live Pod, built from a custom resource with no
template this scan can read, can still be missed if the scan runs while
that Pod is absent, and applying the default then rejects its
recreation. At 32Gi that is unreachable in this tree. It becomes
reachable if an operator lowers the knob below such a workload's
request, which the flag help, values.yaml and the operations page now
say plainly.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:06:48 +03:00
Myasnikov Daniil 81c12bc953 fix(operator): withhold on any foreign memory policy, not two fields
The coexistence guard treated a namespace as already-governed only when
another LimitRange carried a container memory default or max, and said
so in its comment. That is not the set of fields that collide with what
this reconciler writes. A container min above the default request
rejects the pod, and so does a maxLimitRequestRatio, since a 32Gi
default against a 32Mi default request is a ratio of 1024:1 and no cap
allows it. So the namespace-wide admission failure the guard exists to
prevent stayed reachable through fields the guard did not read, and
kube-system is where it is likeliest, this feature having deliberately
brought it in scope.

Enumerating the colliding fields does not fix it, because at the Pod
scope the collision is not decidable here. Whether a per-container
default of 32Gi breaches a pod-wide max depends on how many containers
the pod has, and the pods in question do not exist yet: any arithmetic
about them is a guess, and a wrong guess costs a namespace its entire
pod admission. So the rule is ownership rather than arithmetic. If
anything else in the namespace has an opinion about memory, at either
scope and in any field, the namespace is not this reconciler's to
default. Narrow still on the resource, because nothing here writes cpu
or storage and a LimitRange bounding those must keep coexisting. This
is the rule the tenant-namespace exclusion already rests on.

The pod scan no longer skips Failed pods, only Succeeded. A pod that
ran to completion is never recreated from its spec, but an eviction or
a node crash leaves a Failed carcass holding the same oversized request
that its controller will recreate, and that carcass is frequently the
only surviving evidence the namespace needs a raised ceiling, since the
shape the scan looks for cannot be admitted once the namespace has
settled. Dropping it lowered the ceiling under a workload that was
coming back, which is what the grace period exists to prevent.

A grace stamp dated ahead of the clock is clamped to now. Unclamped it
held the ceiling for the skew plus the grace period, silently, so an
hour of skew bought ninety minutes instead of thirty. Clamping never
shortens the wait, it only stops it being extended.

The two flag defaults are named constants the flag registration and its
test both read. The test previously duplicated the literal and still
asserted 4Gi after the default moved to 32Gi, so it misinformed a
reader about the shipped value and pinned nothing.

The disable claim is narrowed to what the code guarantees. "Removes any
previously created one" was universal, but the reconciler only runs for
namespaces an active Package still targets, so a namespace whose
Package was removed keeps its LimitRange. The flag help, values.yaml
and the operations page now say that, and name the managed-by label for
finding leftovers by hand.

The test asserting a Pod-scoped memory max should be ignored is gone,
since under the ownership rule a Pod-scoped memory bound withholds.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:06:45 +03:00
Myasnikov Daniil bcfa50d7e9 test(e2e): install only the packages the selected suites need
The E2E job installs the whole platform on a fresh cluster before it
knows which suites will run, so every PR pays install time and flake
surface for packages its suites never use. Select the suites first and
pass the complement computed by select-install.sh to the platform
Package as disabledPackages.

The full-e2e label and an empty selection keep the full install, and
the nightly and tag lanes pass no list, so they are unchanged.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:53 +03:00
Myasnikov Daniil 974aaa0bac test(hack): regenerate the selective install fixture
Two optional Cluster API providers, Proxmox and in-cluster IPAM, were
added to the platform sources after the fixture was generated. The
drift test compares the fixture with the current disable list, so it
failed until they were added.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:53 +03:00
Myasnikov Daniil f46a9797b8 fix(hack): keep platform services in selective installs
Dropping every package that no suite owns also drops services the rest
of the platform relies on: multus for secondary networks, the cozystack
and LINSTOR schedulers, flux-plunger, cozy-proxy and ouroboros, whose
hairpin NAT the Kubernetes suites assert. Nothing waits for them, so no
PackageSource edge reaches them, and adding one would also change
reverse test selection.

The remaining packages stay uninstalled on purpose, also when test
selection answers with every suite: nothing in the run looks at them,
so installing them only adds a readiness wait.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:52 +03:00
Myasnikov Daniil 59555c895e fix(hack): retain controllers and APIs for selective installs
Tenant HelmReleases need a shard reconciler, and the root SeaweedFS
release creates COSI resources even when no backup suite is selected.
Without those controllers, a fresh reduced install cannot become ready.

The Kubernetes suites create worker pools through KubernetesNodes, and
MongoDB renders an operator-owned CR. Selecting only the cluster API or
application chart drops resources those suites require.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:52 +03:00
Myasnikov Daniil 361893e6f4 fix(hack): retain backup APIs in selective installs
Backup roundtrip suites create Bucket and backup resources that the
application graph does not require. A graph-only closure therefore
omits the APIs those suites need on a fresh install.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:51 +03:00
Myasnikov Daniil 610578be16 test(hack): verify selective install contract
A missing suite owner or a discarded discovery failure can leave a
reduced install without the APIs its tests exercise. Graph-only checks
cannot establish that the chart preserves the requested package set.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:50 +03:00
Myasnikov Daniil b6c33926dc fix(dashboard): re-arm the 401 latch once the session recovers
unauthorizedHandled was set once per client and never cleared, so it
collapsed every later 401 too, not just the concurrent ones it was meant
to. Harmless for the shipped default handler, which navigates away and
takes the client with it, but the onUnauthorized hook exists precisely so
a caller can re-auth in place, and such a handler would fire exactly once
for the client's lifetime and then go silent.

Clear the latch on any 2xx from either fetch site: a success means the
session is alive again, so the next 401 is a new episode. The storm guard
still holds, since concurrent 401s have no success between them.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:58:15 +03:00
Myasnikov Daniil 7e5f9ab152 fix(dashboard): re-auth via /oauth2/sign_in so token-proxy mode works
The dashboard gatekeeper runs one of two mutually exclusive proxies,
chosen by the cluster's oidc-enabled flag (templates/gatekeeper.yaml).
/oauth2/start exists only in oauth2-proxy. The token-proxy branch
registers only sign_in, sign_out and userinfo under the /oauth2
prefix, so there /oauth2/start falls through to the catch-all
reverse-proxy handler, which does not re-verify the token. With a
present but expired cookie it proxies the request to the console
upstream, the SPA reloads, gets a 401 again and loops.

/oauth2/sign_in is the one entry point both modes implement, so the
default 401 handler goes there. oauth2-proxy's sign_in hands straight
off to OAuthStart because the gatekeeper passes --skip-provider-button,
and it resolves rd through the same app-director, so the OIDC path
behaves as it does through /oauth2/start. token-proxy serves its token
form there. It does not honour rd yet, so after sign-in the user lands
on / instead of looping.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:58:14 +03:00
Myasnikov Daniil badbec63fe test(dashboard): cover the 401 -> oauth2-proxy redirect path
The 401 handling is what brings an expired session back to sign-in,
and a regression in it stays invisible until a cookie expires. The
test drives K8sClient through the vi.stubGlobal("fetch", ...) harness
the sibling k8s-client tests use. It checks that the hook fires once
per 401 from both the request and the watch fetch sites, once across
concurrent 401s, never on other responses, and that the default
handler passes the current location as a relative rd, which
oauth2-proxy's open-redirect check accepts.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:58:14 +03:00
Myasnikov DaniilandIvanHunters 8bf78d3bfe fix(dashboard): redirect to oauth2-proxy on 401 instead of failing silently
When the kc-access cookie expires, k8s API calls return 401 and the
SPA stays on a stale page until the user clears the cache by hand.
K8sClient now treats 401 as an auth boundary and makes one top-level
navigation into the oauth2-proxy sign-in flow, passing the current
location as rd so the user comes back to the same page. The handler
can be replaced through K8sClientConfig.onUnauthorized and does
nothing outside the browser. The unauthorizedHandled latch keeps
concurrent 401s from redirecting more than once.

Re-applied from cozystack/cozystack-ui#34 onto the in-tree console
after the UI was vendored (#2963), since the source repo is being
archived. Related: #2788.

Co-authored-by: IvanHunters <49371933+IvanHunters@users.noreply.github.com>
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:58:13 +03:00
Myasnikov Daniil 3d901ca9b4 test(clickhouse): add backup API render assertions
The port the backup strategy dials, the two secretKeyRefs the
clickhouse-backup sidecar authenticates with, and the auth Secret that
backs them are all rendered by this chart. A helm-unittest suite
catches a regression in them on every pull request, while the e2e
assertions on the same wiring run only when test-impact analysis
selects the ClickHouse suite. The e2e assertions stay, since they
check the operator-created Pod and the admitted Secret.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:52:28 +03:00
Myasnikov Daniil 9fe93a3d7e docs(operator): fix three claims that overstated what the code does
The static-pod note said containers built from an on-disk manifest are
unaffected "in both directions". True of the container, which the
kubelet runs from disk whatever the API says, but the mirror pod is an
ordinary API object that passes admission like any other, so a static
pod declaring a memory request above the ceiling with no limit of its
own has its mirror pod rejected -- leaving the container running with no
representation in the API. Worth naming, since kube-system is in scope
by design. The scan does cover it: a mirror pod is a live Pod.

The ReplicaSet exclusion was justified on live ones being copies of
their Deployment's template. That does not reach an ownerless
ReplicaSet, which at replicas 0 has neither pods to be seen through nor
a Deployment template standing in for it. It stays out of scope --
nothing here ships a bare ReplicaSet -- but as a gap left open rather
than one the stated reason closes.

The delete guard's comment claimed the enabled apply always adopts and
stamps a same-named foreign object. It does not when that object's spec
already matches, because then the apply is skipped entirely. The
outcome is the same one the guard gives on purpose, but it is reached by
the apply not running.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:54 +03:00
Myasnikov Daniil 1df6e3ac8f revert(linstor): leave the controller memory limit to its own PR
This branch gave linstor-controller a 10Gi memory limit, justified in
the values and pinned in cluster_test.yaml on the grounds that the
namespace default was 4Gi, half the -Xmx8G baked into the image.

Both halves of that have moved. The default is now 32Gi, four times the
heap ceiling rather than half of it, so nothing about the JVM requires a
per-chart limit to sit above the namespace default any more. And #3824
argues the opposite direction for the same component -- extending the
controller's existing limit-free exemption to linstor-satellite, using
that exemption as its precedent. Keeping the 10Gi here would remove the
precedent that PR reasons from, and whichever of the two merged second
would silently invalidate the other's rationale.

So linstor policy belongs to one PR, and it is not this one. The three
files go back to their state on main; this branch is now confined to
the operator and the installer chart.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:53 +03:00
Myasnikov Daniil e131a49514 feat(operator): default the system namespace memory ceiling to 32Gi
The guard in front of this default reads declared memory requests. It
cannot see a container's real working set, so a system component that
requests little but legitimately uses more than the ceiling, and
declares no limit of its own, was handed a deterministic kernel OOM on
its next recreation in exchange for leaving the Talos handler's victim
set. Since the feature ships enabled, that trade fell on the whole
fleet on upgrade.

It is not a trade this feature has to make. The handler discards a
cgroup for carrying a memory.max at all, whatever its value, so
immunity does not require a fitted ceiling -- only a real one. A
default far above any plausible working set buys the immunity and
leaves nothing for the ceiling to bind on, which also keeps it clear of
the failure mode this repository already carries a scar from, where an
undersized limit converted a rare pressure-driven kill into a reliable
one.

Nothing about scheduling changes: only requests are scheduled against,
and the paired default request stays 32Mi. An operator who wants the
ceiling to bind lowers it, per component in a chart or fleet-wide with
this knob, once real usage is measured -- which is the direction that
needs evidence, and now the direction that needs it rather than the
default.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:53 +03:00
Myasnikov Daniil b6cb0def78 docs(operator): say that kube-system is in scope for the memory default
The scope of the default LimitRange is every namespace a Package targets
whose name does not start with `tenant-`, and cozystack-scheduler installs
into kube-system, so a namespace owned by the underlying Kubernetes
distribution is defaulted like any other. Neither the flag help nor the
installer value said so, and both are what somebody setting the knob
reads.

docs/operations/system-memory-limits.md already covers this, including the
part that surprises: containers the kubelet builds from an on-disk
manifest never pass through admission, so a Talos control-plane static pod
gets no memory.max from this feature in the first place and the mirror pod
is the only thing the LimitRange reaches. Both notes now carry the same
statement, so neither reading of the knob leaves it out.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:51 +03:00
Myasnikov Daniil dd36eeddbc docs(operator): describe the raise, not the retract, in the shipped help
The flag help and the installer values comment both still said the
operator skips a namespace holding a request above the configured limit
and retracts any LimitRange it applied there. It stopped doing that in
3127c0fad: it raises that namespace's ceiling to clear the request and
drops back once the request is gone. An operator reading the shipped help
and then finding an 8Gi LimitRange in cozy-monitoring had no way to
connect the two.

Both texts now describe the raise, the drop-back and the log line to grep
for. The values comment also loses its VPA warning, which this branch's
own comment in findRequestAboveDefaultLimit refutes — VPA rescales the
limit in proportion to the request under the default controlledValues, so
it cannot be what puts a namespace over — and gains a pointer to
declaring a limit in the component's own chart, which is how
packages/system/linstor now stays out of the default's reach.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:47 +03:00
Myasnikov Daniil ae431808ea fix(linstor): give linstor-controller a memory limit above its baked-in -Xmx8G
linstor-server bakes `-Xms256M -Xmx8G -XX:+CrashOnOutOfMemoryError` into
the generated bin/Controller (build.gradle, jvmControllerOpts), nothing in
this chart or in piraeus-entry.sh overrides it, and a JVM handed an
explicit -Xmx never reads memory.max. The container was deliberately left
with a 700Mi request and no limit at all, which was the safe choice while
nothing else supplied one.

It stops being safe alongside the system-namespace LimitRange: a container
declaring no memory limit is given the configured default, 4Gi, so the
process's own heap ceiling would sit at twice its cgroup ceiling and a
large enough DRBD resource count would have the kernel kill the storage
control plane where the JVM would otherwise have crashed itself loudly at
8G. Stating the limit here is also what keeps the container out of the
LimitRange's reach, since a container with its own limit is never
defaulted one.

10Gi is the 8G heap plus room for metaspace, code cache, thread stacks and
direct buffers. CPU stays unlimited. The helm-unittest assertion that
pinned "no limits" is replaced by one pinning the memory limit and the
absence of a CPU limit, so neither lowering it below the heap ceiling nor
dropping it again passes silently.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:47 +03:00
Myasnikov Daniil 546d8e6a54 feat(operator): default system namespace memory via LimitRange
The Talos userspace OOM handler (v1.12+) picks its victim by ranking
cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable:
0.5, ...}[class] * memory_current`, discarding every cgroup that scores
zero. Only a memory limit sets memory.max on the pod cgroup, so only a
limit takes a pod out of the candidate set: a request merely demotes
BestEffort to Burstable, which under the v1.13.6 default
strictCgroupClassOrdering still means "killed second". PriorityClass has
no effect at all, because the handler walks cgroupfs and never talks to
the API server.

Victim selection is decoupled from the trigger, so the cgroup that
caused the pressure and the cgroup that gets SIGKILLed are unrelated by
design. Tenant workloads generally carry limits and system components
generally did not, which inverted the intended order: a tenant workload
thrashing against its own limit drives node-wide memory PSI, and
metallb-speaker or linstor-satellite is killed for it while the workload
responsible is never touched.

An audit found roughly 70 workloads across ~30 packages with at least
one container missing a memory limit, so per-chart resources would
neither cover the current set nor keep covering it. Maintain a
LimitRange in every system namespace instead, at the single choke point
that already creates them and already distinguishes system from tenant
(reconcileNamespaces). Tenant namespaces are left alone because
packages/apps/tenant owns LimitRange policy there.

The limit is a ceiling, not a reservation: it defaults well above real
usage (4Gi against a largest observed system request of 512Mi).
defaultRequest is set explicitly and small, because Kubernetes otherwise
defaults each request to the limit and reserves that much at schedule
time. Setting the limit to 0 disables the LimitRange and removes any the
operator previously created.

Two things keep the feature from being able to hurt more than it helps.

A LimitRange default only reaches a container that declares no memory
limit of its own, and the API server then rejects a request above its
limit. So before applying, scan the namespace for a container that
requests more than the configured limit and sets none, and where one
exists skip the namespace, retract any LimitRange applied earlier, and
log the pod, container, request and limit. This is not hypothetical:
packages/system/monitoring and packages/system/monitoring-agents ship
VPAs for cozy-monitoring with maxAllowed.memory of 8Gi and 6G under
updateMode: Initial, so the VPA admission webhook -- which runs after
the in-tree LimitRanger -- can write a request past 4Gi that no chart in
the tree declares. That is why the check runs against live pods rather
than rendered templates, and why it reads through APIReader rather than
starting a cluster-wide Pod informer.

And no failure inside the feature propagates: a LimitRange error, or a
pod list that cannot be read, is logged and stepped over instead of
failing the Package reconcile. A namespace without a LimitRange is back
to the behaviour of every earlier release, while a namespace that never
reconciles leaves its components uninstalled.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:36:41 +03:00
Myasnikov Daniil f318157d69 docs(networking): qualify webhook translation fallback
Limit the fail-open discussion to the actual counterexample: another component translating the Service after Cilium source-side egress policy. A disabled translator by itself leaves no working Service path.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:00:04 +03:00
Myasnikov Daniil 79e236aa7a test(networking): harden webhook regression proof
Probe one exact ready admission EndpointSlice backend throughout the measurement window and reject missing or ambiguous Service ports. Ensure the kubeovn-webhook Helm suites run through package discovery.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:00:03 +03:00
Myasnikov Daniil c9558f5658 fix(networking): correct webhook policy assumptions
Read the ingress admission port from the effective subchart values and fail if the port is absent. Describe kube-proxy replacement as a required service-routing invariant while retaining target-side enforcement.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 05:00:03 +03:00
Myasnikov Daniil 361311c5c9 docs(monitoring-agents): note what object count the 2Gi limit covers
The sizing comment named the measured peak without saying how much
room 2Gi leaves above it, which left open why the chart pins 2Gi when
the default LimitRange limit proposed in #3494 is 4Gi.

2Gi is 6.7x the measured 307Mi peak working set. Read as linear in
object count, that is roughly 2100 pods and 170k series against the
322 pods and ~25.6k series the peak was measured on, so the limit
stays a ceiling rather than a bound on a cluster several times larger
than the one behind the numbers.

Comment only. The requests, the limit and the deliberate absence of a
CPU limit are all unchanged.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 04:56:09 +03:00
Myasnikov Daniil 2e533acf4e fix(monitoring-agents): give kube-state-metrics a sized memory limit
It ran BestEffort with no resources at all, so it set no memory.max on
its pod cgroup and the Talos userspace OOM handler kept it permanently
in the victim set: any tenant workload driving node memory pressure
could evict it, and it is the source of every kube-state metric we
alert on.

Numbers are measured rather than guessed. A 14d peak on a 3-node
cluster with 322 pods and ~25.6k series is 307Mi working set, 280Mi
RSS, 176Mi mean. The request sits above the mean and the limit carries
wide headroom, because this component scales with object count and
--metric-labels-allowlist widens every pod and deployment series
further, so a larger cluster lands well above what was measured.

No CPU limit, following the node DaemonSet sizing in c9ba7a3cc: CPU is
irrelevant to the OOM ranking, and a binding ceiling on a scraper turns
into scrape timeouts rather than backpressure.

Supersedes #2632, whose 256Mi limit sits below the measured peak.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 04:56:03 +03:00
Aleksei Sviridkin 48e3d1d611 feat(redis,postgres): certify the per-engine external hostname
The tenant Gateway's TLS-passthrough listeners are designed to route
databases by SNI under a per-engine subdomain (*.redis.<apex>,
*.postgres.<apex>), and a passthrough listener forwards the handshake
untouched, so the engine's own server certificate must carry
<release>.<engine>.<apex> or a verifying client fails the hostname
check.

The name is issued next to the existing flat <release>.<apex>, under
the same TLS-and-external condition, so that a later move between the
two hostname schemes reissues no certificate and leaves clients on the
same trust anchor. Nothing renders a route yet; the certificate is
prepared ahead of it. Adding the name changes the SAN list, so existing
external TLS releases get their server certificate reissued once on
upgrade, from the same CA.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 04:43:15 +03:00
Aleksei Sviridkin 92bf90cc7d fix(kubernetes): force the kubevirt-csi temp NFS unmount after a timeout
The NFS publish path temp-mounts the export root and unmounts it in a
defer with a plain umount, which has no deadline. If the NFS server is
gone by then, that umount can block indefinitely: the publish hangs and
the temp mount stays behind for the life of the node plugin container.

mount-utils' UnmountWithForce cannot fix this. It retries with -f only
after its first umount returns, and that umount forks the umount.nfs
helper: killed on the timeout, umount leaves the helper stuck on the
vanished server with the output pipe still open, and CombinedOutput
waits for the pipe, so -f never runs. The plugin therefore runs umount
itself with WaitDelay set, which closes the pipe shortly after the
kill, and falls back to umount -f under a timeout of its own. As in
mount-utils, "not mounted" counts as done: umount2 detaches the mount
before it blocks tearing the superblock down, so the -f after a killed
umount can find nothing left. A mounter that is not the exec-based one
keeps its plain Unmount.

An umount in uninterruptible sleep outlives any kill, so the cleanup
also waits at most one fixed budget, well inside the 2m deadline
kubelet puts on NodePublishVolume, and leaves a stalled unmount
running in the background, where it logs its result once it ends. The
temp dir is removed by that background unmount once it succeeds, not
by the publish: removed at the deadline it is still a mount point, and
nothing would retry.

A forced unmount runs nfs_umount_begin, which kills every RPC in
flight on the superblock, and NFS shares one superblock between mounts
of the same export with the same options. The temp mount therefore
gets nosharecache, so -f cannot fail the I/O of pods on the node that
mount the same volume.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 04:38:53 +03:00
Aleksei Sviridkin 520af57905 chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out
internal mirror block, an internal product name, and a reviewer's
name in a comment do not belong in a public repository; drop them
and replace what needs a stand-in with neutral placeholders.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 04:38:06 +03:00
Aleksei Sviridkin d2e3eb4f82 fix(kubernetes, e2e): unblock autoscaling below min size, size the computeplane pool, fix the backupcontroller build (#4768)
## What this PR does

Makes the backupcontroller test package compile again on main. #4587
added a flavor argument to `createCNPGBackupArtifact` and
`marshalCNPGBackupSnapshot`, and #4669, whose CI ran before that, added
two test calls with the old argument lists. Both PRs were green on their
own, and together they break `go test ./internal/backupcontroller/`. The
two calls now pass the same default flavor as the other tests in the
file.

The second commit fixes an e2e suite that fails the required E2E check
here. It is the `computeplane` suite from #4713, and its worker pool was
one node with a single vCPU. After the kubelet reservations that node
has 900m of CPU to allocate. The tenant addons (cilium, coredns, csi,
ingress-nginx, metrics-server, cert-manager) request exactly 900m
between them. If cert-manager's startupapicheck hook pod is created
last, it can't be scheduled and the install times out after 10m. The
reinstall after remediation runs into the same full node. Whether that
pod comes last depends on install order, so the suite passed on #4713
and failed here.

The pool now has two workers with one vCPU each, the same shape the
kubernetes suites use. One vCPU per worker stays on purpose, because of
#3513.

Two workers exposed a deadlock in the tenant cluster-autoscaler. The
suite's pool is created at zero replicas and then gets a min size of 2,
which CAPI does not apply to the live replica count on upgrade (#4786).
The autoscaler's scale-up for a pending pod asks for one node, and the
clusterapi provider refuses it because one is under the min size,
although it is an increase. That refusal backs the group off, and the
min-size enforcement skips groups in backoff, so the pool never leaves
zero. Upstream had the same report in kubernetes/autoscaler#4942, closed
as stale, and it is now tracked in kubernetes/autoscaler#10400. Our
autoscaler image already patches the mirror case for max size, and the
patch now refuses only a decrease below the min size.

`go vet` and `go test ./internal/backupcontroller/...` pass, and so do
the bats pins for the computeplane suite. The autoscaler patch adds a
test for each direction, and the clusterapi package tests pass with it
applied to 1.32.0.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved cluster autoscaling for worker groups outside configured size
limits: groups below their minimum can scale up toward it, and nodes can
be removed when a group exceeds its maximum. Scale-down requests below
the minimum remain restricted.
* **Tests**
* Updated backup and restore test calls with PostgreSQL and empty flavor
values.
* Updated ComputePlane checks for a two-worker pool and clarified
cert-manager installation timing expectations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 04:36:10 +03:00
Aleksei Sviridkin 82ae9f6cf7 fix(kubernetes): let the autoscaler grow a node group that sits below min
The clusterapi provider refuses any SetSize under the group's min size,
including an increase. A pool whose min size is raised while it stands
at fewer replicas then deadlocks: the scale-up for pending pods asks for
fewer nodes than the min, is refused, and puts the group in backoff,
and the min-size enforcement skips groups in backoff. The group stays
where it is for good.

Refuse only a decrease below min size. The guard it keeps is the one
DecreaseTargetSize relies on, and DeleteNodes checks min size itself.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 00:58:11 +03:00
Aleksei Sviridkin 9620be0103 feat(postgres): add per-user createRole and bypassRls attributes (#3505)
## What this PR does

- Adds `users.<name>.createRole` to the postgres chart, which sets the
`CREATEROLE` role attribute on a managed login user.
- Adds `users.<name>.bypassRls`, which sets `BYPASSRLS` on the same
user. The two attributes are needed together to provision a privileged
service role, see below.
- Sets `createrole_self_grant` as a role-level default alongside
`createRole`, without which the attribute is granted but not usable.
- Refuses `createRole` on PostgreSQL 15 and older, where it is an
escalation path rather than a bounded privilege.
- Makes the init job record the roles it manages in a superuser-only
marker instead of trusting role comments and refuses to adopt a role it
did not create once `createRole` has been used, see below.
- Adds `tests/user_createrole_test.yaml`: defaults, both explicit values
for each attribute, composition with `replication` and with each other,
the `createrole_self_grant` set and reset paths, and the version guard
on v15 and v13. Adds `tests/managed_roles_test.yaml` for the marker and
the adoption checks.

Today a tenant that needs to create its own roles has no way to do it
through the chart, because role attributes are not inherited through
role membership: granting `CREATEROLE` to the generated `<db>_admin`
group role has no effect on the users that are members of it, and
`<db>_admin` is `NOINHERIT` anyway. The attribute has to be set on the
login user itself, which only the init script can do. The remaining
option is to hand the tenant the `postgres` superuser, which also gives
it `COPY ... FROM PROGRAM`, arbitrary file access and command execution
inside the instance pod. These flags are the narrow version of that: the
tenant manages its own roles, and the superuser stays internal.

Both attributes are reconciled on every init-script run, like
`replication`, so setting one back to `false` emits the `NO...` form and
revokes it.

## Why `bypassRls` belongs in the same PR

`CREATEROLE` on its own does not let a tenant reproduce a role layout
that uses row-level security, which is the common case this feature
exists for. PostgreSQL only lets a role create other roles carrying
`BYPASSRLS` if it holds the attribute itself, so `CREATE ROLE
service_role ... BYPASSRLS` fails with `Only roles with the BYPASSRLS
attribute may create roles with the BYPASSRLS attribute`. Exposing
`createRole` without `bypassRls` therefore ships a feature that stops
one statement short of the goal.

`bypassRls` is not gated on a PostgreSQL version, because it carries no
escalation of its own: `BYPASSRLS` waives row-level security inside the
tenant's own databases, and the database owner already reads every row
in them. It grants no reach outside the tenant's data.

## Why `createrole_self_grant` is set with the attribute

This is the non-obvious half. PostgreSQL 16 grants a `CREATEROLE` holder
only `ADMIN OPTION` on the roles it creates, governed by
`createrole_self_grant`, whose default is empty. `ADMIN OPTION` allows
granting the role onward but not `SET ROLE` to it, and creating an
object owned by another role requires `SET`. The attribute therefore
looks functional -- `CREATE ROLE` succeeds -- while the very next
statement a tenant writes, `CREATE SCHEMA ... AUTHORIZATION <new role>`,
fails with `must be able to SET ROLE`. Every tenant granted `createRole`
would hit this and have to discover the parameter, so the chart sets it
as a role-level default and resets it when the attribute is withdrawn.

## Why the init job stops trusting role comments and role names

The init job runs as the superuser, and it trusted two things a
CREATEROLE holder controls for the roles it creates. It picked the users
and roles to drop by their comment, and on PostgreSQL 16 a CREATEROLE
holder may `COMMENT ON ROLE` every role it created, so it could hand the
chart for deletion a role the chart never created. It also adopted any
role that already existed under a declared name, so a role created under
a name an administrator declares later stayed under its creator's
control, for example `debezium` before `users.debezium.replication:
true`, or `newdb_admin` before `databases.newdb`.

Neither case can be caught from the catalog. Creating a role through an
intermediate role and then dropping that role leaves the squatted role
with no row in `pg_auth_members` at all, while the objects, role
settings and default privileges planted in it stay. On 17.11 such a
`debezium` had zero memberships, and after the chart adopted it
`migrator` could still run code as `debezium` with `REPLICATION` through
a function it had planted. The chart therefore keeps its own record:

- Chart-managed roles are members of `cozystack:managed`, a NOLOGIN role
that only the superuser can grant. On 16+ the membership is `WITH
INHERIT FALSE, SET FALSE`. The name is outside the username and database
charsets. The delete passes act only on these members. On its first run
the chart marks every role it had commented before, when no role could
yet hold CREATEROLE through it.
- Once any user has held `createRole`, recorded as
`cozystack.createrole_granted` on the `postgres` database (the same
mechanism as `cozystack.adopted_restore`), the chart refuses to adopt an
existing role it did not mark, both for users and for
`<db>_admin`/`<db>_readonly`. The init job fails with the role's name,
and a platform administrator must drop the role. Before that point
adoption works as before, so a fresh install that declares `users.app`
still adopts the role CNPG bootstraps.

## The PostgreSQL 16 floor on `createRole`

The floor is a security boundary, not a cosmetic check. Before
PostgreSQL 16, `CREATEROLE` allowed granting membership in any
non-superuser role, so a user holding it could grant itself
`pg_execute_server_program` and reach command execution on the instance,
which is exactly what not exposing the superuser is meant to prevent. I
verified both sides on a real cluster: on 15.15 `GRANT
pg_execute_server_program TO <createrole_user>` succeeds and `COPY ...
FROM PROGRAM 'id'` then returns `uid=26(postgres)`; on 16.11 and 18.1
the same grant is refused because it now requires ADMIN option on the
role. PostgreSQL 16 restricted `CREATEROLE` to roles the grantee
administers, which is what makes the attribute safe to expose. The guard
uses the same fail-fast `SECURITY:` idiom as the blocked
`postgresql.parameters` list in `templates/db.yaml`.

Note that a grant obtained on PostgreSQL 15 survives a major upgrade:
after moving an instance to 16.11 the user kept its membership in
`pg_execute_server_program` even though new attempts were refused.
Operators who exposed `CREATEROLE` out of band on an older major should
audit and revoke memberships in `pg_execute_server_program`,
`pg_read_server_files` and `pg_write_server_files` before relying on the
floor.

## Verification

The marker, adoption and `createRole` paths were rendered and run
against live PostgreSQL 13, 15, 16 and 17 containers. An upgrade from
the previous chart marks the roles it already managed and removes a
dropped user. What happens to a dropped role's objects comes from #4574
and is covered by its tests. A fresh install adopts the CNPG `app` role
in the same run that grants `createRole`. A forged comment is ignored,
and squatted `debezium` and `newdb_admin` roles stop the init job with
nothing granted to them. The orphaned-role pass still drops the roles of
a removed database, and every re-run is clean. Reverting each part of
the fix in turn fails `tests/managed_roles_test.yaml`.

Beyond the chart unit tests, I replayed a real migration checklist
against a live PostgreSQL 17.7 instance, connected as the managed login
user and never as the superuser: creation of four roles with their
attributes including `CREATEROLE` and `BYPASSRLS`, `CREATE SCHEMA ...
AUTHORIZATION` landing on the requested owner, trusted extensions into a
named schema, per-role and per-database `search_path`, per-role
`statement_timeout` and `idle_in_transaction_session_timeout`, RLS
including `FORCE ROW LEVEL SECURITY` verified end to end (a plain role
sees zero rows, the `BYPASSRLS` role sees them all), and `SECURITY
DEFINER` functions. All of it succeeds without the superuser. The one
thing that remains out of reach is `CREATE EVENT TRIGGER`, which
PostgreSQL restricts to superusers and no chart flag can change.

Other attributes and the predefined roles that a managed-service admin
role usually wants (`pg_monitor`, `pg_read_all_settings`, `pg_maintain`)
can follow the same per-user pattern in a later PR, minus the three that
grant OS access (`pg_execute_server_program`, `pg_read_server_files`,
`pg_write_server_files`).

### Screenshots

No UI change.

### Downstream repositories

`packages/apps/postgres/values.schema.json` gains two fields, which per
the trigger map reaches `cozystack/terraform-provider-cozystack`: its
postgres resource carries a hand-maintained schema plus an
expand/flatten pair for `users`, with no codegen from
`values.schema.json`, so `createRole` and `bypassRls` will be invisible
to Terraform users until they are added there by hand. I searched that
repository and found no open PR or issue covering it (the only open PR
is a dependabot Go module bump). I have not filed the follow-up, so I am
leaving the box unticked rather than claiming work that does not exist.
Happy to open the issue or the provider PR if a maintainer confirms that
is the preferred route.

I walked the rest of the map against the diff and found nothing else: no
package is added, renamed or removed, no version enum changes, no
default changes, `release.prefix` and the output Secret and Service
names are untouched, and no CRD outside the regenerated
`packages/system/postgres-rd/cozyrds/postgres.yaml` is affected. The
`README.md` parameter table is regenerated here and postgres is already
in the website generator's app list, so the website path stays
automated.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: not filed yet, see the note above
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
feat(postgres): add `users.<name>.createRole` and `users.<name>.bypassRls`, which grant the CREATEROLE and BYPASSRLS attributes to a managed user so a tenant can create and manage its own roles, including roles that bypass row-level security, without the postgres superuser. `createRole` also sets `createrole_self_grant` on the user so that roles it creates can own objects. `createRole` requires PostgreSQL 16 or newer: on older majors CREATEROLE can be used to grant oneself pg_execute_server_program and reach command execution on the instance, so the chart refuses to render the attribute there. The init job now records the roles it manages as members of a superuser-only `cozystack:managed` role instead of trusting role comments. Once `createRole` has been used on an instance, it refuses to adopt an existing role it did not create.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* PostgreSQL user configuration now supports `createRole` and
`bypassRls` privileges.
* These privileges can be combined with replication access. Enabling
`createRole` also configures the role’s self-grant behavior.
* `createRole` requires PostgreSQL 16 or newer; configuring it on older
versions is rejected.

* **Documentation**
* Updated configuration schemas, examples, and guides to describe the
new options and PostgreSQL version requirements.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 20:03:34 +03:00