Commit Graph
600 Commits
Author SHA1 Message Date
Aleksei Sviridkin 641faa7fde build(kafka-operator): vendor Strimzi 1.2.0
Strimzi 0.45 is out of upstream support. Strimzi 1.x serves only the
v1 API, so the release now replaces the CRDs on upgrade;
helm-controller's default of skipping CRD upgrades would leave the
v1beta2-only CRDs from the first install in place. Strimzi 1.x runs no
ZooKeeper, so the ZooKeeper alert group goes too.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-17 12:22:54 +03:00
Aleksei Sviridkin af0db99c71 feat(platform): move Strimzi resources to the v1 API before Strimzi 1.x
Strimzi 1.0 removed v1beta2 from its CRDs, and the API server refuses
a CRD that drops a version its status still lists as stored. Every
stored object has to be rewritten as v1 before the new operator
arrives, which needs the CRDs of Strimzi 0.49 to 0.51 that serve both
versions and Strimzi's own conversion tool. The 0.45 operator is
stopped first: Strimzi supports the conversion only under an operator
that knows v1, and 0.45 keeps rewriting the Kafka status through
v1beta2. The conversion Job's ClusterRole can patch CRDs and write
ConfigMaps in every namespace, so it goes with its binding and
ServiceAccount once the Job has finished, successfully or not; a
failed Job stays for diagnosis, and a Job still running when the wait
times out keeps them, because the retry waits for that Job.

Two references to v1beta2 outlive the stored objects. Helm builds the
objects of a release before it upgrades or uninstalls it, so every
stored revision of every Kafka application is rewritten to v1. And the
API server refuses server-side apply to an object whose managedFields
still name a version the CRD no longer serves; when helm-controller
recovers by wiping them, a field a later chart drops is never removed.
Those entries move to v1 as well. A Helm action already running stores
the releases it read when it finishes, v1beta2 included, so the version
is not stamped while any Kafka release revision is still pending.

Strimzi 1.x also runs neither ZooKeeper nor Kafka 3.x. A ZooKeeper
cluster, or a shared broker pool still being drained, left by a
platform that moved past v1.7 before its ZooKeeper to KRaft migration
finished cannot be finished afterwards, so the migration stops
instead; Kafka applications on v3.8 or v3.9 move to v4.3. The
migration also stops, before it changes anything, while a Kafka
application topic or a KafkaTopic sets a config Kafka 4 removed or a
segment.bytes below the 1 MiB minimum of Kafka 4: the new kafka chart
refuses to render such a topic, and a Kafka 4 broker whose log holds
the small segment.bytes does not start. segment.bytes is read the way
Kafka reads it: trimmed, with an optional sign, leading zeros decimal.

Strimzi's conversion tool refuses some v1beta2 objects outright, and
convert-resource stops at the first one, with the operators already
stopped. While v1beta2 is served the migration lists every Strimzi
kind first and refuses, naming each object and the reason, before it
stops anything. The list follows the 0.51.0 tool the conversion Job
runs, and a test fails when that image changes.

The conversion deletes the Kafka's KRaft metadata state, and KRaft is
the state Strimzi never leaves. Before it stops the operators, the
migration records an annotation on every Kafka it accepts, so a retry
after the conversion still accepts a NotReady Kafka that passed on
its KRaft state.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-17 12:02:45 +03:00
Aleksei Sviridkin 06ca603f4f feat(platform): expose Keycloak truststores through platform values (#5144)
## What this PR does

Fixes #5140.

#5039 added `truststores` to the Keycloak chart, but that capability was
not reachable through the administrator-owned platform Package: the
system bundle rendered `Package cozystack.keycloak` without a components
override. This PR adds the optional
`authentication.oidc.keycloakTruststores` platform value and forwards it
unchanged to `spec.components.keycloak.values.truststores` on that child
Package.

Administrators can configure it under `spec.components.platform.values`
on `Package cozystack.cozystack-platform`:

```yaml
authentication:
  oidc:
    keycloakTruststores:
      - configMapName: organization-ca
        key: ca.crt
      - secretName: partner-ca
        key: certificates.pem
```

When OIDC is enabled and the list is non-empty, the complete list is
forwarded, including entry order, explicitly selected keys, and entries
that omit `key` so the chart's `ca.crt` default remains effective. When
the value is unset or explicitly empty, the platform continues to use
the default Package helper without introducing `spec.components`. When
OIDC is disabled, the value is ignored because no Keycloak Package is
emitted.

The platform layer only forwards references; it does not create Secrets
or ConfigMaps. Referenced objects and keys must already exist in
`cozy-keycloak`, and the Keycloak chart remains responsible for
validation and mounting. Its existing semantics are unchanged: trust
applies server-wide across all realms, only public certificate material
should be supplied, and changing an object's contents requires a
Keycloak restart or a versioned object reference to reload the
truststore.

### Validation

- `make test` in `packages/core/platform`: all 295 tests pass. The new
suite covers unset, empty and null defaults, complete list forwarding,
false-y malformed values, both `isp-full` and `isp-slim`, and disabled
OIDC; removing the forwarding path or restoring truthiness-based empty
checks makes the regression suite fail.
- `make test` in `packages/system/keycloak`: all 121 tests pass.
- With the system bundle and OIDC enabled, the unset render is
byte-identical to `upstream/main`. The explicitly empty case is
separately verified to omit `spec.components`; malformed false-y values
reach the Keycloak chart and are rejected by its existing `truststores
must be a list` validation.
- `make generate` has no target in `packages/core/platform`; there are
no generated platform schema artifacts to update.
- `make show` could not run because `cozyhr` is unavailable. A direct
`helm template` render of the system bundle was checked in a temporary
chart copy with `templates/repository.yaml` removed because its `lookup`
requires a live source. This verifies rendering and forwarding, not a
live-cluster deployment or an end-to-end Keycloak TLS connection.

### Screenshots

Not applicable; no UI changes.

### Downstream repositories

Changing `packages/core/platform/values.yaml` directly triggers the
website's hand-written platform-package reference. The existing
cozystack/website#765 follow-up covers both the chart-level value and
this platform configuration path, including
`content/en/docs/next/operations/configuration/platform-package.md`; no
other downstream trigger applies to this diff.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/765 (covers the
platform value and platform-package reference as well as the chart-level
guide)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(platform): add `authentication.oidc.keycloakTruststores` so administrators can forward existing Secret or ConfigMap CA references from the platform Package to the bundled Keycloak server for private-CA LDAPS, external identity-provider, and SMTP TLS connections.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added the `authentication.oidc.keycloakTruststores` setting for
configuring additional CA certificates from Secrets or ConfigMaps in
`cozy-keycloak`. Certificate entries default to `ca.crt`, and configured
certificates apply across all Keycloak realms.
* The setting defaults to an empty list. Configured truststores are
passed to Keycloak when OIDC is enabled; when OIDC is disabled, the
setting is ignored and existing Keycloak package settings remain
unchanged.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 00:40:57 +03:00
Aleksei Sviridkin 53078a1d29 feat(observability-proxy): publish the proxy on its own host (#5267)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Adds an optional public endpoint for the observability proxy, so that
clients outside the cluster (kubectl, scripts, third-party tools) can
call it directly with their Kubernetes bearer token. The existing paths
do not deliver the token: kube-apiserver removes the `Authorization`
header after it authenticates a request, so the service proxy of
kube-apiserver cannot carry it, and the console route goes through the
dashboard gatekeeper, which with OIDC enabled answers a client without a
session cookie with 401 and exists only where the dashboard is
published.

The endpoint is off by default. It is published at
`observability-proxy.<publishing.host>` once `observability-proxy` is
added to `publishing.exposedServices`, in the same two modes the
dashboard uses:

- `gateway.enabled: false`: an Ingress on the publishing ingress class,
with a per-host cert-manager certificate, or the operator's wildcard
certificate when `publishing.certificates.wildcardSecretName` is set
(the TLS section stays, so HTTP is still redirected to HTTPS).
- `gateway.enabled: true`: an HTTPRoute on the publishing tenant's
Gateway. The platform now lists `cozy-observability-proxy` in
`gateway.attachedNamespaces` so that the Gateway admits the route. A
cluster that overrides `gateway.attachedNamespaces` replaces the whole
list and has to add the namespace itself.

In Ingress mode the request crosses the publishing tenant's egress
policies, which deny everything except the `allow-to-*` destinations, so
the tenant chart gains `allow-to-observability-proxy`. It matches the
whole `cozy-observability-proxy` namespace, as `allow-to-linstor-gui`
does, because cert-manager runs the HTTP-01 solver pods for the proxy's
certificate there.

The route passes the request through unchanged: no path rewrite and no
authentication at the edge, so `/apis/...` and the bearer token reach
the proxy as sent. The edge limits are set so that they never end a
request before the proxy does:

- The tenant ingress-nginx reads upstream with a 10s timeout by default,
while the proxy allows a non-tail request up to `--request-timeout` (1m)
and keeps a log tail open until it has been silent for
`--tail-idle-timeout`. Before the first byte of a response, the token
and access reviews can add up to another minute. The Ingress sets a 360s
read timeout and keeps response buffering off, so tail lines reach the
client as they arrive.
- The Ingress sets a 1 MiB body limit, the same limit the proxy enforces
on a request body.
- On an HTTPRoute that sets no timeout, Cilium keeps Envoy's 15s route
timeout, which would cut every tail. The route disables it
(`timeouts.request: 0s`), because the proxy limits each request itself.
Envoy's stream idle timeout (Cilium
`envoy.streamIdleTimeoutDurationSeconds`, 300s by default) still applies
and cannot be changed from an HTTPRoute, so the chart now runs the proxy
with `--tail-idle-timeout=4m30s` instead of the 5m default. A tail that
stops sending lines is then ended by the proxy as a complete response,
not reset by Envoy. Before the first byte, slow token and access reviews
together with a log store that never sends headers can still exceed
300s; that case ends in an error either way.

The proxy never reads the client address or `X-Forwarded-*`. All its
limits and its logs use the authenticated user, so nothing changes
behind an ingress and the client cannot spoof anything through headers.

### Security

This adds a new internet-facing entry point when it is enabled. What
protects it:

- Every request to a tenant root needs a bearer token that
kube-apiserver accepts through a TokenReview, and a SubjectAccessReview
for the requested namespace. Paths the proxy does not serve return 404
before any review, and a request without a bearer token returns 401
without a review.
- Per-user limits apply after authentication: request rate, concurrent
requests, concurrent tails per user, and a total cap on open tails.
Every request is bounded by the request timeout, and every tail by the
idle timeout.
- The body is capped at 1 MiB at the edge and in the proxy.
- The new tenant egress policy lets every pod of a tenant open a
connection to the proxy namespace, whether or not the proxy is
published. This gives a tenant workload nothing it cannot already do
through the published host, and the proxy asks for a token in both
cases.

What it does not protect against: an unauthenticated client can make the
proxy send one TokenReview per distinct token it invents. Rejected
tokens are cached by hash for the auth cache TTL, so a repeated token
costs nothing more. This is the same cost a client can already cause by
calling the published Kubernetes API (`api` is in
`publishing.exposedServices` by default). No per-IP rate limit is added
at the edge, because with `publishing.externalIPs` the ingress Service
uses `externalTrafficPolicy: Cluster`, which can hide client addresses
behind node addresses, so a per-IP limit would throttle all clients
together.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#787
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff changes `packages/core/platform/values.yaml`, which reaches
cozystack/website:
`content/en/docs/next/operations/configuration/platform-package.md`
lists the possible values of `publishing.exposedServices` and the
default `gateway.attachedNamespaces` list, and both need the new
entries. The observability-proxy documentation is tracked in
cozystack/website#787, which does not cover the public endpoint yet; the
endpoint and the new list entries will be added there once this PR is
accepted. No other repository in the trigger map is reached by this
diff.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(observability-proxy): the observability proxy can be published at observability-proxy.<host> for clients outside the cluster that use a Kubernetes bearer token. It is off by default; add `observability-proxy` to `publishing.exposedServices` to publish it, through an Ingress or, with `gateway.enabled`, an HTTPRoute. Clusters that override `gateway.attachedNamespaces` must add `cozy-observability-proxy` to it for the Gateway mode. Tenants gain an egress policy to the proxy namespace, which the Ingress mode needs.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
  * Tenant workloads can now connect to the observability proxy.
* The observability proxy can be published through an Ingress or Gateway
route when enabled in the service exposure settings.
* Proxy routing supports long-running requests and streams with adjusted
timeout handling.
* Ingress publishing supports per-host or wildcard certificates and
configurable certificate challenge settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 20:41:32 +03:00
Aleksei Sviridkin d2c5bb05c8 Merge branch 'main' into fix/operator-crds-upgrade
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:20:08 +03:00
Aleksei Sviridkin 2a0dfa6ae5 Merge branch 'main' into feat/platform-keycloak-truststores
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 14:07:55 +03:00
Maxim Belyy 86a484d56b Merge branch 'main' into feat/observability-proxy-ingress
Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-13 16:07:33 +07:00
Aleksei Sviridkin aa97451bc2 fix(platform): leave tenant-root's ingress empty when the root tenant has none (#5226)
## What this PR does

Enabling SeaweedFS on the root tenant without ingress no longer hangs.
`cozystack-basics` wrote `_namespace.ingress: tenant-root` for the root
tenant whatever its `spec.ingress` said, so `seaweedfs-system` depended
on an `ingress-nginx-system` release that nothing creates, and the
platform backup bucket waited with it. Every other tenant already
derives the key in `packages/apps/tenant/templates/namespace.yaml` and
leaves it empty without ingress.

`cozystack-basics` cannot read the root Tenant, so the platform now
carries its ingress state the same way it carries monitoring: the new
platform value `ingress.rootEnabled` reaches the `cozystack-values`
Secret as `_cluster.ingress-enabled`, and `_namespace.ingress` is
written only when it is true. The default is `true`, which keeps the
value every existing cluster renders today, and a Secret written before
the key existed keeps it as well. An operator running the root tenant
without ingress sets `ingress.rootEnabled: false`, kept in sync with
`Tenant/root spec.ingress` by hand, as with `monitoring.rootEnabled`.

With the key empty, `seaweedfs-system` drops the dependency through the
check it already had (`eq $ingress .Release.Namespace`), and nothing
reads live cluster state for it.

An empty `_namespace.ingress` now reaches tenant-root and every tenant
inheriting from it, so the consumers that would break on it are adjusted
in the same PR. The API rejects `ingressClassName: ""` (checked with a
server-side dry run), and two consumers quote the value:

- `kubernetes`: the proxied ingress-nginx addon Ingress is not rendered
without an ingress; its Service still is.
- `monitoring`: Grafana's `spec.ingress` is left out, since
grafana-operator copies it into the Ingress verbatim.
- `seaweedfs`: the S3 Ingress is disabled instead of rendering with no
controller to serve it.

The `KamajiControlPlane` `network.ingress` block is unchanged on
purpose: its `hostname` is what the Kamaji provider uses as the control
plane endpoint, and Kamaji itself skips an empty class name. Harbor,
bucket, bootbox and alerta write the class unquoted, so an empty value
renders as an unset class, which the API accepts.

Tests:
`packages/core/platform/tests/apps_ingress_enabled_wiring_test.yaml`
pins the flag reaching `_cluster` both ways and with `ingress` set to
null;
`packages/system/cozystack-basics/tests/cozystack-values-secret_test.yaml`
pins `_namespace.ingress` for a missing, string, bool and `"false"`
flag, and that the other keys stay when ingress is off;
`packages/extra/seaweedfs/tests/ingress_dependency_test.yaml` pins the
dependency and the S3 Ingress for an own, empty and inherited ingress;
the kubernetes and monitoring suites pin the gated Ingress. Reverting
any of the gates, or flipping the missing-key default, fails the suites.

Not covered here:

- `_namespace.etcd` has the same hardcoded shape (#4783) and is left for
its own change.
- Toggling `Tenant/root spec.ingress` without changing
`ingress.rootEnabled` leaves the value stale, as
`monitoring.rootEnabled` does today.

### Screenshots

Not a UI change.

### Downstream repositories

The diff adds a key to `packages/core/platform/values.yaml`, which the
website documents by hand.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#800 documents `ingress.rootEnabled` in the
platform Package reference
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a root tenant without ingress no longer leaves seaweedfs-system waiting for an ingress-nginx-system release that is never created. Set the new platform value ingress.rootEnabled to false to match Tenant/root spec.ingress: false.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added a root-tenant ingress setting, enabled by default, that can be
turned off.
* **Bug Fixes**
* Kubernetes and Grafana ingress resources are now rendered only when
ingress is configured.
  * SeaweedFS S3 ingress is enabled only when ingress is configured.
* Ingress-dependent components now respect the root-tenant setting,
while other platform services remain available when root ingress is off.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 15:06:12 +03:00
Alexey Artamonov 5b6708c7f9 fix(platform): leave tenant-root's ingress empty when the root has none
cozystack-basics wrote _namespace.ingress: tenant-root whatever the root
Tenant's spec.ingress said, so applications in tenant-root were told an
ingress-nginx runs there when nothing deploys it. SeaweedFS then waited
forever on ingress-nginx-system, and the platform backup bucket with it.
Every other tenant already derives the key and leaves it empty without
ingress.

cozystack-basics cannot read the root Tenant, so the platform carries
its ingress state the way it carries monitoring: ingress.rootEnabled
reaches the Secret as _cluster.ingress-enabled and gates the key. The
default keeps the value existing clusters render, and a Secret written
before the key existed keeps it too.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 18:04:35 +03:00
Aleksei Sviridkin 9978771858 feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does

Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.

Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

What is included:

- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.

#### Known limitations

- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.

### Release note

```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 16:24:25 +03:00
Maxim Belyy 77acd23bdd feat(platform): let the observability proxy attach to the tenant Gateway
With the Gateway API enabled, the publishing Gateway admits routes only
from the publishing tenant and the namespaces in
gateway.attachedNamespaces. The observability proxy publishes its
HTTPRoute from its own namespace, so without an entry the route renders
and is never served.

Clusters that override gateway.attachedNamespaces replace the whole
list and have to add the namespace themselves.

Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-10 18:27:09 +07:00
myasnikovdaniil 91f03dd90e fix(platform): remove the dead registries value and patch-containerd Secret (#5190)
Nothing reads the `patch-containerd` Secret: the kubeadm certs.d mount
it fed is gone, and the v1.6.0 changelog already says the passthrough is
removed. The `registries` value stayed in the Package spec and takes
mirrors that do nothing, which is worse than no knob for someone setting
up an air-gapped cluster. Removed both.

A Package that still sets `registries` keeps installing, the platform
chart has no values schema so the key is ignored. On upgrade the Secret
in `cozy-system` is deleted by the release.

Mirrors for tenant workers are the per-pool `talos.registryMirrors` of
KubernetesNodes.

I checked that nothing is left: `git grep` for `.Values.registries` and
for `patch-containerd` outside the changelogs finds nothing but the
kubernetes README sentence that says the Secret was removed. Full `make
unit-tests` passes.

Fixes https://github.com/cozystack/cozystack/issues/4006


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration Changes**
* The platform chart no longer supports the `registries` configuration
option or generates containerd registry secrets. Mirror endpoint
settings, including credentials and TLS verification options, are no
longer applied through this chart.
* Registry mirrors can be configured per worker pool using
`talos.registryMirrors` on `KubernetesNodes`. Existing packages that set
`registries` continue to install, but the setting is ignored.
* **Documentation**
  * Updated guidance for configuring registry mirrors on worker pools.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 19:36:30 +05:00
Aleksei Sviridkin d61db784a6 fix(mariadb-operator): derive the cluster name from the platform domain (#5123)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Fixes #2348.

`packages/system/mariadb-operator/values.yaml` pins
`mariadb-operator.clusterName: cozy.local`, and the vendored chart turns
it into the operator's `CLUSTER_NAME` and into a SAN of the webhook
certificate. On a platform installed with any other
`networking.clusterDomain`, the operator resolves its own pods as
`<name>.<ns>.svc.cozy.local` and gets NXDOMAIN. Observed on a cluster
with `clusterDomain: lab.local`: a 2-replica MariaDB never gets
replication configured (`status.replication` stays `{}`, the secondary
runs as a standalone semi-sync master), and User, Database and Grant
fail with `lookup mariadb-<name>-primary.<ns>.svc.cozy.local ... no such
host`, while the operator only retries at verbosity 1.

`clusterName` is a value of the vendored subchart, which the `_cluster`
lookup used by package templates cannot reach. The platform now carries
`networking.clusterDomain` under the `mariadb-operator` subchart key in
the shared `cozystack-values` Secret, the same route
`keda.clusterDomain` and `opensearch-operator.manager.dnsBase` already
take. The literal in the package stays as the fallback for a render
without that Secret, so a cluster on `cozy.local` renders exactly as
before.

Tests cover both ends of the path:
`packages/core/platform/tests/apps_mariadb_operator_cluster_name_test.yaml`
pins that the Secret carries the domain under the anchored subchart key
and tracks a non-default value (it fails with the `apps.yaml` change
reverted), and
`packages/system/mariadb-operator/tests/cluster_name_test.yaml` pins
that a domain set on that key reaches the operator's `CLUSTER_NAME` and
the webhook certificate, with `cozy.local` kept as the standalone
default. `make helm-unit-tests` passes locally.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(mariadb-operator): the MariaDB operator now uses the platform's networking.clusterDomain instead of a hard-coded cozy.local, so replication and User/Database/Grant reconciliation work on clusters with a non-default cluster domain.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* MariaDB now uses the configured networking domain as its cluster name
and webhook certificate address. If no custom domain is configured, it
continues to use the default cluster name and address.
* **Tests**
* Added coverage for default and custom domain settings, including their
effect on the MariaDB cluster name and webhook certificate address.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:07:37 +03:00
ZverGuy bbdf05c6c7 feat(platform): deploy the observability-proxy
The proxy only answers for stores that the monitoring stack runs, so it
follows monitoring-agents: on by default in the full variants and
opt-in through enabledPackages on the slim ones, where monitoring is
opt-in too. It is a stateless reader with no CRDs, so networking is its
only dependency.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
Myasnikov Daniil e79946e79c fix(platform): remove the dead registries value and patch-containerd Secret
Nothing reads the Secret: the kubeadm certs.d mount it fed is gone and
the Talos worker config never took it. The value stayed in the Package
spec as a knob that accepts mirrors and does nothing, which is worse
than no knob for an operator in an air-gapped setup. Mirrors for tenant
workers are a per-pool field of the KubernetesNodes chart.

A Package that still sets registries keeps installing: the platform
chart has no values schema, so the key is ignored.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-04 10:08:59 +05:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
Fedor Batonogov 3990268195 feat(platform): expose Keycloak truststores through platform values
Assisted-by: LLM
Signed-off-by: Fedor Batonogov <fedor@batonogov.ru>
2026-10-28 12:48:17 +03:00
Mattia Eleuteri c70f83b375 fix(mariadb-operator): derive the cluster name from the platform domain
The package pinned clusterName to cozy.local, so on a cluster installed
with any other networking.clusterDomain the operator resolved its own
MariaDB pods as <name>.<ns>.svc.cozy.local and got NXDOMAIN. Replication
between replicas was never configured, and User, Database and Grant
resources failed to reconcile, with the operator only retrying at a
verbose log level.

clusterName is a vendored subchart value, out of reach of the _cluster
lookup that package templates use, so the platform now carries it under
the subchart key in the shared values Secret, the same route KEDA and
the OpenSearch operator already take. The literal in the package stays
as the default for a render without that Secret.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-10-26 22:11:09 +01:00
Myasnikov Daniil 59932e5098 fix(platform): report FoundationDB patch drift in migration dry runs
The carry path skipped diagnostics in dry-run mode, so its summary could
report no releases needing attention even when the upgrade would lower a
FoundationDB patch version. Skip only the write and share the warning and
summary bookkeeping with a real run.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-23 19:14:17 +05:00
Andrei Kvapil 4cb505a512 fix(platform): record 7.1 FoundationDB clusters instead of carrying them
The operator no longer carries the 7.1 client, and the chart no longer
offers v7.1, so a 7.1 cluster has no line to be carried to. Migration
59 now records it, together with any release whose version already
names a line that is no longer offered; both have to be upgraded or
repointed before the chart will render them.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil d5ebb62632 fix(platform): keep migration 60's record whole across a retry
A failed patch stops the migration before the stamp, and the retry
finds version already written on the releases it carried, so it never
reached the step that records a patch-level move. The move is now
worked out from the values alone, so a retried run records it too.

An OpenSearch release storing version: null rendered the old chart's
v2 default and is refused by the new one; it now goes on the record
before it turns up as a stuck release.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil 8c2e20539d fix(platform): record every release migration 60 moves
A FoundationDB release carried to its line still moves to the one tag
the chart ships for it, which can be a step below what it runs, and
that showed up only in the Job log. Kubernetes and KubernetesNodes
releases pointing the removed image overrides at a mirror were not
listed at all. Both now go into the needs-attention record next to the
OpenSearch ones.

The migrations image carries no charts, so the tag map is repeated in
the migration, and the bats suite fails when it drifts from the
chart's versions.yaml.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:16 +05:00
Andrei Kvapil 033bec74fb fix(platform): keep one bad release from stopping migration 60
spec.values on a HelmRelease takes any shape, so a release holding a
scalar there, or a cluster.version the apiserver stored as a number,
crashed jq and stopped the carry-over for the whole management cluster:
every healthy FoundationDB release then went into the upgrade without
its version. Those releases are now reported and recorded like any
other value that cannot be carried over.

The same record covers OpenSearch releases that still set
images.opensearch. Nothing can be carried over for them, so what the
operator needs is the list.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:15 +05:00
Andrei Kvapil f3058d2353 fix(platform): guard the FoundationDB version carry-over on all values
Migration 60 tested only cluster.version before adding version, so a
version set on the release between the scan and the patch would have
been overwritten. The patch now tests the whole values object it read.

resourceVersion is not used for this: helm-controller keeps writing the
release status, and a mismatch there would abort the platform upgrade
for no change to the values.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:15 +05:00
Andrei Kvapil 930051280b fix(platform): carry FoundationDB cluster.version over to version
The foundationdb chart no longer reads cluster.version, so a cluster
that set it to 7.1 or 7.4 would move to the v7.3 default on the first
reconcile after the upgrade: a full-bounce upgrade from 7.1, or a
downgrade from 7.4.

Migration 60 sets version to the release line of each cluster.version
it finds. It keeps cluster.version, because the old chart still renders
the release between the migration and the chart upgrade and would fall
back to its own default without it. A value outside the supported lines
is recorded on the cozystack-version ConfigMap and does not block the
upgrade.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:15 +05:00
Andrei Kvapil 873bd6712d fix(platform): stop carrying images.kubectl into adopted node pools
Migration 56 copied the parent cluster's kubectl image override into
each kubernetes-nodes release it creates. Neither chart reads an images
map any more, so the copy only planted an inert key in the pool values.

Migration 56 has not shipped in a stable release, so no cluster has run
the old mapping outside a pre-release.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:14 +05:00
Myasnikov Daniil b83cc087cc docs: move the repo-only guides onto the website
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.

docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 17:50:46 +05:00
Yan Bondarenko 8e96a3e9b7 fix(foundationdb): apply foundationdb-operator CRD updates on upgrade
The foundationdb-operator PackageSource sets no upgradeCRDs policy, so
a cluster installed before v1.6.0 runs the v2.30.0 operator on the
v2.13.0 CRDs it first installed. The apiserver prunes every field added
since, status.backupDetails.restorable on FoundationDBBackup among
them, so a backup never shows as restorable however long it has run.

The v2.30.0 CRDs only add fields. They keep the single v1beta2 version
and drop or retype nothing, and the one new constraint, a pattern on
perpetual_storage_wiggle_locality, sits on a field the platform does
not set, so CreateReplace is safe.

Nothing in the foundationdb-operator package changes here, so a cluster
picks the new CRDs up at the next upgrade of that release, which comes
with the next change to the package.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-18 23:11:02 +03:00
Yan Bondarenko b74029cb39 fix(keycloak): apply keycloak-operator CRD updates on upgrade
keycloak-operator ships its CRDs in the chart's crds/ directory and its
PackageSource sets no upgradeCRDs policy, so helm-controller never
replaces them. A cluster installed before v1.2.0 therefore runs the
v1.32.0 operator against the v1.25.0 CRDs, whose ClusterKeycloakRealm
schema has no spec.sessions, spec.login or spec.organizationsEnabled.
The apiserver drops those fields from the realm keycloak-configure
renders, so the operator never applies the session limits the platform
sets. On a v1.6.4 cluster in that state Keycloak also rejected the
dashboard client with HTTP 400, and replacing the CRDs from the chart
by hand, with an operator restart, cleared it.

The v1.32.0 CRDs are not purely additive. They drop the realm,
targetRealm and keycloakOwner fields that v1.25.0 marked deprecated and
require realmRef and keycloakRef instead. Every Keycloak object the
platform renders already uses the refs, and each CRD keeps its single
version, so replacing them loses nothing the platform relies on.

helm-controller applies the policy only when it upgrades the release,
and it runs an upgrade only when the release artifact or its values
change, which a PackageSource edit does not do. The package test added
here also changes the keycloak-operator artifact, so the replace runs
on the upgrade that ships this change even where nothing else in the
package changes. The test itself pins the realm fields
keycloak-configure sets against the CRD the chart ships.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-18 23:11:02 +03:00
Myasnikov Daniil f3486f4e3a docs: move the operator guides out of docs/ to where readers find them
Five guides in docs/ were written next to the feature code that needed
them and left in the source tree, where no operator looks. They are on
the website, so a second copy here only drifts: edits kept landing in
docs/ after the website copy was made.

docs/vm-external-vlan.md is split rather than moved. Its section on
staging the reference CNI plugins is a contract that
hack/cni-plugins-staging-contract.bats asserts against the multus
Dockerfile, the daemonset manifest and the platform values, so that half
is package documentation and goes to packages/system/multus/README.md,
which the suite now reads. The operator half goes to the website.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-14 16:24:16 +05:00
Aleksei Sviridkin 1f74ad4ee7 fix(multus): drop the CPU limit and make resources configurable (#4830)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The multus daemon was too small to recover a node after a CNI outage,
and its limits could not be raised from values. The 100m CPU limit and
the 1Gi memory limit of `kube-multus` were hardcoded in the patched
upstream template.

On a node with about 600 dead sandboxes and 220 pods waiting, multus was
throttled in every CFS period. Its calls took longer than kubelet's
two-minute CRI deadline, so every call was retried and the queue only
grew. With the CPU limit lifted it OOM-looped at 1Gi. With 2 CPU and 4Gi
it cleared the backlog in about three minutes.

Now the resources come from a new `resources` value. The default keeps
the same requests (100m CPU, 100Mi memory), has no CPU limit, and raises
the memory limit to 4Gi. The memory limit stays on purpose. Without it
the pod cgroup has no `memory.max`, and multus becomes a candidate for
the Talos userspace OOM handler again.

The chart now refuses to render when an override removes the memory
limit or the memory request. Without a request Kubernetes uses the limit
as the request, so a node with less allocatable memory than the limit
would be left without multus.

The change is in `patches/customize-deployment.patch`, so `make update`
keeps it. I applied the patch to upstream v4.3.0 and got the committed
template, except the image references. The platform reverts a direct
edit of the child `cozystack.multus` Package, so a running cluster sets
the value through the new `networking.multus.resources` key of the
platform Package. The key is passed on only when set, and a value that
is not a map stops the render. The system-defaults LimitRange in the
namespace sets memory only, so it does not bring a CPU limit back.

Closes #4829

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/747
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff adds a platform value, so the hand-written key table on the
website needs it. The follow-up PR there is a draft until this one
merges.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(multus): the kube-multus container no longer has a CPU limit, and its memory limit is raised from 1Gi to 4Gi, so a node with hundreds of pods can recover after a CNI outage. Its resources can be overridden through the new `networking.multus.resources` platform value. An override that removes the memory limit or the memory request now fails the render.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional CNI plugin staging during Multus startup, configurable
through chart values. Staging skips when the plugin directory is
unavailable and reports individual copy failures while continuing.
* Made Multus resource settings configurable, with defaults of 100m CPU
and 100Mi memory requests and a 4Gi memory limit. Rendering requires
memory requests and limits.
* Updated the Multus deployment to use the `cozy-multus` namespace and
DaemonSet name, and Cilium as the master CNI configuration.
* **Bug Fixes**
  * Added a 128Mi memory limit to the CNI installation step.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 17:24:46 +03:00
Aleksei Sviridkin 938abdc4d2 fix(platform)!: refuse a networking variant change on a live cluster (#4790)
## What this PR does

Changing the platform variant on a running cluster can delete its CNI,
and nothing stopped that before. A new platform variant (`spec.variant`
of the `cozystack.cozystack-platform` Package) can give the
`cozystack.networking` Package a different variant, and the operator
then deletes every HelmRelease of that Package the new variant does not
install. Moving from `isp-full` to `isp-hosted` deletes Cilium and
Kube-OVN, so every pod loses networking. Moving from `isp-full` to
`isp-slim` deletes Kube-OVN.

With this PR the platform render looks up the live
`cozystack.networking` Package. If its variant differs from the one
about to be rendered, the render fails, and the error names the value
that lets it through: `networking.acknowledgeVariantChange: "<new
variant>"`. I made the acknowledgement a variant name instead of a
boolean like `publishing.proxyProtocolAcknowledgeUnclean`. A leftover
`true` would let every later switch through unchecked, while a variant
name covers only the switch it names.

The guard does nothing when there is no live Package to compare against:
`helm template`, a client-side dry-run, or a first install. It is also
skipped when `cozystack.networking` is listed in
`bundles.disabledPackages`, because the platform does not render the
Package then. It lives in the shared Package helper and not in
`bundles/system.yaml`, so it covers both places that emit the networking
Package, including the `noop` call in the `isp-hosted` branch. The other
Packages whose variant follows the bundle, `cozystack-scheduler` and
`tenant-application`, keep the same release names in every variant, so
changing their variant deletes nothing and they need no guard.

The new helm-unittest cases cover the refused switches, an unchanged
variant, a matching and a mismatching acknowledgement, a bare boolean,
`disabledPackages`, an offline render, and a live cluster with no
networking Package yet.

A cluster where someone set the Helm-owned networking Package to
`cilium-kilo` by hand now fails its platform upgrade, and the error
points it at `bundles.disabledPackages`. Acknowledging does not help
there, because the render writes the bundle's variant over
`cilium-kilo`. helm-controller v1.5.0 installs a release with
server-side apply by default, which takes over every rendered field. On
a release it installed, the upgrade replaces the variant and the
operator deletes Kilo. A release first installed with Helm 3 keeps the
apply method recorded in its history, so there the outcome depends on
that history.

Known gap, left for a follow-up: the guard fires on any variant change,
including Talos to generic (`kubeovn-cilium` to
`kubeovn-cilium-generic`). Those pairs share release names, so nothing
is deleted and only values change. Refusing is safe, and the error does
not claim a deletion for such a switch.

Closes #4596

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

This PR adds a key to `packages/core/platform/values.yaml`, and the
website keeps a hand-written table of those keys in
`operations/configuration/platform-package.md`.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/746
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): the platform render now refuses to change the variant of a live cozystack.networking Package, because the operator would then delete the CNI HelmReleases the new variant does not install (isp-full to isp-hosted removes Cilium and Kube-OVN). To go ahead with such a change, set networking.acknowledgeVariantChange to the new networking variant named in the error. A cluster whose cozystack.networking variant was set by hand (for example cilium-kilo) needs cozystack.networking in bundles.disabledPackages before the upgrade, otherwise the platform render refuses.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Changing the networking variant on a running cluster now requires
explicit confirmation naming the requested variant. A change without a
matching confirmation is blocked.
* The confirmation applies only to the specified switch and must be
updated for a later variant change. If the running package has no
recorded variant, it is treated as using the default variant.
* New installations without an existing networking package are
unaffected, and networking can still be disabled through its package
setting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 17:24:09 +03:00
Andrey Kolkov 647407545d fix(backupstrategy-controller): run backup clients on a package-owned image
The Altinity and RabbitMQ strategy Pods ran platform-migrations, a 349 MB
image carrying kubectl, helm and etcd-migrate whose ENTRYPOINT is the
migration runner, and whose pins were stamped from packages/core/platform
so every migration change moved the backup clients too.

Build images/backup-http-client (alpine + curl + jq + ca-certificates,
non-root) from this package's Makefile and stamp both client keys from
it. The cross-package stamp in packages/core/platform/Makefile and the
platform-migrations-specific check in hack/image-pin-consistency.bats go
away; the general one-digest-per-repository check still covers the two
keys sharing one image.

Fixes #3420

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-12 09:54:52 +04:00
Aleksei Sviridkin ddd00f8639 feat(platform): pass multus resources from networking.multus.resources
The multus chart takes its resources from values, but the platform
renders the cozystack.multus Package with stageCniPlugins alone, and an
edit to that child Package is reverted by the platform reconcile. A
running cluster therefore had no way to raise the multus limits.

networking.multus.resources is passed to the Package only when set, so
an unset key leaves the chart default in charge. A value that is not a
map stops the render here; the multus chart keeps refusing a render
without a memory limit or request.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 20:59:26 +03:00
Aleksei Sviridkin 1245b32a32 fix(platform)!: refuse a networking variant change on a live cluster
Changing the variant of the cozystack.cozystack-platform Package on a
running cluster changes bundles.system.variant and can re-render the
cozystack.networking Package with another variant. The operator then
deletes every HelmRelease of that Package the new variant does not
install, so isp-full -> isp-hosted removes Cilium and Kube-OVN and the
cluster loses its CNI; isp-full -> isp-slim removes Kube-OVN.

The platform render now looks up the live Package and fails when its
variant differs from the one about to be rendered, unless
networking.acknowledgeVariantChange names the new variant. The
acknowledgement is a variant name rather than a boolean so that a value
left behind after one switch does not cover a later one. The check is
inert where lookup returns nothing (helm template, client-side dry-run,
first install) and skipped when cozystack.networking is in
bundles.disabledPackages, since the platform then does not render it.

A live variant set by hand, such as cilium-kilo, is pointed at
bundles.disabledPackages rather than at the acknowledgement: the
acknowledged render would write the bundle's variant over it.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 20:57:17 +03:00
Aleksei Sviridkin a2e10312de fix(platform): stop treating a failed ConfigMap read as a fresh cluster (#4792)
## What this PR does

A single failed API read at the start of the platform migration Job
could skip every pending migration and still end green.
`run-migrations.sh` checked whether the `cozystack-version` ConfigMap
exists with a silenced `kubectl get` and looked only at the exit code. A
timeout, a dropped connection or an apiserver restart during the upgrade
all counted as "no ConfigMap", so the script stamped the target version
and exited 0. The hook Job is rendered only when that ConfigMap already
exists, so in practice an error was the only way into this branch.

Now the script creates the ConfigMap only when it is really missing. The
check uses `--ignore-not-found --output name`: kubectl drops NotFound
and nothing else, so empty output means a genuine NotFound. Any other
error keeps a non-zero exit, `set -e` aborts the script, and the Job
retries the pod. The rest of the script already works this way: the read
of the stamped version aborts on error, and the stamp runs under
`pipefail`.

The tests cover both branches. A new case in `hack/run-migrations.bats`
makes the existence check fail with a connection error and expects a
non-zero exit, no migration run and the stamp left where it was. It
fails against the old check. The fake kubectl returns empty output for a
missing ConfigMap only when `--ignore-not-found` is passed, as real
kubectl does, so dropping the flag breaks the existing "no ConfigMap"
case.

No migration is added, so `targetVersion` stays. The script ships in the
migrations image, and the release build restamps that image's digest as
usual.

Several older migrations have the same problem in a smaller form. They
check for an object with `kubectl get ... >/dev/null 2>&1` and skip
their own work when the check fails, for example migrations 31, 32, 36,
48, 50 and 51. A transient error there skips one migration's work, not
all of them. Editing migrations that already shipped has its own risk,
so I'd track that separately.

Closes #4579

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a transient API error while the platform migration Job checks for the cozystack-version ConfigMap now fails the pod so the Job retries, instead of stamping the target version and silently skipping every pending migration.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Migration checks now distinguish a missing version record from an
error while checking for it. When the check fails, the pod stops instead
of treating the record as missing and proceeding with migrations.
* Added coverage for TLS-handshake timeouts during the check, confirming
that migrations do not run and the stamped version remains unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 20:37:53 +03:00
Aleksei Sviridkin 87a64071a7 fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out (#4778)
## What this PR does

fixes #3740.

on the kube-ovn variants cilium allocates no pod ips, but the agent
still takes its router ip (`cilium_host`, `CiliumInternalIP`) and the
gateway api ingress ip from its own allocation cidr. with `ipam.mode:
kubernetes` that's `Node.spec.podCIDR`, a /24 out of the same /16
kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an
address cilium already holds. we hit it on three clusters, details in
https://github.com/cozystack/cozystack/issues/3740#issuecomment-6022394243.

**what changes**

- the kube-ovn variants run cilium with `ipam.mode: cluster-pool`, a /29
per node out of the last /21 of `networking.podCIDR` (`10.244.248.0/21`
for the default /16). the platform derives the block from
`networking.podCIDR` and passes the same range to kube-ovn as
`--default-exclude-ips`, so a fresh install never hands it to pods.
- the block sits inside the pod cidr on purpose. envoy reaches gateway
api backends from the node's ingress ip, and kube-ovn masquerades host
originated traffic only when the source is in `ovn40subnets`. with the
`100.65.0.0/16` pool from the issue a nodeport through the gateway
answered 200 only on the backend's node and 503 `upstream connect error
... connection timeout` on the other two. with the block inside the pod
cidr it's 200 from all three. worth a look on the ci side too. the ci
sandbox sets talos `podSubnets` to `100.65.0.0/16`, so cilium's ingress
ips there already come from outside `ovn40subnets`, and the gateway e2e
checks Accepted, Programmed and the lb service but sends no request
through a listener, so it wouldn't notice. with this pr the sandbox
takes its block from the pod cidr like everyone else.
- the end of the /16 is what kube-ovn reaches last. it hands out the
lowest free address and reuses released ones only when the free range
runs dry (`pkg/ipam/subnet.go` and `pkg/ipam/ip_range_list.go` at
v1.15.26). a /21 cut into /29s covers 256 nodes.
- the slim variants and guest clusters stay on `ipam.mode: kubernetes`.
cilium is the only cni there and `Node.spec.podCIDR` is the pod range.
- `networking.podCIDR` now needs /20 or wider on the kube-ovn variants,
otherwise the render fails and says why. an install that rendered before
can stop rendering at upgrade, hence the `!`. a /21 cut into /29s also
caps these variants at 256 nodes, a fresh install included.

**existing clusters**

a post-upgrade hook in the cilium chart moves them
(`templates/ipam-migration.yaml`, `files/ipam-migration.sh`). before it
touches anything it checks that the block has a /29 for every CiliumNode
and that it lies inside kube-ovn's default subnet, and stops with both
cidrs printed otherwise. a node cleared past the pool's capacity would
lose pod networking on its next agent restart, and a block outside the
subnet puts the agents outside `ovn40subnets` again. it has to be
post-upgrade. platform migrations run as the platform's pre-upgrade
hook, while every agent still runs `ipam.mode: kubernetes` and writes
`Node.spec.podCIDR` back into a cleared CiliumNode
(`pkg/nodediscovery/nodediscovery.go` at v1.19.5), so the move can only
happen after the cilium release switched the mode. the job runs on the
host network against the agent's own api endpoint (kubeprism on talos),
because it restarts the agents.

1. adds the block and every node's current ingress ip to `excludeIps` of
the default subnet, in one json patch with a `test` op, and reads the
subnet again on conflict, since kube-ovn rewrites that list into its own
order right after a patch. the old ingress ip goes in because cilium
1.19.5 keeps a former ingress ip in the agent's ipcache as
`reserved:ingress` after the move. we saw 2 to 3 such entries per node
hours later. old router ips leave the ipcache and go back to kube-ovn.
2. stops right there if an address in the block is still allocated,
names it, and moves no agent. it first checks the subnet status it
patched against, then waits until `status.v4availableIPrange` stops
offering the block, which is kube-ovn having applied the exclusion, and
then reads the `ips.kubeovn.io` objects of the default subnet. the
status alone can't answer it, kube-ovn leaves excluded addresses out of
`v4usingIPrange` (`GetSubnetIPRangeString` in `pkg/ipam/ipam.go` at
v1.15.26), so a pod still in the block would be invisible on a retry
where the exclusion is already in place. recreate those pods and the
next upgrade retry goes through.
3. one node at a time, clears the inherited /24 from
`spec.ipam.podCIDRs` (again with a `test` op, so a concurrent writer
fails the patch instead of losing an update), waits for the operator to
allocate a /29 from the block and restarts that node's agent. a node
that isn't ready is skipped and its agent picks up the block when it
starts. nodes already moved are left alone, so a retry after a timeout
just continues. the hook runs inside the cilium upgrade, so where
restarting every agent takes longer than the upgrade timeout, 10m by
default, the release first reports a failed upgrade and the move
finishes on the retry.

running pods keep their addresses. every agent restarts twice on
upgrade, once when the config change rolls the daemonset and once from
the hook, and while it restarts host to pod traffic on that node blips
for a few seconds because the router ip moves. a pod that sits on a
router ip today is reachable again after the move, the node no longer
holds that address. a pod on a node's current ingress ip keeps it and
that node keeps seeing it as `reserved:ingress`, so recreate it. the
hook logs those pods by name.

### Validation

- `helm unittest` passes cilium 23/23, platform 246/246 and kube-ovn
22/22. the new suites pin the rendered `cilium-config` keys per variant,
the hook shape, env and rbac, and the platform wiring for the talos and
generic branches, a custom pod cidr, host bits in the network address,
the refusals for a too small or non ipv4 cidr, and the slim variant.
- on a trial merge with main at cbc9d0c4e, where #3849 runs
`hack/*.bats` under real bats 1.14 with `set -u`, the 27 bats tests pass
after the second commit loads `test_helper`, the strict setup audit is
clean, and helm unittest passes cilium 23/23, platform 263/263 and
kube-ovn 22/22.
- `hack/cilium-ipam-migration.bats` passes 27/27 and runs the real
script against a fake kubectl that models the operator, the agent
daemonset and kube-ovn publishing the subnet status. thirteen mutations
turn it red each (exclusion after the move, no ingress exclusion, no
retry, no status check, an ingress ip inside the block listed on its
own, no `test` op, no wait for kube-ovn, no ip object check, no subnet
filter on the ip objects, no capacity check, a block outside the subnet
let through, overlap turned into containment, no log of ingress ip
holders). the containment one is pinned by a usage range that crosses
the start of the block, `10.244.247.250-10.244.248.3`.
- live on a 3 node talos test cluster with cozystack v1.6.4, cilium
1.19.5 and kube-ovn v1.15.26
- moved from `kubernetes` to the block, exclusion first and then the
node moves this script does. 3 nodes in about 40 seconds, every agent
took its router and ingress ip from its /29, no pod changed its address.
- gateway api through a nodeport 9/9 answered 200 and through the
loadbalancer ip 6/6, against 3/9 with `100.65.0.0/16`.
- the final hook job on the moved cluster is a no-op in 2 seconds,
including the ip object read with the job's own rbac. the same ip object
pipeline over that cluster's real objects finds 77 in a populated /24
and none in the block.
- the exact `test` plus `replace` patch on `ovn-default` goes through,
kube-ovn re-sorts the list right after, and the same patch against the
old list is rejected. that's the conflict path the retry covers.
- shellcheck is clean and `hack/check-commit-trailers.sh` passes.
- not run here. the upgrade move from an older release in ci, e2e only
covers the fresh install.

### Screenshots

not a ui change.

### Downstream repositories

walked the trigger map against the diff.
`packages/core/platform/values.yaml` only gains a comment, but
`networking.podCIDR` gets a new constraint on the kube-ovn variants,
which the hand written key table on the website should say. the ansible
role sets `networking.podCIDR` to `10.42.0.0/16`, which yields
`10.42.248.0/21` and needs nothing there. the talm preset keeps
`podSubnets: 10.244.0.0/16`, which no longer matters on these variants
because cilium stops reading `Node.spec.podCIDR` there. the terraform
provider doesn't model platform values.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/pull/739
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cilium)!: on the kube-ovn variants cilium now takes its router and gateway api ingress ips from the last /21 of networking.podCIDR, which kube-ovn no longer hands out, so pods stop landing on addresses cilium holds. existing clusters move on upgrade through a post-upgrade hook, and each cilium agent restarts twice, once for the config change and once in the hook. networking.podCIDR needs /20 or wider on these variants, and a /21 of /29s caps them at 256 nodes.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Kube-OVN-enabled variants now reserve the final `/21` of the pod CIDR
for Cilium router and Gateway API Ingress addresses, excluding it from
Kube-OVN pod allocation. Pod CIDRs must be IPv4 with a prefix of `/20`
or shorter.
* Added a post-upgrade migration that moves existing Cilium IP
allocations into the cluster pool and updates affected node networking.
* **Documentation**
* Documented the pod CIDR requirements and node capacity limit for
Kube-OVN variants.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-08 21:41:23 +03:00
Aleksei Sviridkin 0a2bb5ef98 fix(cert-manager): enable the Gateway API solver in the host cert-manager (#4039)
## What this PR does

With the platform Gateway enabled, the ClusterIssuers in
`cert-manager-issuers` and the per-tenant Issuers rendered for
`TenantGateway` carry an `http01.gatewayHTTPRoute` solver, but the
controller refuses it: the vendored chart never turns Gateway API
support on, so every HTTP-01 challenge behind the Gateway fails with
`gateway api is not enabled` and no certificate is issued.

This passes a `ControllerConfiguration` with `enableGatewayAPI: true` to
the host cert-manager through the component values of its platform
Package. The host ships the Gateway API CRDs on every variant and orders
cert-manager after them, so the flag is safe there.

It is deliberately not a default of the `system/cert-manager` chart. The
same chart is installed into tenant Kubernetes clusters, where the CRDs
are an opt-in addon, and cert-manager v1.20 exits at startup when
Gateway API support is on and the CRDs are missing. The tenant
HelmRelease in `packages/apps/kubernetes` already passes the same
`ControllerConfiguration` when its `gatewayAPI` addon is enabled, so
both clusters now opt in from the layer that knows whether the CRDs are
there, and the chart itself stays neutral.

The new tests pin both halves: the host Package carries the flag and the
preconditions that make it safe, and the chart wires `config` through to
the controller while shipping none by default.

Closes #3725

### Screenshots

Not applicable, no UI changes.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(cert-manager): enable the Gateway API solver in the host cert-manager, so ACME HTTP-01 challenges succeed when the platform Gateway is enabled
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Supported system bundles can now use cert-manager’s Gateway API
HTTPRoute solver to issue certificates.
- Hosted deployments include the required Gateway API custom resource
definitions, installed before cert-manager.
- Gateway API solver support remains opt-in in environments that do not
provide the required definitions, avoiding unintended changes to
certificate management there.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-07 17:47:41 +03:00
Aleksei Sviridkin 76b9e55102 fix(cert-manager): enable the Gateway API solver in the controller
With the platform Gateway enabled, the ClusterIssuers in
cert-manager-issuers and the per-tenant Issuers rendered for
TenantGateway carry an http01.gatewayHTTPRoute solver, but the
controller only honours that solver when Gateway API support is
switched on, and the vendored chart leaves it off. Every HTTP-01
challenge behind the Gateway then fails with "gateway api is not
enabled" and no ACME certificate is issued.

Pass a ControllerConfiguration with enableGatewayAPI to the host
cert-manager through the component values of its platform Package.
The host always ships the Gateway API CRDs and orders cert-manager
after them, so the flag is safe on every variant.

It is not a default of the system/cert-manager chart: the same chart
is installed into tenant clusters, where the CRDs are an opt-in addon
and a controller started with Gateway API support and no CRDs exits
at startup. The tenant side keeps opting in on its own.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
2026-10-07 14:14:35 +03:00
Aleksei Sviridkin cbbb69afc3 fix(platform): stop treating a failed ConfigMap read as a fresh cluster
run-migrations.sh decided whether cozystack-version exists by the exit
status of a silenced kubectl get, so a timeout or a dropped connection
counted as "no ConfigMap": the runner stamped the target version and
exited 0, and every pending migration was skipped for good. The hook Job
only renders when the ConfigMap already exists, so in practice an error
was the only way to reach that branch.

--ignore-not-found suppresses NotFound alone; any other error keeps its
non-zero status and now fails the pod, so the Job retries the read.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-07 12:06:18 +03:00
Myasnikov Daniil d61fed46cc feat(local-storage): add opt-in OpenEBS LocalPV hostpath package
LINSTOR needs the DRBD and ZFS kernel modules, so a cluster without
them has no dynamic PV provisioning. OpenEBS LocalPV hostpath needs no
kernel modules and also serves deliberately node-local workloads.

The package is opt-in so it cannot change an existing cluster. Its
class is named cozystack-local-hostpath so it cannot collide with an
existing class name, and it is not the default so PVCs that omit
storageClassName keep their current binding. The vendored RBAC is
patched down to what volume reconciliation calls; helper Pods, Leases
and analytics state stay in the install namespace.

Images stay on docker.io, pinned by index digest.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 11:30:45 +03:00
Myasnikov Daniil 974aaa0bac test(hack): regenerate the selective install fixture
Two optional Cluster API providers, Proxmox and in-cluster IPAM, were
added to the platform sources after the fixture was generated. The
drift test compares the fixture with the current disable list, so it
failed until they were added.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:53 +03:00
Myasnikov Daniil f46a9797b8 fix(hack): keep platform services in selective installs
Dropping every package that no suite owns also drops services the rest
of the platform relies on: multus for secondary networks, the cozystack
and LINSTOR schedulers, flux-plunger, cozy-proxy and ouroboros, whose
hairpin NAT the Kubernetes suites assert. Nothing waits for them, so no
PackageSource edge reaches them, and adding one would also change
reverse test selection.

The remaining packages stay uninstalled on purpose, also when test
selection answers with every suite: nothing in the run looks at them,
so installing them only adds a readiness wait.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:52 +03:00
Myasnikov Daniil 59555c895e fix(hack): retain controllers and APIs for selective installs
Tenant HelmReleases need a shard reconciler, and the root SeaweedFS
release creates COSI resources even when no backup suite is selected.
Without those controllers, a fresh reduced install cannot become ready.

The Kubernetes suites create worker pools through KubernetesNodes, and
MongoDB renders an operator-owned CR. Selecting only the cluster API or
application chart drops resources those suites require.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:52 +03:00
Myasnikov Daniil 361893e6f4 fix(hack): retain backup APIs in selective installs
Backup roundtrip suites create Bucket and backup resources that the
application graph does not require. A graph-only closure therefore
omits the APIs those suites need on a fresh install.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:51 +03:00
Myasnikov Daniil 610578be16 test(hack): verify selective install contract
A missing suite owner or a discarded discovery failure can leave a
reduced install without the APIs its tests exercise. Graph-only checks
cannot establish that the chart preserves the requested package set.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-07 06:01:50 +03:00
Yan Bondarenko dc362fa591 fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out
On the kube-ovn variants Cilium allocates no pod addresses, but the agent
still takes its router IP and the Gateway API Ingress IP from its
allocation CIDR. In ipam.mode kubernetes that is Node.spec.podCIDR, which
the documented installs slice from the same range kube-ovn allocates pods
from, so kube-ovn sooner or later hands a pod an address Cilium already
holds.

Cilium now runs ipam.mode cluster-pool on the kube-ovn variants, with a
/29 per node from the last /21 of networking.podCIDR, and kube-ovn
excludes that /21 through --default-exclude-ips. The block stays inside
the pod CIDR on purpose. Envoy reaches Gateway API backends from the
node's Ingress IP, and kube-ovn masquerades host-originated traffic only
when the source lies in one of its subnets, so a block outside them such
as 100.65.0.0/16 breaks Gateway API traffic that crosses nodes.

A post-upgrade hook moves existing clusters. It first checks that the
block has a /29 for every node and lies inside kube-ovn's default
subnet, and stops without changing anything otherwise. It adds the block
and every node's current Ingress IP to excludeIps of that subnet,
because the agent keeps a former Ingress IP in its ipcache as
reserved:ingress. It then waits until kube-ovn stops offering the block
and stops if an address in it is still allocated. kube-ovn leaves
excluded addresses out of the subnet status, so that check reads its IP
objects. Then, one node at a time, it clears the inherited /24 from the
CiliumNode, waits for the operator to allocate from the block and
restarts the agent.

networking.podCIDR now needs a prefix of /20 or shorter on the kube-ovn
variants, and the render fails otherwise. A /21 cut into /29s also caps
these variants at 256 nodes.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-06 22:11:02 +03:00
Aleksei Sviridkin bcda5c893a feat(kubevirt)!: drop HotplugVolumes for declarative volume hotplug (#4453)
## What this PR does

This turns off the deprecated KubeVirt `HotplugVolumes` gate and moves
the platform to declarative volume hotplug. It changes how every volume
hotplug on the platform works, so the title carries `!`. Closes #4032.

This PR is stacked on #4451 and has to merge after it. #4451 adds the
kubevirt-csi flag this PR flips.

The replacement gate is already on in KubeVirt 1.9, and the old gate is
the only thing keeping it off. `DeclarativeHotplugVolumes` is Beta at
v1.9.0, and Beta gates are on by default. virt-controller skips
`handleDeclarativeVolumeHotplug` while `HotplugVolumes` is on, so
dropping the old gate is what switches the new one on.

The kubevirt-csi fallback goes off in the same change. With the old gate
off, virt-api rejects the VMI-level `removevolume` on a VM-owned VMI, so
the chart now passes `--enable-vmi-hotplug-fallback=false`. The comments
next to the gate and the flag point at each other.

This needs the rebuilt kubevirt-csi image. A driver built before #4451
does not know the flag and exits on it at startup. The image is stamped
at release preparation and ships in the same release as this change, so
only a `make apply` from main before then would crash-loop the CSI
controller.

What changes at runtime on an upgrade:

- The VM-level `addvolume` and `removevolume` calls now edit `VM.spec`
directly. virt-controller then makes the VMI hotplug volumes match it,
instead of going through `VM.status.volumeRequests`.
- Volumes hot-plugged through the VM subresource are in `VM.spec`, so
they stay attached.
- During a tenant teardown, when the VM goes before its VMI, the driver
no longer calls the VMI subresource. It waits for the VMI status to drop
the volume, and the garbage collector deletes the VMI with its VM.
Upstream waits up to two minutes. If the VMI is still there after that,
the unpublish fails and is retried.

A new platform migration keeps old VMI-only volumes attached. On its
first pass virt-controller removes every hotpluggable VMI volume that is
missing from `VM.spec`. Until July 2025 kubevirt-csi attached volumes at
the VMI level (the Role then granted
`virtualmachineinstances/addvolume`), and live migration carries such a
VMI across KubeVirt updates. Without help, a tenant node not recreated
since then would lose a mounted disk on upgrade.

Migration 59 copies each such volume and its disk from the VMI into the
VM template before anything else changes. Adding a hotplug volume to the
VM spec is a valid live update in KubeVirt, so no VM needs a restart. It
runs as the platform chart's pre-upgrade hook, before the upgrade
re-points `cozystack-packages`. The KubeVirt CR and the tenant
kubernetes charts both render from that source, so the gate and the flag
cannot change before the migration has finished. It only touches VMs
that need it, logs each one, and a second run changes nothing.

The migration also makes ephemeral hotplugs permanent. A volume a user
added at the VMI level, for example with `virtctl addvolume` without
`--persist` from an older virtctl, has the same shape as kubevirt-csi's
old attachments, because virt-api builds both the same way. There is
nothing reliable to tell them apart by, so both are copied. Leaving the
user's volume out would not keep it ephemeral: the first reconcile would
unplug it from the running VM. After the upgrade it is part of `VM.spec`
and attaches again on a restart, and `virtctl removevolume` drops it.

Ephemeral hotplug stops working on VMs. With the old gate off, virt-api
refuses VMI-level `addvolume` and `removevolume` on a VMI that a VM owns
(`vmiVolumePatch` in v1.9.0). A volume on such a VMI is added and
removed through the VM from then on. A VMI without a VM is not affected.

Only running VMIs are copied. virt-controller reconciles no other VMI,
and a stopped VM builds its next VMI from `VM.spec`, so copying from a
stopped one would bring back volumes the restart was going to drop.

Migrations run only on the isp-full, isp-full-generic and isp-hosted
variants. KubeVirt comes from the platform only on isp-full and
isp-full-generic, so those are covered. On the default variant, where
packages are installed by hand, nothing runs the migration.

This optional check lists the VMIs and volumes the migration would copy.
It uses the same rule. On the default variant, run it before upgrading
and recreate each tenant node it lists:

```bash
kubectl get vm,vmi --all-namespaces --output json | jq --raw-output '
  ([.items[] | select(.kind == "VirtualMachine")
    | {key: "\(.metadata.namespace)/\(.metadata.name)", value: [.spec.template.spec.volumes[]?.name]}]
    | from_entries) as $vm
  | .items[] | select(.kind == "VirtualMachineInstance")
  | ([.metadata.ownerReferences[]? | select(.controller == true and .kind == "VirtualMachine" and (.apiVersion | startswith("kubevirt.io/"))) | .name][0]) as $owner
  | "\(.metadata.namespace)/\($owner)" as $key
  | select(.status.phase == "Running" and $owner != null and $vm[$key] != null)
  | .spec.volumes[]? | select(.dataVolume.hotpluggable == true or .persistentVolumeClaim.hotpluggable == true)
  | .name as $n | select($vm[$key] | index($n) | not)
  | "\($key) \($n)"'
```

Empty output means nothing is affected. I checked the query against the
migration's test fixtures only. Neither the query nor the migration has
run on a live cluster.

#4287 also adds a migration 59. If it merges first, this one moves to
60: the file, the version it stamps, `migrations.targetVersion`, the
bats file (its `run_migration` argument and expected `STAMP`) and
`hack/testdata/migration-59-vmi-hotplug`.

A VM with a pending hotplug request at upgrade time may also get stuck.
virt-controller still processes `VM.status.volumeRequests` before the
declarative handler, and it applies them through the VMI subresource,
which virt-api rejects once the old gate is off. The sync then fails
before the VM spec is updated, so nothing clears the request, and every
later sync of that VM fails the same way. A retry through the VM
subresource edits the spec and clears it, which covers kubevirt-csi's
own requests. A one-shot request stays stuck until the VM restarts. I
read this from the v1.9.0 code and haven't reproduced it. Waiting until
this lists nothing before upgrading narrows the window, but a request
made during the rollout can still hit it:

```bash
kubectl get vm --all-namespaces --output json | jq --raw-output '.items[] | select((.status.volumeRequests // []) | length > 0) | .metadata.namespace + "/" + .metadata.name'
```

The two halves live in different HelmReleases, so there is a short
window during the rollout where the driver and the gate disagree. In
that window an unpublish fails and is retried until both are updated.

A tenant cluster whose `kubernetes` HelmRelease is suspended or failing
keeps its old driver for longer. Drivers from before July 2025 attach
through the VMI subresource, so on such a tenant every attach is refused
until its release reconciles. Such a driver can also attach a volume
after the migration has run and before the gate flips. The migration
doesn't see that volume, and the first declarative reconcile unplugs it
from the running VM.

The migration lists the VMs once and patches each one with a
`resourceVersion` test, so a VM that changes in between fails the hook
Job. The Job has `backoffLimit: 3`, and each retry lists again and skips
the VMs already patched. Re-reading only the conflicting VM would avoid
the retry; I left that out.

The new gate is listed in the CR even though it is on by default. With
the old gate gone it is the only hotplug path, and the chart refuses a
render that disables it. I found no other user of `HotplugVolumes` in
the platform or in the downstream repositories.

The VMI-level fallback may have been failing already. The kcsi Role has
no rule for the VMI-level `removevolume` (#4452), so that call likely
got 403 with or without the gate. I haven't checked that on a cluster.

`hack/migration-59-vmi-hotplug-volumes.bats` runs the migration against
a fake kubectl: a VM that needs the copy, VMs and VMIs that must be left
alone (a VMI that is not Running among them), a second run, a cluster
without KubeVirt and the fail-closed paths. The helm-unittests for the
KubeVirt CR pin the whole gate list, accept `HotplugVolumes` in
`disabledFeatureGates` and refuse `DeclarativeHotplugVolumes` there. The
CSI test pins the flag at false, and the `make update` idempotency test
still passes. None of that exercises hotplug itself. The real test is
e2e with volume attach and detach on a tenant cluster, including a
teardown. No CI lane upgrades from the previous release yet, so an
upgrade with a live VMI-only volume is untested; #4463 tracks that test.

### Screenshots

No UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(kubevirt)!: KubeVirt now runs with DeclarativeHotplugVolumes instead of the deprecated HotplugVolumes feature gate, and kubevirt-csi no longer uses the VMI-level hot-unplug. Volume hotplug requests are applied through VM.spec. The upgrade copies volumes that older kubevirt-csi versions attached only to the VMI into VM.spec, so they stay attached; the check in the PR description is optional. This also applies to ephemeral volumes added to running user VMs at the VMI level, such as with virtctl addvolume without --persist: they become part of VM.spec and attach again after a VM restart. virt-api now refuses VMI-level addvolume and removevolume on a VMI owned by a VM, so such volumes are added and removed through the VM. A tenant cluster whose kubernetes release is suspended or failing keeps its old kubevirt-csi, and a driver from before July 2025 cannot attach volumes there until that release reconciles, and a volume it attaches during the upgrade itself can be unplugged again. A values override that puts DeclarativeHotplugVolumes into disabledFeatureGates now fails the kubevirt render. Clusters on the default variant do not run platform migrations and should run that check before upgrading.
```





<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
  * Declarative hotplug volumes are enabled by default for KubeVirt.
* Running virtual machines keep eligible hotplugged volumes and disks
when their configuration is migrated.
* **Bug Fixes**
* The migration avoids adding duplicate volumes or disks and does not
mark the migration complete if required operations fail.
  * The Kubernetes CSI driver no longer uses the VMI hotplug fallback.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-06 17:06:03 +03:00
Alexey Artamonov e63b9e439c fix(monitoring): count a central request as the receiver admitted it
The limiter refuses for good any request it counts above burst, and
burst was the receiver's 4 MiB cap. A request grows on its way to the
limiter: the sampler adds a tracestate to every span it keeps, and the
limiter counts a request as the collector re-encodes it, which writes
the empty value of every attribute the wire may omit, so a valid
request of spans with empty-valued attributes is counted 1.4x. A
request the receiver admitted near its cap was therefore refused on
every retry.

The limiter now runs right after memory_limiter, so it counts the
request before sampling. The receiver caps a request at 3 MiB and
burst stays at 5 MiB, which holds 1.4x the cap; a larger burst would
also be what each pod start lets through above the rate. The rate is
counted before sampling, so the store keeps only the sampled share of
it. The texts say that a request above the cap gets a 400, which
senders do not retry, and that each collector rollout a tenant causes
lets that burst through again on every pod.

The replica bound that keeps that allowance small now applies only to
a central tenant's collector, whose render fails above ten replicas: a
collector writing to the tenant's own store has no rate, and the
replica counts it shipped with stay valid.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00
Alexey Artamonov 3497855e15 fix(platform): read the tracing rate only as plain decimal digits
Sprig's int64 reads a string with a leading zero as octal and turns a
parse error into 0, so "0100" would reach the collectors as 64, and
"08" or digits beyond int64 would leave the default in place without
a word. A float beyond int64 converts differently on each CPU
architecture, and the collector adds its replica count to the rate
before splitting it.

A string rate must now be plain decimal digits, and any rate above
1 TiB/s fails the render; the collector also falls back to the default
on digits it cannot hold rather than start with a rate of zero, which
the processor refuses.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-10-06 09:39:04 +03:00