Commit Graph
3019 Commits
Author SHA1 Message Date
Aleksei Sviridkin 045f1a9bbc fix(seaweedfs): remove identities left behind by deleted BucketAccesses
Since SeaweedFS 4.08 the S3 gateways move every identity from
/etc/iam/identity.json into /etc/iam/identities/<name>.json, and the
COSI driver up to v0.3.1 revokes only from identity.json. Every
BucketAccess deleted on such a filer left its identity file behind,
with its keys still accepted (seaweedfs/seaweedfs-cosi-driver#11). A
driver that deletes the file on revoke does nothing for the files
already left.

A post-install/post-upgrade hook removes a ba-<uid>.json file when no
BucketAccess in any namespace names it, by uid or by
status.accountID. The uid covers a BucketAccess still being granted,
the accountID one restored with its status. The identity directory is
listed before the BucketAccesses: a BucketAccess exists before its
grant writes a file, so a file seen in the first listing whose
BucketAccess still exists is always named by the second. File ages are
not used, because grants and gateway migrations rewrite left-over
files as well. A failed or empty listing removes nothing. Each removal
is logged and recorded as an Event on the HelmRelease. The hook runs
before the Object Lock hook, so in delete mode a left-over identity is
removed before that hook grants it anything.

A filer published for Client-topology clusters holds identities whose
BucketAccesses live in those clusters, so cosi.orphanIdentities
defaults to auto: report there, delete otherwise. An operator can
force either.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-17 22:07:25 +03:00
Aleksei Sviridkin c49f58c4a4 fix(seaweedfs): keep Object Lock working for COSI credentials on SeaweedFS 4.48 (#4893)
## What this PR does

Existing bucket credentials need additional Object Lock actions to work
with SeaweedFS 4.48. Since v0.4.0 (#5284), the upstream COSI driver
grants them to new credentials. It does not grant access again for a
ready BucketAccess, so older identities stay unchanged after the driver
update. The custom COSI driver build and its image repository are no
longer part of this PR.

This PR adds a post-install and post-upgrade hook that grants the
actions implied by each identity's Read and Write permissions. Bucket
restrictions and credentials stay unchanged, and Write does not grant
governance bypass. The `anonymous` identity that public-read buckets
share is skipped on purpose. The driver revokes public read by removing
only `Read:<bucket>`, so added Object Lock reads would stay open to
unsigned requests. The hook runs after the release update and does not
guarantee Object Lock access during a server rollout.

The wrapper chart also declares its local dependency so Helm lint can
check it. The chart remains on 4.45. #4879 is stacked on this PR for the
4.48 upgrade.

The hook's bats suite (22 cases), the chart tests and shellcheck pass
locally on 65708a81b8. The Linux unit lane and E2E are pending. Live
cluster validation has not been run.

Known hook limitations remain tracked in #4894. A pod that never starts
can still fail the Helm action. Deletion between the dry run and apply
can leave a keyless identity. Warning Events expire, and the Job stays
behind for its logs.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(seaweedfs): grant existing bucket credentials the Object Lock actions required by SeaweedFS 4.48 on install and upgrade, preserving their bucket restrictions and credentials.
```
2026-11-17 18:05:41 +03:00
Aleksei Sviridkin 87f160d473 fix(vm-disk, kubernetes-nodes)!: refuse golden images whose import keeps failing (#4802)
## What this PR does

New clones can wait forever on a golden image whose import keeps
failing. CDI v1.66.1 retries ordinary HTTP failures without setting
phase `Failed`.

The guards now also refuse goldens with at least three importer restarts
unless they reached `Succeeded`. kubernetes-nodes and the picker require
`Running=False`; vm-disk refuses during running attempts too. Existing
vm-disks and already-cloned pools remain exempt. CDI never lowers the
restart count, so a healthy import with three early restarts stays
refusable until `Succeeded`, including while a recreated importer is
still starting.

An empty worker-pool StorageClass now resolves to the virtualization
default, then the GA Kubernetes default. Beta is a last fallback for the
class written explicitly onto a worker disk, not a CDI fallback. The
strict same-class policy stays.

This can refuse a previously accepted builtin pool on upgrade. The
affected pool has an empty StorageClass, no cloned disk yet, and a
golden on the Kubernetes default while another class is the
virtualization default. StorageClass is immutable: point
`osImage.builtin` at a catalog entry on the virtualization default or
switch to `osImage.factory`. Already-cloned pools keep their disk's
class.

A VMDisk's own HTTP import, golden-image health checks and imports that
hang without restarting remain outside this change.

Closes #4564

Refs #4059

### Screenshots

No UI changes.

### Downstream repositories

The platform values change only updates a comment. The application
schema change updates the StorageClass description without changing
fields, defaults or enums.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(vm-disk, kubernetes-nodes)!: refuse new clones of golden images whose import has not succeeded after at least three importer restarts. kubernetes-nodes and the image picker require the importer to be not running; vm-disk refuses during running attempts too. kubernetes-nodes now resolves an empty storageClass to CDI's virtualization default before the Kubernetes default. A builtin pool with an empty storageClass, no cloned disk yet, and a golden on a different class can now be refused. Point it at a catalog entry on the virtualization default or switch to osImage.factory. Already-cloned pools are unaffected.
```
2026-11-17 18:05:01 +03:00
Aleksei Sviridkin 638f0ffbe7 feat(redis): add affinity configuration (#2665)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, security (a fix that comes through a security
advisory), docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

Adds independent Kubernetes affinity configuration for Redis and
Sentinel pods through `RedisFailover.spec.redis.affinity` and
`spec.sentinel.affinity`. Empty or null rules preserve operator
defaults. Configured rules replace default soft pod anti-affinity;
include `podAntiAffinity` when replica spreading is required.

### Validation

- All 99 chart tests pass with helm-unittest v1.1.2; affected standalone
API package compiles.
- Package generation with cozyvalues-gen v1.7.0 is repeatable; embedded
ApplicationDefinition schema matches the generated values schema.
- Twenty-four offline render scenarios cover default/null/configured
affinity and TLS on/off; Redis checks also cover independent component
rules, single-replica Sentinel behavior, and preservation of upstream
cache configuration.
- Manifest generation, diff checks, and commit trailer checks pass. Each
branch contains one feature commit rebased onto upstream main.
- Root `make generate` remains blocked by the pinned code-generator
parser rejecting Go 1.27's `math/rand/v2` generic method. Affected API
deepcopy generation passes separately with controller-gen v0.16.4;
unchanged files deleted by the failed root generator were restored.
- cozyhr v1.6.1 is installed and checksum-verified. `make show` with
`--plain` still queries Kubernetes discovery, which this environment
cannot access; offline Helm rendering passes.

### Screenshots

Not applicable; no UI changes.

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/85.
Provider implementation is tracked separately from this application
change.
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(redis): add affinity configuration for Redis and Sentinel pods.
```
2026-11-17 15:28:41 +03:00
Aleksei Sviridkin 6efc0a0dc9 feat(rabbitmq): add affinity configuration (#2664)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, security (a fix that comes through a security
advisory), docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

Adds optional Kubernetes affinity configuration to RabbitMQ pods through
`RabbitmqCluster.spec.affinity`. Empty or null affinity keeps existing
scheduling behavior.

### Validation

- All 106 chart tests pass with helm-unittest v1.1.2; affected
standalone API package compiles.
- Package generation with cozyvalues-gen v1.7.0 is repeatable; embedded
ApplicationDefinition schema matches the generated values schema.
- Six offline render scenarios cover default/null/configured affinity
and TLS on/off.
- Manifest generation, diff checks, and commit trailer checks pass. Each
branch contains one feature commit rebased onto upstream main.
- Root `make generate` remains blocked by the pinned code-generator
parser rejecting Go 1.27's `math/rand/v2` generic method. Affected API
deepcopy generation passes separately with controller-gen v0.16.4;
unchanged files deleted by the failed root generator were restored.
- cozyhr v1.6.1 is installed and checksum-verified. `make show` with
`--plain` still queries Kubernetes discovery, which this environment
cannot access; offline Helm rendering passes.

### Screenshots

Not applicable; no UI changes.

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
https://github.com/cozystack/terraform-provider-cozystack/issues/84.
Provider implementation is tracked separately from this application
change.
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(rabbitmq): add support for configuring RabbitMQ pod affinity rules.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an optional affinity setting to control where RabbitMQ pods are
placed, including anti-affinity rules to spread pods across nodes.
* Configured affinity rules are applied to RabbitMQ pods. When affinity
is empty or unset, no affinity rules are added.
* **Documentation**
* Updated RabbitMQ configuration guidance to describe the affinity
setting, its empty default, and how to use anti-affinity to distribute
pods across nodes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-17 15:28:13 +03:00
Aleksei Sviridkin d99e960c00 feat(platform): package address-controller and its MetalLB driver (#3382)
## What this PR does

Adds IP addresses as a first-class resource
([cozystack/community#35](https://github.com/cozystack/community/pull/35))
as two optional system packages, off by default and enabled through
`bundles.enabledPackages`:

- **`cozystack.address-controller`** — the class-agnostic core
([cozystack/address-controller](https://github.com/cozystack/address-controller)):
`IPAddressClass` / `IPAddress` / `IPAddressClaim` in
`local.sdn.cozystack.io`, the PVC/PV-binding analog. It owns
claim–address binding, status and reclaim, and never allocates anything
itself.
- **`cozystack.metallb-iad`** — the per-class driver for addresses
announced by MetalLB
([cozystack/metallb-iad](https://github.com/cozystack/metallb-iad)). A
class names an existing `IPAddressPool` with `autoAssign: false` in
`cozy-metallb`, which the admin provisions together with its
advertisement. The driver holds reservations as placeholder Services in
its own namespace, and attaches an address to a Service that names a
claim with the `local.sdn.cozystack.io/ip-address-claim` annotation.

### Packaging

Both are ordinary wrapper packages under `packages/system/`, sourced
from the `cozystack-packages` artifact like every other package. The
upstream chart is vendored verbatim under `charts/` by `make update` at
a pinned commit, since neither project publishes a chart artifact: the
`v0.2.0` release-preparation commits of
[cozystack/address-controller#17](https://github.com/cozystack/address-controller/pull/17)
and
[cozystack/metallb-iad#11](https://github.com/cozystack/metallb-iad/pull/11).
Each controller image is pinned by digest to a `main-<sha>` build
carrying the same controller code: `address-controller:main-90e65ca`,
whose commit differs from #17 only in chart version and docs, and
`metallb-iad:main-1faf937`, whose commit differs from #11 only in chart
version and the address-controller module pin, a purely additive API
change the driver does not use. The pins move to the `v0.2.0` tags and
images once both are released. The images live under
`ghcr.io/cozystack/<name>`, outside the `ghcr.io/cozystack/cozystack/`
repositories the release tooling owns, so promotion and the nightly
mirror pass over them. The core installs its v1alpha1 CRDs with
`upgradeCRDs: CreateReplace`.

### What the wrappers add

- **Tenant access.** The upstream chart is Cozystack-agnostic and grants
only the controller itself. `cozy:addresses`, `cozy:addresses:view` and
`cozy:addresses:admin` aggregate into `cozy:tenant`, `cozy:tenant:view`
and `cozy:tenant:admin` respectively, so a tenant can manage
`IPAddressClaim`s in its own namespace, and the grant exists only while
the package is installed. The cluster-scoped `IPAddress` and
`IPAddressClass` are not granted: `cozy:tenant*` reaches a tenant
through a per-namespace RoleBinding, so a rule for either would never be
consulted. A tenant reads its address from the claim's
`status.addresses[]`.
- **The driver's admission policy, turned on for tenant namespaces.**
MetalLB honours a pin or pool selector from any writer of a Service: the
`metallb.io/` annotations, their `metallb.universe.tf/` predecessors and
`spec.loadBalancerIP`. A tenant cluster's cloud provider copies a guest
Service's annotations and `loadBalancerIP` onto the Service it creates
in the tenant namespace. The upstream `metallb-iad-service-pins` policy
reserves all of these, plus the sharing annotation, to the driver's
ServiceAccount and the cluster-admin groups. An admission policy binds
every principal regardless of RBAC, so the wrapper adds
`cozystack-cluster-admin`, the group Cozystack's OIDC administrators
authenticate as, to the chart's `system:masters` and
`kubeadm:cluster-admins`. The wrapper enables it in `Deny` mode for
namespaces carrying `namespace.cozystack.io/host`, which `tenant-root`
and every tenant namespace have and nothing in `cozy-*` does. Unchanged
values always pass, so the driver's pin on a tenant's own Service does
not lock the rest of that Service.
- **The driver's MetalLB namespace**, `cozy-metallb` instead of the
upstream default `metallb-system`.

### Variant scoping

`cozystack.metallb-iad` depends on `cozystack.address-controller`, which
is opt-in, and on `cozystack.metallb`, which is absent on `isp-hosted`,
on a slim variant that did not opt into it, and wherever
`disabledPackages` names it. The driver is emitted from the
`common-packages` helper only when the includes of MetalLB and of
`cozystack.address-controller` both rendered a Package, so it cannot be
enabled where it would sit in `DependenciesNotReady` forever, and it
follows either dependency automatically if its rules change. The core
carries no MetalLB dependency and remains available everywhere.

### Validation

`helm unittest` covers the aggregation labels and rules on the tenant
roles and the driver's emission, which requires both MetalLB and the
address core. `hack/select-install.sh --validate` passes, and the
selective-install fixture is regenerated from `hack/select-install.sh
--disabled postgres`. The admission policy is upstream's and carries no
render test here.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
[cozystack/website#810](https://github.com/cozystack/website/pull/810)
(component cards),
[cozystack/website#811](https://github.com/cozystack/website/pull/811)
(operator guide)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The website is the one repository this change reaches:
[cozystack/website#810](https://github.com/cozystack/website/pull/810)
adds the two new platform components to
`guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`, and
[cozystack/website#811](https://github.com/cozystack/website/pull/811)
adds an operator guide under `networking/`. The guide restates both
image refs, so a repin here needs the same edit there.

### Release note

```release-note
feat(platform): add IP addresses as a first-class resource as two optional system packages, `cozystack.address-controller` (IPAddressClass / IPAddress / IPAddressClaim) and its MetalLB driver `cozystack.metallb-iad`. Tenants can reserve an address with an IPAddressClaim. A LoadBalancer Service a tenant Kubernetes cluster creates can name that claim with the `local.sdn.cozystack.io/ip-address-claim` annotation, or draw one through auto-claim once an admin marks an auto-claim class. An admission policy reserves MetalLB's pool, pin and sharing inputs on tenant Services to the driver. The driver is only emitted where MetalLB and the address core are.
```




<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added address management for requesting and tracking IPv4, IPv6, or
dual-stack addresses, with configurable address classes and lifecycle
status.
* Added the MetalLB IAD driver for address provisioning through MetalLB.
It is included only when the required MetalLB and address-management
components are enabled.
* Added optional admission controls for address claims and protected
MetalLB service settings.
* Added optional address-controller metrics, monitoring, and alerting
resources.
  * Added tenant access roles for viewing and managing address claims.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-17 15:15:03 +03:00
Aleksei Sviridkin b6ef231d44 docs(kubernetes-nodes): state why a pool must share its golden's class
The worker image guard, its error message and the storageClass docs
said CDI cannot clone across StorageClasses and always falls back to a
host-assisted copy. CDI falls back only when the two claims have
different CSI drivers or volume modes; with one driver it asks that
driver for a clone into the other class. The guard stays, the reason it
gives is corrected, and the docs name the virtualization default class
an empty storageClass now resolves to.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-17 11:20:20 +03:00
Aleksei Sviridkin 263407a73f fix(api): leave golden images whose importer keeps failing out of the image options
CDI moves an import to Failed only on a fatal error such as a checksum
mismatch, so the picker still offered a golden whose URL is unreachable:
its importer restarts with back-off while the DataVolume stays in
progress. Drop a golden whose importer restarted at least three times
and is not running, the test the kubernetes-nodes chart applies before
cloning.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-17 11:20:18 +03:00
Aleksei Sviridkin 513d250a68 fix(kubernetes-nodes): refuse a golden whose importer keeps failing
The worker image guard refused only a golden in phase Failed, which CDI
reaches on a fatal error such as a checksum mismatch. An unreachable URL
restarts the importer with back-off while the DataVolume stays in
progress, so a pool pinned to such a golden waited forever for its first
disk. Refuse an importer that restarted at least three times and is not
running, under the same already-cloned exemption as Failed.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-17 11:20:17 +03:00
Aleksei Sviridkin 0d239fab81 Merge branch 'main' into feat/kubernetes-nodes-talos-sysctls
The generated cozyrds file is regenerated from the merged values.yaml.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-16 17:17:33 +03:00
kastakhov 977963dda2 feat(redis): add affinity configuration
Signed-off-by: kastakhov <16296930+kastakhov@users.noreply.github.com>
Assisted-by: LLM
2026-11-16 13:16:29 +00:00
kastakhov a0f85886f2 feat(rabbitmq): add affinity configuration
Signed-off-by: kastakhov <16296930+kastakhov@users.noreply.github.com>
Assisted-by: LLM
2026-11-16 13:14:09 +00:00
Aleksei Sviridkin 416abfd8e2 fix(etcd): scope tenant alerts to their scrape job (#5183)
## What this PR does

scopes every tenant etcd alert to its release scrape job. previously
each release evaluated every tenant, so one failing peer could produce
alerts from unrelated releases. tenant peer RTT is evaluated by the
tenant rules, while the remaining global etcd checks stay in place.

validation: both affected chart test runs pass, 46 tests in 17 suites.
both charts render. an expression-only simulated leader-loss check
against existing metrics selected 18 members across six namespaces
before scoping and only the three intended members afterwards.
thresholds are unchanged.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

no API fields, metric names, package interfaces or documented
workarounds change.

### Release note

```release-note
fix(etcd): scope tenant etcd alerts to their release and avoid duplicate tenant peer RTT alerts from the global rules
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Etcd alerts now match metrics from the exact etcd scrape job,
including slow HTTP requests, reducing the chance of unrelated metrics
triggering alerts.
* The peer communication latency alert excludes tenant etcd scrape jobs,
which are covered by tenant-specific rules.
* **Tests**
* Added checks to validate etcd alert job matching and tenant-job
exclusions.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-16 15:15:35 +03:00
Aleksei Sviridkin 65708a81b8 fix(seaweedfs): declare the vendored chart dependency
Helm lint rejects the wrapper chart while its bundled seaweedfs chart is
undeclared. Reference the local chart without constraining its version
so vendoring remains the source of the dependency.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-16 13:28:42 +03:00
Aleksei Sviridkin ca0fcb1c05 Merge branch 'main' into fix/etcd-alert-scope-20261102
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-16 13:28:42 +03:00
Aleksei Sviridkin 78e21fefcc fix(seaweedfs): grant Object Lock actions to existing S3 identities
SeaweedFS 4.48 requires dedicated Object Lock actions, but upgrading the
COSI driver does not re-grant access for ready BucketAccesses. Add the
actions implied by existing Read and Write grants after install and
upgrade, without changing credentials or granting governance bypass.

The anonymous identity that public-read buckets share is skipped: the
COSI driver revokes public read by removing only Read:<bucket>, so
derived Object Lock reads would stay open to unsigned callers.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-16 13:28:42 +03:00
Aleksei Sviridkin 1e5e532737 Merge branch 'main' into feat/monitoring-email-template
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-16 13:24:49 +03:00
Timofei Larkin 39918b7a72 feat(platform): package the address-controller and its MetalLB driver
Two optional system packages bring in IP addresses as a first-class resource
(cozystack/community#35): cozystack.address-controller, the class-agnostic
core (IPAddressClass / IPAddress / IPAddressClaim, local.sdn.cozystack.io),
and cozystack.metallb-iad, the driver implementing the contract for addresses
announced by MetalLB.

Both follow the umbrella shape: the upstream chart is vendored under charts/
from cozystack/address-controller and cozystack/metallb-iad, and what belongs
to Cozystack rather than to a generic Kubernetes controller lives in the
wrapper. Neither project publishes a chart artifact, so each `update` target
clones at a pinned commit, and the wrapper pins the image built from the
same controller code by digest; the charts' own floating :main defaults
would otherwise leave the controllers free to move underneath a cluster.

The wrapper carries the tenant grant. The upstream chart is Cozystack-
agnostic and grants only the controller, so nothing there lets a tenant use
the API it serves; cozy:addresses, cozy:addresses:view and cozy:addresses:admin
aggregate into the platform's base tenant roles, which resolve by label, so
the grant is present exactly while the package is. Only IPAddressClaim is
granted: the other two kinds are cluster-scoped and cozy:tenant* reaches a
tenant through a per-namespace RoleBinding, so a rule for either could never
be consulted.

The wrapper also turns on the driver's admission policy for tenant
namespaces. MetalLB honours a pin or pool selector from any writer of a
Service, through its current annotations, their metallb.universe.tf
predecessors and spec.loadBalancerIP alike, and a tenant cluster's cloud
provider copies a guest Service's annotations and loadBalancerIP onto the
Service it creates in the tenant namespace. Without the policy, a tenant
could steer MetalLB at an address reserved for someone else. Tenant
namespaces are the ones carrying namespace.cozystack.io/host, which
tenant-root and the tenant chart set and nothing in cozy-* does. An
admission policy binds every principal regardless of RBAC, so the
allowlist names Cozystack's OIDC administrators, cozystack-cluster-admin,
beside the chart's own two admin groups.

The driver points at cozy-metallb instead of the upstream default
metallb-system. Its PackageSource declares dependsOn on the core, whose CRDs
it needs, and on cozystack.metallb. The core installs with upgradeCRDs:
CreateReplace since its CRDs are v1alpha1 and still evolving. Neither package
is in the postgres E2E install closure, so both join the selective-install
fixture's disable list.

Assisted-by: LLM
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-11-16 06:49:47 +01:00
Aleksei Sviridkin c1740e7631 test(tests): select resources in absence-only Helm checks (#5174)
## What this PR does

Select the intended resource by kind or name in 23 absence-only Helm
tests: the 21 cases catalogued in #4665, a newer MariaDB backup-script
case, and Kafka's JVM-options case. The requested
kubernetes-nodes/nodegroup and OpenSearch TLS audit found no remaining
unanchored negative-only cases.

Add a diff-based CI guard against new positional selectors in
first-party suites on templates with multiple literal resource blocks.
It preserves single-document uses and unchanged indices, including pure
suite renames; renamed suites are still checked for added index lines.
It follows Helm's template and index precedence, and its fixtures use
portable file writes. The guard runs after the test steps so a guard
failure cannot suppress their results.

The guard is intentionally static: it counts `---`-separated blocks with
a top-level `kind`, not resources produced by helpers or ranges, and
does not evaluate whether whole-document branches are mutually
exclusive. It does not audit vendored suites or existing indices when a
template later gains documents.

Validation:
- `make helm-unit-tests` passes: 106 package test targets, 556 Helm
suite executions, and 4,067 Helm tests, including the dashboard
package’s additional Go checks.
- All 51 focused BATS checks pass. The full BATS sweep initially passed
2,435 of 2,500 cases; six graph-selection failures were then resolved by
using the repository-pinned yq v4.53.3 (127/127 targeted rerun cases
pass). The remaining 56 Docker-dependent and three envsubst-dependent
cases are blocked by missing host prerequisites; no full BATS pass is
claimed. The six yq-related failures also reproduce on the unchanged
base with the older tool and pass with the pinned version.
- All 17 guard regression tests pass, including first-party scope,
inherited index precedence, pure/edited renames, template overrides,
nested default templates, conditional kinds, and multiline index values.
- Missing-target mutations cover all 23 selector changes: original cases
pass against unrelated resources; updated cases fail with `document not
found`.
- Reordered violating-target mutations cover all 22 formerly positional
cases: originals pass against leading unrelated resources; updated cases
find the intended target and fail their content assertion. Kafka's
updated case also passes when the reordered Job remains valid.
- Empty-render mutation confirms that the newer MariaDB backup-script
case previously passed without a document and now fails. Existing
positional cases already fail on an entirely empty render.
- The changed workflow passes the repository-pinned zizmor v1.26.1
offline audit; `git diff --check` is clean.

Closes #4665

### Screenshots

Not applicable (tests and CI only).

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Chores**
* Pull request checks now flag positional document selections in Helm
tests when they target multi-document templates.
* **Bug Fixes**
* Updated chart tests to select Kubernetes resources by kind or name
rather than document position, keeping assertions focused on the
intended resources.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-15 05:59:52 +03:00
Aleksei Sviridkin da17254bbc fix(kubernetes): bring up a replacement worker before draining the old one (#5153)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Worker pool MachineDeployments roll out with `maxUnavailable: 1`. CAPI
therefore drains and deletes an old Machine as soon as its replacement
Machine has been created, not once the replacement has joined. Observed
on a one-node pool changing instance type (`s1.large` to `s1.medium`):
the new Machine was created at 13:07:19, the old node was drained and
deleted 17 s later, and the new node was Ready around 13:11, so the pool
had no node for about four minutes. Larger pools lose one node of
capacity at every step of the rollout for the same reason.

This sets `maxUnavailable: 0`, so a node is drained only once its
replacement is up. `maxSurge` stays at the pool ceiling (`maxReplicas`),
with a floor of 1: CAPI rejects a RollingUpdate where both values are 0,
and `maxReplicas: 0` is accepted by the schema.

Exception: a pool with `gpus` keeps `maxUnavailable: 1`. Its replacement
VM may need the very passed-through GPU the old node holds, and with
`maxUnavailable: 0` it would stay Pending and the rollout would never
finish. A Proxmox pool that passes GPUs through its template carries no
`gpus` value, so the chart cannot recognise it: the new optional
`maxUnavailable` value lets such a pool (or any pool) set the strategy
explicitly, and set to 1 restores the drain-first behaviour.

The value is bounded to 0..2147483647, since CAPI decodes it as an
int32. ComputePlane pools render through the same chart, so
`packages/extra/computeplane` exposes `nodeGroups.<pool>.maxUnavailable`
and its allowlist forwards it.

Trade-off: a rollout now needs room for at least one extra VM. A tenant
at its quota stalls the rollout (the new Machine stays Pending) instead
of temporarily shrinking the pool, which is the safer failure.

Tests: `tests/rollout_strategy_test.yaml` (7 cases: the default, the
ceiling, a zero ceiling, a GPU pool, and the explicit override) fails on
`main` without the template change; the render snapshot is updated. Two
more cases cover the int32 maximum, accepted at 2147483647 and refused
one above. `helm unittest packages/apps/kubernetes-nodes`: 252 passed.
`packages/extra/computeplane/tests/cluster_test.yaml` checks that the
wrapper forwards `maxUnavailable` (it fails without the allowlist entry)
and refuses a value above the int32 range: 15 passed.

Backport: on `release-1.6` the same strategy is also rendered by
`packages/apps/kubernetes/templates/cluster.yaml` (the in-chart
`nodeGroups`), and the snapshot file does not exist there, so the
automatic backport cannot apply. A manual backport PR to `release-1.6`
covering both charts follows. `release-1.7` keeps `kind/backport`.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#79 (tracking issue:
the provider change is out of scope here). `cozystack_kubernetes_nodes`
has no attribute for the new `maxUnavailable`, so a Terraform-managed
pool cannot drain first; the provider has no ComputePlane resource yet.
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(kubernetes): worker pool rollouts (instance type, image or version changes) now bring each replacement node up before draining the old one, instead of removing a node first. A rollout needs room for one extra VM. Pools with GPUs keep the previous behaviour.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added a pool-level `maxUnavailable` setting to control how many
workers may be removed before replacements are ready. It defaults to 0
for non-GPU pools and 1 for GPU pools; configured values override these
defaults.
* Proxmox pools passing through GPUs can set `maxUnavailable` to 1 to
help prevent rollouts from waiting indefinitely.
* **Bug Fixes**
* Rolling updates now use at least one surge node, even when the replica
limit is zero.
* **Documentation**
  * Documented rollout settings, defaults, and capacity considerations.
* **Tests**
* Added coverage for defaults, overrides, surge limits, and invalid
values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-15 02:48:13 +03:00
Aleksei Sviridkin 46cb99df5b fix(controller, platform): give unbounded pool members the root's LimitRange (#5265)
## What this PR does

Since tenant quotas became hierarchical (#3086), an unbounded sub-tenant
(no `resourceQuotas` of its own) shares its parent's pool and receives a
controller-owned `tenant-quota-allocated` ResourceQuota. The tenant
chart, however, renders the `tenant-range-limits` LimitRange only next
to the `tenant-quota` of a tenant that declares `resourceQuotas`, so
these namespaces end up with a quota on `limits.cpu`/`limits.memory` and
no container defaults. Admission then rejects every container that omits
one of those limits.

Pods that predate the quota keep running and fail as soon as they are
recreated. In production this took down CNPG postgres instances with
backups enabled: the `plugin-barman-cloud` sidecar injected by the
barman-cloud plugin sets CPU and memory requests and a memory limit but
no CPU limit, so the instance pod is refused on its next restart. The
same happened to the tenant `kubernetes` app: the `wait-for-kubeconfig`
init container of `cluster-autoscaler`, `kcsi-controller` and the
`bootstrap-token` jobs has no resources, so their ReplicaSets and Jobs
were refused with `failed quota: tenant-quota-allocated: must specify
limits.cpu` in every unbounded sub-tenant hosting a tenant cluster. On a
production cluster, 131 tenant pods currently get their CPU or memory
from the tenant LimitRange (VictoriaMetrics
`config-init`/`config-reloader`, Harbor exporter, MariaDB agent, Strimzi
entity operator, among others), so the defect is not specific to one
chart: any of them breaks once it lands in such a namespace. Workloads
that set every limit explicitly are unaffected.

The controller already decides which namespaces it clamps, so it now
copies the pool root's chart LimitRange into each unbounded member whose
allocated quota bounds `limits.cpu` or `limits.memory` as
`tenant-range-limits-allocated`, keeps it in step with the root, and
deletes it once the namespace leaves the pool (for instance when the
sub-tenant declares its own quota and the chart renders its own
LimitRange). The chart renders its LimitRange under any non-empty
`resourceQuotas`, including one that bounds no compute (#4930), so the
copy is limited to members whose quota bounds a compute limit: elsewhere
it would impose a 128Mi default memory limit the quota never asked for
and refuse pods, virt-launcher among them, that are admitted today. This
keeps the change independent of #4954. The copy has its own name so that
Helm can still create the chart-owned object in that case without an
ownership conflict. The root is the single source of truth for the
defaults, so the values stay defined in
`packages/apps/tenant/templates/quota.yaml` only. The controller gains
`limitranges` RBAC and watches the two LimitRange names so a change
propagates without waiting for the 60 s resync.

Tests: five reconciler cases cover a member under a non-compute quota
(no copy, and a stale one removed), the copy into an unbounded member
(and not into the root), propagation of a change on the root, garbage
collection once the member is bounded (the chart-owned object is left
alone), and a root without a LimitRange. Disabling the copy makes the
first two fail.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff. The code touches only
`internal/controller/tenantquota/` and the `cozystack-controller` RBAC
template, but the website's `system-memory-limits.md` states that tenant
namespaces get defaults only from the chart's `tenant-range-limits`,
which this change makes untrue; the follow-up is an issue, since the
page should change once this is merged.

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#806
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(controller, platform): sub-tenants without their own resourceQuotas now receive their parent pool's container defaults (LimitRange) alongside the allocated quota, when that quota bounds a CPU or memory limit, so pods that omit such a limit, such as the postgres barman-cloud backup sidecar, are no longer rejected at admission.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Unbounded pool members now receive their pool’s CPU and memory limit
settings when their allocated quota sets those limits. Pool setting
updates are reflected in members during reconciliation.
* When a member has its own quota, inherited limit settings are removed
while settings configured directly for that member remain unchanged.
* No inherited settings are created when the pool has no limit settings
or the allocated quota does not set CPU or memory limits.
* Existing inherited settings are retained if an update temporarily
fails.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-15 02:33:34 +03:00
Aleksei Sviridkin eafe1adde5 Merge branch 'main' into feat/kubernetes-nodes-talos-sysctls
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-14 03:55:52 +03:00
Aleksei Sviridkin f8217be501 Merge branch 'main' into chore/cilium-drop-unread-values
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-14 03:55:46 +03:00
Aleksei Sviridkin 3ede7c4d33 feat(observability-proxy): include platform exporter series in the namespace root (#5277)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Builds on #5219 and #5220, both merged.

The namespace metrics root
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/metrics/...` now also
returns series that a platform exporter attributes to `<ns>`. Example:
the ClickHouse operator's exporter runs in `cozy-clickhouse-operator`.
vmagent scrapes it without `honorLabels`, so its series carry
`namespace="cozy-clickhouse-operator"`, and the tenant namespace the
exporter reported moves to `exported_namespace="<ns>"`. Until now, any
client of the namespace root (kubectl, automation, the console) never
saw those series.

### Behavior changes

- Namespace root, every endpoint (`query`, `query_range`, `series`,
`labels`, `label/<name>/values`):
  - was: one proxy-set `extra_filters[]={namespace="<ns>"}`;
- now: two values, `{namespace="<ns>"}` and
`{namespace=~"cozy-.*",exported_namespace="<ns>"}`. vmselect ORs them.
- Label APIs without a client `match[]`: the proxy already repeats the
scope in `match[]` for `-search.ignoreExtraFiltersAtLabelsAPI`; it now
repeats both values. vmselect ANDs every `match[]` with every
`extra_filters[]`. Each cross pair is narrower than one of its halves,
so the result is still exactly the union.
- Client `extra_filters`, `extra_filters[]` and `extra_label` are still
dropped.
- New flag `--metrics-platform-namespace-prefix` (default `cozy-`) and
chart value `observabilityProxy.metrics.platformNamespacePrefix`. An
empty value turns the second branch off and restores the single-value
scope. The proxy quotes the prefix and anchors it, so the value is
matched literally and cannot become a wider regex. The flag accepts only
namespace name characters, because any other value would match no
namespace and silently drop the exporter series.
- Exporter series keep `namespace="cozy-..."`. A query that also selects
`namespace="<ns>"` excludes them; select them by `exported_namespace` or
by the exporter's own labels (for ClickHouse, `chi`).
- The branch is not specific to ClickHouse. For example,
kube-state-metrics in the platform monitoring namespace sets
`exported_namespace` from `metadata.namespace` on the Flux
`gotk_resource_info` series
(`packages/system/monitoring-agents/values.yaml`), so a namespace root
now also returns the state of that namespace's own HelmReleases and
Kustomizations. This is the namespace's own data, but it is a visible
change with the default prefix.
- Queries grouped `by (namespace)` can now return an extra group
labelled with the exporter's platform namespace. The exporter's series
also carry its own target labels (`pod`, `instance`, `job`) into the
tenant's results.
- An empty scope is refused with 500 instead of forwarding a request
without `extra_filters[]`.
- The application roots of #5220 are not affected: they replace the
scope with their own selectors.

### Threat model

1. **Scrape path.** vmagent sets `namespace` from service discovery,
while `exported_namespace` is whatever the exporter wrote. Any tenant
pod can write a `namespace` label, which arrives as
`exported_namespace`, so the extra branch only trusts it on series
scraped from platform namespaces. Tenant roles
(`packages/system/cozystack-basics/templates/clusterroles.yaml`) grant
no access to `operator.victoriametrics.com` and no verbs on `Namespace`.
A tenant can therefore neither run a target in `cozy-*` nor point a
scrape object at one.
2. **Remote-write writers.** A tenant's nested Kubernetes cluster runs
its own vmagent, which remote-writes into the same store when the tenant
has no monitoring of its own, and the labels it writes are not enforced.
Inside that cluster the tenant can create `cozy-*` namespaces or any
other, so it can already write `namespace="<any ns>"` and pass the
existing single filter. This is an existing limitation that this PR
neither introduces nor widens: anything that can write
`namespace="cozy-x",exported_namespace="<ns>"` can write
`namespace="<ns>"` directly. Closing it needs enforcement on the write
path and is out of scope here.
3. **Impact of a forged series.** The scope only adds series to a
namespace's results; it never lets a caller read another namespace. A
forged series can spoil or flood another namespace's graphs, up to the
series limits that answer 422.

### Rejected alternatives

- **Bare `{exported_namespace="<ns>"}`.** Every namespace is scraped by
default, so any tenant pod could inject series into another namespace's
root.
- **Relabeling the ClickHouse operator's scrape so that `namespace`
carries the tenant namespace.** It fixes one exporter at a time, each in
its own chart, and it changes the label set of every existing series,
which splits their history at the change and breaks any query that
selects them by the operator's namespace. The proxy-side branch covers
every platform exporter and leaves stored series as they are.
- **Scraping the ClickHouse pods directly.** The metric names differ
from the operator's exporter, so the shipped ClickHouse dashboard
(`dashboards/clickhouse/altinity-clickhouse-operator-dashboard.json`),
which reads the exporter's `chi_clickhouse_*` series, would have to be
rewritten.
- **A free regex flag instead of a prefix.** One typo (`.*`) would trust
`exported_namespace` from every namespace.

### Tests

- Table of namespace-root requests: exactly two `extra_filters[]` values
in order with the default prefix, one value with an empty prefix, on all
five endpoints. Client `extra_filters[]` still dropped.
- A model of vmselect's selection (OR over `extra_filters[]`, AND with
each `match[]`) shows that `namespace=cozy-x, exported_namespace=<ns>`
is visible, and that `namespace=tenant-y, exported_namespace=<ns>` and
`namespace=xcozy-a, ...` are not.
- Escaping of `<ns>` in `exported_namespace`, literal prefix quoting,
empty scope refused, flag validation, and a chart test that an empty
value reaches the binary as an explicit empty flag.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff touches only `cmd/observability-proxy`,
`internal/observabilityproxy` and the system package
`packages/system/observability-proxy` (a new value and flag, no app,
platform values, CRD, `hack/` or naming change). None of the triggers in
`docs/agents/contributing.md` under "Downstream Repositories" apply.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(observability-proxy): the namespace metrics root also returns series that exporters in platform (`cozy-*`) namespaces attribute to the namespace through `exported_namespace`, such as the ClickHouse operator's. The prefix is set by `--metrics-platform-namespace-prefix` / `observabilityProxy.metrics.platformNamespacePrefix`; an empty value disables it.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Metrics queries now include series attributed to a tenant namespace by
exporters in platform namespaces with the configured prefix. The default
prefix is `cozy-`; setting it to an empty value disables this behavior.
* Namespace scoping is applied across supported metrics endpoints,
including label lookups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 02:44:48 +03:00
Ahmed Kamal b1e067f6e1 chore(tests): merge upstream main into resource selector branch
Preserve the Helm selector guard alongside main's independently gated
backport tests when refreshing the branch's upstream base.

Validated with actionlint and 31 focused Bats tests.

Signed-off-by: Ahmed Kamal <email.ahmedkamal@googlemail.com>
Assisted-by: LLM
2026-11-14 00:58:47 +02:00
Aleksei Sviridkin 7e069356b9 Merge branch 'main' into feat/kubernetes-nodes-talos-sysctls
Generated files (types, README, schema, cozyrds) are regenerated from
the merged values.yaml.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-14 00:43:22 +03:00
Aleksei Sviridkin c93af24ea2 Merge branch 'main' into fix/vpa-admission-webhook-timeout
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-14 00:42:05 +03:00
Mattia Eleuteri 2568db67b1 Merge branch 'main' into fix/kubernetes-nodes-surge-before-drain
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 22:40:18 +01:00
Aleksei Sviridkin 5dbfa1cfad fix(linstor): restore snapshots on a schedulable node (#5309)
## What this PR does

When linstor-csi restores a snapshot (a PVC with a `VolumeSnapshot` data
source, as the Velero data mover creates for every backup), it restores
it on exactly one node, and the new volume is pinned there. Two things
go wrong in how that node is chosen.

- The candidate filter is `slices.Contains(preferredNodes, snapNode)`,
but `preferredNodes` is the output of `SortByPreferred`, which holds
every node and only orders them. The filter therefore never filters, and
the node the scheduler selected for a `WaitForFirstConsumer` volume
(`preferredNodes[0]`) gets no priority. When the snapshot is not on that
node, the pod's `PreBind` fails with `node affinity doesn't match` and
needs another scheduling round. That happens even when the selected node
does hold the snapshot.
- `NodeAvailable` only checks that the LINSTOR satellite is `ONLINE`. A
cordoned Kubernetes node passes. A volume restored there can never be
used, because no new pod can be scheduled on that node: the pod stays
`Pending` until the node is uncordoned. On a production cluster with two
storage nodes cordoned ahead of a reboot, this failed 42 of 44 restores
that landed on those nodes, while 0 of 777 restores on the other nodes
failed. Each failure surfaced as a Velero `DataUpload` ending in
`timeout on preparing data upload`, although the snapshot also existed
on two schedulable replicas.

Patch `003-snapshot-restore-node-selection.diff` makes the controller:

- restore on the scheduler-selected node when it holds the snapshot;
- otherwise pick a random snapshot node, keeping the spreading that
`001-relocate-after-clone-restore.diff` introduced, but skip cordoned
nodes as long as another node holds the snapshot. When only cordoned
nodes hold it, behaviour is unchanged.

Cordon state is read from the Kubernetes `Node` object
(`spec.unschedulable`), through the in-cluster typed client that
`utils.KubernetesClient` already provides. The `linstor-csi-controller`
ServiceAccount already has `get` on `nodes` for the
external-provisioner. Outside Kubernetes, or when the lookup fails,
every node counts as schedulable, as before.

Tests: `TestLinstor_reconcileSnapshotResourcesSkipsCordonedNodes`
reproduces the production case through `reconcileSnapshotResources` with
mocked LINSTOR providers. Without the cordon check, 16 of 50 restores
landed on the cordoned node; with it, none do. `TestSelectRestoreNode`
covers the selected node, the cordoned selected node, only-cordoned
snapshot nodes, the preferred-node fallback and the spreading. The
Dockerfile runs `go test` in `pkg/client` as part of the image build.

### Downstream repositories

The diff only adds a patch under
`packages/system/linstor/images/linstor-csi/patches/`. I walked the
trigger map against it and found no entry that it matches.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(linstor): restore a snapshot on the node the scheduler selected when it holds the snapshot, and never on a cordoned node while another node holds it, so pods using a volume restored from a snapshot (such as Velero data mover backups) no longer stay Pending while a storage node is cordoned
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Snapshot restores now avoid cordoned nodes when another usable node is
available and prioritize preferred nodes when possible.
* Restore selection remains available when Kubernetes node information
is missing or lookup errors occur.
* When no preferred or schedulable node is available, restores can fall
back to any available node.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 00:13:49 +03:00
myasnikovdaniil aa36d3fad6 fix(kubernetes)!: remove the talos block and other leftovers of the worker pool split (#5191)
## What this PR does

Cleanup after the Kubernetes app was split into `Kubernetes` (control
plane and addons) and `KubernetesNodes` (one release per worker pool):
what the split left behind in values, docs, templates and tests.

### Kubernetes

The `Kubernetes` application has no `talos` field any more. Worker pools
are `KubernetesNodes` resources and take their Talos release, image,
installer and registry mirrors from their own `talos` and `osImage`, so
the cluster's copies did nothing. The last one, `talos.version`, fed a
duplicate of the Talos to Kubernetes support-matrix check. The pool
chart already runs that check against the Talos release each pool
actually boots, including one picked through an `osImage` override; the
cluster's copy compared the Kubernetes version against a release no node
runs. A stored spec that still carries `talos.*` keeps rendering,
because the values schema does not close the object and the API does not
validate `spec` against it, and a unit test pins that.

The pool's check now has render coverage. Its own suite pins the
pairings it must accept, and a bats test renders the refusals through
`talos.version`, `osImage.factory.version` and `osImage.builtin.version`
with schema validation skipped, since the schema offers no Kubernetes
version the matrix refuses.

I fixed the README and field descriptions that described behavior the
chart no longer has, or never had: a dedicated etcd cluster, a
`blockSize.matchVolume` setting that does not exist, a default `md0`
pool, the host default, the substrate mismatch behavior, and rollout
phase names. The README's Breaking Changes section becomes a pointer to
the per-release upgrade notes on cozystack.io. Unused helpers and stale
comments are removed; the chart renders the same bytes.

### KubernetesNodes

Five field descriptions and the `talos` block told users to keep them in
sync with parent fields that do not exist. NOTES warned about colliding
with a parent-managed `md0` pool, although the parent renders no pools
and `md0` is the normal name of a pool. The installer image was credited
to an upgrade Job that never runs one. The values descriptions, README
intro and NOTES now say what the chart renders today, including which
fields have no effect on `substrate: proxmox`: `storageClass`,
`diskSize`, `instanceType`, `talos.imageFactoryURL` (beyond its format
check) and `talos.imageFactoryCA`.

The render error for a pool whose MachineDeployment belongs to another
release told users to remove the pool from the parent's `nodeGroups`,
which cannot clear the conflict. It now names a remedy that works.
Split-era history is gone from Helm and test comments, along with two
`storageClass` fallbacks that can only return the value they fall back
from, a constant variable, and two `.helmignore` rules that match
nothing.

The tenant pod and service CIDRs are literals in both charts, and
nothing checked that they agree. A new bats test renders both charts and
fails when they differ.

No rendered manifest of the pool chart changes. I rendered it for the
snapshot values, GPU, kubelet-override, factory/CA, storageClass
variants and three Proxmox variants on main and on every commit, and the
output is byte-identical. The snapshot and TalosConfigTemplate hash
tests pass unchanged, so no worker rolls. YAML comments inside the
hashed machine config and machine template are left as they are for that
reason.

### The rule

`docs/agents/overview.md` gains a rule so this does not come back with
the next change: app READMEs and field descriptions describe current
behavior, a change users must act on goes to the website's upgrade page,
and a description does not ask the reader to keep it in sync with
another chart.

```release-note
[kubernetes] The Kubernetes application no longer has a `talos` field. Talos settings for workers, and the Talos to Kubernetes compatibility check, live on each KubernetesNodes pool. Existing clusters that still store `talos` values keep working; the values are ignored. KubernetesNodes field descriptions, README and install notes now describe the worker pool as it works today, including which fields have no effect on Proxmox pools.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration**
* Worker OS image, installer, and registry-mirror settings are
configured per worker pool; these settings are no longer available at
the cluster level.
* Kubernetes and Talos version compatibility is checked for each worker
pool. A worker Kubernetes version cannot be ahead of its parent cluster.
* Worker pools support KubeVirt and Proxmox, with pool-specific sizing
and disk settings.
* Rendering is refused when a pool uses a different substrate from its
cluster or conflicts with an existing worker pool owned by another
release.
  * The default cluster hostname now includes the `kubernetes-` prefix.
* Existing cluster configurations that still include Talos settings
receive a warning; those settings are no longer used.

* **Documentation**
* Updated guidance for cluster and worker-pool settings, storage,
upgrades, and version compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 00:34:44 +05:00
Maxim Belyy b24a9b4cb9 Merge branch 'main' into feat/observability-proxy-exported-namespace
Resolve the overlap with the per-application roots: the namespace root
keeps its platform exporter branch, the application root still replaces
the whole scope with its own selectors, and the label APIs mark the
repeated scope so that replacement reaches match[] as well.

Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-14 00:49:51 +07:00
Aleksei Sviridkin 4f7eb63447 feat(observability-proxy): serve per-application logs and metrics (#5220)
## What this PR does

Depends on #5219. This PR is stacked on that branch; the description
covers only what it adds on top.

Adds the per-application roots to the observability proxy, as described
in the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94):

```text
/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/log/select/logsql/<endpoint>
/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/metrics/api/v1/<endpoint>
```

This lets the console show the logs and resource consumption of one
application rather than of the whole namespace.

What is added:

- Plural-to-Kind mapping from an informer over `ApplicationDefinitions`.
The Kind is looked up only after the `SubjectAccessReview` (`get
<plural>/log` or `get <plural>/metrics`), so an unauthenticated caller
cannot probe which plurals exist. An unknown plural answers `404`; an
unsynced cache or a plural declared by two kinds answers `503`.
- Logs: a second `extra_stream_filters` value with the application Kind
and name, on top of the unchanged namespace filter. VictoriaLogs ANDs
the two, on every endpoint including `tail`.
- Metrics: the forced `extra_filters[]` becomes an OR of three branches,
each still bound to the namespace:
  1. series carrying the application identity labels;
  2. series of the application's pods, found through `kube_pod_labels`;
3. series of the application's PVCs, found through
`kube_persistentvolumeclaim_labels`.
- The pod and PVC lookups cap the number of names
(`--metrics-app-max-pods`, `--metrics-app-max-pvcs`); over the cap
answers `422`. A failed pod or PVC lookup fails the request (`502`, or
`504` on the deadline) rather than answer without the application's
series, and so does a lookup that vmselect marks `isPartial`, since a
storage node that did not answer leaves names out (a partial list
already over the cap still answers `422`); vmselect's series-limit
refusal of a lookup is passed through as `422`. A query that reads
further back than `--metrics-app-lookback` (default 720h, chart value
`observabilityProxy.metrics.appLookback`), or by an unknown amount, and
a query whose samples lie at a time the lookup cannot cover answer `422`
on the application root; this includes every query with an `@` modifier,
`WITH` with a negative offset, and offsets that are not plain durations.
The namespace root still serves them. `rollup_candlestick`, a backslash
or a `#` comment outside a string, an offset directly followed by a
minus (compound durations such as `-1h-30m` or `1h-2h`), and a subquery
step that is not a plain duration are refused as well. The lookup
reaches past the evaluation time by any negative offset and by two steps
per subquery level.
- Application routes require read access to the namespace root as well
as the application subresource; a grant narrowed with `resourceNames`
alone answers `403`.
- The proxy's ClusterRole gains `list`/`watch` on
`cozystack.io/applicationdefinitions`. No tenant RBAC change is needed:
the `view` role already allows `get` on every `apps.cozystack.io`
resource and subresource.
- `/readyz` now also waits for the `ApplicationDefinitions` cache, so a
missing CRD or RBAC rule keeps the whole proxy unready, namespace roots
included.

#### Data comes from the identity labels

The application roots find data only through identity labels added by
three PRs that are now in main:

- #5215 `feat(monitoring-agents)`: export PVC labels from
kube-state-metrics (PVC branch).
- #5216 `feat(monitoring)`: stamp application identity on scraped series
(identity branch).
- #5217 `feat(monitoring-agents)`: keep application identity in
container log streams (logs root).

Merge after #5219.

The pod branch does not depend on them, because `kube_pod_labels`
already exports all pod labels. Logs and series written before those
changes rolled out carry no identity and are reachable only through the
namespace root.

#### Security condition: a convenience filter, not an isolation boundary

The application roots narrow the namespace roots; they do not isolate
applications from each other. Each application route therefore requires
two grants: the application subresource (`get <plural>/log` or `get
<plural>/metrics`) and read access to the namespace root (`list
cozystack.io/logs` or `cozystack.io/metrics` in the namespace). A grant
narrowed with `resourceNames` alone is refused. Every tenant role that
can read applications already has both. The application scope cannot
isolate yet because:

1. The lineage webhook keeps `apps.cozystack.io/application.*` labels
that a user sets on a pod or PVC that has no application owner, so such
an object can claim another application's identity.
2. The webhook's `objectSelector` skips any object created with
`internal.cozystack.io/managed-by-cozystack`, pods included, so forged
labels on such an object are never examined.
3. The pod and PVC branches match by name over the lookup window, so an
object that reuses the name of another application's recently deleted
pod or PVC pulls that object's older series into scope.

The proposed fix (a separate change to the lineage webhook) is to strip
the identity labels when no owner resolves, and to stop skipping CREATE
requests for objects carrying the managed-by label.

#### Other known gaps

- PVCs created from a StatefulSet's `volumeClaimTemplates` usually get
no lineage labels, so their volume series are not in the application
scope.
- ingress-nginx series intentionally carry no application identity, so
they are reachable only through the namespace root.
- A request for an application that does not exist answers `200` with an
empty result rather than `404`: the proxy checks authorization, not
existence. Nothing leaks; whether a `404` is wanted is open for review.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages, including a
narrowed-grant test (access to application `a` only): every application
path for `b` answers `403` and no request reaches the store. Name
escaping in the pod and PVC branches is covered for regex metacharacters
and quotes.
- helm-unittest for `observability-proxy` (new RBAC rule).
- Verified on a test cluster, with the three identity PRs above applied:
the application logs root of a Redis instance returned only that
instance's lines and none from a neighbouring pod in the same namespace;
its application metrics root returned only the Redis pods' series.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: no trigger matched. Candidate: none affected, because the
diff on top of #5219 is confined to the proxy's Go code and its own
chart RBAC; it reads `ApplicationDefinitions` but does not change their
semantics or the CRD. The security condition above may deserve a note in
the design proposals in `cozystack/community`, which is not a
trigger-map item.

### Release note

```release-note
feat(observability-proxy): add per-application logs and metrics roots under `/apis/apps.cozystack.io/v1alpha1/namespaces/<ns>/<plural>/<name>/{log,metrics}/`, scoped by application identity labels. Data appears for logs and series collected after the application identity changes to Fluent Bit, VMAgent and kube-state-metrics are rolled out.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added a tenant-aware observability proxy for logs and metrics, with
namespace and application scoping, access checks, and per-user request
limits.
* Enabled the proxy in the system package bundle and routed console log
and metrics requests through it.
* Added live log streaming and health, readiness, and metrics endpoints.
* **Bug Fixes**
* Invalid or excessive query parameters are rejected; requests with
uncertain application-metric scope are refused rather than returning
incomplete results.
* **Tests**
* Added end-to-end and integration coverage for tenant isolation,
console routing, access controls, and log streaming.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 20:47:10 +03:00
Aleksei Sviridkin 526f08e17d fix(mariadb): restore the dump a Backup was taken as (#5227)
## What this PR does

A RestoreJob that names a MariaDB `Backup` restored the newest dump of
the application, not the dump of that `Backup`. The driver created the
operator `Restore` with only `backupRef`, and mariadb-operator then
restores the dump nearest to the moment the restore starts, from the
prefix every run of the app shares. Restoring an older `Backup` brought
back the latest data, and when its own dump was already gone it silently
restored another one.

The `Backup` now records the dump its operator `Backup` wrote, the file
name in `k8s.mariadb.com/dump-file` and the timestamp from that name in
`k8s.mariadb.com/dump-time`. They come from the ownership-proven match
the size probe already makes, and nothing is recorded when ownership is
not proven. A restore first checks with a bounded S3 listing of the
operator `Backup`'s own storage that the dump still exists, then sets
`targetRecoveryTime` on the operator `Restore` to the dump's own
timestamp, which mariadb-operator matches exactly. If the dump is gone
the RestoreJob fails and names it. If the check cannot be made it waits
with `Ready=False/DumpCheckPending` until the restore deadline and then
fails with the cause.

`Backup`s taken before this stay restorable. The restore makes the same
ownership-proven match against their operator `Backup` and pins the dump
when it is provable, otherwise it restores the newest dump as before and
a `BestEffortDump` warning event on the RestoreJob says so. One case is
not covered: the `Backup`'s own dump is gone, a neighbour's dump from
within the minute of slack is still there and its operator `Backup` was
deleted, so that dump is taken as the match. Nothing in the platform
deletes operator `Backup`s, it needs someone to delete one by hand. A
`Backup` made of an operator `Backup` whose Job failed (`Complete` with
reason `JobFailed`) has no dump, so its RestoreJob now fails instead of
restoring another run's dump.

Stacked on https://github.com/cozystack/cozystack/pull/5223 and targets
its branch until that merges, only the last three commits belong here.

Checked on a test cluster with mariadb-operator 25.10.2. An operator
`Restore` of an older `Backup` with `targetRecoveryTime` set to its dump
timestamp restored exactly that dump (`obtained target backup` names it
and a row inserted after it is absent from the target), the same
`Restore` without the target restored the newest one. Against the real
bucket and operator `Backup`s the recording probe attributed to each
`Backup` the dump its backup pod logged, the existence check passed for
those dumps and refused a pruned one, legacy `Backup`s with a provable
dump got pinned while back-to-back runs fell back with the warning, and
the two failed runs on that cluster were refused. The controller itself
was not run on the cluster, only its code driven by a test against the
real bucket.

Fixes https://github.com/cozystack/cozystack/issues/5196

### Screenshots

Not a UI change.

### Downstream repositories

The diff is confined to the backup controller's Go code. It changes no
CRD or API type (`driverMetadata` is an opaque map), no chart, values,
package layout or `hack/` script, so no entry of the trigger map
applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(mariadb): a RestoreJob now restores the dump of the Backup it names instead of the newest dump of the application. It fails with a clear message when that dump no longer exists in storage, or when the Backup was made of a failed backup run. Backups taken before this release are restored by the one dump that can be matched to them, or else, as before, by the newest dump, with a BestEffortDump warning event on the RestoreJob.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* MariaDB backups now record the selected dump’s filename and timestamp,
allowing restores to target the corresponding recovery time.
* For older backups without dump metadata, restores use a uniquely
matched dump when possible or fall back to the latest dump with a
warning.
* Restore processing verifies recorded dumps, retries temporary storage
checks, and reports failures for invalid or missing recorded dumps.
* **Bug Fixes**
* Deleting a MariaDB backup now removes its associated dump and operator
backup when ownership is verified and no restore is active. Cleanup
avoids deleting artifacts when it cannot safely identify them.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 20:46:45 +03:00
Aleksei Sviridkin 53078a1d29 feat(observability-proxy): publish the proxy on its own host (#5267)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Adds an optional public endpoint for the observability proxy, so that
clients outside the cluster (kubectl, scripts, third-party tools) can
call it directly with their Kubernetes bearer token. The existing paths
do not deliver the token: kube-apiserver removes the `Authorization`
header after it authenticates a request, so the service proxy of
kube-apiserver cannot carry it, and the console route goes through the
dashboard gatekeeper, which with OIDC enabled answers a client without a
session cookie with 401 and exists only where the dashboard is
published.

The endpoint is off by default. It is published at
`observability-proxy.<publishing.host>` once `observability-proxy` is
added to `publishing.exposedServices`, in the same two modes the
dashboard uses:

- `gateway.enabled: false`: an Ingress on the publishing ingress class,
with a per-host cert-manager certificate, or the operator's wildcard
certificate when `publishing.certificates.wildcardSecretName` is set
(the TLS section stays, so HTTP is still redirected to HTTPS).
- `gateway.enabled: true`: an HTTPRoute on the publishing tenant's
Gateway. The platform now lists `cozy-observability-proxy` in
`gateway.attachedNamespaces` so that the Gateway admits the route. A
cluster that overrides `gateway.attachedNamespaces` replaces the whole
list and has to add the namespace itself.

In Ingress mode the request crosses the publishing tenant's egress
policies, which deny everything except the `allow-to-*` destinations, so
the tenant chart gains `allow-to-observability-proxy`. It matches the
whole `cozy-observability-proxy` namespace, as `allow-to-linstor-gui`
does, because cert-manager runs the HTTP-01 solver pods for the proxy's
certificate there.

The route passes the request through unchanged: no path rewrite and no
authentication at the edge, so `/apis/...` and the bearer token reach
the proxy as sent. The edge limits are set so that they never end a
request before the proxy does:

- The tenant ingress-nginx reads upstream with a 10s timeout by default,
while the proxy allows a non-tail request up to `--request-timeout` (1m)
and keeps a log tail open until it has been silent for
`--tail-idle-timeout`. Before the first byte of a response, the token
and access reviews can add up to another minute. The Ingress sets a 360s
read timeout and keeps response buffering off, so tail lines reach the
client as they arrive.
- The Ingress sets a 1 MiB body limit, the same limit the proxy enforces
on a request body.
- On an HTTPRoute that sets no timeout, Cilium keeps Envoy's 15s route
timeout, which would cut every tail. The route disables it
(`timeouts.request: 0s`), because the proxy limits each request itself.
Envoy's stream idle timeout (Cilium
`envoy.streamIdleTimeoutDurationSeconds`, 300s by default) still applies
and cannot be changed from an HTTPRoute, so the chart now runs the proxy
with `--tail-idle-timeout=4m30s` instead of the 5m default. A tail that
stops sending lines is then ended by the proxy as a complete response,
not reset by Envoy. Before the first byte, slow token and access reviews
together with a log store that never sends headers can still exceed
300s; that case ends in an error either way.

The proxy never reads the client address or `X-Forwarded-*`. All its
limits and its logs use the authenticated user, so nothing changes
behind an ingress and the client cannot spoof anything through headers.

### Security

This adds a new internet-facing entry point when it is enabled. What
protects it:

- Every request to a tenant root needs a bearer token that
kube-apiserver accepts through a TokenReview, and a SubjectAccessReview
for the requested namespace. Paths the proxy does not serve return 404
before any review, and a request without a bearer token returns 401
without a review.
- Per-user limits apply after authentication: request rate, concurrent
requests, concurrent tails per user, and a total cap on open tails.
Every request is bounded by the request timeout, and every tail by the
idle timeout.
- The body is capped at 1 MiB at the edge and in the proxy.
- The new tenant egress policy lets every pod of a tenant open a
connection to the proxy namespace, whether or not the proxy is
published. This gives a tenant workload nothing it cannot already do
through the published host, and the proxy asks for a token in both
cases.

What it does not protect against: an unauthenticated client can make the
proxy send one TokenReview per distinct token it invents. Rejected
tokens are cached by hash for the auth cache TTL, so a repeated token
costs nothing more. This is the same cost a client can already cause by
calling the published Kubernetes API (`api` is in
`publishing.exposedServices` by default). No per-IP rate limit is added
at the edge, because with `publishing.externalIPs` the ingress Service
uses `externalTrafficPolicy: Cluster`, which can hide client addresses
behind node addresses, so a per-IP limit would throttle all clients
together.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

Not a UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#787
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The diff changes `packages/core/platform/values.yaml`, which reaches
cozystack/website:
`content/en/docs/next/operations/configuration/platform-package.md`
lists the possible values of `publishing.exposedServices` and the
default `gateway.attachedNamespaces` list, and both need the new
entries. The observability-proxy documentation is tracked in
cozystack/website#787, which does not cover the public endpoint yet; the
endpoint and the new list entries will be added there once this PR is
accepted. No other repository in the trigger map is reached by this
diff.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(observability-proxy): the observability proxy can be published at observability-proxy.<host> for clients outside the cluster that use a Kubernetes bearer token. It is off by default; add `observability-proxy` to `publishing.exposedServices` to publish it, through an Ingress or, with `gateway.enabled`, an HTTPRoute. Clusters that override `gateway.attachedNamespaces` must add `cozy-observability-proxy` to it for the Gateway mode. Tenants gain an egress policy to the proxy namespace, which the Ingress mode needs.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
  * Tenant workloads can now connect to the observability proxy.
* The observability proxy can be published through an Ingress or Gateway
route when enabled in the service exposure settings.
* Proxy routing supports long-running requests and streams with adjusted
timeout handling.
* Ingress publishing supports per-host or wildcard certificates and
configurable certificate challenge settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 20:41:32 +03:00
Mattia Eleuteri f7ee616f6c fix(linstor): rebase the snapshot restore patch on linstor-csi v1.13.2
The import block of pkg/client/linstor.go changed in v1.13.2, so the
patch no longer applied and the linstor-csi image failed to build.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 17:27:48 +01:00
Nimesh Kulkarni 700600ee3c chore(cilium): drop externalIPs.enabled and nodePort.enabled
The vendored Cilium chart reads neither key: there is no externalIPs
value upstream, and from the nodePort block the templates read only
range, addresses, bindProtection, autoProtectPortRange,
directRoutingDevice, enableHealthCheck, enableHealthCheckLoadBalancerIP
and enableDynamicSourceLookup. Both features follow
kubeProxyReplacement, which stays true, so the keys only misled readers
into thinking they toggle anything.

Fixes #4816

Assisted-by: LLM
Signed-off-by: Nimesh Kulkarni <167794819+GitNimay@users.noreply.github.com>
2026-11-13 21:39:41 +05:30
Aleksei Sviridkin ba5948ed29 Merge branch 'main' into fix/linstor-csi-snapshot-restore-node
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 18:54:57 +03:00
Aleksei Sviridkin f61cfdf0b8 feat(bucket)!: add publicRead to serve a bucket without credentials (#5252)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, security (a fix that comes through a security
advisory), docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

Adds `publicRead` to the Bucket application. A bucket cannot serve its
objects without a signature today: every BucketAccess issues a key, and
those keys grant neither `WriteAcp` nor `PutBucketPolicy`, so `aws s3api
put-bucket-acl --acl public-read` is refused and a tenant has no way to
serve static assets (images, downloads) straight from a bucket. The only
workaround is an operator running `weed shell` `s3.anonymous.set` by
hand, which nothing reconciles and which outlives the bucket.

With `publicRead: true` the chart renders one more BucketAccess,
`<bucket>-public-read`, on a new `-public` BucketAccessClass whose
`accessPolicy` is `public-read`. The class is added in the three places
that define access classes: the Client topology, the per-storage-pool
classes, and the vendored chart (through its existing
`cosi-bucket-class-lock-readonly.patch`). The COSI driver handles the
policy by adding `Read:<bucket>` to the gateway's `anonymous` identity
and, on revoke, removing only that action, so turning the flag off or
deleting the bucket closes it again without touching other public
buckets. Anonymous callers can `GET`/`HEAD` objects at
`<endpoint>/<bucket>/<key>`; listing and writes stay refused.

A user named `public-read` would own the same BucketAccess name, so that
combination fails to render with an explicit message rather than having
two owners fight over one object.

The seaweedfs pool name check now also reserves `public` and the
`-public` suffix, next to the existing `-lock` and `-readonly`: a pool
`<x>-public` would render a readwrite BucketAccessClass with the same
name as pool `<x>`'s public-read class, and a pool `public` would take
the instance's own `-public` class. This is a breaking change for an
instance that already defines a pool with such a name: its seaweedfs
release stops rendering with the pool name error. Pool keys are part of
the volume server and PVC names, so renaming such a pool moves its data
to new volumes; that migration is the operator's to plan before
upgrading.

**Known limitation until the next driver release.** On v0.5.0, revoking
public read (turning `publicRead` off or deleting the bucket) can leave
the bucket readable anonymously if an S3 gateway migrates
`identity.json` while the revoke is in flight: the migrated
`identities/anonymous.json` keeps `Read:<bucket>`, and SeaweedFS loads
it over `identity.json` without ever repairing it. The fix is
seaweedfs/seaweedfs-cosi-driver#22 (the revoke always writes
`anonymous.json`, after the legacy file). The driver release carrying it
will be pinned here if it lands before this PR merges, otherwise in a
follow-up. Until then, an operator can check `s3.anonymous.list` in
`weed shell` after a revoke and clear a leftover with `s3.anonymous.set
-bucket <bucket> -access none`.

**Driver: pins the upstream seaweedfs-cosi-driver v0.5.0.** The
`public-read` policy merged upstream in
seaweedfs/seaweedfs-cosi-driver#21 and v0.5.0 is tagged on that merge,
so this PR moves the image pin from v0.4.0 (#5284) to v0.5.0 in
`packages/system/seaweedfs/values.yaml`, the client-mode `.tag` file and
`cosi_image_test.yaml`. An earlier revision of this branch built the
driver in-tree while no release carried the policy; that build is gone,
and the intermediate commits that added it are undone by the merge of
`main`, so squashing leaves only the feature and the pin.

v0.5.0 differs from v0.4.0 only by the `public-read` policy
(seaweedfs/seaweedfs-cosi-driver#21) and its README note. Upstream
documents one `public-read` BucketAccess per bucket as the supported
shape, which is what the chart renders.

Tested: new helm-unittest suites for the bucket app (default off, class
name with and without storage pool, locked bucket, coexistence with
users, the name collision), the seaweedfs extra chart (Client and
storage-pool classes) and the vendored chart; they fail when the class
suffix or the policy is removed. `helm unittest` passes on
`packages/apps/bucket`, `packages/extra/seaweedfs`,
`packages/system/seaweedfs` and `packages/system/bucket`, as do
`test-guard-fail-closed` and `test-image-version`;
`cosi_image_test.yaml` pins
`ghcr.io/seaweedfs/seaweedfs-cosi-driver:v0.5.0`, which is published for
amd64 and arm64. The anonymous-read mechanism itself was checked by hand
on SeaweedFS 4.45 (`weed shell` `s3.anonymous.set -access Read`):
anonymous `GET` returns 200 on both gateways,
`ListObjects`/`ListObjectsV2` and `PUT` return 403, and the identity
survives grants and revokes of regular BucketAccesses. The end-to-end
path through the new driver has not run on a cluster yet.

### Screenshots

No UI change.

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [x]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up: cozystack/terraform-provider-cozystack#80
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Walked the trigger map against the diff: the only hit is the new field
in `packages/apps/bucket/values.schema.json` (and
`api/apps/v1alpha1/bucket/types.go`), which the Terraform provider
models by hand. Follow-up filed as
cozystack/terraform-provider-cozystack#80. The website's bucket page is
hand-written and could mention the flag, but no listed trigger covers an
added field, so it is left unticked.

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
feat(bucket): add `publicRead` to serve a bucket's objects without credentials at `<endpoint>/<bucket>/<key>`. Anonymous callers can read objects but cannot list the bucket or write to it. The COSI driver is bumped to seaweedfs-cosi-driver v0.5.0, which adds the `public-read` access policy. The seaweedfs pool names `public` and `*-public` are now reserved: an instance that already defines such a pool fails to render until the pool is renamed, which moves its data to new volumes.
```





<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Bucket configurations can enable anonymous reads of objects through
the bucket endpoint. Anonymous access does not allow listing buckets or
writing objects.
* SeaweedFS provides public-read access classes across supported bucket
configurations, alongside read-only and read-write access.
* Added a retained bucket class with object locking and a 365-day
compliance retention period.
* **Documentation**
* Added guidance for the `publicRead` setting, which is disabled by
default.
* **Updates**
* Updated the SeaweedFS COSI driver to v0.5.0. Pool names `public` and
names ending in `-public` are reserved.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:47:57 +03:00
Aleksei Sviridkin 29e8522f4b fix(cozystack-basics): refuse a direct delete of a tenant namespace (#5011)
## What this PR does

Fixes #5002. A DELETE of a tenant namespace is refused unless it comes
from the identities that remove it on the normal path, and the denial
says to delete the Tenant instead.

The new ValidatingAdmissionPolicy in cozystack-basics matches DELETE of
core namespaces, and its binding selects namespaces labelled
`tenant.cozystack.io/tenant-root`. The tenant chart puts that label on
every tenant namespace below tenant-root (`tenant.ancestorTenantLabels`,
and migration 49 backfilled older ones). tenant-root itself has no such
label and stays with `cozystack-no-delete-guardrail`. The `tenant-*`
probe namespaces of the e2e suites carry only
`namespace.cozystack.io/host` and are not selected either.

Two groups pass. `system:serviceaccounts:cozy-fluxcd`, because
helm-controller and its shards run as `flux` there, and a Tenant
uninstall is helm-controller deleting the namespace after the pre-delete
hook. `system:serviceaccounts:kube-system`, for the garbage collector
following the ownerReference of a child namespace once its parent is
gone. The namespace controller never sends a DELETE for a namespace, it
updates `finalize`, so it is not affected
(`pkg/controller/namespace/deletion` and
`pkg/registry/core/namespace/storage` at v1.33.12 and v1.34.1).
`system:masters` is not exempt, the same as with the no-delete
guardrail.

This is the denial kube-apiserver 1.35 returns.

```
namespaces "tenant-foo" is forbidden: ValidatingAdmissionPolicy 'cozystack-tenant-namespace-delete-policy' with binding 'cozystack-tenant-namespace-delete-policy' denied request: Namespace tenant-foo belongs to the Tenant foo in tenant-root. Delete the Tenant instead with kubectl delete tenants.apps.cozystack.io foo --namespace tenant-root. Deleting the namespace directly can remove the tenant etcd before the Kubernetes clusters that use it and leave the namespace stuck in Terminating. If the Tenant is already gone, remove the label first with kubectl label namespace tenant-foo tenant.cozystack.io/tenant-root-
```

There are a few limits.

- The bypass is removing the label, like with the guardrail. That also
takes the namespace out of the tenant-root egress policy and the
dashboard tree, which only matters for a namespace that is about to go.
- Migration 49 labelled every `tenant-*` namespace that existed when it
ran, so a non-tenant namespace with such a name is refused too and needs
the same label removal.
- Because the garbage collector is exempt, a parent namespace whose
label was removed and that is deleted with foreground propagation still
cascades into its children directly.
- A DELETE of a namespace that is already Terminating is answered before
admission, so the policy does not see it. It changes nothing either.
- A cluster that made helm-controller impersonate another account
(`--default-service-account`, or `spec.serviceAccountName` on Tenant
HelmReleases) would need that account allowed. Cozystack sets neither.
- It does not change how Kamaji cleans up once the order is broken,
which is the second idea in #5002.

The helper rename in `external_tls_policy_test.go` (`loadPolicy(t,
path)` and `compilePolicy`) is byte for byte the one in #2622, so the
two merge in either order.

Checked locally.

- `go test ./pkg/registry/apps/application/ -run
'TestTenantNamespaceDeletePolicy|TestDatabaseExternalTLSPolicy' -count=1
-v` passes 4 tests, the new `TestTenantNamespaceDeletePolicy_Decisions`
with 11 cases. `go test ./pkg/registry/... ./pkg/config/...
./pkg/cmd/server/...` passes in all 11 packages that have tests.
- `helm unittest` in `packages/system/cozystack-basics` with
helm-unittest 1.1.2 in docker passes 12 suites and 70 tests, against 11
and 66 on main. The new tests on main without the template fail, both Go
tests and 4 helm tests in 2 suites.
- 12 mutations of the template each fail a test. They exempt
`system:masters`, drop either exempt group, select on the no-delete
label, add UPDATE, always name tenant-root as the parent, drop the
capability gate, label tenant-root, switch to Warn, switch failurePolicy
to Ignore, drop the bypass from the message and drop the objectSelector.
- The rendered policy, the guardrail and namespaces rendered from the
tenant chart were applied to kube-apiserver v1.33.0, v1.35.0 and v1.36.2
from setup-envtest, 16 checks on each. Real and dry-run deletes by a
cluster admin, by an OIDC user bound to cluster-admin and by a
cozy-system and a tenant ServiceAccount are refused with 403. The flux
and garbage collector identities get their delete accepted. tenant-root
gets the guardrail's denial, and with the guardrail binding removed a
dry-run delete of it goes through, so this policy does not select it. A
`tenant-*` namespace without the label and a tenant namespace after its
label is removed are accepted for deletion. With this binding removed, a
dry-run delete by the cluster admin goes through. That harness is not
part of the PR.
- `hack/bats-no-negated-assert.bats`, `hack/bats-no-exit-trap.bats`,
`hack/bats-runner-coverage.bats`, `hack/helmrelease-readiness.bats` and
`hack/route-hostname-policy-version-coverage.bats` pass under
`hack/cozytest.sh`.

The new e2e case in `hack/e2e-install-cozystack.bats` creates an empty
Tenant, checks that a server dry-run delete of its namespace is refused
with the message, deletes the Tenant and waits for the namespace to go.
It was not run here, there is no Cozystack cluster at hand. Its kubectl
steps were run against the envtest apiserver 1.35, the selector and
field selector read, the refused dry-run with the exact strings it greps
for, and `kubectl wait --for=delete` on a namespace that is gone. The
Chainsaw suites that delete Tenants, the gateway inheritance ones among
them, go through the uninstall path as well.

The release-1.6 backport is #5012.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

The trigger map was walked against the diff. It adds a policy and tests
to cozystack-basics and an e2e case, and no entry covers that. The
website docs do not tell users to delete a tenant namespace directly,
and terraform-provider-cozystack deletes Tenants through the apps API.

### Release note

```release-note
fix(cozystack-basics): refuse a direct DELETE of a tenant namespace unless it comes from helm-controller or the garbage collector, and point to deleting the Tenant instead, so a tenant namespace deleted by hand no longer gets stuck Terminating on Kamaji finalizers
```
2026-11-13 18:25:04 +03:00
Aleksei Sviridkin fa09cdc184 fix(keycloak, foundationdb): apply operator CRD updates on upgrade (#4946)
## What this PR does

The generated HelmReleases leave `spec.upgrade.crds` unset unless the
PackageSource sets `upgradeCRDs`, so helm-controller never replaces a
CRD that a chart ships in `crds/`. keycloak-operator and
foundationdb-operator ship their CRDs that way and set no policy. On a
v1.6.4 cluster that came through earlier releases, the CRDs of both
operators were older than the ones in the v1.6.4 charts. The v1.25.0
ClusterKeycloakRealm schema has no `spec.sessions`, `spec.login` or
`spec.organizationsEnabled`, so the apiserver dropped the session
settings keycloak-configure sets and the operator never applied them to
the realm. Keycloak also rejected the dashboard KeycloakClient with HTTP
400 on that cluster, and replacing the 17 CRDs from the charts and
restarting keycloak-operator cleared it.

This opts both PackageSources into `upgradeCRDs: CreateReplace`, the
policy mongodb-operator, redis-operator and velero already use, and pins
it with a platform test for each. The keycloak-operator change is not
purely additive. v1.32.0 drops the `realm`, `targetRealm` and
`keycloakOwner` fields that v1.25.0 marked deprecated and requires
`realmRef` and `keycloakRef` instead. Every Keycloak object the platform
renders already uses the refs, and each of the 14 CRDs keeps its single
version, so the replace loses nothing the platform relies on and drops
no stored version. A hand-made object that still uses a deprecated field
would lose it on its next write, and that write needs the ref. The
foundationdb-operator CRDs only add fields from v2.13.0 to v2.30.0, plus
a pattern on `perpetual_storage_wiggle_locality`, which the platform
does not set.

helm-controller applies the policy only when it upgrades a release, and
it upgrades only when the release artifact or its values change. A
PackageSource edit changes neither. The PR therefore also gives the
keycloak-operator package a helm-unittest suite, which pins the realm
fields keycloak-configure sets against the ClusterKeycloakRealm CRD the
chart ships and changes the keycloak-operator artifact, so the CRDs are
replaced on the upgrade that brings this change even where nothing else
in the package changes, as on a backport to release-1.6. The platform
release writes the new PackageSource in the same apply that moves
`cozystack-packages` to the new revision, so the policy should reach the
HelmRelease before the new keycloak-operator artifact does. The
foundationdb-operator package does not change here, so its CRDs are
replaced at the next change to that package, and main already carries
one since v1.6.5.

Objects written while the CRDs were stale get the pruned fields back
only when their chart applies them again. For the cozy realm that is the
next keycloak-configure upgrade. On main it comes with the same platform
upgrade, since keycloak-configure changed after v1.6.5, and its
HelmRelease depends on the keycloak-operator one through the oidc
variant. On a branch where keycloak-configure does not change, the realm
keeps its current session limits until it does or until that release is
forced to upgrade.

#4901 adds the same `upgradeCRDs: CreateReplace` to the
foundationdb-operator PackageSource as part of the FoundationDB backup
work, and it changes the foundationdb-operator package as well. The
foundationdb commit here is separate from the keycloak one and does not
depend on it, so it can be dropped if #4901 lands first.

The other packages that ship CRDs only in `crds/` and set no policy are
bootbox, clickhouse-operator, clustersecret-operator,
cozystack-scheduler, external-dns, grafana-operator, kafka-operator and
snapshot-controller. None of their CRDs changed since v1.0.0, or since
the package was added, apart from description text in external-dns, so
no cluster is behind on them today. kafka-operator gets CreateReplace
with the Strimzi move in #4900. gpu-operator replaces its CRDs with its
own pre-upgrade Job, and the remaining packages render their CRDs from
templates or leave them to an operator. Each of the eight will need the
same review and opt-in the next time its CRDs change.

`make -C packages/core/platform test` passes with 58 suites and 287
tests, and `make -C packages/system/keycloak-operator test` passes. With
either PackageSource reverted to main, its suite fails with `unknown
path spec.variants[0].components[0].install.upgradeCRDs`. With the
v1.25.0 ClusterKeycloakRealm CRD in the chart, the keycloak-operator
suite fails all four assertions. The full `hack/helm-unit-tests.sh`
sweep passes.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Two PackageSources under `packages/core/platform/sources/` gain an
install option, three test suites are new, and
`packages/system/keycloak-operator/Makefile` gains the same `test`
target other packages have. Platform `values.yaml`, variants, bundles,
CRDs, `hack/` and the installer are untouched. The website's
`development.md` does not describe per-package test targets, so the
Makefile change needs nothing there, and no website page documents a
manual CRD replace for these operators.

### Release note

```release-note
fix(keycloak, foundationdb): keycloak-operator and foundationdb-operator now replace their CRDs on upgrade. Clusters upgraded from before v1.2.0 kept the v1.25.0 keycloak-operator CRDs, under which the apiserver dropped the Keycloak realm session settings and Keycloak could reject the dashboard client, and clusters upgraded from before v1.6.0 kept the v2.13.0 foundationdb-operator CRDs.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Existing FoundationDB and Keycloak installations can now upgrade their
custom resource definitions with replacement behavior, including updates
to schemas and removal of deprecated fields.
* **Tests**
* Added checks for the operator upgrade settings and Keycloak CRD schema
fields.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:24:45 +03:00
Aleksei Sviridkin 51282a1243 fix(piraeus-operator): keep the LINSTOR error-report alerts quiet when the controller pod moves (#4963)
## What this PR does

`linstorSatelliteErrorRate` fires for every node that has any old
LINSTOR error report whenever the LINSTOR controller pod moves to
another node, and it keeps firing for 15 minutes.
`linstor_error_reports_count` is exported by the controller, so its
series carry the controller pod's `instance` and `controller_node`, and
a moved pod starts new series that hold the existing counts.
VictoriaMetrics'
[`increase()`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase)
starts from the last sample before the window, and when a series has
none,
[`rollupDelta`](https://github.com/VictoriaMetrics/VictoriaMetrics/blob/v1.136.0/app/vmselect/promql/rollup.go#L1859-L1901)
counts it from zero whenever its first value is below ten times the next
change plus one. That is the case the docs describe as `increase`
ignoring "the first value in a series if it is too big" under
[`increase_pure`](https://docs.victoriametrics.com/victoriametrics/metricsql/#increase_pure),
and report counts are never that big, so every new series counts all of
its reports as new. The `up` guard from #2265 matches `on(instance,
job)`, and the new controller is up, so it lets them through.

Seen on a v1.6.5 cluster with VictoriaMetrics v1.136.0. During the
upgrade the controller pod moved to another node, and the alert fired
for all three nodes with values 1, 1 and 2 for 15 minutes, while
`linstor error-reports list --since 1h` was empty and the four reports
that exist were weeks old. The day before, it fired the same way for
about 15 minutes when only the `cluster` external label of vmagent
changed.

A satellite coming back to the metric does the same. In LINSTOR v1.33.3
the controller asks every satellite for its reports on each scrape and
leaves out any satellite that is not connected
([`prepareErrRequestApi`](https://github.com/LINBIT/linstor-server/blob/v1.33.3/controller/src/main/java/com/linbit/linstor/core/apicallhandler/controller/CtrlErrorListApiCallHandler.java#L240-L259)),
so after a node reboot or any other disconnect longer than five minutes
the satellite's old count is counted again.

Both rules now take the count per `hostname`, hold the last value for an
hour with `last_over_time`, and run `increase()` over that as a
one-minute
[subquery](https://docs.victoriametrics.com/victoriametrics/metricsql/#subqueries),
whose inner points VictoriaMetrics aligns to that step. Keyed on
`hostname`, a moved controller or a changed label continues the same
series, and the hold bridges a disconnect or a slow controller start of
up to an hour. The `up` guard matches `on()`, since the aggregated
result has no `instance` any more. A new report still fires with the
same value and stays for 15 minutes, at most a minute later than before
because of the one-minute step. The alerts now carry `hostname` and
`severity`, and no longer the controller's `instance`, `job` and
`controller_node`, the `module`, or the external labels of the scrape.

`linstorControllerErrorRate` gets the same form. Its `hostname` is the
controller pod name and the piraeus controller keeps its reports in an
[`emptyDir`](https://github.com/piraeusdatastore/piraeus-operator/blob/v2.10.2/pkg/resources/cluster/controller/controller-deployment.yaml#L111-L112),
so a new pod starts with no reports and a move alone does not trip it,
but a label change does whenever the controller has reports.

A gap longer than the hour is not covered. After it the series starts
over, and as before the `up` guard holds the alert back only if the
controller target recorded failed scrapes during the gap.

I checked it against the stored metrics of the cluster above with
read-only queries to vmselect. Over the upgrade the current expression
returns the three nodes from 15:45:30 to 16:00:00 UTC at a 30-second
step, the same interval as vmalert's `ALERTS` series, and over the label
change from 20:27:30 to 20:42:00. The new expression returns nothing
over both windows, and without `> 0` it returns each of the three nodes
with 0 at every step, 88 and 121 points. The same storage has six gaps
of 8 to 113 minutes over one night with no failed scrape recorded in
them. Replaying vmalert's instant query of the `increase()` part every
30 seconds in the 154 evaluations after those gaps, the current
expression returns at least one node in 99, the `hostname` key without
the hold in 84, and the new expression in 30, all of them after the
113-minute gap. Retention there does not reach back to a real new
report, so the firing side is covered by the tests.

`hack/linstor-error-report-alerts.bats` replays both rules through
`vmalert-tool` v1.136.0, pinned by digest, which is the VictoriaMetrics
release victoria-metrics-operator v0.68.4 [deploys by
default](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38),
with 30-second samples and 15-second evaluations like the platform's
vmagent and vmalert. The suite needs docker, like the migration suites,
and only the image pull touches the network.
`hack/testdata/linstor-error-report-alerts/unittest.yaml` has 20 cases
in five groups, which cover a controller move followed by a new report,
a changed external label for both rules, one satellite away for 40
minutes and one that comes back with a new report, a new report and a
first report on a node for both rules, and a controller that cannot be
scraped for 70 minutes. Against the rule on main 13 cases fail, 9 of
them because old reports fire again after the move, the label change or
the disconnect, and the other 4 on the label set alone, since the
current alerts carry the controller's labels. Each of seven mutations of
the new expressions turns at least one case red, namely no hold, a
15-minute hold, the guard on `instance`, no guard, the key on `hostname`
and `instance`, the key on `hostname` and `cluster`, and the controller
rule as on main. With Bats 1.14.0 at the commit CI pins, the new suite
with `alert-severity-contract`, `bats-strict-setup`,
`bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`
and `image-pin-consistency` passes 74 of 74, and `helm template` renders
the same 15 rules as main. I have not run the new rules in vmalert on a
live cluster.

The commit cherry-picks onto release-1.7 as is, and the suite passes
there. On release-1.6 it applies without conflicts, but that branch runs
`hack/*.bats` through `hack/cozytest.sh` and has no
`hack/test_helper.bash`, so the suite there has to drop its `load
test_helper` line.

### Screenshots

Not a UI change.

### Downstream repositories

A rule file under `packages/system/piraeus-operator/alerts/`, a new Bats
suite under `hack/` and its test data. No values, schema, README, image,
make target or `ApplicationDefinition` change, and the website does not
mention these rules or the metric, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(piraeus-operator): the LINSTOR error-report alerts no longer fire on old reports when the LINSTOR controller pod moves to another node, when an external label of the scrape changes, or when a satellite reconnects within an hour. They now count reports per node, and their alerts carry the node's hostname instead of the controller pod's instance, job and controller_node labels.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved LINSTOR controller and satellite error alerts to account for
recent error reports over a 15-minute window, even when metrics are
intermittent or scrape labels change.
* Updated alert matching to better handle controller moves between nodes
and satellites that disappear from metrics while disconnected.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:18:47 +03:00
Aleksei Sviridkin 868b97dbe6 fix(kubevirt): let tenant admins upload a VMDisk image with virtctl (#5114)
## What this PR does

A VMDisk created with `source: upload` waits in `UploadReady` for an
image, but no tenant role can fill it. `virtctl image-upload` reads the
disk PVC and CDI's PVC Prime and requests an upload token, and the
tenant roles grant neither `get` on persistentvolumeclaims nor `create`
on `upload.cdi.kubevirt.io/uploadtokenrequests`, so the command stops on
a forbidden PVC read.

`cozy:tenant:admin:base` now grants both. Tenant admins and super-admins
get them in their own namespaces through the existing RoleBindings, the
use and view tiers do not. Reading the DataVolume is already granted for
the disk's own name by the vm-disk chart's dashboard role, so no
DataVolume rule is added. The vm-disk notes print the command with
`--no-create`, because the chart has already created the DataVolume, and
the security self-assessment and threat model list the new grants.

This is the RBAC part of #3847, split out so tenants can upload while
the console upload panel and the upload proxy certificate stay in review
there. The rules, their tests, the notes change and the docs lines are
from @myasnikovdaniil, who is co-author on the commit.

Checked live on a cluster with KubeVirt v1.8.4 and CDI v1.66.1, with
virtctl v1.8.4 impersonating a user in the `<tenant>-admin` group.

- With the stock roles `virtctl image-upload dv vm-disk-<name>
--no-create` fails with `persistentvolumeclaims "vm-disk-<name>" is
forbidden`.
- With exactly these two rules added as a temporary Role, the same
command reads `prime-<uid>`, uploads a 16 MiB image and the DataVolume
reaches `Succeeded`. `get` on the DataVolume came from the vm-disk
dashboard role.
- The upload went through a port-forward to `cdi-uploadproxy` with
`--insecure`. A trusted certificate on the published proxy is the other
half of #3847 and is not changed here.

Unit tests with `helm unittest` 1.1.2, the version CI pins.
`packages/system/cozystack-basics` passes 72 of 72, against 66 on main,
and `packages/apps/vm-disk` 117 of 117, against 116. Removing the upload
token rule fails one case, and dropping `--no-create` from the notes
fails one case.

Fixes #3759

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. Two tenant RBAC rules, their tests, the vm-disk notes and two docs
files. No values key, schema, default, README or package change, so no
trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubevirt): tenant admins can upload an image into a VMDisk created with source upload using virtctl image-upload. The tenant admin and super-admin roles get get on persistentvolumeclaims and create on upload.cdi.kubevirt.io/uploadtokenrequests, and the vm-disk notes print the command with --no-create.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
  * Tenant administrators can now upload images into existing VM disks.
* **Improvements**
* The displayed upload command now explicitly targets the release
namespace and avoids creating a new disk.
* **Documentation**
* Security guidance now clarifies the access needed for uploads and
notes that persistent-volume permissions remain limited by namespace
scope.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 18:12:46 +03:00
Aleksei Sviridkin df42104b13 fix(seaweedfs, cozy-lib): cap the Go heap so components survive their memory limit (#4234)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Every SeaweedFS component is a Go binary running under a cgroup memory
limit, and none of them set `GOMEMLIMIT`. The collector then works from
`GOGC` alone, which is a target relative to the live heap with no
absolute ceiling, so a burst of allocation grows the heap past the limit
and the kernel kills the container.

This is not a leak. On the cluster where it showed up, the S3 gateway
held 6.4 GiB of `heap_sys` against an 8 GiB limit while `heap_inuse` was
389 MiB: the container was killed over memory the runtime was holding
and not using. Ten kills in seven days, each one an interruption of the
object store every tenant writes its backups to. Raising the limit does
not help, because nothing bounds the growth; it only moves the ceiling
the heap climbs to.

This PR adds a `cozy-lib.resources.goMemLimit` helper next to the
existing `javaHeap` one and wires it into the master, volume, filer and
s3 components and into every volume pool and zone, each sized from the
memory limit already computed for it. Pools need their own value because
the subchart merges `volume` into every entry of `volumes`: without one,
a `t1.nano` pool would inherit the value sized for the default volume's
512Mi limit, above its own 128Mi. The value is 75% of the limit, the
same fraction `javaHeap` uses for the JVM; the remaining quarter covers
what `GOMEMLIMIT` does not account for, goroutine stacks and mmap'd
regions among them.

The lifecycle admin and worker run the same binary under limits the
parent chart does not set (256Mi and 512Mi in
`packages/system/seaweedfs/values.yaml`), so their value is pinned next
to those limits at the same 75%: 192MiB and 384MiB.

A limit under 4/3 Mi (1398102 bytes) floors to `0MiB`, which Go accepts
and then collects almost continuously, so the helper fails the render
instead.

The upstream chart already supports `extraEnvironmentVars` on every one
of these components, so nothing had to change there. No values key is
added, so this needs no configuration and changes no public schema.

Tested with `helm unittest`: a suite on the seaweedfs package asserts
the value tracks the memory limit rather than a fixed string, for the
default volume, Simple pools, MultiZone zones and zone pools; the system
seaweedfs suite asserts the admin and worker env; and a helper suite in
`cozy-lib-tests` pins the unit grammar and the rejection of a zero
value. Reverting each fix makes its suite fail.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(seaweedfs): set GOMEMLIMIT on master, volume (and each pool and zone), filer, s3 and the lifecycle admin and worker so the Go runtime collects before the container hits its memory limit instead of being OOM-killed
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* SeaweedFS master, volume, pool, zonal volume, filer, and S3 workloads
now receive a Go memory limit set to 75% of their configured memory
limit.
* Pool workloads use their own resource settings when provided, falling
back to the default volume preset when needed.
* SeaweedFS admin and worker components now have Go memory limits
aligned with their configured memory limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 17:53:25 +03:00
Aleksei Sviridkin 4fa6bfe30b feat(monitoring): alert when pods of one workload share a node (#4993)
## What this PR does

On three 3-node clusters running v1.6.4 and v1.6.5 we found a tenant
etcd with two of its three members on one node, a tenant Redis with two
of its three Sentinels on the node that also ran the master, and several
VictoriaMetrics and VictoriaLogs pairs (vmstorage, vmselect,
vmalertmanager, vlstorage) sharing a node. Nothing alerted. Losing one
of those nodes would have cost the etcd or Sentinel quorum, or both
copies of a storage tier. Spread constraints keep new pods apart once
they are in place (#4986 for etcd, #4989 for monitoring, #4990 for
Redis), but a placement that already exists, or that comes from a soft
rule or a drain, stays as it is, because the scheduler does not move
running pods.

This adds `PodReplicasColocated`, a warning in a new
`workload-placement` group under
`packages/system/monitoring-agents/alerts/`. It fires when two or more
running pods of one StatefulSet, ReplicaSet, CNPG `Cluster` or
`StrimziPodSet` have been on one node for an hour while the cluster has
at least as many schedulable nodes as the workload has running pods, so
five replicas on three nodes stay quiet. The alert carries `cluster`,
`namespace`, `owner_kind`, `owner_name` and `node`, plus the `severity`,
`exported_instance` and `service` labels the other platform alerts set,
and its value is the number of pods on that node. DaemonSets, Jobs and
every other owner kind are not in the list, so a DaemonSet surge pod
next to its predecessor does not count.

etcd-operator gives every member pod its own `EtcdMember` as controller
owner
([etcdmember_controller.go#L546](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L546)),
so grouping by owner would see one pod per group. Members get their
names from `GenerateName: <cluster>-`
([etcdcluster_controller.go#L434](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L434),
[#L839](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L839)),
and members adopted from the old StatefulSet keep names like `etcd-0`
([adopt.go#L103](https://github.com/cozystack/etcd-operator/blob/v0.5.6/internal/migrate/adopt.go#L103)).
The rule drops the last dash-separated part of the member name and
reports the group as `EtcdCluster <name>`. With the etcd package that is
one group per tenant namespace, and two etcd clusters in one namespace
would still be told apart. CNPG instance pods carry a controller
reference to their `Cluster`
([cluster_create.go#L1580](https://github.com/cloudnative-pg/cloudnative-pg/blob/v1.30.0/internal/controller/cluster_create.go#L1580)),
and Strimzi pods one to their `StrimziPodSet`
([StrimziPodSetController.java#L369](https://github.com/strimzi/strimzi-kafka-operator/blob/0.45.1-rc1/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/StrimziPodSetController.java#L369)).

Owner and node both come from `kube_pod_info`. kube-state-metrics fills
`created_by_kind` and `created_by_name` from the controller owner
reference and `node` from `spec.nodeName`
([pod.go#L638-L653](https://github.com/kubernetes/kube-state-metrics/blob/v2.15.0/internal/store/pod.go#L638-L653)).
`kube_pod_owner` and `kube_pod_status_phase` have no node label of their
own, and the kube-state-metrics scrape adds `node` from
`__meta_kubernetes_pod_node_name` with `honorLabels: true`
(`templates/kube-state-metrics-scrape.yaml`), so on those series `node`
is the node of the kube-state-metrics pod. The test data keeps that
label on the phase series. A pod counts while
`kube_pod_status_phase{phase="Running"}` is 1, so a Failed or Pending
pod on the same node does not. `kube_pod_labels` does carry every pod
label here (`--metric-labels-allowlist=pods=[*]`), but the rule does not
depend on it.

A node counts as schedulable unless it has a `NoSchedule` or `NoExecute`
taint. A cordon shows up as the
`node.kubernetes.io/unschedulable:NoSchedule` taint
([node_lifecycle_controller.go#L549-L555](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L549-L555)),
and not ready and unreachable nodes get theirs from their conditions
([#L87-L99](https://github.com/kubernetes/kubernetes/blob/v1.33.12/pkg/controller/nodelifecycle/node_lifecycle_controller.go#L87-L99)),
so one clause covers all three, and a tainted control plane does not
count either. Pods that a drain had to put together on the remaining
nodes therefore stay quiet while the node is out, and the hour starts
once it is back.

The alert waits an hour. A replacement pod runs next to its terminating
predecessor only for the predecessor's grace period, and node
maintenance takes the drained node out of the count, while a placement
that has held for an hour does not fix itself. The alert is about a risk
rather than an outage, so the wait costs nothing. The VMAlert sets both
`remoteWrite` and `remoteRead`, so the pending state survives a vmalert
restart ([vmalert
docs](https://docs.victoriametrics.com/victoriametrics/vmalert/#alerts-state-on-restarts)).

Not covered are workloads that run one pod per owner, such as ClickHouse
with one StatefulSet per replica
([patterns.go#L45-L53](https://github.com/Altinity/clickhouse-operator/blob/release-0.25.2/pkg/model/chi/namer/patterns.go#L45-L53))
and the KubeVirt VMs behind a tenant Kubernetes node group. They would
need grouping by pod labels, which this rule avoids. The node count is
cluster-wide, so a workload that a node selector or affinity restricts
to fewer nodes than it has pods fires too.

`hack/workload-placement-alerts.bats` takes `.spec` of the VMRule and
replays it through `vmalert-tool` v1.136.0 pinned by digest, the release
vm-operator v0.68.4 deploys by default
([config.go#L38](https://github.com/VictoriaMetrics/operator/blob/v0.68.4/internal/config/config.go#L38))
and the same image #4963 and #4985 use.
`hack/testdata/workload-placement-alerts/unittest.yaml` has 13 cases
with 24 checks. Two of three StatefulSet pods on one node fire at 61
minutes and not at 59, with labels and annotations pinned. Spread pods,
five ReplicaSet pods on three nodes, a DaemonSet surge pod, two pods of
one Job, a Failed pod next to a running one, and one pod each in two
clusters that share workload and node names stay quiet. etcd members
fire as one `EtcdCluster` per namespace, for generated and adopted
member names. A replacement pod next to its terminating predecessor for
twenty minutes stays quiet, and the same overlap held past the hour
fires and resolves when it ends. A cordoned node keeps the alert quiet
until it is uncordoned, and a node with unreachable taints keeps it
quiet throughout. ReplicaSet, CNPG `Cluster` and `StrimziPodSet` pods
fire with their owner kind.

```console
$ bats hack/workload-placement-alerts.bats
1..1
ok 1 workload placement alerts pass their vmalert-tool cases
```

With the expression replaced by one that never fires, the suite prints
`not ok 1` and vmalert-tool reports 5 of the 24 checks, which are all
the checks that expect an alert. On main, where the rule file does not
exist, the suite fails as well. Each of these changes to the rule fails
at least one check, with the number of failing checks in brackets. `for:
15m` (5), no phase filter (1), no node count (6), no `cluster` label
(6), DaemonSet and Job in the kind list (3), no taint clause (4), `<`
instead of `<=` (5), etcd members grouped per member (1).

With Bats 1.14.0 at the commit CI pins, the suite passes together with
`alert-severity-contract`, `bats-strict-setup`,
`bats-no-negated-assert`, `bats-no-exit-trap`, `bats-runner-coverage`,
`bats-runner-contract`, `image-pin-consistency` and
`check-commit-trailers`, 109 of 109, and the severity contract lists the
new alert as `warning`. helm-unittest 1.1.2, the version CI pins, passes
the monitoring-agents chart with 19 of 19 tests in 6 suites, and `helm
template` with Helm 3.17.3 renders the VMRule the same as the file. Like
the migration suites the new suite needs docker, and only the image pull
touches the network. I have not run the rule in vmalert on a live
cluster.

On release-1.6 kube-state-metrics 2.15.0, its scrape config, the alerts
template, the VMAlert and vm-operator v0.68.4 are the same, and
etcd-operator v0.5.5 and CNPG 1.28.2 set the same owners and member
names. The new files apply there without changes. That backport is a
separate PR with one more commit that drops the `load test_helper` line,
because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`.

### Screenshots

Not a UI change.

### Downstream repositories

A new alert file under `packages/system/monitoring-agents/alerts/`, a
Bats suite under `hack/` and its test data. No values, schema, README,
image or `ApplicationDefinition` change, and the website does not list
platform alerts by name, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(monitoring): new warning alert PodReplicasColocated fires when two or more running pods of one StatefulSet, ReplicaSet, CNPG Cluster, StrimziPodSet or etcd cluster have shared a node for an hour while the cluster has a schedulable node for each of them.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added monitoring for workloads whose running pod replicas are
colocated on the same node.
* Alerts include workload, node, cluster, and severity details, along
with quorum-loss guidance.
* Coverage includes StatefulSets, ReplicaSets, Clusters, StrimziPodSets,
and EtcdMembers.

* **Tests**
* Added validation for workload-placement alerts across scheduling,
availability, cluster-isolation, and pod-replacement scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 17:38:58 +03:00
Aleksei Sviridkin 5fe356f57e fix(postgres): accept a standard five-field backup schedule (#5119)
## What this PR does

The postgres app hands `backup.schedule` to a CloudNativePG
`ScheduledBackup`, and the field asks for the six-field CloudNativePG
cron with leading seconds. The mongodb app takes a standard five-field
cron in the same field, and so do backup Plans and the Plan form in the
console, so tenants copy a value from one app into the other and support
has to explain the difference.

CloudNativePG does not reject a five-field value. CloudNativePG 1.30.0,
which main pins, parses the schedule with `cron.Parse` from robfig/cron
v1.2.0 in both the ScheduledBackup webhook and the controller. That
parser takes five or six fields, reads the first one as seconds and
makes day-of-week optional, and the webhook only returns a warning when
the count is not six. CloudNativePG's own webhook test "warn the user if
the schedule has a wrong number of arguments" covers this with `1 2 3 4
5`. So `0 2 * * *`, meant as every day at 02:00, fires every hour at
minute 2, and `5 * * * *` fires every minute.

The chart now counts the fields. A five-field value gets a leading `0`
seconds field, so `0 2 * * *` renders `0 0 2 * * *`. A six-field value
and a descriptor such as `@daily` or `@every 6h` render exactly as
before. Any other field count fails the render with a message that shows
both forms, where before it reached the webhook and was rejected there.
An empty schedule still renders no ScheduledBackup. The field
description that the dashboard form shows and the README now name both
forms.

One thing changes on upgrade. A release that already sets a five-field
value gets the standard meaning of it, so `0 2 * * *` moves from hourly
to daily. Before this change such a value was outside the documented
format and drew the webhook warning.

`packages/system/keycloak` renders its own CloudNativePG ScheduledBackup
from its `backup.schedule`, with a six-field default and a values
comment that says so. It is a platform component with its own template,
so it is not changed here. The other apps with a backup schedule
(mongodb, mariadb, clickhouse) and backup Plans already take five
fields.

- Ran robfig/cron v1.2.0 `Parse` locally from 10:00:00. `5 * * * *`
fires at 10:00:05, 10:01:05 and 10:02:05, and `0 2 * * *` at 10:02:00
and 11:02:00. With the leading `0` they fire at 10:05:00 and 11:05:00,
and daily at 02:00.
- Rendered the whole chart with helm 3.17.3 and the `configured` render
fixture. The output matches main for the defaults, for `0 2 * * * *` and
for `@daily`, and for `0 2 * * *` only the `schedule` line differs.

Unit tests with `helm unittest` 1.1.2, the version CI pins.
`packages/apps/postgres` passes 216 of 216 in 27 suites, against 209 in
26 on main. The new suite has seven cases. With the old template five of
them fail. The six-field and descriptor cases pass on both, and they
fail when the template prepends to six fields too or drops the
descriptor check.

### Screenshots

Not a UI change. The dashboard form picks up the new field description
from the schema.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff. A chart template, a new test suite, one field description and the
README, schema, Go types and cozyrd regenerated from it. No field is
added, removed or renamed and no default or enum changes. The terraform
provider does not model `backup.schedule`. The website reference page
for postgres is generated from this README on a stable tag, and the Go
types example there uses a six-field schedule, which keeps working.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(postgres): backup.schedule accepts a standard five-field cron such as 0 2 * * *, and the chart prepends a 0 seconds field before handing it to CloudNativePG, which reads the first field as seconds. Six-field values and descriptors render as before, and any other field count fails the render. A release that already sets five fields gets the standard meaning on upgrade, so 0 2 * * * moves from hourly to daily.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* PostgreSQL backup schedules support standard five-field cron
expressions as well as six-field expressions with a leading seconds
field. Five-field schedules are converted automatically; six-field
schedules and cron descriptors are used as provided.
* **Schedule Changes**
* After upgrading, `0 2 * * *` runs daily at 02:00 instead of hourly at
minute 2. To keep the previous hourly schedule, use `0 2 * * * *`.
* Invalid schedule formats produce a rendering error. Empty schedules
continue to disable chart-managed backups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 17:38:05 +03:00
Aleksei Sviridkin 8a37a70681 Merge branch 'main' into fix/mariadb-restore-dump
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 16:33:50 +03:00
Aleksei Sviridkin a4537127e7 fix(vertical-pod-autoscaler): run two admission controllers spread over nodes (#5098)
## What this PR does

The VPA admission controller runs as one replica, and its webhook is
registered with `failurePolicy: Ignore`. While that one pod is being
rescheduled, every pod created in the cluster is admitted without its
VPA recommendation, and with `updateMode: Initial` it keeps those
resources until it is recreated. On a v1.6.5 cluster
`vmstorage-shortterm-0` was created during a node drain while the
admission controller moved to another node. It has no `vpaUpdates`
annotation and runs with the vm-operator defaults, requests 250m/500Mi
and limits 1/1500Mi, while its VPA recommends 920m and 3.42Gi and the
other replica got 5.13Gi through VPA. It was OOMKilled 18 days later.

This PR sets `replicaCount: 2`, enables the chart's PodDisruptionBudget
with `minAvailable: 1` and spreads the two replicas over nodes with a
`ScheduleAnyway` topology spread, so single-node clusters still schedule
both. A drain then leaves one controller serving. The controller still
deletes and recreates its MutatingWebhookConfiguration on start
(`selfRegistration` in `pkg/admission-controller/config.go`, VPA 1.5.1),
so a restart leaves only that short gap instead of the whole reschedule.
`failurePolicy` stays `Ignore`, a VPA outage should not block pod
creation.

Two new cases in `tests/vpa_test.yaml` check the replica count, the
spread selector against the pod labels and the PDB. `helm unittest .` in
`packages/system/vertical-pod-autoscaler` passes 18 of 18 with
helm-unittest 1.1.2, and against the values on main the two new cases
fail and the other 16 pass. `helm template` renders the Deployment with
2 replicas and the PDB with the admission controller selector. I have
not run the change on a live cluster.

#4540 is the same trap from the other side, a component that keeps a bad
size because Initial never revisits it.

### Screenshots

Not a UI change.

### Downstream repositories

Values of `packages/system/vertical-pod-autoscaler` and its tests only.
No platform values, app schema, default exposed to tenants or package
change, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(vertical-pod-autoscaler): the admission controller runs two replicas spread over nodes with a PodDisruptionBudget, so pods created during a drain still get their VPA recommendation.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Improvements**
* The VPA admission controller now runs with two replicas, increasing
availability.
* Its pods are spread across hostnames when scheduling permits, helping
reduce the impact of node-level disruptions.
* A disruption budget requires at least one admission-controller pod to
remain available during voluntary disruptions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 16:32:43 +03:00
Aleksei Sviridkin 24f2b8009f feat(redis): add maxmemory, eviction policy and snapshot toggle (#5310)
## What this PR does

Lets a tenant run Redis as a pure cache without giving up the persistent
volume. Until now the chart only turned snapshots off together with the
volume (when `size` is empty), and offered no way to bound memory: a
`CONFIG SET maxmemory` is lost when the pod restarts. On a cache-only
instance, RDB snapshots and replica full syncs then filled the volume,
and each SIGKILL during a save left an orphaned `temp-*.rdb` behind, so
the instance crashlooped.

New values:

- `maxmemory` (quantity, unset by default): rendered in bytes. Unset
renders `maxmemory 0`, the Redis default (no limit).
- `maxmemoryPolicy` (enum, default `noeviction`): rendered as
`maxmemory-policy`.
- `snapshots` (bool, default `true`): `false` renders `save ""` while
keeping the volume. Snapshots stay off when `size` is empty, as before.

Every directive is rendered with an explicit value, defaults included.
The operator applies `customConfig` with `CONFIG SET` on every reconcile
and never resets a directive whose line disappears
(`service/redis/client.go` in freshworks-oss/redis-operator v3.3.5), so
leaving the defaults implicit would turn reverting a setting into a
no-op until the next pod restart. As a consequence, existing instances
gain `save 3600 1 300 100 60 10000`, `maxmemory 0` and `maxmemory-policy
noeviction` on upgrade. These equal the Redis 7 and 8 defaults, but they
overwrite any value an operator set by hand with `CONFIG SET`.

`maxmemory` is refused at render time above 90% of the Redis memory
request: above that threshold (`reservedPodMemoryPercent`, default 10)
the operator logs an error and skips the line, which would leave Redis
unbounded with nothing visible to the tenant.

Not changed here: a replica still writes a temporary RDB to disk during
a full sync (`repl-diskless-load` defaults to `disabled`), even with
`snapshots: false`. `stop-writes-on-bgsave-error` keeps its default,
since no background save runs once snapshots are off.

Tests: `tests/cache_config_test.yaml` covers the rendered defaults, the
snapshot toggle with and without a volume, byte conversion from presets
and explicit resources, and both refusals. `helm unittest
packages/apps/redis` passes (93 tests).

### Screenshots

Not a UI change.

### Downstream repositories

The new fields in `values.schema.json` reach
`cozystack/terraform-provider-cozystack`, whose redis schema and
expand/flatten pair are hand-written; no follow-up has been opened there
yet, so the box is left unticked. The website reference page is
regenerated from `README.md` by the release bot.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(redis): add `maxmemory`, `maxmemoryPolicy` and `snapshots` values so Redis can run as a pure cache (bounded memory, eviction, no RDB snapshots) while keeping its persistent volume. Existing instances now render the Redis defaults for these directives explicitly, which resets any value set by hand with `CONFIG SET`.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added configurable Redis memory limits, with a default of unlimited,
and eviction policies, defaulting to `noeviction`. Configured limits
must be whole-byte quantities and cannot exceed 90% of the pod’s memory
request.
* Added controls for scheduled Redis snapshots. Snapshots are enabled by
default when storage is configured; disabling them does not remove
existing snapshot files, and snapshots are off when storage size is
empty.
* **Documentation**
* Documented the new Redis memory and snapshot settings, including
defaults, constraints, and effects.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 16:32:31 +03:00
Aleksei Sviridkin 03a2edf3db Merge branch 'main' into feat/observability-proxy-app-scope
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 16:31:05 +03:00
Aleksei Sviridkin fe7edffa10 Merge branch 'main' into feat/observability-proxy-ingress
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-13 16:27:58 +03:00